tag:blogger.com,1999:blog-9959776.post6459609912774698289..comments2024-02-15T04:42:41.606+00:00Comments on James' Empty Blog: Reliability of the IPCC AR4 (CMIP3) ensembleJames Annanhttp://www.blogger.com/profile/04318741813895533700noreply@blogger.comBlogger35125tag:blogger.com,1999:blog-9959776.post-88186250398921976932010-01-21T07:31:47.313+00:002010-01-21T07:31:47.313+00:00Thanks!
There is certainly a much broader discuss...Thanks!<br /><br />There is certainly a much broader discussion to be had than merely focussing on the contents of this one short paper.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-35085360268419384052010-01-21T07:08:25.216+00:002010-01-21T07:08:25.216+00:00Inline comment on RC on my pointer to the Reliabil...Inline comment on RC on my pointer to the Reliability of CMIP3 Ensemble paper:<br /><br />[Response: This will be topic of the week, next week.... - gavin]<br /><br />So I guess he will discuss the IPCC Expert meeting and the paper.Deep Climatehttps://www.blogger.com/profile/07739846320319167391noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-14952449904656364902010-01-21T06:06:03.893+00:002010-01-21T06:06:03.893+00:00Thanks DC, I would suggest waiting to see what the...Thanks DC, I would suggest waiting to see what the IPCC Experts advise as "Best Practice" for the use of the ensemble. I think they may actually be aiming to complete this during the workshop next week, though that may be a stretch. Probably Gavin will want to say something about that anyway.<br /><br />Hank,<br /><br />researchpages.net is a sort of hosting site for scientists - we know the guy who runs it, through the <a href="http://researchpages.net/GENIE/" rel="nofollow">GENIE</a> project. However in practice these things tend to be just another monkey to feed, or rather, neglect :-)James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-67530631707720722722010-01-21T01:33:55.433+00:002010-01-21T01:33:55.433+00:00So, um, I followed a post of Eli's about your ...So, um, I followed a post of Eli's about your pajamas and ended up reading this:<br />http://researchpages.net/JUMP/<br /><br />How ... um ... what ... um<br />Well? What's the overlap here?Hank Robertshttps://www.blogger.com/profile/07521410755553979665noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-72489936553285449122010-01-20T16:25:18.426+00:002010-01-20T16:25:18.426+00:00For what it's worth, I left a comment at RC in...For what it's worth, I left a comment at RC in Unforced Variations, with a pointer to the paper and this post, along with the abstract.<br /><br />I added the following comment:<br /><i>Definitely merits a head post update, and I would suggest, a guest post from Annan and Hargreaves.</i>Deep Climatehttps://www.blogger.com/profile/07739846320319167391noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-17553371993128738102010-01-19T22:47:41.230+00:002010-01-19T22:47:41.230+00:00Well, putting on my cynical hat for a minute (did ...Well, putting on my cynical hat for a minute (did I ever take it off?) there does seem to be a bit of a culture in some parts of reliance on nice rhetoric rather than a sound mathematical basis. They write something that sounds nice and uncontentious, then try to translate it somehow into a mathematical form, and assume that the resulting methods and numbers therefore have some validity.<br /><br />I certainly noted this in the promotion of uniform priors, which I believe almost everyone now accepts was a mistake (I still don't know what Myles Allen thinks).<br /><br />The "<a href="http://julesandjames.blogspot.com/2008/09/how-not-to-combine-multiple-constraints.html" rel="nofollow">averaging the constraints</a>" thing is yet another example, from members of the same clique. Sounds good in words, but has absolutely no mathematical foundation and in fact directly contradicts the axioms of probability.<br /><br />You might find it hard to believe, but at one point they even wrote that where their intuition ran contrary to the axioms of probability, they would rather use their intuition!<br /><br />However, returning to the point of this post, I note that the paper is now published (post is updated with the link at the top) and I will wait to hear how the "experts" react to it at their meeting next week.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-52372399333297323992010-01-19T18:49:18.523+00:002010-01-19T18:49:18.523+00:00James -
Thanks for this writeup and for answering...James -<br /><br />Thanks for this writeup and for answering my question. I don't think that your answer gets to the heart of the matter, though.<br /><br />I have very minimal training in statistics, yet I was able to detect this flawed approach as I read about ensemble averaging. How can it be that specialists with strings of letters after their names, who spend their lives doing this stuff, could have pursued this line of reasoning?<br /><br />I think there is something else going on here, namely thinking that a model output has the same status of a measurement vis-a-vis the truth. How else can people talk abut "random errors" in a model? Measurements have random errors, models are "always wrong but sometimes useful". <br /><br />It's probably not an accident that the meterologists, who have dealt with these concepts for decades, are more cautious about the model outputs.Tom Chttps://www.blogger.com/profile/03793192912187740419noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-81845844425149080892010-01-19T06:52:11.453+00:002010-01-19T06:52:11.453+00:00Hank,
I'll see your GISS averaging nonsense, a...Hank,<br />I'll see your GISS averaging nonsense, and raise you with the latest from the National Post's Lawrence Solomon - "Google censors climategate search results". I kid you not.<br /><br />http://deepclimate.org/2010/01/19/national-posts-lawrence-solomon-claims-google-censors-search-results/Deep Climatehttps://www.blogger.com/profile/07739846320319167391noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-67209276096532711132010-01-19T06:15:15.149+00:002010-01-19T06:15:15.149+00:00PS, a cross-reference for anyone who finds this la...PS, a cross-reference for anyone who finds this later and doesn't recognize what it's about, the bunk I mentioned is debunked in a paragraph here:<br />http://www.realclimate.org/index.php/archives/2010/01/unforced-variations-2/<br /><br />The relevant part begins:<br />"Update: Some comments on the John Coleman/KUSI/Joe D’Aleo/E. M. Smith accusations about the temperature records. Their claim is apparently that coastal station absolute temperatures are being used to estimate the current absolute temperatures in mountain regions and that the anomalies there are warm because the coast is warmer than the mountain. This is simply wrong. What is actually done is that temperature anomalies are calculated locally from local baselines, and these anomalies can be interpolated over quite large distances. This is perfectly fine and checkable...."Hank Robertshttps://www.blogger.com/profile/07521410755553979665noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-24216985028193861762010-01-19T06:11:37.047+00:002010-01-19T06:11:37.047+00:00I, unfortunately, went and read the blog mentioned...I, unfortunately, went and read the blog mentioned in the presser. My favorite gripe: "why does GISS throw away data before 1880?!". <br /><br />I hope I don't have to answer that question for anyone reading here.Rattus Norvegicushttps://www.blogger.com/profile/03449457204330125792noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-82184646170100838822010-01-19T03:32:20.204+00:002010-01-19T03:32:20.204+00:00Hank,
That is just too funny...I'd expect to ...Hank,<br /><br />That is just too funny...I'd expect to see that on the Onion or Daily Mash!James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-45451817592063562862010-01-19T02:07:43.629+00:002010-01-19T02:07:43.629+00:00You'll be shocked to hear that NASA has been c...You'll be shocked to hear that NASA has been caught averaging numbers; this arithmetrickery is apparently being explained in a full hour of television programming. I wonder who's sponsoring this one.<br /><br />http://www.prnewswire.com/news-releases/nasa-caught-in-climate-data-manipulation-new-revelations-headlined-on-kusi-tv-climate-special-81507392.htmlHank Robertshttps://www.blogger.com/profile/07521410755553979665noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-63927688087881283122010-01-19T02:06:28.236+00:002010-01-19T02:06:28.236+00:00Roger,
Thanks. Of course with 23 models, even in ...Roger,<br /><br />Thanks. Of course with 23 models, even in the best case you will still see reality outside the model range one time in 12, which means a lot of these cases if you are looking at a lot of predictands - and it's well known that in short-term weather prediction mode, models tend to underestimate extreme cases. However multimodel ensembles are generally far better than single model (initial condition) ensembles in that respect.<br /><br />Guthrie, I don't think any denialist will read the paper, still less understand it if they do. Besides, the main message is that the models are <i>more</i> reliable than others have argued, which is hardly a point they will want to promote...<br /><br />(and, what Chris says)James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-78462411630779823552010-01-18T23:32:15.059+00:002010-01-18T23:32:15.059+00:00I noted that the 2.7 to 3.4 simply got called absu...I noted that the 2.7 to 3.4 simply got called absurd and completely horrible. A range of 0.7C - could we be that sure? Obviously not even James thinks the range could be that narrow.<br /><br />"Is it going to be 2.1 or 4.4? Either doesn't sound nice"<br /><br />Remember this is climate sensitivity which is not directly observable/feelable. Even if CO2 does get to 560ppm it would be some time before full effects are felt. So probably not in our lifetime unless we get a big methane burp. Maybe 'either doesn't sound nice for our children' would be better?<br /><br />In the absense of a methane burp, which end of the temp rise we get depends on policy as well as uncertainty in climate sensitivity. So AFAICS although transitory rate of change is more certain than climate sensitivity, the uncertainty of policy reintroduces the uncertainty. However if effects are severe, policy will likely curtail the worse end scenarios.crandleshttps://www.blogger.com/profile/15181530527401007161noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-51952051373269498962010-01-18T22:25:27.357+00:002010-01-18T22:25:27.357+00:00I'm afraid that going from 2.7 to 3.4 to 2.1 t...I'm afraid that going from 2.7 to 3.4 to 2.1 to 4.4C somehow doesn't make any difference to me, being a simple member of the public. Is it going to be 2.1 or 4.4? Either doesn't sound nice given the change a mere 0.5C makes to the climate I was used to. <br /><br />How long until we see this crop up in the denialosphere? <br />(Not that they should get too much attention)guthriehttps://www.blogger.com/profile/17992984293423290387noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-12157660313211773082010-01-18T17:36:12.773+00:002010-01-18T17:36:12.773+00:00Of course, I meant:
The y-axis is percentage of b...Of course, I meant:<br /><br /><i>The <b>y-axis</b> is percentage of binned observations. 23 models = 24 bins = <b>1.67</b> each on average.</i><br /><br />OK, there are 24 models = 25 bins (not 23 models). Normalizing to 40 d.f. gives an average score of 1.6. I'll get there.<br /><br />I see that 0.5C is not a huge bias, given the spread. But the direction of the bias does argue against the meme put forth in some quarters that climate models have a warm bias in general.Deep Climatehttps://www.blogger.com/profile/07739846320319167391noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-47542330592571873252010-01-18T17:06:41.847+00:002010-01-18T17:06:41.847+00:00This comment has been removed by the author.Deep Climatehttps://www.blogger.com/profile/07739846320319167391noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-36133428783884808962010-01-18T12:42:57.854+00:002010-01-18T12:42:57.854+00:00James,
glad you've written this. All the pape...James,<br /><br />glad you've written this. All the papers/reports I have done using model-derived data in risk analyses since 1999 have used the assumption that the truth is more likely than not to be somewhere in there, but not truth centred.<br />By the time various uncertainties are combined, uniform or rather flat beta distributions of a rather wide scatter are fine. Actually, if you do this for regional climate, as we have found out in SE Australia, climate variability <b>not</b> represented in the consensus model set and probably not in any of the individual runs, can deliver a climate not in the sample. <br /><br />Also, when climate policy uncertainties are added, the precision afforded by erroneous (hubristic?) assumptions becomes even less relevant. In most cases, policy does not need such high precision. <br /><br />The application of probabilities needs to be mindful of what information is most needed to manage risk.Roger Joneshttps://www.blogger.com/profile/13558413693023988451noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-601285506957388092010-01-17T11:05:30.033+00:002010-01-17T11:05:30.033+00:00Ian,
Theoretically, it's just as I said, as l...Ian,<br /><br />Theoretically, it's just as I said, as long as our uncertainties are honestly reflected in the range of models produced, we should be ok. Many have argued that this is not the case and that the model range is too small, but I don't believe the evidence supports them.<br /><br />Beyond the rank histogram, there are a range of other ways of investigating whether what we can observe of reality is interchangeable with the models, and we are playing around with some of them right now. I'm expecting to write a longer paper significantly extending this work in the next month or two (or four, you know how it goes).<br /><br />Martin,<br /><br />As independent samples their average correlation is necessarily zero. I think you may be confusing this with the correlation of x and y (labelling the diagram in the obvious way).<br /><br />DC, yes the obs are a bit hot, ie the models are cool. It's only about 0.5C on average though. We normalised the graphs to sum to 40 for reasons discussed in the paper - there are really 72x36 data points (5 degree grid) but with substantial spatial correlation.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-9501827425310938992010-01-16T21:13:32.652+00:002010-01-16T21:13:32.652+00:00Second time around, it all made more sense to me.
...Second time around, it all made more sense to me.<br /><br />The x-axis is percentage of binned observations. 23 models = 24 bins = 1.67% each on average.<br /><br />Wrt surface temps the model ensemble looks to have a bit of a cool bias though, right?<br /><br />Sorry about posting the same rank histogram explanation you did. In my defence, the links don't stand out very much, so honestly I just missed it.Deep Climatehttps://www.blogger.com/profile/07739846320319167391noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-51935131394661090552010-01-16T10:33:47.158+00:002010-01-16T10:33:47.158+00:00James, thanks!
So, are (m_i - E(O)) and (m_j - E(...James, thanks!<br /><br />So, are (m_i - E(O)) and (m_j - E(O)) correlated? Your second pic seems to suggest so (ellipses). Or is this unintentional?Martin Vermeerhttps://www.blogger.com/profile/04537045395760606324noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-51233073381349698172010-01-16T06:24:59.819+00:002010-01-16T06:24:59.819+00:00Tutorial on rank histogram (using simple weather p...Tutorial on rank histogram (using simple weather prediction example):<br /><br />http://satreponline.org/vesa/verif/www/english/msg/ver_prob_forec/uos4b/uos4b_ko1.htmDeep Climatehttps://www.blogger.com/profile/07739846320319167391noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-75017955282220414352010-01-16T05:51:31.320+00:002010-01-16T05:51:31.320+00:00Very interesting! I hadn't thought about this...Very interesting! I hadn't thought about this before, but your description above makes sense. So, hypothetically, under what conditions would you have some confidence that models (climate-related or not) really are randomly scattered in some model-space, allowing inferences from an ensemble mean?Ianhttps://www.blogger.com/profile/11298186803453096185noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-49065966396666784292010-01-16T05:35:06.209+00:002010-01-16T05:35:06.209+00:00If the effect is systematic looking at climate mod...If the effect is systematic looking at climate models is a bad tactic because there is only one realization. How about you do the same thing with local weather models. Lots of models, lots of realizations.EliRabetthttps://www.blogger.com/profile/07957002964638398767noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-73589065456035412992010-01-16T00:30:07.886+00:002010-01-16T00:30:07.886+00:00Tom C and Chris, I firstly just mean the same sent...Tom C and Chris, I firstly just mean the same sentence was a direct quote from the previous post :-) However, there does seem to be something more systematic going on.<br /><br />In both this case and the previous one involving uniform priors, we seem to have climate scientists building some edifice around some intuitively appealing rhetoric which rapidly falls apart under critical analysis. In this case, I tried to trace back the origins, and say in the paper:<br /><br /><i>This truth-centred paradigm appears to have arisen as a post-hoc interpretation of the ad-hoc weighting procedure known as “Reliabilty Ensemble Averaging” or REA</i><br /><br />It wouldn't be fair to exclude myself entirely from criticism, as (again, in both cases) I was aware of what people were saying, and even accepted it myself, before getting directly involved. OTOH it is worrying how quickly and easily we were able to knock holes in theories that many influential people had been using and accepting for some years.<br /><br />It is not difficult to conclude that the IPCC process has played a part here, in focussing power in private committees and favouring consensus-forming over debate. Some people appear to be rewarded more for echoing the majority view rather than for actually coming up with anything new.<br /><br />Wikipedia has an interesting page on <a href="http://en.wikipedia.org/wiki/Groupthink" rel="nofollow">groupthink</a> which seems highly relevant.<br /><br />Hence my comments in the last paragraph of the post.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.com