Wednesday, May 02, 2007

More on Schneider v Hegerl et al

I recently had some brief correspondence with Tapio Schneider, who had found my blog comments on the comment and reply in Nature. So prompted, I went and had a more careful look at what he wrote.

It seems very clear that his main criticism is correct - at least, based on what Hegerl et al said they had done in their paper and the supplementary information they supplied at the time. In brief (although the various manuscripts themselves are brief enough to read in full for those who have access) Hegerl et al used a regression to estimate past temperatures anomalies as a function of proxy data, and estimated the uncertainty in reconstructed temperature as being entirely due to the uncertainty in the regression coefficient. The problem with this manifests itself most clearly when the tree ring anomaly is zero, as in this event the uncertainty in the reconstructed temperature is also zero! Steve McIntyre's plot of the data that Hegerl et al supplied as supplementary information illustrates the effect neatly (and the problem is quickly diagnosed in the comments following).

Shortly before the publication of the comment and reply (but well after they were accepted for publication), the supplementary info was changed. There is now a file giving the reconstruction back to 1500 with new confidence intervals, which no longer vanish or swap over. This new data doesn't match the description of their method, or the results they plotted in their Fig 1 (which is almost surely a smoothed version of the original supplementary data).

Here are their Fig 1 and a quick plot of the new supplementary data. I've added lines at 0, 0.2 and 0.4 to aid visual comparison. Their fig is smoothed and the data are vertically offset by some unknown amount, so I just did my best with the supplementary to make a decent match. Spot the difference especially around 1570 and 1750-1800, where their old confidence interval seems to vanish and is certainly nowhere near filling the 0-0.4 range. The width of new confidence interval has a lower bound of about 0.45.





I won't speculate on what Hegerl et al actually did in their calculations but Schneider's criticism seems entirely reasonable to me, based on the information that was available.

I stand by my comment that their choice of a uniform prior is a bigger problem though :-)

7 comments:

Steve McIntyre said...

"I won't speculate on what Hegerl et al actually did in their calculations."

I'd be interested in your speculations. I re-read their materials and it makes less sense than ever. Maybe it would be worth your inquiring, you're likely to have more luck in finding out than me. It would save much decoding if they would just provide code. Maybe a statistical reference as well.

BTW if you're trying to decode confidence intervals, have you tried figuring out how MBH99 confidence intervals are calculated? MBH98 is from calibration period residuals, but MBH99 is different. There's no statistical reference and it's another mystery. It's stumped a couple of statistics post-docs and me at climateaudit.

James Annan said...

I'd be interested in your speculations.

Well, my primary interest is mainly in going from observational evidence (and physical theory as expressed in models) to predictions of future climate change, and not in the derivation and analysis of the observational evidence itself. So I haven't looked into the data in any great detail, and in fact I've never used any of the last 1000 years tree-ring stuff at all. However, there's a fair chance I will be moving in that direction over the next year or two (but it won't happen in a hurry).

Whether or not Hegerl et al underestimated the uncertainty at some points in their reconstruction might not have a huge effect on their final result for climate sensitivity, since uncertainty in the forcing and ocean heat uptake means there would be uncertainty in climate sensitivity even with a perfectly known temperature record. It's unclear how much difference would be made by adding uncertainty in the record (but it would be worth their checking: AIUI Schneider spent some time trying and failing to get this out of them). Of course in terms of the temperature reconstruction itself a zero width confidence interval is nonsense.

EliRabett said...

I think the same mistake was made by Huang Pollack and Shen, and certainly shows in Monckton's version of their long term reconstruction. This appears to be a common error.

Steve McIntyre said...

It's very annoying that they won't say how they got their results, but, as we know, this isn't the first such instance in this field.

This error in confidence interval estimation is not a "common error" outside of climate science. However, Eli, as you say, it has occurred on more than one occasion in recent climate science and it would be worthwhile pinning down exactly how the error is occurring.

In econometrics, people are obliged to archive working code in their articles so these silly sort of guessing games and trying to winkle details out of authors are unnecessary. This is merely one more unfortunate example of bad behavior in climate science - it shouldn't be up to me to be the only one to condemn this sort of stuff.

Heiko said...

Is that criticism by Steve McIntyre for all of climate science, or just for temperature reconstructions?

stephen.mcintyre said...

One of the climateaudit readers has a very plausible explanation of the error in the original Hegerl et al confidence interval calculations. See http://www.climateaudit.org/?p=1531#comment-108174

James Annan said...

Yes, that looks right to me.

However, I must correct the "deliberate error" I made in my previous comment. Zero-width confidence intervals are not necessarily wrong. They are rather unconventional, perhaps, but that doesn't make them incorrect.