Time for a quick round-up of "what we know that just ain't so".
I noticed a letter in Physics Today pointing out an error in the Bryden et al paper that I spotted back in 2005:
Another story that you might remember from a year or so ago is that shocking news that plants were pumping out loads of methane. Well, apparently they don't really. As Carl Zimmer notes, this contradiction isn't quite attracting the same coverage as the original claim (to put it mildly). Of course the original authors may now produce some more evidence to support their claim, but I'm not holding my breath.
Saving the best to last, an analysis of the ARGO data published in GRL last year suggested that the oceans had cooled sharply over the last couple of years - to an extent that appeared to cast some doubt on the validity of climate models. Roger Pielke Snr has been pushing this particular peanut tirelessly since before the paper was even published, so the correction to the data has left him with plenty of egg on his face. In the latest episode, he's got a transcript of an interview with Marcel Crok on his web site, which he's had to correct:
The moral of this story? Don't uncritically believe any single paper that gives a surprising result, because there's a fair chance it will turn out to be flawed, and perhaps completely wrong. Be especially sceptical of papers that you particularly wish to be true, because you are probably kidding yourself as to their credibility.
It is interesting to speculate on what a betting market would have made of these papers. I strongly suspect that claims written based on the main results of each of these papers would have rapidly settled at prices indicating a rather low probability that they were valid (I certainly got the impression that there was significant scepticism of these results in the scientific community, which even extended to the authors of the papers in some cases). Wouldn't it have been useful to have that information in the public domain?
I noticed a letter in Physics Today pointing out an error in the Bryden et al paper that I spotted back in 2005:
According to Bryden and coauthors, the 1957 transport in a layer shallower than 1000 m was 22.9 ± 6 Sverdrups (1 Sv = 106 m3/s) compared with the transport of 14.8 ± 6 Sv in 2004. The ± 6 Sv represents an uncorrelated error of each measurement. Bryden subtracts the two quantities and presents the results as 8.1 ± 6 Sv (instead of 8.1 ± 12 Sv or ± 8.5 Sv, depending on the character of errors), which is an incorrect result. It is a mystery how such an error was missed by Levi and by the editors and reviewers of the original paper.IMO one could reasonably add the authors to that list. In fairness to Bryden, the letter-writer also mentioned that the submitted manuscript had a question mark at the end of its title, (ie Slowing of the Atlantic meridional overturning circulation at 25° N ?) which the Nature editors insisted on removing. When White House staff spuriously add uncertainty there's no shortage of criticism, but apparently no-one raises an eyebrow when Nature editors do the reverse. [There are also more fundamental problems with the result, in that the variability is so high that snapshots tell us little about any trend.]
Another story that you might remember from a year or so ago is that shocking news that plants were pumping out loads of methane. Well, apparently they don't really. As Carl Zimmer notes, this contradiction isn't quite attracting the same coverage as the original claim (to put it mildly). Of course the original authors may now produce some more evidence to support their claim, but I'm not holding my breath.
Saving the best to last, an analysis of the ARGO data published in GRL last year suggested that the oceans had cooled sharply over the last couple of years - to an extent that appeared to cast some doubt on the validity of climate models. Roger Pielke Snr has been pushing this particular peanut tirelessly since before the paper was even published, so the correction to the data has left him with plenty of egg on his face. In the latest episode, he's got a transcript of an interview with Marcel Crok on his web site, which he's had to correct:
Over the last 50 years there is a warming trend in the oceans. However between 2003 and 2005 some 20 percent of the accumulated heat was lost. The key paper which published these results was Lyman et al. 2006. This means there was radiative cooling of the earth climate system in those years. [As readers of Climate Science know, this conclusion has been corrected; there is not evidence of any loss of heat over this time period; see].I hope that Marcel Crok has managed to correct his article too :-)
The moral of this story? Don't uncritically believe any single paper that gives a surprising result, because there's a fair chance it will turn out to be flawed, and perhaps completely wrong. Be especially sceptical of papers that you particularly wish to be true, because you are probably kidding yourself as to their credibility.
It is interesting to speculate on what a betting market would have made of these papers. I strongly suspect that claims written based on the main results of each of these papers would have rapidly settled at prices indicating a rather low probability that they were valid (I certainly got the impression that there was significant scepticism of these results in the scientific community, which even extended to the authors of the papers in some cases). Wouldn't it have been useful to have that information in the public domain?
3 comments:
It was in the public domain. The RealClimate postings in each of those instances indicated that the results were preliminary, contradictory to other work and would need very strong additional support before being widely accepted. This is same message that I gave to journalists who interviewed me on the subject.
However, the 'methane from plants' press debacle, was in no small measure due to the scientists' original press release, the other two stories could have been better handled though.
What more do you think should have been done?
I'm trying to figure out how one could test whether a paper is "right" for the purposes of a betting market.
My best guess is to compare number of citations with a directly contradictory paper, starting after enough time has passed for the dust to settle and a consensus to emerge.
What more do you think should have been done?
There's nothing wrong with publishing an interesting and unusual analysis of new data, but I think scientists could sometimes benefit from taking a slightly more detached view of their results. IIRC Feynman had something to say about pointing out the potential flaws in your own work. They could also be less willing to collude in the overselling, whether this be through Nature editors sexing it up or press officers misrepresenting it in their press releases. But even with perfectly honest reporting, mistakes will still be made and ultimately perhaps the last line of defence is simply for the reader to remember to not believe everything they read in the papers, even peer-reviewed ones let alone the newspapers! The main worry I have is that the more wrong stuff gets hyped, the harder it will be to argue that the science really is settled on the main questions...
I think it is very plausible that a betting market would have give a better summary of the honest opinions of informed scientists compared to the partial summaries provided by people all of who have their own personal axes to grind.
Brian, I agree that actually fixing a specific verifiable claiim isn't always easy. However, in discussing the Bryden result, the Guardian made some specific comment about substantial future cooling over the UK. Such an event would be easily testable and it would have been IMO interesting to find out if anyone would assign a non-negligible probability to such an event.
Robin Hanson suggests that in general, bets could be decided by a panel of experts (paid from the pool of bets). He argues that in practice this mechanism would usually not need to be invoked given a reasonably liquid market, as people would choose to cash in their bets for whatever they could get rather than throw good money after bad on a lost cause.
Post a Comment