Eventually, after 9 1/2 weeks, we got the reviews back from GRL for our revised "Comment on Frame..." (see here and here for previous).
To our surprise, the Editor did not return our manuscript to the original referees to check that we'd dealt with their criticisms, but instead seems to have found 2 new ones. As happened last time, one reviewer clearly recommends publication of this "useful" discussion which will be "good to bring out into the published literature". However, the other one recommends rejection, on the grounds that he cannot see anything confusing or wrong in the original Frame et al paper (F05)! He views the entire debate entirely as our misunderstanding of their work, explicitly describing them as having presented merely a "subjective opinion", but a quick glance at the paper itself shows that the term "subjective" is only used once, in the abstract, as a derogatory term regarding the work of others - and they regard this subjectivity as a problem which they explicitly claimed to have resolved. [I'd be interested to hear whether others honestly think that we have misrepresented or misinterpreted F05's work. In particular, does anyone disagree with our reading that F05 are claiming that the use of a uniform prior is the correct choice when generating estimates of climate sensitivity (inter alia)? If any other readers besides ourselves draw that conclusion, then Houston, we have a problem, and at the very least clarification is urgently required.]
So now GRL have a total of 4 independent reviews of essentially the same manuscript. One thinks the original F05 has such fundamental flaws that the best course of action is not to draw attention to it (as we mentioned in our reply to him, but he will not have seen, this seems a vain hope given the paper's citation by others including the AR4 draft, and the authors' own repetition of the same comments in a forthcoming book chapter). Another referee cannot see any problem with the original paper, but it seems hard to reconcile their view with what F05 actually wrote. And 2 more reviewers clearly recommend publication of what they see as a useful discussion of an important issue.
Given the diversity and strength of opinions presented, and the significance of the issue, I find it very hard to understand how GRL can consider it in the interests of their readership or indeed climate science that this discussion should not appear at all. I would have thought that this is precisely the sort of situation that a comment-and-reply system exists for. I have written to the Editor saying as much.
Based on the new reviewer's comment about the "subjective opinion" of F05, there may be a bit of a nudge-nudge-wink-wink let's all walk away whistling and pretend it never happened going on behind the scenes. But if there is, it's not a discussion that I've been party to, and Allen and Frame (as it now seems to be) appear unrepentant in their own review of our Comment. I don't particularly blame them for playing the game in the way they have, but if they had the courage of their convictions and were prepared to stand behind (or even merely clarify what they meant by) the original paper, you might think they would welcome the chance to publish their own rebuttal alongside our comment, rather than ask the Editor to suppress the whole matter.
Recommended further reading: Chuck Doswell on peer review (especially the discussion of Comments and Replies, towards the bottom of the page, and Ronald Errico's "On the Lack of Accountability in Meteorological Research". I'll finish with Errico's 4th recommendation:
To our surprise, the Editor did not return our manuscript to the original referees to check that we'd dealt with their criticisms, but instead seems to have found 2 new ones. As happened last time, one reviewer clearly recommends publication of this "useful" discussion which will be "good to bring out into the published literature". However, the other one recommends rejection, on the grounds that he cannot see anything confusing or wrong in the original Frame et al paper (F05)! He views the entire debate entirely as our misunderstanding of their work, explicitly describing them as having presented merely a "subjective opinion", but a quick glance at the paper itself shows that the term "subjective" is only used once, in the abstract, as a derogatory term regarding the work of others - and they regard this subjectivity as a problem which they explicitly claimed to have resolved. [I'd be interested to hear whether others honestly think that we have misrepresented or misinterpreted F05's work. In particular, does anyone disagree with our reading that F05 are claiming that the use of a uniform prior is the correct choice when generating estimates of climate sensitivity (inter alia)? If any other readers besides ourselves draw that conclusion, then Houston, we have a problem, and at the very least clarification is urgently required.]
So now GRL have a total of 4 independent reviews of essentially the same manuscript. One thinks the original F05 has such fundamental flaws that the best course of action is not to draw attention to it (as we mentioned in our reply to him, but he will not have seen, this seems a vain hope given the paper's citation by others including the AR4 draft, and the authors' own repetition of the same comments in a forthcoming book chapter). Another referee cannot see any problem with the original paper, but it seems hard to reconcile their view with what F05 actually wrote. And 2 more reviewers clearly recommend publication of what they see as a useful discussion of an important issue.
Given the diversity and strength of opinions presented, and the significance of the issue, I find it very hard to understand how GRL can consider it in the interests of their readership or indeed climate science that this discussion should not appear at all. I would have thought that this is precisely the sort of situation that a comment-and-reply system exists for. I have written to the Editor saying as much.
Based on the new reviewer's comment about the "subjective opinion" of F05, there may be a bit of a nudge-nudge-wink-wink let's all walk away whistling and pretend it never happened going on behind the scenes. But if there is, it's not a discussion that I've been party to, and Allen and Frame (as it now seems to be) appear unrepentant in their own review of our Comment. I don't particularly blame them for playing the game in the way they have, but if they had the courage of their convictions and were prepared to stand behind (or even merely clarify what they meant by) the original paper, you might think they would welcome the chance to publish their own rebuttal alongside our comment, rather than ask the Editor to suppress the whole matter.
Recommended further reading: Chuck Doswell on peer review (especially the discussion of Comments and Replies, towards the bottom of the page, and Ronald Errico's "On the Lack of Accountability in Meteorological Research". I'll finish with Errico's 4th recommendation:
Encourage publication of scientific criticisms. It takes courage and hard work to write a good scientific criticism, and this is in part what science is about. Such works should not be termed “negative,” but instead should be rewarded if they are sound.
12 comments:
So... will you be publishing it elsewhere instead?
Re the Errico piece: interesting, but without specific examples rather frustrating.
Re: other publications. Not sure it's worth the candle. I'll try to do some other things before wasting any more time on it, especially if people are prepared to claim it is all due to a misunderstanding and row back in a face-saving manner. But if people keep on producing pathological pdfs then I'll certainly have more to say at some time...
Errico: true, but he's not easily dismissed as a crank or embittered.
you're something like 0-for-10 in bashing Allen & Frame; is it possible that you are wrong or misguided? Or will your ego not let you admit any deviation from your quixotic Bayesian quest?
Anon,
I really don't think there is any room for debate over the technical details, which are quite trivial. Eg, the method of F05 does not generate probabilities, under any standard definition of what probability is. Moreover, a uniform prior (between two specified limits, the choice of which still needs to be made in any case) represents a very specific set of beliefs, which cannot credibly be described as mere ignorance.
The only real issue seems to be whether we really did misinterpret what was written, as one ref asserts - and if so, were we alone, and is it not worth clarification? Perhaps it ultimately doesn't really matter, so long as they (and others who are aware of our Comment) don't re-assert the same errors. I guess a polite withdrawal allows everyone to save face. But there's a rather strange bit in their review which suggests they don't understand our first point (about coherence) at all, so it's not really clear that they accept our argument.
As for "Bayesian quest", one point that I hope all protagonists agree on is that it is essentially a question of Bayesian probability.
Corrections and comments are a queer kettle of fish. Editors hate them and will only publish when pushed to the wall. There appears to be an editor located in Yokohama, are you submitting through him? If not a telephone call might be useful.
Again, the "rabid necessity" and "push to the wall" of these "criticisms" is only in the minds of Annan & Hargreaves. Perhaps it's time to get back to doing original research? Or perhaps join the McIntyre & McKitrick camp --- just bitch about others, do nothing original --- and whine about "Nature."
Anon,
So far only 1 out of 4 referees agrees with your characterisation of the matter. What is the source of your special insight that over-rules their judgement? Do you really not think that F05 were presenting "the solution" which "resolved" the the problem of how to choose a prior? If not, what did they mean by those terms? What do you think a reasonable reader is going to conclude?
If it was as clear-cut as you claim, then don't you think Allen and Frame would be happy to let us publish and make ourselves look foolish in public?
I've not yet completely give up hope on GRL, actually. My impression is that the Editor merely looks at the scores given and makes an automatic assessment without much real consideration. In this latest case, one ref gave a top mark of 1 for science, but the other gave no grade. However he has already shown himself of being capable of listening to reason (the first version got exactly the same treatment, but a reasonable email changed his mind). So I'll definitely give him a chance to reconsider before doing anything else.
Give us a break, Anon. A&H do plenty of original work. Comparing them to M&M is both inaccurate and nasty.
Is that so, Steve? Yeah, the Annan & Hargreaves "original work" is so great that in a new paper from M. Crucifix, the holes are exposed for all to see -- namely the poor data/model they use, the bad analysis & weak conclusions. And Julie helped herself & her husband get hoist by their own petard, haha!
It's funny how "cock sure" James continually is over things that have such high error, and for things on the low probability side. Pretty poor career choice for the charmed couple methinks, then again, that's why they're stuck out in Japan with no recovery...
Oh, is Michel's paper out yet? I don't think it has much bearing on this discussion, but feel free to expand...once you've had a coffee, or a valium perhaps.
Regarding the career choice, if I'm as poor a scientist as you seem to think, then having any job at all is quite an achievement :-) I'm happy with my decision to come here. There is some bad with the good, of course. But I'm certainly not in the job market.
No Valium required -- MIROC stinks, your analysis is feeble, your paleo/LGM constraints are laughable:
"(MIROC3.2) has the smallest LGM cooling over Antarctica (even when the effect of the ice sheet elevation change, which is not exactly the same in all models due to their different resolution, is accounted for). Likewise, MIROC3.2 is one of the two models showing the smallest tropical cooling at the LGM, while it is one of the two models showing the largest temperature increase in this area in response to CO2 doubling. It is therefore inappropriate to simply scale an observational estimate of global LGM temperature to predict
climate sensitivity.
...
Therefore, climate sensitivity cannot be directly estimated from the Last Glacial Maximum global temperature."
"Therefore, climate sensitivity cannot be directly estimated from the Last Glacial Maximum global temperature."
Well, we certainly haven't claimed otherwise. But I wonder if you dare to write snarky anonymous emails to Ganapolski and Rahmstorf and their co-authors others too?
Post a Comment