After about 12 weeks (is that a record?), I eventually managed to wring a reply out of GRL (with reference to our Comment on Frame et al).
Given the lengthy delay, I suppose the decision (rejection) was not that much of a surprise, but the reviews that it was based on leave us feeling like we're somewhere in the twilight zone. The Editor didn't actually explain his decision at all, but merely enclosed the reviews. Maybe it is supposed to be obvious, but we are rather baffled by it all.
There are a total of 3 reviews - itself something of a surprise, as AGU policy indicates that generally only 2 (including the original author) will be used to review a Comment and Reply. Perhaps the 3rd reviewer was some sort of tie-breaker - this would help to explain the delay, but in that case the decision is even more surprising, since Reviewer 3 (a named climate scientist working in this sub-field) strongly recommends publication of this "highly valuable contribution".
Reviewer 1 is Dave Frame, and his review was pretty much as expected. I don't have any major gripes with that, athough perhaps if he was more confident of his ground he could have actually recommended publication rather than not. I'm unconvinced by his promises to sort it all out in forthcoming publications, as he has not conceded on some of the (IMO unarguable) points that we have made.
That leaves the enigmatic Reviewer 2. He spends a full 3 pages on a diatribe about the faults of the original Frame et al paper, a further page criticising their Reply as inadequate and evasive, and only devotes a single page to discussion of our Comment, the gist of which is that we didn't criticise things in quite they way he would have done himself had he bothered to. His summary recommendation against publication is essentially on the grounds that it would be best not to draw attention to the original Frame et al paper because it is so bad!
Unfortunately, the genie is well out of the bottle on that point, with the paper already cited a number of times, including extensively by the IPCC AR4 drafts (I'll probably get my head chopped off for daring to say that). We consider that, given the impact of the paper, it is irresponsible of GRL to not allow our criticism of it to be published - or at least give us a chance to improve our Comment in the light of the reviews received. But regardless, we now feel like we are sitting in the twilight zone, where criticism of Hegerl et al is disallowed specifically because many others have made similar mistakes, and criticism of Frame et al is apparently disallowed because the original paper is so bad it should never have been published. (We should make it clear that we find R2's comments to be rather harsh in this respect - we sense a bit of a power struggle over who should be considered an authority in this field. IOO Frame et al did represent some progress on what had gone before, but also took a step or two backwards in some respects.)
No doubt we will eventually be able to publish our ideas in a stand-alone paper, but it will take time and effort that could have been more fruitfully expended elsewhere. Part of the reason we haven't already done this is the suspicion that some might say our criticism is too trivial to publish, unless we justify it by tying it directly to the existing literature. And if we are going to name and criticise specific authors, it is only reasonable to give the original authors the chance to respond directly in Comment-and-Reply format. But it seems that this is no longer an option. Meanwhile, climate scientists (including, but not limited to, the IPCC authors) press on regardless, blissfully unaware of the problems in what they are doing. They may even feel more strongly justified now in what they are doing, having successfully shaken off some attacks :-(
On the positive side, one more climate scientist (R3) seems to be pretty much on-board and on-message (I only used these buzzwords to annoy jules who hates them). On top of one more who we've had some recent email correspondence with, there are some signs that the tide will start to turn eventually. Jules and I have been fortunate enough to get invited to a workshop in the UK later this month on uncertainty and probability in climate science. I'm sure these issues will get a good airing there - unfortunately, it coincides with an IPCC lead author meeting, so some of those who need to hear it most urgently will not be able to do so!
Given the lengthy delay, I suppose the decision (rejection) was not that much of a surprise, but the reviews that it was based on leave us feeling like we're somewhere in the twilight zone. The Editor didn't actually explain his decision at all, but merely enclosed the reviews. Maybe it is supposed to be obvious, but we are rather baffled by it all.
There are a total of 3 reviews - itself something of a surprise, as AGU policy indicates that generally only 2 (including the original author) will be used to review a Comment and Reply. Perhaps the 3rd reviewer was some sort of tie-breaker - this would help to explain the delay, but in that case the decision is even more surprising, since Reviewer 3 (a named climate scientist working in this sub-field) strongly recommends publication of this "highly valuable contribution".
Reviewer 1 is Dave Frame, and his review was pretty much as expected. I don't have any major gripes with that, athough perhaps if he was more confident of his ground he could have actually recommended publication rather than not. I'm unconvinced by his promises to sort it all out in forthcoming publications, as he has not conceded on some of the (IMO unarguable) points that we have made.
That leaves the enigmatic Reviewer 2. He spends a full 3 pages on a diatribe about the faults of the original Frame et al paper, a further page criticising their Reply as inadequate and evasive, and only devotes a single page to discussion of our Comment, the gist of which is that we didn't criticise things in quite they way he would have done himself had he bothered to. His summary recommendation against publication is essentially on the grounds that it would be best not to draw attention to the original Frame et al paper because it is so bad!
Unfortunately, the genie is well out of the bottle on that point, with the paper already cited a number of times, including extensively by the IPCC AR4 drafts (I'll probably get my head chopped off for daring to say that). We consider that, given the impact of the paper, it is irresponsible of GRL to not allow our criticism of it to be published - or at least give us a chance to improve our Comment in the light of the reviews received. But regardless, we now feel like we are sitting in the twilight zone, where criticism of Hegerl et al is disallowed specifically because many others have made similar mistakes, and criticism of Frame et al is apparently disallowed because the original paper is so bad it should never have been published. (We should make it clear that we find R2's comments to be rather harsh in this respect - we sense a bit of a power struggle over who should be considered an authority in this field. IOO Frame et al did represent some progress on what had gone before, but also took a step or two backwards in some respects.)
No doubt we will eventually be able to publish our ideas in a stand-alone paper, but it will take time and effort that could have been more fruitfully expended elsewhere. Part of the reason we haven't already done this is the suspicion that some might say our criticism is too trivial to publish, unless we justify it by tying it directly to the existing literature. And if we are going to name and criticise specific authors, it is only reasonable to give the original authors the chance to respond directly in Comment-and-Reply format. But it seems that this is no longer an option. Meanwhile, climate scientists (including, but not limited to, the IPCC authors) press on regardless, blissfully unaware of the problems in what they are doing. They may even feel more strongly justified now in what they are doing, having successfully shaken off some attacks :-(
On the positive side, one more climate scientist (R3) seems to be pretty much on-board and on-message (I only used these buzzwords to annoy jules who hates them). On top of one more who we've had some recent email correspondence with, there are some signs that the tide will start to turn eventually. Jules and I have been fortunate enough to get invited to a workshop in the UK later this month on uncertainty and probability in climate science. I'm sure these issues will get a good airing there - unfortunately, it coincides with an IPCC lead author meeting, so some of those who need to hear it most urgently will not be able to do so!
Please don't take this the wrong way, but are you perhaps just flogging a dead horse? Or redoing the "Charge of the Light Brigade?" I don't see any sort of "smoking gun" that would discredit the Frame et al GRL paper. It seems to be that some people like a uniform prior and others (like you) love the expert prior.
ReplyDeleteIs it simply all about egos, jealousy, schadenfreude, and who's considered an expert, rather than doing science? It's starting to smack of the "climateaudit" lunacy.
My suggestion is that you publish your comments in the Arkiv. It gets it out there immediately, and since you have to revise and add stuff to make a paper you should not have already published problems.
ReplyDeleteSorry, that should be www.arxiv.org You may need an endorsement, but fortunately, I don't think Lumo can do it in the Atmospheric and Oceanic area.
ReplyDeleteAnon,
ReplyDeleteYou can characterise it how you like, but it is very clear to us that there are substantial problems both with what has been published, and how it is interpreted. If you had read the AR4 draft carefully the significance of this would be apparent. There is much more to it that merely liking a particular prior.
It might be worth pointing out to those looking either for conspiracy theories, or for a reason to dismiss me as a crank, that the barrier to publication has not actually been the reaction of climate scientists at all - the comment on Hegerl was blocked by an over-zealous editor, and the negative ref for this comment on Frame et al is not in fact a climate scientist, but someone who (by his own admission) is very much on the fringe of the field with a very unconventional (and clearly unflattering) view of the whole field. I agree there is clearly something about ego and jealousy in his review!
The climate scientists who have looked at our work have been much less hostile - even those who were the subject of the critical comments have acknowledged that there is merit in at least some of our points.
I spent some time yesterday reading their paper, your response, and then some Wikipedia and Google Books info on Bayesian induction.
ReplyDeleteIt's quite amazing that such fundamental philosophical issues are being discussed in the pages of Geophysical Research Letters! The question of how best to do induction, indeed whether the practice of induction is logically valid, goes back to David Hume in the 18th century, and his thoughts are still considered relevant today. One might expect to see this kind of debate occuring in a social science or philosophy journal, not one regarding the hard sciences.
In terms of Bayesian reasoning and updating, the issue in their paper is the search for the "noninformative prior", an attempt to formalize what it means to say that we "know nothing beforehand" about an issue. And as the article mentions, this leads to Bertrand's paradox, where an attempt to say that "all possibilities are equally possible" does not work when you start transforming the possibility space. They talk about the question of climate sensitivity, where it might also be valid to parameterize in terms of 1/sensitivity. Clearly a prior which is uniform in sensitivity cannot be uniform in 1/sensitivity.
The bottom line message I took from my reading is that in theory, these issues are intractable, but in practice, they are usually unimportant. No matter what prior you adopt, before long enough evidence has been collected from experience, study, and analysis, that your posterior conclusions will be much the same, independent of the prior. Hence it is not all that important whether your prior is uniform in sensitivity or 1/sensitivity; by the time you look at all the information available today from all the independent studies, you will (or should) have pretty much the same idea about what the probability distribution is for this parameter.
It should be possible to show this quantitatively. You could take your earlier paper where you put together several lines of evidence, do the analysis separately for different priors, and verify that the choice of prior doesn't much matter. This would then contradict the point of this most recent paper, which is that choosing the uniform prior in sensitivity is better than being uniform in 1/sensitivity, or other such choices. You would show that it doesn't matter in practice, hence it is an empty philosophical distinction.
Well, that's my "expert" opinion after about an hour of study, make of it what you will.
Hi James - I think you are right: a large part of this is who exactly gets to be the expert? After all, you can't have outsiders getting in the easy papers stating the (in retrospect) bleedin' obvious. Thats the priviledge of the great and the good in that field.
ReplyDeleteHal,
ReplyDeleteBroadly speaking I agree regarding the suitability of this topic for GRL (and perhaps that could be expanded to include the suitability of me to discuss it :-). However, even though, as the Yorkshireman said when asked for directions, "I wouldn't start from here", here is where we are.
Ultimately the point I am hoping to get across is that any belief in high climate sensitivity (say P(S>6) at the 5% level) depends on both
A) choosing a prior that initially assigns very high probability to this (eg 40% or 70% for U[0,10] and U[0,20] respectively)
and
B) only using a small subset of the relevant data, and thereby implicitly asserting that the rest is either useless or dependent.
I don't think the literature justifies either one of these points. And of course if either one of A or B does not hold (let alone both of them) then the alarm about S>6 simply vanishes. In our previous GRL paper we made what was perhaps a mistake in only addressing B - you are right that the prior doesn't matter much when all the data are used, but the question of independence is perhaps a little more nuanced than our simplified calculation (or for that matter Hegerl et al) showed. And many scientists have indeed argued - using logic that I consider to be faulty, but they are the ones publishing in Science and Nature, not me - that the data are insufficient to rule out high S with high confidence.
James wrote:
ReplyDelete"Ultimately the point I am hoping to get across is that any belief in high climate sensitivity (say P(S>6) at the 5% level) depends on both
A) choosing a prior that initially assigns very high probability to this
and
B) only using a small subset of the relevant data, and thereby implicitly asserting that the rest is either useless or dependent."
I think A is kind of what I want to focus on in the reply, if possible, because F05 didn't have much to say about B. I'd say B is a live issue, but that there's far from a consensus on that at the moment - I doubt that one will be settled in a two-pager in GRL! I've heard quite strong opinions both ways on the value of volcanic and paleo constraints on sensitivity, and it seems pretty clearly to be a debate that has a way to run. On the other hand I think clarifying different some issues to do with A would be a really useful thing to do. I think you're right in your comment that "what does this data stream tell me about X assuming no prior information about X?" is not the only question we're interested in, but I think it's still a question we ought to be interested in, and one that seems well-matched to the sort of approach in F05. But that doesn't mean I think F05 is anything like the last word on the subject. [Actually I saw it mostly as a paper that said "here's why these studies disagree so much, and here's one workaround." If other people develop alternatives, great.]
As Al Gore has recently said:
ReplyDeleteTo begin with, there is a reason why new scientific research is peer-reviewed and then published in journals such as Science, Nature, and the Geophysical Research Letters, rather than the broadsheets. The process is designed to ensure that trained scientists review the framing of the questions that are asked, the research and methodologies used to pursue the answers offered and even, in some cases, to monitor the funding of the laboratories — all in order to ensure that errors and biases are detected and corrected before reaching the public.
You may insert a hollow laugh right here ->
Oh, I don't think anyone with any direct experience would describe peer-review as much more than a least-bad method of weeding out some of the worst dross...of course some review(er)s are excellent, but the worst can certainly be worse than useless.
ReplyDelete