Friday, October 08, 2010

It's so unfair!

Probably some of you have seen this video regarding the dreaded "third reviewer". Some of you engaged in "normal science" may even have had experiences where the third reviewer throws a spanner in the works, ruining what would otherwise have been a straightforward path to publication. Well, I smile ruefully at your your tales of woe, pat you patronisingly on the head and introduce the post-normal....

4th reviewer!

The 4th reviewer is the grumpy curmudgeon who the editor calls on, when the first three reviews have all proved disappointingly positive. Having got the grump to give his negative comments (in which they freely admit that they don't actually understand the work very well, but nevertheless judge it to be simultaneously trivial and wrong), the editor gleefully brandishes this review like a rapier, consequently rejecting the manuscript as worthless and conveniently ignoring the fact that the first three (yes, 3, labelled as A, B and C) reviewers all clearly recommended the paper for acceptance (albeit subject to some revision, but that's entirely standard - I don't think I have ever had "accept unchanged" on initial submission).

Any similarity between the above description and the fate of this manuscript at JClim is entirely frustrating. I have politely asked the editor if he might reconsider and give a little more weight to the first three positive reviews, but prior to a decision on that can do little but shout IT'S SO UNFAIR!





I am however substantially encouraged by the fact that the three positive reviewers appear to be climate scientists working in this area (I obviously recommended several at submission, and before you accuse me of picking my friends, bear in mind that I don't actually have any) whereas I have some reason to hope that curmudgeonly Reviewer D may be a bit out of their field. So I am reasonably optimistic that the ideas in the manuscript may find a receptive audience at some point.

Can anyone else report having had a paper rejected with either a higher proportion, or number, of openly positive reviews? (Excluding the tabloids where entirely valid and interesting science may be judged by the editor to be simply not sexy enough, that is.) Both a 3:1 ratio of positive to negative reviews, and a total of 3 positive ones, are surely records for me.

21 comments:

Arthur said...

Hi James,

while this didn't happen to me personally, I am aware of a paper that received *6* (supposedly) independent mostly positive reviews, and nevertheless was rejected by the editor :)

http://vacanteyes.com/blog/download/fifth_force.pdf

I'm also aware of a more serious case where the editor decided to publish a paper after receiving 5 negative reviews (due to the importance of the work on the chance it was right, despite it likely being wrong).

I.e. editors are expected to exercise their judgment, not to blindly follow referees. There are always other journals...

eduardo said...

A few years ago, I had one manuscript in Climate Dynamics that was not even sent out for review, without explicit reason

James Annan said...

Arthur, well of course i acknowledge the ed has the authority to do what they want - but I also have the right to complain when it seems they are making a partial and unreasonable judgement based on picking a few words out of a more balanced picture :-)

Eduardo - ouch! I'd expect at least a reasonable excuse in that case. Climate Dynamics will probably be our second choice - hopefully taking these reviews with us, as I'm completely confident that at least three (and maybe even 4 after revision and reply, because a fair bit of his hostility is easily attributable to a simple misunderstanding) will support publication. But JClim is where almost all of the relevant work in this area has appeared so I'm disappointed this manuscript seems so far beyond the pale to them (well, to one editor at least).

noxrch said...

I had a paper some years ago which finally, after a full year in review, got rejected by nature. 3 rounds of reviewing with 4 different reviewers. All reviewers except one was satisfied. His final comment, after our complete rebuttal of his onslaught of red herrings, was simply that he did not "believe" the significance. I suspect that the true reason was that we openly criticized his work.

I was pretty pissed with the editors choice to listen to an administrative old fart.

Roger said...

In astronomy it's not common for papers to be reviewed by more than one person. All my papers have been accepted subject to moderate revision, except for one (my very first) that was accepted in its original form. I should have retired on the spot - I don't think that will ever happen again.

Telescope observing proposals are a different matter. Oversubscription rates are such that it's common to get a report on your application that basically says it's great, the science will be important, the need for time is well demonstrated, the technical case is robust, but no, you won't get time. It's always a bit confusing!

David B. Benson said...

I had a paper once which received (mildly) positive reviews which an editor rejected and eventually (voluntarily) explained that it wasn't significant.

Up[on mature reflection, the editor was right. I never resubmitted elsewhere.

Rattus Norvegicus said...
This comment has been removed by the author.
Rattus Norvegicus said...

Ah, so the conspiracy cuts both ways!

Oh, wait, maybe that means it isn't a conspiracy afterall....

Word verif: matcous. Is that anything like couscous?

crandles said...

Penultimate line of page 9 there is a 'then' which should be 'them'.

That will make all the difference to the decision to publish ;o)

crandles said...

On page 20 you say:

"The central 90% “very likely” confidence interval, therefore, is precisely the full range of the values obtained, i.e. 2.1–4.4C. With such a small sample,
however, the endpoints of the interval are determined entirely by the outlying samples which
therefore makes this analysis rather sensitive to sampling error

...

The implied 70% confidence interval of
2.3–4.3C is much more robust in respect of sampling error"

I am just wondering if this also applies to the analysis of number of nearer neighbours esp. when there is only one or no nearer neighbours. Is it possible that a more robust analysis can be performed by making the test easier to pass.

For example, number of models that are better than (the model mean adjusted by say 1/10 of a standard deviation of the models performance range). (Even if that makes any sense at all, you would want to ajust it to make the test as robust/least sensitive to sampling error as possible.)

My other comment would be that I would like to see the synthetic isotropic curves for 4 through 15 and extented outwards to 50 ensemble members. This could easily be a misunderstanding but I would have thought it is the level at which saturation is reached that is the level that is most important to determine. I don't know how much error there would be likely to be with a curve fitting exercise. The shapes of the curve for real data look like the saturation levels would be noticably higher.

Probably all nonsense, but those are my initial reactions, for what little they are worth.

crandles said...

I presume from this post that you are confident that the 4th reviewer will not read this blog. (Though I suppose it is possible that you know who it is and this is some sort of (private?) joke.)

James Annan said...

Hey, those are interesting ideas, thanks for them. As for readers, I always take into consideration that anyone may read what I've written. I don't think I've been unfair to either the reviewer or indeed the editor. There is a rather straightforward misunderstanding which underlies much of the reviewer's hostility, and I'm entitled to feel frustrated that as things stand I won't even have the chance to argue the point with him. Bear in mind that his review has real consequences for me, whereas my post has no power at all.

crandles said...

So nothing like calling someone a 'grumpy curmudgeon' to make them less grumpy? I suppose likening yourself to Kevin does lighten the mood. :o)

I assumed in such a situation you could and have asked the editor to pass on a reply in case it causes the 4th reviewer to change their opinion even if you cannot compell the 4th reviewer to read and reply nor the editor to consider and/or ask the 4th reviewer for a response. However, I know nothing of the process and your "I won't even have the chance to argue the point" make me doubt this assumption.

David B. Benson said...

I don't follow what's going on. Did the editor resign? Called to the Great Scientist in the Sky?

crandles said...

Now I am confused. Was David Benson's post meant to be on Publication limbo or road trip part 2 rather than this thread?

James Annan said...

(deleted and reposted)

Chris, the ed has rejected the paper and I don't know if he would be prepared to pass on any further response from me - it's not something I have ever tried in the past, nor have I ever received similar messages as a reviewer.

David, as Chris said, your comment appears to be misplaced. Assuming you are referring to the climatic change case, the ed was Stephen Schneider. The ed I am arguing with here is very much alive, or at least he was very recently.

EliRabett said...

If the editor was not Broccoli, write an appeal to him enclosing the reviews and your comments on the fourth reviewer. If it was Broccoli, give up

David B. Benson said...

Less confused now, but baffled over Eli's reference to one of my favorite vegatables.

crandles said...

My only guess is that Eli is trying to avoid using such a pejorative term as suggesting someone might be in a persistent vegitative state.

guess which troll? said...

All of my first author papers (I think I had 5) were accepted without revision. My professor had 150 previous publications and had never just seen an "accepted". He'd been an editor for 300 (at a decent journal) and never seen it either.

James Annan said...

Now I think about it, our recent GRL paper probably had the best set of reviews I've ever seen - one accept as is, and one v minor suggestion. It didn't go back to the reviewers. Almost always even a strongly favorable reviewer asks for some minor point to be clarified, and I am happy to take their advice since I'm often accused of being a little terse (or efficient, as I like to think of it).