In a recent blog post, James Annan (hereafter JDA) suggested the sending of a rude letter to the AGU, in which he criticises the peer review process currently employed at their journals.
The first point JDA makes is that the AGU editors should not sit down so much and should get some reviewers outside of the list supplied by the author. I believe that JDA has no evidence that this does not already occur. Indeed I have it on good authority that has himself been one of the reviewers of one of these supposedly "rubbish" papers. I find it hard to believe that the authors would have suggested him as a reviewer. Indeed, IIRC, JDA supplied the editor of the aforementioned "rubbish" paper with an incisive and damning review. So what went wrong? As I recall, JDA was subsequently so outraged that the paper was not rejected and that instead it came back with revisions, that he refused to have any more to do with it.
So, where does the blame lie? With the editors?
Many papers are neither right nor wrong - they are simply the scientifically obtained results of experiment - however, so far I have once been in the happy situation of dealing with a paper that I thought was very likely to be wrong. Since it was in an EGU journal, some of the process is out there to be seen. Here it is.. This paper is mostly either right or wrong since it proposes a theory about the way the climate system works.
I have had so many papers stupidly rejected by ignorant reviewers, that I always give authors the right of reply to at least one round of reviews. I also think that if the scientific consensus were the truth then there would be no more science to be done. Mediocre papers fly through peer review, whereas both the brilliant ones and the crap ones struggle. How to discern the difference?
So, despite one negative and two unimpressed reviews I invited the author to attempt a revision. After one revision the EGU process goes underground so the further correspondence is, unfortunately, not available on the web. But basically what happened is that the most critical reviewer refused to participate further! The other reviewers remained kind of unimpressed, but not entirely damning - I felt they could not actually disprove the hypothesis being presented in the paper although they thought it extremely unlikely. Eventually I hit on the idea that the author should outline in full details what would be required for his theory to be true. I quite liked the result, which outlined how three or four things would have to all be at the unlikely end of the current scientific understanding for the theory to have a chance of being right. At that point I felt the paper was scientifically true, even if the proposed hypothesis was unlikely, and that this was properly represented in the manuscript, and so I accepted the paper.
So, yes, I do think it is the fault of the editors that utter "rubbish" gets published. I think far too many editors abdicate their responsibility and base their decision on the number of good or bad votes from the reviewers, rather than actually switching on their brains and considering what the authors need to do in order to make the paper publishable. However, as an editor, I do also wish that reviewers would be prepared to stick with the process until the end rather than storm off in a hissy fit, even if this means they submit increasingly exasperated reviews to the editor - they will still appreciate them.
I would also remind JDA that he has himself said on many occasions that the peer review process is just the start, that it is OK that wrong papers are occasionally published, and that the true science will win in the end. This is, of course, why it is absolutely imperative that point 2 made by JDA is fixed. The initial publication is just the start of the discussion so why are the AGU so frightened of Comments? Especially in the politicised world of Climate science, it is shameful that they go so out of their way to discourage them.
All I can say is vote with your feet.. :-)
The first point JDA makes is that the AGU editors should not sit down so much and should get some reviewers outside of the list supplied by the author. I believe that JDA has no evidence that this does not already occur. Indeed I have it on good authority that has himself been one of the reviewers of one of these supposedly "rubbish" papers. I find it hard to believe that the authors would have suggested him as a reviewer. Indeed, IIRC, JDA supplied the editor of the aforementioned "rubbish" paper with an incisive and damning review. So what went wrong? As I recall, JDA was subsequently so outraged that the paper was not rejected and that instead it came back with revisions, that he refused to have any more to do with it.
So, where does the blame lie? With the editors?
Many papers are neither right nor wrong - they are simply the scientifically obtained results of experiment - however, so far I have once been in the happy situation of dealing with a paper that I thought was very likely to be wrong. Since it was in an EGU journal, some of the process is out there to be seen. Here it is.. This paper is mostly either right or wrong since it proposes a theory about the way the climate system works.
I have had so many papers stupidly rejected by ignorant reviewers, that I always give authors the right of reply to at least one round of reviews. I also think that if the scientific consensus were the truth then there would be no more science to be done. Mediocre papers fly through peer review, whereas both the brilliant ones and the crap ones struggle. How to discern the difference?
So, despite one negative and two unimpressed reviews I invited the author to attempt a revision. After one revision the EGU process goes underground so the further correspondence is, unfortunately, not available on the web. But basically what happened is that the most critical reviewer refused to participate further! The other reviewers remained kind of unimpressed, but not entirely damning - I felt they could not actually disprove the hypothesis being presented in the paper although they thought it extremely unlikely. Eventually I hit on the idea that the author should outline in full details what would be required for his theory to be true. I quite liked the result, which outlined how three or four things would have to all be at the unlikely end of the current scientific understanding for the theory to have a chance of being right. At that point I felt the paper was scientifically true, even if the proposed hypothesis was unlikely, and that this was properly represented in the manuscript, and so I accepted the paper.
So, yes, I do think it is the fault of the editors that utter "rubbish" gets published. I think far too many editors abdicate their responsibility and base their decision on the number of good or bad votes from the reviewers, rather than actually switching on their brains and considering what the authors need to do in order to make the paper publishable. However, as an editor, I do also wish that reviewers would be prepared to stick with the process until the end rather than storm off in a hissy fit, even if this means they submit increasingly exasperated reviews to the editor - they will still appreciate them.
I would also remind JDA that he has himself said on many occasions that the peer review process is just the start, that it is OK that wrong papers are occasionally published, and that the true science will win in the end. This is, of course, why it is absolutely imperative that point 2 made by JDA is fixed. The initial publication is just the start of the discussion so why are the AGU so frightened of Comments? Especially in the politicised world of Climate science, it is shameful that they go so out of their way to discourage them.
All I can say is vote with your feet.. :-)
1 comment:
Sadly, Eli's experience of editors is that they pick the reject, accept button first and there is not much you can do in between, esp with the higher impact journals. A major advantage of open review appears to be that the editors can no longer be arbitrary
Post a Comment