Thursday, November 26, 2009

Reply

In her comment on my earlier post, Julia Hargreaves (henceforth JCH) makes a number of points regarding the peer review process, illustrating her points with reference to some specific examples.

I am pleased to note that JCH agrees strongly with point 2 in my original article. That is, the current Comment and Reply system is largely broken, due in no small part to the obvious hostility that some journals (at least) exhibit towards them. While one can often make a good case for simply ignoring a poor paper, there are also occasions when a direct rebuttal - or even a public debate to clarify points of reasonable disagreement - is appropriate.

As for her point about choice of reviewers, I remind her that we have both certainly had experiences of having our manuscripts reviewed exclusively by those on our lists of suggestions, the reviewers having waived their anonymity in these cases. Of course, I do try to suggest reviewers who I consider to be independent and authoritative, but the Editor must also be prepared to take some responsibility for ensuring this.

In the case that JCH refers to where a paper I rejected was published anyway, I remind her that the decision to publish was not mine, and reviewers are not infrequently over-ruled. It turns out that in their revised version the authors did address my main criticism, but substantial problems remain in their work. While with hindsight it may have been a mistake for me to decline to re-review the paper, I consider it harsh to blame me for the decision I played no part in making, other than to oppose. I am confident that any number of competent reviewers would have easily identified the problems, in fact two of them have already dismissively cited the paper as an example of an obviously flawed approach.

I realise that the job of the editors is a difficult one, especially in a highly politicised field where not all participants are primarily interested in the truth. It will not be easy to filter out the rubbish without rejecting the genuine science, especially in cases where the latter presents a valid challenge to an existing status quo. I agree it is probably better to accept there will always be some of the former that gets through, than reject too many of the latter. But in this case, and irrespective of blame, the journals have a responsibility to allow corrections to be made.

JCH does not comment on the issue of open review, which I consider to represent a significant improvement over the standard approach. I am in no doubt that many (maybe all) of the obviously flawed papers that the AGU has recently published would have been easily weeded out by such a process (or, to take JCH's generous approach, they would at the very least have had to account openly for the limitations in the methods employed, which would probably have weakened their conclusions to the point of irrelevance).

15 comments:

jules said...

I like open review too, but I don't think it is necessarily the solution to all problems. Lazy editors and idiotic reviewers can exist in both systems. What I like most about the open review system is that there is a citable record of the original submission - so that if, in the future a rejected manuscript turns out to be have been valid after all, then the authors can claim the credit due to them.

The system also does, I hope, prevent the shoddy practice of reviewers writing up their own related papers while very slowly reviewing and rejecting the work of their rivals.

The other potential advantage of open review, an example of which I have yet to experience, is where reviewers disagree markedly with some loving the paper, others disagreeing and more thinking it is trivial. AGU journals seem to always reject such papers. With open review the end result would be the same, but at least the discussion is already out there in the world, rather than being a secret fight between only 3 or 4 people, all of which aren't supposed to talk about it.

The first rule of fight club is: you must hold all your fights in public....That doesn't sound quite right..? Oh no, even the Mona Lisa's falling apart!

georgesdelatour said...

If I read that a paper has been peer reviewed, should I assume its data has been thoroughly cross-checked, audited, replicated etc? Or is that part mostly taken on trust, and peer review is more about people arguing whether the logic of the argument is sound?

James Annan said...

Definitely the latter, and it works well enough so long as (1) people aren't dishonest and (2) you don't mind if the occasional incorrect result is published.

In principle, most scientists aren't generally interested in precise replication, but rather validating and testing theories with different methods. There are no prizes (or even publications or grants) to just repeat previous work.

If something wrong is published, it can lead to a bit of wasted time and effort for those who are misled, but detailed replication of all results prior to publication would cost a lot more, and is essentially impossible for complex situations where the equipment may only exist in one place in the world. In my field, it is not uncommon for a computer code to be tailored to run on a specific supercomputer and take several months to run.

EliRabett said...

What journals have long fought against is turning into a comment magazine. You REALLY have to be either a Nobel Prize winner or have the killer argument to get a comment published in a good journal (weird rules apply to Nature, Science and Physical Review so we will leave them out). Even if you have a killer argument it takes forever to get a comment published, and the editors pretty much force you to hold to a single strong point. Anyone needing a reminder of this, only has to re-read (Eli assumes everyone here read it at least once) Rick Trebino's lament about how his FROG got squashed.

Thus, outside of an open review/comment format there is not a chance that there will be more comments. As a result, what you get are comment papers, papers that are really comments built around a small factoid from the commenting lab's results, but not strongly linked to the original.

Hank Roberts said...

Sorry, Eli, citation needed.
My friend Google has no recollection of ever having read that.

James Annan said...

Eli is talking about the article I link to half way down this post (not the Roe and Baker stuff itself).

I don't really like the comment-that's-not-a-comment as it rarely makes a genuinely worthwhile paper and denies the original authors the chance of a reply. Gavin has a couple of these recently, I have heard on the grapevine there is another in the pipeline relating to one of the junk papers that has been mentioned a lot recently. IMO it leaves a bit of a nasty taste in the mouth, I think the criticisms should be both made, and dealt with, head-on. But all too often the journals won't allow this.

David B. Benson said...

One of the journals from a long established society has a comment policy. To wit, comments must be received within a fixed period after a paper appears (6 moths?). Then the author(s) write what is called a "closure" as a reply. The comments and the closure all appear togethr in some single issue. As far as that journal goes, that ends the matter.

James Annan said...

That seems pretty reasonable, so long as the time scale is not over-long. With the AGU, the process seems to involve 2 separate rounds of peer review at a minimum (perhaps 4 with re-reviews) and the odds are stacked heavily against publication.

bigcitylib said...
This comment has been removed by the author.
bigcitylib said...

So, you have an open review process for a Climatology journal.

Does this mean Pielke Sr. for example could mount am email campaign on behalf of his latest, complete with angry accusations and maybe even death threats (I hear Mr. Jones has recieved a few in the past two weeks)from his supporters? Might not attract many reviewers if so.

James Annan said...

BCL,

I don't think it is helpful to link Pielke Snr to some nutter's death threats. Policy on anonymity in the "open" system is a matter for individual journals to decide. The EGU uses some official reviewers who may maintain anonymity even though their reviews are published. So while their judgements may be criticised, the people themselves are protected. However, uninvited commentaries must be published under the authors' names, not that there is any real way of authenticating in the case of pseudonyms.

In the event of denial-of-service type spamming attacks the Editors would have to take control, but this has not happened in practice even when there have been controversial manuscripts concerning hockey sticks and proxies.

David B. Benson said...

James Annan --- I didn't check the details about the peer-review (if any) of "comments" or how long the author(s) have to write a "closure" and the degree (if any) of the peer-review of that.

hecking a recent issue, discussions and closures for one paper appearing in the March 2007 issue and one in the October 2007 issue both appear in the October 2008 issue.

Chuck said...

"The system also does, I hope, prevent the shoddy practice of reviewers writing up their own related papers while very slowly reviewing and rejecting the work of their rivals."

Has anyone used the CRU hack to look for these? It seems to me like a far more effective way of gotchaing people than these inanre rants that get posted...

skanky said...

"What journals have long fought against is turning into a comment magazine."

Would it not be possible for journals to have comment "supplements" (in whatever form, separate, tacked in, pull-out centre-fold, etc.)? That way there could be any number of comments, without having to impact the content of the main journal.

It could even have a different editor to the main journal, so there would be less incentive to refuse comments (they'd need some content, and they haven't already passed a paper that's now being criticised).

There could be an issue of whether comment supplements get incorporated into the literature, but I'm sure that's more down to implementation details than a show-stopper.

There's probably some fault I'm missing though.

Chuck said...

The fault is that journals aren't interested in being reflective. They want to be cutting edge. If you look at "aims & scope" for the journals of your choice, "groundbreaking" and "original" will appear far more than "pedantic" or debatable.

And there is no real penalty for journals being wrong- just look at Nature.