The Hwang stem cell thing has provoked some soul-searching about the peer review process and whether it could be improved. Kevin Vranes and John Fleck both suggest making reviewers' names public. While there is no harm in posing the question, I don't think the idea has got much going for it in general terms, and certainly cannot see how it could make a difference in this or similar cases.
For a start, it's best to be clear about what peer review does and does not (cannot) do. It does not provide any guarantee that a result is correct (this should be self-evident from the fact that new papers get published contradicting old ones). It does not even mean that anyone has checked that the work has actually been carried out as described - referees are not forensic scientists with the time and resources to do this, even if they wanted. It does however generally indicate that the described work and results seem credible and relevant. Peer review cuts out some poor research, both by actually rejecting stuff that is clearly wrong and (probably) dissuading larger numbers of people from even trying to publish work that is weak. In my experience, it also generates significant improvements to the quality of manuscripts that make it through to publication, both in terms of making them more accessible to as broad a readership as possible, and also correcting minor (even major) errors. Perhaps peer review is best thought of as a sort of "moderation" task analogous to moderators of usenet and other web-based discussion fora (eg censoring Lumo's silly comments on my blog), although it is of course performed at a much more detailed and careful level than that requires.
Reviewers are doing an unpaid job which gets very little reward - we do it mainly because we know we have to in order for the system to survive, and the possible perks (getting a peek at some results early, encouraging the authors to add a few citations of your own work) are pretty trivial in comparison. Rarely, the position of power it affords could be used to do down a rival, but given that the other referees might give glowing reviews and the Editor is likely to work out what is going on, that would be a rather risky course of action. I've had a (very) small handful of reviews that I consider unfair, but even so they cannot prevent publication, only slow it down if the author is persistent enough (ie keeps trying new journals).
Peer review can never be expected to catch deliberate premeditated fraud. That's not what it is for. And anyway, does it matter if occasionally fraudulent (and more often, flawed) papers get through? Not much, IMO. If they are wrong, and the science is important enough to matter, they will get caught out fairly soon (as in this recent case). It might be an unwelcome distraction but it's hardly the end of the world.
Removing the right to anonymity would have the single immediate effect of cutting the number of willing referees substantially. Few people would be eager to risk offending more important scientists who might be awarding them grants or jobs in the future - that's not to say that the author would necessarily (or even likely) retaliate, but why take the risk? Even when the author has no possible authority over the reviewer, there's nothing to be gained by making an enemy in the field. I've encountered a few reviewers who have signed their reviews, but I don't know if this is their general policy or if they only do it when saying nice things. I'm not sure that the latter is worth a great deal. I don't see how naming the reviewers in the Hwang case would achieve anything more than perhaps allowing us to make scapegoats out of a few people who are actually victims as much as the rest of us. They didn't ask to be sent a fraudulent paper, and it's not reasonable to expect them to have caught it.
On a slightly different tack, some EGU journals have an open discussion phase, where as well as a formal peer review, there is an interval during which the paper is put on the web and anyone can comment (HESS and CP) . It's an interesting idea but doesn't seem to have caught on widely. And their system of using pdf documents makes it incredibly tedious to follow. The reviews there also seem to be published, and many are signed, but there is no compulsion (at least for the latter). There has been the suggestion that openly publishing reviews may generate a "me too" syndrome where later comments merely echo the first rather than providing an independent perspective. Perhaps there's room for some fine-tuning on that.
Another idea that's doing the rounds, that I am much more sympathetic to, is that authors should "detail their specific contributions to the research submitted" (Science's new policy). Check the accompanying example too. Some journals have encouraged this for some time, and I can see how it might act as some sort of an extra incentive to honesty if the fraudulent author has to specifically claim a particular bit of the work as their own rather than hope to deflect blame onto the whole group. I've never bothered with this procedure in the past - since I rarely have more than about 1 co-author there seems little point - but it might help to discourage one or two who seem overly cheeky about claiming co-authorship when they have at best a tenuous link to the work, and also result in those who deserve it being properly credited.
For a start, it's best to be clear about what peer review does and does not (cannot) do. It does not provide any guarantee that a result is correct (this should be self-evident from the fact that new papers get published contradicting old ones). It does not even mean that anyone has checked that the work has actually been carried out as described - referees are not forensic scientists with the time and resources to do this, even if they wanted. It does however generally indicate that the described work and results seem credible and relevant. Peer review cuts out some poor research, both by actually rejecting stuff that is clearly wrong and (probably) dissuading larger numbers of people from even trying to publish work that is weak. In my experience, it also generates significant improvements to the quality of manuscripts that make it through to publication, both in terms of making them more accessible to as broad a readership as possible, and also correcting minor (even major) errors. Perhaps peer review is best thought of as a sort of "moderation" task analogous to moderators of usenet and other web-based discussion fora (eg censoring Lumo's silly comments on my blog), although it is of course performed at a much more detailed and careful level than that requires.
Reviewers are doing an unpaid job which gets very little reward - we do it mainly because we know we have to in order for the system to survive, and the possible perks (getting a peek at some results early, encouraging the authors to add a few citations of your own work) are pretty trivial in comparison. Rarely, the position of power it affords could be used to do down a rival, but given that the other referees might give glowing reviews and the Editor is likely to work out what is going on, that would be a rather risky course of action. I've had a (very) small handful of reviews that I consider unfair, but even so they cannot prevent publication, only slow it down if the author is persistent enough (ie keeps trying new journals).
Peer review can never be expected to catch deliberate premeditated fraud. That's not what it is for. And anyway, does it matter if occasionally fraudulent (and more often, flawed) papers get through? Not much, IMO. If they are wrong, and the science is important enough to matter, they will get caught out fairly soon (as in this recent case). It might be an unwelcome distraction but it's hardly the end of the world.
Removing the right to anonymity would have the single immediate effect of cutting the number of willing referees substantially. Few people would be eager to risk offending more important scientists who might be awarding them grants or jobs in the future - that's not to say that the author would necessarily (or even likely) retaliate, but why take the risk? Even when the author has no possible authority over the reviewer, there's nothing to be gained by making an enemy in the field. I've encountered a few reviewers who have signed their reviews, but I don't know if this is their general policy or if they only do it when saying nice things. I'm not sure that the latter is worth a great deal. I don't see how naming the reviewers in the Hwang case would achieve anything more than perhaps allowing us to make scapegoats out of a few people who are actually victims as much as the rest of us. They didn't ask to be sent a fraudulent paper, and it's not reasonable to expect them to have caught it.
On a slightly different tack, some EGU journals have an open discussion phase, where as well as a formal peer review, there is an interval during which the paper is put on the web and anyone can comment (HESS and CP) . It's an interesting idea but doesn't seem to have caught on widely. And their system of using pdf documents makes it incredibly tedious to follow. The reviews there also seem to be published, and many are signed, but there is no compulsion (at least for the latter). There has been the suggestion that openly publishing reviews may generate a "me too" syndrome where later comments merely echo the first rather than providing an independent perspective. Perhaps there's room for some fine-tuning on that.
Another idea that's doing the rounds, that I am much more sympathetic to, is that authors should "detail their specific contributions to the research submitted" (Science's new policy). Check the accompanying example too. Some journals have encouraged this for some time, and I can see how it might act as some sort of an extra incentive to honesty if the fraudulent author has to specifically claim a particular bit of the work as their own rather than hope to deflect blame onto the whole group. I've never bothered with this procedure in the past - since I rarely have more than about 1 co-author there seems little point - but it might help to discourage one or two who seem overly cheeky about claiming co-authorship when they have at best a tenuous link to the work, and also result in those who deserve it being properly credited.
1 comment:
Obviously you have never been coauthor on a materials paper with 15 or more co-authors. One grew the stuff, another purified it, six or seven others used specialized analytical techniques involving half million $ instruments, two integrated it into a device, and a bunch more tested it. Of course, any one of these authors had a graduate student or a post-doc involved. Of course, we could have published 15 fragments.....
Post a Comment