Friday, January 25, 2008

Can you say "survivor bias"?

There's a (presumably unintentionally) funny report on Nature's "Peer-to-Peer" blog today. Entitled "Researchers like the peer-review system", it discusses a study in which "more than 3,000 senior authors, reviewers and editors were asked about the peer-review system". Surprise surprise, they basically liked it (although with some reservations).

Now think about that for a second.

For an example of what I mean by "survivor bias", drawn directly from Peer-to-Peer's previous posting(!), I'd bet dollars to doughnuts that the "more than 3,000 senior authors, reviewers and editors" did not include many of the women who find that they have a much tougher time publishing as first author. Because of course as a result of this gender discrimination, they are less likely to be "senior". One could hardly expect those who have benefited most through the system, to admit that their success is substantially a matter of luck and nepotism, whether or not this is the case. (By "nepotism", I mean that if you want "entry" to the top journals, having a famous supervisor to co-author your papers is very helpful.) One might as well ask sitting British MPs if they support the constituency-based simple majority electoral system that effectively guarantees many of them a job for life (actually, they do).

My ears pricked up when I noticed that the respondents were reported as being hostile to "open peer review". However, on checking the summary, they use this term to mean signed reviews, and it is unrelated to the open system of many EGU journals that I have mentioned before.

[Note that I'm intentionally not saying what I think about the peer review system here. That's not my point. In fact I've had no real problems with peer-review myself, until I started to tread on the toes of some "probabilistic climate prediction experts" at which point my work became simultaneously (1) well-known, (2) interesting and important, and (3) wrong-headed - and most importantly, (4) not appropriate for this journal. But that is a separate rant you've all heard many times. Indeed overall I probably just about qualify as one of those "senior authors" who supports the system pretty much as it is].


EliRabett said...

It is not just women. Try and get something into PRL if you are not part of the nomenklatura let alone Science or Nature.

Not to be bitter of course, but about ten years ago we sent something off to them that came back at rocket speed as not interesting. About two years later after we published in a lower impact journal I got sent a bunch of proposals to review written by the consignetti that started from our observation. Eli was middling kind. As Brecht said, the world is mean and man uncouth and high level journal editors don't tell the truth.

Heiko said...

It's not right that private companies should get articles for free from government funded researchers and then charge a little fortune for access, and that's also part of the current peer review system.