Someone asked me recently how I thought journalists should cover science stories. I'm not sure I answered very well at the time, but since then I came up with a couple of thoughts.
I think the golden rule to remember is that a new item of research, even if it's appeared in a prestigious peer-reviewed journal, is not in itself a new "truth" about the world. It is only the current opinion of a couple of researchers working in a particular area. It's always worth bearing in mind that the researchers themselves might change their minds in a few months and even if they don't, it is quite possible that the rest of the scientific community will decide they are talking nonsense and either criticise or (what is perhaps worse) ignore the study. Peer review is only the first step of the evaluation process by which new ideas become part of the established body of knowledge. It merely means that another couple of researchers - who may know the authors quite well on a professional or even social level - have read at the paper, seen no glaring errors and are prepared to take most of the details on trust. So the whole process amounts to someone saying "this is our idea, what do you all think?" with a filter that helps to keep out the most obvious errors. It's rare that a paper will be both highly significant (in terms of overturning an established consensus), and accepted without a murmur of disagreement from all quarters. So if it's hot off the press, it's perhaps best thought of as an opinion more than a fact, even though the opinion may (and hopefully in most cases does) have some strong grounding in reality. In time, it will generally become clear that a significant majority of researchers either accept the work or reject it (or perhaps understand its strengths and limitations).
I dont mean to denigrate the contribution of the reviewers - I've been grateful for the many helpful comments I've received through the process - but I once saw an estimate that each reviewer takes about 1hr per paper (this does seem low to me, and I suspect it may be different across fields, but then again I don't get asked to review much myself). So I think it's important to keep a sense of perspective about things.
This limited review is potentially a bit of a problem with the IPCC reports, which may use hot off the press results which have not yet really stood the test of time. Of course, ignoring the most recent stuff would be just as easy to criticise. I guess the authors have to use their judgement as to how trustworthy the 2006 papers are.
I think the golden rule to remember is that a new item of research, even if it's appeared in a prestigious peer-reviewed journal, is not in itself a new "truth" about the world. It is only the current opinion of a couple of researchers working in a particular area. It's always worth bearing in mind that the researchers themselves might change their minds in a few months and even if they don't, it is quite possible that the rest of the scientific community will decide they are talking nonsense and either criticise or (what is perhaps worse) ignore the study. Peer review is only the first step of the evaluation process by which new ideas become part of the established body of knowledge. It merely means that another couple of researchers - who may know the authors quite well on a professional or even social level - have read at the paper, seen no glaring errors and are prepared to take most of the details on trust. So the whole process amounts to someone saying "this is our idea, what do you all think?" with a filter that helps to keep out the most obvious errors. It's rare that a paper will be both highly significant (in terms of overturning an established consensus), and accepted without a murmur of disagreement from all quarters. So if it's hot off the press, it's perhaps best thought of as an opinion more than a fact, even though the opinion may (and hopefully in most cases does) have some strong grounding in reality. In time, it will generally become clear that a significant majority of researchers either accept the work or reject it (or perhaps understand its strengths and limitations).
I dont mean to denigrate the contribution of the reviewers - I've been grateful for the many helpful comments I've received through the process - but I once saw an estimate that each reviewer takes about 1hr per paper (this does seem low to me, and I suspect it may be different across fields, but then again I don't get asked to review much myself). So I think it's important to keep a sense of perspective about things.
This limited review is potentially a bit of a problem with the IPCC reports, which may use hot off the press results which have not yet really stood the test of time. Of course, ignoring the most recent stuff would be just as easy to criticise. I guess the authors have to use their judgement as to how trustworthy the 2006 papers are.
No comments:
Post a Comment