Hudsongate: the mystery deepens

OK, it seems that the CRU thing isn't going to go away any time soon, so I might as well fan the flames a little.

Recently some journalists have been getting very excited about BBC journalist Paul Hudson's supposed complicity in all this. When the story first broke, Hudson quickly reported "I was forwarded the chain of e-mails [just the small subset relating to an article he wrote] on the 12th October" - a month before the widespread release. Many people initially attributed this to a careless slip really meaning 12 November, but Hudson confirmed explicitly that he meant October, saying "I was copied in to them at the time".

But....he wasn't. The email addresses of all participants in that sequence of messages are clearly identifiable, and none of them are he.

The obvious conclusion, which may explain why he has been a bit coy about all of this, is that one of the participants forwarded these email to him rather than including him in the conversation. That is, they secretly divulged private email conversations to a journalist. Not that there is anything particularly noteworthy or scandalous in the emails, but if I was talking about someone behind their back, I'd be a little surprised to find a colleague was forwarding my comments directly to them. It seems that at least Hudson is adhering to his journalistic ethics by not revealing his mole. Not so long ago, a prominent climate scientist was sacked for unauthorised contact with the media, so perhaps it's just as well!

[OK, it's theoretically possible that he was openly forwarded the emails with the prior agreement of all participants, but that those messages have not found their way into the released set, but it seems a bit improbable.]

Empirical Study of Data Sharing by Authors Publishing in PLoS Journals

Apropos of nothing in particular :-)

Conclusions

We received only one of ten raw data sets requested. This suggests that journal policies requiring data sharing do not lead to authors making their data sets available to independent investigators.

(via Andrew Gelman, in my list of old starred items)

Those Japanese budget cuts, in full, in brief

There has been much wailing and gnashing of teeth to be heard in Japanese scientific circles recently. News reports were talking of 50% cuts here, 20% there, a complete mothballing of major projects like the new petaflop supercomputer. None of this directly affected us, although cuts all around would probably lead to some loss from our institute as the budgets of different departments seem highly fluid. And though we may make some use of the new computer, it's not central to any of our plans. But still, it was rather shocking news.

But it turns out that it's just hot air. What the Govt did was set up some "Peoples' Panels" to give advice on where to cut the budget. This nakedly populist move is aimed at cutting money to pay for things like promise to remove or reduce the road tolls (so much for their green credentials, but one can hardly expect a politician to put principles above popularity). The panels are made up of non-specialists and public representatives, and their suggestions are not based on any detailed evaluation of value for money but are just arbitrary numbers plucked out of thin air based on their prejudices after a brief meeting.

Faced with the predictable howls of outrage from not only the scientists but also the ministries responsible for funding the research, the deputy PM has already distanced himself from the recommendations and the PM also seems keen to emphasise that the actual decisions will be made by the Govt. So I expect the gravy train of Japanese science has some way to go before it hits the buffers.

It's a shame they've made such a cack-handed effort, as there is massive waste in the Japanese system that could easily be addressed. As a fairly typical example, I was suddenly told to spend $15,000 at short notice a couple of months ago, for fear that we would otherwise have to hand it back and lose face in the process. So I bought some computers and software for colleagues who didn't really need it. There is some sort of cap on salaries, but every year we struggle to waste all the money we are given for equipment and travel. And if larger savings were required there are research groups who basically do nothing for years on end, like the 100-person-year project that has cost tens of millions of dollars and generated about 10 minor publications (and nothing else). Not that the number of papers is a perfect measure of output, but if a whole group isn't writing anything at all, it might as well not exist as far as the rest of the world is concerned. The project leader is influential, however, so there is no chance of anyone asking whether this is a sensible way of spending money...

On related news, jules learnt about the annual performance evaluation system for staff yesterday. Apparently there has been one for years, only no-one bothered to tell us :-)

[jules' pics] 11/26/2009 11:36:00 PM

Japanese snow drop!, originally uploaded by julesberry2001.

Showing that Jomyoji tea shoppe really does have a British-style garden, the first snow drops I have seen for 8 years.

--
Posted By jules to jules' pics at 11/26/2009 11:36:00 PM

And now for something completely different

Sorry, I know you all want more of the "Swifthack" email theft and smear campaign, but you'll have to put up with fat Japanese bellies instead:

An obesity research group is considering revising as early as next year the waistline threshold, a key gauge used to diagnose obesity and metabolic syndrome, to better reflect the nation's actual health

They have apparently realised that 85cm for a man's waist is not really a useful indicator of significant health problems, more likely a sign of a decent dinner.

Some may be surprised that the phrase "considering revising as early as next year" is attached to such an apparently trivial decision. I'm wondering if they will raise the threshold (for that is the direction it will go, if it moves) more rapidly than my waist will increase.

Reply

In her comment on my earlier post, Julia Hargreaves (henceforth JCH) makes a number of points regarding the peer review process, illustrating her points with reference to some specific examples.

I am pleased to note that JCH agrees strongly with point 2 in my original article. That is, the current Comment and Reply system is largely broken, due in no small part to the obvious hostility that some journals (at least) exhibit towards them. While one can often make a good case for simply ignoring a poor paper, there are also occasions when a direct rebuttal - or even a public debate to clarify points of reasonable disagreement - is appropriate.

As for her point about choice of reviewers, I remind her that we have both certainly had experiences of having our manuscripts reviewed exclusively by those on our lists of suggestions, the reviewers having waived their anonymity in these cases. Of course, I do try to suggest reviewers who I consider to be independent and authoritative, but the Editor must also be prepared to take some responsibility for ensuring this.

In the case that JCH refers to where a paper I rejected was published anyway, I remind her that the decision to publish was not mine, and reviewers are not infrequently over-ruled. It turns out that in their revised version the authors did address my main criticism, but substantial problems remain in their work. While with hindsight it may have been a mistake for me to decline to re-review the paper, I consider it harsh to blame me for the decision I played no part in making, other than to oppose. I am confident that any number of competent reviewers would have easily identified the problems, in fact two of them have already dismissively cited the paper as an example of an obviously flawed approach.

I realise that the job of the editors is a difficult one, especially in a highly politicised field where not all participants are primarily interested in the truth. It will not be easy to filter out the rubbish without rejecting the genuine science, especially in cases where the latter presents a valid challenge to an existing status quo. I agree it is probably better to accept there will always be some of the former that gets through, than reject too many of the latter. But in this case, and irrespective of blame, the journals have a responsibility to allow corrections to be made.

JCH does not comment on the issue of open review, which I consider to represent a significant improvement over the standard approach. I am in no doubt that many (maybe all) of the obviously flawed papers that the AGU has recently published would have been easily weeded out by such a process (or, to take JCH's generous approach, they would at the very least have had to account openly for the limitations in the methods employed, which would probably have weakened their conclusions to the point of irrelevance).

[jules' pics] 11/25/2009 08:09:00 PM

Jomyoji English tea garden bakery, originally uploaded by julesberry2001.

Jomyoji, one of the top five Zen temples of Kamakura has an English tea shoppe in its grounds. I have avoided it for 8 years, because English tea shoppes in Japan sell cakes that look realistic but taste of salt rather than of sugar. This is so crushingly disappointing that its better to avoid the places altogether. I thought the Zen temple version would be the worst of the worst, but I was wrong. Not only were the cakes actually sweet and the coffee quite acceptable, but the garden was actually very realistic too.

There was even an on-site bakery, which is what the picture is of.

--
Posted By jules to jules' pics at 11/25/2009 08:09:00 PM

Comment on "Advice to the AGU regarding their journals", by James Annan Esq.

In a recent blog post, James Annan (hereafter JDA) suggested the sending of a rude letter to the AGU, in which he criticises the peer review process currently employed at their journals.

The first point JDA makes is that the AGU editors should not sit down so much and should get some reviewers outside of the list supplied by the author. I believe that JDA has no evidence that this does not already occur. Indeed I have it on good authority that has himself been one of the reviewers of one of these supposedly "rubbish" papers. I find it hard to believe that the authors would have suggested him as a reviewer. Indeed, IIRC, JDA supplied the editor of the aforementioned "rubbish" paper with an incisive and damning review. So what went wrong? As I recall, JDA was subsequently so outraged that the paper was not rejected and that instead it came back with revisions, that he refused to have any more to do with it.

So, where does the blame lie? With the editors?

Many papers are neither right nor wrong - they are simply the scientifically obtained results of experiment - however, so far I have once been in the happy situation of dealing with a paper that I thought was very likely to be wrong. Since it was in an EGU journal, some of the process is out there to be seen. Here it is.. This paper is mostly either right or wrong since it proposes a theory about the way the climate system works.

I have had so many papers stupidly rejected by ignorant reviewers, that I always give authors the right of reply to at least one round of reviews. I also think that if the scientific consensus were the truth then there would be no more science to be done. Mediocre papers fly through peer review, whereas both the brilliant ones and the crap ones struggle. How to discern the difference?

So, despite one negative and two unimpressed reviews I invited the author to attempt a revision. After one revision the EGU process goes underground so the further correspondence is, unfortunately, not available on the web. But basically what happened is that the most critical reviewer refused to participate further! The other reviewers remained kind of unimpressed, but not entirely damning - I felt they could not actually disprove the hypothesis being presented in the paper although they thought it extremely unlikely. Eventually I hit on the idea that the author should outline in full details what would be required for his theory to be true. I quite liked the result, which outlined how three or four things would have to all be at the unlikely end of the current scientific understanding for the theory to have a chance of being right. At that point I felt the paper was scientifically true, even if the proposed hypothesis was unlikely, and that this was properly represented in the manuscript, and so I accepted the paper.

So, yes, I do think it is the fault of the editors that utter "rubbish" gets published. I think far too many editors abdicate their responsibility and base their decision on the number of good or bad votes from the reviewers, rather than actually switching on their brains and considering what the authors need to do in order to make the paper publishable. However, as an editor, I do also wish that reviewers would be prepared to stick with the process until the end rather than storm off in a hissy fit, even if this means they submit increasingly exasperated reviews to the editor - they will still appreciate them.

I would also remind JDA that he has himself said on many occasions that the peer review process is just the start, that it is OK that wrong papers are occasionally published, and that the true science will win in the end. This is, of course, why it is absolutely imperative that point 2 made by JDA is fixed. The initial publication is just the start of the discussion so why are the AGU so frightened of Comments? Especially in the politicised world of Climate science, it is shameful that they go so out of their way to discourage them.

All I can say is vote with your feet.. :-)

Advice to the AGU regarding their journals

I intend to write a proper letter (well email) based on the following thoughts. Before I send it, I'd be grateful for any suggestions.

Dear AGU,

Your previously-respected journals are starting to build a reputation for publishing rubbish papers. Here are a few suggestions for how to remedy the situation.

1. Please advise your journal editors to get off their backsides and try to find at least one reviewer outside of the list of suggestions that you insist all authors provide. I'm sure some conscientious editors already do this, but not all. Under the current system all one has to do in order to publish complete crap is nominate a couple of friends who will wave things through the review process.

2. Please fix your Comment and Reply system. You already have a published policy, please ask your editors to adhere to it. Currently, it seems common for editors to impose an additional "pre-review" stage before even starting the process of asking for a Reply. This means that while any old crap can get published in a few weeks on the say-so of a couple of friends (see 1), even in the best case scenario it takes up to a year, and about 6 reviews (including those nominated by the authors of the original work), for anyone to have any chance of pointing out the problems however glaring they are. Comments are generally relatively urgent in nature and are required to be short by your policies, so a time scale rather closer to that of GRL would seem more appropriate. From what I have experienced and heard from others, editors not infrequently decide that they simply can't be bothered with dealing with comments regardless of their content or validity. If you don't want to deal with comments pointing out errors in misleading and shoddy work, you should work harder to prevent the publication of the erroneous work in the first place rather than blocking any criticism!

3. Please let us know which editor is responsible for each paper. There is no reason for this information to be secret, and EGU journals routinely publish it. I'm sure many hard-working and conscientious editors are upset that the reputation of the journals is being tarred by the actions of a few. Being held to account even in this small way may encourage people to be a little more careful.

4. Please consider introducing a meaningful open review system, like the one in place at many EGU journals. I'm aware of your plans to take baby steps in this direction, but it seems that you are trying to make it as ineffective as possible. Rather than creating some half-hearted process, why not just simply copy one that already works pretty well? (If you feel the need to differentiate yourselves from the EGU, an obvious improvement on their process would be to post all the invited reviews together, rather than letting the later reviewers read the earliest ones.) I'm not aware of any obviously crap papers being published in the EGU journals that practice open review, and their reputation and status is rising as rapidly as yours is falling. Although I'm personally a fan of the EGU open publication system, I would much rather see the AGU journals as a credible and authoritative alternative, rather than sinking into oblivion.

[jules' pics] 11/23/2009 05:56:00 PM

cooked prawn, originally uploaded by julesberry2001.

This time the focus is on data people, but to all climate modellers, whose arts are equally dark: in order to avoid being boiled alive after all is revealed, make sure you have fully published your climate model at the journal created especially for the purpose of improving transparency in modelling science, GMD.

--
Posted By jules to jules' pics at 11/23/2009 05:56:00 PM