Advice to the AGU regarding their journals

I intend to write a proper letter (well email) based on the following thoughts. Before I send it, I'd be grateful for any suggestions.

Dear AGU,

Your previously-respected journals are starting to build a reputation for publishing rubbish papers. Here are a few suggestions for how to remedy the situation.

1. Please advise your journal editors to get off their backsides and try to find at least one reviewer outside of the list of suggestions that you insist all authors provide. I'm sure some conscientious editors already do this, but not all. Under the current system all one has to do in order to publish complete crap is nominate a couple of friends who will wave things through the review process.

2. Please fix your Comment and Reply system. You already have a published policy, please ask your editors to adhere to it. Currently, it seems common for editors to impose an additional "pre-review" stage before even starting the process of asking for a Reply. This means that while any old crap can get published in a few weeks on the say-so of a couple of friends (see 1), even in the best case scenario it takes up to a year, and about 6 reviews (including those nominated by the authors of the original work), for anyone to have any chance of pointing out the problems however glaring they are. Comments are generally relatively urgent in nature and are required to be short by your policies, so a time scale rather closer to that of GRL would seem more appropriate. From what I have experienced and heard from others, editors not infrequently decide that they simply can't be bothered with dealing with comments regardless of their content or validity. If you don't want to deal with comments pointing out errors in misleading and shoddy work, you should work harder to prevent the publication of the erroneous work in the first place rather than blocking any criticism!

3. Please let us know which editor is responsible for each paper. There is no reason for this information to be secret, and EGU journals routinely publish it. I'm sure many hard-working and conscientious editors are upset that the reputation of the journals is being tarred by the actions of a few. Being held to account even in this small way may encourage people to be a little more careful.

4. Please consider introducing a meaningful open review system, like the one in place at many EGU journals. I'm aware of your plans to take baby steps in this direction, but it seems that you are trying to make it as ineffective as possible. Rather than creating some half-hearted process, why not just simply copy one that already works pretty well? (If you feel the need to differentiate yourselves from the EGU, an obvious improvement on their process would be to post all the invited reviews together, rather than letting the later reviewers read the earliest ones.) I'm not aware of any obviously crap papers being published in the EGU journals that practice open review, and their reputation and status is rising as rapidly as yours is falling. Although I'm personally a fan of the EGU open publication system, I would much rather see the AGU journals as a credible and authoritative alternative, rather than sinking into oblivion.

Editorial standards at AGU journals

This ridiculous paper has already been eviscerated by Tamino, RC, and mt, so I won't waste too much time on it, but I have spotted one more error that no-one else has commented on so far before I get to the main point of my post.

So first, the error. It's not as significant as the one Tamino deals with, but here it is anyway. Paragraph 30 reads as follows:

[30] For the 30 years prior to the 1976 shift (i.e., 1946–1975) the SOI averaged +1.93 but in the 30 years after 1976 (i.e., 1977–2006) the average was −3.06, which represents a shift from a La Niña inclination to an El Niño inclination. The standard deviations for the two periods were 9.48 and 10.40 on monthly SOI averages, and 6.56 and 6.35 on calendar year averages, which indicates consistent variation about a new average value. Only the RATPAC-A data are available for lower tropospheric temperatures both before and after this shift, and even then we are limited to 17-year periods for our analysis of RATPAC-A data because monitoring did not commence until mid-1958. From 1959 to 1975 the RATPAC LTT averaged −0.191°C and from 1977 to 1993 it averaged +0.122°C. The standard deviations on the seasonal data were 0.193° and 0.163 C°, and on monthly data 0.162°C and 0.146°C. We have already illustrated the close relationship between SOI and GTTA, but this description of the respective changes before and after the Great Pacific Climate Shift indicates a stepwise shift in the base values of each factor but otherwise relatively consistent ranges of variation.

(SOI and RATPAC are time series data, the definition of which is irrelevant to my point).

So, to parse this clearly, the authors are claiming that when a time series has the properties that the mean of the first half and second half differ, but the variability in each interval is the same, this indicates that there was a step shift in the middle.

Let's take a linear trend plus noise, y=at+e where t (time) runs from -T to T, and e is any additive noise with variance s². The expected mean over the first half [-T,0] is -aT/2, and the mean over the second half is aT/2. The standard deviation of the first half is sqrt(a²T²/12 + s²), where these two contributions come from the linear trend and noise respectively. The standard deviation of the second half is, um, sqrt(a²T²/12 + s²). In other words, when the means of the first and second half of a time series differ, but the variability does not, this tells us precisely nothing about whether there was a step change or just a linear trend. Ooops.

I hate to think what they might have done were it not for Craig Loehle's graciously acknowledged assistance with the statistical analysis. I'm sure he is delighted to be associated with this sorry mess of a paper.

Now to the real point, which is that the AGU journals seem to have become rather prone to publishing this sort of nonsense recently (remember Schwartz, Chylek and Lohmann, to name but two). Although of course no system will ever be infallible (and a system that blocked out all the mistakes would block a lot of interesting and important stuff too) the errors in these papers are so blindingly obvious that it is hard to believe that any reasonably diligent and competent reviewers would miss them.

When you submit a paper to an AGU journal, you are asked to suggest 5 reviewers. It's a common enough practice (pretty much ubiquitous) which helps the editor who may not be well acquainted with the particular subfield that the paper address. However, it also serves as an open invitation to game the system by suggesting people who you think are likely to be particularly generous and uncritical. Of course any editor worth his (or her) salt should also look outside this list, especially if he thinks that the authors have played this game. But if they have a lot of papers to deal with, and no real stake in the outcome, they might not bother.

I'd like to see AGU editors attach their names to the papers they handle. This seems to be standard practice in the EGU journals, which have not (AFAIK) suffered from this sort of nonsense. This leaves the editors somewhat accountable for the mistakes they make, and any pattern of repeated carelessness would be easily spotted. Of course, the main responsibility lies with the authors and reviewers, but as things stand, it seems like a small clique can publish anything they want so long as they all pat each other on the back. Peer review isn't well set up to deal with deliberate gaming of the system.

Of course, under the EGU's open review system, the gaping holes in this paper would have been spotted very quickly and it would never have been published.

Open access publishing

Jim Giles had an article in NewScientist on open access publishing recently, along somewhat similar lines to his article in Nature earlier this year. However, this time it's a comment article rather than mere news, so he's prepared (allowed? Nature's hostility to open-access is hardly a secret) to present his point of view, which is that open access is the way forward. I agree with what he says, although think he misses a detail in his presentation of the matter as one of paper charges versus subscription fees. Many journals charge both! So in fact we are already paying the journals $1000-$2000 to take our work, hide it behind a paywall, and sell it for their own profit.

Since I had it handy, I just checked that the 4-page Comment on Schwartz will cost about $2000 for standard publication (in JGR) assuming some use of colour. In fact I see the AGU has just instigated a new experimental system whereby we can pay the same again (roughly) as an additional charge to have the article made freely available to all readers. So that would make it $4000, just for a short comment. Think I'll pass on that second option, as the AGU (in common with essentially all publishers) do not prevent authors putting papers on their own web sites anyway. Google will usually find the full text for recent and even forthcoming papers these days (old pre-web ones are harder to track down).

The EGU open access journals somehow manage the whole process far cheaper - and I don't think they are heavily subsidised by the EGU itself, at least not in the long term. Their page charges are about €20 per page. Even with their small pages this is lot less less than the AGU ask for (an order of magnitude cheaper than the AGU's free-to-view version), and right at, or even below, the bottom of the range of cost that Jim Giles suggests. I also like their open reviewing system. Now that several of their journals are well-established, it looks like an obvious place to send manuscripts on a wide range of topics. The only thing I really don't much like about their system is that the papers are only available as (awkwardly-formatted IMO) pdfs and not directly as html. But this is a bit of a detail. Sadly, they don't yet have a journal for what I would think of as the bulk of climate science itself (there's clearly a demand for it, as I've seen the occasional paper that I would class as mainstream "Climate Dynamics" material in rather tangentially-related journals like ACP and CP).

Is for-profit publishing dead?

I'm talking about academic journals, of course. People have been grumbling for ages about what a scam it is: academics not only provide the material for free but even pay the journal handsomely for the privilege of appearing in their hallowed pages, then other academics review and edit the material for free, and finally yet more academics pay to be allowed to read it. In these days of electronic transmission and powerful searching and indexing facilities, the added value provided by the publisher appears limited at best, and indeed strongly negative when you consider the sums of money involved. So I am following this initiative with some interest. I haven't signed up yet because I haven't checked how many relevant journals are run by Elsevier, nor whether they are a particularly egregious offender.

As an example of what it actually costs to run a professional outfit, the EGU journals generally charge a publication fee of €24 per "page". The "page" size for this calculation is unusually small (1900 characters), but a typical paper is still only around €600 in my experience, and a previous short comment was the princely sum of €170. This is the total price, after which the resulting paper is open access. Compare to AGU journals, where the basic fee is already often larger (GRL is $500 for 4 pages, and longer JGR papers $1000) even when the final article is paywalled, and the open access option is another $2500-$3500 on top - an obvious indication of the value of our work that we have simply been giving away. I suppose at least with the academic societies the profit might be going to reasonably good causes (such as funding the sterling work of their Ethics Task Force, I presume), but even with their low fees the EGU publication empire runs at a modest profit level, sufficient to fund gradual expansion.

Previously I have grumbled about the EGU stable of journals having a gaping hole in that papers on climate change and prediction - surely one of the largest constituencies of their community - do not have a suitable home (CP and ACP have taken a few climate change papers, but the fit seems tenuous). However, a couple of years ago, they started up "Earth System Dynamics". This looked initially according to its blurb like it might be a bit too Earth-systemy for my tastes, but it's also had a steady stream of more conventional (primarily physical) climate change research. One paper that looks like it could kick off a bit of a bunfight is this one here about diagnosing climate feedbacks from satellite measurements, (which refs previous Spencer+Braswell vs Dessler spats) for which Andrew Dessler was quick to get in a somewhat critical review. I haven't looked at the manuscript carefully enough to judge for myself. Anyway, I'll certainly bear the journal in mind for future manuscripts, and hopefully it will take some business away from the leeches.

Incidentally I think a commenter previously pointed to this paper some time ago, but I can't find where. It seems that enough reviewers noted the somewhat woolly logic behind the climate sensitivity calculations that it did not progress to the final peer-reviewed publication stage. There is also this manuscript which got somewhat similar comments.

McLean's whine part 2

OK, having talked about the (absence of) science in McLean's reply, on with dealing with the whine with which they try to smother it.

Well, the first point is that it may be unprecedented for the authors' reply to fail to pass peer review, although conversely it may very well have happened before without my hearing about it. (FWIW, many years ago I once found out quite by chance that someone had published a "comment on" one of my papers in a journal that I didn't regularly read, and neither the author nor the editor had bothered to contact me at any point in the process.) In the AGU journals there is no automatic right of reply, there is only the right to submit a reply for the consideration of the editor. In this case, it seems that after peer review McLean et al's reply was not found to meet the standards for publication. I don't make any claim to be disinterested but on reading it, I can only agree with this judgment, specifically on the following grounds.

(A) Their reply performs a dishonest bait-and-switch in initially claiming that their analysis was not based on the filtered data[1], but then conversely stating that their statistics only refer to the filtered data and were never even intended to refer to long-term variation[2]. Of course their acknowledgment of this second point means that there is not one scrap of support in the paper for their claim that the analysis "shows the potential of natural forcing mechanisms to account for most of the temperature variation" [over the last 50 years]. Their analysis simply has no bearing on any long-term trends, since they filtered them out of the data.

(B) They don't even try to address the fact that their original paper pretended that the differencing was done to reduce the noise, when in fact it amplifies noise and eliminates long-term variability[3]

(C) they present no defence of their claim of a "stepwise shift" in the mid-1970s, which (as we pointed out) their naive statistics do not support.

There is other stuff that I could criticise in their reply, but this is more than enough to justify a rejection. They simply aren't responsive to our criticisms.

As for the stuff they quote from the hacked emails...well that's all pretty small beer. First, the most (only? IIRC) critical comment that they have reprinted from one of our three reviewers was a very reasonable complaint and I think we were all happy to make some edits to the tone of the original. Since that is available on the web, you can look for yourself to see the minor differences. There were also some useful comments from all three reviewers about clarity and references, but nothing major. As you can see, the people that we discussed proposing for reviewers are all authoritative and respected figures in the field. The complaint that they were "reasonably well known" to one of the more prominent authors is particularly laughable - there could hardly be anyone of similar experience who isn't. However, the actual choice of reviewers is the responsibility of the editor and I don't know who was used. It would certainly be amusing to know who McLean et al proposed for both their original paper and the attempted reply, but somehow I doubt they'll be prepared to divulge either this information or the full reviews that they received. I can only assume that the editor made his own choice of reviewers, which is not only his right but duty if he thinks the suggested reviewers are inappropriate. The list of suggestions is not to enable the authors to choose their reviewers, but rather to provide the editor with some help.

The complaint about "prior publication" due to placing a copy of the submitted manuscript on a personal web-page is just a petty and pathetic attempt at armchair-lawyering. The AGU explicitly endorses publication of manuscripts on a personal web-page, the only minor error was in some carelessness over the formatting which was corrected within a couple of days. This could justify a minor slap on the wrist but it is not "prior publication" by any reasonable definition, including that of the AGU. For one thing, it wasn't even put on the web-site prior to submission!

I don't see on what grounds there could possibly be any criticism of the AGU for using a different editor and reviewers from those who dealt with the original paper. Obviously, the submission of a comment may be considered an implicit criticism of those responsible for the original publication, so it is reasonable to remove this possible source of bias. But anyway, neither of these matters has anything to do with us.

I thought it was supposed to be the Poms that whinged. On this evidence, some Aussies are pushing them pretty hard. If only they had devoted as much effort to science they might have learnt something.

---

[1] "contrary to what Foster et al. (2010) imply, the data in question (Figure 7) were not subjected to contrived statistical analysis" and "we used the filtering technique solely to establish that a 7-month time lag existed between changes in the ENSO and changes in global average lower tropospheric temperature...Our substantive conclusions were then based on applying this time-lagged relationship to the raw data sets"

[2] "Our comments about the change in Southern Oscillation Index (SOI) accounting for 72% of the variance in satellite (MSU) GTAA, 68% of variance in the radiosonde (RATPAC-A) GTAA and 81% of variance in the tropospheric temperature in the tropics were made in the context of the discussion of our derivatives based on differentials between 12 month averages, and we stand by them. Contrary to Fea10 claims, those figures do not refer to long-term variations but only to the derivatives that were used."

[3] "To remove the noise, the absolute values were replaced with derivative values based on variations. Here the derivative is the 12-month running average subtracted from the same average for data 12 months later."

AGU open review trial

The AGU has announced (EOS: sub required) that it is half-heartedly trialling an open review system. Similar to what Nature did some time ago, participation is completely voluntary on the part of the authors, it is only the volunteered reviews that will be published (the "real" reviews will be private as usual) and the authors will be under no obligation to even read, still less respond to, any comments offered. The AGU even take pains to emphasise that comments will not be considered part of the permanent record and the article itself will be removed when the final decision to publish (or not) is taken. So even assuming the comments themselves are kept up, they will be "hanging" with no context - even when the paper is published, it will generally be rather different from the one that the comments refer to.

Oh, as if that isn't enough, the web-sites where the manuscripts and comments are published will be open to AGU members only. Of course it is hard to marry the concept of open review with a paywall, which is just one more nail in the coffin of the latter.

So that will be a hit. Not. I wonder if they really think they are doing something useful, or whether they actually set out to design a "trial" that is guaranteed to fail so they can claim the status quo is just fine?

I predict they will conclude that there is little support for such a system, just like Nature did. I further predict the continuing expansion of the EGU publication empire where an open review system is carried out properly (if not perfectly IMO)! "Popular" or not, I think many scientists can see that such a system enhances the credibility of the peer review process. CP and ACP are very young but already two of the leading journals in their respective fields.

The list of journals included in this trial doesn't include much of interest to me: Geochemistry, Geophysics, Geosystems; Global Biogeochemical Cycles; JGR–Earth Surface; JGR–Planets; and Radio Science.

Comment on "Advice to the AGU regarding their journals", by James Annan Esq.

In a recent blog post, James Annan (hereafter JDA) suggested the sending of a rude letter to the AGU, in which he criticises the peer review process currently employed at their journals.

The first point JDA makes is that the AGU editors should not sit down so much and should get some reviewers outside of the list supplied by the author. I believe that JDA has no evidence that this does not already occur. Indeed I have it on good authority that has himself been one of the reviewers of one of these supposedly "rubbish" papers. I find it hard to believe that the authors would have suggested him as a reviewer. Indeed, IIRC, JDA supplied the editor of the aforementioned "rubbish" paper with an incisive and damning review. So what went wrong? As I recall, JDA was subsequently so outraged that the paper was not rejected and that instead it came back with revisions, that he refused to have any more to do with it.

So, where does the blame lie? With the editors?

Many papers are neither right nor wrong - they are simply the scientifically obtained results of experiment - however, so far I have once been in the happy situation of dealing with a paper that I thought was very likely to be wrong. Since it was in an EGU journal, some of the process is out there to be seen. Here it is.. This paper is mostly either right or wrong since it proposes a theory about the way the climate system works.

I have had so many papers stupidly rejected by ignorant reviewers, that I always give authors the right of reply to at least one round of reviews. I also think that if the scientific consensus were the truth then there would be no more science to be done. Mediocre papers fly through peer review, whereas both the brilliant ones and the crap ones struggle. How to discern the difference?

So, despite one negative and two unimpressed reviews I invited the author to attempt a revision. After one revision the EGU process goes underground so the further correspondence is, unfortunately, not available on the web. But basically what happened is that the most critical reviewer refused to participate further! The other reviewers remained kind of unimpressed, but not entirely damning - I felt they could not actually disprove the hypothesis being presented in the paper although they thought it extremely unlikely. Eventually I hit on the idea that the author should outline in full details what would be required for his theory to be true. I quite liked the result, which outlined how three or four things would have to all be at the unlikely end of the current scientific understanding for the theory to have a chance of being right. At that point I felt the paper was scientifically true, even if the proposed hypothesis was unlikely, and that this was properly represented in the manuscript, and so I accepted the paper.

So, yes, I do think it is the fault of the editors that utter "rubbish" gets published. I think far too many editors abdicate their responsibility and base their decision on the number of good or bad votes from the reviewers, rather than actually switching on their brains and considering what the authors need to do in order to make the paper publishable. However, as an editor, I do also wish that reviewers would be prepared to stick with the process until the end rather than storm off in a hissy fit, even if this means they submit increasingly exasperated reviews to the editor - they will still appreciate them.

I would also remind JDA that he has himself said on many occasions that the peer review process is just the start, that it is OK that wrong papers are occasionally published, and that the true science will win in the end. This is, of course, why it is absolutely imperative that point 2 made by JDA is fixed. The initial publication is just the start of the discussion so why are the AGU so frightened of Comments? Especially in the politicised world of Climate science, it is shameful that they go so out of their way to discourage them.

All I can say is vote with your feet.. :-)

We-who-must-not-be-named

Let's face it: pay-to-view scientific publishing is dead, even though its zombie corpse is still staggering around, thrashing around aimlessly.

There's another article in Nature about open access (and at least this one isn't hidden behind a paywall). Actually, it's not that bad, and certainly doesn't seem to be agitating shamelessly in the way that some past articles appeared to be. Among the open access publishers they discuss, there is however one notable absentee: the entire EGU family of journals is conspicuous by its absence. Of course this is only one publishing house operating in a particular field of research, but it's a very big one, and within that area, certainly ACP, BG, CP and probably TC (I don't really know the latter) are important within their fields. It may be worth emphasising that as well as being economically viable (indeed comfortably profitable), the costs of the EGU journals are extremely low, often lower than the publication charges imposed by paywalled journals even before you consider what they are raking in through subscription fees. That makes the excuses of for-profit publishers hard to take seriously. What are they actually adding for their fees?

Although I recently pointed to some dodgy papers in EGU journals, I don't think they are any worse than the AGU or other publishers who use a paywall paradigm - rather, my concern is that I expect them to be better, given the open review and opportunity for additional unsolicited comments. But even Nature, with it's $30-40,000 of investment in every paper, manages to come up with its share of stuff that is known to be wrong before the ink is dry. One nice feature of the EGU system is that you can see the reviews, and in the cases I mentioned, it seems that the problem (if there is one) is that the eds are bending over backwards to be generous towards papers that have been roundly rubbished in review. It is important to maintain some sort of standards, if reviewers are going to be expected to donate their time and energy. It might be useful to see the second and subsequent rounds of reviews, and I'm not sure why this bit is kept secret.

Incidentally, something I have been agitating for over recent years has recently come to pass: there is now a "subscribe to comments" button on each discussion page! So if you spot an interesting manuscript under review, you can easily keep an eye on what the reviewers are saying. I hope this will lead to an increase in non-invited comments. There are also RSS feeds for both the discussion and final publication phases of the journals.

Of course, Nature aren't stupid, and while trying to defend their cash cow for as long as possible, are also increasingly buying in to the open access model. It's just a matter of time. If they can persuade either authors, or funding bodies, that they add up to $40,000 of value to every paper they publish, then maybe they get their money in other ways, and good luck to them. It's time to stop gouging readers who have already paid for the research with their taxes.

EGU review part 2

After three days of action, we were both basically planning on being observers for the remainder. We even had a bit of a lie-in on Thursday and missed the 8:30 talks. I let the sea level wash over me, there was an interesting review of various projection methods for global sea level change and some detailed local investigations. I then briefly popped into some paleoclimate variability stuff, which is an area that our new project/post-doc will probably be looking into in the near future.

We were planning on staying late into the evening, so chose a gentle option for the afternoon - the panel discussion on blogs and social media in scientific research. It was, perhaps predictably, fairly rubbish. I've been to enough of these sort of things (e.g. at the AGU) that I should have known better, but like I said, I needed a mental break anyway. There was the usual po-faced advice for "professionalism" etc, and the ubiquitous stern advice to under no circumstances blog any research in progress, because the journals won't like it, because of copyright, or prior publication, or some other generally ill-defined bogey-man.

It took all of 30 seconds to confirm what I was already pretty confident about, that Nature explicitly allows pre-publication on a personal blog or server such as Arxiv. The AGU and AMS have basically the same policy for all of their journals. Of course the EGU does too - almost all of its journals provide open access to the submitted manuscript prior to publication anyway. That EGU web page handily has links for similar statements from Elsevier, PNAS, and PLoS. Can anyone find a single case of a journal rejecting a paper based on the fact that an author has discussed some of the content on a blog?

But still, the myth propagates. Sigh. I'm sure he meant well, but that's little excuse for making up stuff (and thereby potentially misleading the audience) that could have been so easily checked.

[Update: but see comment below - apparently, the Geological Societies of America and London both prohibit any prior publication of "results, data, ideas and/or interpretations", including on any electronic media. It's therefore understandable that the speaker might have assumed these conditions would apply more widely. But fortunately, they don't!]

Of course, there is no compulsion to write about work in progress, if it is something you'd rather keep private. There is no compulsion to write about anything at all! Perhaps it's time to present to the world, jules and James' three rules for scientific bloggers and twitterers:

1. There are no rules.
2. See 1.
3. You have the right to remain silent, but anything you do say will be taken down and may be used in evidence.

Rule 3 brings us on to the other big bogey-man which was, remarkably, that some people seemed genuinely concerned about getting sued. Honestly. As if anyone actually cares what inconsequential drivel you write on your personal corner of the internet. Of course it wouldn't be sensible to libel anyone, but that's nothing to do with blogging per se, but rather a matter of ... being careful not to libel anyone, irrespective of the format. You should remember that you are effectively standing in public shouting with a megaphone - most of the time, no-one is listening, but someone could be, and it's the embarrassing bits that will get propagated - like, that someone said that blogging research in progress is a big no-no :-) I would guess that the knee-jerk re-tweeting of a libel might be more of a risk than writing a blog-post on some scientific topic, regardless of whether you are critical or cheer-leading. But it's certainly not a particular issue for scientists that I can see.

The humorous highlight was when one panellist stated that blogging had doubled his citation rate, which he supported by a histogram of his citations over time. I of course immediately assumed (based on the graphic) that he was joking, but it soon became clear that he was being serious.

I may re-post his stats when I have access to WOK next week, but for now a verbal description will have to do. His histogram showed the rapidly increasing quasi-exponential growth in citation rate that just about every mid-career scientist will have seen over the first 10-15 years of their career, though the last few years in his case seemed to showed a distinct plateau. His blogging started shortly before the last year in the increasing phase, and hence (or otherwise) he attributed the final doubling to this. If I was him, I'd be more concerned that his blogging efforts might have hurt his performance sufficiently to kill off the expected subsequent growth in citations, as this seems to me to be a far more plausible interpretation of the data he presented. I don't want to embarrass him too much (hence not naming him, though he'll be easily enough tracked down if you try) but google scholar suggests he hasn't actually published a great deal in the last few years, at least compared to the arbitrary handful of other mid-career scientists of similar overall performance that I bothered to check. Not that I care about his output one way or the other - I'm just pointing out that his claim that blogging gave him a boost is hardly supported by the evidence. Of course there may be an number of alternative explanations for his particular trajectory.

Rant over, I'll demonstrate that I'm also quite capable of posting entirely positive and enthusiastic commentary without snark when the situation deserves it. Jules and I greatly enjoyed the short course on nonlinear time series analysis oven by Reik Donner and Jonathan Donges. I'd seen the former giving a "Young Scientist" prize lecture a few years ago, which at the time was a bit too detailed and I never got round to chasing up the content. The gentler didactic style this time round was much more digestible, even up to the 8pm finishing time. This event was certainly the highlight of Thursday, and one of the highlights of the week for me.

After that ~12h day, we thought we deserved the morning off. I don't think I have ever done 5 full days at a meeting of this nature without missing a session, and I'm not sure it would be sensible to try. Friday always has the feel of a wind-down and this time seemed particularly weak in terms of my interests. It is hard not to feel sorry for the afternoon speakers. We managed a morning run before heading into town for a really good cake. One thing that we have been repeatedly reminded of is that Viennese standards of service are not on the same planet as what we have come to expect in Japan. But once you get used to that, it's pleasant enough really.

In to the conference for the afternoon, there wasn't actually a great deal I could focus on. I wrote most of this post in the climate data homogenisation session, which had a few interesting things, including rather bizarrely a talk on the periodicity (or otherwise) of Dansgaard-Oeschger events. I wonder what the satellite calibration guys made of that!

Then off to Salm Bräu for probably the best meal of the week. I've been pleased with how well the Tripadvisor recomendations have turned out. Next Sunday's half marathon may suffer as a result, however.

Comments welcome?

Following the rip-roaring blockbuster success of the comment on Schwartz, Jules wanted a taste of the action so we did a more detailed analysis and critique of Chylek and Lohmann's attempt at estimating dust forcing and climate sensitivity using paleoclimate data. Of course we sent the comment to GRL first, and after one round of review (including reply from C&L) the editor decided he couldn't be bothered dealing with it any more. Seriously, his email simply said that it wasn't worth his time and that of the reviewers to deal with a revision. Regular readers will recall that I've had trouble in the past with GRL editors finding feeble excuses to avoid dealing with comments on wrong papers, but even so this seemed rather extraordinary. It may not be immediately obvious to non-scientists, but re-reviewing is generally a much less onerous task than the first round, as it only requires the reviewers and editors to check that any significant criticisms have been dealt with and spurious comments have been rebutted. Needless to say there were basically none of the former and the latter would have been easily dealt with. Contrary to their official written policy, the way that AGU journals deal with comments seems to be to use one reviewer suggested by each of the comment and reply authors. Thus however compelling the case may be, there is generally one reviewer predisposed to be sympathetic to the original paper and the editor can often find sufficient doubt to block publication of the comment if he is so minded. In our case, one reviewer was strongly supportive, and the other tried his best to defend the original paper but that's hardly a viable position to take and in doing so he clearly evaded the main issue. Maybe we would have fared better had we enlisted a couple of eminent co-authors...

Anyway, after some brief thought we soon realised that Climate of the Past would be an ideal venue for the paper. So we sent the manuscript, very lightly edited for the different format and in light of the few substantive comments that had been received, and included the original reviews (after checking the GRL editor agreed) along with our responses to inform the CP Editors as to the history of the piece. The paper took some time to be posted up but now can be seen along with the first reviewer comment which seems if anything even more sharply critical of the original paper than we were.

Quite by chance, a couple of weeks ago Jules spotted this comment and the response which had just appeared in GRL. Obviously this had been going through the review system at the same time as our manuscript was at GRL (their first submission predates ours by about a month), but we didn't know anything of it. It is interesting to see that they have a set of entirely unrelated criticisms, illustrating that while the truth is generally constrained to a single path, there are any number of ways of being wrong. We'll surely refer to it briefly in our revision for CP.

In the meantime we still have another reviewer (maybe two) to wait for, and in stark contrst to GRL's attitude, any other relevant comments are welcome either here or there (best at CPD if you have something that impinges on the content). I'm a big fan of the EGU system where reviews are available to view and the decision making process is out in the open, and think it is especially well suited to comments where reviews may have a tendency to be polarised.

Too crap to publish or too hot to handle?

No, don't answer that :-)

By now probably many of you will have seen the discussion regarding this manuscript. Roger Pielke Snr covers the story here and Fergus has added some commentary here and here. For those of you who are interested in how such an odd triple came to co-author the paper, I'll go into that at the bottom. But first to flesh in some of the details of the submission process:

We first sent it to EOS for their Forum section, which as Roger says, it seems well suited for. Note that this is not a formally peer-reviewed publication in the way that most academic journals are - it is a newspaper, not a research journal (their own choice of words). The manuscript sat on the editor's desk for an astonishing 4 months, and Fergus' occasional polite enquiries were fobbed off rather abruptly, until eventually we got a brief rejection email from Fred Spillhaus on the grounds that they wanted to focus only on science, not opinions (it took him 4 months to work out that it was an opinion poll?). He has not replied to any further requests for clarification as to how he squares this explanation with the stated policy:

Forum contains thought-provoking contributions expected to stimulate further discussion, within the newspaper or as part of Eos Online Discussions. Appropriate Forum topics include current or proposed science policy, discussion related to current research in our fields especially scientific controversies, the relationship of our science to society, or practices that affect our fields, science in general, or AGU as an organization. Commentary solely on the science reported in research journals is not appropriate. [my italics]

But with all emails to him simply vanishing into a black hole, it soon became clear there was no point in pursuing that route any further. Anyway, by this time the survey results had been spotted by some sharp-eyed journalists, and it was getting mentioned in various places (such as here and here). So Fergus then approached the Nature Climate Feedback blog, asking if they were interested. Olive Heffernan replied that they weren't open to guest bloggers, but that he should send it to Nature Precedings and after it had appeared there she could write about it herself.

Nature Precedings is basically a non peer-reviewed preprint service (maybe a rival to the Arxiv?) that merely screens for "relevance and quality". So it was rather a surprise to get a one-line rejection that they were "unable to post your document at this time". The email was anonymous and the author(s) did not explain whether it was because they considered the manuscript irrelevant, or rather than it was too poor quality, or both. Only a minuscule proportion of scientific papers get mentioned in the press, and as I've mentioned this has been picked up in a few places despite having no PR, so it is apparently relevant to some. Obviously I'm biased but the quality of the work and presentation seems well up there with the typical middle-of-the-road conference presentation/poster type of thing (remember we aren't talking high-impact peer-reviewed journals here, just a preprint server).

So it seems that no-one wants to publish it, and no-one wants to say why...with Fergus moving on to other things, it seems like we are at a dead end.

As for my participation in this:

The first I heard about it was an email from Fergus (who I know via his blogging, but not otherwise) asking for comments on his proposed poll. I was generally supportive of the idea and offered some suggestions on the questions and format. I also participated in the poll (FWIW I was a 5: although I can see some arguments leaning towards 4 and 6, they are IMO not strong enough to justify actually choosing one of these options, even as a half point). Later on, he sent me the manuscript again asking for comments, and I suggested some edits. It was around this point that the question of co-authorship was mentioned, and although my contribution had been rather minor the other two seemed keen to include me and I was happy to accept. I have certainly known co-authors do less work (though not on papers where I was first author)!

One can always quibble over details of the wording, but IMO the questions are clear enough, the set of scientists polled is very reasonable and carefully controlled (due entirely to Fergus' hard work) and the results are written up fairly and accurately. Indeed I think it stands in striking contrast with the previous survey of Bray and von Storch, where the questions were more ambiguous (how much do you agree with the sentence "Climate change is mostly the result of anthropogenic causes"?) and the survey was open to anyone who found out about it, including the entire readership of the "climate sceptics" mailing list. Of course the main weakness is in the response rate of ~10%: that leaves open the possibility that the 90% non-responders were either all firmly suportive of the IPCC and saw the poll as a bit of irresponsible trouble-making that didn't justify a response, or all so thoroughly alienated and marginalised by the IPCC that they don't have the energy to grumble about it. Personally, I think the first of these is much closer to the truth, but it seems we will never know for sure. Of course, all surveys suffer from this problem to some extent. I bet all the current polls on Clinton vs Obama have enough refusals to completely dominate the result, were they all to end up on one side of the fence. Yet you don't see reports saying "Clinton 22%, Obama 24%, and the other 54% slammed the phone down".

Reply

In her comment on my earlier post, Julia Hargreaves (henceforth JCH) makes a number of points regarding the peer review process, illustrating her points with reference to some specific examples.

I am pleased to note that JCH agrees strongly with point 2 in my original article. That is, the current Comment and Reply system is largely broken, due in no small part to the obvious hostility that some journals (at least) exhibit towards them. While one can often make a good case for simply ignoring a poor paper, there are also occasions when a direct rebuttal - or even a public debate to clarify points of reasonable disagreement - is appropriate.

As for her point about choice of reviewers, I remind her that we have both certainly had experiences of having our manuscripts reviewed exclusively by those on our lists of suggestions, the reviewers having waived their anonymity in these cases. Of course, I do try to suggest reviewers who I consider to be independent and authoritative, but the Editor must also be prepared to take some responsibility for ensuring this.

In the case that JCH refers to where a paper I rejected was published anyway, I remind her that the decision to publish was not mine, and reviewers are not infrequently over-ruled. It turns out that in their revised version the authors did address my main criticism, but substantial problems remain in their work. While with hindsight it may have been a mistake for me to decline to re-review the paper, I consider it harsh to blame me for the decision I played no part in making, other than to oppose. I am confident that any number of competent reviewers would have easily identified the problems, in fact two of them have already dismissively cited the paper as an example of an obviously flawed approach.

I realise that the job of the editors is a difficult one, especially in a highly politicised field where not all participants are primarily interested in the truth. It will not be easy to filter out the rubbish without rejecting the genuine science, especially in cases where the latter presents a valid challenge to an existing status quo. I agree it is probably better to accept there will always be some of the former that gets through, than reject too many of the latter. But in this case, and irrespective of blame, the journals have a responsibility to allow corrections to be made.

JCH does not comment on the issue of open review, which I consider to represent a significant improvement over the standard approach. I am in no doubt that many (maybe all) of the obviously flawed papers that the AGU has recently published would have been easily weeded out by such a process (or, to take JCH's generous approach, they would at the very least have had to account openly for the limitations in the methods employed, which would probably have weakened their conclusions to the point of irrelevance).