I've always been a fan of the EGU open review process, so it's a bit disconcerting to see a number of dodgy papers being published recently.
First we have the Makarieva et al stuff on ACP. This got strongly criticised in review and on a number of blogs. There were several long threads on Judith Curry's blog, and she said in her public comments that the manuscript required significant changes before it could be considered publishable. She was notably more reticent when Willard asked repeatedly whether she thought the published paper had fixed the problems. The editor implies pretty clearly that they think the theory has major problems but deserves a hearing. In some previous cases, heavily debunked and criticised papers have been rejected, partly with the explanation that the argument is already out in the open for all to see. (Update: the particular example I was thinking of, but had forgotten the details, was specifically the previous Makarieva manuscript on the same topic, which went to full editorial board appeal.) It is hard for me to see how the final manuscript addresses the major criticisms levied at it, though it is not my area of expertise.
ESD had a bit of a disaster with this paper. It is clear that the editor had her doubts, but the very limited reviews included a very positive (albeit partisan and nonsensical) one. I know from personal experience that finding reviewers is often a difficult task. The flip side is, that if several reviewers give generously of their time and effort to provide intelligent and useful reviews, they will be unimpressed by an editor who over-rides their input, as has apparently happened in these cases. There's a relevant manuscript under review here.
Another odd CP paper appeared not so long ago, despite a succession of hostile reviews. The Editor Eduardo Zorita has just broken cover and presented his view of what happened (perhaps provoked in part by a poorly-aimed jibe here a couple of days ago). Irrespective of the ins and outs of the tale, I'm a bit surprised by his claim to be satisfied at the way the process has turned out. As editor, he ultimately carries the can for the decision to publish, and the paper seems obviously silly to me, so I don't think it reflects very well on him that he approves of its publication. The paper presents an 18-parameter (yes really) fit to a highly smoothed time series, and concludes:
First we have the Makarieva et al stuff on ACP. This got strongly criticised in review and on a number of blogs. There were several long threads on Judith Curry's blog, and she said in her public comments that the manuscript required significant changes before it could be considered publishable. She was notably more reticent when Willard asked repeatedly whether she thought the published paper had fixed the problems. The editor implies pretty clearly that they think the theory has major problems but deserves a hearing. In some previous cases, heavily debunked and criticised papers have been rejected, partly with the explanation that the argument is already out in the open for all to see. (Update: the particular example I was thinking of, but had forgotten the details, was specifically the previous Makarieva manuscript on the same topic, which went to full editorial board appeal.) It is hard for me to see how the final manuscript addresses the major criticisms levied at it, though it is not my area of expertise.
ESD had a bit of a disaster with this paper. It is clear that the editor had her doubts, but the very limited reviews included a very positive (albeit partisan and nonsensical) one. I know from personal experience that finding reviewers is often a difficult task. The flip side is, that if several reviewers give generously of their time and effort to provide intelligent and useful reviews, they will be unimpressed by an editor who over-rides their input, as has apparently happened in these cases. There's a relevant manuscript under review here.
Another odd CP paper appeared not so long ago, despite a succession of hostile reviews. The Editor Eduardo Zorita has just broken cover and presented his view of what happened (perhaps provoked in part by a poorly-aimed jibe here a couple of days ago). Irrespective of the ins and outs of the tale, I'm a bit surprised by his claim to be satisfied at the way the process has turned out. As editor, he ultimately carries the can for the decision to publish, and the paper seems obviously silly to me, so I don't think it reflects very well on him that he approves of its publication. The paper presents an 18-parameter (yes really) fit to a highly smoothed time series, and concludes:
"It shows that the climate dynamics is governed at present by periodic oscillations."
35 comments:
I recently had a paper that I strongly recommended be rejected for publication get accepted instead.
The reason: "Not enough papers on XXX being published in this journal and we'd like to recommend more."
So there you have it... there aren't enough papers being published in science already. Needs more papers, or cowbell, or both.
Make that "we'd like to *encourage* more"
Yes Carrick, same thing happened to me. This is why quality is going down. Too many journals and too much pressure to publish. The default should be a research report every year and if its really new a paper. But who am I to criticize my betters?
18 parameters. Heck, that's enough to fit a whole herd of de-extincted mammoths! :)
Don't forget PrimaKlima which has been the best on this (yes, it is in German, but the google does a passable job)
If he has accurately characterized the situation, it does seem to me that Eduardo has a bit of a point---the problem here is with the reviewers not appropriately incorporating the public comments, which is their responsibility and not that of the editors.
Perhaps they could fix this by adding more reviewers on "controversial" papers? We do this sometimes in other fields.
I also see a lot of hand wringing by people who don't understand the process. If Eduardo went over the heads of his primary reviewers, well he has the authority to do this, but it doesn't seem like that's what's happened.
Seems more like the process needs to be improved, including communication to non-primary reviewers what their role is. There appeared to be a bit of confusion on that point too.
And things that might hit the fan...
http://www.earth-syst-dynam-discuss.net/4/25/2013/esdd-4-25-2013-discussion.html
Funny thing that Carrick, Eduardo didn't even insist that Luedekke et al make the changes HE called for in his summary.
You appear awfully forgiving in this case.
Eli: Funny thing that Carrick, Eduardo didn't even insist that Luedekke et al make the changes HE called for in his summary.
Has that EVER happened in a paper you've written? There's always back and forth in my experience...
You appear awfully forgiving in this case.
Forgiving? No... sympathetic.
Rather, one might suggest you have a vendetta against Eduardo that preexists this paper....
I guess I do tend to have sympathy for the people who are filling the role of the editor, as long as they don't abuse their role. They get to put up with a lot of flack for basically very little in return, other than kicked in the teeth when a politically unpopular paper such as this one gets published.
Where they lose my sympathy is when they "game the system" to push through their own views over those of the referees. IMO, it's the referees job to act as gatekeepers, and the editors job to ensure they play fair. When the editor starts cheating and bending rules is when you have real problems develop.
The fact this got published points to a need to change their system, though it will always be the case that bad papers make it through. I just don't see a need to personalize this.
As it happens, like so much out there, I think Lüdecke's paper is dreck. So poor, it's not even worth my spending time on.
The real test of a bad paper is whether people who "like the results" will repudiate it or not, rather than how people who "don't like the results" will react. If skeptics "like" the paper and won't admit that it lacks merits, that should make you happy.
Well the thing that puzzles me the most is that Zorita thinks the paper is worth publishing. Ultimately, it is his decision to accept or reject, and he would not have had to look hard to find serious problems with it.
But rather than admitting that his decision was in error, he insists that everything is just fine...ultimately, this reflects on his judgement.
Carrick, the reviewers are responsible for the contents of their reviews, but the editor is responsible both for all decisions about which reviewers to use (and which to ignore), and also regarding the ultimate fate of the manuscript.
Ludecke wrote a rather bad and silly paper, but Zorita chose to accept it for publication in CP.
Carrick's last comment was cross-posted with mine.
Carrick, I think you misunderstand the role and responsibility of the editor. It is made explicitly clear on the relevant web page of CP:
"The editor has complete responsibility and authority to accept a submitted paper for publication or to reject it. The editor may confer with other editors or referees for an evaluation to use in making this decision."
Note: "complete responsibility and authority" and "may confer".
It is not the editor's job to act as a meta-referee between authors and reviewers. It is the editor's job to judge whether the paper should be published. Do you think Zorita made a good decision here?
I agree that some tolerance of mistakes is appropriate. But, it's also important to recognise mistakes when they are made.
Eli: Funny thing that Carrick, Eduardo didn't even insist that Luedekke et al make the changes HE called for in his summary.
Carrick: Has that EVER happened in a paper you've written? There's always back and forth in my experience...
The ones that got published. C'mon, everything.
Eli so you've never seen an editor let a paper through without all of the changes he originally asked for? You must not review very many papers.... authors can effectively say "take it or leave it" even to an editor, I've done it and it works.
James, I agree that Lüdekke's paper shouldn't have been published, but if an editor can choose to reject or accept papers without input from the recommendation of reviewers, what exactly is the role of the reviewers for then? Why bother?
It seems like you are calling for the editors to act capriciously, but that doesn't seem like a very good strategy for improved publication quality. (They do act capriciously already, already don't encourage it. :-P)
If Eduardo is being honest in his characterization of the reports, at least part of the fault lies with the reviewers not providing an appropriate review. I haven't spent much time on the paper, but it is of that category of "what isn't wrong isn't interesting" category. Which does pose the question of "what were the reviewers thinking?" I just don't understand how it could have gotten favorable enough reviews to allow it to be published, without the editor acting capriciously.
I think Mudelsee shares a bit of the fault here too. I will pass this off as him not understanding how the public comments are treated in CP, which he has a responsibility to understand—otherwise "crybaby" might be a better word choice—since he had the opportunity to be a primary reviewer and declined. (And yes if you want to be part of the review system, you have a responsibility to understand how it works.)
I'm also a bit baffled with all of the energy that is being spent over one bad paper, there must be a subtext that I'm not getting. There are so many bad papers, why is this particular one acting as a lightning rod? Makarieva is beyond poor, it's make-up-new-physics bad. At least Lüdekke didn't invent new, wrong physics. Where is the heat over that one?
I'm not surprised that bad papers make it through occasionally. If you're going to set up rules where the editor isn't supposed to act capriciously, then IMO that's bound to happen.
Carrick: If Eduardo is being honest
Eli calls for an engineering level review of the reviews. Audit! Audit!
As to what the editor wants, the editor usually gets that. You can push back on the referees (all except the third one), but the editor, as James says, has the final word.
Eli: As to what the editor wants, the editor usually gets that. You can push back on the referees (all except the third one), but the editor, as James says, has the final word.
Depends on what he wants more, don't you think? You're saying, with your skills at persuasion, you've never been able to "bend" an editor's position to a "more reasonable one"?
Besides, everything's negotiable, if you have something valuable to trade for it (otherwise it's groveling, not negotiating ;-)).
Carrick,
if an editor can choose to reject or accept papers without input from the recommendation of reviewers, what exactly is the role of the reviewers for then? Why bother?
The reviewers are people who the editor asks for advice. No more, no less. The editor is in charge, and can accept or reject their advice as they see fit.
Magnus Westerstrand, I suggest that you submit an interactive comment on my discussion paper that I can respond to your criticism.
Carrick, I think there has been a huge amount of discussion regarding the Makarieva paper, I only linked to one thread on JC but I think there are about 4 in total with vast comment threads, it was also covered on several other blogs.
Maybe Lüdecke isn't worth much time, but it's always a bit random as to what catches the attention of the internet, and it's not every day that the editor responsible gets involved in such a public manner. I note in passing that you have commented 7 times on this short thread alone, which makes you the most prolific contributor here :-)
did anyone get pictures of the editorial meetings? http://kotaku.com/latest-japanese-schoolgirl-trend-fake-dragon-ball-atta-460482170
Carrick,
I think there is some interesting history with the Ludecke paper. The first time I heard mention was about a year-and-a-half ago when Judith Curry decided to promote a couple of Ludecke's self-published articles on her blog - these have subsequently merged into this single CPD paper.
That posting prompted Richard Tol to speak out on Twitter, declaring that Curry was spreading disinformation. There followed a series of five or more posts in which she attempted to defend herself from charges of disinformation, then provided a platform for Tol to critique and Ludecke to respond.
Another factor is that Ludecke is affiliated with EIKE, which is apparently a German equivalent to Heartland. Presumably in Germany and surrounding lands the paper has been publicised by news organisations and politicians with links to EIKE.
Makarieva, did Eli hear someone mention Makarieva
James: Maybe Lüdecke isn't worth much time, but it's always a bit random as to what catches the attention of the internet, and it's not every day that the editor responsible gets involved in such a public manner. I note in passing that you have commented 7 times on this short thread alone, which makes you the most prolific contributor here :-)
I do try to keep things lively when I can. ;-)
While I'm not very interested in Lüdecke, Eduardo's response raises interesting "meta" questions, because of what it suggests about the editorial process.
I agree that the editor can choose to ignore a particular reviewer's advice, but what happens when all of the reviewers give the same advice? The editor still has the authority to ignore these (seen it happen), but that's the "capriciousness" issue I was raising. (Don't encourage them!)
Anyway, my response about Eduardo's position was "he had a bit of a point", which is different than "he had a very strong point". Meaning only that I can see how he came to the decision he did based on his perception of the facts, not that I think this was the best decision based even on his perception of the facts.
Without wasting much time reviewing this paper, it does seem that the methodology used in Lüdecke is okay (as Tamino says, he basically manages to show you can describe an arbitrary time series with a Fourier series, possibly neglecting the "set of measure zero" clause).
When I see entire "industries" built on i>fundamentally flawed —in other words successful execution is impossible—Lüdecke just seems like such a minor glitch in the system.
Paul S, thanks for the context...I admit I have spent very little time reading Judith's blog. I could see there was a lot of energy over this paper, I genuinely had managed to avoid stepping in it, till now.
One weakness of the current EGU system is that the review process goes underground after the first phase. So we have no way of knowing whether the reviewers recommended acceptance, or argued for rejection, or just said "stop sending me this crap, I've wasted enough time on it already". I suspect a fair bit of cranky stuff gets the latter treatment. Wrestling with a pig comes to mind (you both get muddy, the pig enjoys it).
But Zorita certainly seems to have bent over backwards to get it published, and that's what puzzles me, because it is his name, and his reputation, rather than that of the anonymous reviewers, that is permanently tied to it.
Pehr, thank you for the invite however I do not think it is worth my time. We already covered much after your ridiculous claim about a climate sensitivity of around 0,6... and it is clear you can not take things in as shown about our discussion were you claimed that the CO2 increase could be from natural sources...
in Swedish (do not know if google translate can make it understandable)
http://uppsalainitiativet.blogspot.com/2010/10/stockholmsinitiativets-forvirring.html
http://uppsalainitiativet.blogspot.se/2010/10/pehr-bjornbom-och-carl-gustaf-ribbing.html
http://uppsalainitiativet.blogspot.se/2009/06/stockholmsinitiativets-vetenskap.html
Magnus Westerstrand, citing a published research work stating some results about for example climate sensitivity is not the same as claiming this result on once own behalf. The variations in CO2 mixing ratio connected to natural causes such as the decrease after the Pinatubo eruption and the variations due to ENSO oscillations are well accepted in the scientific community.
Furthermore, your blog posts from 2010 and 2009 are not relevant for my discussion paper in ESDD. I am a little astonished that you say that you have no time for commenting on ESDD but when it came to writing the blog posts obviously there was time enough.
Carrick: Without wasting much time reviewing this paper, it does seem that the methodology used in Lüdecke is okay (as Tamino says, he basically manages to show you can describe an arbitrary time series with a Fourier series, possibly neglecting the "set of measure zero" clause).
Please, Carrick, GEABOAFB. You learn this in freshman calculus at the latest. But you are right, the paper is of Lilian Hellman class, so the question becomes WHY did the "honest broker" push it through. Why did he even take it to the second level of review which gave cover to the eventual publication?
Google says:
"No results containing all your search terms were found.
Your search - GEABOAFB - did not match any documents."
Hint: Give Eli a bit of
FWIW, the case I was thinking of in the first para ("In some previous cases, heavily debunked and criticised papers have been rejected, partly with the explanation that the argument is already out in the open for all to see") was the earlier Makarieva attempt at the same topic.
http://www.atmos-chem-phys-discuss.net/8/17423/2008/acpd-8-17423-2008-discussion.html
I've updated the post to put in the link. So it's even more surprising that a policy that was obviously discussed internally in some depth, was apparently ignored in this case. That's assuming one accepts that the newer paper is a re-hash of the previous that does not solve its problems.
James: But Zorita certainly seems to have bent over backwards to get it published, and that's what puzzles me, because it is his name, and his reputation, rather than that of the anonymous reviewers, that is permanently tied to it.
That is a good way to put it. By pushing the paper through like this, you do take "part ownership".
Eli, I'll leave guessing other people's motivations to you.
It really is stunning to me that they published Makarieva's work in any form.
Carrick, guessing the obvious ain't so hard, but OTOH the problem Eli has always had with Makarieva is that people didn't trust their lying eyes but rather enjoyed mutual mathturbation, which is exactly the same thing with Luedecke.
Pehr it is relevant for the discussion in the paper. And no I do not have the time, lets hope the reviewers does the job.
And that you try to sneak out of responsibility for the 0,6 degree sensitivity when you first do not have a published article that gives that and secondly still use it to say that with such a low sensitivity we do not have much to worry about... just shows how meninges it is do discuss things with you. Just as the occasions when you try to argue that the increase in CO2 is from natural sources. When basic stuff like that goes past you I see no meaning to continue the discussion.
Magnus Westerstrand, you seem to confuse my comments in the climate debate in Sweden with my interest for climate science as a hobbyist. I am not associated with a political party, as you are in being a spare time politician (KD), but I have sometimes some opinions on Swedish climate policy in my right as an ordinary citizen. Although you misrepresent my views it would be OT to elaborate on the Swedish debate on climate policy in this thread.
Note that the results in my discussion paper do not give support for a low climate sensitivity of 0.6 C in the long run. I have found what could be interpreted as a climate feedback parameter value of around 6 W/(m2 K) for limited time periods. However, my results also suggest that this parameter value may change due to distinct changes in the state of the climate system, for example the change seen in mid-2006 in the relation between the temperature anomaly and the radiative flux anomaly.
Furthermore, the new paper of Armour et al. (2012) give more evidence for the difficulties to relate the climate feedback parameter from observations to sensitivity. On the other hand their new theoretical results on feedback also offer an explanation to my results both concerning the phase shift between the oscillations in the temperature anomaly and the radiative flux anomaly and concerning the high value of the climate feedback parameter according to the phase plane plot with lag. I have discussed this in a recent author comment in the interactive discussion:
Earth Syst. Dynam. Discuss., 4, C137–C143, 2013
www.earth-syst-dynam-discuss.net/4/C137/2013/
I really think discussion of Pehr's paper and/or other views belongs elsewhere.
(/dev/null might be one suitable destination...)
Post a Comment