BlueSkiesResearch.org.uk: Like a phoenix redux

Even odder than finding that our old EnKF approach for parameter estimation was particularly well suited to the epidemiological problem, was finding that someone else had independently invented the same approach more recently…and had started using it for COVID-19 too!

In particular, this blogpost and the related paper, leads me to this 2013 paper wherein the authors develop a method for parameter estimation based on iterating the Kalman equations, which (as we had discovered back in in 2003) works much better than doing a single update step in many cases where the posterior is very small compared to the prior and the model is not quite perfectly linear – which is often the case in reality.

The basic idea behind it is the simple insight that if you have two observations of an unknown variable with independent Gaussian errors of magnitude e, this is formally equivalent to a single observation which takes the average value of the two obs, with an error of magnitude e/sqrt(2). This is easily shown by just multiplying the Gaussian likelihoods by hand. So conversely, you can split up a precise observation, with its associated narrow likelihood, into a pair of less precise observations, which have exactly the same joint likelihood but which can be assimilated sequentially in which case you use a broader likelihood, twice. In between the two assimilation steps you can integrate the model so as to bring the state back into balance with the parameters. It works better in practice because the smaller steps are more consistent with the linear assumptions that underpin the entire assimilation methodology.

This multiple data assimilation idea generalises to replacing one obs N(x_o,e) with n obs of the form N(x_o,e*sqrt(n)). And similarly for a whole vector of observations, with associated covariance matrix (typically just diagonal, but it doesn’t have to be). We can sequentially assimilate a lot of sets of imprecise obs in place of one precise set, and the true posterior is identical, but the multiple obs version often works better in practice due to generating smaller increments to the model/parameter samples and the ability to rebalance the model between each assimilation step.

Even back in 2003 we went one step further than this and realised that if you performed an ensemble inflation step between the assimilation steps, then by choosing the inflation and error scaling appropriately, you could create an algorithm that converged iteratively to the correct posterior and you could just keep going until it stopped wobbling about. This is particularly advantageous for small ensembles where a poor initial sample with bad covariances may give you no chance of reaching the true posterior under the simpler multiple data assimilation scheme.

I vaguely remembered seeing someone else had reinvented the same basic idea a few years ago and searching the deep recesses of my mind finds this paper here. It is a bit disappointing to not be cited by any of it, perhaps because we’d stopped using the method before they started….such is life. Also, the fields and applications were sufficiently different they might not have realised the methodological similarities. I suppose it’s such an obvious idea that it’s hardly surprising that others came up with it too.

Anyhow, back to this new paper. This figure below is a set of results they have generated for England (they preferred to use these data than accumulate for the whole of the UK, for reasons of consistency) where they assimilate different sets of data: first deaths, then deaths and hospitalised, and finally adding in case data on top (with some adjustments for consistency).

The results are broadly similar to mine, though their R trajectories seem very noisy with extremely high temporal variability – I think their prior may use independently sampled values on each day, which to my mind doesn’t seem right. I am treating R as taking a random walk with small daily increments except on lockdown day. In practice this means my fundamental parameters to be estimated are the increments themselves, with R on any particular day calculated as the cumulative sum of increments up to that time. I’ve include a few trajectories for R on my plot below to show what it looks like.

Social Nonscience again

So, prompted by Doug McNeall's tweet, I went and read that much tweeted (and praised) paper by Iyengar and Massey: "Scientific communication in a post-truth society". My expectations weren't high and it was just as bad as I'd feared.

It starts off with a encouraging abstract:

"Here we argue that in the current political and media environment faulty communication is no longer the core of the problem. Distrust in the scientific enterprise and misperceptions of scientific knowledge increasingly stem less from problems of communication and more from the widespread dissemination of misleading and biased information. [...] We suggest that, in addition to attending to the clarity of their communications, scientists must also develop online strategies to counteract campaigns of misinformation and disinformation that will inevitably follow the release of findings threatening to partisans on either end of the political spectrum."

Great. They realise that the problem is not because scientists communicate badly. It's long been obvious to many of us that there are lots of excellent public communicators in science, certainly within climate change. Some are excellent at both research and communication, some make more of a career out of the communication than the science, and that's fine too. Blaming scientists has long been a lazy excuse by those who should know better.

And even better, these social scientists have a recommendation! They actually have a proposal for how to break the policy logjam. Us scientists should "develop online strategies to counteract campaigns of misinformation". Yes! Let's do that! Though my spidey senses are tingling a bit, is this really the scientists' job? We do research and communicate it, I'm not really sure our expertise is in developing communication strategies in an adversarial environment. Sounds to me like that might be a whole new area of research in itself. Well never mind, let's see what they are actually recommending.

[...reads on through several pages of history and analysis relating to scientific communication in a post-truth society, which is interesting but hardly news...]

On to the section entitled "Communicating Science Today". At last, they are going to explain and expand on their recommendation. Aren't they?

Here is the last paragraph in full

"At this point, probably the best that can be done is for scientists and their scientific associations to anticipate campaigns of misinformation and disinformation and to proactively develop online strategies and internet platforms to counteract them when they occur. For example, the National Academies of Science, Engineering, and Medicine could form a consortium of professional scientific organizations to fund the creation of a media and internet operation that monitors networks, channels, and web platforms known to spread false and misleading scientific information so as to be able to respond quickly with a countervailing campaign of rebuttal based on accurate information through Facebook, Twitter, and other forms of social media. Of course, this is much easier said than done, and — given what research tells us about how the tribalization of US society has closed American minds — it might not be very effective."

Oh congratulations. The authors have invented groups like Sceptical Science and RealClimate. Sadly, they don't seem to realise that the scientists are more than a decade ahead of them. Granted, they do seem to be talking about something a bit more grandiose than those sites but it would be nice if they'd had some awareness of what was already going on, and perhaps offered some sort of useful critique.  They seem to be moving from a position of blaming scientists for not communicating adequately, to blaming them for not inventing some sort of magical unicorn for which they have no roadmap and which, they admit, probably wouldn't work even if it could be created. This is progress?

"...because the stakes are so low"

In an earlier blog I alluded to the fact that someone had written a paper on ensemble means. I didn't think it was particularly interesting or useful since we had written a more correct and complete analysis some years previously (as blogged variously here here here). The particular point that the author Bo Christensen got most obviously wrong was his claim that the CMIP ensemble behaved like it was sampled from a high-dimensional space. In fact, as we showed via a variety of analyses, it behaves much more like it was sampled from an intermediate-dimensional space. Specifically, we found an effective dimensionality of about 3-11 depending on the details of the analysis. On top of getting that wrong, it didn't seem like his analysis actually added anything of significance to our results. Furthermore, Bo's analysis relies on the ensemble being perfect in the sense that the verifying observational data set (ie reality) is drawn from the same distribution as the ensemble members. We already demonstrated that the good performance of the ensemble mean does not depend on this very strong (and generally false) condition.

Amusingly, when I commented critically on his analysis after his EGU talk, he assumed I was Jonty Rougier, who seems to have been a dissatisfied reviewer of the paper (Jonty had also extended and improved on our work in some ways back in 2016).

So far, so boring, although it's a bit frustrating to see our own work so comprehensively misunderstood and denigrated. Bo did cite our paper, and Jonty's too, so was not unaware of them, but claimed (without much justification) that their explanations were not compelling.

What prompted me to bother writing it all down now is that a comment/reply pair has just appeared in Journal of Climate (full versions on sci-hub for anyone who cares). The comment is from Jonty Rougier of course. It's good to see that academic politics continues in our absence...

More D&A and FvB.

By chance I happened to notice another paper with an interesting title appearing in Climatic Change on the very same day as the recent Mann et al paper: Is the choice of statistical paradigm critical in extreme event attribution studies? While my noticing it was fortuitous, the publication date was no coincidence, as it was clearly intended as a "comment on" in all but name. I am not particularly impressed by such shenanigans. I know that Nature often publishes a simultaneous commentary along with the article itself, but these are generally along the lines of a sycophantic laudation extolling the virtues of the new work. The climatic change version seems to be designed to invite a critical comment which does not provide the authors under attack any right to reply. Jules and I were previously supposed to be a victim of this behaviour when we published this paper. However the commentary never got written, so in the end we suffered nothing more than a lengthy delay to final publication.

Anyway, back to the commentary. Is the choice of statistical paradigm critical? I can't really be bothered discussing it in any detail. The arguments have been hashed out before (including on this blog, e.g. here). The authors provide a rather waffly defence of frequentist approaches without really providing any credible support IMO, based on little more than some rather silly straw men. Of course a badly-chosen prior can give poor results, but so can an error in your computer program or a typo in your manuscript, and no-one argues that it's better to just take a guess instead of doing a calculation and writing down the answer. Well, almost no-one.

Encyclopedia of Geosciences

As per title. A work in progress (well, hardly started) so I wouldn't want to be too critical. A prize for the first person to find out where “climate change” is located.

That's all for now :-)

Open review vs open access.

Now that Boris Johnson has finally Godwinated the EU referendum debate I don't need to discuss that, and can talk about this instead. Though actually, it's about at the same intellectual level as Johnson's comments. Everyone agrees it's junk, the questions are why this stuff gets written, and how it gets published. I won't speculate here on the why (I already did here), just the how.

It has appeared in a new journal of unknown quality, the AGU (partnering with Wiley) seemingly jumping on the open access pay-to-publish bandwagon. It is worth noting, Bates has not previously published there, he didn't choose it out of familiarity or convenience, but had (apparently) been shopping around since at least 2014 with this idea trying to find a home. Pretty much everything gets published eventually, by the way. It just takes longer if it's either rubbish, or a revolutionary idea that is well ahead of its time.

It's a shame Bates didn't have the guts to try ACP or ESD, where his manuscript would have been shredded by reviewers in short order. It is all too easy for a lazy or overstretched editor at any journal to simply use the author's “suggested reviewers” without also looking for an independent view. At least in the case of the EGU open review process, an incompetent reviewing process is in principle discouraged by the fact that it's out in the open, and there is of course the opportunity for other interested parties to add their views. Score +1 for open review.

New publication model for EGU journals

There's been longstanding debate over the way in which manuscripts are handled by the EGU/Copernicus journals. The current system is that publication fees are charged for submitted manuscripts, which are professionally typeset (hence incurring significant costs) and appear in an on-line “Discussion Journal” like CPD, GMDD etc where they remain as part of the public record (with DOI etc) irrespective of whether a final peer-reviewed paper appears or not in the non-D journal. There are several drawbacks with this approach, though its originators and advocates have always argued that the benefits outweigh these.

Well, it's all changing now anyway. From January, submitted manuscripts will not be professionally typeset, the Discussion journal will cease to exist (though the submitted manuscripts will still be public and citeable), and paper charges will only be applied after the paper has been through peer-review. It will be interesting to see how this affects the journals, and whether for better or worse. As for me, I'm pretty ambivalent about it. Paying charges in advance was always a bit awkward and puts editors in a difficult position when rejecting papers (at GMD, the unofficial policy was to offer a hefty discount on fees for their next submission). On the other hand, perhaps it does dissuade a certain amount of dross - though it apparently didn't help with this nonsense. Anyway, the new system will be much closer to the standard model for journal publication, while retaining the special feature that not only the final manuscript, but also the review process, is on-line and open to all.

GMD raises the bar again

Not content to rest on their laurels, the executive editors at GMD have revised and updated their policies, the new version of which can be found here. Of particular note, it is now a requirement that code must always be made available to the editor at minimum, and each manuscript must include a paragraph outlining code availability. (In practice we expect that most model code will be publicly available.) This represents a logical and incremental change to the previous policy, and once again sets a new standard for open and accessible science.

It is increasingly common for journals to have code and/or data availability policies, whether or not they are enforced. This BAMS paper is also relevant, focussing on the analysis code rather than numerical model itself. 

(Disclaimer: I helped to found GMD and was an executive editor for several years. However I no longer hold this position though I continue to act as a topical editor [which basically means I handle manuscripts but don't set the editorial policy].)

EGU Open discussions

The Hansen paper (submitted manuscript!) has provoked me into mentioning that the EGU are in the process of revamping the discussion phase of their on-line open access publications. There have been a number of complaints over the years that the open publication of the pre-review manuscript can be confusing, as it may lead people to think that these are peer-reviewed publications (which they are not). I've never been particularly convinced by this, and you only have to look at the process to see what is going on, but then again, if people are going to be confused, maybe it is reasonable to wonder if the process could be clarified. A “watermark” on the title page was already added a few years ago, and they have also recently improved the way that the website links and indexes the manuscripts - pointing more clearly to the final paper when available, so people do not read and cite the original version unless they specifically want to.

I'm not sure quite how finalised the new plans are, but at the time of the last EGU meeting one of the proposals was that the authors' own typset manuscript (eg a fairly plain LaTeX template) would be posted up as-is rather than being formatted into the in-house EGU online style. I don't really like the EGU style so that's already an improvement from my perspective. Another potential benefit is that this would eliminate one source of up-front cost which may pave the way for normalising the publiction fee model to a pay-on-(peer-reviewed)-publication model rather than pay-on-submission. There are arguments on both sides of this - pay on submission may reduce the number of poor submissions, but it can be administratively difficult and may lead to both a presumption of acceptance and hard feelings when a paper is rejected.

Ultimately, I don't think these changes will have much effect on the rare cases where people deliberately publicise their submitted manuscrips, as Hansen appears to have done. Most journalists won't understand, or care about, the details of the peer-review process. Nevertheless, I don't have much sympathy for Ken Caldeira's claim that publishing the review process is a form of “pollution” of the peer reviewed literature. No-one sane would routinely read the discussion phase of the journal, and the point behind the publication of these manuscripts is not to double the amount of material that scientists are supposed to read, but to give people the chance both to look at how a particular paper passed through the peer-review process, or to comment on a manuscript that is particularly noteworthy. To that end, I'm happy to see that the Hansen manuscript is well on its way to being the most heavily-commented paper on ACPD, though predictably most of the comments seem to be from crazies. At least they are thereby distracted from commenting on my blog :-)

Peer review as a business transaction

Amusingly hidden behind a paywall as usual (ironic considering that Nature bases its content and profit on the free labour of others, including the reviewers), there's a letter suggesting peer reviewers should be paid. I've found this idea a handy way of turning down reviewing invitations that I'm not really that desperate to do, without appearing too rude or selfish. "Please Dr Esteemed Scientist, we would greatly value your expertise in reviewing this manuscript." "Yes, of course I'd be happy to review it, and my fee will be £X." "Oh, we don't really think your input is valuable after all. " Kthxbye.

Another journal editor resigns!

Regular readers will have noticed that I follow the goings-on at EGU journals with some interest. So in that vein I'd like to point out there have been some recent changes at GMD. Perhaps most notably, our Dear Leader Dan Lunt has stepped down from the position of Chief Executive Editor, which he has held since the journal's inception about 6 years ago. Jules is the incoming chief. (Chief doesn't actually have any extra powers that I'm aware of, but is expected and trusted to take the lead on many decisions with or without discussion.) Bob Marsh has been added to the list of execs - this happened last year actually - having been a topical editor for some time. And...drum roll...I am no longer on the list of execs, though I'll remain a topical editor. All the execs feel that the journal (indeed all EGU journals) should be regarded as community assets rather than personal fiefdoms. So although it made sense to stick with a core team who shared a clear vision though the early years, we realised some time ago that it was time to bring in new ideas and let things evolve a bit. This feeling has been informally formalised though a rough plan to swap execs off the board on a biannual biennial every two year basis - Bob's induction was the start of this, staggered with my resignation to allow a bit of settling in time - and also rotate the chief exec position among the board members. I'm happy to leave the journal in the capable hands of the new board.

Incidentally, it is rumoured that the new Impact Factor for the journal will be approaching 6, up from 5 last year. That should put us even closer to the top of the list for journals in the geosciences! I'm sure that GMD, and all the other EGU journals, will continue to go from strength to strength as the open access movement continues to gain momentum.

First!

For various reasons mostly related to the IPCC rumour-mill, the "hiatus" (seems to be the politically correct term these days) in global temperature is in the news again. Which brings to mind this manuscript which was rejected by GRL a few years ago (and which I just put on the arXiv a few days ago):

Our results indicate cause for concern regarding the consistency between climate model projections and observed climate behavior under conditions of increasing anthropogenic greenhouse-gas emissions.

The analysis was extensively discussed back at the time, and the paper submitted to (and rejected by) GRL at about the same time. From memory, it got quite an involved treatment from the reviewers. Rejection from GRL isn't something I can't get too worked up over. I'm confident that the paper was fundamentally correct, worth publishing, and that it would have had plenty of impact. However, the peer review filter is pretty noisy at journals like GRL with high rejection rates, and decisions can't be parsed too finely. I did subsequently encourage submission to other journals, but for various reasons that didn't happen. Of course it's easy for a minor author to encourage other people to do the work for a new submission :-) In case it isn't already clear, my listing as last author is not an indication that I'm the Machiavellian brains masterminding this nefarious plot to discredit climate models, but instead just a fair reflection on the minor magnitude of my contribution.

3 years later, it seems reasonable to conclude that our main error was merely in being several years ahead of the rest of the field.

Reasons for reviewer anonymity #7,654

An Nth-hand comment from elsewhere, no names no pack drill:

I reviewed a paper a few years ago, and only when it was published did I realise I was a co-author!

I literally LOLed. And by "literally", I mean the literal sense of "literally", not the new aliteral definition of "not literally". Move over "pal review", the new kid on the block is "self review".

Polynomial cointegration tests of anthropogenic impact on global warming

Apparently global warming has been refuted, buy some economists writing in Earth System Dynamics:

Therefore, greenhouse gas forcing, aerosols, solar irradiance and global temperature are not polynomially cointegrated, and the perceived relationship between these variables is a spurious regression phenomenon.

Were I Judith Curry, I would probably be saying "wow" at this stage. Alternatively, it could just be some dross that has accidentally found its way into print after having been rejected at least twice at different journals.

The review comments are interesting, to say the least. Reviewer #2, in particular, seems awfully keen on a number of silly sceptic claims that have been presented in recent years.

I suppose it just goes to show that you can fool at least one person sometimes, and if that person happens to be a journal editor, you're in luck.

I see this Beenstock thing has done the rounds before, though I'm not sure any of the commenters quite get to grips with it.

Multi-Stage Open Peer Review

You might have realised from jules' posts, though not from (the absence of) mine, but I'm back.

Here's a well-written account of the strengths of the EGU's open peer review system, from one of its strongest advocates (and Chair of the Publications Committee). The journals continue to grow steadily and (moderately) profitably, and their success can only be helped by policies such as this. It is worth noting (again) that the publication charges of the EGU journals, which make their papers freely available, are only comparable to those of for-profit journals which then sell on the papers at a huge additional profit. Many paywalled journals do offer open access, but only for a fat additional fee. It is hard to see how they add value to the publication process. Both approaches rely on unpaid reviewers to do the bulk of the (post-authoring) work.

The issue of open review is additional to the open publication, of course. Uli makes strong arguments for the EGU system, including that although the number of unsolicited comments seems low, it is far higher than you get in the traditional journals. However, writing "Comments on" is hardly the same thing as suggesting changes to the design of the figures or even asking the author to cite one's own papers more :-) I do like that he doesn't pull any punches in slagging off the silly Nature designed-to-fail "experiment".

I wanna start a fight

I don't really, but it looks a bit like some people do:

On the determination of the global cloud feedback from satellite measurements:

It is shown that the results of a previous analysis, which suggested a likely positive value for the short-term cloud feedback, depended upon combining all-sky radiative fluxes from NASA's Clouds and Earth's Radiant Energy System (CERES) with reanalysis clear-sky forecast fluxes when determining the cloud radiative forcing (CRF). These results are contradicted when ΔCRF is derived using both all-sky and clear-sky measurements from CERES over the same period.

This reads a bit like a criticism of a previous Dessler paper, and the man himself had plenty to say during the open review process. Which based on my armchair viewpoint, seems to have worked fairly well in this contentious case, which is ranked highly on the "most commented papers" list.  Whether it is the end of the matter, remains to be seen...

Anyway, ESD seems to be turning into quite an interesting journal.

Phew

I'm relieved to say that our paper on the CMIP3 ensemble and its mean has finally been accepted. The final version can be found here, though it seems that the website is not working properly (it is finally being reorganised to match reality, the institute "FRSGC" ceased to exist about 5 years ago and the division "d5" about 3 years ago IIRC). It's not hugely changed from the original and the main results are the same, but it has been improved a bit by the peer-review process. I was a bit unimpressed with how that worked this time around, actually. I already talked about the initial rejection, which was made on the basis of three overtly favourable and one hostile review. We managed to talk round the editor by showing that the rejection was based on a simple misunderstanding, at which point he conceded that we should be allowed to submit a revised version. We did that, and for the next round we had three clear acceptances (I think the paper might have only gone back to the most critical of these three, actually). The hostile reviewer, however, having had his initial complaint refuted, decided this time that our paper was too trivial to publish (using a simple argument that was not actually correct and didn't cover our main results anyway), and the editor asked for "major revision" to deal with this criticism. We complained again, on the grounds that three of four reviewers had already found it interesting enough and that any further revision was hardly likely to change the mind of the 4th (we did also show why their argument failed, though I'm not sure if the editor really believed us). And this got us upgraded to a "minor revision", and finally acceptance.

So in the end it all worked out OK, and certainly a lot more smoothly than this lengthy saga, but I can't help but think it was rather more of a struggle than it needed to be. I suspect that the editor probably knows and respects the 4th reviewer rather better than the others, but that's a bit of a guess. Incidentally, the editor is a very clever guy whose work I admire greatly (though we've not met and he probably doesn't know who I am), so perhaps that is partly why I found the process a bit disappointing. Anyway, I can now move on to the next part of the plan...

More on open access publishing

Some time ago I got a questionnaire from Project SOAP (Study of Open Access Publishing) and now they have various survey data, presentations, and videos of their symposium on their web site. Go and have a look if you're interested. I don't think there are any big surprises in what they found, but it is clear that the open access movement is gathering momentum.

Of course the EGU journals don't stop at open access but also have a fairly open review system too. I say fairly open, as there is still some behind-the-scenes stuff (which may include substantial revision and re-review) between the initial open discussion phase and the final publication. I'm not sure why this part of things is hidden from view, perhaps they have their reasons. It is interesting to ponder how their procedure contrasts with the various attitudes that have been expressed recently about people publishing reviews that they have received. I see no fundamental ethical problem with that in principle, I haven't been particularly tempted to publish them in full but have certainly discussed reviews received in general terms, and I recall Jim Hansen doing the same in the past in one of his email circulars. So long as the reviewers are not identifiable, I don't see how there can be much of a problem with this.

Of course, that should not be interpreted as excusing someone who asks about the identity of a suspected reviewer while promising confidentiality, and then decides to blab about it all over the internet...

E&E caves in

I don't seem to have posted much recently...hope you've been enjoying jules' pics in the interim. Here's a little snippet of interest. Probably most of you know of the threat that Bill Hughes, publisher of possibly the worst journal in the world, E&E, made to Gavin Schmidt: "At the moment, I’m prepared to settle merely for a retraction posted on RealClimate". RC replied robustly, and there was an article in the Guardian about it a few days later.

It now seems that E&E are planning a graceful withdrawal, according to chief editor Sonja Boehmer-Christiansen as reported here:

There is quite a wide discussion about what to do. Multi-science is not rich and I have not time for legal matters!! Most think it is just not worth it. I think we could thank Gavin for the publicity and withdraw our threat, but this decision is not really mine. Benny certainly does not want to sue.

It occurs to me that, at the time he issued the threat, Bill Hughes might actually have not known quite how crap and biased E&E is, but perhaps that's too naive of me. Anyway, it serves a useful purpose as a dustbin for lunatics to publish in, which may reduce their incentive to sneak into real journals (though they still succeed there occasionally, of course). Long may it flourish!

Thanks but...(2)

Another for the annals of useless invitations. It is not rare to receive publishing mill spam - usually from Hindawi, inviting me to publish my esteemed research in one of their obscure and unread journals - on payment of a publication fee, of course. I don't usually take too long pondering their generous offers, to put it politely.

Recently, I got one for a new journal "Natural Science" published by SCIRP, which I had not heard of. Helpfully, their email included a list of current contents. One article in particular caught my attention: On the recovery from the Little Ice Age, by Akasofu.

I think that tells me everything I need to know about their standards of peer review... (more digging uncovers that in fact the organisation seems to have form).