Tuesday, February 08, 2011

Tsukuba

As you will have seen, we recently spent a few days in Tsukuba, which must be my least favourite place in Japan, by quite a distance. It's a horrible windswept boring new town, built about 40 years ago as a "science city" (Nature article here). There are no interesting shops, restaurants or other facilities that I am aware of, just a regular grid of offices and apartment blocks. Going to Tsukuba is about on a par with visiting Milton Keynes. Unfortunately, one of our main collaborating labs is based in Tsukuba, so we have to show our faces there occasionally.

When first built, it was almost impossible to travel to (being in the middle of nowhere, with a poor bus service) and had a famously high suicide rate even by Japanese standards. This web page says the university still has the highest suicide rate in Japan, even though it's supposed to have gone down a lot. A few years ago, the new train line was completed and now the scientists seem happier in the knowledge that they can get whisked straight to the geek heaven of "electric town" Akihabara in under an hour. The train terminus is in the basement of Yodobashi, no less. If our colleagues are representative, Tsukuba also has an unusually high birth rate, so it seems there still isn't a lot to do there :-) I think all 5 of our closest associates have young children, whereas no-one in our institute does. Of course the different employment system (tenure vs short contracts, basically) may also be a factor in that...

Anyway, enough about Tsukuba. The reason we were there for was a workshop on downscaling, which in practice really meant regional modelling and prediction. It's not that much of a focus of mine but it does seem a natural progression from global down to smaller scales. And being an "International Workshop", all the Japanese-side presentations were in English, so it was a good chance to hear what was going on. As well as the Japan-based work (which I'm not really part of), there were quite a few invitees from overseas, who mostly said interesting and useful things. One of them was Roger Pielke Snr. He didn't actually come over, but gave a presentation via videolink, which was described as "Skype" but looked like it might have been something else - Google video chat perhaps? Technically it worked very well and while I accept there are benefits to physical attendance, the time and cost savings of such an approach make it very attractive.

His main theme was that neither statistical nor dynamical downscaling of multidecadal global climate model projections generates any value. The basic reasoning was based on the typology expounded here, where "Type 1" is basically hindcasting or nowcasting forced by observations, and "Type 4" is basically pure modelling as is required for long-term climate projections. Pielke's article explicitly states:
Observational constraints on the solution become less as we move from Type 1 to Type 4. Thus forecast skill will diminish from Type 1 to Type 4.
This final sentence is, of course, completely false, at least when you realise that Type 1 and Type 4 are likely to be used for different purposes (we don't have obs for forcing regional climate models in 2100...). While it is true that model accuracy will (other things being equal) generally degrade as observational conditions are replaced by model outputs, this does not mean that skill will diminish for the simple reason that skill is always a comparative measure against some alternative hypothesis (typically a null hypothesis such as persistence, or climatology, or indeed persistence of climatology, when talking about multidecadal forecasts) and the performance of this null hypothesis will ALSO degrade as we look further ahead. We all agree (including Roger) that stationarity is dead (if it was ever truly alive), and thus the climate in 100 years is likely to be substantially different to that of today, or the next 10 years. In order for Roger's claim to be true, it would have to be the case that the model performance degrades MORE RAPIDLY than the performance of whatever null hypothesis he would use in the absence of the models. Which could in principle be true but he has not, as far as I'm aware, ever attempted to show it. I'm disappointed to see him still playing fast and loose with this terminology about "skill", several years after I pulled him up on it previously.

Lest you think I'm being too technical and pedantic in my usage of the term "skill", it is quite clear from reading what Roger says, that he is also using it in this sense, at least when it suits his purpose. Eg, he explicitly states that as a consequence of his argument, model runs are not suitable for use as projections or even scenarios.

I challenged Roger on this at the end of his talk, and he retorted that the models had yet to demonstrate skill over these time and space scales. I agree with that to some extent, but it's a very much weaker claim than his original one that the models necessarily have no (or at least negligible) skill. It is perhaps debatable to what extent we can consider skill in a rigorous sense, as we aren't dealing with a lot of repeatable experiments like daily weather forecasting, but we are working on it to the extent that is possible. (The analysis presented there only considers global mean temp, because that is all that was available. However it's pretty obvious that if the data had been available, it would have given similar results at the continental scale, since the spatial patterns are very consistent between all models and obs. That's still some way from really fine scale of course, but it's a start.)

10 comments:

Anonymous said...

And again,


http://www.youtube.com/watch?v=UM6yfhFULYI

David B. Benson said...

The "skill" most needed is in projecting regional precipitation [with much more than just annual averages required].

crandles said...

It seems to me that the response to forcing of a few years CO2 accumulation is likely to be small unless something creates some non-linear behaviour.

So it would appear to me to be difficult to demonstrate skill (or lack thereof) over small time and space scales to a general CO2 accumulation forcing.

Would it be easier to demonstrate skill where the forcing is something more like the arctic becoming seasonally sea ice (nearly?) free?

If so, I would have thought that there should be lots of modeling of the effects of this.

Perhaps there is still lots of time for that to be done as forecast rather than as hindcast?

James Annan said...

Yes, you are right that predicting changes over a few years doesn't make much sense, as the natural variability will swamp any forced change. I'll have to think about the sea ice...you mean predicting what will happen in (say) 20 or 40 years when there is no sea ice? That still doesn't help us evaluate the models now...

Some participants talked about hindcasting recent forced changes...but they haven't been large on the local scale, especially not precip...

crandles said...

I hope you are right, and win you bet against J Romm. Presumably that would make Maslowski's projection of 2016 +/-3 years wrong.

Are you still willing to take more bets on same terms as Romm?

5 years data after first down to 10% of pre 2000 coverage to have any hope of verifying predictions would appear to mean it could be as little as 10 available years if Maslowski is right. So there could well be sufficient spare time to collect a couple of years of cryostat2 data before getting around to talking about whether it is worth doing, decide how to force the models, prepare grant proposals, prepare and run models, collate results, attempt to get published....

Obviously there is plenty of scope for me to be way out on estimating such timings.

James Annan said...

TBH I can't remember exactly what I bet with Romm, but I bet I was right whatever it was :-)

I suppose rather than waiting for a large change it might be possible to hindcast situations where there has been a directly imposed change in land cover (urbanisation or cropping) and see whether the models got the change right. Actually now I htink about it, surely this must have been done on lots of occasions, even if the analysis was not pursuing this precise question ("skill" is a rather technical concept).

Steve Bloom said...

Aha, a bet on a bet! May I package that for sale in several tranches? :)

Chris, bear in mind that Romm is now hiring staff, so he probably needs the money more, and anyway all J+J are likely to do with any winnings is blow it on more Apple paraphernalia. :)

Speaking of sea ice and model skill, note the new paper discussed here. I haven't read it yet (have asked for a copy), but Mahlstein and Knutti's near-contemporaneous effort springs immediately to mind. I think you said it was on your reading list when I mentioned it a few months ago, James, but now I'm wondering if there will be any models left standing after the filters proposed by these two papers. I would be extremely interested in your take on both papers.

James Annan said...

Tell you what, I bet you can't work out how to tranche that bet effectively, but if you do, I'll take a percentage off each trade rather than a position either way :-)

At a (very quick) glance, that linked paper is purely observational, that is the comments about models seem entirely hypothetical (well, other than that they are generally evaluated firstly on means rather than on variability). So it's not clear that the models actually do have any related failings, though the authors may have some reasons to suspect it. Not that this is meant as criticism, there is only so much work one can be expected to do per paper, especially these days when the count matters more than the content.

crandles said...

Perhaps
http://julesandjames.blogspot.com/2007/12/another-bet-on-not-so-much-climate.html

should be added to

http://julesandjames.blogspot.com/2005/06/betting-summary.html

James Annan said...

Well found - with you around, maybe I don't need to keep note :-)

OK, I have updated the page...