This was the title of an interesting workshop I recently attended in the UK. It's not really that good blogging fodder, as mostly it was rather technical stuff to do with estimation methods, and there wasn't a whole lot of new climate science. Apparently there will be a web page put up with the talks (and maybe other info) some time.
Not surprisingly, the mix of attendees reflected the interests of the organiser, so about half of the presentations were from climate scientists, and half were from Bayesian statisticians who have developed and formalised a general framework by which inferences can be drawn from models. There was a heavy focus on and promotion of the use of emulators, which seem like a really cool idea but do appear to have some limitations (I'll probably post more about that separately). In fact one of the climate scientists said that he felt like he was being sold a 2nd-hand car...a sentiment I have some sympathy with. Jules sugested this might be somewhat cultural, due to the commercial background (backing) of some of the Bayesian statisticians who have previously worked with the oil industry for things like forecasting well ouput. Perhaps there is some defensiveness and resistance to new ideas from within the climate science community too.
I enjoyed giving my rant about climate sensitivity, and heard nothing to change my opinion (as outlined on the last slide of that talk) that the vast majority of published (and widely publicised) "pdfs of climate sensitivity" are pathological by construction and virtually worthless as a result. Indeed, I got the distinct impression that what I had to say was basically teaching about half the audience to suck eggs. There were of course some quibbles about the details of the calculation I performed, but no substantive criticism of the basic idea. I was pleased to note that several of the following speakers were rather apologetic for their use of uniform distributions as representing "ignorance", so at least that idea is catching on. Unfortunately, several members of the potential audience either left before or arrrived after my talk. In the following discussion, someone did suggest that we ought to deliberately exaggerate the probability of high S, in order to counter the tendencies of others to ignore the range of uncertainty. That's not a viewpoint that I have any sympathy for.
There was one bizarre moment when someone paused in the middle of their scientific presentation to show this Greenpeace advert. Really. I'm not making it up.
Overall, it was a very interesting week, but there were a few minor disappointments for us. I think we failed to adequately convey the amazing power of the EnKF in generating solutions to nominally difficult problems (ie those involving high dimensional, computationally intensive, nonlinear and chaotic models). This became apparent in the final discussion when people started talking about the potential for using adjoints to solve these problems, citing this paper in particular...perhaps they didn't realise, or didn't accept, that we developed the EnKF as a means of addressing the specific, well-known (or so I thought) and intractable problems that adjoints have in such situations. I guess if anyone starts to go down that path they'll learn the hard way soon enough. [I shouldn't be too thoroughly negative - there are some things an adjoint is useful for, and maybe they will get something worthwhile out of it. But I'm certainly not tempted to try it myself.] Also, although we had gone there expecting to learn how emulators are the solution to all our problems, it seems that they are very hard work for what seems like a rather modest gain. Of course it is useful to have found that much out prior to putting in all that hard work ourselves.
On the plus side, although thanks to the trains we barely overlapped with one or two of the attendees, it was good to meet for the first time several people who I had previously only corresponded with via email (and to meet again others whom I only see occasionally). Electronic communication is all very well and good but it's not a full substitute for some face-mail.
After the workshop, we popped into London for the Friday evening, and had a pleasant dinner in the vicinity of Broadcasting House where we heard some of the back-story surrounding the "Overselling climate change" radio programme. That was an interesting tale...
Not surprisingly, the mix of attendees reflected the interests of the organiser, so about half of the presentations were from climate scientists, and half were from Bayesian statisticians who have developed and formalised a general framework by which inferences can be drawn from models. There was a heavy focus on and promotion of the use of emulators, which seem like a really cool idea but do appear to have some limitations (I'll probably post more about that separately). In fact one of the climate scientists said that he felt like he was being sold a 2nd-hand car...a sentiment I have some sympathy with. Jules sugested this might be somewhat cultural, due to the commercial background (backing) of some of the Bayesian statisticians who have previously worked with the oil industry for things like forecasting well ouput. Perhaps there is some defensiveness and resistance to new ideas from within the climate science community too.
I enjoyed giving my rant about climate sensitivity, and heard nothing to change my opinion (as outlined on the last slide of that talk) that the vast majority of published (and widely publicised) "pdfs of climate sensitivity" are pathological by construction and virtually worthless as a result. Indeed, I got the distinct impression that what I had to say was basically teaching about half the audience to suck eggs. There were of course some quibbles about the details of the calculation I performed, but no substantive criticism of the basic idea. I was pleased to note that several of the following speakers were rather apologetic for their use of uniform distributions as representing "ignorance", so at least that idea is catching on. Unfortunately, several members of the potential audience either left before or arrrived after my talk. In the following discussion, someone did suggest that we ought to deliberately exaggerate the probability of high S, in order to counter the tendencies of others to ignore the range of uncertainty. That's not a viewpoint that I have any sympathy for.
There was one bizarre moment when someone paused in the middle of their scientific presentation to show this Greenpeace advert. Really. I'm not making it up.
Overall, it was a very interesting week, but there were a few minor disappointments for us. I think we failed to adequately convey the amazing power of the EnKF in generating solutions to nominally difficult problems (ie those involving high dimensional, computationally intensive, nonlinear and chaotic models). This became apparent in the final discussion when people started talking about the potential for using adjoints to solve these problems, citing this paper in particular...perhaps they didn't realise, or didn't accept, that we developed the EnKF as a means of addressing the specific, well-known (or so I thought) and intractable problems that adjoints have in such situations. I guess if anyone starts to go down that path they'll learn the hard way soon enough. [I shouldn't be too thoroughly negative - there are some things an adjoint is useful for, and maybe they will get something worthwhile out of it. But I'm certainly not tempted to try it myself.] Also, although we had gone there expecting to learn how emulators are the solution to all our problems, it seems that they are very hard work for what seems like a rather modest gain. Of course it is useful to have found that much out prior to putting in all that hard work ourselves.
On the plus side, although thanks to the trains we barely overlapped with one or two of the attendees, it was good to meet for the first time several people who I had previously only corresponded with via email (and to meet again others whom I only see occasionally). Electronic communication is all very well and good but it's not a full substitute for some face-mail.
After the workshop, we popped into London for the Friday evening, and had a pleasant dinner in the vicinity of Broadcasting House where we heard some of the back-story surrounding the "Overselling climate change" radio programme. That was an interesting tale...
13 comments:
>There were of course some quibbles about the details of the calculation I performed, but no substantive criticism of the basic idea.
Presumably this was things like separating signals for current warming from the volcano record from one dataset and treating them as independent and the greater scope for unknown unknowns with LGM information as Dave Frame mentioned?
crandles
I was thinking of adding to here a question something like:
I know Carl did not get funding for Ensemble Kalman Filter(EnKF) work with CPDN, so it is hardly for want of trying that this hasn't been done. However, as Dr Annan reckons his EnKF method is substantially more efficient than the CPDN approach, does the extra work returned provide sufficient extra analysis to make it worth the inefficiency?
Would you ask this sort of question in a different way?
crandles
Chris,
Yes, mostly stuff about how all the "other" constraints (ie not the C20th warming) were really rather uncertain due to forcing and response uncertainties...but of course that is not a sufficient reason to ignore them completely, and I do think that we've shown that our result is rather robust to such details (IMO the "headline" result that we published is rather pessimistic as an overall pdf). In fact, after thinking further about the choice of underlying prior, I've probably hardened my position somewhat on that. Note that in the GRL paper, the C20th "prior" is actually built from a uniform prior and C20 data - using a less alarming underlying prior would already have shortened its tail before we even started including the other data...
As for the EnKF and CPDN - the practicality of it does depend a bit on the architecture of their system, in particular how many (serial) runs they get from the fastest 100 or so machines. Maybe they also dug themselves a bit of a hole at the outset by insisting that brute force was the only method (when they were trying to motivate support for the project). Without having seen the details of what they were actually trying to do, it's hard to comment further.
so actually running a lot of models to quantify uncertainty is "brute force?" HAHAHA
Let's ask an expert panel of climate scientists what they would prefer -- 10-100K Hadley Centre model simulations or a small EnKF of MIROC? :-)
Anon,
You make my point admirably for me. I believe I've seen CPDN say that the computing resources at their disposal exceed that of the entire Earth Simulator, and they have an army of staff and tons of funding at their disposal. I, on the other hand, have my own time (jules also spends some of her time on this stuff) and run in the "test area" corner of the ES - less that 1% of the total machine at any one time (and then only for a few hours at a time). If you want to argue that the CPDN results are 100 times more important than my work, fire away (I'll not dispute that they have garnered 100 times the newspaper column inches) - but if their approach isn't "brute force" I'd like to know what is!
So at what point do you & your wife deem it "brute force?" Is the MetOffice QUMP group, with 100-200 HadCM3 models "brute force?" And, OK, say these big modelling efforts are "brute force" --- why would that be bad? Again -- no scientist would pick the real thing (big ensemble of models) over some emulator games or EnKF. Unfortunately you're too close to see the forest for your statistical trees.
CPDN & QUMP are still at 1/1000th what Donald Rumsfeld has run on top US supercomputers for military purposes. Not to mention all the work expended on biomedical modelling. I hardly feel that any climate modelling effort to date is "brute force" compared to what is being expanded on research labs worldwide on military or medical models.
Do you really think your EnKF with MIROC is the be-all end-all? What a laugh. I've seen preprints of a paper coming out that pretty much shows that you are "garbage in garbage out" with MIROC & EnKF. So we'll wait and see that published I guess, unless your "realclimate heros" can get it ditched I guess. Meanwhile your RC buddies can't even handle a couple of Canadian hacks who successfully snapped the vaunted hockey stick. It's a shame when egotism gets in the way of science...
Um...yeah..whatever.
a "bold" response, worthy of your realclimate brethren ("run away run away").
Post some relevant and interesting comment, I'll try to answer it.
I thought your previous was just a rhetorical flourish, but on the off-chance that you really were asking, I suppose I could deal with it more fully:
"Brute force" is commonly used to apply to simple Monte Carlo sampling or multifactorical experiments: CPDN is the latter, QUMP is neither. The only "wrong" thing about BF is that there is usually a more efficient way to get to as good an answer, and indeed it is often completely infeasible (eg CPDN only used 3 levels for each of 6 parameters, QUMP and us have been using ~25 parameters sampled continuously).
I certainly don't think that the EnKF is the "be-all end-all": it has well-known theoretical limitations and I'm working on ways to overcome them (although I've not seen evidence that the impact is large). As for GIGO, one could perhaps describe our SOLA paper in those terms :-)
>As for the EnKF and CPDN - the practicality of it does depend a bit on the architecture of their system, in particular how many (serial) runs they get from the fastest 100 or so machines. Maybe they also dug themselves a bit of a hole at the outset by insisting that brute force was the only method (when they were trying to motivate support for the project). Without having seen the details of what they were actually trying to do, it's hard to comment further.
Fair enough, I don't know all the details either and hence the need for a rather general question I suggested.
I do know that one of the aims was to make the dataset available to other researchers (CMIP like?). Presumably other climate researchers, if interested in a dataset, would prefer a large array of models with the parameters held constant rather than a small ensemble where the parameters are varying over time?
(Progress with this has been disappointing.)
There was also some talk of finding islands of viable parameter space. Would your method be more likely to miss these?
(I would expect there to be some acceptable level of missing islands to compensate for the efficiency effect.)
A couple of computers have completed 160 year TCM runs since March. I would guess over 1000 computers have done 80 years with another 40,000 having done over 5 years and less than 80 years.
You don't really want the fast computers to wait for a revised set of parameters before starting to crunch again. It may also need a bit of BOINC development to allow the new parameters to be sent out. Therefore, I imagine you would have to send out each 5 or 10 year chunk as a separate work unit. This would mean the different chuncks could be processed by different computers. I think BOINC can handle the requirement to only give out Work units to the same type of processor. This wouldn't guarentee the same math library so models could calculate differently on different computers but hopefully this would only be like different ic members.
There would be extra challenges to this approach. After receiving back say 20 runs of the first chunk you would want to start issuing the second chunk but the first chunk runs would continue to come in and you would then want to amend the distribution of parameters for the second chunk. Not sure if that would be an interesting problem or just a nightmare.
>I believe I've seen CPDN say that the computing resources at their disposal exceed that of the entire Earth Simulator, and they have an army of staff and tons of funding at their disposal.
Hmm. I did a calculation of the funding which worked out to 4 full time equivilent staff members and was told this was about right.
There are many more people mentioned as being involved. So some are part time and some don't count in the funding figure. I don't suppose there is much spare time on the Earth simulator or on other supercomputers so compared to the cost of building a new supercomputer .....
It seems clear to me that they could do a lot more with the CPDN system with more funding. I expect more science would be done if half the participants stopped their computers and gave the cost of electricty saved to the project. It seems to me suggesting this would be rather counterproductive to the public understanding of science goals of the project. (In short I don't agree with your "army of staff and tons of funding" it sounds to me like a case of the grass is always greener. If it really is greener, why don't you put in a proposal to do it better?)
crandles
"grass is greener"...
Funding always looks better from the outside:-) But really, I wasn't trying to whinge, just providing a bit of context for the anonymous troll's comments. Of course from the funder's POV, the fact that CPDN got their computing donated for free is a big plus point.
If you think I've got a snowball's chance in hell of getting funded for something that cpdn have failed on twice, you are wearing seriously rose-tinted specs - not least in ignoring the fact that they would be the most likely reviewers of any application :-) (Actually I have tried here in Japan, but there isn't really a suitable funding stream).
OK I probably was pushing it too far to suggest you might get funding.
AFAIUI CPDN are pushing for other researchers to do similar things to what they are doing with different models. Make the model used just another parameter. This obviously isn't the same thing as using a EnKF with hadcm3. So depending on what model you intend to use, I thought you might find them supportive.
Perhaps I should have concentrated more on: CPDN's approach is certainly a brute force approach. However while "brute force" often goes with ignorance, it isn't always the case. If you have a situation where all the cost is in the setting up and not at all in the computing power, then maybe the brute force approach is the sensible option?
crandles
If you have a situation where all the cost is in the setting up and not at all in the computing power, then maybe the brute force approach is the sensible option?
Well, as I already said, if the power is available, BF does work. It also has the advantage of being some/much easier to set up than the alternatives! I've done my share of Monte Carlo sampling when I couldn't be bothered setting up anything more sophisticated for a particular problem...but it is easy to run into the situation where the computing resources limit the problem which can be atttacked. In that case, it's worth thinking about how to use resources effectively.
Post a Comment