Comment on Frame et al

As I hinted recently, I've got a few things to say about Frame et al: "Constraining climate forecasts: The role of prior assumptions", GRL, 32(L09702) (F05). In fact, jules and I submitted them as a comment to GRL a couple of weeks ago.

I'll start off with the bits I like: F05 give a nice demonstration of how the choice of prior can have a significant influence on the resulting pdf in a probabilistic estimation, especially when the observational evidence is quite weak. Furthermore, markedly different results can be generated by different, but apparently natural and plausible, ways of describing initial ignorance. Therefore the same evidence could be used to draw quite different conclusions due to what are ultimately fairly arbitrary choices. It's obviously an important point that doesn't appear to have been adequately considered in at least some previous work.

If they'd stopped there, I'd have had nothing to complain about. But now on to the bits I don't like so much. Firstly, they present what they see as the solution to this problem - they assert that we should choose the prior to be uniform in the variable which we are trying to estimate (ie uniform in climate sensitivity, if we are wishing to estimate this). This, in their words, "resolves" the "arbitrariness" and allows them to generate what they describe as "objectively determined" estimates. "Objective" is a dangerous word to use here, as the probability cannot be objective in the sense of a frequentist probability - what they presumably mean is merely that they are providing an automated rule that removes this element of choice from the procedure (or perhaps, imposing their own subjective judgement in place of anyone else's). But that's not the biggest problem I see in their suggestion. Where their method really falls down is that it generates results which are not self-consistent. As we demonstrate in our comment, their method will generate the pair of results P(X>3)=2.3% and P(X⁴>3⁴)=7.8% from a single observation of X=2+-0.5. Under any standard definition of Bayesian probability, P must be a function, which (again by definition) means it must be single-valued. But X > 3 and X⁴ > 3⁴ are precisely the same proposition (there's no sleight-of-hand with negative values here: X is positive definite, and I could equally have used X³ or X⁵). Therefore their P cannot be a probability at all!

There are some generalisations of probability (Dempster-Schafer theory) in which probabilities are defined as taking a range of values. Elmar Kriegler is the only person I know who's gone any distance down this path within climate science. Arguably, this provides a better framework in situations of deep uncertainty, but handling these issues correctly is far from trivial (note that a uniform prior in X does not actually represent a state of true "ignorance", but rather the specific belief that 10 < X < 20 is ten times as likely as 2.5< X < 3.5, for example). It is not at all clear to us that F05 have provided adequate theoretical justification and underpinnings for what is in fact a rather drastic challenge to the standard view of Bayesian probability, and they certainly haven't (IMO) drawn sufficient attention to the radical implications of their work.

The other complaint we have about their paper is in their description of the sort of problems that we are all attempting to answer. They say:

Unless they are warned otherwise, users will expect and answer to the question "what does this study tell me about X, given no knowledge of X before the study was performed?"

(and they use "no knowledge" to justify their choice of a uniform prior). In context, this could reasonably be re-written as (A) "what would our estimate of climate sensitivity be, if we had no data and knowledge other than that directly considered by this study?"

However, it seems clear to us that what users really want to know is (B) "what is our estimate of climate sensitivity, using all of our data and knowledge?"

The answer to question A will necessarily have greater uncertainty than the answer to question B. If someone wants to generate an estimate of climate sensitivity, they should use all of the data, either by explicitly considering it, or by the use of a prior which encapsulates (as accurately as possible) the information which the study doesn't directly look at! This is precisely the issue that our recent paperaddresses, so perhaps it is a bit harsh to pick on F05 in this respect (rather than the numerous other papers that have apparently mixed up the two questions in a similar way). On the other hand, these guys are specifically presenting a theoretical analysis of probabilistic estimation, together with recommendations as to how we should all go about it in future (rather than just having a go at producing an estimate themselves), so it's surely more important that they get it right. We certainly don't think that their opinions should be accepted by default, without some meaningful debate over the issues.

Inevitably, given the space constraints of a 2 page comment, it is hard to get the points across clearly without running the risk of appearing overly hostile. That's life, and I'm sure they have thick enough skins to cope. Indeed, depending how they reply, our comment might end up in the bin anyway - unlike most papers, where I only have to convince some neutral referees and can therefore be pretty confident of publication, in this instance there is (at least potentially) an opponent who will try their best to point out weaknesses in our case.

Both these comments, and the contents of our recent paper, are summarised on our poster for the EGU in a couple of weeks. It will be interesting to see how they go down. Unfortunately I'll not be there, so jules will have to face the angry horde by herself :-)