Monday, April 17, 2006

Subjectivity, objectivity and belief in probabilistic estimation

As I've discussed before (eg here and here) there is a certain amount of confusion over what subjective and objective really mean, in the context of Bayesian estimation (such as estimating climate sensitivity S).

There is a good discussion of these issues in Jonty Rougier's forthcoming paper, which I could do worse than to simply quote from at length (it's in Section 2 of his paper):
The first thing to understand about probability in this context is that there is no such thing as ‘the’ probability. To ask for ‘the’ probability is to commit a category error. A probability is a numerical summary of a person’s state of knowledge about a proposition: it is inherently subjective (i.e., it relates to the mind of a subject). Therefore probability takes the possessive article, not the definite one: better to say your probability.

Some readers will be concerned about this characterisation of probability. There appears to be a syllogism that runs “Science is objective, your type of probability is subjective, objective and subjective are antonyms, therefore your type of probability has no place in science.” It comes up in discussions with scientists often enough to warrant a brief comment. The error is to confuse two meanings of ‘subjective’. ‘Objective’ in this context may be taken to mean disinterested, or uninfluenced by personal prejudice: obviously a hallmark of good science. There is a meaning of ‘subjective’ which is antonymous to this: emanating from a person’s emotions or prejudices. But in dictionaries this is the second meaning. The first meaning of ‘subjective’ is relating to the mind of the subject, and this is the appropriate sense when probability is used to describe uncertainty: uncertainty is a property of the mind.

A scientist’s prediction will be perforce subjective, but he should aim to be objective as well, by making a disinterested appraisal of the probabilities he attaches to events—this is not paradoxical. Objectivity is not always easy to achieve. For example, if a climate scientist thought too little attention was being given to a certain type of future climate catastrophe, he might be tempted to overstate his probability of the event, in order to attract attention. For the policymaker, though, it is not just what a scientist thinks that is important, but also the extent to which that scientist can justify his assessment. Even though probabilities are subjective statements, not all such statements are demonstrably valid, and of those that are, not all are authoritative. Valid statements are those that are consistent with the probability calculus, the axioms of which were clarified by Kolmogorov in the 1930s.
However, scientists routinely talk about the probability, eg "the probability of S>6". I'm sure I've done it myself. It is a much more natural way for a scientist to write than to talk about "my probability", and I suspect that many of them would be squeamish to talk openly about their own beliefs, partly for the reasons Jonty elaborates above. OTOH, Bayesian probability only exists as a belief. One obvious way of squaring this circle is to interpret the scientist's statement as meaning "for anyone who accepts the stated (and unstated) premises underlying this calculation, their personal probability would be ...". Or, in more formal terms, A implies B. Indeed most papers can be reduced to this form, where A is a set of assumptions, observations and approximations, and B are the conclusions.

While this might clarify what scientists mean by "the probability", it merely shifts the real question one stage further back: does the author of the paper actually accept these premises (A) and therefore the conclusions (B) themselves? I think it is reasonable to expect and assume that they do, unless there are clear statements to the contrary in the paper. For example, we tried to make it clear that we do not believe P(S>4.5)=5%, for example, but instead consider this probability to be an overestimate (given a high/low bet at 19:1 odds, I'd choose the low side). OTOH, the authors of various other probabilistic estimates have sometimes hidden behind circumlocutions that make it hard to know to what extent they really believe their premises (and therefore conclusions). (I don't really want to pick on anyone in particular here, I think we've all done it to a certain extent.) I'd like to see greater clarity on this point in the future. By all means write a paper showing that A implies B, but if you don't actually believe either A or B, you ought to say so!

The main purpose of this comment is to lay the groundwork for a subsequent one I'm planning...


Jani said...

First, I apologize for not using my real name, as I do not want to be anyhow associated with, in my opinion, highly political climate science.

Second, since you seem to be very much promoting Bayesian statistics, I'd like to point out to your readers that this controversy, although it may be new to climate science, is a really old one, and definitely there is no "concensus" about meaning of probabilities as James seems to be implying. For a good overview of different approaches I recommend: Stanford Encyclopedia of Philosophy. For arguments against subjective probabilities, see, e.g., Burdzy's Probability is symmetry.

My main reason for writing here, however, concerns your recently published (congratulations!) paper, "Using multiple observationally-based constraints to estimate climate sensitivity". As you seem to agree (reading your new submission to GRL available on your web page), the prior plays rather significant (or at least some role) in your estimation. Now your prior (as well as the volcanic cooling) in your work is described with a gamma-distribution. I find this rather curious to say the least, since a gamma distribution gives zero probability for non-positive values. In other words, your a priori "belief" is that it is impossible for the climate sensitivity to be negative! My question is how do you justify this? Also is your analysis affected by this, and if so, how much? For other candidate choices for (skewed) priors, see e.g. here.

James Annan said...

Hi Jani/Pikkupoika,

I think you've misunderstood the origin of the "prior" in our paper. This is actually already a posterior generated by other researchers who used a uniform prior and observations of 20th century warming. Any prior belief in negative sensitivity is effectively ruled out by the observations anyway.

We did play around with a number of different shapes, and found that it didn't make a huge amount of difference. We did specifically encourage other researchers to perform their own more detailed calculations - the paper was written in a hurry, and space in GRL is highly limited, so there is certainly room for further investigations.

Note that although there may be some philosophical points to debate on the meaning of probability, any classical frequentist interpretation is impossible in this context.