This is something I've been meaning to blog about for some time. It comes up a lot in the context of the hurricane wars, over at RPJnr's blog. A recent comment of his provides a nice opening:
There is, however, an entirely different but equally valid approach that could also be used from the outset, which is: what is our estimate of the magnitude of the effect? The critical distinction is that the null hypothesis has no particularly priviledged position in this approach.
This distinction between detection and estimation is related to that between a frequentist and Bayesian approach to probability. Ruling out a null hypothesis (or not) at some level of significance is essentially a frequentist approach: estimating the probabilities of various competing hypotheses, about which we have prior beliefs (but not necessarily a strong bias towards a null hypothesis) is Bayesian. The answers that these two approaches provide may be very different in any given situation, and neither is necessarily right or wrong a priori, but it is surely self-evident that the Bayesian approach is more relevant to decision-making. If we have any reasonable expectation that certain policies would have particular bad effects, it would be ridiculous to wait until such effects could be shown to have occurred at some arbitrary level of statistical significance (that's not a point specific to climate change, of course).
IMO, the IPCC slightly muddies the waters with its discussion of D&A. It defines attribution, as Roger implicitly does above, as a strictly stronger statement than detection - one can only attribute once the effect has already been detected. I suppose they can define the term "attribution" in that way if they choose, but it is certainly not then valid to equate this with the general estimation problem as they do further down. It is trivial to create situations in which a currently undetectable effect can be reasonably estimated to be large, and the converse is equally possible - an easily detectable (statistically significant) influence may be wholly irrelevant in practical terms. I suspect that this forms a large part of the difference in presentation between various parties in the hurricane debate - the evidence may not yet rule out the null hypothesis of no effect, but some people estimate that AGW is likely to have a substantial effect (even if the ill-defined error bars on their estimate do not exclude zero). In principle, exactly the same evidence could support both of these conclusions, although I don't personally know enough about hurricanes to make a definitive statement in that particular case.
It is amusing to see Roger, very much at the sharp end of policy-relevant work, promoting the scientifically "pure" but practically less useful detection/frequentist approach rather than the more appropriate estimation/Bayesian angle. It's not surprising, although perhaps a little disappointing, that the IPCC explicitly endorses that view. But by placing the null hypothesis in a priviledged position from which it can only be dislodged by a mountain of observational evidence, this approach provides a strong inbuilt bias for the status quo which cannot be justified on any rational decision-theoretic grounds.
As you well know much of science works through hypothesis - falsification. Across climate science the null hypothesis used to guide research has been that a human signal is NOT present, and research is then done to falsify this hypothesis. In some cases such research has been done allowing attribution of climate effects to an anthropogenic forcing. This null hypothesis is chosen because of Occam's razor, it is a simpler explanation for what is observed.He is essentially posing the question as initially one of detection - can we show that the AGW has had an effect, and that the observations are not just the result of climate variability? - before moving on to attribution - how much of a change can we describe as being due to this particular cause?
There is, however, an entirely different but equally valid approach that could also be used from the outset, which is: what is our estimate of the magnitude of the effect? The critical distinction is that the null hypothesis has no particularly priviledged position in this approach.
This distinction between detection and estimation is related to that between a frequentist and Bayesian approach to probability. Ruling out a null hypothesis (or not) at some level of significance is essentially a frequentist approach: estimating the probabilities of various competing hypotheses, about which we have prior beliefs (but not necessarily a strong bias towards a null hypothesis) is Bayesian. The answers that these two approaches provide may be very different in any given situation, and neither is necessarily right or wrong a priori, but it is surely self-evident that the Bayesian approach is more relevant to decision-making. If we have any reasonable expectation that certain policies would have particular bad effects, it would be ridiculous to wait until such effects could be shown to have occurred at some arbitrary level of statistical significance (that's not a point specific to climate change, of course).
IMO, the IPCC slightly muddies the waters with its discussion of D&A. It defines attribution, as Roger implicitly does above, as a strictly stronger statement than detection - one can only attribute once the effect has already been detected. I suppose they can define the term "attribution" in that way if they choose, but it is certainly not then valid to equate this with the general estimation problem as they do further down. It is trivial to create situations in which a currently undetectable effect can be reasonably estimated to be large, and the converse is equally possible - an easily detectable (statistically significant) influence may be wholly irrelevant in practical terms. I suspect that this forms a large part of the difference in presentation between various parties in the hurricane debate - the evidence may not yet rule out the null hypothesis of no effect, but some people estimate that AGW is likely to have a substantial effect (even if the ill-defined error bars on their estimate do not exclude zero). In principle, exactly the same evidence could support both of these conclusions, although I don't personally know enough about hurricanes to make a definitive statement in that particular case.
It is amusing to see Roger, very much at the sharp end of policy-relevant work, promoting the scientifically "pure" but practically less useful detection/frequentist approach rather than the more appropriate estimation/Bayesian angle. It's not surprising, although perhaps a little disappointing, that the IPCC explicitly endorses that view. But by placing the null hypothesis in a priviledged position from which it can only be dislodged by a mountain of observational evidence, this approach provides a strong inbuilt bias for the status quo which cannot be justified on any rational decision-theoretic grounds.
6 comments:
Actually James, I am pretty much in agreement with you. I am not promoting the IPCC approach at all, only noting that it is the IPCC approach. It seems quite suspect to me for scientists to advocate selectively defecting from the IPCC approach only in those cases where the IPCC approach does not generate conclusive attribution. (i.e., when it can't be proven, then assume it!) This is an all-too-convenient cherry picking of methodological orientation as a function of whether-or-not D&A is achieved.
Shouldn't the IPCC use a consistent approach to D&A across climate science? And BTW I do not think that the Bayesian/frequentist distinction is what lies behind the hurricane debate, it is much more about data quality and paradignmatic orientation.
And on the policy issues, as I've written many times before, the question of D&A with respect to hurricanes may be politically exciting, but it is of little policy relevance. The policies that make sense will still make sense whether or not D&A is achieved.
Roger,
Cheap jibes like this aren't really very helpful, are they:
(i.e., when it can't be proven, then assume it!)
I'm certainly not claiming that people are using a formal Bayesian estimation procedure to justify their comments. But at a less formal level, people do estimate, and "we can't prove anything has yet happened" and "we expect an effect to occur" are in no way mutually exclusive interpretations of the same evidence - it is partly a matter of what question is being asked. The latter approach is clearly more relevant to policy development.
I don't think it is the IPCC's business to prescribe any particular approach to D&A, or estimation. I'm certainly not advocating "selectively defecting from the IPCC approach when it does not generate conclusive attribution", but merely suggesting that people should try to answer questions that could in principle have some policy relevance. I don't see that rejecting (or not) a null hypothesis (especially when no-one believes it is at all plausible in the first place) at some level of significance is necessarily the most constructive approach. Explicitly Bayesian methods are widespread in my corner of the field, although some seem now to be arguing (unconvincingly, IMO) that even this framework is too limited.
James-
I'm sorry for the miscommunication, but my comments weren't directed at you. We do seem to be pretty much on the same page here.
However, this statement of yours deserves a raised eyebrow:
"I don't think it is the IPCC's business to prescribe any particular approach to D&A, or estimation."
The IPCC sure thinks that this is central to its business. Implementation of the FCCC and Kyoto depend crucially upon such business.
And I'll stand by my "cheap jibe" just because its true! -- yikes, are we really so humorless! For truely cheap jibes come visit the comments on my blog ;-)
Roger,
I agree that we probably largely agree. OTOH I perceive a hint of "you can't prove it" in your various comments. It's not a valid basis for rational decisions.
I'm a bit puzzled by what you say about the IPCC - it's primary (sole?) function is to write assessment reports, and whatever science I do, if it's relevant and not obviously invalid, it's going to get assessed whether or not I use an approved technique. Besides, that page I referred to explicitly mentions extension to Bayesian methods, and that was back in the TAR.
For truely cheap jibes come visit the comments on my blog
Maybe, but I try to keep the tone higher here :-))
James-
The reality is that D&A related to hurricanes is not a particularly policy relevant question. I'd be happy with an approach like the following:
"Given that GHGs afect tropical cyclones, what policy actions are likely ot be most effective with respect to modulating future damages?"
On this question existing research is unequivically unambiguous.
And I appreciate your higher tone;-)
Roger is a bit of a hot house flower. He should come visit sci.environment on occasion.
However, to the subject at hand, I agree that the IPCC should not perscribe an approach to D&A, but it is silly to claim that it can do its work without adopting an approach. This means it has to be clear to itself and others what approach has been taken.
Post a Comment