Thursday, February 02, 2006

Probability, prediction and verification V: Commotion in the comments

Well, having posted what I thought was an interesting but essentially uncontentious discussion of Bayesian and frequentist approaches to probability, I was rather surprised to see the reaction in the comments.

I'm going to pick primarily on Wolfgang at first, because he has given his opinions most explicitly - which exposes the inconsistencies most clearly :-) From what I can glean of their comments, I think the others are displaying similarly inconsistent attitudes, however.

He starts out:
In physics statements about probability makes sense in two cases:
i) quantum theory
ii) systems where we do not know the initial state exactly.

In all cases we must be confident about the theory (kinematics and dynamic), otherwise the use of probability is misleading.

Probability is not a good replacement for missing or wrong theories. Your example of the prime number illustrates this very well.

I could perhaps quibble over exactly what is meant by "we do not know the initial state exactly", but this looks like it is at least trying to be a fundamentally frequentist position. Fine, if that's what he really wants.

But later on, I pose the following problem in the comments to CIP's post (nb the electron neutrino is just some arbitrary poorly-known particle: I am assuming the Wikipedia page is reasonably valid when it says its mass is thought to lie between 0.1eV and 2.5eV, but note that even this statement is only meaningful in a Bayesian worldview!):
Say you are trying to measure the mass of the electron neutrino. One possible experiment costs $1 million, and would work down to 0.5eV. Another possible experiment would cost $10 million but would work to 0.1eV. You have other demands on the money. Do you consider there is any information in the above that can help you choose one experiment rather than the other? Do you consider that questions along the lines of "how likely do I think it is that the mass of the neutrino is less that 0.5eV" are even legitimate in the first place, or do you simply reject that as an ill-posed question? If that is the case, how do you actually make the decision in practice? If the cost of the second experiment were 1.1 or 100 million, would your decision change?
After some prompting, Wolfgang replied that
if you have reason to believe that the neutrino mass is in the range m = 0.1-2.5eV you can calculate probabilities.

This is because you have different scenarios ( m = 0.11, m = 0.21, etc.) which are all consistent and compatibel with what you know.

But this is precisely espousing the Bayesian attitude that he rejected previously! So it seems that he's a Bayesian without realising it, or perhaps a Bayesian depending on how he feels at the time. He has not yet explained how he would calculate these probabilities, or what they could possibly mean in a frequentist world.

CIP makes similarly odd noises in his posts here and here. He says:
Subjective plausibility makes my skin crawl. I have no idea what the hell it is supposed to mean. I can't think of any case when an expressed probability isn't either based on an explicit or implicit frequentist interpretation or else, like Mr. Spock's invariably misguided predictions, merely an ignorant prejudice.
but has so far ducked my question about the electron neutrino.

There were other various comments along vaguely similar lines. I think that some of these people don't actually understand how the Bayesian and frequentist views of probability actually differ. I'll try to explain further with a simple example.

I have a max-min thermometer in my garden, which I reset each day. It measures the max and min, and let's assume that I know that the error on each day's measurement is well-characterised as a standard gaussian deviate - ie a sample from N(0,1). This morning I went out and saw that the min from the last night was -5C. What is the probability that the actual minimum was below -5C?

Most people will instinctively answer that the probability is 50%, reasoning that the error is just as likely to be positive as negative. Most people would be wrong. The probability depends on the prior distribution before you looked at the thermometer! This is an unavoidable consequence of how Bayes' Theorem works. The posterior is equal to the prior multiplied by the likelihood of the true temperature given the observation. Those who do not understand this will repeatedly get themselves into trouble when discussing probabilistic estimation.

To those who insist that 50% is the right answer, consider this situation: Say I know that the record low temperature ever recorded in my town (of which my garden is representative) is -4C. Will I still believe that there is a 50% chance that last night's temperature was below -5C? Of course not. Temperature records are virtually never broken by such a large margin. In this case, my prior probability density function has its mean (and the bulk of its support) above -5C, so even after measuring -5C, the posterior mean is above -5C. Even with a less extreme example - say the record low is -6C, but the typical February daily minimum temperature is -1C - a rational Bayesian will still (probably, based on the info supplied) conclude that the temperature was probably warmer than -5C.

The naive frequentists will say that they can interpret this in a frequentist sense - consider sampling the temperature each day, and on those days when you see a temperature of -5C, was it warmer or colder? But what distribution do you sample your days from? Statistics for every day in the last 30 years? Every February in the last 30 years? Every February 2nd when there had been rain on the previous day (as was the case this time)? Note that if you actually sample from the finite population, you can never assing a non-zero probability to breaking any record, and more generally you can never assign a non-zero probability to any state you have not already observed. So you have to make some subjective (ugh) decisions about how to define a sensible continuous prior distribution (is it Gaussian? Weibull? Spline through the histogram of data?) and you have to make subjective (ugh) decisions about which historical data you base your distribution on, and then you perform some hypothetical thought experiment in which days are drawn at random from this subjective (ugh) and hypothetical prior distribution and this is precisely the Bayesian paradigm!

In the above example, the whole concept of probability is only valid (to a frequentist) when the prior distribution is given a priori (groan). For example: "If we sample the Feb daily min temperature from a particular pdf f and then measure this temperature (with error as stated before), then if the temperature measurement is -5C, what is the probability that the sampled temperature is less than -5C?" Any frequentist or Bayesian can answer that question easily and correctly. But I did not define the prior in the prior version (groan again). I just asked for the probability. A Bayesian has to (and is prepared to) choose a reasonable prior, and realises that his answer is not an "objective" statement of the "true" probability but a somewhat subjective one contingent on the prior he chose. A competent frequentist will say the question has no meaning without a clearly-defined prior. A naive frequentist will probably provide an answer (like the original "50%") without even realising that they are implicitly making an assumption of a particular (and likely unrealistic) prior. Which category are you in?

40 comments:

wolfgang said...

> But this is precisely espousing the Bayesian attitude that he rejected previously!

Your statement is a great example of beating a strawman.
I tried to make the case what "subjective uncertainty" can mean on CIP's thread.
Before you put me in one category or another you should perhaps read what I wrote.

wolfgang said...

> A Bayesian has to (and is prepared to) choose a reasonable prior

And to (try to) make myself clear one more time:

A "reasonable prior" requires some theory, prior experience etc.

You cannot just make it up, otherwise you are just arbitrarily wrong.

In your neutrino example: If you use a uniform distribution for the mass m, because it seems "reasonable" or something else, you already have a theory that m is your "natural" variable.

A uniform distribution for ln(m), 1/m, etc. give you completely different results and as long as you do not know the physics you do not know whcih one to use.

PS: I will post more comments on CIPs blog only.

James Annan said...

Wolfgang,

Your statement is a great example of beating a strawman.

Well give me something solid to work with then. A worked example illustrating anything along the lines of

if you have reason to believe that the neutrino mass is in the range m = 0.1-2.5eV you can calculate probabilities

would be great. Post it here, there or anywhere. Thanks!

BTW before you get too indignant over this, bear in mind that Lubos has been conspicuous by his absence, and he would certainly jump at the chance to attack me if he thought there was the opportunity...

Wolfgang said...

James,

if you use my name in your page, then you could at least put a link there as well.
I have linked to your site in the past.

Wolfgang said...

> Well give me something solid to work with then.

Let me try the very last time.

YOUR statement "that the neutrino mass is in the range m = 0.1-2.5eV" can mean only two things:

i) you know n different theories with different values for m (and I made those values up) and all theories describe current knowledge equally well.

ii) you have measured m experimentally (which requires a lot of theory!) and found N different values in the range 0.1 - 2.5eV.

You can also have a combination of i) and ii)

In case i) you count the theoretical scenarios and in case ii) you use the distribution u(m) determined by the experiment(s). If you have several experimental samples you will use modern re-sampling techniques and respect conditional probabilities.

This is what I meant.

What is also clear is that without theories (i) or experiments (ii) you have nothing. You cannot just make up a prior.
E.g. assuming that a uniform distribution u(m), instead of a measured one, is "natural" or "reasonable" as a Bayesian prior would be misleading.
Because it might be just as natural to assume that u(1/m) or u(ln(m)) should be uniformly distributed.

In short: A frequentist/physicist has no problem with unique events, as long as there are reasonable ways to dtermine plausible scenarios.

In other words, the "subjective uncertainty" of Bayes means that you have different scenarios in your head.

If you have problems to put me in one of your categories (is he a Bayesian or a frequentist?), this is your problem not mine. I consider myself as a physicist. If you doubt my credentials you can read my webpage.

I do not expect that this answer will clarify my opinion for you, but perhaps it will clarify it for your readers.

Wolfgang said...

Perhaps I should answer a question you or your readers might have in advance.
Part of the theory in case (ii) is the knowledge that we are dealing with measurement errors, since our theory is that the neutrino mass is a constant which is the same for all neutrinos. (The neutrino mass is not a stochastic variable, governed by a complicated Markov process).
The "deep questions" on how to fit a distribution to our data are thus not really so deep. Gauss has done most of the work for us many years ago.

James Annan said...

OK,

let's try to keep this simple. I'm still looking for a numerical worked example. Extracting from your comment:

ii) you have measured m experimentally (which requires a lot of theory!) and found N different values in the range 0.1 - 2.5eV.
[...]
in case ii) you use the distribution u(m) determined by the experiment(s). If you have several experimental samples you will use modern re-sampling techniques and respect conditional probabilities.


Can you explain how this can work in practice? Say you have one estimate of 1eV, with a gaussian uncertainty of +-1. Use more than one observation if you like (need). Tell me how to turn this into a distribution for the mass.

You seem to have decided that you accept the Bayesian view again.

(I'll fix a link up, sorry - but your comments that I was interested in are only in the comments, not your blog)

Wolfgang said...

> I'm still looking for a numerical worked example

I am sorry, but I do not know what to do or say other than point to some standard text.
You calculate the mean, variance (the error) and determine the confidence interval etc.

If you are fancy, calculate the higher moments to see if your distribution is skewed.
If you have a PC use Numerical Recipes 8-)

Do you try to test my statistics knowledge?

Wolfgang said...

> You seem to have decided that you accept the Bayesian view again.

This is really strange.
What difference does it make what "view" I have?

Wolfgang said...

> Say you have one estimate of 1eV, with a gaussian uncertainty of +-1. Use more than one observation if you like (need). Tell me how to turn this into a distribution for the mass.

Well, your 1st sentence specifies already a distribution.
I can read your stuff up and down, I do not see what you want me to do or write at this point.

OK, lets turn your description into a distribution: James found a Gaussian distribution 1eV +- 1eV

As your number of data points increases, your significance of the mean will increase and the n-sigma confidence interval will narrow like 1/sqrt(N).
Use t-statistics or something even fancier to show your statistics skills.

If you have more than one experiment you may deal with systematic errors.
There is a whole literature how to identify and deal with them.
You may want to ask a real experimental physicist how to do this right.

James Annan said...

Wolfgang,

I think rather than brush me off, we would both benefit from you trying to explain what you mean. According to my understanding, you cannot construct a posterior distribution from a measurement without using some sort of prior. If you want to say different, please explain how.

This is not so much a test of "statistics knowledge" from a book as to see what you understand of the concepts. Your comments seem confused and inconsistent to me, and appealing to "some standard text" is not convincing.

For example, by "confidence interval" do you really mean the standard frequentist usage, or instead the bayesian "credible interval"? I admit I have used confidence interval in the latter context myself. It seems widespread in my field, although perhaps a little sloppy.

Your task, should you wish to accept it, is to use some measurements to construct a posterior distribution (eg bayesian credible interval) for a physical parameter without recourse to a prior. The world awaits!

Wolfgang said...

As I wrote previously,
I did not expect that you would (want to) understand what I have to say. Perhaps your readers do, if they are not already tired of the frequentist vs. Bayesian debate.

And perhaps you want to still check out the text I linked on CIPs board,
whcih explains how Bayesian reasoning can be quite wrong when applied to thermodynamics.

As I said previously,
I wish you good luck with your research.

James Annan said...

Wolfgang,

Do you not understand the difference between a likelihood function (that arises from a measurement of, say, xo=1eV with uncertainty N(0,1)), and a probability distribution (x has the distribution N(1,1))? They are not equivalent! One is a distribution on the measurement error, the other is a distribution on the mass!

This distinction is absolutely fundamental to this whole discussion. You can't just pretend that by specifying one, I've specified the other.

James Annan said...

If my use of the word "estimate" was unclear, let me rephrase:

Say you have one observed value of 1eV, with an observational uncertainty of +-1eV. How do you turn this into a distribution for the mass, other than by using a prior and Bayes' Theorem?

wolfgang said...

> One is a distribution on the measurement error, the other is a distribution on the mass.

Yes, very profound. I did not know that 8-)
Except that this was not your original question.
Your question was at which energy range
I want to schedule my next experiment.

If your question is about the likelihood of the mean,
I told you already that confidence intervals are
a function of the measured variance and the number of data points. Look up t-statistics in a standard text ...
So far we assume a gaussian distribution, which you can either assume knowing the theory, or you can check it with enough data points (checking higher momentum, autocorrelations etc.)

But perhaps you want to think about a much simpler example, much simpler than neutrino physics.
If I want to determine my age, I have to count the number of my birthdays.
If I want to know a confidence interval, I need to count the number of days as well.
This requires a theory, which is explained in elementary school. This does not mean the theory is trivial by the way.

If you want to determine the age of a (female) movie star, you may need to use Bayesian statistics 8-)

James Annan said...

Wolfgang,

There still seems to be quite a gulf between your vague claims and any actual demonstration of what you are on about. Telling me to look it up is not adequate. Remember the maxim that if you can't explain it, then you don't understand it.

I'm still hoping for some credible explanation of what this means in practice:

if you have reason to believe that the neutrino mass is in the range m = 0.1-2.5eV you can calculate probabilities.

This is because you have different scenarios ( m = 0.11, m = 0.21, etc.) which are all consistent and compatibel with what you know.


followed by


ii) you have measured m experimentally (which requires a lot of theory!) and found N different values in the range 0.1 - 2.5eV.
[...]
in case ii) you use the distribution u(m) determined by the experiment(s). If you have several experimental samples you will use modern re-sampling techniques and respect conditional probabilities.


All I'm asking for is the simplest example you can think of where you can generate a posterior distribution without needing to assume a prior. Is that really too much to ask that you will do this small thing rather than just wander around the subject, occasionally referring to it obliquely?

wolfgang said...

You make N measurements which gives you a mean and variance and a confidence interval.
You make n more measurements which gives you a new mean and variance and and a new confidence interval. I will not calculate it here because your comments do not take LaTex, but you can look it up.

IF you are dealing with measurment errors, i.e.
you have a theory, you will find that the confidence interval narrows from N to n, assuming that N and n
are sufficiently large.

If you want to frame this example in Bayesian priors so be it, go ahead. I have no problem with it.

This was really the last time I will write a comment here, because I need to go to work.

wolfgang said...

Sorry, another typo:
Narrows from N to n should have been
Narrows from N to N+n

James Annan said...

Wolfgang,

This is all very trivial and point-dodging.

The fundamental question is: How do you go from measurements with uncertainties, to a distribution on the parameter in question?

Of course the obvious answer is that you can't. Your comments remain incoherent and inconsistent.

There are two obvious alternatives to resolving this disagreement. You can either agree that any probabilistic estimate of a physical parameter (fixed but unknown, like the mass of a neutrino) is necessarily bayesian and relies on a subjective prior, or you can try to demonstrate a counterexample.

So far, you have both said that such a probabilistic estimate makes no sense, and also claimed that you can generate one without recourse to a subjective prior. Whether or not the approach implied by the first comment is a sensible one, the second statement is simply false.

wolfgang said...

> How do you go from measurements with uncertainties, to a distribution on the parameter in question?

For the n-th time.
You need to have a theory to do this.
In your example you specified the theory: neutrino mass, which is a constant in a physical theory, plus measurment errors (you need to have a theory about your measurment device etc.)

I suggested to think about a simple example, like counting birthdays, with a simple theory.

I am at my job now and really have to stop this discussion.
Feel free to write one more comment or a whole post about what a moron and wannabee I am.

I see that Lubos showed up on CIP's blog, maybe you want to discuss with him. I think it may be easier with him, becuase he likes to put people into categories just like you do.

PS: By the way, I really am a wannabee 8-)

wolfgang said...

PPS: Thank you for the link!

James Annan said...

Wolfgang,

I'm disappointed that you have not attempted to clarify what you mean.

You comment that "you need to have a theory" seems to be where the prior is concealed, but you won't admit this openly. Since you started off by saying that such probabilities make no sense, you must be rather confused.

wolfgang said...

> You comment that "you need to have a theory" seems to be where the prior is concealed

If you want to call "prior" what I call theory, this is fine with me.
I really do not care if you put me in the Bayesian or frequentist category.

But, I think it is very misleading to call a theory of physics a "prior".
Lubos explained (much better than I could) what a theory of neutrinos really looks like.
This is very different from somebody who tries to fit some model from some data (he or she does not really understand).

Please, think about my example 2 again. You have three very good data points (a plane landed at 2pm +- 10min at point B the last three days).
As long as you have no theory (e.g. this is a commercial airport, commercial airlines have flight plans etc.) you cannot estimate a probability for a plane to land tomorrow.
If you do you are, you are a practitioner of cargo-cult science, as described by Feynman.

Best regard,
Bayesian Man

James Annan said...

Wolfgang,

You started out by saying that probability made no sense in cases such as the mass of the neutrino. You then decided you could estimate such probabilities. You still don't seem to understand that such an estimate necessarily requires subjective judgement. That's simply the way the world works.

Your childish jibes about "cargo cult science" sugest you are more interested in point-scoring than trying to understand. Of course in your problem it would be reasonable to think there was a non-zero likelihood of a plane landing on the next day. After it failed to land, each day the bayesian would revise their estimate downwards until after a few days they would give upn waiting. This is of course the precise opposite of "cargo cult science" where the belief is maintained in the face of evidence to the contrary.

You are correct that there is no frequentist approach to this problem (or indeed any problem where the prior is not explicitly or implicitly defined). Nevertheless, we have to make multiple decisions every day regarding such problems. Bayesianism provides the mechanism for making such decisions.

The frequentist would throw their hands up in horror at how ill-posed this plane problem is, but what would they actually do? They either go and wait, or they do not. There is no "null decision" - simply two alternatives, one of which has to be chosen. On what basis would they make that decision?

CapitalistImperialistPig said...

James, you say I never answered your neutrino question, but I think I answered pretty clearly that I didn't know enough to answer sensibly. If I were forced to answer, I would of course guess - what else could I do.

Meanwhile, Lubos has pointed out that the electron neutrino isn't a mass eigenstate, so that arbitrarily accurate measurements of its mass really should give a distribution.

I have attempted to absorb your lesson in Bayesian probability. You can check whether I actually learned anything from my post So, a Bayesian and a Frequentist go into a Bar

James Annan said...

What would you guess? Can you imagine any circumstances in which such a decision could be made on purely objective grounds, by any expert?

Another angle on it: you have the choice of two equally expensive and time-consuming experiments, one of which searches the space between 0.1eV to X, and the other covers X to 2.5eV. Does your decision about which experiment to perform depend in any way on the value of X, or do you just flip a coin irrespective of the value of X? Or do you just stay in bed all day, saying you have no grounds for making a decision either way?

wolfgang said...

> The frequentist would throw their hands up in horror at how ill-posed this plane problem is, but what would they actually do?

I do not know what a "frequentist" does, but I can tell you what a physicist would do: Try to come up with a reasonable theory, e.g. by talking to people etc., instead of relying on some arbitrary "probabilities".

By the way, the reason I like to discuss simple examples is not because I want to duck and weave, but because simple examples make it easier to explain an idea to your readers.

And it turned out that your neutrino probem was indeed a great example. The mass of the electron neutrino is not necessarily an eigenstate and the theory would tell you what distribution to expect for the mass!

And I am sure the Cargo-Cult people gave up after a while; But they never understood anything.
And I am sure a lot of people bought internet stocks in 1999 and 2000, because their Bayesian priors made it plausible to buy stocks which were going up in previous years and they may have thought they understood something in an uncertain market - until March 2000.

I hope you were not one of them. But I see that you try to understand climate change by studying financial markets.
Good luck with that!

James Annan said...

Your description of a bayesian is an extremely naive caricature, a straw-man entirely of your own creation. Perhaps this helps you to justify your prejudice, but if instead you were prepared to look at the subject with an open mind you might find you learn something useful. If you can manage the doublethink required to outwardly reject bayesianism while simultaneously using it, then good luck to you. You are clearly not alone in this.

wolfgang said...

Sorry,
I really am "typo man". I should have typed:

"The electron neutrino is not necessarily an eigenstate and the theory would tell you what distribution to expect for the mass!"

above, insetad of the nonsense I produced. These comment windows on blogger are too small for me.

CapitalistImperialistPig said...

what would I guess?

Heck, I can't even remember the question. If your point is that one should take into account any prior knowledge you have, I don't think either Wolfgang or I would disagree. My argument was that prior knowledge of that sort is a kind of a theory, and that if you want to quantify it, you are in effect assuming some kind of frequency or measure in your theory.

You, on the other hand, have not told me whether or not I correctly understood your explanation of Bayesian reasoning.

CapitalistImperialistPig said...

James - Another angle on it: you have the choice of two equally expensive and time-consuming experiments, one of which searches the space between 0.1eV to X, and the other covers X to 2.5eV. Does your decision about which experiment to perform depend in any way on the value of X, or do you just flip a coin irrespective of the value of X?

I would choose the big side of X, because I don't know anything else, or, more realistically, ask Lubos and other experts. And your point is?

James Annan said...

CIP,

This decision implies a roughly uniform prior on the mass. Don't worrry, I'm not going to hold you to it in detail or anything, but the point is that there has to be a threshold at which you are indifferent, and a uniform prior puts that at X=1.3eV (I'm assuming you mean "bigger part of the interval" not "the section that is greater than X"). Perhaps someone else might say that there are many more small particles than large ones, and their density is uniform in log-space, and to them the point of indiference would be 0.5eV (don't bother telling me this is silly - I know nothing abut particle physics, it's just an example). So far, so subjective.

If someone came along with some more information (say a measurement of 1eV, with an uncertainty of 0.2eV), you could (and would, assuming you are rational) update your priors. As the evidence increases, your different distributions would in fact converge.

But the point of my example is there is no getting away from the fact that there is always a subjective component to each of your estimates, and your posteriors will never truly coincide (even when the difference has become entirely negligible in practice). Unless and until God tells you what the "real" prior is from which reality was actually chosen, you have no option but to acknowledge that there will always be a subjective component in your estimate. Even Lubos is not God, and his opinion is also subjective!

You might feel happier if you don't try to think of bayesian probability as directly telling you "the truth" about the real world but instead as describing and formalising your imperfect understanding of the real world. It's not that the world is probabilistic, but that you are uncertain.

It is quite deliberate that many of my examples have explicitly concerned decision-making, since when faced with the need to make a decision, the frequentist attitude of "I can't give an answer until someone tells me the prior" is simply not an adequate response. A bayesian approach gives you the tools to make decisions, which of course will not be perfect, but which will improve as you iteratively incorporate knowledge into your prior. It's not about generating a grand theory of everything, but about dealing with the real world.

I'm not sure that your bayesian joke tells me much about whether you understand what I'm trying to say...

Wolfgang said...

> Unless and until God tells you what the "real" prior is from which reality was actually chosen

Yes, and there is always a 50% chance that everything is just a dream and we are all wrong.
Or is the probability that we live in The Matrix only 25% ?

If I watch the evening news I sometimes think its 100%.

Which is it ?

And after all there is a small chance that 12 is a prime number, because once my daughter was not able to factor it when she was 6 years old.

Arun said...

I'd like an explanation of what this means:

"that the error on each day's measurement is well-characterised as a standard gaussian deviate"

On what sample of days was measurement made to verify this standard gaussian deviate? "Statistics for every day in the last 30 years? Every February in the last 30 years? Every February 2nd when there had been rain on the previous day (as was the case this time)?"

What you're arguing is that the error on each day's measurement is not characterised as a standard gaussian deviate; especially when the temperature falls below the previous record low.

James Annan said...

Arun,

"that the error on each day's measurement is well-characterised as a standard gaussian deviate"

On what sample of days was measurement made to verify this standard gaussian deviate?


It doesn't matter. Assume it was checked across a huge range of artificially-generated temperatures, with the real temperature measured precisely and compared to my thermometer. What it means is that for a given real temperature (and therefore when integrated across all real temperatures) the measurement discrepancy of my thermometer has zero mean with specified variance. It's simply not the case that

"measuring error is symmetric with zero mean"

implies that

"for a given observation, the real temperature is as likely to be higher as lower."

In order to evaluate the latter statement, we also have to consider the prior distribution of real temperatures. I know it sounds a little counterintuitive at first but that's the way probability works.

A simple discretised version: assume my thermometer rounds up or down randomly with 50% probability to the nearest degree. I have two bowls of water, with exact temperatures 4.5C and 5.5C. I choose the former bowl with probability p and the latter with probability q=1-p. If my thermometer says 5C, what is the probability that I chose the cooler bowl?

There are 4 cases of (true,obs) to consider with their associate probabiities:

(4.5,4) p/2
(4.5,5) p/2
(5.5,5) q/2
(5.5,6) q/2

Of the cases where I see 5C, exactly p/(p+q) = p of them are the cooler bowl, even though for any particular bowl, the observation is as likely to be warmer as cooler. In fact if we see 6C then we _know_ the real temp was only 5.5C, not 6.5C, because the prior probability of the latter is zero by construction.

The continuous case works similarly.

Anonymous said...

>For the n-th time.
>You need to have a theory to do this.

Wolfgang,

Is your answer to the question you seem to want to ask - 'Is there a theory?' always a boolean Yes/No or is there a range from knowing nothing about a system to knowing everything about it.

If you know nothing about a systems, I don't think you would dare dream about trying to estimate probabilities without doing some investigation of the system first. If you know everything about the system then you can go about trying to estimate probabilities.

So at both extremes, I think we seem to agree. What about the situation when we know a bit about the system but not everything?

crandles

Arun said...

James,

Still trying to understand what the meaning of your error bounds on your thermometer mean.

Is it that for a uniform distribution of calibration temperatures as determined by a precise thermometer, your thermometer shows a normal distribution of error, with zero mean error?

Arun said...

Yet another question - if I had hundreds of your min-max thermometers, then I would not need any knowledge of the a priori temperature distribution, I'd simply take the average of all the readings, these are uniformly Gaussian distributed about the true value?

James Annan said...

Arun,

It doesn't matter what the real temperature distribution is. The meaurement error is symmetrical for any distribution of real temperatures. The distribution of measured temperatures will be broader than the real temperatures. Like in my simple example, a real temp of 4.5 with equiprobable +-0.5C errors gives a symmetric observational distribution over 4-5C, and a skewed distribution over inputs 4.5-5.5 with symmetrical errors gives a skewed distribution over 4-6C. However, an observed temperature of 5C does not imply a symmetric distribution over real temperatures 4.5-5.5C, in the same way as an observation of 6C does not imply a symmetric real distribution over 5.5-6.5C because the latter (6.5C) did not happen at all!

The wikipedia bayes pages have a few more similar examples, such as making inferences from medical tests. Just because a test has a false positive rate of 20% does not mean that someone with a positive result has a 20% chance of having the disease!

And yes, under this simple description of errors, if you had 100 independent thermometers and took the average, then this would give you an observation with only 0.1C error. However, you still need a prior and Bayes theorem in order to form a probabilistic estimate of the real temperature! Even if you are now very confident that you really have smashed the old record, the new record is still likely to be marginally less extreme than the measured temperature!

Anonymous said...

>the new record is still likely to be marginally less extreme than the measured temperature!

Did you mean the new record is marginally more likely to be less extreme than the measured temperature?

crandles