tag:blogger.com,1999:blog-9959776.post8109634039530421122..comments2024-02-15T04:42:41.606+00:00Comments on James' Empty Blog: BlueSkiesResearch.org.uk: How confident should you have been about confidence intervals?James Annanhttp://www.blogger.com/profile/04318741813895533700noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-9959776.post-75364071665716801472019-06-06T11:33:54.663+01:002019-06-06T11:33:54.663+01:00D'oh that last post is full of errors. I forgo...D'oh that last post is full of errors. I forgot to divide -249995 by 20 and as it is a 95 CI I should have written 2.5-97.5% ranges not 5-95. Hope these errors don't distract too much.crandleshttps://www.blogger.com/profile/15181530527401007161noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-7478781761853684502019-06-06T01:23:27.698+01:002019-06-06T01:23:27.698+01:00Well you are the expert and if you say my example ...Well you are the expert and if you say my example is not valid, you are almost certain to be correct. Yes I accept that there is at least an error in the auxiliary hypotheses and that may well make it a bad example to use. However, does this really matter if no-one knows about the 1 in 1,000,000 chance?<br /><br /><br />Re "the CIs do not have correct coverage under repeated experimentation".<br /><br />My 1 in 1000000 chance is well outside a 5%-95% range. If the experiment is take the mean of 20 numbers drawn from this distribution, then 1 in a 1,000,000 chance coming up is a rare case well outside the 5-95 range. Does it matter to the CI whether whether the 1 in a 1,000,000 chance has value of -250,000 or +100,000? I don't see that the CI is changed much if at all whether you repeat the experiment 100 times, 10,000 times or 10,000,000 times.<br /><br />If the experiment is take the mean of 10,000,000 numbers drawn from this distribution and this is repeated 10,000,000 times then the CI will be centred near 0 and and I accept that here the 0.1-0.4 range does not have correct coverage under repeated experimentation. In that case fair enough, but this wasn't what I was intending.<br /><br />Maybe this quibble is my fault that I didn't adequately explain my example was finding the mean of just 20 numbers drawn from this distribution. (Or more likely I am still wrong maybe in more different ways.) <br /><br />Yes the distribution is still not gaussian, there is a small very low probability peak near -249,995, but this seems such a small divergence from the gaussian assumption that I am inclined to try to dismiss it as a rather trivial error in the auxiliary hypotheses. Maybe I shouldn't get away with that?<br />crandleshttps://www.blogger.com/profile/15181530527401007161noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-65591163489370968252019-06-05T18:14:00.980+01:002019-06-05T18:14:00.980+01:00Yes, the fact that "CI can be weird" mea...Yes, the fact that "CI can be weird" means that you can't deduce anything meaningful from the CI. Where CI means confidence interval, a credible interval on the other hand is extremely useful as it describes where you believe the parameter to lie.<br /><br />I think however your counterexample may not be valid, as the CI calculation is (probably) making a distributional assumption that is false (ie using the mean and sd of a sample is only valid if the sample is gaussian). Whereas CIs are not useful even if there are no errors in the auxiliary hypotheses involved in their creation. In your case, the CIs do not have correct coverage under repeated experimentation.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-1341710787763664562019-06-04T14:20:49.588+01:002019-06-04T14:20:49.588+01:00>"We can generate valid confidence interva...>"We can generate valid confidence intervals for an unknown parameter with the following procedure: with probability 0.95, say “the whole number line”, otherwise say “the empty set”. If you repeat this many times, the long-run coverage frequency tends to 0.95, as 95% of the intervals do include the true parameter value."<br /><br />Seems an odd confidence interval and I am not sure what we learn from this other than CI can be weird.<br /><br />I found it helpful to consider that the true distribution might be:<br /> <br />1 in 1,000,000 chance of -250,000<br />rest 0.25 with small amount of noise.<br /><br />If researcher's sample size is less than 100,000 then likely to generate 0.1 to 0.4 confidence interval. But the true mean is -0.00000025.<br /><br />Clearly there are an infinite number of examples like the above or more normal that can produce a 0.1 to 0.4 confidence interval.<br /><br />Clearly there was no information on whether the particular distribution in question was likely or highly likely to be a normal or highly skewed example. (Also no info on researchers sample size.)<br /><br />So clearly the statements are false as in don't follow. <br /><br />Was expecting more to it and probably tried too hard to find other issues as well.crandleshttps://www.blogger.com/profile/15181530527401007161noreply@blogger.com