tag:blogger.com,1999:blog-9959776.post1933764875830729834..comments2024-02-15T04:42:41.606+00:00Comments on James' Empty Blog: Why the multi-model mean is so good!James Annanhttp://www.blogger.com/profile/04318741813895533700noreply@blogger.comBlogger31125tag:blogger.com,1999:blog-9959776.post-56084212514689199112010-06-24T13:22:17.287+01:002010-06-24T13:22:17.287+01:00Well I don't think it will really cause a revo...Well I don't think it will really cause a revolution, but since it seems like it may be new in fields as diverse as NWP and "Forecasting" (financial) I would expect quite a few to at least take note. One thing it will do is more securely justify the use of the MMM. It (or a modified version) may help with unequal weighting too, but I haven't thought about that yet.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-19496085379709964532010-06-24T10:59:30.662+01:002010-06-24T10:59:30.662+01:00>"It seems like it might have a bit of an ...>"It seems like it might have a bit of an impact."<br /><br />An equation allows the effect to be quantified. Do you foresee people calculating the RMSE for MMM then increasing this answer by a factor to reflect the advantage it has before comparing with the other models RMSE?<br /><br />Or something similar, or some other quantitative use, or is the impact you refer to something completely different?crandleshttps://www.blogger.com/profile/15181530527401007161noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-83391424330350571922010-06-23T22:16:56.159+01:002010-06-23T22:16:56.159+01:00I think those papers were talking more about somet...I think those papers were talking more about something like the standard way of combining gaussian estimates, ie the x1*s2^2/(s1^2+s2^2)+ x2*s1^2/(s1^2+s2^2) formula (or something like that). But I haven't yet had time to read them very carefully. They are also only talking about combining two things, which may be a special case.<br /><br />Anyway, if Eugenia Kalnay says she hasn't seen it before (and she does), that's good enough for me! I'm still baffled by the idea that this equation can really be new though. It seems like it might have a bit of an impact.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-6986295745384698742010-06-23T20:43:30.204+01:002010-06-23T20:43:30.204+01:00First of two papers linked by anonymous referenced...First of two papers linked by anonymous referenced Gauss 1809 so obviously your search won't be complete until you have gone back at least that far ;)crandleshttps://www.blogger.com/profile/15181530527401007161noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-4287927700956926012010-06-22T23:30:48.734+01:002010-06-22T23:30:48.734+01:00Belette,
I'll post about the mean being bette...Belette,<br /><br />I'll post about the mean being better than all models, but this result explains precisely the "thing" that I set out to explain, which is that the mean outperforms most models (though not necessarily all) by a significant margin.<br /><br />I can't really bring myself to dis Armstrong when it seems that no-one else knows this result (even though some can at least prove that the mean always outperforms a typical model). I'm relieved that my co-author vetoed some slightly sarcastic comments in our manuscript about climate scientists being unaware of this well-known result!James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-59933331325001353182010-06-22T18:12:11.345+01:002010-06-22T18:12:11.345+01:00Closest I can find are these:
http://journals.ame...Closest I can find are these:<br /><br />http://journals.ametsoc.org/doi/abs/10.1175/1520-0493%281977%29105%3C0228%3AHTIABC%3E2.0.CO%3B2<br /><br />http://journals.ametsoc.org/doi/abs/10.1175/1520-0493%281977%29105%3C1198%3ACOTIAB%3E2.0.CO%3B2<br /><br />From a reference in an ECMWF paper, which also discusses the issue.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-9959776.post-91776861027175924362010-06-22T17:16:03.544+01:002010-06-22T17:16:03.544+01:00I'm not convinced your result explains things....I'm not convinced your result explains things. Yes it is very neat and all, but it only explains what you say: why |avg(m)-O| is better than avg(|m-O|) (or whatever :-).<br /><br />But the "observation" is that avg(m) is generally better than most if not all of the models. I don't think you've explained that.<br /><br />Go on, write a paper dissing Armstrong and his ilk.William M. Connolleyhttps://www.blogger.com/profile/05836299130680534926noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-20587719154622936652010-06-21T05:30:51.864+01:002010-06-21T05:30:51.864+01:00Wow, there's decades of literature on combinin...Wow, there's decades of literature on combining forecasts surveyed there, and it seems that none of them know this result.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-91585064355791845782010-06-19T16:36:20.524+01:002010-06-19T16:36:20.524+01:00James,
Armstrong does seem to rely on empirical e...James,<br /><br />Armstrong does seem to rely on empirical evidence rather than a proof like yours.<br /><br />Looking though his <em>Principles of Forecasting</em> (available to browse at <a href="http://www.amazon.com/Principles-Forecasting-International-Operations-Management/dp/0792374010/ref=sr_1_1?ie=UTF8&s=books&qid=1276960282&sr=1-1" rel="nofollow">Amazon</a>), especially Armstrong's section on Combining Forecasts in Chapter 13, it seems dominated by empirical arguments.<br /><br />You could always ask him if the mathematical solution is well-known... :^)<br /><br />-ChipChip Knappenbergerhttp://www.worldclimatereport.comnoreply@blogger.comtag:blogger.com,1999:blog-9959776.post-71786109093056374992010-06-18T23:29:07.454+01:002010-06-18T23:29:07.454+01:00Oh my, that was easy!
But I've not seen exact...Oh my, that was easy!<br /><br />But I've not seen exactly that before, as best as I can recall.David B. Bensonhttps://www.blogger.com/profile/02917182411282836875noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-6527310903249353652010-06-18T22:56:33.717+01:002010-06-18T22:56:33.717+01:00Chip, thanks for that.
Coincidentally I saw somet...Chip, thanks for that.<br /><br />Coincidentally I saw something related a while back...<a href="http://www.forecasters.org/ijf/journal-issue/300/article/6157" rel="nofollow">this Armstrong article on significance testing</a> criticises a paper <a href="http://www.forecasters.org/ijf/journal-issue/282/article/5997" rel="nofollow">KFHS</a> which states "Finally, we find that the M3 conclusion that a combination of methods is better than that of the methods being combined was not proven."<br /><br />Well it has been now :-)<br /><br />(The Armstrong paper is unaware of the proof, but focuses on the inappropriate use of significance testing - another of my frequent complaints.)<br /><br />I can't help but be amused that these self-appointed experts in forecasting don't know this result (though as DM showed, it is definitely well known in some quarters). Anyway I can hardly be too snarky about Armstrong when the climate scientists were unaware of it too!James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-54802483066940094702010-06-18T18:26:32.974+01:002010-06-18T18:26:32.974+01:00James,
I came across a reference to a similar top...James,<br /><br />I came across a reference to a similar topic in the forecasting literature recently. Perhaps you may find it to be of interest:<br /><br /><a href="http://www.forecasters.org/ijf/journal-issue/310/article/6222#abstract" rel="nofollow">http://www.forecasters.org/ijf/journal-issue/310/article/6222#abstract</a><br /><br />-ChipChip Knappenbergerhttp://www.worldclimatereport.comnoreply@blogger.comtag:blogger.com,1999:blog-9959776.post-54221443090243682112010-06-18T12:46:09.360+01:002010-06-18T12:46:09.360+01:00Yes, the second part of my comment was essentially...Yes, the second part of my comment was essentially James' point - however he hadn't exactly specified the problem completely, and I was thinking more in terms of the question of whether observations remain within the bounds set by the variance in the "model" - well slightly confused on the point anyway. Within the context of RMS differences, this post does nicely explain why the MMM is always "better".Arthurhttps://www.blogger.com/profile/06249922708053689717noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-90397224169315283172010-06-18T12:32:01.628+01:002010-06-18T12:32:01.628+01:00Thanks DM, I'm sure this idea pops up all over...Thanks DM, I'm sure this idea pops up all over the place where people use ensembles. The book is <a href="http://books.google.com/books?hl=en&lr=&id=-aAwQO_-rXwC&oi=fnd&pg=PA1&dq=Neural+networks+for+pattern+recognition%22+Chris+Bishop&ots=FIMTBkCRgu&sig=b7atP2i_e0Ni_TOMO9cVomOIR78#v=onepage&q=committee&f=false" rel="nofollow">here</a> for anyone interested. I agree that my version is neater :-)James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-8004135922168536842010-06-18T10:07:41.785+01:002010-06-18T10:07:41.785+01:00The result that the sum-of-squared error (SSE) of ...The result that the sum-of-squared error (SSE) of the mean of a committee of models is less than the mean SSE of the individual committee members is quite well known in machine learning, where committees of models are often used as a variance reduction technique. For example, see pages 365-6 of "Neural networks for pattern recognition" by Chris Bishop (an excellent book on the topic - 14,000 cites according to Google scholar!), which obtains the same result by a slightly different method.<br /><br />The reason this happens is because the MMM is smoother and towards the middle of the distribution of the individual models. The SSE punishes large errors much more harshly than small errors (due to the squaring). This means that the average of the errors for the individual models will be dominated by the models with the greatest errors. This happens for two reasons, either they lie on the other side of the mean to the observations (high bias) or they are very noisy and the noise is badly correlated with the observations (high variance).<br /><br />Jame's proof is still neat though, and easier to understand without the use of expectations.Dikran Marsupialnoreply@blogger.comtag:blogger.com,1999:blog-9959776.post-17650762571838385522010-06-18T09:28:59.106+01:002010-06-18T09:28:59.106+01:00>Hey, the ENSO thread is that way ^
OK
>It...>Hey, the ENSO thread is that way ^<br /><br />OK<br /><br />>It turns out that this probability varies widely depending on a handful of factors.<br /><br />Clearly the greater the variation between models, the greater the advantage the MMM has.<br /><br />The nearer the obs trend is to the MMM trend, the more likely the MMM is to beat all models.<br /><br />If the individual models have very similar trends then the MMM is more likely to beat all models.<br /><br />A model having very similar deviations from trend to the obs may make it more likely to beat the MMM.<br /><br />I doubt I am getting up to the 'handful' yet.<br /><br />Oops, that also was "for another time".crandleshttps://www.blogger.com/profile/15181530527401007161noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-73621770640489911802010-06-18T06:37:19.647+01:002010-06-18T06:37:19.647+01:00Hey, the ENSO thread is that way ^
(I know it'...Hey, the ENSO thread is that way ^<br /><br />(I know it's my fault for not writing it yet!)James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-83177934216440251432010-06-18T06:02:22.687+01:002010-06-18T06:02:22.687+01:00Looking at GISTEMP, I was surprised to see that th...Looking at GISTEMP, I was surprised to see that the Jan-May 2010 is net .54 degrees ahead of 1998, and indeed after August 1998 anomalies dropped off to levels that haven't been seen for 10 years, so at this point I'd have to say I'll be surprised if 2010 isn't a new record.Steve Bloomhttps://www.blogger.com/profile/12943109973917998380noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-43734526052254121142010-06-18T05:10:56.939+01:002010-06-18T05:10:56.939+01:00Chris, remember that 1998 dropped off sharply at t...Chris, remember that 1998 dropped off sharply at the end of the year, so all is not lost. All else equal I suppose we can expect this year to drop off less sharply. So for now the odds look to me to be no worse than even, probably a bit better.<br /><br />Also, which source was being used for temps?Steve Bloomhttps://www.blogger.com/profile/12943109973917998380noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-7794134834526518272010-06-18T02:41:02.317+01:002010-06-18T02:41:02.317+01:00ac, I'd definitely be interested in seeing any...ac, I'd definitely be interested in seeing any other written version of this. I suspect it has "folklore" status and is rarely written down as people are usually working on more advanced topics. I certainly can't believe it is unknown in the broader community, indeed it jumped into my mind immediately when a reviewer asked about it so I'm sure I must have seen it in a different context, but I don't know where. The multimodel NWP stuff I've read doesn't seem to quite state it though.<br /><br />Alastair, we can easily create ensembles where one model is closer to the obs than the mean is - just add a perfect model (ie one that matches the obs) to the ensemble! Also, error cancelling will often occur in practice, but it is not necessary for the formula to work.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-39375245635059663012010-06-18T01:39:26.059+01:002010-06-18T01:39:26.059+01:00Answering my own question:
-1, 3 -> 1 + 9 = 10...Answering my own question:<br /><br />-1, 3 -> 1 + 9 = 10<br />-3, 1 -> 9 + 1 = 10<br />-2, 2 -> 4 + 4 = 8<br /><br />so that works, but what about <br /><br />2, 4 -> 4 + 16 = 20<br />4, 6 -> 16 + 36 = 52<br />3, 5 -> 9 + 25 = 34<br /><br />Mean of 34 > 1st model of 20!Alastairhttps://www.blogger.com/profile/15152292130415788120noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-14470785416851058092010-06-18T00:36:43.808+01:002010-06-18T00:36:43.808+01:00Neat. I recall reading a similar analysis in the s...Neat. I recall reading a similar analysis in the seasonal prediction literature, but the focus was correlation, not RMSE, and the theme was potential predictability. Could have been a tech report or just a set of slides. If it's of interest I might be able to scrape my memory for an author or other unique identifier.achttps://www.blogger.com/profile/08817718132877704613noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-6059869759822049132010-06-18T00:04:20.747+01:002010-06-18T00:04:20.747+01:00What if the errors are (-1, 3) and (-3, 1)?What if the errors are (-1, 3) and (-3, 1)?Alastairhttps://www.blogger.com/profile/15152292130415788120noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-37650807017307699442010-06-17T23:43:03.214+01:002010-06-17T23:43:03.214+01:00Chris, an ENSO post is long overdue, and on my lis...Chris, an ENSO post is long overdue, and on my list. In brief, yes I suspect you are right...James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-33309888954895375062010-06-17T23:25:44.874+01:002010-06-17T23:25:44.874+01:00Chris, I didn't get that out of what Arthur sa...Chris, I didn't get that out of what Arthur said, but I might have misunderstood it (the first part of his comment may have put me off the track).<br /><br />Belette, fixed, but not exactly as you suggest!<br /><br />Anon #1, well if the mean is better than the typical model (in practice by a significant margin), it is surely going to be better than *most* models (for reasonable distributions). As Tim says, the question of the mean being better than *all* models is not amenable to a direct proof. In fact in the paper we consider it in a probabilistic form - that is, what is the probability of a single model being better than the mean? It turns out that this probability varies widely depending on a handful of factors.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.com