tag:blogger.com,1999:blog-9959776.post3339262346793851592..comments2024-02-15T04:42:41.606+00:00Comments on James' Empty Blog: How not to combine multiple constraints...James Annanhttp://www.blogger.com/profile/04318741813895533700noreply@blogger.comBlogger7125tag:blogger.com,1999:blog-9959776.post-26938825173240244412009-03-14T23:39:00.000+00:002009-03-14T23:39:00.000+00:00I mean the Bayesian emulator ideas introduced by K...I mean the Bayesian emulator ideas introduced by Kennedy and O'Hagan, now being developed in the climate context by <A HREF="http://www.maths.bris.ac.uk/~mazjcr/" REL="nofollow">Jonty Rougier</A> and people at the Hadley Centre. I think this is probably what "Gaussian process emulators" refers to in your paragraph.<BR/><BR/>The random forests abstract is already up on the EGU meeting site, so I don't think you need to worry about breaching confidences! Interestingly, the poster seems to be withdrawn, which is a shame.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-62495570648130090912009-03-14T20:53:00.000+00:002009-03-14T20:53:00.000+00:00>"The NN stuff is potentially interesting,...>"<I>The NN stuff is potentially interesting, although there are alternative (standard) methods for emulation which have a much more rigorous statistical foundation. It is not clear to me why they chose this approach, or how its performance compares to the standard method.</I>"<BR/><BR/>What do you consider "the standard method"? <BR/><BR/><BR/>I now find out there is work going on to use "a technique called Random Forests as an emulator, which benefits from requiring very little tuning to get good results and it also provides a measure of uncertainty in each prediction.....<BR/>sending out some verification runs on CPDN .... each with an initial condition ensemble of 4 members, based on a continuous sampling of parameter space (rather than the discrete sampling used in the original ensemble), and am currently analysing the results. [student] plans to compare different emulation techniques, namely neural nets v random forest v gaussian process emulators and highlight the strengths and weaknesses of each.<BR/><BR/>(I expect the above will be made publically available soon so I hope there isn't too much harm in posting this now in this obsure location.)crandleshttps://www.blogger.com/profile/15181530527401007161noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-30830355888385210512008-09-27T17:41:00.000+01:002008-09-27T17:41:00.000+01:00Making no pretense at understanding the math -- th...Making no pretense at understanding the math -- this showed up in "related items" searches for climate articles, and vice versa. It appears from this that the economists have rediscovered Catton's "Overshoot" caution about removing redundancy from the human economy to the point where one hiccup causes a crash. They are warning one another about risks of low probability events -- like whatever happened to the credit system.<BR/><BR/>Brief excerpt in case it tempts any of the climate scientists to comment:<BR/><BR/>http://www.edge.org/3rd_culture/taleb08/taleb08_index.html<BR/><BR/>"3) Beware the "atypicality" of remote events. There is a sucker's method called "scenario analysis" and "stress testing"—usually based on the past (or some "make sense" theory). Yet I show in the appendix how past shortfalls that do not predict subsequent shortfalls. Likewise, "prediction markets" are for fools. They might work for a binary election, but not in the Fourth Quadrant. Recall the very definition of events is complicated: success might mean one million in the bank ...or five billions!<BR/><BR/>4) Time. It takes much, much longer for a times series in the Fourth Quadrant to reveal its property. At the worst, we don't know how long. Yet compensation for bank executives is done on a short term window, causing a mismatch between observation window and necessary window. They get rich in spite of negative returns. But we can have a pretty clear idea if the "Black Swan" can hit on the left (losses) or on the right (profits).<BR/><BR/>The point can be used in climatic analysis. Things that have worked for a long time are preferable—they are more likely to have reached their ergodic states.<BR/><BR/>5) Beware Moral Hazard. Is optimal to make series of bonuses betting on hidden risks in the Fourth Quadrant, then blow up and write a thank you letter. Fannie Mae and Freddie Mac's Chairmen will in all likelihood keep their previous bonuses (as in all previous cases) and even get close to 15 million of severance pay each.<BR/><BR/>6) Metrics. Conventional metrics based on type 1 randomness don't work. Words like "standard deviation" are not stable and does not measure anything in the Fourth Quadrant. So does "linear regression" (the errors are in the fourth quadrant), "Sharpe ratio", Markowitz optimal portfolio, ANOVA shmnamova, Least square, etc. Literally anything mechanistically pulled out of a statistical textbook.<BR/><BR/>My problem is that people can both accept the role of rare events, agree with me, and still use these metrics, which is leading me to test if this is a psychological disorder. <BR/><BR/>The technical appendix shows why these metrics fail: they are based on "variance"/"standard deviation" and terms invented years ago when we had no computers. One way I can prove that anything linked to standard deviation is a facade of knowledge: There is a measure called Kurtosis that indicates departure from "Normality". It is very, very unstable and marred with huge sampling error: 70-90% of the Kurtosis in Oil, SP500, Silver, UK interest rates, Nikkei, US deposit rates, sugar, and the dollar/yet currency rate come from 1 day in the past 40 years, reminiscent of figure 3. This means that no sample will ever deliver the true variance. It also tells us anyone using "variance" or "standard deviation" (or worse making models that make us take decisions based on it) in the fourth quadrant is incompetent.<BR/><BR/>7) Where is the skewness? Clearly the Fourth Quadrant can present left or right skewness. If we suspect right-skewness, the true mean is more likely to be underestimated by measurement of past realizations, and the total potential is likewise poorly gauged. A biotech company (usually) faces positive uncertainty, a bank faces almost exclusively negative shocks. I call that in my new project "concave" or "convex" to model error....Hank Robertshttps://www.blogger.com/profile/07521410755553979665noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-19830410023805519072008-09-12T11:03:00.000+01:002008-09-12T11:03:00.000+01:00Their paper really discussed methods and generalit...Their paper really discussed methods and generalities with no solid numerical answers, so there is nothing really to "correct" in that sense. The difficulty in real applications is probably in deciding what the matrix should be rather than actually solving the resulting problem (it may depend on the size of the problem too).<BR/><BR/>I'd be very happy if the whole sorry mess died a death, but the worst case scenario of course is that some poor sod unconnected with the original work reads: "we have chosen to average the quadratic cost functions rather than adding them. ... to combine errors in this [latter] way would be incorrect", takes this to mean that averaging is correct, and does some completely bogus calculation as a result. Of course I may be fussing over nothing. We'll have to wait and see...but it seems they have gone to some lengths to promote this averaging method while maintaining plausible deniability That is, they have not actually made any explicit claims that averaging is <I>ever</I> correct in <I>any</I> situation at all, but it is easy to see how this paper - and indeed that specific section I quoted - could be interpreted as supporting the approach.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-80742185631285717642008-09-11T16:38:00.000+01:002008-09-11T16:38:00.000+01:00>"Try plotting x^2, 0.01x^2 and 100x^2 (we...>"Try plotting x^2, 0.01x^2 and 100x^2 (well you hardly need to go to the trouble...) and of course they all have the same minimum, which in mathematical terms is perfectly well defined, but in the context of the paper this phrasing refers to the spread in x for which the cost does not vary very much (for some rather vague definition of "very much") and thus the minimum of 100x^2 is much more tightly defined than 0.01x^2."<BR/><BR/>Yes I expressed that badly it needed a relative comparison (ie relatively how well defined it is). If you plot x^2+30 and y^2+30 and find x is better defined than y for some vague 'not very much' (a%?) variation in cost. If instead you plot 100x^2+3000 and 100y^2+3000 you will still end up with the same x is relatively better defined than y conclusion.<BR/><BR/>So if I have now understood correctly (but demonstrated I have failed to express that clearly in writing), the conclusion in the abstract that I quoted remains unaffected in the idealised case of independent observations. <BR/><BR/>>"The main point remains that the operation of averaging the constraints cannot correspond to learning from new observations."<BR/><BR/>I don't, at present, see any reason to dispute this or of this being the main point of your post. So I have nothing much to say. I would prefer to discuss the extent to which the problem you have identified and explained remains a problem to the the conclusions that have been left in. ie could co-varying observations change a well defined relative comparison if your correct full covariance matrix method was used instead of the papers averaging approach?<BR/><BR/>I am just wondering if no answer means the full matrix solution could well be either an intractable problem which might make it not an available alternative or that the effect are so negligable it is drowned out by other issues. <BR/><BR/>Either of these would appear to make some sense for your decision to not write a comment ... but wait until the method is copied somewhere else in a way that it can affect a papers conclusion?<BR/><BR/>Probably too soon for such pondering, absence of evidence ...crandleshttps://www.blogger.com/profile/15181530527401007161noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-66321477687676349042008-09-11T15:02:00.000+01:002008-09-11T15:02:00.000+01:00I get the impression that an "unjustifiable multip...<I>I get the impression that an "unjustifiable multiplicative scaling factor of sqrt(N) on the width of the implied joint likelihood function" would not have any effect on where the minimum is or how well defined it is.</I><BR/><BR/>It won't affect the location, but in context "how well defined it is" is precisely the "sharpness" of the minimum. Try plotting x^2, 0.01x^2 and 100x^2 (well you hardly need to go to the trouble...) and of course they all have the same minimum, which in mathematical terms is perfectly well defined, but in the context of the paper this phrasing refers to the spread in x for which the cost does not vary very much (for some rather vague definition of "very much") and thus the minimum of 100x^2 is much more tightly defined than 0.01x^2. The main point remains that the operation of averaging the constraints cannot correspond to learning from new observations.<BR/><BR/>I'm surprised the final version is not on the CPDN web site, as they usually put their stuff up pretty quickly.<BR/><BR/>The NN stuff is potentially interesting, although there are alternative (standard) methods for emulation which have a much more rigorous statistical foundation. It is not clear to me why they chose this approach, or how its performance compares to the standard method.James Annanhttps://www.blogger.com/profile/04318741813895533700noreply@blogger.comtag:blogger.com,1999:blog-9959776.post-75619280400913118342008-09-11T11:21:00.000+01:002008-09-11T11:21:00.000+01:00Cannot comment much without reading the paper.The ...Cannot comment much without reading the paper.<BR/><BR/>The abstract includes:<BR/><BR/>"The use of annual mean or seasonal differences on top-of-atmosphere radiative fluxes as an observational error metric results in the most clearly defined minimum in error as a function of sensitivity, with consistent but less well-defined results when using the seasonal cycles of surface temperature or total precipitation."<BR/><BR/>I get the impression that an "unjustifiable multiplicative scaling factor of sqrt(N) on the width of the implied joint likelihood function" would not have any effect on where the minimum is or how well defined it is. However that assumes the observations are independent or so close as to not matter. If there are correlations, would you consider that this would be likely/very likely/etc to have much(?) effect on where in parameter space the minimum is or how well defined it is? <BR/><BR/>If it is likely to have a noticable effect, is this likely to be intractable or could estimates be made?<BR/><BR/>I imagine errors in the neural emulation might (also?) be quite significant.crandleshttps://www.blogger.com/profile/15181530527401007161noreply@blogger.com