Police left red-faced after arresting Japanese woman they thought was a foreigner - MSN-Mainichi Daily News

This is an amazing bit of news: Police left red-faced after arresting Japanese woman they thought was a foreigner:

At around 7:40 p.m. on Saturday, three officers spoke to a 28-year-old woman walking on a street in Kawaguchi, and asked her name and nationality because she looked like a woman from Southeast Asia, according to the officials.

After saying, "I'm Japanese," she refused to talk to the officers, who took her to the police station. After she refused to respond to the questions officers asked her in Japanese, police deemed that she was a foreigner.

<>They held her for 14 hours, merely for "looking Asian" (like the UK attitude to "European", some Japanese use "Asian" to mean "Asian excluding Japanese"). Of course, the idea that Japanese are visibly different from other Asians is pretty ridiculous in the first place (as this case proves), considering this to be a justification for arrest is absurd.
<>

Guardian Unlimited: Climate scientists issue dire warning

There's an interesting article by David Adam in the Grauniad today, entitled Climate scientists issue dire warning:

The Earth's temperature could rise under the impact of global warming to levels far higher than previously predicted, according to the United Nations' team of climate experts.

A draft of the next influential Intergovernmental Panel on Climate Change (IPCC) report will tell politicians that scientists are now unable to place a reliable upper limit on how quickly the atmosphere will warm as carbon dioxide levels increase.

So apparently one of the lead authors has been "leaking". However, later the article says:

According to sources who have seen it, the draft now assumes a doubling of carbon dioxide would cause a likely temperature rise of between 2 and 4.5C, but says higher increases are possible.

Peter Cox describes this as a "significant" move, but it doesn't actually sound like much of a change at all to me. The previous 1.5-4.5C range was only confident at the "likely" level, obviously implying a significant chance of a higher figure. Maybe something got a bit lost in translation, or maybe he knows of some more details not reported here. Anyway, us mere mortals will have to wait another month or so before we get to see the second draft (even though I was listed as a "contributing author" on the first draft of one of the chapters). It will be interesting to see how the lead authors attempt to reconcile the various opinions expressed in the literature.

Further on in the article, Peter Cox is quoted as making this strange statement:

"The scientific agenda has moved from improving the predictions to thinking about what are the chances of something awful happening."

and Dave Stainforth talks about the "destruction of society". I think that amply vindicates my comment about people "talking up the possibility of disaster".

I'm Scottish!

Since coming to Japan, I've frequently been referred to as "English". It's not something I make a big fuss over, particularly as Japanese doesn't seem to have a clear distinction between British and English. Once or twice when I've tried to explicitly describe myself as Scottish, people end up drawing maps and trying to place this small country somewhere in the vicinity of Norway and Sweden, so I've pretty much given up bothering. But every so often, I feel it's worth reminding people where I come from :-)

Scotland is mostly famous for whisky, haggis, and deep-fried delicacies such as Mars bars, pizza, and the aforementioned haggis. On the sporting front we specialise in curling and elephant polo (unfortunately, this time the curlers finished 4th and 5th in Turin - the women even lost to Japan). Many famous scientists and inventors are actually Scots, including people like Alexander Graham Bell who many USAians may think of as one of their own (he emigrated and founded Bell Labs). As well as his telephone, probably all readers will have benefited from the work of John Logie Baird (TV), Watt (steam engine), Dunlop (pneumatic tyre), McAdam (tarmac roads), Kirkpatrick Macmillan (bicycle), Napier (logarithms), Fleming (penicillin), Maxwell (his equations) and others too numerous to mention.

Of course jingoistic nationalism isn't really a rationally defensible attitude. These white men all died before I was born, and I had precisely nothing to do with the Scottish rugby victory. But that won't stop me from enjoying it :-)

No. of foreigners given refugee status hits record high

Japan granted 46 people refugees status last year, a record high. A further 97 were granted permission to stay.

The equivalent figure in the UK is generally in the region of 10,000 per year. This is not particularly exceptional compared to other European nations.

In other news, Tottori prefecture gave up on its attempt to introduce legislation protecting human rights, including outlawing racial discrimination. The reason given is that the lawyers didn't like it. Just to clarify this, they aren't merely giving up, but are actually introducing a new bill to suspend enforcement of the HR legislation. As one person here comments, they could justifiably call this new bill the "Tottori Inhuman Rights Act of 2006".

Farmers' response to bird flu

I am struck by some similarities between the approaching bird flu epidemic, and Foot and Mouth disease which hit the UK a few years ago.

At the start of the F&M outbreak, the National Farmers' Union was strongly opposed to any basic biosecurity measures such as restrictions on movement of cattle. Instead, they issued sterotypical "get orf moi laaaaaand" calls for tourists to stay away (the economic value of tourism is massively bigger than farming, and of course the disease isn't spread by tourists anyway, but it was a good chance to flex their muscles). The Govt (especially MAFF, which despite officially having responsibility for the countryside was essentially ruled by farming interests) went along with this, and imposed various fines and restrictions. The delay in cattle movement restrictions resulted in a massive national epidemic of this virulent and highly infectious disease (which fortunately has no implications for human health), and the entirely pointless shutdown of the tourist industry wasted billions of pounds.

Eventually, the Govt realised that the shutdown of the tourist industry was a massive over-reaction, but by then the damage had been done. The farmers got their compensation (ok, I'm sure that plenty of them also lost some money) but the tourism-related industries are still struggling to recover. MAFF deservedly ended up on the funeral pyre also, and its replacement DEFRA has a more explicit responsibility for Rural Affairs in general rather than the special interest of agriculture.

Now we have bird flu approaching, and what do the NFU do? In response to a suggestion that it might be sensible to keep poulty indoors, they call this a massive over-reaction. All we need is for them to do is to follow this up with exhortations for the townies to stay away, and history will have repeated itself in only 5 short years.

(If you've got some time to kill, you might enjoy this usenet thread from a few years ago - at 1134 messages, I think it's my most successful troll ever. Unlike most trolling, it was however entirely true, and was clearly vindicated within a few months.)

Spring time

At last, it feels like spring is here. It was a warm bright day today, so we took the day off to enjoy the ume blossom. Ume is usually translated into English as "plum", but it's not really a plum as eaten in the West - in fact it is more of an apricot, and the fruit are small, hard and bitter, often eaten as a sour pickle (umeboshi).

We also heard some Japanese nightingales singing today for the first time this year. I didn't realise they were traditionally associated with ume blossom until reading the Wikipedia pages.


Apparently the Japan Meterological Agency is thinking of officially naming the recent bad weather "Heisei 18 Heavy Snow" (Heisei 18 is 2006, in the Japanese naming system - the 18th year of the current Emperor's reign). It's been the worst winter in decades. There's more snow forecast tomorrow, but warmer weather should be just around the corner.

Retirement age 'should reach 85'

It's interesting to see this idea getting more air-time, and also that people are starting to make more radical suggestions (the UK Govt's position is in comparison some rather feeble kite-flying which talked of about a 3 year increase in retirement age, starting some time after 2030).

As a foreign researcher in Japan, I'm compulsorily enrolled into the pensions system, which means that money is automatically removed from my salary with no realistic possibility of my ever getting a payout (I'd have to contribute for 20 years continuously, which considering it will take another 5 years of annually-renewed contracts before it would be worth even bothering to apply for permanent residency rights let alone citizenship, and there is virtually no chance of tenure here, is hardly a serious possibility at this stage). This state-sponsored theft of my salary is just one of those minor niggles that no-one is ever going to fix because (a) it would cost money and (b) there are no votes in it. Since I'm not a taxpayer in the UK, there is no incentive to contribute to a pension fund there either, although I should be able to make some of the shortfall up if and when I return - unless I'm kicked out at age 60 or sooner, which is currently the way things work in the UK research labs. If retirement age is up to 85 by 2050 then I've got plenty of time! (No, I'm not really stupid enough to think this is likely to be the case.)

73,000,000:1

This is the famously erroneous statistic quoted by Roy Meadow, supposedly giving the odds of two children dying of SIDS (cot death) in the same family, when appearing as an "expert witness" in a murder trial. His "evidence" neatly illustrates two distinct fallacies in the same calculation, and as well as resulting in some wrongful convictions which were eventually overturned, he was struck off by the General Medical Council, but has just been reinstated following an appeal.

Firstly the fallacies: he got his number by squaring the odds of a single child dying in this way, of about 8,500:1. The implicit assumption is that such deaths strike entirely at random. However, if there are any genetic or environmental factors involved, a the probability second death in the same family would be far greater. Ignoring this factor is known as the ecological fallacy. In fact medical research suggests a second death happens in about 1 out of 100 cases where a first death has happened.

A second and more pernicious fallacy is the prosecutor's fallacy, so called because it seems pretty near ubiquitous in the presentation of statistical evidence (eg DNA tests). It is the incorrect interpretation of a probability of an event happening to an innocent person, as implying the probability that this person is in fact guilty.

An example: assume that you are a juror, and you've been told that DNA testing provides a match to the suspect at the 1,000,000:1 level (ie, a random person will match with probability 1 in 1,000,000). What is the probability that this person committed the crime? 999,999/1,000,000? No! Given a population of 60,000,000 in the UK, there will be about 60 people whose DNA matches that well. A priori, any match only indicates a 1 in 60 chance that a particular person did it - if there's no other evidence, this is all but worthless. However, it would be a rare juror who would understand this (and probably a rare lawyer). In practice, matches are often quoted at a much higher level of significance, but by the time we are up to billions to one, the chances of accidental contamination or fraud must be higher than that anyway.

Back to Meadow. He was reinstated recently, with the judge ruling that expert opinion given in court should be "priviledged" in the sense that incompetence is not grounds for the GMC to punish the witness. While that may seem rather bizzarre, I note that no-one has called for the defence lawyers to be themselves disbarred, for failing to produce any expert witness who understood elementary statistics well enough to destroy (that aspect of) Meadow's evidence. That is surely incompetence on a similar level to Meadow's.

Probability, prediction and verification VI: Verification

At last, and after getting slightly sidetracked in various ways, I'll get back to the meat of things.

Forecast verification is the act of checking the forecast against the reality, to see how good it was. The basic aim is to see if the forecast was valid, in the sense that reality did not throw up any major surprises. You don't want your forecast to be confident of sunshine and warmth, but reality to be cold and rainy.

Obviously, for any current forecast, this check cannot even be attempted before the valid time of the forecast has arrived. So anyone who complains that today's multidecadal climate forecasts cannot be verified is merely stating a truism based on the definitions of the terms. A weather forecast also cannot be verified in advance of its valid time (say, tomorrow). But this in itself obviously does not mean that weather forecasts cannot be trusted and used. On the contrary, they prove themselves to be highly valuable on a daily basis, with industries ranging from agriculture to the military depending heavily on them. (That's despite there being ultimately no objective rigorous basis for the way in which the epistemic uncertainty in weather prediction is handled, as I've explained in more detail here, here and here.)

In fact, even after the valid time of the forecast has passed, and even assuming that precise observations are available, verification is still not a trivial matter. Returning to my previous example of a rain forecast, if the forecast said "70% chance of rain", then either a rainy or dry day is an entirely acceptable outcome. So was that forecast inherently unverifiable? The inevitable answer is that yes, of course it was! Even for a quantitative forecast ("tomorrow will have a max of 12C, with an uncertainty of 1C"), it will only be on the rare occasion that the observed temperature falls far enough outside the plausible range of forecast uncertainty that one might be able to say that the forecast failed to verify. In fact, if we assume the forecast uncertainty is Gaussian (or any other continuous unbounded function), there is no threshold at which the forecast fails to verify in absolute terms - you might simply have got the 1 chance in 1000 that the target was 3 standard deviations from the forecast mean. Indeed, with one forecast every day, you'd expect to see this roughly once every 3 years. [Note that we check whether the data lie within the uncertainty of the forecast, not whether the (central) forecast falls within the observational uncertainty of the data - see here for more on this.]

Once you have more than a handful of forecasts, however, you can usually make a realistic assessment of the reliability of the system as a whole - if many days are 3 standard deviations from the mean, you'll probably judge it more likely that the system is bad than that you happened to hit the 1 in 10¹⁰⁰ unlucky streak in a good system :-) But the latter can never be truly proven false, of course. Conversely, if the forecast system has validated consistently over a period of time, we will probably trust today's forecast, but even if the system is known to be statistically perfect, there is still a 1 in 1000 chance that it will be 3 standard deviations wrong tomorrow. Each day is a unique forecast based on the current atmospheric state, which has not occurred before. As I explained before, the forecast uncertainty is fundamentally epistemic not aleatory, so there is no sense in which there is a "correct" or "objective" probabilistic forecast in the first place. The uncertainty is fundamentally a description of our ignorance, not some intrinsic randomness.

Obviously it would take a long time to collect adequate statistics from successive 100 year climate forecasts if we started now. And given the rate of ongoing model development, this approach could never tell us much about the skill of the most up-to-date model anyway, since they are replaced every few years. We can however, use simulations of the historical record (and the present) to test how well the models can hindcast variations in the climate which are known to have occurred. In its simplest form, this sort of test provides only a lower bound on forecast errors, since the models are largely built and tuned to simulate existing observational data.

When the models fail to reproduce the data, of course it calls their validity into question - at least, it does if the data are reliable. A striking example of models teaching us about reality is in the recent resolution of the tropospheric data/model incompatibility in favour of the models (OK, I'm over-egging things a little perhaps). Looking back over the longer scale, we have Hansen's famous forecast from 1988, which has proved to be spot on over the subsequent 17 years. In fact, the simplicity of the physics means that one thing we really can forecast quite confidently is a continued global warming in coming decades: the IPCC TAR said it was likely to continue at 0.1-0.2C/decade for several decades to come, and although this perhaps could be nudged marginally higher (we are getting close to the 0.2 limit), it won't be far wrong.

A slightly more sophisticated general technique known as cross-validation involves witholding some historical data, training the model on the rest of the data, and seeing if it correctly predicts the data which were witheld. In order to avoid accusations of cheating, it is necessary to use some sort of automatic tuning technique. If the data take the form of a time series which is split into an initial training interval followed by a forecast interval, then this accurately mimics the situation of a real forecast. It is also how new versions of weather prediction systems are tested prior to introduction - repeat the forecasts of the past year (say), and adopt the new system if it shows greater skill than the current one. We demonstrated a simple example of this cross-validation approach in this paper a few years ago, and broadly similar methods can be found throughout the more prediction-focussed corners of the climate research literature (eg Reto Knutti used a neural network in this forthcoming paper, training it on half the data and verifying it on the other half). These sort of formal forecast methodologies have not been widely undertaken in the GCM-building community in the past, partly because until recently there were no computationally-affordable automatic tuning methods, and partly because most climate scientists don't have much of a background in prediction and estimation - they are primarily physical scientists with an interest in understanding processes, rather than forecasters whose main aim is to predict the future. But there is now plenty of work going to bridge this gap, and here's the obligatory plug for the modest contribution we're making in this area :-) Climate scientists may never get to the level that weather forecasting is at, in terms of attaching clear and reliable probabilities to all of our predictions, but we are definitely making progress.

Tea tray tobogganing

The UK doesn't have a winter really, just a colder part of the annual 12-month rainy season. So we don't take "winter sports" too seriously (by which I mean things involving snow and ice - football and rugby in the pouring rain is quite wintry enough for my tastes).

The one thing we do enjoy, when a snowy day comes along, is sliding down hills on tea trays. So it is only fitting that our first medal in this year's Winter Olympics is a silver in the women's tea tray event. (OK, they don't really call it that - but they might as well.) She has to go abroad to do any proper training, so it's really quite a feat.

The Japanese are rather miffed as they have a serious winter sports industry and have yet to get a medal :-) I'm sure they will overtake us soon enough though. The only other thing we do moderately competently is curling, and there are two more medal chances there, but I don't think there is anything else to hope for. In fact Britain has won an average of one medal per Games since the Winter Olympics began in Chamonix in 1924.

Alarming new research?

A report from the Tyndall Centre is getting some attention on the the BBC:

The UK could face major flooding and tropical temperatures by the year 3000 if greenhouse gas emissions are not sharply reduced, a new study says.

The report, from the Tyndall Centre for Climate Change Research, claims Britain could look radically different with sea levels rising as much as 11.4m.

That's a rather garbled take on what is a very speculative bit of research. The high-end numbers are based on projections in which mankind burns not just all the oil and coal, but even the "exotic" fossil fuel resources like methane clathrates, at a rapidly increasing rate over the next few centuries. They base the emissions on the old IS92a from the IPCC SAR, which was already known to be an outdated and fairly extreme scenario at the time of the TAR, and then extrapolate up from that over the subsequent centuries. While it is obviously fun to play around with ideas like this, the suggestion that it has much to do with current policy decisions is tenuous at best. I'm sure that the People's Federation of Great Europia (or however the world is organised by then) will be quite capable of making their own decisions over carbon emissions in the 24th century, and I doubt they will feel bound by our opinions on the matter. Anyway, if the last remnants of humanity are going to be eking out a perilous existence on the fringes of the Arctic, it's hard to see how they will also be burning fossil fuels at more than 6 times the current rate :-)

Before blaming the BBC for misrepresenting the research, have a look at the press release (New science shows urgent action needed today on climate change) and the report itself. For a much more realistic look at mitigation strategies, try this recent well-written RealClimate article instead.

[I note in passing that the Tyndall Centre report misrepresents the climateprediction.net results, although it doesn't directly affect the results they generated (they only use it to support their claim that things could turn out even worse). Given the misleading manner in which those results themselves were presented, I suppose it's hardly surprising.]

More wailing about whales

In another you-just-can't-believe-they'd-do-that attempt to deal with the whale meat mountain, it's getting fed to schoolchildren again. Of course there has been a full and frank discussion in the media about the astonishingly high level of mercury routinely found in whalemeat, as any search on a major news site will show. Perhaps the real goal of this "whale research" is to look into Minamata disease.

This is the same Japan that is currently banning USA beef imports due to the supposed threat of BSE!

UNHCHR slams Japan on racism

Doudou Diene's report on his visit to Japan is published at last, available here. It's not exactly diplomatic:

69. After having collected and analysed the views of all parties concerned, the Special Rapporteur reached the conclusion that racial discrimination and xenophobia do exist in Japan, and that these affect three circles of discriminated groups: the national minorities - the Buraku people, the Ainu and the people of Okinawa; people from and descendants of people from former Japanese colonies - Korea and China; and foreigners and migrants from other Asian countries and from the rest of the world.
70. The Special Rapporteur noticed that the manifestations of such racial discrimination and xenophobia are manifold. First of all, they are of a social and economic nature. All surveys and indicators point to the fact that minorities live in a situation of marginalization and economic and social vulnerability, in the fields of employment, housing, marriage, pensions, health and education. Such inequalities vis-à-vis the rest of the Japanese society should urgently be addressed.
71. Secondly, the discrimination is also of a political nature. The Special Rapporteur noticed the invisibility of the national minorities in State institutions, in particular the Parliament and the Government. For example, the Ainu have only had one congressman in the national Parliament, whom the Special Rapporteur met, but have none at present. Such invisibility shows the depth of exclusion, and increases the sense of discrimination and marginalization of the communities concerned, who are given no opportunity to participate in the managing of their present and future affairs.

The main conclusions are that the Japanese Govt should do something about it:

V. RECOMMENDATIONS
74. The Government, at the highest levels, should officially and publicly recognize the existence of racial discrimination and xenophobia in Japanese society. It should be done by conducting a survey to find out the present conditions of each discriminated group in Japan. The Government, at the highest levels, should also officially and publicly recognize historical and cultural roots of racial discrimination and xenophobia in the Japanese society, and express in clear and strong terms its political will to combat it. Such a message will not only create the political conditions of combating discrimination and xenophobia at all levels of society, but also facilitate the promotion of the complex but profound process of multiculturalism in Japanese society. Moreover, in the context of globalization, such a message will undoubtedly enhance the standing and image of Japan in the world and in particular in the countries economically related to Japan and whose citizens or people migrate or visit Japan. Japanese citizens, who are increasingly visiting foreign countries for tourism or business-related reasons, will be in a stronger moral position not only to combat the manifestations of discrimination they may be subjected to, but also to promote the image of their country.
75. The Government should strongly condemn and oppose to any statement by public officials which tolerates or even encourages racial discrimination and xenophobia, in accordance with article 4 of the International Convention on the Elimination of All Forms of Racial Discrimination, ratified by Japan, and in particular its paragraph (c), which provides that States “shall not permit public authorities or public institutions, national or local, to promote or incite racial discrimination”, and in accordance with article 20 of the International Covenant on Civil and Political Rights, also ratified by Japan, which prohibits “any advocacy of national, racial or religious hatred that constitutes incitement to discrimination, hostility or violence”.
76. The Government and the parliament (Diet) should as a matter of urgency proceed to the adoption of a national law against racism, discrimination and xenophobia, giving effect into its domestic legal order to the provisions of its Constitution and of the international instruments to which Japan is a party, which include the International Convention on the Elimination of All Forms of Racial Discrimination, the International Covenant on Civil and Political Rights and the International Covenant on Economic, Social and Cultural Rights. Such a domestic law should:
Penalize racial discrimination in all its forms, and specifically discrimination in the field of employment, housing and marriage, and guarantee access to effective protection and remedies, including compensation, to victims;
Declare an offence all propaganda and all organizations which are based on racial superiority or hatred and promote or incite racial discrimination, as provided for in article 4 of the International Convention on the Elimination of All Forms of Racial Discrimination. In this regard, the Special Rapporteur shares the view of the Committee on the Elimination of Discrimination that the reservation made by Japan to article 4 (a) and (b) of the Convention is in conflict with Japan’s obligations under article 4, which is of a mandatory nature, and that the prohibition of the dissemination of all ideas based upon racial superiority and hatred is compatible with the rights to freedom of opinion and expression. Therefore, the inclusion in the domestic legal system of a prohibition of all propaganda and all organizations which promote or incite racial discrimination cannot validly be avoided by invoking the rights to freedom of opinion and expression.
The communities concerned should be consulted and should participate in the process of elaboration of this law.

So we can expect to see (at best) a brief flurry of bland comments from the politicians, assurances that they will look into it when the time is right...and the whole matter being quietly forgotten.

The Japan Times on the Royal Pregnancy

I thought I was blunt enough in my previous comment. But the JT pulls no punches it its editorial. Some highlights:

No wonder the Crown Princess gets depressed. The spectacle of the chasm between the Imperial family and the 21st century has long been enough to depress anyone. But then, just when the princess must have thought the gap might be closing a bit, given the prime minister's efforts to win the right of succession for the family's female members, along comes an unexpected pregnancy to send everything back to square one.

It is not that the princess would not wish to congratulate her brother- and sister-in-law, Prince Akishino and Princess Kiko, on their joyous news. The whole nation does. It is just that she must dread having to explain to her 4-year-old daughter why people's joy seems to be so dependent on this new cousin being a boy. Whatever happened to the idea that girls are just as special, just as valued, as boys? How do you explain why some people think being a girl is such a crippling defect it automatically disqualifies you from a job that carries no power anyway? Or why it would still be empowering to women for a woman to accede to a position of such bizarre powerlessness?

[...]

If Japan was truly ready for a female emperor, why is everyone so thrilled about this pregnancy? Television announcers all but wept breaking the news on Tuesday. And opponents of the prime minister's plan appear giddy with relief at the thought that a boy could yet appear and save the nation from the frightful prospect of a reigning empress who could be succeeded in turn by her own daughters.

[...]

Some might argue that this is all a tempest in a teacup, because the emperor system is purely symbolic, anyway. But that is exactly why it is important. What better vehicle than the monarchy to set a symbolic example on social issues? Last week's news has set that effort back -- but there is still room for optimism. The new baby might be a girl, thus putting this crucial debate back on track. Here's hoping.

Ouch!

A week is a long time in politics

3rd February:

Koizumi, however, stressed to reporters on the same day the necessity of having the bill [to allow females and their descendants to ascend Japan's imperial throne] passed during the ongoing session, saying, it is better to "hurry for a stable imperial succession."

"I don't think it is good to discuss the matter too long," he added and reiterated that the party will require all member lawmakers to vote for the bill.

10 February

Prime Minister Junichiro Koizumi indicated Friday he has given up a plan to submit to the current Diet session a bill to allow women to succeed to the Chrysanthemum throne [...]

Koizumi told reporters at his official residence, "We can make a decision after closely examining" developments on the matter within his ruling Liberal Democratic Party.

So what happened in the intervening week to cause such an abrupt U-turn? What happened is that we all found out that the wife of the younger Crown Prince is pregnant, raising the hope of a male heir. Of course, consummate politician that he is, Koizumi is making the natural decision to chicken out of addressing the issue, performing a handbrake turn without so much as a look in the wing mirror. So much for his "gender equality plan".

Obviously I'm hoping the child turns out female, so these old misogynists who run the country will have to face up to life in the 21st century. But I can't help think that Koizumi wouldn't have made this decision unless he knew that the child was indeed going to be male...he will certainly look very stupid if it's yet another girl. He is standing down in September anyway, so it will be someone else's problem.

Detection, Attribution and Estimation

This is something I've been meaning to blog about for some time. It comes up a lot in the context of the hurricane wars, over at RPJnr's blog. A recent comment of his provides a nice opening:

As you well know much of science works through hypothesis - falsification. Across climate science the null hypothesis used to guide research has been that a human signal is NOT present, and research is then done to falsify this hypothesis. In some cases such research has been done allowing attribution of climate effects to an anthropogenic forcing. This null hypothesis is chosen because of Occam's razor, it is a simpler explanation for what is observed.

He is essentially posing the question as initially one of detection - can we show that the AGW has had an effect, and that the observations are not just the result of climate variability? - before moving on to attribution - how much of a change can we describe as being due to this particular cause?

There is, however, an entirely different but equally valid approach that could also be used from the outset, which is: what is our estimate of the magnitude of the effect? The critical distinction is that the null hypothesis has no particularly priviledged position in this approach.

This distinction between detection and estimation is related to that between a frequentist and Bayesian approach to probability. Ruling out a null hypothesis (or not) at some level of significance is essentially a frequentist approach: estimating the probabilities of various competing hypotheses, about which we have prior beliefs (but not necessarily a strong bias towards a null hypothesis) is Bayesian. The answers that these two approaches provide may be very different in any given situation, and neither is necessarily right or wrong a priori, but it is surely self-evident that the Bayesian approach is more relevant to decision-making. If we have any reasonable expectation that certain policies would have particular bad effects, it would be ridiculous to wait until such effects could be shown to have occurred at some arbitrary level of statistical significance (that's not a point specific to climate change, of course).

IMO, the IPCC slightly muddies the waters with its discussion of D&A. It defines attribution, as Roger implicitly does above, as a strictly stronger statement than detection - one can only attribute once the effect has already been detected. I suppose they can define the term "attribution" in that way if they choose, but it is certainly not then valid to equate this with the general estimation problem as they do further down. It is trivial to create situations in which a currently undetectable effect can be reasonably estimated to be large, and the converse is equally possible - an easily detectable (statistically significant) influence may be wholly irrelevant in practical terms. I suspect that this forms a large part of the difference in presentation between various parties in the hurricane debate - the evidence may not yet rule out the null hypothesis of no effect, but some people estimate that AGW is likely to have a substantial effect (even if the ill-defined error bars on their estimate do not exclude zero). In principle, exactly the same evidence could support both of these conclusions, although I don't personally know enough about hurricanes to make a definitive statement in that particular case.

It is amusing to see Roger, very much at the sharp end of policy-relevant work, promoting the scientifically "pure" but practically less useful detection/frequentist approach rather than the more appropriate estimation/Bayesian angle. It's not surprising, although perhaps a little disappointing, that the IPCC explicitly endorses that view. But by placing the null hypothesis in a priviledged position from which it can only be dislodged by a mountain of observational evidence, this approach provides a strong inbuilt bias for the status quo which cannot be justified on any rational decision-theoretic grounds.

Petfood!

So, as you may have seen in the news, Japan's whale meat mountain is getting turned into petfood. This shop is specifically advertising "whale jerky for pets" (that's a direct translation of the red text across this image on the left - the black writing says "it's our recommendation!!") I previously suggested that putting the unwanted whale meat in school dinners would be crass. I'm obviously hopelessly naive!

Food glorious food

Two contrasting food books caught my eye recently.

A couple of weeks ago, the Guardian featured a roadkill recipe book, in which the author describes such delicacies as "Hedgehog spaghetti carbonara". I'm not sure I would go that far myself, but I did once pick up a freshly-killed pheasant. In fact it was so fresh it was not even quite dead when we got to it - we'd seen a puff of feathers as the car in front disappeared round a corner. Sadly we were only spending a weekend at my parents' house, and Dad insisted (correctly) that it had to hang for a week, by which time we had gone home. So we didn't even get to taste it.

The JT takes a look at a more conventional approach which has brought long life and health to Japan. I'm not sure there is any particular magic ingredient - it's common sense that a population which eats modest portions of fresh fish and vegetables, and generally walks and cycles moderate distances rather than sitting in a car, is going to be healthy (the only real vices are too much salt and pickles, the latter of which leads to an abnormally high rate of stomach cancer). It has to be said that having to chow down a bowl of plain steamed white rice with every meal leaves me, if not exactly losing the will to live, certainly losing the will to eat. So it's not surprising that people are so thin - it's rare I see someone here who would be considered overweight in the UK. On our first visit to Japan, jules and I each lost about half a stone in a week: we now keep our weight up with a weekend diet of baked potatoes and a regular circuit of several really good restaurants (mostly "Western-style", if you count a proper Indian curry as such) round here. Our canteen is really excellent, actually. But there's a limit to how much rice and pickles I can face in a week.

Last Chance Saloon

After huge strings of comments, most of which can be found here, here and here, I'm still not sure that I've made much progress in explaining how Bayesian probability works, and how it differs from the frequentist view.

I'll have one last go before I give up....

---

A man picks an apple out of a barrel and gives it to me. I weigh it on a set of moderately accurate scales, which give a reading of 105+-5g (as previously, this uncertainty is assumed to be Gaussian, with the quoted magnitude being 1 standard deviation). What is the probability that the mass of the apple is less than 100g? Less than 105g?

To a frequentist, these questions are simply ill-posed, and no answer is possible. We don't know the distribution of apple masses in the barrel from which this one was sampled.

To a Bayesian, there are an infinite range of possible priors that could in principle be used to describe the initial uncertainty in the apple's mass. One obvious choices would be to take it as a uniform distribution over a wide range (eg 0-200g). In that case, the posterior pdf for the apple's mass is the Gaussian distribution N(105,5), and the two probabilities requested are 16% and 50% respectively. Another plausible prior would be to use our prior knowledge of the apple being a common medium-sized fruit, and reckon that the overall distribution of apple masses is roughly Gaussian of the form N(125,20). In that case, the posterior pdf of the individual apple is marginally different from before, at N(106.2,4.85), and the two probabilities are 10% and 40% for the apple being less than 100g and 105g respectively.

For each of those answers, of course there is also a perfectly straightforward frequentist interpretation - consider sampling apples from the U(0,200) distribution, weighing each, and of those whose weight is given at 105g, the distribution has the N(105,5) shape. And similarly for the Gaussian prior. So what's the difference? The only difference is, someone with a Bayesian approach to probability will be prepared to actually pick a prior (which might be one of the above, or something else), and give an answer. Someone who claims to take a frequentist approach to probability cannot do so. And indeed if a frequentist experiment is actually performed in which apples are repeatedly sampled from the barrel and weighed on scales with +-5g random error, then the resulting distribution of those which are measured to be 105g will not coincide exactly with the Bayesian's answer, unless he got very lucky and actually guessed the distribution of apples in the barrel correctly.

Note, however, that in my two examples above, the answers are not really so different from each other. Unless the prior is something really pathological (like each apple in the barrel is actually less than 100g) then an apple which measures 105+-5g is "likely" to be greater than 100g ("likely" being the IPCC definition of about 60-90%). So even though the Bayesian won't be exactly "correct", he'll probably be near enough right for his answer to be a useful basis for decision-making. And note that even if you decide you need a better answer, and try to take a more accurate measurement, you still can't get round the fact that you always need to choose a prior which will affect the answer (if only marginally). In the real world, you might need to make a decision - shall I buy the apple or reject it as potentially undersized - with the available data and no opportunity to improve your observation. Without a Bayesian approach to probability, you don't have the tools to address that question at all.

Here endeth the lesson.

How to measure the height of a building with the help of a barometer

A small twist on an old tale...it's not really related to the bayesian/frequentist fuss, but rather this...

It was the end of term, and the three top students (one each of engineering, physics, and maths) had one practical exam left to determine which one got the overall prize.

The tutor took them to a clock tower, and gave each one a barometer, paper and pencil. "Your task is to determine the height of the tower in the next 2 hours. You should assume a priori that it is a typical example of its type, which have heights uniformly distributed in the range 20-40m. We've examined the foundations and found that they are not safe for a building of more than 30m height. Obviously we are very worried. Unless we can rule out the possibility that it exceeds 30m at the 99% level or better, we'll have to spend a million pounds reinforcing it. Can you tell us whether this is necessary?" He then walked off.

The students sat down on the front step to think. The physicist and engineer started scribbling ideas. The mathematician thought for a few seconds, then went off to the pub for lunch.

2 hours later, they all assembled at the clock tower, and the tutor asked for their answers. The physicist says, "I observed the air pressure at the bottom of the tower, climbed to the top and observed the air pressure again. Based on the pressure difference, I estimate the height to be 25+-2.5m, which means there is a 2.3% chance of it exceeding 30m. You'd better do the reinforcements." (All uncertainties are assumed Gaussian and quoted as 1 standard deviation.) The engineer says, "I climbed to the top of the tower, and dropped the barometer, timing the fall with the clock. Based on the time it took, I estimate the height to be 27+-2m, which means there is a 7% chance of exceeding 30m. You definitely have to reinforce it."

The mathematician says, "I estimate the tower's height to be 26.2+-1.56m. The chance of it exceeding 30m is less than 1%. Let's spend the money on renovating the bar instead."

The physicist and engineer are aghast. "But we've both proved there is a substantial chance of disaster! Something Must Be Done!"

How did the mathematician calculate her answer, and was her decision the correct one? If she is right, what did the others do wrong?

---

Updated 6/02

Well, this wasn't really intended as a serious problem - I'm sure that you all realised that the mathematician combined the previous two estimates using Bayes' Theorem. Note that the question as posed specifically defines the tower as a sample from a uniform distribution - so it's a perfect well-posed frequentist problem and the Bayesian/frequentist rambling in the comments is completely misplaced.

The vaguely amusing point I was making is that although the engineer and physicist did nothing wrong initially, and both concluded that there was a significant (>1%) danger, as soon as each of them hears the other one agree with their conclusion, their position immediately becomes untenable. Their only fault is to not realise this as quickly as the mathematician did :-)

Sutaabakkusu

Sutaabakkusu is Japanese for Starbucks. And I'm going to come out of the closet and admit that I am a fan. "Boo, hiss" I hear my reader cry. "Why are you supporting such an evil culturally-imperialistic faceless multinational corporation as it stamps its ugly jackboot in the face of timeless Japanese culture and beauty?"

It's 100% no smoking, that's why.

One of the more annoying features of everyday Japan is the omnipresence of smokers. The proportion of the male population who smoke is one of the highest in the world - and there's no getting away from them. I wouldn't care if they kept it to themselves, but they don't. Until recently, a lot of cafes and restaurants didn't even have a no-smoking section. So although you might go in, spot a quiet smog-free corner and sit down there, you would always have the risk that 5 minutes later some old man would sit right next to you, and start coughing and spluttering and blowing smoke over you and your food. This "cigarette of Damocles" hanging over us is hardly conducive to a relaxing meal. Even when a cafe had a no-smoking section, it was never anything more than a few tables barely visible through the haze - about as much use as a "no pissing" end in a swimming pool.

And then, after we'd been here for a year or so, the first Starbucks opened in Kamakura. And it was 100% no smoking (there are a few tables outside for the addicts). And everyone flocked there. For - surprisingly enough - the non-smokers actually outnumber the smokers, and even the smokers like to sit down in a nice clean environment every now and again (you must have noticed how the smoking carriages on trains are always the last to fill up, with smokers preferring to sit in a non-smoking area and pop out for a smoke once or twice).

There was an article about the success of Starbucks in the JT a couple of years ago. OK, here it is. I remember thinking at the time that the author was astonishingly out of touch in the way he seemed to think that their growth was despite, rather than substantially because of, their no smoking policy. Perhaps in his sneering superiority, he had never actually been to one. Maybe he is one of the addicts for whom clean air is an unwelcome assault on the senses. Maybe he is just a nutcase. Anyway, the rest of Japan is scrabbling to follow in Starbucks' footsteps, with no smoking areas now springing up all over the place. Even the newly rebranded in-house JapanRailways "Becks" cafes - haunt of the haggard overworked salaryman - now generally have large, well-segregated no smoking areas. The small branch in Ofuna station is 100% no-smoking for at least part of the day. Our favourite restaurant is also fully no smoking (it's the best Indian restaurant in Japan, so be sure to try it out when you're in the area). There are still plenty of smoky dives to be found, but they are slowly dying out. I'd prefer it if they went no-smoking themselves, but as it is, when I want a coffee or a snack, I look out for the green and white sign.

Recently, a new and even better Starbucks has opened in Kamakura. It might have happened some time ago - it's in a part of town we rarely visit (west of the centre, in front of the city office). It's a big bright airy building and even has a view into a garden. So I can often be found there on a Sunday morning, worshipping the roasted bean at the Church of St Arbucks. In fact, I wrote most of this blog there. Who's for a green tea jelly frappuccino with whipped cream and caramel?

Update 5/2/06

And I'd just like to mention that British-style pub The Tavern in Yokohama has just implemented a no-smoking side too, which is great news! The owner said that far from chasing away customers, it had proved a great success so far. With the bar splitting the room into 2 sections, it looks like the smoke-free part should stay fairly clear. The pub does good British-style food so it will be much improved without someone puffing away right next to us. I know I'll be more likely to pop in for a drink and a bite to eat. Thanks John!

Woohoo!

Our latest paper was just accepted by GRL. I think the contents may be of general interest, so I'll be blogging about it some time soon when I'm less tired and emotional. Those who are interested in the meantime will find it on my work web page.

Probability, prediction and verification V: Commotion in the comments

Well, having posted what I thought was an interesting but essentially uncontentious discussion of Bayesian and frequentist approaches to probability, I was rather surprised to see the reaction in the comments.

I'm going to pick primarily on Wolfgang at first, because he has given his opinions most explicitly - which exposes the inconsistencies most clearly :-) From what I can glean of their comments, I think the others are displaying similarly inconsistent attitudes, however.

He starts out:

In physics statements about probability makes sense in two cases:
i) quantum theory
ii) systems where we do not know the initial state exactly.

In all cases we must be confident about the theory (kinematics and dynamic), otherwise the use of probability is misleading.

Probability is not a good replacement for missing or wrong theories. Your example of the prime number illustrates this very well.

I could perhaps quibble over exactly what is meant by "we do not know the initial state exactly", but this looks like it is at least trying to be a fundamentally frequentist position. Fine, if that's what he really wants.

But later on, I pose the following problem in the comments to CIP's post (nb the electron neutrino is just some arbitrary poorly-known particle: I am assuming the Wikipedia page is reasonably valid when it says its mass is thought to lie between 0.1eV and 2.5eV, but note that even this statement is only meaningful in a Bayesian worldview!):

Say you are trying to measure the mass of the electron neutrino. One possible experiment costs $1 million, and would work down to 0.5eV. Another possible experiment would cost $10 million but would work to 0.1eV. You have other demands on the money. Do you consider there is any information in the above that can help you choose one experiment rather than the other? Do you consider that questions along the lines of "how likely do I think it is that the mass of the neutrino is less that 0.5eV" are even legitimate in the first place, or do you simply reject that as an ill-posed question? If that is the case, how do you actually make the decision in practice? If the cost of the second experiment were 1.1 or 100 million, would your decision change?

After some prompting, Wolfgang replied that

if you have reason to believe that the neutrino mass is in the range m = 0.1-2.5eV you can calculate probabilities.

This is because you have different scenarios ( m = 0.11, m = 0.21, etc.) which are all consistent and compatibel with what you know.

But this is precisely espousing the Bayesian attitude that he rejected previously! So it seems that he's a Bayesian without realising it, or perhaps a Bayesian depending on how he feels at the time. He has not yet explained how he would calculate these probabilities, or what they could possibly mean in a frequentist world.

CIP makes similarly odd noises in his posts here and here. He says:

Subjective plausibility makes my skin crawl. I have no idea what the hell it is supposed to mean. I can't think of any case when an expressed probability isn't either based on an explicit or implicit frequentist interpretation or else, like Mr. Spock's invariably misguided predictions, merely an ignorant prejudice.

but has so far ducked my question about the electron neutrino.

There were other various comments along vaguely similar lines. I think that some of these people don't actually understand how the Bayesian and frequentist views of probability actually differ. I'll try to explain further with a simple example.

I have a max-min thermometer in my garden, which I reset each day. It measures the max and min, and let's assume that I know that the error on each day's measurement is well-characterised as a standard gaussian deviate - ie a sample from N(0,1). This morning I went out and saw that the min from the last night was -5C. What is the probability that the actual minimum was below -5C?

Most people will instinctively answer that the probability is 50%, reasoning that the error is just as likely to be positive as negative. Most people would be wrong. The probability depends on the prior distribution before you looked at the thermometer! This is an unavoidable consequence of how Bayes' Theorem works. The posterior is equal to the prior multiplied by the likelihood of the true temperature given the observation. Those who do not understand this will repeatedly get themselves into trouble when discussing probabilistic estimation.

To those who insist that 50% is the right answer, consider this situation: Say I know that the record low temperature ever recorded in my town (of which my garden is representative) is -4C. Will I still believe that there is a 50% chance that last night's temperature was below -5C? Of course not. Temperature records are virtually never broken by such a large margin. In this case, my prior probability density function has its mean (and the bulk of its support) above -5C, so even after measuring -5C, the posterior mean is above -5C. Even with a less extreme example - say the record low is -6C, but the typical February daily minimum temperature is -1C - a rational Bayesian will still (probably, based on the info supplied) conclude that the temperature was probably warmer than -5C.

The naive frequentists will say that they can interpret this in a frequentist sense - consider sampling the temperature each day, and on those days when you see a temperature of -5C, was it warmer or colder? But what distribution do you sample your days from? Statistics for every day in the last 30 years? Every February in the last 30 years? Every February 2nd when there had been rain on the previous day (as was the case this time)? Note that if you actually sample from the finite population, you can never assing a non-zero probability to breaking any record, and more generally you can never assign a non-zero probability to any state you have not already observed. So you have to make some subjective (ugh) decisions about how to define a sensible continuous prior distribution (is it Gaussian? Weibull? Spline through the histogram of data?) and you have to make subjective (ugh) decisions about which historical data you base your distribution on, and then you perform some hypothetical thought experiment in which days are drawn at random from this subjective (ugh) and hypothetical prior distribution and this is precisely the Bayesian paradigm!

In the above example, the whole concept of probability is only valid (to a frequentist) when the prior distribution is given a priori (groan). For example: "If we sample the Feb daily min temperature from a particular pdf f and then measure this temperature (with error as stated before), then if the temperature measurement is -5C, what is the probability that the sampled temperature is less than -5C?" Any frequentist or Bayesian can answer that question easily and correctly. But I did not define the prior in the prior version (groan again). I just asked for the probability. A Bayesian has to (and is prepared to) choose a reasonable prior, and realises that his answer is not an "objective" statement of the "true" probability but a somewhat subjective one contingent on the prior he chose. A competent frequentist will say the question has no meaning without a clearly-defined prior. A naive frequentist will probably provide an answer (like the original "50%") without even realising that they are implicitly making an assumption of a particular (and likely unrealistic) prior. Which category are you in?

Female on throne could marry foreigner, Hiranuma warns

I shouldn't be too hard on the Japanese for this, because of course every country has its share of reactionary bigots:

"'If Aiko becomes the reigning empress and gets involved with a blue-eyed foreigner while studying abroad and marries him, their child may be the emperor,' Hiranuma told about 40 lawmakers, academics and supporters at a Tokyo hall. 'We should never let that happen.'"

This is nothing more than misogyny dressed up in the apparently more aceptable clothes of racism. It's always been the case that a male Emperor could marry a foreigner, so there seems no obvious reason why this should afect the female Empress question (in fact the current Crown Prince met his wife while "studying abroad" at Oxford). It's doubly bizarre because not only does the Japanese royal family already have Korean blood in it, but there have also been a couple of female Empresses too. Of course all this was back in feudal times, and to some people such ideas are clearly inappropriate in the 21st century...

The Ravage Of The Planet 2006

This is the title of a scientific conference later this year. I'll let the introductory blurb speak for itself:

The state of our planet continues to deteriorate at an alarming rate. We have arrived at a situation where we need to determine urgent solutions before we reach a point of irreversible deterioration.

Our current civilisation has fallen into a self-destructive process by which natural resources are consumed at an increasing rate. This process has now spread across the planet in search of further sources of energy and materials. The aggressiveness of this quest is such that the future of our planet is in the balance. The problem is compounded by the pernicious effect of the resulting pollution.

Further details on this website: The Ravage Of The Planet 2006. I'm especially impressed to note that the (predominantly European) organisers turned down the choice of numerous local ravaged sites like Milton Keynes or Runcorn that would have been inappropriately modern and unnatural, instead going for the hair-shirt option of the "unspoil ecosystem" of Patagonia as their conference venue. No doubt they'll all be paddling there in dugout canoes :-)

Slash and burn

The bad news about the UK's Centre for Ecology and Hydrology got a mention on PM's Question Time today (might not be on "listen again" for a few hours, but as an alternative here's the NERC press release). I'm surprised this story hasn't popped up previously in the blogosphere (where was Stoat?). It concerns an outrageously audacious slash and burn by the new NERC Chief Executive to wipe out up to half of the UK's research capability in this area. Oh, NERC = Natural Environment Research Council, ie the main governmental funding agency in this area with a ~UKP300million annual budget.

CEH is one of these "virtual centres" that have been increasingly fashionable in the UK in recent years. It doesn't exist as a physical entity, but is an administrative and scientific concept currently linking 9 sites across the UK. But no more, as 4 of them are planned for closure (and the admin department, which was separately located, is moving too). About 1/3 of the 600 staff are likely to lose their jobs. That's a significant chunk of the UK's environmental science community.

According to the newspaper reports, this UKP45million plan is expected to save UKP1million a year. It's clearly not an economically-justifiable decision, but instead forms part of NERC's long-term strategy to divest itself of any in-house capability for research. It's no secret that NERC intends to become a purely administrative funding body that contracts out its research to universities. That's more "responsive", because they don't have any long-term liabilities such as tenured staff or lab facilities to worry about. Of course, it also means that national research capabilities are increasingly fragile and disintegrated, and it pretty much kills morale and productivity for however long the process takes.

(Disclosure: I'm a former employee of the NERC POL lab, where similar - athough less extreme - events played out several years ago.)