Sunday, February 19, 2006

73,000,000:1

This is the famously erroneous statistic quoted by Roy Meadow, supposedly giving the odds of two children dying of SIDS (cot death) in the same family, when appearing as an "expert witness" in a murder trial. His "evidence" neatly illustrates two distinct fallacies in the same calculation, and as well as resulting in some wrongful convictions which were eventually overturned, he was struck off by the General Medical Council, but has just been reinstated following an appeal.

Firstly the fallacies: he got his number by squaring the odds of a single child dying in this way, of about 8,500:1. The implicit assumption is that such deaths strike entirely at random. However, if there are any genetic or environmental factors involved, a the probability second death in the same family would be far greater. Ignoring this factor is known as the ecological fallacy. In fact medical research suggests a second death happens in about 1 out of 100 cases where a first death has happened.

A second and more pernicious fallacy is the prosecutor's fallacy, so called because it seems pretty near ubiquitous in the presentation of statistical evidence (eg DNA tests). It is the incorrect interpretation of a probability of an event happening to an innocent person, as implying the probability that this person is in fact guilty.

An example: assume that you are a juror, and you've been told that DNA testing provides a match to the suspect at the 1,000,000:1 level (ie, a random person will match with probability 1 in 1,000,000). What is the probability that this person committed the crime? 999,999/1,000,000? No! Given a population of 60,000,000 in the UK, there will be about 60 people whose DNA matches that well. A priori, any match only indicates a 1 in 60 chance that a particular person did it - if there's no other evidence, this is all but worthless. However, it would be a rare juror who would understand this (and probably a rare lawyer). In practice, matches are often quoted at a much higher level of significance, but by the time we are up to billions to one, the chances of accidental contamination or fraud must be higher than that anyway.

Back to Meadow. He was reinstated recently, with the judge ruling that expert opinion given in court should be "priviledged" in the sense that incompetence is not grounds for the GMC to punish the witness. While that may seem rather bizzarre, I note that no-one has called for the defence lawyers to be themselves disbarred, for failing to produce any expert witness who understood elementary statistics well enough to destroy (that aspect of) Meadow's evidence. That is surely incompetence on a similar level to Meadow's.

10 comments:

William M. Connolley said...

I'm with the judge, in two ways: firstly, Meadows incompetence was (a) with stats and (b) pretty common, and doesn't have much to do with his paediatrics. And secondly, the argument that witnesses should be allowed to speak their mind is very strong.

As you say, the real incompetents here are the defence lawyers: if (as is now claimed) Meadows mistake is so egregious, how come no-one noticed it?).

James Annan said...

OTOH, it is their "expert" status that makes "expert" witnesses useful, and if their regulatory body is forbidden by law from removing their accreditation when they make such horrible errors, how can their expertise be meaningfully certified?

The most worrying thing is that even now Meadow apparently doesn't understand or accept that he is wrong. It's entirely to do with paediatrics since it is specifically this argument-by-statistics that leads him to assert that abuse/murder has happened in the first place.

As I recall, the erroneous statistic was widely criticised even during the trial, with the RSS writing an open letter to the Lord Chancellor. It certainly stuck out like a sore thumb to me!

Anonymous said...

I don't see why you couldn't write a letter suggesting that the defence team's incompetence with statistics is sufficiently bad that it brings the legal profession into disrepute and they should take action.

I imagine that if you did write such a letter it would have to receive serious consideration.

Question is; would you be comfortable doing that knowing it could wreck a professional person’s career when their expertise is in the law not in statistics?

Incidentally, I think accountants suffer much greater risk from deep pocket syndrome. This is more the equivalent of making partners of the defence lawyers bankrupt even though they had nothing to do with the case concerned. Does that make you more comfortable with writing such a letter?

crandles

James Annan said...

It's not that they should be expert, but that they should have hired one and put up a decent defence. IMO their client would have strong grounds for complaint. I see that the Guardian published a broad criticism of the whole thing last year. As far as the recent legal decision goes, it's not a matter of what I or anyone else thinks, but whether regulatory bodies such as the GMC should be able in principle to form these sort of judgements based on the behaviour of experts in court. Whatever the rights and wrongs of this particular case, it seems absurd that they should be prohibited from doing so.

William M. Connolley said...

I don't really buy this stats-is-to-do-with-paed stuff. Paed is doctoring, bodies, and squishy stuff. I'm sure you could make a career tending off to the stats side, but I don't think thats what Meadows was doing. AFAIK he was testifying mainly about the medical evidence (if his testimony and expertise was mainly stats, then I'm wrong).

But in fact the judge, indeed, seems to have gone somewhat further and said that even if it was their mainstream expertise, they shouldn't be de-accredited for it. I still think even that is plausible. Firstly because if not, it makes testifying too risky. Secondly, because if you deserve to be de-accredited, their really ought to be evidence from your actual *work*, not from your non-work activities. If you make some duff blog posts, would you accept jamstec de-accrediting you for it?

Anonymous said...

>As far as the recent legal decision goes, it's not a matter of what I or anyone else thinks, but whether regulatory bodies such as the GMC should be able in principle to form these sort of judgements based on the behaviour of experts in court.

I think regulatory bodies such as the GMC should be able, in principle, to form these sorts of judgements.

I think you are brushing aside what I asked. Yes it is (at least in part) a matter of what you and others think - if the legal profession's discipinary scheme does not get a complaint from anyone they can't be expected to take action. If the client is relying on expert advice from their solicitors and the solicitors are pointing the finger of blame at Meadows and not themselves/barrister, the client is being let down again. Does the solicitor feel sympathy for the barrister and find it easier to point the finger at Meadows?

Unless you put yourself in the position of asking yourself whether you would feel comfortable writing a letter that could distroy someones career when it is not their area of expertise then I don't think you are fully appreciating the situation.

crandles

James Annan said...

I'm not sure I know how to define what level of competence should be required, but the GMC obviously felt that Meadow fell below their standards. OTOH there is apparently plenty of evidence that many doctors misinterpret medical test results along the lines of the prosecutor's fallacy, so maybe it's not fair of them to be so harsh on him in particular. I don't think it is unreasonable to wonder if the defence is also at fault - after all, you expect the prosecution to make the most of their case, and it's precisely the defence's job to pick holes in it. While you are feeling sorry for the lawyers, how about the people who spent time in prison as a result of wrongful convictions?

Anonymous said...

Ok I'll accept that it was remiss of me not to express sorry for people wrongly imprisoned. I do feel very sorry for them.

However I am not going to let that sway me into recommending punishments that are too harsh for the circumstances.

>While you are feeling sorry for the lawyers

I don't think I am feeling sorry for the lawyers. I think that between us we are making a powerful case for the lawyers to face some sort of disciplinary action on the grounds that justice should be seen to be done. Whether the layers concerned should be found guilty or not, I am not sure or qualified to say. I wouldn't be suprised if it required gross negligence rather than just negligence for such a complaint to succeed. If found guilty of negligence and not gross negligence, I don't think the punishment should be too severe.

This might seem little comfort to those wrongly imprisoned to let professionals get away with some sort of slap on the wrist, but they are entitled to justice but not revenge.

Meadows has faced some sort of disciplinary hearing and I doubt he could make a career of being an expert witness even if he wanted to. Should we also throw away his pediatric skills - I don't think so.

crandles

Anonymous said...

Well ..., how would you argue for the "DNA case" in a court say in Luxembourg (population 0.5 mln) ?

James Annan said...

I guess it doesn't depend directly on the population of the country, but rather the number of people who could plausibly have been in contact with the scene. But in any case, I would expect that the probability of accidental or deliberate contamination in the testing would generally be larger than the possibility of a false positive. In a case where the only evidence is from DNA, it seems to me that it would be very easy for the police to fit up a suitable person, who they may even genuinely think did it.