Wednesday, April 08, 2020

Dumb and dumber

All these people exhorting amateurs to "stay in their lane" and not muddy the waters by providing analyses and articles about the COVID-19 pandemic would have an easier job of it if it wasn't for the supposed experts churning out dross on an industrial scale.

Two new forecasts for the UK caught my eye yesterday. First from IHME, it predicts a rising death toll to a daily value of over 200 by the end of the week, peaking at about 3000 later in the month (with much uncertainty). It seems to be some sort of curve-fitting exercise. They are regarded as "world-leading disease data experts" by the Guardian at least. Here is their forecast for the UK as reported in the Guardian:

and again from their own report with uncertainties (lines are at 1k intervals). So you can see they expect a strong rise and guarantee that it will go over 1k deaths later in the month.

The article describing their method is here, it's some sort of fancy curve fitting that doesn't seem to make much use of what is known about disease dynamics. I may be misrepresenting them somewhat but we'll see below what a simple disease model predicts.

Next, the famous IC group. And these are the real ones with Ferguson as last author, not the jokers who generated the silly 5700 number that got a brief bit of attention. This is the MRC-funded, WHO-collaborating lot. And what have they done here? They've also basically fitted a curve through historical death data,  and extrapolated it into the future. Their prediction is for a whopping 14k deaths with a range of 8-16k - just in the next week! A minimum of 1500 on the 7th day in that forecast too.

So why do I think these are bonkers? Because we KNOW that a lot changed two weeks ago. And we KNOW that this feeds though into deaths in about 3 weeks. Typical time from infection to death may be a little over 3 weeks but a bunch of people die significantly quicker and this means the effect of the social distancing controls should be feeding though into death rate, oh, about now.

This below is what I get from fitting my simple SEIR model to the available death data, backdated to Sunday to form a fair comparison with the IC group. 

Now, before looking at it too carefully, I do need to give a "health warning" on its validity. In particular, I don't expect the uncertainty range to be very well calibrated. I'm really focussed mainly on the best fit, and the parametric uncertainty included here is a rather subjective component. I suspect the uncertainty intervals to be rather too broad, especially at the top end (which essentially allows for continued strong growth of the epidemic, contrary to what has been seen in other countries that clamped down sooner). The corollary being that I'm pretty confident of reality lying inside the bounds, and optimistic of my median being a much better estimate of the truth than the central forecasts provided by the experts above :-)

The forecast is necessarily vague at this point, because although I believe R changed at the breakpoint, I don't yet know what it changed to. "About 1" is a plausible estimate, it might be higher but we can certainly hope that it's lower as it proved to be in Wuhan and also Italy after they clamped down.

My model predicts a total of 8k deaths next week, with a 5-95% range of 4-19k. Yes it's a wide uncertainty range, I think my prior on Rt is probably still too broad as I don't really expect to see a value lower than 0.5 or higher than 1.5 (and these are just the 1sd spread limits in the above). But I am very optimistic that the median estimate generated by this method is better than the experts have provided, and they don't seem to believe that anything in the lower half of my range is possible at all. Ominously for them, the recent data are already consistently lower than the median, albeit marginally.

I'll come back to these in a week to see how we fared.


Ben McMillan said...

Probably R about 1 after lockdown is even rather pessimistic, given decreasing death rates in places that have implemented lockdowns (and that the fit looks better with a lower R).

Will be interesting to see what happens in places like Australia, where case counts are getting low enough that eradication might be possible.

On the slow UK response: one gets the impression that once the 'government experts' had decided in January that it was probably not a big deal, they had effectively locked-in that view, and it took extraordinarily public evidence to shift them into a response. There was never a 'we need to commit resources and prepare for the tail-risk right now'. Committees aren't much fun for the Cassandras of the world or those who want to stick out.

James Annan said...

Yes I agree about R, but given the simplicity of the model and uncertainty about time to death, I can't really claim to be sure from that based on the available data. It doesn't make a big difference to the central forecast over the next week if R is 0.8 or even 0.6 but the top end of my above forecast does look excessive to me.

There was certainly a "take it on the chin" approach baked in to the flu pandemic plan. And when all is said and done, we might end up there anyway - there is no clear exit plan and we might never find one. But even so, the curve needed flattening. It wasn't even a tail risk, it was the central estimate based on available data by early Feb!

James Annan said...

Just ran it for fun: setting Rt = 0.6±0.1 we get a median of 6.5k and top end of 10k. We'd expect to see the peak with the week but the actual numbers aren't so different apart from ruling out the high end.

PeteB said...

I think the reporting delays are significant

it's not just time to die - it is time for death to be reported

James Annan said...

That's important, but under-reporting *in a consistent manner* (big assumption, I know) doesn't affect estimates of growth *rate*, thanks to the magic of exponentials. It's the same reason why we can use deaths to indicate growth rate of infections in the first place. Of course the delays matter hugely when you are looking at changes in the rate. And changes in reporting methods matter hugely too (eg testing all suspects and then moving to testing only the seriously ill leads to a big bias if uncorrected).

Bryan Lawrence said...

Why did you post your 14 day prediction when you had a 16 day available? is that for better comparison? In any case, I think it would be good if your "forecasts" had a "using data available on X-Y-Z", since no one really can remember what x days since lockdown means, and the data is being updated anyway.

James Annan said...

Yes I didn't want to cheat by using more days than the IC group did. Good suggestion about the label/title. Processing dates in R is a bit of a hassle but I ought to make myself....

Jim Hunt said...

Pete - It is certainly the case that "it is time for death to be reported"

James - I am taking your name in vain over at:

I hope that's OK with you? In particular please let me know at your earliest convenience if you'd prefer to be labelled as something other than a "climate modeller"!

P.S. Is there any way to edit a typo on here. The only option seems to be to delete the offending comment and start again?

James Annan said...

No, I think delete and start again is the only possibility. BTW I slightly improved my forecasting methodology to get a better initialisation from "today" (just by weighting more towards recent data than earlier data, a basic heuristic rather than any fancy theory). It cuts a little off the ends but not much different overall.

Yes I guess climate modeller though probably doing more epidemiology in the last couple of weeks :-) My background includes a rather broad range of modelling actually, I even briefly did a very little epidemiology some decades ago (fungal disease in crops).

Improved forecast..

Jim Hunt said...

Thanks James,

I have duly amended your "label".

I'll just have to get used to clicking the Blogger "Preview" button!

James Annan said...

well I failed to give a proper link so here is the new forecast in cut and paste style...

Jim Hunt said...

Thanks James,

Duly reported at my end.

David Young said...

There is no doubt that the eeeevil skeptics like Ioannidis have been right about the science elite’s total screwup. In fairness, China is a police state where virtually everything in the ‘media’ is a party line lie. Plenty of blame to go around. Why is it that for some people every discussion has to devolve into a focus on their Favorite witches who must be constantly tarred and feathered? Epidemiology has had a woke hangover ever since the AIDS lie that “everyone is at risk.”

Jim Hunt said...

I'll just have to get used to keeping a close eye on your Twitter feed!

FYI - COVID-19 in South West England:

My helpful advice to those considering taking a traditional holiday in South West England over the Easter break?

Stay the f*ck at home!

winston said...

Careful now. You're in danger of sounding like Steve McIntyre. All of these fools with their mad models predicting the wrong numbers, but here's a short R program I knocked up in the shower. Not saying you're wrong, but imagine how the IC guys might feel when hearing this sort of thing no doubt multiplied ad nauseam.