I'm sure many of you spend your mornings glued to Parliament TV but for anyone who missed it, there was a House of Lords Science and Technology Select Committee hearing on Tuesday, which discussed the science of Covid-19 with a particular focus on modelling. There are transcripts (two parts) here. and there is a video of the whole event here.
Two things particularly caught my attention. First bit is Question 31 in the transcript about 10:47ish when they are asked about the 20,000 death estimate that Vallance had hoped to undershoot. Matt Keeling first excused this by saying that the lockdown could have been stricter. However, Ferguson had already confirmed that the lockdown was in fact adhered to more strictly than the modellers had anticipated. (Keeling also went on to talk about the failure to protect care homes which is probably fair enough though the 20,000 target would have been badly overshot in any case). It would have surely been appropriate at this point to mention that the 20k target was already missed at the point that the lockdown was imposed on the 23rd March. By this point we had well over a million infected people, which already guaranteed well over 10k deaths, and there was no plausible way of stopping the outbreak dead in its tracks at that point. The decay was always going to be slower than the growth, meaning more cases and deaths.
Then in the follow-up question, the witnesses were asked what had most surprised them compared to their earlier predictions. Ferguson answered with the biggest factor being that more infection had come from Spain and Italy and this is why they were further ahead on the curve than anticipated! The doubling time has nothing at all to do with how quickly people came with the infection, of course, and getting this wrong made a far greater difference to how things turned out.
It's curious that in two hours of discussion about the modelling, it never once came up that the modellers' estimate of the doubling time was 5-7 days up to the 18th march, and abruptly changed to 3-5 days on the 20th (reaching SAGE on the 23rd, according to the documentation)
15 comments:
Hence why Fergusson's model abruptly increased R0 from 2.4 to 3.0 The code as released on GitHUB uses R0=3.0 (620,000 unmitigated deaths) while their March 16 paper used 2.4 (504,000 unmitigated deaths. A couple of days ago he released the original code. It is a nightmare to understand !
I took a policy decision not to look at their model...I know it would have been a huge time sink and I don't have easy access to big computers any more. Would be fun to do the calibration that they didn't bother with though. What sort of throughput do you get for simulations - say running just up to the end of March?
Well. A couple of things. John Carmack, who is way smarter than me, indeed a genius in his field, did look at the code and thought it was surprisingly good from a software engineering perspective, while being careful to point out that he, as a non-expert, couldn’t really cooment on the underlying model. Which doesn’t stop various random lockdown sceptics and TDD voodoo aficionados whining about code quality.
And. Constant doubling time implies a continuous process and sufficiently large numbers? Were those evident back then, or was everyone just doing their best with limited data?
But TBF some of the stuff from epidemiologists is starting to sound like some of the stuff Mann used to say to justify some of his more questionable numerical procedures. And some of the criticism from the other side is starting to sound like the comments on gistemp.
That was more than a couple.
James - Ken Rice had a good look at the IC model, and seems to have a plentiful supply of CPU cycles close at hand!
https://andthentheresphysics.wordpress.com/2020/05/16/the-imperial-college-code/
James,
How would you do the calibration with their model?
It's only really R0 that needs to vary (and that can be constrained), so I would just run a range of values and weight according to likelihood. Using case numbers in the early days for the UK, similarly to my medrxiv manuscript. The start size/time would also have to vary with R, it sounds like this is done quasi-automatically which might complicate (or simplify!) things.
It could be as simple as doing 10 runs or so....though probably more like 100 in reality...but only short runs.
Thanks for writing this, I missed the hearing.
Regarding the doubling time, in his March 13th interview Edmunds takes issue with doubling time numbers of 2.5 days cited by blogger-guy Tomas Pueyo, saying the following:
“it’s true that if you look crudely at the numbers, then the cases are doubling every 2.5 days, but that’s because they are doing more contact tracing. The actual underlying rate of doubling is actually about every 5 days” https://youtu.be/C98FmoZVbjs?t=1399
I am not really sure what he means by 'underlying rate', but maybe this provides some clue as to why SAGE persisted with their estimate.
Thanks for the link! That's pretty amazing that they were so much in love with their models that they couldn't accept the evidence in front of their eyes even though it was happening in multiple countries. By underlying, he just meant the true number of infections as opposed to the diagnosed/reported number. And he was catastrophically wrong.
The 5 day doubling stuff only ever came from some limited analyses of early data in Wuhan, I am baffled as to why it became such a fixed point in their beliefs.
Not so much reifying as deifying their models.
Thanks. Right I see, he is implying there was a larger number of undiagnosed cases, and if they are taken into account then the doubling time in the whole population is longer. Presumably there would be some way of estimating this from the differential in testing rate, but seems like there would be an awful lot of uncertainty involved in that.
Interesting last point. As you may know, Mackenzie proposed a 'trough of uncertainty' regarding the extent to which various actors perceive uncertainty in the models. In theory, the modellers should be extremely aware of the uncertainties in their own models. However, I wonder that as modellers become more institutionalised/involved with science advice, they drift rightwards into a reduced perception of uncertainty? Link to Mackenzie diagram: https://proxy.eplanete.net/galleries/broceliande7/uncertainty-social-construct-certainty-trough
I still don't know what to make of it all. Even at that time the LHSTM lot were using a wide range of R values which actually did reach to high enough levels at the top end. And yet there is one of them saying confidently that they don't think those numbers were plausible, despite the data agreeing with it. Makes no sense to me.
James,
Running a suite of simulations with a range of R is easy enough. Changing the start time is less obvious (or, I haven't worked out how to do this yet). I presume that you also only run them up to the point at which interventions start.
Yes, in the new Lancet PH paper they say they estimated R0 to be 2.7 but the 95% conf interval is 1.6-3.9. Of course, we would expect a lot of caveats to get washed away in day-to-day scicomm, but for one of the researchers involved to focus down on a single number is...surprising.
ATTP, If I was wanting to show how they could have calibrated their model for the mid-march paper, I'd only do it up to that date...
An alternative to changing start would be to change the target they have from a single number to some spread. So long as it covers a big enough range (so that this doesn't cut out a region of the high-likelihood space) it should work ok. Might have to work out what this means as a prior and adjust the calculation accordingly, but nothing too complicated. May be easier than trying to hack the code.
If it all scales linearly (which it probably does, near enough) then just varying R would be enough.
WP: if there are far more actual cases than tests, then the growth rate in confirmed cases is mostly a function of the grow rate of testing, not of the growth rate of cases. This happened first in Wuhan. At the beginning of the epidemic testing was very limited, and only those admitted to hospital with Covid-like symptoms were tested. As testing ramped up, so did the confirmed cases, even while the actual cases were likely falling due to the lockdown.
I don't have data to confirm that was the case in the UK during any specific time period.
Post a Comment