Imperial College model used to justify UK and U.S. lockdowns deemed ‘buggy mess’ & ‘total unreliable’ by experts
One expert’s damning assessment: “In our commercial reality, we would fire anyone for developing code like this and any business that relied on it to produce software for sale would likely go bust.”
Last week our update included the report that Dr. Neil Ferguson, who developed the Imperial College Model predicting the spread of the Wuhan coronavirus, had resigned from his government position.
The reason for his departure was thought to be that he was discovered to be violating quarantine orders to see his mistress. However, it turns out there may be more motivation than a romantic affair.
The model United Kingdom experts, as well as many others around the world, have largely used to guide their coronavirus policies has been deemed “totally unreliable” by experts. To start with, the Daily Telegraph‘s report on the assessments done by technology professionals is damning.
The model, credited with forcing the Government to make a U-turn and introduce a nationwide lockdown, is a “buggy mess that looks more like a bowl of angel hair pasta than a finely tuned piece of programming”, says David Richards, co-founder of British data technology company WANdisco.
“In our commercial reality, we would fire anyone for developing code like this and any business that relied on it to produce software for sale would likely go bust.”
…The Imperial model works by using code to simulate transport links, population size, social networks and healthcare provisions to predict how coronavirus would spread. However, questions have since emerged over whether the model is accurate, after researchers released the code behind it, which in its original form was “thousands of lines” developed over more than 13 years.
In its initial form, developers claimed the code had been unreadable, with some parts looking “like they were machine translated from Fortran”, an old coding language, according to John Carmack, an American developer, who helped clean up the code before it was published online. Yet, the problems appear to go much deeper than messy coding.
Other scientists have also found troubling problems with the model as well:
Scientists from the University of Edinburgh have further claimed that it is impossible to reproduce the same results from the same data using the model. The team got different results when they used different machines, and even different results from the same machines.
“There appears to be a bug in either the creation or re-use of the network file. If we attempt two completely identical runs, only varying in that the second should use the network file produced by the first, the results are quite different,” the Edinburgh researchers wrote on the Github file.
A fix was provided, but it was the first of many bugs found within the program.
“Models must be capable of passing the basic scientific test of producing the same results given the same initial set of parameters…otherwise, there is simply no way of knowing whether they will be reliable,” said Michael Bonsall, Professor of Mathematical Biology at Oxford University.
It is hard to overstate how important the Imperial College model was in altering the approach to the coronavirus being taken by both the U.S. and Britain at that time.
In fact, just prior to the model’s pronouncement that 2.2 million people would die from coronavirus with no mitigation in this country and 1.1 million with mitigation, the White House was using the protective approaches applicable to a severe flu strain.
As a reminder, it is an approach I highlighted in early March. If we had continued with the “severe flu model,” including the decontamination of commonly touched surfaces (like those in subways) and promoting vitamins (like Vitamin D, which seems to be protective against coronavirus), it appears the severe flu model would have offered all the protections without the economy-crushing side effects.
Perhaps the best fisking of the code comes from viriologist and computational epidemiologist Chris von Csefalvay, who has issues with the use of 13-year old code for such an critical project. Specifically, von Csefalvay called the use of the code “somewhere between negligence and unintentional but grave scientific misconduct.”
First of all, the elephant in the room: code quality. It is very difficult to look at the Ferguson code with any understanding of software engineering and conclude that this is good, or even tolerable. Neil Ferguson himself attempts a very thin apologia for this:
I’m conscious that lots of people would like to see and run the pandemic simulation code we are using to model control measures against COVID-19. To explain the background – I wrote the code (thousands of lines of undocumented C) 13+ years ago to model flu pandemics…
— neil_ferguson (@neil_ferguson) March 22, 2020
That, sir, is not a feature. It’s not even a bug. It’s somewhere between negligence and unintentional but grave scientific misconduct.
For those who are not in the computational fields: “my code is too complicated for you to get it” is not an acceptable excuse. It is the duty of everyone who releases code to document it – within the codebase or outside (or a combination of the two). Greater minds than Neil Ferguson (with all due respect) have a tough enough time navigating a large code base, and especially where you have collaborators, it is not unusual to need a second or two to remember what a particular function is doing or what the arguments should be like.
Or, to put it more bluntly: for thirteen years, taxpayer funding from the MRC went to Ferguson and his team, and all it produced was code that violated one of the most fundamental precepts of good software development – intelligibility.
The entire post by von Csefalvay is worthy reading. His conclusion about the public’s view of epidemiology in the wake of this disastrous miscalculation is spot on.
There will no doubt be public health consequences to the loss of credibility the entire profession has suffered, and in the end, it’s all due to the outdated ‘proprietary’ attitudes and the airs of superiority by a few insulated scientists who, somehow, somewhere, left the track of serving public health and humanity for the glittering prizes offered elsewhere. With their abandonment of the high road, our entire profession’s claim to the public trust might well be forfeited – in a sad twist of irony, at a time that could well have been the Finest Hour of computational epidemiology.
Based on these new determinations, it seems clear that the model we should be relying on is common sense: Practice good hygiene and open up the economy.
DONATE
Donations tax deductible
to the full extent allowed by law.
Comments
So GET OUT OF THE DAMNED WAY SO WE CAN OPEN THE ECONOMY AGAIN, ABBOTT!!!!
But it got that dopey dude laid, so it was worth it.
In his defense, his field is math/programming. He is not an epidemiologist, infection control specialist or even an family MD. WHY did Governors in the US listen to this guy? Being incompetent is on Ferg, looking to one, single dipwit fo make policy is on the execs who did that.
The whole point of being a CEO or executive branch official is that you have and use judgement, seeing through the fog of multiple, often conflicting demands and advisors to make a robust decision.
The lady in Idaho did exactly that, as did Eisenhower. A CEO of any medium size co that lasts over 10 years has probably done that, else their biz would have been run over by the competition, regulators, lawyers, etc. Every ceo/gov/mayor has people that was xzher to let them be surrogate leaders. It’s up to the top name to get a 2nd and 3rd opinion and then use their own judgement on big decisions.
Imperial College is soooo elite and a fountain of knowledge… until now.
In complete fairness, his code is junk, as code.
How is this not on the CEO of our country – Trump? IMO he has repeated made poor personnel choices. Although it looks like he was “had” this time, perhaps if he had surrounded himself with better people the country would not have been left with something that will take years, if ever, to recover from. For example, I do not think any of us will live to see 3.5% unemployment again. There are plenty of studies that show that increased unemployment and poverty hurt morbidity and mortality.
By the way, why is no one making a very big noise about liability issues, for businesses, K-12 schools, colleges, churches, restaurants, sports arenas, etc. Is it really not a serious issue on reopening anything?
It was a deliberate trap aided and abetted by the communist Chinese also.
Coupe V.nth degree….
Look at all the DEMS lies inflating the deaths by thousands fold….
How is this not on Trump? Because he is not an epidemiologist or programmer. He is a businessman trying to enrich the country economically. He depends upon “experts” in those fields to advise him on matters outside of HIS area of expertise. And, those “experts” engaged in some incredibly bizarre behavior.
In the beginning, the experts, including Fauci, were treating COVID exactly as what it was, a novel flu type disease which produced out-of-character, negative results in a small target population. But, through media action and for some unknown other pressure, suddenly the “expert” did a 180 and began treating this as an apocalyptic disease which was going to destroy the world. And, again, for unknown reasons, they began prescribing total societal sequestration as the ONLY cure. And, they are still doing so.
Trump did the only thing he could do in an election year in an anti-Trump world, he punted. He gave the responsibility for the response to state and local leaders, saying that they knew their areas and problems better and could direct services more reliably. He threw the full power of the federal government into giving them everything that they asked for to fight the virus. He knew, or suspected, that this was all BS, so he distanced himself from the overzealous actions that the media and others wanted to pursue. And as the dust settles, he is STILL being blamed for the actions of state and local politicians. He would be blamed no matter what he did or did not do. But, in this case, many of these local officials are not going to be able to dodge responsibility for the results of their actions.
Look at the trend line for the US, that had a lock-down, and Sweden, that didn’t do anything as radical.
Very similar.
The ‘swamp’ is an international Mafia: we citizens of the US are the victims, China is the Godfather, and pelosi on down are the ‘made members’.
It’s truly amazing.
Bet that isn’t the only Wuhan virus modelling software that’s a buggy unreliable mess.
Bet Ferguson’s code is also undocumented. Which would mean no one but Ferguson knows how it’s supposed to work. Assuming he even knows.
The entire covid response has, overall, been a fiasco.
IMO, there are several reasons:
1. The PRC refusing to accurately and immediately report to the other Nation’s pubic health agencies the severity of the situation. The PRC continues to fail in this.
2. Faulty data sets based upon incomplete/false information being fed into predictive models.
3. The reliance upon the predictive models after the point at which it became clear that the facts on the ground didn’t match the prediction by orders of magnitude.
4. Our collective willingness to put our faith in public health professionals, both members of the general public and those with a medical/research background with a different take or hypothesis than the consensus view point.
5. Government reliance upon the consensus view, again after it was exposed to be flawed.
6. The MSM pressure on government to adhere to the flawed models or be accused of killing people.
7. The willingness of the American People to voluntarily comply with this fiasco even when they become aware the policies being put in place were not going to stop covid.
8. The willingness of some LEO and DA to enforce the lockdown, sometimes enthusiastically and in some cases where they had to know the enforcement was not constitutional.
There are others but these are the main issues.
Fortunately or unfortunately we are all now in this fiasco and everyone has had the opportunity to observe all this and draw their own conclusions. Personally, my willingness to voluntarily comply with, what I consider to be unconstitutional actions, is done. As is my willingness to defer to the opinions of the ‘experts’ or the ‘authorities’.
I feel certain that many others have come to a similar point.
As one example of the lasting impact. Those of us who, in the past reflexively supported LEO and gave them the benefit of the doubt, are no longer willing to do so. I have come to the conclusion that, based upon the willingness of some individual LEO and some departments to support clearly unconstitutional actions, every department should have a civilian review board looking over it’s shoulder.
A board that must sign off on departmental policy and procedures. A board that creates and enforces some sort of LEO equivalent to the UCMJ, with financial penalties, career ending penalties, non-judicial punishment and a process to make criminal referrals to the State or Federal government.
No more deference to any sort of ‘expert’ without concrete data being provided, to include the basis of the subjective parameters, the source code and the proven ability to replicate the results.
Lock me down once… shame on them… lock me down twice..shame on me. I have talked to people and they will NOT do this again.
This has turned out to be the most expensive and civilization obliterating self-inflicted experiment in the history of mankind.
Oh Mann,did they just adapt global warming.
(speaking of which, here out west the mountain pass roads are still closed and we are still getting snow!)
Indeed! https://nypost.com/2020/05/14/the-sun-has-entered-a-lockdown-period-which-could-cause-freezing-weather-famine/
It has always been primarily the sun, except when you get a Krakatoa or meteor impact every once in a while. Hopefully we can save some big bucks on this to help cover some of the damage from the Covid shutdown. Poor Greta Thunberg.
Are we making snap judgments? Please, may I?
He looks like one of those political-engineering parasites. The stuff they build doesn’t work. He’s the guy who learned in college how to screw up a team project n take all the credit — that’s his only competence.
On a totally unrelated note — no idea why this comes to mind — what’s it been like trying to get a look at the (in)famous “Global Warming” models; the actual code. (Not the various measurement data sets that you can’t see.) BTW, you will get sued if you poke at Mann & company about either. Ask Mark Setyn.
“Thousands of lines of code” isn’t all that much. LibreOffice’s office suite has about 12.5 million lines of code.
The C language is a poor choice for modeling. Perl or Python are easier to understand, would require less lines of code, and make it easier to avoid programming errors.
When attempting to do analysis, it is best to use at least two entirely different algorithms. If they get the same result, you have much more confidence in it. A computer will always give you numbers. It’s your job to determine whether or not they have any meaning.
As the article says, if a model cannot predict the present, it may not be entirely worthless, but it is quite suspect.
Scott Adams has some helpful comments about models.
You are right in saying Python would be a better choice, but it is easier to document with C because it has a more robust data structure that allows the data to be understood regardless of the complexity. That is difficult to do in Python when the project gets to several thousand lines.
Python would be much faster to program.
From the sound of the program, it seems that the optimum language for it is Mathlab, and the whole project looks like it should take less than week.
In addition, as the data is collected, the model should be adjusted to fit the data. There was never any data in the last 8 months that suggests that millions of people were going to die. This model doesn’t pass a sanity check.
Those governors who shut their states down based on this software should be immediately replaced with someone who has some idea of how to run something.
Kidding / not-kidding aside…
I’ve worked with building high-stakes numberical models n modeling systems in multiple domains.
Doing one of these models takes great skill in three domains: the application, the numerical wrangling, n the engineering of software.(*) At a glance it’s obvious that epidemic modeling failed at all three.
The other distinction is softare run to support discovering a new clue, vs. software built for the modeling results to be consumed, ongoing.
A colleague wrote a book — on Springer, AIR — about how “research support” software is different. Net, it only has to run once, for someone who knows all it’s idiosyncracies. Software for use by others has to run repeatedly, reliably, with results that can be naively trusted by people who don’t understand the internal hacks. Software supporting research is a hot rod, that only has to hang together for the one quarter-mile run. Software for use is more like a Civic: robust, reliable, indifferent to surroundings.
The last thing you want to do is let the research idea guy represent what the modeling SW is good for. Researchers are good at making the case for the new thing they’re claiming. Great at “defending”; not at being generally right, not at helping you use what they’ve uncovered.
(*) For example, making pseudo-random number generators that are actually “random” the way you need is a hard problem … starting with you don’t need them to be “random”; you the distribution of the values they return to be right, with enough precision.
Sorry for the bad copy.
Post-stroke — six years later this month — I sometimes don’t see that kind of error, still.
So many apologies… (2 of 2):
I didn’t mean to imply above that I’m expert in the three domains it takes to make any robust simulation systems. Even post-broken brain, I know you need all three, can name them, can suss out whether they’re working, and help get what’s missing in place.
It’s weird what comes back. I can’t write a loop again … yet.
Even the septics can’t get it right.
“Models must be capable of passing the basic scientific test of producing the same results given the same initial set of parameters…otherwise, there is simply no way of knowing whether they will be reliable,” said Michael Bonsall, Professor of Mathematical Biology at Oxford University.
A model which makes bogus predictions isn’t reliable, it’s useless, even if it gives the same bogus predictions every time. Unless that’s what you define as “reliable”—we can rely on it to be garbage.
Hear. Hear.
Those models are just Voodoo and those cases.
“in”
From what I’ve read, it wasn’t just bogus predictions – it was also inconsistent predictions: they couldn’t get similar results out of the software in successive runs from the same parameters.
So it wasn’t actually modelling anything: it was just spewing out white noise.
Fyi
Horowitz:
We now know that the majority who contract COVID-19 are asymptomatic. That changes everything
Daniel Horowitz · May 12, 2020
Finally, it’s important to remember that now that it’s clear the majority of those who contract the virus do not develop pronounced symptoms, we must revisit the death count itself. We now know that even seniors, while more vulnerable to fatal cases that the rest of the population, still often develop asymptomatic versions of the virus. Coding every death, especially among nursing home residents, as a COVID-19 deaths so long as they test positive – even if they don’t exhibit those particular symptoms – is indefensible.
Given how much we’ve learned about the virus, why does the governing class continue to resist any adjustment to its strategy?
Your question – TDS and personal/professional embarrassment. This may be Phase III of the effort to get rid of Trump and it may work. His best hope may be the Dems apparently overplaying their hands in many states, in denying people their constitutional rights and destroying their livelihoods. I also suspect that how people respond to a poll and actually voting for someone who looks to me like he can’t be let outdoors without a minder may be different.
It is also the case that if they are asymptomatic, then they are not sick. The antibodies usually are developed by exposure, alone. When enough people have antibodies (in this case about 50%), the virus dies.
The problem lies with the fact that these term are never defined by the people who use them.
In the case of a pandemic, the definition is any disease which is found to be present, to a significant degree, in more than one country or area. Under the definition of pandemic, seasonal flu is a pandemic.
In the case of asymptomatic, all this means is that a person is infected by a disease organism, but that infection does not produce any overt symptoms. People are infected, by disease organisms constantly. That is why we have an immune system. Spend time around a child with measles, and you will be infected with the measles virus. The same is true of seasonal flu. However, you do not display symptoms, become ill, because you have anti-bodies which destroy the virus before it causes enough damage to produce symptoms. So, being infected but asymptomatic is a normal state of affairs for human beings. Where this become of concern, is when the person is asymptomatic but is still capable of spreading the disease. So far, I have seen NO proof that asymptomatic infectees are, in fact, capable of producing viral infectants. They probably are. But, we have no proof of that, or, if they are, at what point in the post infection cycle they become infectious. Many viral diseases only are capable of being transmitted from the host for a short time before symptoms appear. The level of free virus, in the host;s system has to reach a certain point before it can be transmitted to another.
So, what people do not understand here, is that the medical profession usually only classifies a person as being an official case of a disease, if the person displays symptoms of that disease, not because they have simply been exposed to the disease and may be infected.
My first computer was an Italian made Olivetti in 1980 used in an accounting firm. 40 years later the ole saying of ” Garbage In is Garbage Out. Obviously it still holds true.
What was missed in the articles quoted was that Ferguson’s code gave faulty inflated results for the Swine flu and other infectious events.
But for some reason in spite of that they *still* gave him money and a seat at the table!
https://statmodeling.stat.columbia.edu/2020/05/08/so-the-real-scandal-is-why-did-anyone-ever-listen-to-this-guy/
One of the problems is that governments and universities are so strongly linked, that governments would be hard pressed to ignore what an ‘expert’ at a university says. Governments provide untold amounts of taxpayer dollars to post secondary education – operating grants, ‘research’ grants etc. If they didn’t listen/use the results, taxpayers would begin to wonder if their money was being used effectively. Plus, universities would beat the drum that governments are not listening to them and the whole faculty would rise up in anger that they were being ignored.
The second problem is that because of ‘academic freedom’, which as been translated to be almost anything these days, academics producing crap like this model and the many global warming models, are not subject to any type of serious peer review. The peer review is essentially confirmation because those reviewing it will be looking for a similar confirmation in the future. No outsider looks at the model/research (because of academic freedom), and even through public funds paid for the model, no one is allowed to see the actual code.
The whole system is corrupt. I can only hope that after covid, somebody will cry that the emperor has not clothes, but probably not.
Everyone remembers Eisenhower’s warning about the “Military/Industrial” complex in his final speech. No one remembers that he also warned about the threat of worshiping scientists.
Look at the damage caused by this malarkey.
The software was so bad it wouldn’t even reproduce it’s own results.
It didn’t matter what you put in, garbage came out.
Yes, same input, inconsistent output.
This is the model Fauci used as well.
That he put faith into anything without qualifying it first is sheer incompetence, negligence, and stupidity.
And partly based upon his recommendations;
Millions out of work, losing their homes, all else…
Trump has to see this by now. The damage is already permanent, but the larger problem is that it is on-going.
This nonsense has to stop.
Fauci ….. “sheer incompetence, negligence, and stupidity.”
I do not rule out Deep State, Phase III of the effort to remove Trump. On its face, the Imperial College London (up to 2.2 million deaths) model was so inconsistent with the low Chinese deaths – now known to be false, but being taken at face value at the time – that I think a malign motive might be a better assumption. I think that Trump realizes that he was “had”, but is playing a bad hand by delivering supposedly much lower deaths than the models the “great” Dr. Fauci brought him.
The President is not going to deliver a booming recovery by November and having claimed too many times we would have a strong recovery he has opened himself up to a “We can do better economically” campaign against him – “We did it after 2008.” His repeated claims about great virus testing have been false. For example, my grandson had Covid and could not get a test until, finally, the doctor met he and my daughter at some facility to demand one.
Finally Trump has put a lot of political capital on a vaccine, which I think is most unwise, in general, and poorly timed relative to November, in particular. We do not need a vaccine, which never may be successfully developed for a variety of reasons, if we have a successful therapy – like an office test for strep throat and antibiotics if positive. NYU Langone last week reported 44% fewer deaths from the HCQ regimen (hydroxychloroquine + azithromycin + zinc) and Laura Ingraham had on a TX doctor who said all patients given it on an outpatient basis showed rapid improvement – its best application, per doctors using it. My money is on HCQ as the key to Trump’s re-election, not an iffy vaccine.
Remember the absolute explosion of rage when the Trump EPA announced that it would no longer use ‘secret science’ for its policy formations? that the science had to be public or they would not use it? Oh, how thunderous the editorials and talking heads exploded on them. Academia across the board denounced it as anti-science. Surely the Trump EPA was out to deliberately kill us all!
And this coronavirus / Ferguson thing is exactly why you don’t use ‘secret science.’