The Undark Interview: John Ioannidis Responds to His Critics
In mid-March, the Stanford University scientist John Ioannidis wrote a short, viral essay for STAT arguing that the global response to the Covid-19 pandemic could be “a once-in-a-century evidence fiasco.” Without more data about the virus’s spread, Ioannidis, a professor of medicine, epidemiology, and population health, argued, the lockdowns in place in much of the world may not be justified. Covid-19 infections could be more widespread, and less lethal, than many experts feared.
Over the years, Ioannidis has earned accolades from his colleagues for his sharp, reasoned skepticism of sloppy research practices and unsupported assertions. His Covid-19 claims, though, have earned him vigorous pushback from many of his fellow scientists, who argue that, while he may be correct, Ioannidis is getting out ahead of the data — and likely understating the risks of Covid-19.
At issue here is a simple question: How many people actually have Covid-19? Ioannidis and other researchers from Stanford tried to answer that in a draft paper, or preprint, last month. Other experts began pointing out problems in the study, raising concerns about statistical errors, possible issues with a Covid-19 test kit, and shoddy sampling technique.
Undark published a story about the controversy late last month. Ioannidis did not respond to multiple requests for comment before publication. But, less than an hour after the story went up, he sent me a warm note expressing appreciation for the scientists who had criticized him. We arranged a time to talk.
A few weeks later, the team released a revised version of the paper. The new draft, which, like the original version, has not yet received formal peer review, softens some of the more controversial claims, and acknowledges more uncertainty about the true number of infections.
The following interview — which covers the papers as well as Ioannidis’ appearances on partisan television — has been edited for length and clarity.
Undark: What kind of responses have you been getting so far to the revised draft?
John Ioannidis: Well, we’ve heard from several people. And I think that they’re happy that we have addressed the main issues that were raised on the first round. I also saw that [Columbia University statistician] Andrew Gelman, who was probably the most critical voice in the first round, has posted his appraisal of the revised version, which I think is very reasonable.
It’s hard to recall another paper that has been so extensively peer reviewed. [He laughs]. And lots of accounts were very useful and very constructive. The revised version has tried to address all the major concerns. I think the results are still very robust.
But it’s a single study. You can never say that a single study is the end of the story. You need to see all studies that are done, and by now there’s more than a dozen serology studies, and I think they pretty much paint the same picture.
UD: How so? In terms of estimates for how many infections there are, and what the infection fatality rate actually is — there still seem to be some substantial differences.
JI: Yeah, but this is entirely expected. Infection fatality rate is not like Avogadro’s number. It’s not a constant, like in a chemical experiment, the Km of an enzyme reaction. It’s affected both by how you count the [numerator] and how you count the denominator, and who are the people in the [numerator] and who are the people in the denominator. So, the case mix is very different in different locations. And the way that the serious cases were managed, or could be managed, is very different in different locations.
So, depending on the setting and the population, the infection fatality rate may be from far less than influenza to far more — from the mild infection all the way to “This is disaster.”
UD: The revised draft has a new line that says, “Our prevalence and [infection fatality rate] estimates do not advocate for or refute the usefulness of any nonpharmaceutical interventions.” What do you think are policy implications from this study?
JI: Our point was to really just present the results of a scientific study. There are policy implications, of course, from any scientific study, especially from one that tries to address such an important question, but I’m afraid that many people just got entangled into a fight over, “So this study means that our policy is right, or our policy was wrong.” So then, “You’re in favor of Republicans or Trump, or you’re in favor of…” [laughs]
We felt it’s important to dissociate the specific paper from making policy recommendations, because this is taking things to a different level. Now, if you ask my opinion about whether it does have policy recommendations or implications separate from the study, I think what it says is that this is a very common infection, and very often it is asymptomatic, so it goes below the radar screen. If you just wait for people to show up to get tested and then you track the contacts, that’s not going to work, because you will miss the majority, probably the large majority, of people who will just not show up.
I just didn’t want us to see that study as being a test of the hypothesis, “Did lockdown work?” That’s not what the study did. That was not the intention. It was trying to get a particular piece of information, and trying to do that as carefully as possible, with all the limitations that exist in these types of surveys.
UD: The day the study came out, one of your coauthors published an op-ed saying that the study suggests lockdowns might be too strict. That same week, you and some of your coauthors were talking to the media about policy implications. How is that not making it political?
JI: You’re right about that. But at the same time, I want to give a little bit of a different perspective. So we did this study — the samples were run in the first days in April. And we were very careful not to say anything in the news, not to appear and make statements about the study until we had a full paper, written up and deposited.
[Compare that] with what practically every other study did, where they completely rushed to press release and press conferences and appearing in the news, like the same day that they got the results.
Our stance was — and this is entirely aligned with my beliefs — this is important science. It can have repercussions, there’s no doubt about that.
So it is important to not just go and make press releases, but really write out the full paper — admittedly without the appendices, which we added in that second version.
This was something that was not possible to just hide it under the carpet. It was a major finding, and I worried that it would be misinterpreted in different ways. So, yes, I did show up on Fox, I did show up on CNN. … You know, on BBC. Does it mean that I have a conservative agenda if I appear on Fox, or that I have a Democratic agenda if I appear on CNN? I think that’s [laughing] just a complete, complete, complete misunderstanding. I’m just a scientist. I have no political party affiliation and absolutely no interest to turn this into a political debate, or to have a political agenda supported.
 |
Thoughts or questions on Covid-19? |
UD: I’ve seen your work being widely cited by people — including Fox News’ Tucker Carlson, who had you on as a guest — who are saying that the pandemic is not that serious, that it’s been overblown.
JI: I think that every citizen has the right to read science and try to make some sense of it. I cannot stop people’s interpretations of scientific findings, and it would not be appropriate for me to do that. So, yes, different people from different ideologies and from different backgrounds will read these results in different ways. But, this is not necessarily my reading or my interpretation. To be honest, as a scientist, I prefer to offer the data and try to be as calm as possible.
I think the main question is, should I or others, not talk to anyone — either Fox or CNN or BBC or Der Spiegel or Reuters or whatever? In principle, I have a problem with this perspective. It think that it creates a notion that scientists should not present the work. If you see some of these interviews — I mean, you mentioned Tucker Carlson, you can see my interview. And it’s obvious that I do not agree with lots of things that Tucker Carlson was proposing. At the same time, when he says that this virus is less lethal than we thought, this is accurate. It’s not who is saying it. It is whether this is an accurate statement or not. We started thinking that one out of 30 people will die. You know, when the [World Health Organization] made the announcement.
UD: Who thought that? The WHO said that 3.4 percent was the case fatality rate. Epidemiologists I’ve talked to said that it was clear the true infection fatality rate would likely end up being much lower. One scientist described the argument you’re making right now as “a straw man.”
JI: Well, let’s go back and check the exact announcement. [Note: The WHO announcement in question, from early March, specifies that “3.4 percent of reported cases have died.”] That was at the time when WHO had sent an envoy to China. And [the WHO envoy] came back and he said there’s no asymptomatic cases. Just go back and see what the statement was. He said there’s hardly any asymptomatic cases, it’s very serious and has a case fatality of 3.4 percemt.
Of course, that [fatality rate] was gradually dialed back to 1 percent or 0.9 percent. And these are the numbers that went into calculations, and these are the numbers that are still in many of the calculations, you know, until very recently.
You know, 1 percent is, is probably like the disaster case, maybe in some places in Queens, for example, it may be 1 percent, because you have all that perfect storm of nursing homes, and nosocomial infection [an infection that originates in a hospital], and no hospital system functioning. In many other places, it’s much, much lower.
I’m trying to disentangle the accuracy of a scientific statement from who is making that scientific statement. Because if we don’t agree with who is making the scientific statement, then we run the risk of attacking the science, because it was just stated by that person.
We have to be very cautious here. I think that that’s going to be highly detrimental to science. My first opening statement in my first Fox interview was that science is the best thing that has happened to humans. I’m very proud to say that again and again and again.
UPDATE: In the original version of this article, John Ioannidis misspoke in referring to the “nominator” for the fraction representing the presumed infection fatality rate of Covid-19. The correct term is “numerator.” This mischaracterization has been corrected.
Comments are automatically closed one year after article publication. Archived comments are below.
WHY ARE NONE OF YOU ‘REASONABLE’, ‘OBJECTIVE’ libertarian “thinkers” from “Austrian persuasion” or whatever talking about the fact that Ioannidis “science” was funded by JetBlue’s highly political anti lockdown founder? Just as with ‘cigarettes don’t cause cancer’ or ‘climate change isn’t real’ studies being funded by industry, this is invalidating.
Well, being Austrian I am not sure if I can satisfy your need for an unbiased reply, but I will do my best good Sir.
Every study has to be paid for, can we agree on that? Pro and Con alike. So this would invalidate every single study, ever, by definition. What you tend to do, is to set up a pool for funding, that gets anonymized.
Do I say that one or the other is un-/biased? Nope. Does it cancel the findings of the study? I see no connection that invalidates the general findings – nor did the heaviest scientific critics of the study so far. Problem is more: it seems a lot like Harvard vs. Standford in USA. Whilest I couldn’t care less on those two, I hope that Ioannidis is right in the end, because this would mean we all win.
The question about the case fatality rate and the inserted excerpt from the 3/3/2020 WHO release are extremely misleading and border on being a lie of omission. The relevant part of the WHO’s statements reads, “Globally, about 3.4% of reported COVID-19 cases have died. By comparison, seasonal flu generally kills far fewer than 1% of those infected.” In those two sentences the WHO directly compared the CFR at that time of COVID-19 to an infection fatality rate for the flu.
THAT was Dr. Ioannidis’ original, and correct, criticism: The 3.4% number, which the WHO itself was insinuating to be an infection fatality rate, was meaningless because no one knew the denominator.
Just a month ago Ioannidis predicted fewer than 40,000 deaths in USA, and we’ve passed more than double that many.
Much larger and statistically relevant studies like Spain with >60,000 participants, are getting ~1.2% IFR both nationally and separately in Madrid.
https://twitter.com/_MiguelHernan/status/1260625031119409156?s=20
Ultimately too much airtime has been given to one scientist who hasn’t been particularly insightful on COVID-19. I suspect it’s less of Ioannidis own design, and more because the political right has sought any contrarian scientists they could find.
“If I were to make an informed estimate based on the limited testing data we have, I would say that covid-19 will result in fewer than 40,000 deaths this season in the USA,”.
As he said, its based on the limited data. The other thing to understand in that interview was the distinction between “deaths with covid” and “deaths of covid”. Presumably most reported deaths are those with covid. Every hospital and country has different protocol and methods for determining cause of death and it can get very complicated and skew information very quickly.
All serological studies are showing that the disease is much more prevalent and this is going to drive the mortality statistics down. and breaking the mortality rate by age will drive it even lower and lower.
The debate on the lock-downs is a losing one for you guys and as more information becomes available, it becomes more clear it is a wrong decision.
Not only it exposes us to second and third waves which will eventually increase the death toll. it creates a state of fear which resulted in decrease ER visits for MI and strokes meaning more deaths from ordinary causes that could have been prevented. Increased death from tumors will also follow.
And of-course no need to say it is anti-freedom and unconstitutional.
What’s the point to add the editor’s note about the WHO 3.4% CFR when Ioannidis says it at the end of the paragraph? Why add a paragraph to say that Ioannidis misspoke numerator instead of just fixing it? Why insert the Ioannidis laugh track?
You guys are so petty.
“You know, 1 percent is, is probably like the disaster case, maybe in some places in Queens, … … In many other places, it’s much, much lower.” – JI
Santa Clara has a population of 100k vs 2m in Queen. You have a massive problem if the disaster scenarios are happening in the most populous locations in the US.
And Santa Clara might have been a little hotspot, but obvously still had just a tiny outbreak. So …
1) No stressed health system
2) less death could habe been just random
and there is still the possibility of self selection bias in the study (despite all modifications)
The study was for Santa Clara County. NOT the City of Santa Clara
“If I were to make an informed estimate based on the limited testing data we have, I would say that COVID-19 will result in fewer than 40,000 deaths this season in the USA.” – John Ioannidis
As we sit at double that number of deaths one month after he made that statement, he looks like he just might not know what he’s talking about.
But it seems that he already turns more careful.
I think the major problem is: All scientists estimated real IFR lower than the CFR heard about. *All*. But I think he had a private estimate before the study and Santa Clara “confirmed” his estimate, thus he did not question his own results in the study, but went public quickly. And now many with a political agenda just refer to his estimate/result as “true” IFR.
This was a disappointingly shallow set of responses.
Ioannidis doesn’t seem to address the WHO 3.4% straw-man in very good faith.
He seems to be justifying getting ahead of his study’s data to make statements about severity and policy, with yet another straw-man argument: other scientists released their serology results without prints.
Lastly on the general idea that IFR is not a constant and depends on who gets infected: yes, and (1) this is important to mention when talking about the data or making inferences, and (2) other good scientific work has provided age-stratified results (as recommended by WHO), so that those results can be applied across different populations, and mitigation strategies can be benchmarked.
Have you read WHO report? They’re stating there that case fatality rate is 3.4% and only 1% of all cases are without symptoms , which indirectly suggests that if you get this virus, with 99% chance you will have symptoms and be included in total count of cases.
You’re missing the point. WHO’s figures aren’t what’s being debated. I suggest looking up the definition of “straw man argument”.
That was at a time with less knowledge. Many scientists had already estimated IFR a bit 1 % from the beginning because of the fact that now all cases could be known in China and also in Italy because testing cannot catch all cases in such a major outbreak.
The problem is rather that Ioannidis obvisouly had his private estimate at the lower end near the flu from the beginning and thus did not question his own Santa Clara results.
He is responding to a particular WHO statement and he is clearly correct. But instead of admitting that the WHO was wrong and moving on, you throw straw-man accusation at him.