Wed 2021-Aug-04

Vaccine Efficacy, Delta Variant, and the CDC Changes

Tagged: COVID / MathInTheNews / PharmaAndBiotech / Politics / Statistics

Last week several bombs were dropped: COVID vaccine efficacy with the Delta variant, breakthrough infections, asymptomatic carriers, and changes in CDC guidelines. Was the panicky media coverage at all useful?

Vaccine efficacy vs Delta in the real world

STAT News: Pfizer/BioNTech vaccine efficacy at 6 months First, let’s consider the news about “waning efficacy” of the mRNA vaccines. STAT News, as usual, had a very even-handed account. [1] They noted that a Pfizer preprint showed after 6 months:

  • The Pfizer vaccine had overall efficacy of 91% averaged over the period following 44,000 volunteers. This is still outstandingly good.
  • Efficacy was 96% in the first 2 months, and 84% by the end. So, a decline of about 3%/month, which is actually pretty good.
  • It is unknown if that is due to fading immune response or the evolution of the virus into more virulent forms, like Delta.
  • Against severe disease requiring hospitalization, it was 97% efficacy. Very good, indeed.
  • At this rate of decline, efficacy would reach the benchmark threshold of 50% efficacy at 18 months, which would be the time for a booster. (Preferably a Delta-specific booster, but the original vaccine would also do.)

medRχiv: Thomas, et al. on Pfizer/vaccine efficacy at 6 months Other outlets were less measured in their reporting, along the lines of: “Glaah! The vaccines are fading, we’re all gonna die!” So let’s ignore them and look directly at the primary literature, namely the preprint itself. [2] (It’s a preprint, so it’s undergoing peer review right now. But take it with a small grain of salt.)

  • They had 44,165 subjects studied in a placebo-controlled, observer-blinded, multinational trial. This isn’t some tiny little study!
  • Overall efficacy was 91% (95% CI: 89.0% - 93.2%). That’s a nice, narrow confidence interval, so this looks statistically solid.
  • Efficacy vs severe disease was 97% (95% CI: 80.3% - 99.9%).

Overall, it’s a good, solid result: the vaccines still work even with the variants present. At the rate of decline of efficacy, it should be about a year before boosters become de rigueur.

Conclusion: Don’t panic. At least not about this.

The CDC on Delta: ‘the war has changed’

The WaPo, the internal CDC slide deck, and media reaction

WaPo: CDC internal report on Delta, vaccines, and transmissibility

The WaPo “obtained” a CDC internal slide presentation and reported on the state of thinking at CDC. [3] They describe it as “striking an urgent note”, which no doubt led to some of the media reaction. They summarize the content in basically 2 arms:

  • There is significant frustration at CDC with public communications. Given the amount of superstitious fear and general interference by nearly every Republican politician, this is entirely understandable. People are frustratingly becoming convinced that the vaccines no longer work. But as you saw above, the efficacy is still 85% - 90%, which is the very definition of “works”!
  • We’ve been underestimating the the ease of transmission of Delta; it’s worse than we generally thought, and we already thought it was pretty bad. Vaccinated people can get asymptomatic Delta and spread it as easily as the much sicker unvaccinated. So universal masking is a good idea, until herd immunity (late 2021/early 2022?).

That provoked a lot of somewhat inflammatory reaction in other media. One of the slightly less inflammatory is shown here, from MSNBC. Keep in mind if you watch it that almost everything else I saw was more inflammatory than this. The things that went wrong here are:

  • He’s not interviewing anybody with domain expertise or COVID knowledge; he’s interviewing one of the WaPo reporters. Basically reporters interviewing each other. This is how small errors get amplified.
  • There is, at best, minimal discussion of the content of the CDC document, just what “people are saying” about it. The little content that is discussed is the communication problems, not the actual data about the Delta variant (which is the important part).
  • His tone of voice is rather alarming, as if urging viewers to panic and outrage.

CNN: White House frustrated with hyperbolic Delta coverage Predictably, this did not escape reaction from senior political folk, as reported when CNN talked to White House aides. [4] They used words like “hyperbolic”, “irresponsible”, and “alarmist”. Of course, they requested anonymity, which sort of lowers their credibility at least with me. Still…

“The media’s coverage doesn’t match the moment,” one of the Biden officials told me. “It has been hyperbolic and frankly irresponsible in a way that hardens vaccine hesitancy. The biggest problem we have is unvaccinated people getting and spreading the virus.”

Axios: Less than 0.1% of vaccinated tested positive At least some media outlets didn’t “bury the lede”. A prime example is at Axios, where Gonzalez & Alberti make the point sharply [6]:

  • The rate of breakthrough infections among the vaccinated is very, very low: possibly around 0.1%.
  • The hospitalization rate is better, at around 0.004%.
  • The death rate is even lower, around 0.001%.

Now that’s the right message: yes, there are breakthrough infections; but they are very rare, and almost none result in hospitalization or death.

As the CDC reported, breakthrough infections can be spreaders, so it’s a good idea to wear a mask even if you’re vaccinated. If you’re unvaccinated, then just realize you should also mask, but being voluntarily unvaccinated at this point is kind of suicidal.

Ok, so what exactly is in the internal CDC slide deck?

CDC slide 1: Improving communications around vaccine breakthrough and vaccine effectiveness Here’s the (frustratingly anonymized!) CDC slide deck [5], archived here in case it gets removed from the WaPo web site.

CDC slide 3: Vaccines dramatically reduce risks of infection, hospitalilzation, and death There’s some entirely understandable venting about the problems of communicating with the public, especially at the same time as disinformation actors like Republican officials. Still, they manage to show that vaccines are _dramatically reducing risks as shown on slide 3:

  • Infection risk is reduced 8 fold, corresponding to an efficacy of 100% * (1 - 1/8) = 87.5%.
  • Both hospitalization and death risks are reduced 25 fold, for an efficacy of 100% * (1 - 1/25) = 96%.

That’s fantastically good evidence that vaccines work, and work well. Why wasn’t this one of the headlines in the news?

CDC slide 15: Mortality vs transmissibility for various viral diseases One item that got some rather nervous side-eye is shown on slide 15. Here the plot shows $R_0$ on the horizontal axis (basically how many new cases of infection does each case generate; it measures the ease of transmission for a disease). The vertical axis is the mortality on a log scale (what fraction of the infected people die). Obviously, the upper right corner (high transmission, high mortality) is where we don’t want to be.

The inital strain of SARS-CoV2 is sown in the pink box at the lower left. The box size apparently shows the uncertainty band, though they don’t quite come out and say so. It’s about as transmissible as the common cold, but it kills somewhere between 0.1% – 1.0% of patients. That’s pretty bad, but…

The blue box is the Delta variant. It’s not remarkably higher in mortality (maybe just a little). But it’s way more transmissible! It looks like they’re estimating $R_0 \sim 5-9$, vs $R_0 \sim 1.5 - 2.5$ for the original strain. So maybe 4 times more transmissible, on average? That’s a huge deal. It also puts Delta on a par with the highly transmissible chicken pox, though thankfully not on a par with the horror that would be measles ($R_0 \sim 15$).

CDC slide 17: Breakthrough infections in vaccinated people can cause spread Another bit of bad news is in slide 17: although breakthrough infections in vaccinated people are rare, once they happen those people can be asymptomatic spreaders to the unvaccinated. So the conclusion is: everybody needs to mask up, so the vaccinated won’t be spreaders of Delta to the unvaccinated.

Cue: infinite whining about masks, mask mandates, changing policy, etc. This is why the CDC said “the war has changed”: Delta is worse enough that we have to go back to masking, until the remaining unvaccinated can be made safe with vaccination.

The Weekend Conclusion

The bad news:

  • Delta is more contagious than we thought, maybe on a par with chicken pox.
  • Vaccinated people can (rarely!) get asymptomatic Delta, and be spreaders.

The good news:

  • The rate of breakthrough infections in the vaccinated is very, very low.
  • Vaccine efficacy remains high, both as measured in the Pfizer study and by the CDC study of cases, hospitalizations, and death rates.

So to prevent any (rare!) asymptomatic Delta cases in the vaccinated from being spread, everybody should mask up again.

Notes & References

1: D Garde & M Herper, “Efficacy of Pfizer/BioNTech Covid vaccine slips to 84% after six months, data show”, STAT News, 2021-Jul-28.

2: SJ Thomas, et al.,, “Six Month Safety and Efficacy of the BNT162b2 mRNA COVID-19 Vaccine”, medRχiv 2021-Jul-28. DOI: 10.1101/2021.07.28.21261159.

3: Y Abutaleb, CY Johnson, & J Achenbach, “‘The war has changed’: Internal CDC document urges new messaging, warns delta infections likely more severe”, Washington Post, 2021-Jul-29.

4: O Darcy, “White House frustrated with ‘hyperbolic’ and ‘irresponsible’ Delta variant coverage, sources say”, CNN Businesss, 2021-Aug-02.

5: Authors redacted, “Improving comunications around vaccine breakthrough and vaccine effectiveness”, 2021-Jul-29. Originally obtained by the Washington Post.

6: O Gonzalez & D Alberti, “Chart: Less than 0.1% of vaccinated Americans tested positive for COVID-19”, Axios, 2021-Jul-31.

Published Wed 2021-Aug-04

Gestae Commentaria

Weekend Editor, Sat 2021-Aug-28 00:50

Hey, Kto –

I think I now understand the reason for our confusion: Simpson’s paradox.

See this new post. Basically, age is a correlate of both being vaccinated, and being hospitalized if there’s an infection. So you have to stratify by age.

There’s more, including calculating the right Bayesian posterior, but that’s the main thing.

All of this was uncovered by Morris, writing at COVID-19 Data Science. I just did a little “explainer” about it.

Thanks for planting a seed of criticism that made me go read some cool stuff.


I’m aware of Simpson’s paradox, but I don’t think it resolves our original issue. Morris’ data shows that when you stratify the data by age, you find ~80-90% efficacy against severe cases among the most vulnerable groups. Given ~90% vaccination rate, this is consistent with the vaccinated people making ~50% of all deaths. So, there is still a missing factor of ~125, when you compare the UK and Israeli data to the Fauci’s 0.8% claim for the US.

Thank you for the detailed response.

“Summary: I don’t quite believe that the death rates in other countries are now mostly among the vaccinated”

You don’t have to take my word on it. Here is a direct link to the official UK government data:

According to the UK report, 54% of all deaths from delta are in fully vaccinated patients. By contrast, according to Fauci’s claim only 0.8% of recent deaths in the US came from the fully vaccinated people (i.e., the odds ratio of 125:1 instead of approximately 1:1). There is a missing factor of ~125 here.

Weekend Editor, Tue 2021-Aug-17 19:02

Yeah, you convinced me there’s something contradictory here.

All the other stories report very low death rates among the vaccinees, e.g. here.

Perhaps there’s some time skew, where the low death rates are pre-Delta? Or is somebody using a different denominator without actually describing what they’re doing? (My experience: more confusion comes from trying to “avoid math” and not write an equation. A bowl of word salad vaguely explaining it is an invitation to trouble.)

Hi Weekend Editor,

I found your blog through the ACX classified thread. I have a question about your post:

How can your statement (“Vaccinated people can (rarely!) get asymptomatic Delta”) and the claim that “Against severe disease requiring hospitalization, it was 97% efficacy” be reconciled with the data from UK and Israel where the majority of deaths come from fully vaccinated?

Given that the fraction of vaccinated among the elderly there is around 90% (less among the younger age groups), with the 97% efficacy the deaths among the vaccinated should be under 25%.

Weekend Editor, Sat 2021-Aug-14 23:15

Hi, Kto:

Welcome, and thanks for taking the time to read a post!

It’s nice to hear from somebody who is neither a Russian casino spammer nor a personal acquaintance/family. :-) This crummy little blog has a global readership of about 6, but 3 of them are my spouse, my cat, and myself. And the cat isn’t a particularly close reader.

Let me rephrase your question, so we can at least share a qualitative understanding of what we’re trying to wrap our heads around:

  • In the US, breakthrough infections are rare, and almost never lead to hospitalization or death. (See: first 2 references here, documenting the claim that 99.5% of US COVID-19 deaths are now among the unvaccinated.)
  • But in the UK and Israel, most deaths are from the fully vaccinated.

I’ll start by saying I haven’t read data saying that about the UK and Israel. This is not at all unexpected, since I haven’t sought it out. In fact, my (only very lightly informed) impression was the opposite: see here for evidence from a Qatari study and an Israeli study documenting near-100% efficacy vs severe disease after both doses of mRNA vaccines.

But for the sake of discussion I’ll take you at your word that you think this is the case, and you may even be right since I’m not especially informed here. The question then is: why do we think vaccination is so protective against death in the US, but in the UK and Israel it looks less so?

If I had access to actual data showing that, then maybe we could do some statistical hypothesis testing. But I don’t have that data, and given the chaotic patchwork of EMR systems in the US, it’s pretty much unobtainable. There’s a reason some of the best studies I’ve seen were the Israeli & Qatari ones: they have actual universal health care and universal EMR systems so the can get at reasonable datasets.

But with things as they are, I’ll just content myself with pointing out some of the problems of comparing across countries and medical systems:

  • Differential viral variant mixes: The Delta variant didn’t appear simultaneously in all countries. (That would have been terrifying.) It came first out of India, then began appearing in the UK & Israel, and didn’t emerge in the US until April 2021. The plot here shows the preponderance of the variants vs time in the US; Delta is the scary green thing that grew rapidly after April. That means for a study at a fixed point in time, the US is looking at different viral variants than the UK & Israel.
  • Differential population behaviors: These 3 countries have populations who behave vastly differently among each other.
    • Israel was admirably fast with vaccination, doing the elderly first. The UK was pretty OK. In the US, we whined, and moaned, and complained, and stomped our feet, and made up conspiracy theories to try to squirm around vaccination. Basically I’m saying you probably have much greater compliance with public health in the UK & Israel (though I’m so sick & tired of the US vaccine resistance that it might be confirmation bias on my part to say that.)
      • So in the US we’re more likely to have willfully unvaccinated middle-aged to elderly people, who are at most danger of death from COVID-19. Any serious statistical modeling would have to take into account public health compliance.
      • Even inside the US, you have different subpopulations with radically different responses to public health guidance. New England masked up, socially distanced, and got vaccinated pretty fast. In the South, they’re still debating about whether the vaccine is from Satan, or just from Bill Gates & George Soros. Very different, and hard to compare!

      • Also, it depends on when the waves hit each country, since best-practice medical care changed. Basically we learned how to care for COVID-19 patients as time went on: use of remdesivir & dexamethasone, intubating in prone position, supplemental antibodies, and so on. See, for example here where I looked at viral mRNA recovered from sewage processing in Boston over about a year, and 2 different waves of infection. I built regression models predicting hospitalization, ICU admission, and ventilator useage from the sewage viral mRNA levels. It sort of worked and sort of didn’t, since the 2nd wave had much less hospitalization useage per sewage mRNA, because we learned better how to treat patients. All the more so, the deat rates between the 3 countries will depend on exactly when the waves occurred. Serious statistical models have to take into account that medical best practices changed with time, as well.
  • Differential immunity waning: We don’t know how long immunity lasts. All the data I’ve looked at (and blogged about) says it lasts pretty well so far, especially against hospitalization & death. But let’s suppose it fades. It will then fade first in the group first vaccinated. Israel was very good about vaccinating its elderly very fast. The US… somewhat less so. Then more immunity waning would have happened in Israel, leading to more deaths among the vaccinated. (I’m pretty sure I don’t believe the immunity fades that fast, but I could be wrong. See the Israeli & Qatari data referred to above for a real-world example.)
  • Differential $R_0$: The viral $R_0$ depends on the virus/host biology (binding to the ACE2 receptor, endocytosis mechanism, reproduction speed, etc.). But it also depends on host behavior: masking, social distancing, no indoor gatherings, etc. We can alter the effective $R_0$. The evidence is clear that some populations do a better job of this than others; see the American South for an example of how not to do it.
  • Rising vaccinated population: Every time we calculate a case rate for the vaccinated, we need a ratio of 2 numbers: the number of diagnosed cases in the vaccinated population, divided by the size of the vaccinated population. The problem with comparisons over time is that the denominator has been growing (and that’s a good thing!) as more people get vaccinated. So the virus, over time, has become both more infective (Delta variant) and it has more shots on goal, i.e., more vaccinated people to attack.

Summary: I don’t quite believe that the death rates in other countries are now mostly among the vaccinated, and have some data that makes me think that. However, whatever the difference in vaccinated death rates are between countries, comparisons over different populations exposed to different variants with different medical care best practices and different compliance with public health guidance is… unlikely to be straightforward?

Comments for this post are closed pending repair of the comment system, but the Email/Twitter/Mastodon icons at page-top always work.