Scary 22% vaccine efficacy in South Africa comes with heaps of caveats

Dismal preliminary data on AstraZeneca’s COVID-19 vaccine in South Africa—where the B.1.351/ 501Y.V2 coronavirus variant is spreading widely—led the government there to rethink its vaccination rollout and raised further international concern about the variant.

But the small study has so many limitations and caveats, experts caution that drawing any conclusions from it is difficult.

The study, which has not been published or peer-reviewed but presented in a press conference Sunday, began in June and enrolled only around 2,000 participants, about half of which received a placebo. Early in the study—before B.1.351 emerged—the vaccine appeared over 70 percent effective at preventing mild-to-moderate cases of COVID-19. That is largely in line with the conclusion of an international Phase III trial released by AstraZeneca and vaccine co-developer Oxford University, which showed mixed results for the replication-deficient adenovirus-based vaccine but an overall efficacy of around 70 percent.

But toward the end of the year—when cases of B.1.351 began taking off—the South African results shifted dramatically. The overall efficacy of the vaccine dropped to just 22 percent against mild-to-moderate COVID-19. In an analysis comparing efficacy solely in cases known to be due to B.1.351, the vaccine fared even worse, showing an abysmal efficacy of just 10 percent.

On first glance, those numbers are startling, and they suggest that the B.1.351 variant rendered the otherwise effective vaccine nearly useless. But, with a closer look, those numbers are almost uninterpretable. The trial was just too small to produce statistically solid results. Thus, each efficacy calculation has huge confidence intervals. For instance, the vaccine’s ending efficacy of 22 percent—based on a total of 42 coronavirus infections in the trial—had a plausible estimate range of between -50 percent effective and 60 percent effective.

Lingering questions

The study also was not designed to answer the critical question of whether the vaccine can prevent severe disease, hospitalization, and death. The people enrolled in the trial were relatively young—median age of 31—with very low rates of underlying conditions. Both of those factors make people far less likely to develop severe COVID-19 and die from the disease.

Experts emphasize that preventing severe disease and death is the most crucial goal of vaccines—not preventing transmission or mild illnesses. Last month, Johnson & Johnson announced that its vaccine—which is an adenovirus-based vaccine, like AstraZeneca’s—was only 57 percent effective against moderate and severe COVID-19 in South Africa. But, experts applauded the finding that it was 85 percent effective at preventing severe disease and death. And the company reported that no one vaccinated in its Phase III trial ended up being hospitalized or dying from the disease. Many experts championed the not-yet-authorized vaccine as a potential new tool for ending the pandemic.

The critical looming question now is whether AstraZeneca’s vaccine will likewise protect against severe disease and death in cases involving B.1.351, which has now been detected in more than 30 countries. Given the uncertainty, researchers and officials in South Africa said on Sunday that they would, for now, pause the rollout of AstraZeneca’s vaccine, which was authorized for use in the European Union on January 29. (It has not yet been submitted to the US Food and Drug Administration for authorization.) South Africa had recently begun offering it to health care workers. Instead, officials there are now looking into the possibility of a phased rollout, which might involve vaccinating just 100,000 people and monitoring them over some time to look at hospitalization rates. If the vaccine looks effective at that point, they would then roll the vaccine out further.

“Put very simply, we don’t want to end up with a situation where we vaccinated a million people or two million people with a vaccine that may not be effective in preventing hospitalization and severe disease,” Salim Abdool Karim, the co-chair of South Africa’s Ministerial Advisory Committee on COVID-19, said in a press conference Monday.