Epidemiologist Says CDC's Flu Vaccine Effectiveness Numbers Are Built on Shaky Math

Every flu season, the CDC puts out a number: the vaccine was X% effective. Doctors use it. Reporters repeat it. Most people never ask how it was calculated.

Epidemiologist Eyal Shahar did ask, in an analysis published through the Brownstone Institute and recirculated by ZeroHedge, the Robert Scott Bell Show, and several smaller outlets. His target is the test-negative case-control design, the statistical method behind most of the CDC's flu vaccine effectiveness estimates, including those produced through the VISION Vaccine Effectiveness Network, a CDC-affiliated research collaboration.

Shahar notes that the NIH director has already criticized the test-negative design publicly. Shahar agrees, and digs into one specific VISION study from the 2022-2023 flu season, a year when the vaccine was considered well-matched to the circulating strain.

How the Design Works, and Where Shahar Says It Breaks

Test-negative studies compare vaccination rates among people who showed up at a clinic or ER with flu-like symptoms and tested positive for flu against those who tested negative. The logic is that restricting the sample to people already seeking care should cancel out differences in who goes to the doctor in the first place.

Shahar argues that fix creates a new problem: collider bias, a statistical distortion that can appear when you restrict analysis to people who share a common consequence, like seeking medical care, in a way that scrambles the relationship between vaccination and infection. He says the net effect of this tradeoff is unknown even to the people running the studies.

His bigger complaint is about timing. Flu risk in the 2022-2023 season was high from October through December 2022 and dropped off from January through March 2023. But vaccination rollout tracked the same calendar. According to Shahar's reading of the published VISION data, 61% of unvaccinated patient encounters fell in the high-risk October-December window, versus only 47% of vaccinated encounters. In the low-risk January-March window, the numbers flipped: 53% vaccinated versus 39% unvaccinated.

In plain terms, vaccinated people in the study banked more of their exposure time during the safer months. Shahar illustrates the risk with a thought experiment: if nobody were vaccinated during the dangerous months and everyone got a saline shot right as risk dropped, the saline would look protective purely from the calendar mismatch.

He also flags the study's exclusion of flu events occurring within 14 days of vaccination, a standard practice meant to account for the time it takes the vaccine to work. Shahar argues this can introduce immortal-time bias, where excluded time periods favor one group. Nearly 2,000 outpatient encounters were dropped from the study because vaccination happened 1 to 13 days before the record date or vaccination status was unclear. The published study didn't break those cases out separately, according to the Robert Scott Bell Show's summary of Shahar's piece.

Shahar points to a separate pattern that caught his attention: statistical adjustment reduced many of the effectiveness estimates, and he reads the reductions as a sign that confounding is present, with residual confounding likely remaining uncorrected. He also notes effectiveness against hospitalization came in lower than effectiveness against outpatient visits, and that protection against hospitalization appeared far stronger in older adults than younger ones. Among hospitalized elderly flu patients in the study, mortality was 4.0% in the vaccinated group versus 2.7% in the unvaccinated group. Shahar is explicit that this comparison cannot establish whether the vaccine affected mortality, since the groups weren't randomized and differed in baseline health.

What This Is, and Isn't

Shahar's conclusion is that the CDC-linked estimates shouldn't be treated as precise, causal measurements of how well the flu vaccine works, and he calls for alternative observational designs.

Defenders of the test-negative design would likely say in response that the method exists specifically because a yearly placebo-controlled trial of flu vaccines in the general population is considered impractical and, by most public health officials, unethical once a vaccine is already recommended. Observational designs are the tool available, and the CDC isn't the only agency relying on them. (None of these defenders appear in this round of coverage.)

No CDC response to Shahar's specific critique appears in the available coverage, and no agency review or retraction has been announced. The piece circulated across ZeroHedge, Brownstone, and allied sites as a single argument from a single author, not as independently corroborated reporting. Whether the CDC or the VISION network addresses the calendar-bias and exclusion-window critiques directly remains an open question.