Hospitals Are Deploying Uncleared AI Tools on Patients While Researchers Warn the Evidence Isn't There Yet

The Study Everyone Cited — and the Warning Everyone Ignored

In April, a research team led primarily by Harvard and Stanford scientists published a study in the journal Science that pitted ChatGPT against hundreds of physicians in a structured diagnostic competition using real-world patient cases. The AI won.

Adam Rodman, a lead author on the study, immediately tried to pump the brakes. At a press conference ahead of publication, Rodman said the work "was an academic exercise" and that it did not prove ChatGPT or any other AI tool was ready to become a standard part of medical practice. He said he felt "a little bit queasy about how some of these results might be used."

He was right to be queasy. The deployment was already underway before the paper hit print.

What's Actually Happening in Hospitals Right Now

Daniel Oppenheimer, a pathologist writing in The Atlantic, described receiving another email from his medical center's administrators while watching Rodman's press conference. The email announced that an "AI-powered clinical reasoning tool" was now available for clinical staff to use. He noted it was not the first such email. Not the second or third either.

None of these tools, he reported, had been approved for medical use by the FDA.

Hospitals are rolling out generative AI under a generic disclaimer that "AI can make mistakes." The statement is technically true but offers little practical safeguard.

The Studies Doctors Should Know About

The diagnostic competition result isn't the whole picture. A randomized trial published in NEJM AI — just one week before Rodman's study dropped — found that intentionally erroneous AI output easily led doctors to wrong conclusions. The AI didn't just fail quietly; it failed in a direction that pulled trained physicians with it.

For patients trying to use AI to manage their own care, the picture is similarly mixed. Oxford scientists published research finding that AI did NOT significantly improve patients' ability to diagnose themselves or others. A separate study led by Mount Sinai researchers found chatbots may fail to alert users to potential medical emergencies.

These are peer-reviewed findings from major academic institutions.

The Case For AI in Medicine Is Real — and Incomplete

The strongest argument for moving fast here deserves a fair hearing. Physician shortages are acute in rural and underserved areas. If AI can reliably flag a rare disease, catch an unusual symptom pattern, or serve as a second set of eyes when no specialist is available, the upside is enormous.

Rodman's own study showed the technology can perform at that level in controlled conditions. And clinicians who dismiss AI entirely risk leaving a genuinely useful tool on the table while patients suffer from the status quo's known failures.

The problem is using that argument to justify deploying tools in live clinical environments before the evidence base supports it. A tool performing well in a structured academic challenge is not automatically safe in a busy emergency department at 2 a.m. with incomplete patient records and a fatigued physician.

The Regulatory Gap

FDA clearance exists for exactly this reason: to separate tools that work under controlled study conditions from tools that are safe at the point of care. The AI products being pushed to hospital staff are not going through that gate.

Oppenheimer points out that health care is normally one of the last fields to adopt new technology. Not out of stubbornness, but because the culture recognizes that a poorly timed glitch can be fatal. That conservatism served patients reasonably well. It is now being abandoned in one of the least conservative ways possible: hospital administrators enthusiastically emailing staff to try uncleared software with minimal structured guidance.

The government waste angle runs in both directions here. If AI genuinely improves diagnostic accuracy and cuts downstream costs, refusing to develop a regulatory pathway quickly is its own failure. But deploying tools that generate confident-sounding wrong answers inside a system that bills insurance and makes treatment decisions is a liability exposure that taxpayers and patients will ultimately absorb.

What Hasn't Been Answered

The critical unresolved question is liability. When an AI-assisted diagnosis is wrong and a patient is harmed, who is responsible: the physician who used the tool, the hospital that deployed it, or the vendor who built it? No clear legal framework exists yet, and hospital administrators sending out cheerful rollout emails are not, as far as the public record shows, answering that question before they hit send.

The FDA has not announced an enforcement timeline for unapproved AI tools currently operating inside health systems. Until it does, the gap between what these tools have proven they can do and what hospitals are trusting them to do will keep widening.