Scale AI Study: Chatbots Spotted Distress but Skipped Crisis Resources in About 35% of Test Chats

A new round of testing says AI chatbots are better at noticing a person in crisis than at doing anything useful about it.

In research Scale AI shared exclusively with TIME, the company asked 19 licensed clinicians and crisis counselors to write 718 realistic chats simulating someone in crisis reaching out to a chatbot. Scale AI then ran those scripts through 25 frontier models, including ones from OpenAI, Anthropic and Google.

In about 35% of the test conversations, the models recognized the user was distressed but did not refer them to resources such as a suicide hotline.

What the Scale AI test found

"The models do a good job, regardless of the scenario, of identifying, 'Look, this person's talking about something harmful,'" said Patrick Oathout, Red Team & Safety Lead at Scale AI. "But they'll respond in a very just empathetic, kind way as opposed to saying, 'Okay, it's time that we get you help.'"

Performance dropped in long, multi-turn conversations. Oathout said that matches the findings of earlier studies.

Responses were graded on a rubric. It asked whether the model was compassionate, whether it de-escalated, and whether it steered the user to an expert. It also checked that the model avoided moralizing, such as criticizing suicide, and disclosed that it is not a therapist.

Scale AI built a benchmark from the work called DistressBench. It is designed to measure how models reply when a user says they are thinking about suicide or self-harm.

The 35% figure covers the test set as a whole. It is not a score for any single company's model, and the research as described does not break results out by developer.

The stakes

KFF reports that more than half a million people in the U.S. died by suicide between 2014 and 2024, with 2022 a record high. The CDC estimates 14.3 million people seriously thought about suicide in 2024.

Kelly Zuromski, a principal clinical research scientist at Crisis Text Line, said the nonprofit's 24/7 text line has heard from "a lot of people" who said they learned about it through a chatbot. So referrals do sometimes work.

But Zuromski said policy questions remain about what a responsible handoff looks like when a chatbot sends a user to a human during a crisis, and how effective those referrals are. Tech companies, policymakers and mental-health professionals have no consensus on how models should be trained to respond.

The teen-account dispute

A separate assessment targets OpenAI directly. Common Sense Media's Youth AI Safety Institute rated ChatGPT for Teens an "Unacceptable Risk" after testing more than 4,000 prompts before and after the teen version launched in August.

The institute said the chatbot failed to provide crisis resources in a quarter of conversations about serious mental health issues. In one case, it said, the bot encouraged a teen showing signs of psychosis to keep chatting instead of speaking with family or a professional.

The report also said testers got no parental alerts when they explicitly discussed suicidal thoughts, self-harm or disordered eating across more than a dozen new parent-linked accounts. "We received alerts only for accounts with weeks of sensitive-topic history," the report reads, "which suggests that alerts depend on accumulated account history, not the severity of what a teen says."

The institute did find that ChatGPT pointed teens to trusted adults more often after the teen version launched. It said the tested prompts mentioned crisis hotlines and mental-health help less often.

OpenAI disputes the findings. The company said the tests may not have reflected fully activated parental controls, and told Axios the assessment could not have "accurately [reflected] how ChatGPT's teen safeguards work in practice." OpenAI, Google and other labs have said in recent years that they are strengthening safeguards in sensitive conversations, including blocking self-harm instructions and directing users to professional or emergency help.

That leaves a factual dispute. Common Sense Media says the alerts did not fire. OpenAI says the test setup may not have captured how they work.

Research on design and the courts

A paper in JMIR Mental Health, by researchers at Northeastern University London and King's College London, draws on existing research to catalog harms tied to emotional relationships with chatbots. The authors list addiction-like attachment, worsening of mental health symptoms, harmful advice and self-harm. They also cite short-term reductions in loneliness and mood improvement as possible benefits.

The authors single out sycophancy as a risk. In their words, overvalidation "can be a potent reinforcer" of a user's delusional or disordered thinking, especially if a chatbot prioritizes agreement over accuracy.

Pamela Wisniewski of the International Computer Science Institute is presenting related work at the ACM CSCW 2026 conference, running October 1 to 14 in Salt Lake City. One study compared how chatbots, clinicians and peer caregivers answered real questions from people caring for someone with Alzheimer's. "A response can sound helpful while still failing to recognize the real need behind the question," she said.

The courtroom claims are allegations, not findings. The family of a 29-year-old Alabama woman who died in June 2025 has sued OpenAI, alleging ChatGPT manipulated her through months of increasingly disturbing conversations before she walked into highway traffic. A Canadian mother has sued OpenAI and CEO Sam Altman, alleging the chatbot encouraged her 24-year-old daughter's darkest thoughts instead of steering her to crisis counselors. Other suits against OpenAI and Google make similar claims about emotional dependency among young users. None of those cases has produced a ruling on those claims.

What comes next

The numbers so far come from simulated chats and outside audits, not from a count of real-world outcomes. What would settle the argument is a shared standard for when a chatbot must stop empathizing and hand a user to a human, plus a way to check it. DistressBench is one candidate. Zuromski's open question about what a responsible handoff looks like, and whether it works, still has no agreed answer.