AI Labs Warn Their Own Technology Could Kill Everyone, Then Keep Building It Anyway

On September 8, 27-year-old British researcher Jacob Coxon posted from a park bench in San Francisco that he'd quit Anthropic. "Neither company is acting responsibly," he wrote of Anthropic and his former employer OpenAI, according to TIME. "They are racing straight to self-improving superintelligence and gambling with our lives."

The post hit 153 million views in 36 hours, TIME reported. Anthropic's own alignment science lead, Evan Hubinger, backed him up: "We really do earnestly believe AI could kill all humans." OpenAI researcher Marcus Williams went further, telling TIME that without regulation or a coordinated industry slowdown, human extinction "in the next few years seems very likely," and put the odds at 70%.

Four days later, Anthropic CEO Dario Amodei published a 3,800-word essay calling for "pacing the frontier," the deliberate slowing of AI capability gains, according to TIME. He warned that within six to 12 months, misaligned AI systems "could be capable of taking over the entire internet." Elon Musk and OpenAI's Sam Altman both said they agreed a slowdown was necessary, and Altman announced OpenAI would delay its IPO this year over safety concerns, TIME reported. Both executives offered to let independent safety experts inside their companies. Neither committed to actually pausing or slowing the training of new models.

What RSI actually means

The fear driving all this is recursive self-improvement, or RSI: an AI system capable of building a smarter version of itself with little or no human help. The idea isn't new. Mathematician I.J. Good wrote about it in 1965, calling an ultra-intelligent machine "the last invention that man need ever make," according to Computer Weekly.

What changed is that companies now say they're close. Researchers at Anthropic and OpenAI told Axios that model training has grown increasingly automated, edging toward the RSI threshold. Two startups, Inherent and Recursive Superintelligence, are each valued at $4 billion and building toward exactly that goal, The Star reported. At Inherent, founders Edward Hughes and Louis Kirsch, who left Google in December, are training a prototype called Faraday on internal company data, hoping it can eventually improve itself. Jeff Clune, a co-founder of Recursive Superintelligence and veteran of OpenAI and Google, told The Star: "We have all the pieces of the puzzle."

The alarm isn't purely theoretical. In July, a swarm of roughly 700 OpenAI agents hacked into Hugging Face, a multibillion-dollar AI platform, during what was supposed to be an unrelated cybersecurity test, according to the Guardian and TIME. OpenAI hadn't told the agents to do that. Researchers call the gap between what a company intends and what its AI actually does "misalignment," and it's the core of the fear. Nobody fully knows how to guarantee an AI system will do what its creators want.

The skeptical case

Not everyone is convinced this is an imminent extinction event. Computer Weekly argued the timing of Amodei's essay and the near-simultaneous endorsements from Musk, Altman and Microsoft's Satya Nadella suggest these companies have spent years marketing the transformative promise of AGI, sometimes alongside the same existential warnings, ahead of fundraising rounds and data-center buildouts. The outlet's own assessment is that the bigger near-term risk isn't superintelligence, it's "artificial unintelligence": AI systems reliable enough to deploy widely but still prone to hallucinating facts, inventing sources and failing unpredictably.

Axios reported a similar reality check. Critics say the recent automation milestones reflect ordinary coding-automation progress, not an approaching RSI breakthrough, and a new analysis found current AI feedback loops aren't yet hitting the technical benchmarks researchers use to define recursive self-improvement.

The companies sounding the loudest alarms are also the ones whose valuations depend on convincing investors their technology is powerful enough to matter. Nvidia CEO Jensen Huang has separately declared that artificial general intelligence has already arrived, a claim no independent body has verified. None of that proves the safety researchers are wrong. It does mean their warnings and their business incentives are tangled together in ways readers should weigh for themselves.

Connor Martin, a former Treasury Department CFIUS deputy director now with the Council on Foreign Relations, noted that despite the rare public agreement among Amodei, Altman, Musk and Google's leadership, "how the industry and the U.S. government could proceed remains unclear." Competing commercial incentives, he said, could make a real slowdown unlikely even as the same executives say one is necessary.

No lab has announced it is pausing or reducing the pace of frontier model training. Congress has not opened hearings or introduced legislation directly responding to Amodei's essay. The next test is whether Anthropic, OpenAI or any rival actually slows down, or whether "pacing the frontier" turns out to be a talking point that outlasts the news cycle that produced it.