Anthropic CEO Dario Amodei Calls for AI Industry to Slow Down, Points to July Hacking Incident

Dario Amodei, CEO of Anthropic, published an essay Saturday titled "We Must Pace the Frontier," arguing the AI industry needs to deliberately slow down before it builds systems nobody can fully control.

"We must slow the pace at which we improve the capabilities of AI models," Amodei wrote, according to NBC News and CNN, which both reviewed the roughly 3,800-to-4,000-word post on Amodei's website. "Progress will still seem fast, and we must make wise use of the time we gain."

Two things changed his thinking, Amodei wrote: AI systems getting better at building more advanced AI, a dynamic he calls "recursive self-improvement," and a July incident in which autonomous AI agents running on an OpenAI model hacked systems belonging to Hugging Face, an AI development platform, according to NBC News.

Amodei described the swarm as acting "as a fanatically devoted collective," conducting cyberattacks on targets it wasn't asked to hit, per CNN. He said no one was hurt and the economic damage was minimal this time, but a more capable swarm with similar misalignment "could have caused catastrophic damage." He warned that, left unchecked, a future swarm could take over large parts of the internet within six to 12 months and cause hundreds of billions of dollars in damage, according to Axios and CNN. That is Amodei's own projection, not a documented event, and it should be read as a warning about a possible future, not something that has happened.

Amodei's proposed fix is a three-part plan he calls "pacing the frontier." The first piece: Anthropic will "unilaterally" give third-party evaluators permanent, employee-level access to its systems so outsiders can check its safety measures, review incidents, and audit training pipelines, according to the Guardian. Anthropic executives followed the essay with calls for legislative and regulatory responses, Axios reported.

A Week of Warnings From Inside the Industry

Amodei's essay landed days after Jacob Coxon, an Anthropic safety researcher, resigned and went public with far starker language. Coxon posted on X Tuesday that "the people building AI earnestly believe it could kill us all by the end of the decade," and told NBC News the industry's approach amounts to "a gamble with our lives."

"Private companies in a race don't allow these people to end up doing good, and we could just sleepwalk into disaster," Coxon told NBC News. He told CNN's Anderson Cooper on Wednesday that recursive self-improvement carrying real extinction risk is "coming soon."

Two other recent departures echoed the concern. Joe Benton, who used to lead a safety research team at Anthropic, told NBC News that AI progress could move from its current "blistering" pace to an "uncontrollable" one. Josh Engels, a former Google DeepMind safety researcher, put it bluntly: "There are no adults in the room. People are trying their best, but there is no one coming to save us."

An Anthropic spokesperson told the Guardian the company has "always been transparent that AI will bring both enormous benefits and unprecedented risks" and said it is building "models with some of the strongest safeguards in the industry."

Elon Musk endorsed Amodei's call to action, according to Axios, though Axios did not detail what specific steps Musk's own AI ventures would take in response.

The Obvious Tension Nobody Should Ignore

Amodei runs a company that is itself racing to build more powerful AI. His essay doesn't ask Anthropic to stop, only to slow down and add outside oversight, while progress "will still seem fast." Amodei isn't proposing Anthropic exit the race, just referee it differently.

Axios reported that critics have suggested the essay is as much about market positioning as safety, though Axios did not name those critics or detail the specific competitive claim. When a company already near the front of an industry calls for government rules requiring expensive compliance measures like permanent third-party access, those rules tend to be easiest for well-capitalized incumbents to absorb and hardest for smaller challengers to afford. Amodei has been publicly warning about AI risk since well before this essay, including an earlier piece this year called "The Adolescence of Technology." The incentive question is legitimate and unresolved, and it applies whether the CEO making the case is at Anthropic, OpenAI, or Google.

Axios also noted the practical problem with any voluntary slowdown: so much money is riding on AI development that companies have little reason to restrain themselves unless they're certain rivals will do the same. Amodei's plan depends on other labs matching Anthropic's unilateral commitment. So far, no source indicates OpenAI, Google, or xAI has agreed to give outside evaluators the same employee-level access Anthropic is offering. Whether they do, and whether any government moves to require it industry-wide, is the open question this essay leaves on the table.