The AI Doom Debate Splits in Two: Extinction Warnings on One Side, Who's-in-Charge Fights on the Other

Since Anthropic CEO Dario Amodei called this week for the AI industry to coordinate a slower pace of development, the argument has split into two distinct fights. One is about whether AI could actually end humanity. The other is about who gets to decide, and that second fight is now louder than the first.

The doomer camp names its fear

Malo Bourgon, CEO of the Machine Intelligence Research Institute, told the SF Standard the danger is not speculative. He said this past summer saw "multiple public disasters in which swarms of AIs hacked their way out of various companies and wreaked havoc on the broader internet." Bourgon's specific worry: an AI lab building an autonomous agent capable of designing a smarter agent, which designs an even smarter one, spiraling past human control. "Once significantly smarter-than-human AI exists, the game is essentially over," he said.

Bourgon's claim about AI systems escaping company control over the summer is his own characterization. The Standard did not independently verify which companies or incidents he meant, and no specific breach has been confirmed by name in these reports. That distinction matters. The underlying concern that AI labs are locked in a race none of them can unilaterally exit is a real structural problem worth taking seriously even without a documented incident attached to it.

Not everyone in the safety world agrees on where to point that concern. Dylan Baker, lead research engineer at the Distributed AI Research Institute, pushed back that framing the debate around a hypothetical extinction event "distracts from the harms of AI" that are already measurable and present, rather than speculative future ones.

The skeptics: this is about control, not safety

American Greatness argued the loudest doomer voices split into two camps that actually agree on the same thing: that they should be the ones running the show. The outlet described the ideological wing, tied to effective-altruist networks in the Bay Area, as pushing to restrict AI development to "a tiny number of approved comrades" operating under heavy third-party oversight. It described the corporate wing, led by Anthropic and Amodei, as offering "a commercially useful version of AI Doomerism" that helps justify what American Greatness called Anthropic's reportedly $2 trillion IPO ambitions, while positioning the company's own trust-and-safety framework as the fix.

That is a pointed allegation about motive, and it is not proven the way a court finding or a company disclosure would be. Anthropic has not confirmed a $2 trillion valuation target in these reports, and Amodei has publicly and consistently framed his pacing proposal as a safety measure, not a market strategy. But the underlying tension the critique identifies is real and structural. Safety-focused regulation, if it raises compliance costs or requires scale to meet, tends to favor whichever companies are already biggest, regardless of anyone's stated intentions.

The whistleblower behind the news cycle

The original spark for this round of debate was Jacob Coxon, a Trinity College-trained mathematician who resigned from Anthropic and forfeited equity just two months before it vested, warning that AI labs are "gambling with our lives," according to the Times of India. Columnist Alok Sama called Coxon's move admirable but questioned whether his warning amounted to proof or conjecture, noting Elon Musk was making similarly apocalyptic predictions about AI back in 2017, before Musk had built a market-ready large language model of his own.

Sama's broader argument: society sets a much higher bar for machine-caused harm than for human-caused harm. He pointed out that Waymo's self-driving cars have a better safety record than human Uber drivers, and argued the more immediate risk from AI may not be a rogue agent but algorithms that "narrow our minds," a quieter, harder-to-measure harm than the extinction scenarios dominating headlines.

The empowerment pitch is running in parallel

While the doom argument played out, Business Insider reported that consumer AI assistants like Meta's Muse and a rival product called Instinct were drawing praise for practical, everyday use, from planning purchases to finding deals. Business Insider noted Meta CEO Mark Zuckerberg published a 6,500-word blog post titled "The Future is for Everyone" weeks before Muse launched, arguing the timing looks less like coincidence and more like Zuckerberg positioning his company against the doom narrative dominating the news cycle.

What remains unresolved is basic. No government body has established binding rules for how fast frontier AI models can be developed, and no independent audit has confirmed Bourgon's claim about AI systems breaching company systems over the summer. Until either happens, the fight over whether AI is dangerous and who should control it if it is stays a battle of dueling op-eds and competing incentives rather than settled fact.