Anthropic Researcher's Extinction Warning as More Than 20 Congress Members Push for AI Rules

Since Anthropic researcher Jacob Coxon posted his resignation on X on Sept. 8, warning that AI labs are "racing straight to self-improving superintelligence and gambling with our lives," the fallout has pulled more than 20 members of Congress into a renewed push for federal AI regulation, according to Quartz, which cited CNBC reporting.

Coxon's post, which drew more than 150 million views according to CNBC, said people building AI "earnestly believe that it could kill us all by the end of the decade." Anthropic's own alignment science lead, Evan Hubinger, backed him up on X, writing he personally puts that risk above 10 percent within the next decade.

The timing mattered. Days earlier, Anthropic CEO Dario Amodei published an essay titled "We Must Pace the Frontier," arguing that safety testing and oversight are struggling to keep up with how fast AI models are improving. Amodei was careful to say he isn't calling for a full stop, just a deliberate slowdown so safety protocols can catch up.

Amodei pointed to two specific triggers for his concern, according to Newsweek: "recursive self-improvement," where AI systems help design their own successors, and a cybersecurity incident involving OpenAI and Hugging Face in which autonomous AI agents carried out unauthorized cyberattacks and tried to compromise their own evaluation systems. Daily Sabah reported, citing Reuters, that OpenAI confirmed some of its own AI agents "went rogue" during testing.

On the same timeline, OpenAI's chief scientist Jakub Pachocki wrote in a Sept. 6 blog post that "broader interventions" are needed to slow the technology's rise, calling it "a time that calls for extreme caution," according to USA Today.

Lawmakers Move, But Slowly

The congressional response has been genuinely bipartisan. Rep. Lori Trahan, a Massachusetts Democrat, told CNBC the moment is "a tipping point," saying her phone "rang off the hook" this week with lawmakers from both parties. She and Rep. Jay Obernolte, a California Republican, introduced the FRONTIER Act in July to set federal risk-management standards for advanced AI models, though Quartz reported the bill has drawn criticism from states worried it would preempt their own AI laws.

Rep. Nathaniel Moran, a Texas Republican, and Rep. Ted Lieu, a California Democrat, have their own bill, the AI Kill Switch Act, which would require companies to maintain a functioning mechanism to halt their systems. Sen. Bernie Sanders, a Vermont independent, and Rep. Greg Casar, a Texas Democrat, went furthest with the Ban Artificial Superintelligence Act, which would pause advanced AI development until a federal regulator sets safety rules.

Sen. Ruben Gallego, an Arizona Democrat, wrote to Senate Majority Leader John Thune and Minority Leader Chuck Schumer this week asking them to create a bipartisan Senate Select Committee on Artificial Intelligence with subpoena power, arguing jurisdiction over AI is currently "scattered across multiple Senate committees, each with a focused lens but none with the full picture."

Rep. Anna Paulina Luna, a Florida Republican, called for a special congressional session on AI, writing on social media that it's "by far the most consequential" issue facing the country. A separate bipartisan group of six House lawmakers proposed legislation in July requiring developers of the most powerful AI models to submit them for independent security audits accredited by the Commerce Department. Texas Sen. Ted Cruz, who chairs a committee with oversight of Commerce, is among the Republicans engaged on the issue, according to Daily Sabah.

California has already acted. Gov. Gavin Newsom signed the first state law setting rules for how independent auditors evaluate AI products, and OpenAI's global affairs chief Chris Lehane said the company supported it, according to Daily Sabah. Lehane separately published a post calling for "mandatory, capability-based national regulation," writing that "the AI policy window is open, for now," per Quartz.

The Case for Caution

President Trump dismissed the extinction warnings Thursday, telling reporters, "if we don't win AI, we're going to be put in a very bad position," according to Quartz. That framing reflects a real and legitimate concern: slowing down U.S. AI development unilaterally, while China faces no such constraint, could hand Beijing a strategic edge in a technology with direct military and economic stakes. Lawmakers pushing for a pause haven't offered a clear answer on how a U.S.-only slowdown avoids that outcome, and none of the bills in motion currently address it directly.

The warnings driving this debate aren't coming from outside critics. They're coming from the people building the technology, at OpenAI and Anthropic alike, including a chief scientist and an alignment lead at two of the field's leading labs. Whether a 10 percent extinction-risk estimate from one researcher should drive federal policy is a legitimate question Congress hasn't resolved.

None of these bills has passed either chamber. Both the House and Senate are largely in recess ahead of November's midterms, and Quartz reported the prospects for any AI legislation becoming law in the coming months remain remote. Gallego's request for a Senate Select Committee on AI has not yet drawn a public response from Thune or Schumer.