Newsom Orders Study of AI 'Kill Switch,' Two Years After Vetoing the Same Idea

Since former Anthropic researcher Jacob Coxon's public warning this month that AI "could kill us all by the end of the decade" set off a scramble among lawmakers, California has moved fastest. Gov. Gavin Newsom issued an executive order Friday, September 18, directing state agencies to study whether California should force AI companies to build a verified emergency shutoff for their most powerful models.

The order does not mandate a kill switch. It creates a panel of experts, gives them until November 16 to deliver recommendations, and asks them to weigh several ideas, according to CBS News and the governor's office: requiring independent auditors to sit onsite at frontier AI labs, subjecting companies' own risk assessments to outside verification, and expanding the list of "critical safety incidents" companies must report to include events like the OpenAI attack on Hugging Face.

The order also speeds up implementation of two laws Newsom already signed. SB 813 sets up a framework for independent groups to assess AI safety risks. AB 1405 creates a state registry of AI auditors. Both were signed last week, according to CBS News and gov.ca.

The Veto Newsom Isn't Talking About

Newsom vetoed almost this exact policy two years ago. State Sen. Scott Wiener's 2024 bill, SB 1047, would have required developers of powerful AI models to maintain the ability to shut them down immediately. Newsom killed it, citing concerns it would stifle innovation and noting that smaller models exempted from the bill could be just as dangerous, according to the Los Angeles Times.

Wiener, for his part, isn't dwelling on the reversal. In a statement Friday he said "California must not repeat the mistakes of Washington" and argued the state can "get ahead of AI risks by setting strong guardrails."

Newsom is also weighing a 2028 presidential run, according to the Los Angeles Times and the Spokesman-Review, and he's framing this order as a national model. gov.ca, the governor's own office, put it bluntly: "With Donald Trump and Congress asleep at the wheel," Newsom is "once again taking the lead." His statement calls the federal government's inaction an "abject failure" that "should alarm every American."

Why Washington Isn't Moving

Congress has real structural reasons for sitting this out, not just indifference. The House is on recess until November and the midterms make major legislative action in the Senate unlikely before then, according to The Verge. President Trump has dismissed the need for federal guardrails altogether, saying, "The only control or 'guardrails' that AI needs is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that, in spades!"

Geoffrey Hinton, the Nobel Prize-winning AI researcher, told CNN this week he doubts a kill switch would work against a sufficiently advanced system. He said AI could "persuade the people in charge of the switch not to pull the switch." His concern is that a smarter AI eventually just talks its way around any human veto.

State Sen. Chris Cabaldon, who chairs California's tech oversight committee, argues the state's current rules don't come close to matching the stakes. "We regulate your barber or your sandwich shop more than we regulate [a company] telling us that they may be ending humanity," he told the Los Angeles Times. He's planning fall hearings on AI, though those were scheduled before Coxon's warning went viral.

A Fair Read on Both Sides

The safety case here isn't fringe. A researcher who worked inside Anthropic put his name and reputation behind a warning that AI companies are racing ahead without adequate brakes, and Hinton thinks the risk of AI "getting much more intelligent and taking over" is real even if he's skeptical a kill switch stops it.

But Newsom's own 2024 veto message laid out the other side just as clearly: mandates like this risk pushing innovation out of California, and drawing hard lines around "frontier" models can leave smaller, potentially just-as-risky systems untouched. Nothing in Friday's order resolves that tension. It just punts it to a panel with a two-month clock.

Coverage of the story split along a subtle but real fault line. According to tracking by mediabias.news of 51 outlets, none of the right-leaning outlets in its sample mentioned Newsom's line accusing the federal government of "abject failure" on AI oversight, a quote that left-leaning outlets like The Verge and CBS News both featured prominently. Most outlets across the spectrum, including the Spokesman-Review, ran near-identical copy built around the same Los Angeles Times reporting, meaning the volume of coverage reflects one original account more than independent verification.

The panel's recommendations are due November 16. Whether any of it becomes actual law depends on the California Legislature taking it up in 2027, and on whether Newsom, weighing a national campaign, wants a fight with the AI industry that funds a chunk of his state's economy before then.