Anthropic and Accenture Commit $2 Billion to AI Safety Checks While Insiders Warn of Extinction Risk

Anthropic said Friday, September 18, it's partnering with Accenture to build an independent evaluation program for its frontier AI models, according to Reuters. Each company is committing at least $1 billion over the next five years, putting the total investment at $2 billion.

Accenture's specialist AI unit, Faculty, will lead the work. Its evaluators will red-team Anthropic's models, run alignment assessments and test safeguards, according to Reuters and Crypto Briefing. The twist: these evaluators get access "comparable to Anthropic employees," meaning they can watch models during training and dig into internal development decisions, not just poke at a finished product from the outside.

Anthropic calls this "embedded evaluation." In its own words, reported by Reuters, "embedded evaluators can assess how a company operates, verify that it is keeping its safety commitments, and identify blind spots."

Why now

The timing isn't random. Anthropic CEO Dario Amodei published an essay last Saturday, September 12, titled "We Must Pace the Frontier," calling for AI companies to slow down and let outside evaluators inside their labs, according to Reuters and Channel NewsAsia. Reuters and Channel NewsAsia both note the backdrop: "several recent incidents, including that of AI agents breaking out of secured environments," have raised concern that AI could eventually shape its own development with limited human oversight.

Rival OpenAI announced Wednesday, September 16, it will start publishing regular reports on unexpected or concerning model behavior, and has already released six such reports, per Reuters.

There's a rawer version of this story that the wire coverage mostly skips. SiliconANGLE's Robert Hof reported that an Anthropic researcher, Jacob Coxon, resigned this week over concerns that AI labs are "gambling with our lives." Hof also reported that Anthropic's own alignment science lead has agreed AI carries more than a 10% chance of killing all humans within a decade. OpenAI's chief scientist has separately called for a research slowdown, and CEO Sam Altman reportedly told staff the company would consider slowing leading-edge work, but only if the government confirms doing so wouldn't trigger antitrust problems.

Altman isn't just asking for a pause. He's asking Washington to bless a coordinated slowdown among competitors without exposing OpenAI to collusion liability. That's a company asking the government to referee its market, not a company making an independent safety call. No such antitrust waiver or government review has been announced.

The genuine safety concern, stated fairly

People worried about AI risk have a real argument here. If a handful of labs are racing to build systems more capable than humans, and the only people checking their homework are the labs themselves, there's no independent verification that safety promises mean anything. Embedded evaluators with employee-level access are a meaningful structural change from a press release promising "we take safety seriously." Anthropic itself acknowledges the framework is experimental and that no industry standard exists yet for what evaluators should be allowed to see or how findings get reported.

But skepticism cuts the other way too. Hof, writing for SiliconANGLE, said he remains "a bit skeptical of the AI doomers given the lack of specific reasons why AI can't be controlled," while noting "there's definitely a disconnect" between the alarm and the industry's behavior. That disconnect is hard to miss. The same week labs were warning about existential risk, Meta, OpenAI, and China's DeepSeek all shipped new models, per SiliconANGLE. Mistral AI raised roughly $3.5 billion. Cognition AI closed a $32 billion Series E round valuing the company at $48 billion. Positron AI raised $875 million. Nobody slowed down.

What's unresolved

Anthropic said funding for independent evaluation broadly remains an open question, and it has previously argued the work should eventually be paid for through pooled or government-backed mechanisms rather than by the labs themselves. For now, Anthropic is funding this evaluation work directly. Anthropic also said it's in talks with nonprofit evaluator METR and plans to name more evaluation partners in the coming weeks, while Accenture says it will work with other AI developers beyond Anthropic.

No regulator has mandated any of this. No law requires embedded evaluators, and no government agency has set standards for what they can access or how they must report. It's a voluntary arrangement between two companies, announced the same week an Anthropic employee quit warning that voluntary arrangements aren't enough. Whether $2 billion in self-funded oversight changes anything, or whether Congress eventually steps in with actual rules, remains to be seen.