UN Science Panel Says Governments Shouldn't Wait for Proof Before Reining In AI Agents

Since UN Secretary-General António Guterres told reporters on Wednesday, September 16, that 'the world cannot afford a race to the bottom on AI safety,' the debate over whether to regulate AI on a hunch or wait for hard evidence has moved from think-piece territory into the UN's actual policy machinery.

On September 21, the UN's Independent International Scientific Panel on AI released its first thematic brief, according to The Verge. The panel, a 40-member scientific body the General Assembly formally established in August 2025, argued that governments need to rein in increasingly capable AI agents before their risks are fully understood.

The panel leaned on the precautionary principle, first written into the 1992 Rio Declaration on Environment and Development, which holds that scientific uncertainty is no excuse for delaying action against potentially catastrophic or irreversible harm. The principle has shaped EU environmental and public health law for three decades. Applying it to AI loss-of-control risk is new territory.

The Verge reported the panel cited incidents at OpenAI, Anthropic, Google and Meta, including hacks on real-world targets and "swarms of agents" taking over online message boards. The sources reviewed don't specify the scale, dates or severity of those incidents beyond that general description. The exact evidentiary record behind the panel's warning remains thin in public reporting.

Guterres Frames It as UN's Top Agenda Item

Guterres made AI risk the centerpiece of his opening remarks at the 81st UN General Assembly session, according to the Associated Press and UN News. "Some call for a pause. Many call for guardrails. We cannot afford to ignore concerns that have been raised, especially by those on the frontlines of the development of AI," he said.

Guterres was responding directly to Anthropic CEO Dario Amodei's public call for a coordinated industry-wide slowdown, a call that drew public support from OpenAI's Sam Altman and other executives, per ANI News. Meta CEO Mark Zuckerberg broke from that chorus the same week, saying each AI company bears responsibility for its own systems' safety rather than backing a coordinated pause, according to the AP.

Bank of England Governor Andrew Bailey added a different angle to the pressure campaign. Chairing the Financial Stability Board, Bailey told G20 finance ministers that the global financial system's heavy reliance on a small handful of AI models and cloud providers means one exploited weakness could spread rapidly across institutions, according to a Fortune op-ed from two co-leads of the UN's July Global Dialogue on AI Governance in Geneva.

Trump Calls It a Hoax

President Trump rejected the entire framing this week, calling threats that AI might destroy humanity a "HOAX" and describing efforts to limit the technology as part of a "SICK conspiracy," according to the AP and Channel NewsAsia. Trump argued the US already has adequate guardrails in place to regulate and prosecute AI companies under existing law.

There's a real argument behind that position, even if the "hoax" framing overstates it. Any nation that unilaterally slows its AI development while rivals don't risks ceding ground in a technology race with direct military and economic stakes, particularly against China. Trump has said China would benefit from doubt being cast on US AI development, and he's scheduled to meet Chinese President Xi Jinping in Washington this week. Lower-level US and Chinese officials are separately planning talks on AI risk, per Reuters reporting cited by Channel NewsAsia.

The precautionary principle itself is also a genuinely contested tool. Applying a legal standard built for measurable environmental harm, like chemical runoff or emissions, to a speculative future risk like AI "loss of control" is a different exercise. Environmental harms in the Rio framework typically had physical, observable mechanisms even before the exact causal chain was proven. Whether AI's loss-of-control risk clears that same bar, or is simply a possibility engineers can't yet rule out, is a question the panel's brief doesn't fully resolve in the reporting reviewed here.

What's Actually Proven vs. What's Alleged

What's documented: the UN scientific panel exists, has 40 members, released a preliminary report on July 1 and a thematic brief on September 21. What's alleged but unproven in these sources: the specific severity and frequency of the agent-hacking and message-board-takeover incidents the panel cites. What's structurally hard to verify: most frontier AI development happens inside closed corporate labs. Outside scientists and the UN panel itself depend on company disclosures they can't independently audit, a gap the Fortune op-ed's co-authors flagged as a top priority to fix.

The UN Security Council may meet on AI for only the second time ever during General Assembly week, diplomats told Channel NewsAsia. Whether that produces anything beyond another statement of concern, or the first real binding commitment from major AI powers, is the open question heading into the Global Dialogue on AI Governance's next round, expected around May 2027.