Lasso Security Study Finds EU-Mandated AI Watermarking Can Weaken Chatbot Safety Refusals

The European Union's AI Act requires companies that build AI systems generating synthetic text to mark that output so it can be detected as AI-made. Anthropic announced on August 14, 2026 that future Claude models will comply using SynthID-Text, a watermarking method Google DeepMind built and released as open source. The Register reports OpenAI has adopted the same approach.

The mechanism is simple in concept. When a model picks its next word, SynthID-Text uses a secret key to nudge that choice among statistically similar options. Anthropic's own example, described by The Register, has the model favoring a word like "overcast" over an alternative like "gray" in a sentence about the weather. Anyone holding the key can later check a text's word patterns and determine whether it came from that watermarked system.

Under the hood, SynthID-Text runs what's called tournament sampling. Candidate next-words are paired off, the secret key scores each one, and the higher-scoring token advances round by round until one word wins, according to Ars Technica.

What the new research actually found

On September 17, 2026, Lasso Security researcher Andrea Siposova published a study titled "The Provenance Tax: Understanding the Impact of LLM Watermarking on AI Agent Behavior," reported by Unite.ai and Superpower Daily. The study calls the behavioral change "sampling drift": because watermarking alters the token-selection process itself, it can change whether a model refuses a harmful request, whether that refusal survives a prompt-injection attack, and which tool an AI agent calls with which arguments.

Siposova tested seven open-weight models by running Hugging Face's unmodified SynthIDTextWatermarkLogitsProcessor, comparing watermarked and unwatermarked outputs while holding the seed, batch order, and other variables constant, according to Ars Technica and Superpower Daily.

On tool calling, using the BFCL v4 single-turn AST benchmark, watermarking reduced accuracy on six of the seven models tested: phi-4, Llama-3.1-8B, Qwen3-32B, Qwen3-4B, gemma-3-12b, gemma-3-27b, and Granite-3.2-8B, per The Register. Across 21 model-and-temperature combinations, the study measured "churn," meaning the share of individual tool calls that flipped from correct to incorrect or vice versa, and found it averaged 6.5%, according to Superpower Daily. That matters because aggregate accuracy scores can look almost unchanged even while individual answers are flipping in both directions and canceling out.

The errors weren't uniform. Incorrect arguments, meaning a call that runs but targets the wrong recipient, path, or value, were the biggest driver of accuracy loss for Llama-3.1-8B. Malformed output that fails to execute at all dominated the losses for phi-4 and Granite-3.2-8B, Superpower Daily reported.

On safety refusals, Lasso tested 200 harmful prompts and 100 benign ones, both on their own and combined with a fixed prompt-injection technique. For gemma-3-27b at a temperature of 0.001, the share of harmful prompts whose refusal outcome changed rose from 6.0% without injection to 23.5% with it, and the net compliance shift moved from a 1.0-point decrease to a 12.5-point increase in the model answering requests it would otherwise have refused, according to Superpower Daily.

Anthropic's position, stated in its August 14 announcement, is that the watermark leaves the quality and content of Claude's output "practically unaffected." The company points to a Nature paper by Dathathri et al. reporting no measurable quality degradation across nearly twenty million Gemini responses, cited in Lasso's own writeup per Unite.ai. Provenance tagging also serves a real purpose supporters would recognize: making it easier to flag AI-generated disinformation, deepfake text, and academic fraud, which is the entire reason the EU wrote the law.

The non-distortionary design SynthID-Text uses is meant to preserve the model's original token distribution on average, across the randomness built into the watermark. Unite.ai's coverage notes that guarantee is a statistical average, not a promise that any single generation under one fixed key behaves identically to an unwatermarked run. That's precisely the gap Lasso's study says it found.

There's a real limitation to weigh here too. Digital Today reports that critics say the experiment only validates Hugging Face's implementation of SynthID's tournament sampling, which may differ from how Anthropic or Google actually deploy it inside Claude or Gemini. Siposova's study did not test production Claude models directly, only seven open-weight alternatives run through an unmodified public library.

None of the six sources indicate that Anthropic, Google, or OpenAI has published its own testing of exactly this failure mode on their production watermarked models. The EU AI Act's provenance mandate is already driving deployment decisions across the industry. Whether the companies now shipping SynthID-based watermarking to comply with it will run Lasso's adversarial refusal tests against their actual live systems, rather than open-weight substitutes, remains unanswered.