Microsoft Benchmark Finds Gemini, Claude and GPT Models Lose Up to Half Their Accuracy on Long AI Tasks

Microsoft Research has published a benchmark that puts a hard number on a problem AI users have complained about for months: chatbots and AI agents fall apart the longer you let them work unsupervised.

The benchmark is called DELEGATE-52. It tests how well AI models preserve the content and accuracy of documents and code across long chains of delegated, multi-step work, spanning 52 professional domains, according to Microsoft Research.

Most AI benchmarks test short, isolated tasks, a single question or a five-minute job. DELEGATE-52 was built to measure something those tests miss entirely: what happens when a model has to iteratively edit, transform, and reconstruct work over dozens of steps, the way a real employee or agentic workflow would operate.

The Numbers

According to Microsoft Research, average document fidelity fell by about 25% after just 20 delegated iterations. Widen the lens across every model and every domain tested, and that degradation figure climbs to roughly 50%.

Catastrophic corruption, defined in the study as fidelity scores dropping to 80% or lower, showed up in more than 80% of the model-domain combinations Microsoft tested.

The models tested were not lightweight or outdated systems. Microsoft's benchmark ran on Gemini 3.1 Pro, Claude 4.6 Opus, and GPT 5.4, three of the most capable AI systems on the market. All three degraded significantly over extended runs.

Gemini 3.1 Pro, specifically, was rated ready for reliable use in only 11 of the 52 professional domains DELEGATE-52 covers, per Microsoft Research's findings.

One Bright Spot: Code

Python workflows were the exception. They held up with less than 1% degradation across extended runs, according to the study. Code is far more resistant to the kind of drift and corruption that hits prose, spreadsheets, and other document formats.

That gap matters. It suggests the reliability problem isn't universal to AI reasoning itself. It's specific to how these models handle unstructured, iterative document work, the exact kind of work companies are being pitched to automate with AI agents.

More Tools, More Problems

Giving models more capability made things worse, not better. When models were given agentic tool access, things like file read/write permissions and code execution, document fidelity degradation increased by an additional 6% on average, according to Microsoft Research. The industry's answer to reliability concerns has largely been to hand AI agents more autonomy and more tools. This benchmark suggests that's making the underlying problem worse.

Microsoft's researchers also tested whether the usual fixes, verification steps and orchestration layers that are supposed to keep agents on track, actually solve the issue. They don't. The study concluded that reliable long-horizon delegation is "a fundamental challenge, not merely an engineering inconvenience that better prompting can fix."

The Marketing Gap

A fair defender of the current AI industry would point out that these models are improving rapidly, that short-horizon benchmarks still have real value for the vast majority of everyday use cases, like answering a single question or drafting one email, and that guardrail technology is still young and will keep getting better. Most consumer AI use isn't a 52-domain, 20-step delegated workflow.

But the marketing numbers that companies use to sell these products, and that enterprises use to evaluate them before writing big contracts, come almost entirely from those short-horizon tests. A model that scores 95% on a five-step task can score 50% on a twenty-step version of essentially the same job, based on Microsoft's data. That's a massive gap between the number on the sales deck and what happens when a company actually deploys an agent for a real, extended workflow.

Microsoft Research notes that parallel industry efforts, including frameworks like AgentRx and monitoring tools such as SentinelBench, have emerged specifically to address reliability in extended agent operations. Their existence is itself evidence that the problem is widely recognized inside the industry, not a fringe complaint.

No AI vendor, including Google, Anthropic, or OpenAI, has publicly committed to adopting DELEGATE-52 or a comparable long-horizon benchmark as a standard for advertised performance claims. Until one does, enterprises signing contracts for AI agents are left comparing vendor-supplied short-task scores against a real-world failure rate Microsoft's own research puts above 80%.