Microsoft's New AI Benchmark Shows Top Agents Fail Repeat Tasks 75% of the Time

Microsoft has a message for every company rushing to put AI agents in charge of customer refunds, insurance claims, and booking changes: your demo is lying to you.

The company released an open-source testing framework called ThinkingBox on August 19, 2026, detailed in a Microsoft Command Line blog post by Principal Machine Learning Engineer Liang-Chun Tsai. The tool does something most AI evaluation methods don't bother with. It ignores what the agent says it did and checks the actual database instead.

That distinction matters.

The Gap Nobody Was Measuring

Microsoft built a companion benchmark, ThinkingBox-Bench, with 507 executable tasks spanning five business domains: retail, hospitality, auto insurance, neobank IT, and consulting support, according to the Nerra Network newsletter. Twelve proprietary and open-weight models ran each task 20 separate times.

The results, also posted to arXiv on August 20, 2026 as paper 2608.19741, expose what Microsoft calls a "discovery-reliability gap." The strongest model, identified by an analysis on co-r-e.com and the newsletter Pebblous as GPT-5.4, scored 65.36% on pass@1, meaning it nailed a task on a single try about two-thirds of the time. That's the number most leaderboards report.

But when the same model had to succeed on all 20 attempts at the same task, its score collapsed to 25.25%. It managed to succeed at least once on 91.12% of tasks, according to Pebblous, which means there's almost always a working path through the problem. The model just can't reliably find it twice in a row.

Counted out in raw terms, per the co-r-e.com analysis of the paper: GPT-5.4 passed all 20 attempts on 128 of 507 tasks, and never passed once on 45. That leaves 334 tasks, roughly two-thirds of the benchmark, sitting in the middle. Sometimes it works, sometimes it doesn't, same model, same tools, same instructions.

Why the Failures Are Invisible

Of the tens of thousands of failed trials Microsoft logged, 80.88% ended cleanly, with the agent invoking a tool that actually changed some record in the system, according to the co-r-e.com review of the paper's data. The agent didn't crash. It didn't throw an error. It confirmed the job was done and moved on. It just did the wrong thing.

Microsoft's own blog post lays out a concrete example: a customer asks a travel agent to add a quiet-room preference to a hotel booking. The agent checks the reservation, tells the customer it's handled. The transcript reads like a success. But the booking's special_requests field stays empty. Nobody finds out until check-in.

A grader reading transcripts or tool-call logs would pass that run. ThinkingBox catches it because it inspects the actual database state left behind, using what Microsoft calls executable assertions rather than a model just judging whether the response sounded right.

What the Numbers Don't Prove

ThinkingBox tests scripted, simulated business workflows in a sandbox, not live production traffic. A separate dataset from Aident.ai, drawn from 4,187 real production agent tool calls between August 4 and August 16, 2026, found a 78.3% terminal success rate for individual executions. That's a different measurement entirely. Aident's own writeup is explicit that it tracks whether an external action reached a successful terminal state, not whether the agent satisfied the user's full task or chose the right action in the first place. The two numbers aren't directly comparable, and neither should be read as a universal reliability score for AI agents generally.

It's also fair to note the strongest models aren't failing randomly or catastrophically. A 91% pass@20 rate means these systems almost always contain a working solution somewhere in their attempt space. The problem Microsoft's data highlights is consistency on repeat, not fundamental incompetence.

For any business currently deciding whether an agent is ready to handle refunds, claims, or account changes without a human checking behind it, the practical takeaway from Microsoft's data is clear: a single successful demo tells you almost nothing about whether the same agent will get it right the tenth time, or the twentieth.

What Happens Next

Microsoft published both the sandbox and the benchmark on GitHub, a deliberate choice to let other companies and researchers stress-test their own agents against the same standard rather than take Microsoft's word for it. The newsletter Nerra Network is already urging engineering teams to run their production evaluation harnesses against ThinkingBox to see where the gaps show up.

The open question is whether any lab closes the discovery-reliability gap in the near term, or whether "sometimes works, sometimes doesn't" becomes the accepted price of putting AI agents into workflows that used to require a human who got it right every time.