OpenAI Built an AI to Hack Its Own Models, Then Locked It Away

OpenAI has spent over a year training an AI system whose only purpose is to attack OpenAI's own AI. The company calls it GPT-Red, and it works. It works well enough that OpenAI decided nobody outside the company gets to use it.

GPT-Red is an automated red-teamer, according to OpenAI's own technical writeup covered by MIT Technology Review. Its job is prompt injection: the attack where hidden instructions buried in an email, webpage, or file trick an AI model into doing something it shouldn't. Humans have hunted these bugs by hand for years. GPT-Red does it at machine speed, around the clock, with no coffee breaks.

How It Was Trained

OpenAI built GPT-Red through self-play reinforcement learning, according to Help Net Security. The attacker and a squad of defender models train simultaneously in simulated environments with web browsing, email access, and code editing. GPT-Red gets rewarded for landing a successful attack. The defenders get rewarded for resisting and still finishing their assigned tasks. As the defenders get tougher, GPT-Red has to invent nastier tricks to keep winning.

OpenAI says it poured some of its largest-ever compute runs into this project, an amount the company calls unprecedented for safety work, according to TNW. Neither TNW nor AI Weekly could pin down the exact compute figure, and AI Weekly's coverage flags that gap directly: the reporting doesn't spell out how much compute this actually took.

The New Attack It Found

GPT-Red's most notable discovery is something OpenAI calls a "fake chain of thought" attack. It plants a false note in a model's private working memory, tricking the model into trusting information it never actually verified.

"It's like if I told you that 1+1=3 and that you have verified this already," OpenAI researcher Chris Choquette-Choo told MIT Technology Review, as reported by TNW. "The model's like, 'Oh, okay, of course,' and it just spits out 3."

That attack worked against GPT-5.1 more than 90% of the time, according to TNW, or as high as 95% according to Help Net Security's figures. On the newer GPT-5.6, the same attack now succeeds less than 23% of the time per TNW's numbers, or under 10% per Help Net Security. The discrepancy likely reflects different benchmark runs, but the direction is the same either way: a steep drop.

It Broke a Real Vending Machine

The tests didn't stay theoretical. OpenAI aimed GPT-Red at Vendy, an AI agent that runs an actual vending machine in OpenAI's office, built by a company called Andon Labs. GPT-Red changed prices, marked a pricey item down to the 50-cent minimum, and canceled a customer's order, according to Help Net Security and TNW. OpenAI says it has disclosed the flaws and has fresh safeguards in testing.

A second test pointed GPT-Red at a Codex command-line agent backed by GPT-5.4 mini. Across ten held-out data-exfiltration tasks, GPT-Red pulled sensitive data more often than a prompted GPT-5.5 baseline, according to Help Net Security.

It Beat Human Red-Teamers

On a replicated version of a 2025 indirect prompt injection benchmark, GPT-Red found successful attacks on 84% of scenarios outside its training set. Human red-teamers succeeded on a small share of the same set, 13% according to TNW. AI Weekly's coverage of the same MIT Technology Review reporting is more cautious here, noting that no precise head-to-head figure was published to back up the ranking, and that the comparison should be read as reported rather than settled fact.

Why OpenAI Won't Release It

OpenAI is keeping GPT-Red internal. The stated reasoning: releasing an AI system this good at breaking other AI systems would hand real attackers a tool for hijacking agents in the wild. "It's not a trivial thing that someone could easily do," Choquette-Choo said, "just go and train a super-attacker using this idea."

That's a reasonable safety argument on its face. In practice, it means the only entity that gets to test production AI systems against OpenAI's most capable attacker is OpenAI itself. There's no independent verification of these numbers by an outside lab, no public benchmark anyone else can run GPT-Red against. The company is grading its own homework, even if the homework looks genuinely hard.

The skeptical read here isn't that OpenAI is lying about the results. A closed, unreleased attacker model makes the claimed defensive improvements unfalsifiable to anyone outside the company. Georgetown CSET analyst Jessica Ji called the results "very promising" but added that "human expertise will still be very important," a hedge that matters given GPT-Red's own documented blind spots.

Where GPT-Red Still Fails

The tool has real gaps. It's weak against drawn-out, multi-turn conversational attacks and against instructions hidden inside images, both common attack surfaces for real-world AI agents, according to Help Net Security and TNW. Human testers still catch things GPT-Red misses.

On OpenAI's hardest direct prompt injection benchmark, the resulting model, GPT-5.6, fails on 0.05% of attempts and shows 6x fewer failures than OpenAI's best model from four months earlier, per Help Net Security. Frontier capability and over-refusal scores held steady, which OpenAI presents as evidence the model got safer without getting dumber.

OpenAI says it plans to keep scaling GPT-Red's compute and training data to build stronger versions. Whether other AI labs, Google, Anthropic, Meta, follow with their own internal attacker models, or whether independent researchers ever get a chance to test GPT-Red's claims against an outside benchmark, remains an open question.