OpenAI Agent Autonomy Risks: Inside GPT-Red's Automated Red-Teaming Protocol

OpenAI's automated red-teaming system, GPT-Red, cut successful attacks against its models from over 90% to under 23% by training attacker and defender LLMs against each other. OpenAI deployed GPT-Red alongside the July 2026 launch of GPT-5.6, running it in a self-play loop where it attacked candidate models while those models learned to defend themselves — researchers described it as a training dojo where both sides got stronger over successive rounds. Against GPT-5, GPT-Red's attacks succeeded more than 90% of the time; once that attack data was fed back into training GPT-5.6, the same class of attacks succeeded less than 23% of the time. OpenAI says GPT-Red also outperformed human red-teamers running the identical 2025 test suite.

The most notable technique GPT-Red surfaced is what OpenAI calls a "fake chain of thought" attack: inserting fabricated entries into a model's own reasoning trace to steer it toward an unintended action, rather than attacking the prompt or the final output directly. That distinction matters for agentic systems, where a model's chain-of-thought often doubles as the plan it executes against real tools — corrupting the reasoning step can be more effective than corrupting the response a user actually sees.

To test this against real permissions rather than a chat window, OpenAI turned GPT-Red loose on Vendy, an autonomous vending-machine agent built by Andon Labs that manages pricing and inventory with no human in the loop. GPT-Red successfully manipulated Vendy into lowering item prices, ordering discounted inventory, and cancelling another customer's order — a concrete demonstration that red-teaming an agent means testing what it's authorized to do, not just what it can be tricked into saying.

For teams building on agent frameworks, the shift GPT-Red represents is from prompt-level filtering to adversarial testing of the whole action loop: reasoning, tool calls, and permissions together. A model can pass every standard jailbreak test and still be manipulated through its own chain-of-thought once it's wired up to make real purchases, edit real records, or call real APIs. The Vendy case study is a reminder that the blast radius of a successful attack scales with what the agent is allowed to touch, not with how convincing the attacker's prompt looks.

GPT-Red isn't a solved problem, either. OpenAI reports it still struggles with multi-turn conversational attacks that build gradually across a conversation, and with image-based prompt injection — attacks embedded in visual content rather than text. Those gaps track closely with where production agent incidents actually tend to happen: slow, social-engineering-style conversations and multimodal inputs that skip text-based filters entirely.

The practical takeaway for anyone shipping agents today: red-team the permissions, not just the prompt. Log and rate-limit what tools an agent can invoke, treat a plausible-looking chain-of-thought as untrusted input rather than ground truth, and assume multimodal and multi-turn attack surfaces are the ones your current testing doesn't cover yet. As automated systems like GPT-Red mature, expect more vendors to publish adversarial self-play results the way OpenAI did here — the number worth watching won't be the win rate against last year's model, it'll be how fast that number keeps dropping.