OpenAI's GPT-Red: AI Learns to Police Itself

OpenAI is turning its AI models against themselves in a bid to bolster safety. The company announced GPT-Red, an internal system designed to act as an automated red-teamer, a critical but often bottlenecked process for identifying vulnerabilities before models are widely deployed. This initiative represents a significant step towards scaling AI safety in lockstep with model capabilities.

Red-teaming, the process of actively probing AI systems for weaknesses, is essential for robustness. However, human-led efforts are time-intensive and struggle to generate the sheer volume and diversity of adversarial data needed to train increasingly powerful models. OpenAI's announcement highlights how current robustness evaluations are already being saturated by their latest models, necessitating a more scalable approach.

GPT-Red is the culmination of OpenAI's work on automated red-teaming. It functions by sending prompts, observing model responses, and iterating to discover vulnerabilities, particularly prompt injection attacks. The company dedicated an unprecedented amount of compute, comparable to some of its largest post-training runs, purely to training GPT-Red.

Self-Improvement Through Self-Play

The system is trained using self-play reinforcement learning. GPT-Red is rewarded for successfully eliciting failures, like executing a prompt injection, while defender models are trained to resist these attacks and complete their intended tasks. This dynamic forces GPT-Red to constantly evolve and discover more sophisticated attack vectors as the defender models improve.

OpenAI utilized GPT-Red in the adversarial training of OpenAI GPT-5.6 Sol. The results show a marked improvement in robustness, with the model exhibiting six times fewer failures on direct prompt injection benchmarks compared to its predecessor from just four months prior. This demonstrates the effectiveness of using AI to improve AI safety.

A Powerful, Evolving Adversary

GPT-Red has proven to be a formidable red-teamer. It can successfully break nearly all models it's pitted against, including internal and production models up to GPT-5.5. In tests on novel safety environments, GPT-Red achieved an 84% attack success rate, significantly outperforming human red-teamers who achieved 13%.

Case studies showcase GPT-Red's capabilities. It successfully manipulated an AI-powered vending machine to change item prices, offer expensive items for $0.50, and cancel customer orders. It also proved more effective and token-efficient than human-prompted models in exfiltrating sensitive data from a Codex CLI agent.

Crucially, OpenAI emphasizes that this robustness is achieved without compromising general capabilities. Models trained with GPT-Red do not exhibit increased refusal rates or a decrease in performance on legitimate tasks. This suggests the gains are in genuine resistance to malicious instructions rather than an overabundance of caution.

OpenAI plans to continue scaling this approach, training even stronger versions of GPT-Red to further enhance the safety and trustworthiness of future AI releases. The company will release more details in an upcoming preprint.