automated-red-teaming

Anthropic's Alignment Science team showed that thousands of Claude agents automatically red-teaming training-data monitors can consistently find datasets that pass per-example screening yet still induce misaligned behavior — hidden in fiction and roleplay — raising questions about whether per-example screening alone is sufficient.

This paper introduces GPT-Red, an automated red-teaming agent trained via self-play at scale to discover novel prompt injection attacks against frontier LLMs, and uses it to adversarially train GPT-5.6, achieving the largest documented LLM safety training run.