Competitive self-play

OpenAI Blog Papers
self-play multi-agent-learning reinforcement-learning emergent-behavior 3d-simulation ppo

Summary

OpenAI demonstrates that competitive self-play in simulated 3D robot environments enables AI agents to discover complex physical behaviors like tackling, ducking, and faking without explicit instruction, suggesting self-play will be fundamental to future powerful AI systems.

We’ve found that self-play allows simulated AIs to discover physical skills like tackling, ducking, faking, kicking, catching, and diving for the ball, without explicitly designing an environment with these skills in mind. Self-play ensures that the environment is always the right difficulty for an AI to improve. Taken alongside our Dota 2 self-play results, we have increasing confidence that self-play will be a core part of powerful AI systems in the future.
Original Article
View Cached Full Text

Cached at: 04/20/26, 02:57 PM

# Competitive self-play Source: [https://openai.com/index/competitive-self-play/](https://openai.com/index/competitive-self-play/) We’ve found that self\-play allows simulated AIs to discover physical skills like tackling, ducking, faking, kicking, catching, and diving for the ball, without explicitly designing an environment with these skills in mind\. Self\-play ensures that the environment is always the right difficulty for an AI to improve\. Taken alongside our Dota 2 self\-play results, we have increasing confidence that self\-play will be a core part of powerful AI systems in the future\. We set up competitions between multiple simulated 3D robots on a range of basic games, trained each agent with simple goals \(push the opponent out of the sumo ring, reach the other side of the ring while preventing the other agent from doing the same, kick the ball into the net or prevent the other agent from doing so, and so on\), then analyzed the different strategies that emerged\. Agents initially receive dense rewards for behaviours that aid exploration like standing and moving forward, which are eventually annealed to zero in favor of being rewarded for just winning and losing\. Despite the simple rewards, the agents learn subtle behaviors like tackling, ducking, faking, kicking and catching, and diving for the ball\. Each agent’s neural network policy is independently trained with[Proximal Policy Optimization⁠\(opens in a new window\)](https://arxiv.org/abs/1707.06347)\.

Similar Articles

Dota 2

OpenAI Blog

OpenAI created a bot that defeats world-class Dota 2 professionals in 1v1 matches using only self-play learning, without imitation learning or tree search. The achievement demonstrates progress toward AI systems that can accomplish complex goals in dynamic, multi-agent environments.

Emergent tool use from multi-agent interaction

OpenAI Blog

OpenAI demonstrates that agents trained in a hide-and-seek environment discover six distinct emergent strategies and tool-use behaviors through multi-agent competition, without explicit incentives for object interaction. This work suggests multi-agent co-adaptation can produce complex intelligent behavior through self-supervised learning.

Is this machine playing?

arXiv cs.AI

MIT researchers placed an AI coding assistant as the mind of a virtual body on a digital island with no specified task or reward, and observed that across thirteen 30-hour runs the agents spontaneously engaged in play-like behaviors—climbing, building towers, drawing mandalas, and running physics experiments. The paper examines whether this behavior meets classical criteria for play and whether play could become a mode of machine development.

Dota 2 with large scale deep reinforcement learning

OpenAI Blog

OpenAI Five became the first AI system to defeat Dota 2 world champions using large-scale deep reinforcement learning with self-play, demonstrating superhuman performance on a complex game with long time horizons and imperfect information.