Competitive self-play

# Competitive self-play Source: [https://openai.com/index/competitive-self-play/](https://openai.com/index/competitive-self-play/) We’ve found that self\-play allows simulated AIs to discover physical skills like tackling, ducking, faking, kicking, catching, and diving for the ball, without explicitly designing an environment with these skills in mind\. Self\-play ensures that the environment is always the right difficulty for an AI to improve\. Taken alongside our Dota 2 self\-play results, we have increasing confidence that self\-play will be a core part of powerful AI systems in the future\. We set up competitions between multiple simulated 3D robots on a range of basic games, trained each agent with simple goals \(push the opponent out of the sumo ring, reach the other side of the ring while preventing the other agent from doing the same, kick the ball into the net or prevent the other agent from doing so, and so on\), then analyzed the different strategies that emerged\. Agents initially receive dense rewards for behaviours that aid exploration like standing and moving forward, which are eventually annealed to zero in favor of being rewarded for just winning and losing\. Despite the simple rewards, the agents learn subtle behaviors like tackling, ducking, faking, kicking and catching, and diving for the ball\. Each agent’s neural network policy is independently trained with[Proximal Policy Optimization⁠\(opens in a new window\)](https://arxiv.org/abs/1707.06347)\.