Evolved Policy Gradients

# Evolved Policy Gradients Source: [https://openai.com/index/evolved-policy-gradients/](https://openai.com/index/evolved-policy-gradients/) The intuition behind EPG comes from something we are all familiar with: trying to pick up a new skill and experiencing the alternating frustration and joy involved in that process\. Suppose you are just starting out learning to play the violin\. Even without instruction, you will immediately have a feel for what to try, and, listening to the sounds you produce, you will have a sense of whether or not you are making progress – that’s because you effectively have access to very well shaped[internal reward functions⁠\(opens in a new window\)](http://www-anw.cs.umass.edu/legacy/pubs/2009/singh_l_b_09.pdf), derived from prior experience on other motor tasks, and through the course of biological evolution\. In contrast, most reinforcement learning \(RL\) agents approach each new task without using prior knowledge\. Instead they rely entirely on external reward signals to guide their initial behavior\. Coming from such a blank slate, it is no surprise that current RL agents take[far longer⁠\(opens in a new window\)](https://arxiv.org/abs/1604.00289)than humans to learn simple skills\. EPG takes a step toward agents that are not blank slates but instead know what it means to make progress on a new task, by having experienced making progress on similar tasks in the past\. There has been a flurry of recent[work⁠\(opens in a new window\)](https://arxiv.org/abs/1611.02779)[on⁠\(opens in a new window\)](https://arxiv.org/abs/1611.05763)[metalearning⁠\(opens in a new window\)](https://arxiv.org/abs/1707.03141)[policies⁠\(opens in a new window\)](https://arxiv.org/abs/1703.03400), and it’s worth asking why learn a loss function as opposed to directly learning a policy? Learning recurrent policies tends to overfit the task at hand, while learning policy initializations has limited expressivity when it comes to exploration\. Our motivation is that we expect loss functions to be the kind of object that may generalize very well across substantially different tasks\. This is certainly true of hand\-engineered loss functions: a well\-designed RL loss function, such as that in[PPO⁠\(opens in a new window\)](https://blog.openai.com/openai-baselines-ppo/), can be very generically applicable, finding use in problems ranging from playing Atari games to controlling robots\. To test the generalization ability of EPG, we conducted a simple experiment\. We evolved the EPG loss to be effective at getting “ants” to walk to randomly located targets on the right half of an arena\. Then, we froze the loss, and gave the ants a new target, this time on the*left*half of the arena\. Surprisingly, the ants learned to walk to the left\! Here is how their learning curves looked \(red lines on graphs\): This result is exciting to us because it demonstrates generalization to a task*outside the training distribution*\. This kind of generalization can be quite hard to achieve\. We compared EPG to an alternative metalearning algorithm, called[RL2⁠\(opens in a new window\)](https://arxiv.org/abs/1611.02779), which tries to directly learn a policy that can adapt to novel tasks\. In our experiment, RL2 was indeed successful at getting agents to walk to targets on the right half of the screen\. However, when given a test time target on the left half of the screen, it qualitatively failed, and just kept walking to the right\. In a sense, it “overfit” to the set of tasks on which it was trained \(i\.e\. walking to the right\)\. As do all metalearning approaches, our method still has many limitations\. Right now, we can train an EPG loss to be effective for one small family of tasks at a time, e\.g\., getting an ant to walk left and right\. However, the EPG loss for this family of tasks is unlikely to be at all effective on a wildly different kind of task, like playing Space Invaders\. In contrast, standard RL losses*do*have this level of generality—the same loss function can be used to learn a huge variety of skills\. EPG gains on performance by losing on generality\. There is a long road ahead toward metalearning methods that both outperform standard RL methods*and*have the same level of generality\.