OpenAI released 'Spinning Up in Deep RL,' an educational toolkit featuring introductory materials, curated paper lists, and clean standalone implementations of key RL algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC) designed to help newcomers learn deep reinforcement learning from scratch.
We’re releasing Spinning Up in Deep RL, an educational resource designed to let anyone learn to become a skilled practitioner in deep reinforcement learning. Spinning Up consists of crystal-clear examples of RL code, educational exercises, documentation, and tutorials.
# Spinning Up in Deep RL
Source: [https://openai.com/index/spinning-up-in-deep-rl/](https://openai.com/index/spinning-up-in-deep-rl/)
At OpenAI, we believe that deep learning generally—and deep reinforcement learning specifically—will play central roles in the development of powerful AI technology\. While there are numerous resources available to let people quickly ramp up in deep learning, deep reinforcement learning is more challenging to break into\. We’ve designed Spinning Up to help people learn to use these technologies and to develop intuitions about them\.
We were inspired to build Spinning Up through our work with the OpenAI[Scholars\(opens in a new window\)](https://blog.openai.com/openai-scholars-2019/)and[Fellows\(opens in a new window\)](https://blog.openai.com/openai-fellows-interns-2019/)initiatives, where we observed that it’s possible for people with little\-to\-no experience in machine learning to rapidly ramp up as practitioners, if the right guidance and resources are available to them\. Spinning Up in Deep RL was built with this need in mind and is integrated into the curriculum for[2019 cohorts\(opens in a new window\)](https://blog.openai.com/openai-scholars-2019/)of Scholars and Fellows\.
We’ve also seen that being competent in RL can help people participate in interdisciplinary research areas like[AI safety\(opens in a new window\)](https://blog.openai.com/concrete-ai-safety-problems/), which involve a mix of reinforcement learning and other skills\. We’ve had so many people ask for guidance in learning RL from scratch, that we’ve decided to formalize the informal advice we’ve been giving\.
Spinning Up in Deep RL consists of the following core components:
- A short[introduction\(opens in a new window\)](https://spinningup.openai.com/en/latest/spinningup/rl_intro.html)to RL terminology, kinds of algorithms, and basic theory\.
- An[essay\(opens in a new window\)](https://spinningup.openai.com/en/latest/spinningup/spinningup.html)about how to grow into an RL research role\.
- A curated list of[important papers\(opens in a new window\)](https://spinningup.openai.com/en/latest/spinningup/keypapers.html)organized by topic\.
- A well\-documented[code repo\(opens in a new window\)](https://github.com/openai/spinningup)of short, standalone implementations of: Vanilla Policy Gradient \(VPG\), Trust Region Policy Optimization \(TRPO\), Proximal Policy Optimization \(PPO\), Deep Deterministic Policy Gradient \(DDPG\), Twin Delayed DDPG \(TD3\), and Soft Actor\-Critic \(SAC\)\.
- And a[few exercises\(opens in a new window\)](https://spinningup.openai.com/en/latest/spinningup/exercises.html)to serve as warm\-ups\.
We’ve designed the code for Spinning Up with newcomers in mind, making it short, friendly, and as easy to learn from as possible\. Our goal was to write minimal implementations to demonstrate how the theory becomes code, avoiding the layers of abstraction and obfuscation typically present in deep RL libraries\. We favor clarity over modularity—code reuse between implementations is strictly limited to logging and parallelization utilities\. Code is annotated so that you always know what’s going on, and is supported by background material \(and pseudocode\) on the corresponding readthedocs page\.
OpenAI held its first Spinning Up in Deep RL Workshop on February 2, engaging ~90 in-person participants and ~300 livestream viewers to provide education in deep RL, robotics, and AI safety through talks, mentorship, and hands-on projects.
OpenAI announces it is standardizing on PyTorch as its primary deep learning framework to improve research productivity and GPU performance at scale. As part of the move, they released a PyTorch version of Spinning Up in Deep RL and plan to open-source PyTorch bindings for their blocksparse kernels.
OpenAI releases OpenAI Gym, a public beta toolkit for developing and comparing reinforcement learning algorithms with a growing suite of environments and a platform for reproducible research. The toolkit aims to standardize RL benchmarks and address the lack of diverse, easy-to-use environments for the research community.
OpenAI announces the completion of its Fall 2018 Fellows program and celebrates the fellows' research contributions. The organization also open-sourced part of the fellowship curriculum, including 'Spinning up in Deep RL,' an educational resource for learning reinforcement learning.
OpenAI releases RL-Teacher, an open-source tool for training AI systems through human feedback instead of hand-crafted reward functions, with applications to safe AI development and complex reinforcement learning problems.