Variational option discovery algorithms

OpenAI Blog 07/26/18, 07:00 AM Papers

Summary

OpenAI researchers introduce VALOR, a variational inference method for option discovery that connects option learning to variational autoencoders, and propose a curriculum learning approach that stabilizes training by dynamically increasing context complexity.

No content available

Original Article

View Cached Full Text

Cached at: 04/20/26, 02:55 PM

# Variational option discovery algorithms Source: [https://openai.com/index/variational-option-discovery-algorithms/](https://openai.com/index/variational-option-discovery-algorithms/) OpenAI## Abstract We explore methods for option discovery based on variational inference and make two algorithmic contributions\. First: we highlight a tight connection between variational option discovery methods and variational autoencoders, and introduce Variational Autoencoding Learning of Options by Reinforcement \(VALOR\), a new method derived from the connection\. In VALOR, the policy encodes contexts from a noise distribution into trajectories, and the decoder recovers the contexts from the complete trajectories\. Second: we propose a curriculum learning approach where the number of contexts seen by the agent increases whenever the agent's performance is strong enough \(as measured by the decoder\) on the current set of contexts\. We show that this simple trick stabilizes training for VALOR and prior variational option discovery methods, allowing a single agent to learn many more modes of behavior than it could with a fixed context distribution\. Finally, we investigate other topics related to variational option discovery, including fundamental limitations of the general approach and the applicability of learned options to downstream tasks\.

Similar Articles

ELVAE: Evidential Learning-Based Variational Autoencoder for Uncertainty-Aware Generation

arXiv cs.LG

Introduces ELVAE, a VAE with evidential learning that models latent coordinates with normal-inverse-gamma posteriors to obtain explicit uncertainty estimates. Experiments on MNIST show that within-class uncertainty ranking can stratify synthetic sample reliability and stress-testing, though results require class-wise normalization and vary across seeds.

CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning

arXiv cs.CL

This arXiv paper introduces CVPO, a reinforcement learning method for LLMs that adapts value-variance for advantage estimation and uses dynamic curriculum learning to match question difficulty, achieving better reasoning performance than VAPO on math tasks.

Variational lossy autoencoder

OpenAI Blog

OpenAI researchers present a Variational Lossy Autoencoder (VLAE) that combines VAEs with neural autoregressive models (RNN, MADE, PixelRNN/CNN) to learn controllable global representations, achieving state-of-the-art results on MNIST, OMNIGLOT, and Caltech-101 Silhouettes density estimation tasks.

Fast and Slow Variational Continual Learning

arXiv cs.LG

This paper introduces the Continual IVON (CoVON) optimizer, which integrates fast and slow adaptation into variational continual learning to balance stability and plasticity, outperforming existing methods in domain-incremental learning, continual pre-training, and fine-tuning of large language models.

Multimodal Continuous Reasoning via Asymmetric Mutual Variational Learning

Hugging Face Daily Papers

Proposes Asymmetric Mutual Variational Learning (AMVL) to resolve train-inference mismatch in multimodal continuous reasoning by using bidirectional calibration to prevent answer leakage and improve latent-space stability, achieving significant gains on the BLINK benchmark.