Tag
SAPO introduces a smooth, temperature-controlled gate to adaptively attenuate off-policy updates in reinforcement learning for large language models, enhancing training stability and performance compared to methods with hard clipping.
Single-stream Policy Optimization (SPO) is a new policy-gradient training method for Large Language Models that eliminates group-based issues like degenerate groups and synchronization barriers, providing a stable, low-variance learning signal and improving performance on hard math benchmarks compared to GRPO.
The paper introduces mmGRPO, a multi-module extension of Group Relative Policy Optimization (GRPO) that improves accuracy in modular AI systems by optimizing language model calls and prompts. It reports an average 11% accuracy improvement across various tasks and provides an open-source implementation in DSPy.
OpenAI trained 9 agents on the CoinRun environment with varying numbers of training levels to quantify generalization in reinforcement learning, finding substantial overfitting even with 16,000 training levels and that IMPALA-CNN architectures generalize significantly better than Nature-CNN baselines.
OpenAI presents parameter noise, a technique that adds adaptive noise to neural network policy parameters rather than action spaces, enabling agents to learn tasks significantly faster than traditional action noise approaches. The method achieves 2x faster learning on HalfCheetah and represents a middle ground between evolution strategies and deep RL approaches like TRPO and DDPG.
OpenAI presents evolution strategies (ES) as a scalable black-box optimization alternative to reinforcement learning for training neural network policies. ES simplifies the optimization problem by treating policy training as a stochastic parameter search that repeatedly samples and selects better parameter configurations based on reward feedback.