ppo

Jerry Tworek discusses the technical insights and challenges in applying reinforcement learning to scale AI models like o1, highlighting the importance of simplicity and techniques such as multiple rollouts.

A writeup that explains the mathematics of reinforcement learning, deriving algorithms such as Policy Gradient, PPO, and GRPO, and discusses their relevance in aligning large language models through techniques like RLHF and RLVR.

DelveRL is an open-source, turn-based roguelike game designed as a benchmark for training game-playing agents, including a Godot-based interface and a recurrent PPO trainer for research and development.

This paper introduces a unified black-box reinforcement learning framework for stable and scalable optimization of agents through complex harnesses, using sandbox execution and trajectory reconstruction with improvements on benchmarks like ClawGym-Bench.

This paper proposes JUROR, a reinforcement learning-based framework that jointly optimizes UAV flight paths and decentralized opportunistic routing in delay-tolerant networks under centralized training and decentralized execution.

Introduces SP3O, a novel reward-model-free, critic-free, gradient-based preference-based RL algorithm that leverages segment-level preferences, demonstrating improved performance in robotic control and LLM fine-tuning, especially for long-horizon tasks.

After 123 failed PPO experiments on Atari Breakout, adding a simple proximity reward for tracking the ball during descent finally achieved reactive, non-scripted play that transfers to unfamiliar brick layouts.

This paper systematically investigates instability in reinforcement learning for small language model agents (70-500M parameters), identifying three failure modes and proposing robust techniques including a merge-and-reinitialize adapter approach and safety mechanisms; it achieves stable convergence and improved win rates.

This paper presents a graph-based traffic signal control interface using a shared graph neural network to assign scores to movements, with deterministic phase construction via incidence matrices. Experiments evaluate transfer across synthetic and city road networks, showing feasibility but sensitivity to distribution shifts.

This paper explores using quasi-Monte Carlo (QMC) methods for weight initialization in meta-reinforcement learning, showing improved convergence in similar control tasks compared to orthogonal initialization.

This paper systematically investigates failure modes in reinforcement learning for small language models (70-500M parameters) using PPO, identifies silent LoRA freezing, numerical overflow, and catastrophic policy collapse, and proposes a robust system with merge-and-reinitialize adapters, float32 precision, and a safety mechanism. The approach converges stably and outperforms baselines with less data.

PPO-HSC introduces a High-order Sampling Coverage reward to encourage exploration of diverse reasoning patterns in RL fine-tuning of LLMs, improving solution diversity and state-space coverage on math and code tasks.

A concise introduction to Reinforcement Learning, from basics to applied algorithms, with PyTorch implementations and supplementary proofs.

The article introduces CARE-PPO, a reinforcement learning framework that integrates confidence estimation with PPO fine-tuning for language-based quantitative prediction, enabling models to produce both accurate numerical estimates and reliable confidence signals, demonstrated in healthcare and finance tasks.

SafeExplorer introduces an unbiased policy gradient estimator for reinforcement learning with recovery interventions, significantly reducing training-time falls on robot tasks while matching or exceeding standard PPO's final reward.

Proposes predictive divergence masks for LLM reinforcement learning that improve upon PPO's direction criterion by predicting whether the next policy gradient step will increase or decrease the divergence used by the trust region, leading to better alignment and improved RL training across model scales.

Explains token-level rejection sampling for RLHF/PPO, where importance ratio M_t is the maximum over vocabulary and tokens are accepted with Bernoulli sampling based on w_t / M_t.

This paper introduces a dose-controllable method for inducing seven psychological disorders in reinforcement learning agents by manipulating cognitive appraisal signals in an appraisal-guided PPO agent. The disorders self-organize into a two-dimensional affective space, and the framework enables modeling of both disorder induction and treatment.

This paper proposes a Proximal Policy Optimization (PPO)-based deep reinforcement learning framework for dynamic battery charging of autonomous mobile robots in warehouses, achieving up to 6% higher order-completion rates over baseline methods.

This paper introduces spectral effective-rank entropy as a metric to measure and control critic complexity in actor-critic reinforcement learning, demonstrating its measurability and controllability in TD3 and PPO experiments.