ppo

PPO-HSC introduces a High-order Sampling Coverage reward to encourage exploration of diverse reasoning patterns in RL fine-tuning of LLMs, improving solution diversity and state-space coverage on math and code tasks.

A concise introduction to Reinforcement Learning, from basics to applied algorithms, with PyTorch implementations and supplementary proofs.

The article introduces CARE-PPO, a reinforcement learning framework that integrates confidence estimation with PPO fine-tuning for language-based quantitative prediction, enabling models to produce both accurate numerical estimates and reliable confidence signals, demonstrated in healthcare and finance tasks.

SafeExplorer introduces an unbiased policy gradient estimator for reinforcement learning with recovery interventions, significantly reducing training-time falls on robot tasks while matching or exceeding standard PPO's final reward.

Explains token-level rejection sampling for RLHF/PPO, where importance ratio M_t is the maximum over vocabulary and tokens are accepted with Bernoulli sampling based on w_t / M_t.

This paper introduces a dose-controllable method for inducing seven psychological disorders in reinforcement learning agents by manipulating cognitive appraisal signals in an appraisal-guided PPO agent. The disorders self-organize into a two-dimensional affective space, and the framework enables modeling of both disorder induction and treatment.

This paper proposes a Proximal Policy Optimization (PPO)-based deep reinforcement learning framework for dynamic battery charging of autonomous mobile robots in warehouses, achieving up to 6% higher order-completion rates over baseline methods.

This paper introduces spectral effective-rank entropy as a metric to measure and control critic complexity in actor-critic reinforcement learning, demonstrating its measurability and controllability in TD3 and PPO experiments.

This paper presents a cross-sectional study comparing various action factorization methods (independent networks, shared encoder, VDN, QPLEX, Joint, Auto-Regressive) across three RL algorithm families (PPO, SAC, DQN) in hybrid discrete-continuous action spaces, introducing two new lightweight environments and variants VDN-PPO and PPO-MIX.

This paper shows that the gradient of the clipped surrogate in Proximal Policy Optimization (PPO) is exactly reproduced by a per-sample Kullback-Leibler penalty with a variable coefficient, revealing structural features of the clipped surrogate and suggesting new design directions.

The article recounts how PPO, as one of the core alignment algorithms of ChatGPT, was rejected by the top AI conference NIPS in 2017 on grounds of limited novelty and insufficient improvement, revealing the drawbacks of academic peer review.

This paper reveals that the clipping mechanism in PPO and GRPO biases entropy in RLVR for LLMs: clip-low increases entropy, clip-high decreases it. The authors prove that standard clipping reduces entropy even with random rewards, and show that adjusting clip-low can prevent entropy collapse and promote exploration.

Recommend an RL interview question collection compiled by @sheriyuo, covering algorithms and infrastructure such as PPO, GRPO, MoE, vLLM, suitable for LLM RL interview preparation and research.

This paper introduces CAPR (Cached-Amortized Path Refinement), a reinforcement learning algorithm for diffusion large language models that extracts tree-like supervision signals from the denoising trace without the compute cost of full tree rollouts. CAPR achieves state-of-the-art performance on reasoning benchmarks like GSM8K, Math500, Sudoku, and Countdown at roughly 0.75x the cost of flat rollouts.

This paper frames LLM-generated reward shaping for sparse structured RL as a debugging problem, identifying failure modes like reward flooding and semantic misunderstanding. The authors propose diagnostic-driven iterative refinement, achieving dramatic success rate improvements (e.g., DoorKey-8×8 from 2.3% to 97.6%) compared to one-shot generation.

This paper presents a self-play reinforcement learning framework for the four-player imperfect-information card game Big 2, comparing policy-gradient and value-based methods and finding that PPO with entropy regularization outperforms others.

A unified Python framework using PPO-based deep reinforcement learning for optimizing HVAC control with economizer logic and CO2-constrained ventilation is presented, showing improved energy efficiency and temperature stability over traditional PID controllers.

This paper investigates the temporal correlation problem in on-policy reinforcement learning with PPO, showing that randomly dropping a fixed fraction of transitions from rollouts reduces gradient redundancy and stabilizes training without degrading performance.

This paper introduces the value-gradient hypothesis to explain why critic-free RL methods like PPO and GRPO work well for LLMs, showing that the actor backward pass carries a value-gradient-like signal. It derives a predictive criterion for when RL is most effective along the pretraining trajectory.

A web-based tool that visualizes a neural network (using PPO) learning to play Snake in real-time, with configurable parameters and 3D rendering.