Tag
The paper formulates regularity conditions for token-level credit in reinforcement learning for LLMs and introduces Policy Aligned Critic Training (PACT), which outperforms GRPO and PPO in mathematical reasoning and coding benchmarks.
The paper introduces On-Policy Reverse Distillation (OPRD), a method that enables stronger AI models to exceed weaker supervisors by amplifying verifier-supported policy gradients along the teacher's shift direction, achieving higher performance with fewer updates in distillation scenarios.
This paper proposes shrinkage baselines for reinforcement learning with verifiable rewards to reduce variance in policy gradient estimators and improve training stability.
The paper proposes Tail-Likelihood Reinforcement Learning (TailRL), an optimization method that focuses on the upper tails of reward distributions to improve policy performance in generative tasks, demonstrated across various applications.
This paper identifies value mismatch in parallel reinforcement learning where a shared critic degrades performance by mixing environment-specific values, and proposes conditioning the critic on environment indices to improve learning stability and return in benchmarks like Procgen.
This paper analyzes a single-loop, entropy-regularized Natural Actor-Critic algorithm and proves accelerated convergence rates for the unregularized objective in stochastic and deterministic regimes under linear function approximation.
This paper introduces Vector-Symbolic Policy Gradient (VSPG), a novel method that uses vector symbolic architecture for discrete-action policy gradients in reinforcement learning, achieving competitive performance with robust degradation under noise for edge systems.
Introduces Boundary-Seeking Policy Gradient (BSPG), a first-order method for safe reinforcement learning that actively drives the policy toward the constraint boundary, with convergence guarantees and improved reward/boundary tracking on a Safety-Gymnasium task.
This paper proves that using error-penalized scoring rules with abstention as a discrete action can kill both the reward gradient and the KL anchor, causing models to collapse toward refusing everything. It proposes a structural repair — training a mandatory confidence report — and validates the mechanism with simulations and language model experiments.
Introduces Policy Gradient Steering (PGS), a method that formulates activation steering as a reinforcement learning problem, using policy gradients to construct removable, composable steering vectors from behavioral objectives. Validated in gridworld, chess puzzle, and football environments.
Introduces Hindsight Policy Optimization (HPO), a novel policy gradient method that uses an intent space and Wasserstein distance to reduce variance in long-horizon language agent training, showing improved stability over GRPO and PPO.
SafeExplorer introduces an unbiased policy gradient estimator for reinforcement learning with recovery interventions, significantly reducing training-time falls on robot tasks while matching or exceeding standard PPO's final reward.
Proposes predictive divergence masks for LLM reinforcement learning that improve upon PPO's direction criterion by predicting whether the next policy gradient step will increase or decrease the divergence used by the trust region, leading to better alignment and improved RL training across model scales.
This paper introduces FADE (Focal Advantage with Dynamic Entropy), a self-adapting advantage function that dynamically schedules gradient weights during RL post-training of LLMs, achieving faster convergence and better accuracy-diversity trade-offs compared to static baselines.
EMAgnet introduces parameter-space exponential moving average regularization for policy gradient self-play in large two-player zero-sum games, achieving lower exploitability compared to uniform regularization targets.
This paper shows that the gradient of the clipped surrogate in Proximal Policy Optimization (PPO) is exactly reproduced by a per-sample Kullback-Leibler penalty with a variable coefficient, revealing structural features of the clipped surrogate and suggesting new design directions.
MIT researchers co-authored a paper showing that general-purpose policy gradient algorithms can outperform specialized game-theoretic algorithms in imperfect-information games, challenging long-held assumptions in the field.
This paper formalizes embedding model routing as an adversarial contextual linear bandit with low-rank experts, proposing the Hypentropy Policy Gradient (HPG) algorithm that achieves O~(s√(MT)) policy regret, avoiding the curse of dimensionality.
DiPOD stabilizes diffusion policy optimization by interleaving self-distillation with policy-gradient updates to maintain a tight ELBO, preventing the double-drift phenomenon and achieving higher rewards in both language and continuous control tasks.
SDPG (Self-Distilled Policy Gradient) is a new RL training framework for LLMs that combines group-relative verifier advantages with on-policy self-distillation and KL regularization to address sparse rewards and instability in RLVR training. The method uses a shared model as both student and teacher by conditioning on privileged context, showing improved stability and performance over RLVR and self-distillation baselines.