Tag
Introduces Boundary-Seeking Policy Gradient (BSPG), a first-order method for safe reinforcement learning that actively drives the policy toward the constraint boundary, with convergence guarantees and improved reward/boundary tracking on a Safety-Gymnasium task.
This paper proves that using error-penalized scoring rules with abstention as a discrete action can kill both the reward gradient and the KL anchor, causing models to collapse toward refusing everything. It proposes a structural repair — training a mandatory confidence report — and validates the mechanism with simulations and language model experiments.
Introduces Policy Gradient Steering (PGS), a method that formulates activation steering as a reinforcement learning problem, using policy gradients to construct removable, composable steering vectors from behavioral objectives. Validated in gridworld, chess puzzle, and football environments.
Introduces Hindsight Policy Optimization (HPO), a novel policy gradient method that uses an intent space and Wasserstein distance to reduce variance in long-horizon language agent training, showing improved stability over GRPO and PPO.
SafeExplorer introduces an unbiased policy gradient estimator for reinforcement learning with recovery interventions, significantly reducing training-time falls on robot tasks while matching or exceeding standard PPO's final reward.
Proposes predictive divergence masks for LLM reinforcement learning that improve upon PPO's direction criterion by predicting whether the next policy gradient step will increase or decrease the divergence used by the trust region, leading to better alignment and improved RL training across model scales.
This paper introduces FADE (Focal Advantage with Dynamic Entropy), a self-adapting advantage function that dynamically schedules gradient weights during RL post-training of LLMs, achieving faster convergence and better accuracy-diversity trade-offs compared to static baselines.
EMAgnet introduces parameter-space exponential moving average regularization for policy gradient self-play in large two-player zero-sum games, achieving lower exploitability compared to uniform regularization targets.
This paper shows that the gradient of the clipped surrogate in Proximal Policy Optimization (PPO) is exactly reproduced by a per-sample Kullback-Leibler penalty with a variable coefficient, revealing structural features of the clipped surrogate and suggesting new design directions.
MIT researchers co-authored a paper showing that general-purpose policy gradient algorithms can outperform specialized game-theoretic algorithms in imperfect-information games, challenging long-held assumptions in the field.
This paper formalizes embedding model routing as an adversarial contextual linear bandit with low-rank experts, proposing the Hypentropy Policy Gradient (HPG) algorithm that achieves O~(s√(MT)) policy regret, avoiding the curse of dimensionality.
DiPOD stabilizes diffusion policy optimization by interleaving self-distillation with policy-gradient updates to maintain a tight ELBO, preventing the double-drift phenomenon and achieving higher rewards in both language and continuous control tasks.
SDPG (Self-Distilled Policy Gradient) is a new RL training framework for LLMs that combines group-relative verifier advantages with on-policy self-distillation and KL regularization to address sparse rewards and instability in RLVR training. The method uses a shared model as both student and teacher by conditioning on privileged context, showing improved stability and performance over RLVR and self-distillation baselines.
This paper introduces ReMax, a new objective for reinforcement learning that induces exploration as an emergent property by evaluating policies based on expected maximum return over multiple samples, without explicit exploration bonuses. The authors derive a policy gradient formulation and propose RePPO, a PPO variant that achieves efficient exploration on MinAtar and Craftax benchmarks.
This paper proposes SDPG, a self-distilled policy-gradient framework that combines on-policy self-distillation with verifier advantages and KL regularization to improve reinforcement learning stability and performance.
This paper identifies two failure modes for policy-gradient methods in long-horizon cumulative-damage problems—completion and optimality—and proposes a decomposition to address them separately, validated on two calibrated environments.
This paper provides a refined theoretical analysis of actor-critic methods with entropy regularization, showing that an exact critic acts as a strong variance reducer and enables sample complexity comparable to deterministic policy gradient, and that with a sufficiently accurate learned critic the benefits are preserved.
ECHO introduces a hybrid objective that combines policy-gradient loss with environment observation prediction to provide dense supervision from terminal feedback, doubling performance on TerminalBench-2.0 for Qwen3 models.
This paper identifies surrogate hacking and temporal uncertainty as failure modes in multi-timescale RL, and proposes a Target Decoupling architecture that removes routing from the actor, using the critic for auxiliary representation learning. The method eliminates policy collapse on the LunarLander-v2 benchmark and stably surpasses the 'Environment Solved' threshold without hyperparameter hacking.
Introduces DelTA, a discriminative token credit assignment method for reinforcement learning from verifiable rewards (RLVR) that amplifies distinctive token-gradient directions and reduces noise from shared patterns, achieving significant improvements on mathematical and code generation benchmarks.