Tag
推荐 HuggingFace 博客文章系列《Score Function Centering》，从直觉出发系统讲解 LLM 异步强化学习中的 trainer-generator 失配问题以及分数函数中心化（SFC）如何修复策略梯度中零期望性质，只需了解 GRPO 即可阅读。
Part 1 of a 3-part series arguing that on-policy RL, distillation, and near-policy SFT updates are sparse, showing that because the score has zero mean under the sampler's distribution, the expected gradient is a covariance and the Fisher a variance, bounding any update regardless of reward.
The paper proposes Fixed-point Flow Matching (FFM), a data-free method for training and fine-tuning flow-based policies in online RL by formulating the actor update as a fixed-point iteration where the critic's action gradients serve as flow-matching regression targets, showing effective offline-to-online fine-tuning and from-scratch training.
ReSPO 提出一种重塑的序列策略优化方法，用平滑的双分支 α-散度核替代裁剪机制，解决 RLVR 中离策略学习的梯度饥饿问题，并在 Qwen3 密集与 MoE 模型上验证了更优的训练分数和基准性能。
This blog post visually explains policy gradients for large language models, deriving the concept from scratch using a math problem example to illustrate how reinforcement learning improves model performance.
The paper formulates regularity conditions for token-level credit in reinforcement learning for LLMs and introduces Policy Aligned Critic Training (PACT), which outperforms GRPO and PPO in mathematical reasoning and coding benchmarks.
The paper introduces On-Policy Reverse Distillation (OPRD), a method that enables stronger AI models to exceed weaker supervisors by amplifying verifier-supported policy gradients along the teacher's shift direction, achieving higher performance with fewer updates in distillation scenarios.
This paper proposes shrinkage baselines for reinforcement learning with verifiable rewards to reduce variance in policy gradient estimators and improve training stability.
The paper proposes Tail-Likelihood Reinforcement Learning (TailRL), an optimization method that focuses on the upper tails of reward distributions to improve policy performance in generative tasks, demonstrated across various applications.
This paper identifies value mismatch in parallel reinforcement learning where a shared critic degrades performance by mixing environment-specific values, and proposes conditioning the critic on environment indices to improve learning stability and return in benchmarks like Procgen.
This paper analyzes a single-loop, entropy-regularized Natural Actor-Critic algorithm and proves accelerated convergence rates for the unregularized objective in stochastic and deterministic regimes under linear function approximation.
This paper introduces Vector-Symbolic Policy Gradient (VSPG), a novel method that uses vector symbolic architecture for discrete-action policy gradients in reinforcement learning, achieving competitive performance with robust degradation under noise for edge systems.
Introduces Boundary-Seeking Policy Gradient (BSPG), a first-order method for safe reinforcement learning that actively drives the policy toward the constraint boundary, with convergence guarantees and improved reward/boundary tracking on a Safety-Gymnasium task.
This paper proves that using error-penalized scoring rules with abstention as a discrete action can kill both the reward gradient and the KL anchor, causing models to collapse toward refusing everything. It proposes a structural repair — training a mandatory confidence report — and validates the mechanism with simulations and language model experiments.
Introduces Policy Gradient Steering (PGS), a method that formulates activation steering as a reinforcement learning problem, using policy gradients to construct removable, composable steering vectors from behavioral objectives. Validated in gridworld, chess puzzle, and football environments.
Introduces Hindsight Policy Optimization (HPO), a novel policy gradient method that uses an intent space and Wasserstein distance to reduce variance in long-horizon language agent training, showing improved stability over GRPO and PPO.
SafeExplorer introduces an unbiased policy gradient estimator for reinforcement learning with recovery interventions, significantly reducing training-time falls on robot tasks while matching or exceeding standard PPO's final reward.
Proposes predictive divergence masks for LLM reinforcement learning that improve upon PPO's direction criterion by predicting whether the next policy gradient step will increase or decrease the divergence used by the trust region, leading to better alignment and improved RL training across model scales.
This paper introduces FADE (Focal Advantage with Dynamic Entropy), a self-adapting advantage function that dynamically schedules gradient weights during RL post-training of LLMs, achieving faster convergence and better accuracy-diversity trade-offs compared to static baselines.
EMAgnet introduces parameter-space exponential moving average regularization for policy gradient self-play in large two-player zero-sum games, achieving lower exploitability compared to uniform regularization targets.