ppo

这篇论文在统一实验协议下，用统计方法系统比较了六种深度强化学习处理（PPO、A2C、DQN，各含/不含动作掩码）与传统启发式算法及随机对照在HPC作业调度上的表现，发现MaskablePPO在异构负载上优于对照，但多数情况下DRL并未显著超越动作掩码带来的收益。

该论文通过反事实干预可视化PPO智能体在Crafter环境中的价值函数，发现模型无关的RL算法在优化奖励的同时隐式学习了环境的世界模型，如物品收集规则和健康机制，并讨论了其安全与对齐含义。

VASPO 将自蒸馏教师与 PPO 中的 critic 结合，通过 token 级优势与有界概率比率改进推理训练，在六个数学推理基准上相比 PPO 显著提升 Qwen3-4B 与 OLMo3-7B 的准确率。

Tencent ARC 的论文提出 Player-1 一套面向游戏控制的后训练框架，通过扩展动作空间、在 6,044 条游戏轨迹上进行 SFT 初始化，并利用 16 个可程序化验证的合成环境进行 PPO 多环境强化学习，显著超越监督基线与现有最强 baseline。

该论文提出 Tracking-Value PPO（TPPO），通过在对数几率空间对前缀值进行贝叶斯分解，构建累积值参数化以更好地跟踪语言模型推理中的值演化。TPPO 在 Qwen3-4B-Base 和 Qwen3-8B-Base 上分别比 PPO 提升 7.0 和 8.4 个数学推理分数点，并改善了分布外泛化能力。

A paper introducing MFPG-PPO, a multi-fidelity policy gradient method that uses cheap low-fidelity simulator data as a control variate to stabilize on-policy RL when high-fidelity real data is scarce, demonstrating stable learning on simulated robot locomotion and a physical Franka arm with only 4 real episodes per update.

This paper proposes complex-valued unitary recurrent networks (uRNNs) as drop-in replacements for recurrent PPO architectures, using phase-preserving dynamics inspired by quantum state measurement to address partial observability. The authors report up to 3x the reward of baselines on memory-improvable tasks such as rocksample and Craftax, with code released publicly.

EasyPPO stabilizes Proximal Policy Optimization for large language models by addressing critic instability through actor-only overlong filtering and noise-normalized regression, achieving performance gains over baseline methods.

ClashRoyaleAi is an open-source, deterministic Clash Royale simulator designed for reinforcement learning, featuring a PPO agent with lookahead search and expert iteration, showing improved win rates through simple lookahead techniques.

RefinePPO introduces iterative action refinement for continuous control policies, achieving performance comparable or better than standard PPO with faster convergence in benchmark tasks.

The paper identifies Value Flattening as a failure mode in PPO critic learning for LLMs and introduces SP3O, a sparse supervision method, to mitigate it, showing consistent improvements in experiments.

Jerry Tworek discusses the technical insights and challenges in applying reinforcement learning to scale AI models like o1, highlighting the importance of simplicity and techniques such as multiple rollouts.

A writeup that explains the mathematics of reinforcement learning, deriving algorithms such as Policy Gradient, PPO, and GRPO, and discusses their relevance in aligning large language models through techniques like RLHF and RLVR.

DelveRL is an open-source, turn-based roguelike game designed as a benchmark for training game-playing agents, including a Godot-based interface and a recurrent PPO trainer for research and development.

This paper introduces a unified black-box reinforcement learning framework for stable and scalable optimization of agents through complex harnesses, using sandbox execution and trajectory reconstruction with improvements on benchmarks like ClawGym-Bench.

This paper proposes JUROR, a reinforcement learning-based framework that jointly optimizes UAV flight paths and decentralized opportunistic routing in delay-tolerant networks under centralized training and decentralized execution.

Introduces SP3O, a novel reward-model-free, critic-free, gradient-based preference-based RL algorithm that leverages segment-level preferences, demonstrating improved performance in robotic control and LLM fine-tuning, especially for long-horizon tasks.

After 123 failed PPO experiments on Atari Breakout, adding a simple proximity reward for tracking the ball during descent finally achieved reactive, non-scripted play that transfers to unfamiliar brick layouts.

This paper systematically investigates instability in reinforcement learning for small language model agents (70-500M parameters), identifying three failure modes and proposing robust techniques including a merge-and-reinitialize adapter approach and safety mechanisms; it achieves stable convergence and improved win rates.

This paper presents a graph-based traffic signal control interface using a shared graph neural network to assign scores to movements, with deterministic phase construction via incidence matrices. Experiments evaluate transfer across synthetic and city road networks, showing feasibility but sensitivity to distribution shifts.