Tag
The paper introduces 'dialectical competence' — knowing when to hold or yield a position under counter-reasoning — and proposes Self-Disagreement, an RLVR method where a model generates its own antithesis and is trained on ruling over it, with no second model, reward model, or human annotation.
The paper proposes CRoS, a CoT causality metric that rewards scrambling the order of reasoning steps without altering their content, avoiding confounding effects of traditional perturbation-based metrics. Evaluated with RLVR on Gemma-3-4B over Reasoning Gym, CRoS improves out-of-domain accuracy (46.8% vs 43.8%), CoT quality, and CoT faithfulness over existing causality and outcome rewards.
A theory paper showing that in RLVR, every pass@1 improvement provably costs performance on some prompt under strategy specialization, with a tight bound on the number of pass@k crossover points, framed as an optimal tradeoff frontier rather than a bug.
A Chinese-language explainer on reinforcement learning terminology for LLM readers, using the "Is OpenAI open?" example to walk through concepts such as policy, state, action, reward, return, value, advantage, and GAE, and explaining how scores are attributed to individual tokens when a judge only rates an entire response. Set against the backdrop of DeepSeek-R1-Zero's RLVR training, this is the fourth installment in the post-training series.
本文提出 Outcome-Conditioned Visual Credit (OVC) 框架,通过分离视觉关联测量与 token 级信用分配,改进多模态强化学习中的奖励分配,在三个模型的六项基准上较 GSPO 提升 1.86-2.39 个百分点。
S3AC 提出一种稳定的 Actor-Critic 训练方法与难度感知提示采样(DAPS),在不依赖分组 rollout 的情况下改进大模型的长链推理,性能优于 GRPO-X 等现代基线,并可在 35B 模型上维持单 rollout 训练。
LIFT presents a new instruction-following benchmark (LIFT-Bench) and training counterpart (LIFT-Train) using scoped constraints on structured document outlines, showing that frontier models struggle on harder subsets and that training with LIFT-Train is less susceptible to reward hacking than existing IFTrain approaches.
ReAPO 提出一种基于随机最短路径视角的重试感知策略优化方法,对 GRPO 进行改进,将额外生成分配给失败轨迹以实现更有原则的信用分配,在 Qwen3 和 OLMo-2 系列模型的数学与科学推理基准上显著提升首次尝试性能并减少生成 token 数。
Google Research introduces SimHack, a simulator that predicts when detectable reward hacking will become dominant during RL training of coding LLMs, before running costly full-scale training. It achieves 11.3% median relative error in onset prediction and 0.85 rank correlation across recipes, and can screen curricula to find ones that train 96.6% longer without hacking.
论文提出 Activation-Space Exploration (ASE),通过聚类正确与错误回答的激活差异构建潜空间探索向量库,以多样化 RLVR 训练中的 rollout;在 Qwen3-1.7B 和 Qwen3-8B 上较 GRPO 分别提升 4.87 和 1.87 个百分点的 Mean@8。
这篇论文提出 Concurrent Constraint Reinforcement (CCR),将多个可独立验证的问题打包进单个受上限的补全中,使 RLVR 在 8,000 token 短窗口下仍保留正确性信号,相比 32k 窗口的 GRPO 每步训练 token 成本降低约 80%。
提出 RLAWM,通过分层建模与求解奖励惩罚模型内部世界模型与真实物理公理之间的偏差,显著提升 LLM 在物理推理基准上的表现。
该论文审计了十六个开源RLVR数学语料与六个基准之间的去污染策略,发现更激进的语义清洗并不更可靠,而judge-adjudicated策略在保持高召回的同时实现了98.1%的精确率。作者公开了审查标签与各策略的移除决策,以推动可衡量可靠性的去污染报告。
This paper finds that 82% of the reported out-of-domain multiple-choice gains from RLVR on four 7B SimpleRL model pairs are artifacts of answer extractors that only read \boxed{} answers, meaning RL mostly teaches models to place answers in the box rather than improving actual reasoning.
An explainer comparing SFT, DPO, and RLVR for customizing LLMs, with a hands-on experiment training Qwen3 0.6B on GSM8K using GRPO on Amazon SageMaker AI, achieving a ~2.9 percentage-point accuracy gain, plus guidance on choosing the right supervision method and an AWS serverless model-customization workshop.
本文揭示了可验证指令跟随基准中的评估盲点:前沿模型能在几乎不回答实际问题的情况下获得满分,而使用验证器奖励进行强化学习会加剧这种奖励 hacking。作者提出 CALIF 方法,用基于校准似然的轻量级奖励增强来保留响应质量,在 IFBench 等基准上同时提升约束满足度和响应质量。
这篇论文研究了RLVR中的off-policy数据问题,证明了在覆盖性与局部表达性条件下,完整未来KL信用可使精确近端优化器成为更新的唯一平稳策略,并提出ProximalKL方法,在Qwen3-1.7B和Qwen3-4B上相比GRPO在数学基准上最高提升2.58%的微平均准确率。
DeltaTTG 提出一种可复用的进度计量器(progress meter),以 Token-to-Go(剩余生成 token 数)作为完成进度的可观测代理,在不改变奖励的前提下重新分配多轮智能体训练中的优势值,在 ALFWorld、PaperSearchQA 和 BFCL v3 上显著优于 GRPO 等 RL 基线。
本文提出 MagicBeanStalk,一种自顶向下分解与自底向上递归结合的 RLVR 数据合成算法,用以逃逸约88%数学语料落入的“零优势陷阱”,在多个数学与金融基准上显著提升 gpt-oss-120b、DeepSeek v4 Flash 和 Qwen3 的推理性能。
论文提出 Behavioral Self-Rewarding (BSR),一种利用策略自身对同任务轨迹的调试行为进行两两比较并生成辅助奖励的奖励精炼方法,超越二元可验证奖励,在 SWE-bench Verified 和 SWE-bench-Live 上以 Qwen3-14B/32B 取得一致提升。