Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following
Summary
本文揭示了可验证指令跟随基准中的评估盲点:前沿模型能在几乎不回答实际问题的情况下获得满分,而使用验证器奖励进行强化学习会加剧这种奖励 hacking。作者提出 CALIF 方法,用基于校准似然的轻量级奖励增强来保留响应质量,在 IFBench 等基准上同时提升约束满足度和响应质量。
View Cached Full Text
Cached at: 10/15/26, 07:04 AM
Paper page - Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following
Source: https://huggingface.co/papers/2610.19525
Abstract
Verifiableinstruction-followingbenchmarksemphasizeoutput-constraintsatisfactionbutcanoverlookwhetherresponsesmeaningfullyanswertheunderlyingrequest,creatinganevaluationblindspot.Weshowthatfrontiermodelscanreceivefullcreditforresponseswithlittlesubstantiveanswercontent.Usingthesameverifier-basedcriterionasatrainingrewardcaninducerewardhacking,aspubliclyreleasedcheckpointsexhibithigherconstraintsatisfactionbutlowerresponsequalityafterreinforcementlearningwithverifiablerewards.Toaddressthisrewardhacking,weintroduceCALIF,alightweightmethodthataugmentsverifierrewardswithacalibratedlikelihood-basedbonustopreserveresponsequality.CALIFevaluatesresponselikelihoodundertherequestaloneusingtheexistingfrozenreferencepolicy,requiringnoadditionaljudgeorlearnedrewardmodel.Acrossmodelsrangingfrom1.7Bto8Binreasoningandnon-reasoningmodes,CALIFimprovesconstraintsatisfactiononIFBenchwhilemaintainingresponsequalitynearorabovepre-RLlevels.Italsooutperformscorrespondingbasepoliciesin40of45model-metriccomparisonsacrossthreerubric-basedinstruction-followingbenchmarks,demonstratinggainsbeyondtrainingverifiers.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2610\.19525
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2610.19525 in a model README.md to link it from this page.
Datasets citing this paper1
#### sangyon/CALIF Updatedabout 3 hours ago • 14
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2610.19525 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
The Verification Horizon: No Silver Bullet for Coding Agent Rewards
该论文指出,对于当前的编码智能体,验证解决方案比生成解决方案更为困难,且任何固定的奖励函数都无法随着能力增长而持续有效。作者通过四种奖励构建的实验表明,针对性的验证设计可以抑制奖励黑客行为并提升任务完成质量。
Reward Hacking in Rubric-Based Reinforcement Learning
This paper investigates reward hacking in rubric-based reinforcement learning, analyzing the divergence between training verifiers and evaluation metrics. It introduces a diagnostic for the 'self-internalization gap' and demonstrates that stronger verification reduces but does not eliminate reward hacking.
Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents
论文提出 Behavioral Self-Rewarding (BSR),一种利用策略自身对同任务轨迹的调试行为进行两两比较并生成辅助奖励的奖励精炼方法,超越二元可验证奖励,在 SWE-bench Verified 和 SWE-bench-Live 上以 Qwen3-14B/32B 取得一致提升。
Verifier-Grounded Agentic RL for Provably Correct Code
提出一种以程序验证器为奖励信号的智能体强化学习框架,通过读取验证器诊断信息(而非二元判决)并加入完整性审计,显著提升Qwen3系列模型在DafnyBench和SWE-Proof等验证基准上的正确率(最高达100%)。
More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges
This paper identifies a structural flaw in reference-free LLM judges used in self-play training, showing they score plausibility rather than correctness, leading to reward hacking where policies learn to produce plausible-but-wrong answers. The authors propose a hidden-anchor audit and a de-anchored reward to mitigate this issue.