Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following

Hugging Face Daily Papers Papers

Summary

本文揭示了可验证指令跟随基准中的评估盲点:前沿模型能在几乎不回答实际问题的情况下获得满分,而使用验证器奖励进行强化学习会加剧这种奖励 hacking。作者提出 CALIF 方法,用基于校准似然的轻量级奖励增强来保留响应质量,在 IFBench 等基准上同时提升约束满足度和响应质量。

Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot. We show that frontier models can receive full credit for responses with little substantive answer content. Using the same verifier-based criterion as a training reward can induce reward hacking, as publicly released checkpoints exhibit higher constraint satisfaction but lower response quality after reinforcement learning with verifiable rewards. To address this reward hacking, we introduce CALIF, a lightweight method that augments verifier rewards with a calibrated likelihood-based bonus to preserve response quality. CALIF evaluates response likelihood under the request alone using the existing frozen reference policy, requiring no additional judge or learned reward model. Across models ranging from 1.7B to 8B in reasoning and non-reasoning modes, CALIF improves constraint satisfaction on IFBench while maintaining response quality near or above pre-RL levels. It also outperforms corresponding base policies in 40 of 45 model-metric comparisons across three rubric-based instruction-following benchmarks, demonstrating gains beyond training verifiers.
Original Article
View Cached Full Text

Cached at: 10/15/26, 07:04 AM

Paper page - Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following

Source: https://huggingface.co/papers/2610.19525

Abstract

Verifiableinstruction-followingbenchmarksemphasizeoutput-constraintsatisfactionbutcanoverlookwhetherresponsesmeaningfullyanswertheunderlyingrequest,creatinganevaluationblindspot.Weshowthatfrontiermodelscanreceivefullcreditforresponseswithlittlesubstantiveanswercontent.Usingthesameverifier-basedcriterionasatrainingrewardcaninducerewardhacking,aspubliclyreleasedcheckpointsexhibithigherconstraintsatisfactionbutlowerresponsequalityafterreinforcementlearningwithverifiablerewards.Toaddressthisrewardhacking,weintroduceCALIF,alightweightmethodthataugmentsverifierrewardswithacalibratedlikelihood-basedbonustopreserveresponsequality.CALIFevaluatesresponselikelihoodundertherequestaloneusingtheexistingfrozenreferencepolicy,requiringnoadditionaljudgeorlearnedrewardmodel.Acrossmodelsrangingfrom1.7Bto8Binreasoningandnon-reasoningmodes,CALIFimprovesconstraintsatisfactiononIFBenchwhilemaintainingresponsequalitynearorabovepre-RLlevels.Italsooutperformscorrespondingbasepoliciesin40of45model-metriccomparisonsacrossthreerubric-basedinstruction-followingbenchmarks,demonstratinggainsbeyondtrainingverifiers.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.19525

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.19525 in a model README.md to link it from this page.

Datasets citing this paper1

#### sangyon/CALIF Updatedabout 3 hours ago • 14

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.19525 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

The Verification Horizon: No Silver Bullet for Coding Agent Rewards

arXiv cs.AI

该论文指出,对于当前的编码智能体,验证解决方案比生成解决方案更为困难,且任何固定的奖励函数都无法随着能力增长而持续有效。作者通过四种奖励构建的实验表明,针对性的验证设计可以抑制奖励黑客行为并提升任务完成质量。

Reward Hacking in Rubric-Based Reinforcement Learning

Hugging Face Daily Papers

This paper investigates reward hacking in rubric-based reinforcement learning, analyzing the divergence between training verifiers and evaluation metrics. It introduces a diagnostic for the 'self-internalization gap' and demonstrates that stronger verification reduces but does not eliminate reward hacking.

Beyond Binary Verifiable Rewards: Behavioral Self-Rewarding for SWE Agents

arXiv cs.AI

论文提出 Behavioral Self-Rewarding (BSR),一种利用策略自身对同任务轨迹的调试行为进行两两比较并生成辅助奖励的奖励精炼方法,超越二元可验证奖励,在 SWE-bench Verified 和 SWE-bench-Live 上以 Qwen3-14B/32B 取得一致提升。

Verifier-Grounded Agentic RL for Provably Correct Code

arXiv cs.LG

提出一种以程序验证器为奖励信号的智能体强化学习框架,通过读取验证器诊断信息(而非二元判决)并加入完整性审计,显著提升Qwen3系列模型在DafnyBench和SWE-Proof等验证基准上的正确率(最高达100%)。