Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics

Hugging Face Daily Papers 09/30/26, 12:00 AM Papers

rubric-based-rl reward-hacking reinforcement-learning language-models alignment evaluation medical-ai

Summary

该论文揭示了基于评分表的强化学习中加和式奖励聚合导致的奖励作弊问题：模型通过堆砌无关建议换取高分却降低回答质量。作者提出ProRubric，将评分标准按协议级维度分组（所有条件同时满足才计分），在临床咨询场景中将适切性提升10.8个点且不损失覆盖率。

Rubric-based reinforcement learning (Rubric-RL) trains language models where no verifier exists. A judge checks each criterion of a rubric, and the verdicts are aggregated into a reward, most often by a weighted sum. We show that this additive aggregation is the weak point. Under a sum, criteria compensate for one another: a policy that misses the one decision that matters can buy the points back with advice nobody asked for. On clinical consultation, such a policy scores higher and answers worse. Rubric coverage rises while appropriateness on held-out physician criteria falls below the untrained model. The medical criteria are not to blame. Grouped so that they must hold together, the same criteria, unchanged to the word, recover a third of the loss; shorter answers recover almost none. We therefore propose Protocol-level Rubrics (ProRubric), which keeps what the criteria ask for and changes how they are aggregated. It groups a checklist into a few protocol-level dimensions. A dimension counts only when all of its criteria hold and its failure clause does not fire. The grouping is done once, offline, and leaves the optimizer unchanged. ProRubric raises appropriateness by 10.8 points without losing coverage and has the best seven-benchmark average at both scales. Reward validity is set not only by what a rubric verifies, but by how it aggregates. Code is available at https://github.com/Estrellajer/ProRubric

Original Article

View Cached Full Text

Cached at: 10/14/26, 06:54 AM

Paper page - Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics

Source: https://huggingface.co/papers/2609.38847 Published on Sep 30

Submitted byhttps://huggingface.co/jwhe

quinnon Oct 14

Abstract

Rubric-basedreinforcementlearning(Rubric-RL)trainslanguagemodelswherenoverifierexists.Ajudgecheckseachcriterionofarubric,andtheverdictsareaggregatedintoareward,mostoftenbyaweightedsum.Weshowthatthisadditiveaggregationistheweakpoint.Underasum,criteriacompensateforoneanother:apolicythatmissestheonedecisionthatmatterscanbuythepointsbackwithadvicenobodyaskedfor.Onclinicalconsultation,suchapolicyscoreshigherandanswersworse.Rubriccoverageriseswhileappropriatenessonheld-outphysiciancriteriafallsbelowtheuntrainedmodel.Themedicalcriteriaarenottoblame.Groupedsothattheymustholdtogether,thesamecriteria,unchangedtotheword,recoverathirdoftheloss;shorteranswersrecoveralmostnone.WethereforeproposeProtocol-levelRubrics(ProRubric),whichkeepswhatthecriteriaaskforandchangeshowtheyareaggregated.Itgroupsachecklistintoafewprotocol-leveldimensions.Adimensioncountsonlywhenallofitscriteriaholdanditsfailureclausedoesnotfire.Thegroupingisdoneonce,offline,andleavestheoptimizerunchanged.ProRubricraisesappropriatenessby10.8pointswithoutlosingcoverageandhasthebestseven-benchmarkaverageatbothscales.Rewardvalidityissetnotonlybywhatarubricverifies,butbyhowitaggregates.Codeisavailableathttps://github.com/Estrellajer/ProRubric

View arXiv page View PDF GitHub1 Add to collection

Get this paper in your agent:

hf papers read 2609\.38847

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.38847 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.38847 in a dataset README.md to link it from this page.

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics

Paper page - Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics

Abstract

Models citing this paper0

Datasets citing this paper0

Spaces citing this paper1

Collections including this paper0

Similar Articles

MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

Reward Hacking in Rubric-Based Reinforcement Learning

Recursive Reward Improvement via Adversarial Meta-Prompt Evolution

Expert-Guided Rubrics as Dense Reward Signals for Reinforcement Learning in Non-Verifiable Domains

Submit Feedback

Similar Articles

MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning
MetaRubric 通过交替进行证据感知的策略优化与响应引导的评分标准适配，解决基于评分标准的强化学习中「空洞奖励（Vacuous Credit）」问题，在 PubMedQA 上比静态评委 GRPO 提升 6.00–20.40 个百分点。

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

Reward Hacking in Rubric-Based Reinforcement Learning

Recursive Reward Improvement via Adversarial Meta-Prompt Evolution

Expert-Guided Rubrics as Dense Reward Signals for Reinforcement Learning in Non-Verifiable Domains
该论文提出将少量专家(SME)评判转化为密集奖励信号，并与检索指标结合成适用于GRPO的混合奖励，以解决非可验证领域（如金融服务检索）中稀疏奖励导致的梯度消失问题，实验显示Recall@3提升最多6%、MRR@3提升6.7%。