Rubric Rewards from Item Response Theory

Hugging Face Daily Papers Papers
reinforcement-learning rubric-rewards item-response-theory reward-modeling judge-efficiency rlhf

Summary

The paper introduces Rubric Response Theory (RRT), which replaces additive rubric scoring with a two-parameter item response model plus a Response Parameter Network (RPN) updated via online EM, achieving higher macro criterion scores than GRPO on Medical and Science benchmarks while halving judge requests.

Many language tasks have no single answer that can be checked automatically. Rubrics provide criteria for judging responses to these tasks. For reinforcement learning, the resulting verdicts must be combined into a scalar reward. A common approach sums the points assigned to satisfied criteria. Distinct verdict patterns can thus receive the same reward, and the fixed points encode how much each criterion should count, not how strongly its verdict distinguishes the current rollouts. Beyond this aggregation problem, judging the full rubric needs more judge requests as the criterion count grows. To address these limitations, Rubric Response Theory (RRT) measures quality and selects criteria when rubric criteria are monotone indicators of a shared target. Rather than adding assigned points, RRT uses a two parameter item response model that treats the verdict pattern as evidence about scalar quality specific to the rubric. Under this model, its likelihood score maximizes the local signal-to-noise ratio for quality. Its Response Parameter Network (RPN) reads the prompt and criterion text to predict criterion difficulty and discrimination. As the policy distribution changes during training, RRT uses online expectation maximization to update the RPN from current rollout verdicts. With Qwen3.5-4B as the policy, RRT's macro criterion score across Medical, Science, Rubrics as Rewards Science, and RubricBench is 1.7 points above that of group relative policy optimization (GRPO). On hard and very hard criteria in Medical and Science, RRT gains 2.8 to 5.6 points over GRPO. At half the criterion budget, adaptive Fisher selection with a frozen RPN keeps the macro criterion score across four datasets within 0.1 points of GRPO with full judging. These results show RRT can reduce judge requests while remaining competitive with GRPO.
Original Article
View Cached Full Text

Cached at: 10/01/26, 08:24 PM

Paper page - Rubric Rewards from Item Response Theory

Source: https://huggingface.co/papers/2609.35646

Abstract

Manylanguagetaskshavenosingleanswerthatcanbecheckedautomatically.Rubricsprovidecriteriaforjudgingresponsestothesetasks.Forreinforcementlearning,theresultingverdictsmustbecombinedintoascalarreward.Acommonapproachsumsthepointsassignedtosatisfiedcriteria.Distinctverdictpatternscanthusreceivethesamereward,andthefixedpointsencodehowmucheachcriterionshouldcount,nothowstronglyitsverdictdistinguishesthecurrentrollouts.Beyondthisaggregationproblem,judgingthefullrubricneedsmorejudgerequestsasthecriterioncountgrows.Toaddresstheselimitations,RubricResponseTheory(RRT)measuresqualityandselectscriteriawhenrubriccriteriaaremonotoneindicatorsofasharedtarget.Ratherthanaddingassignedpoints,RRTusesatwoparameteritemresponsemodelthattreatstheverdictpatternasevidenceaboutscalarqualityspecifictotherubric.Underthismodel,itslikelihoodscoremaximizesthelocalsignal-to-noiseratioforquality.ItsResponseParameterNetwork(RPN)readsthepromptandcriteriontexttopredictcriteriondifficultyanddiscrimination.Asthepolicydistributionchangesduringtraining,RRTusesonlineexpectationmaximizationtoupdatetheRPNfromcurrentrolloutverdicts.WithQwen3.5-4Basthepolicy,RRT’smacrocriterionscoreacrossMedical,Science,RubricsasRewardsScience,andRubricBenchis1.7pointsabovethatofgrouprelativepolicyoptimization(GRPO).OnhardandveryhardcriteriainMedicalandScience,RRTgains2.8to5.6pointsoverGRPO.Athalfthecriterionbudget,adaptiveFisherselectionwithafrozenRPNkeepsthemacrocriterionscoreacrossfourdatasetswithin0.1pointsofGRPOwithfulljudging.TheseresultsshowRRTcanreducejudgerequestswhileremainingcompetitivewithGRPO.

View arXiv pageView PDFGitHub4Add to collection

Get this paper in your agent:

hf papers read 2609\.35646

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.35646 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.35646 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.35646 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning

Hugging Face Daily Papers

MetaRubric 通过交替进行证据感知的策略优化与响应引导的评分标准适配，解决基于评分标准的强化学习中「空洞奖励（Vacuous Credit）」问题，在 PubMedQA 上比静态评委 GRPO 提升 6.00–20.40 个百分点。

C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences

Hugging Face Daily Papers

C2 proposes a scalable rubric-augmented reward modeling framework that trains a cooperative rubric generator and critical verifier exclusively from binary preferences, eliminating the need for costly rubric annotations while achieving up to 6.5 point gains on RM-Bench.

Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges

Hugging Face Daily Papers

本文提出 ReVeR，一种点式（pointwise）rubric 奖励模型，仅通过分析指令本身生成硬性规则与分级原则，并借助工具增强的独立评判来评估响应，避免了从候选响应中推导标准所引入的偏置。实验表明 ReVeR 在偏好准确率、测试时扩展及下游策略优化方面优于成对 rubric 奖励模型。