Paper page - C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences

Source: https://huggingface.co/papers/2604.13618

Abstract

Cooperative yet Critical reward modeling (C2) enhances reward model reliability by enabling critical collaboration between a reward model and a rubric generator trained exclusively from binary preferences, achieving superior performance without requiring costly rubric annotations.

Rubric-augmented verification guidesreward models (https://huggingface.co/papers?q=reward%20models)with explicit evaluation criteria, yielding more reliable judgments than single-model verification. However, most existing methods require costly rubric annotations, limiting scalability. Moreover, we find thatrubric generation (https://huggingface.co/papers?q=rubric%20generation)is vulnerable to a failure of cooperation; low-quality rubrics actively misleadreward models (https://huggingface.co/papers?q=reward%20models)rather than help. Inspired by the principle ofcooperative communication (https://huggingface.co/papers?q=cooperative%20communication), we propose Cooperative yet Critical reward modeling (C2), a framework that significantly improvesreward model judgments (https://huggingface.co/papers?q=reward%20model%20judgments)by having the reward model critically collaborate with a rubric generator trained solely frombinary preferences (https://huggingface.co/papers?q=binary%20preferences). In C2, we synthesize helpful and misleading rubric pairs by measuring how each rubric shifts the reward model toward or away from the correct preference. Using these contrastive pairs, we train a cooperative rubric generator to propose helpful rubrics, and a critical verifier to assess rubric validity before making its judgment, following only rubrics it deems helpful at inference time. C2 outperforms reasoningreward models (https://huggingface.co/papers?q=reward%20models)trained on the samebinary preferences (https://huggingface.co/papers?q=binary%20preferences), with gains of up to 6.5 points onRM-Bench (https://huggingface.co/papers?q=RM-Bench)and 6.0 points length-controlled win rate onAlpacaEval 2.0 (https://huggingface.co/papers?q=AlpacaEval%202.0). Without external rubric annotations, C2 enables an 8B reward model to match performance achieved with rubrics from a 4times larger model. Overall, our work demonstrates that eliciting deliberate cooperation in rubric-augmented verification makesreward models (https://huggingface.co/papers?q=reward%20models)more trustworthy in a scalable way.

View arXiv page (https://arxiv.org/abs/2604.13618)View PDF (https://arxiv.org/pdf/2604.13618)GitHub1 (https://github.com/asahi-research/C2)Add to collection (https://huggingface.co/login?next=%2Fpapers%2F2604.13618)

Community

Paper author

Paper submitter

3 days ago (https://huggingface.co/papers/2604.13618#69e19ba83ce9e902c47d2e98)

•

edited 3 days ago (https://huggingface.co/papers/2604.13618#69e19ba83ce9e902c47d2e98) Rubrics are a powerful harness for aligning reward models with human judgment, but they come with two problems.

1. Human-annotated rubrics are costly.
2. Auto-generated rubrics are often vague or misleading, and can hurt the RM more than help.

We introduce C2, a reward modeling framework that builds robust rubric-based verification from binary preferences alone. The generator learns which rubrics help the verifier reach the correct judgment, and the verifier learns which rubrics to trust.

Upload images, audio, and videos by dragging in the text input, pasting, orclicking here.

Tap or paste here to upload images

Get this paper in your agent:

hf papers read 2604\.13618

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2604.13618 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2604.13618 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2604.13618 in a Space README.md to link it from this page.

Collections including this paper1