Enhancing Rubric-based RL via Self-Distillation

Hugging Face Daily Papers Papers
reinforcement-learning self-distillation llm rubric-based-rl open-ended-tasks

Summary

Introduces CriPO, a method that enhances rubric-based reinforcement learning for LLMs via on-policy self-distillation, addressing unexplored and suppressed criteria. Experiments on medicine and science benchmarks show stronger final performance with roughly 2x fewer optimization steps.

Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks. A widely recognized limitation of rubric-based RL is limited exploration: criteria that no rollout manages to satisfy (Unexplored Criteria, UC) receive no optimization signal. Recent methods address this by incorporating rubric information as external guidance during rollout, yet they introduce a train-inference mismatch: the policy is optimized on rollouts produced under external guidance while this guidance is absent at inference time, causing error accumulation through autoregressive decoding. Moreover, these exploration-focused approaches overlook a fundamentally different failure mode that we term Suppressed Criteria (SC) -- criteria that are satisfied by some rollouts yet whose learning signals are lost during optimization because scalar reward aggregation assigns them non-positive aggregate advantages. Our analysis reveals that SC are remarkably prevalent: over 57% of samples exhibit this failure mode throughout training, with an average of 1.8 SC per sample. To simultaneously address both UC and SC without introducing training-inference mismatch, we propose Criterion-Distilled Policy Optimization (CriPO), which enhances rubric-based RL via on-policy self-distillation. For UC, CriPO constructs a criterion-injection self-teacher and computes a localized forward-KL loss to inject missing behaviors into the policy. For SC, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values, preserving useful patterns that would otherwise be suppressed. Experiments on medicine and science benchmarks demonstrate that CriPO consistently outperforms rubric-based RL, achieving stronger final performance with approximately 2times fewer optimization steps.
Original Article
View Cached Full Text

Cached at: 08/03/26, 05:30 AM

Paper page - Enhancing Rubric-based RL via Self-Distillation

Source: https://huggingface.co/papers/2607.18082

Abstract

Rubric-basedRLhasrecentlyshownpromiseinimprovingLLMsonopen-endedtasks.Awidelyrecognizedlimitationofrubric-basedRLislimitedexploration:criteriathatnorolloutmanagestosatisfy(UnexploredCriteria,UC)receivenooptimizationsignal.Recentmethodsaddressthisbyincorporatingrubricinformationasexternalguidanceduringrollout,yettheyintroduceatrain-inferencemismatch:thepolicyisoptimizedonrolloutsproducedunderexternalguidancewhilethisguidanceisabsentatinferencetime,causingerroraccumulationthroughautoregressivedecoding.Moreover,theseexploration-focusedapproachesoverlookafundamentallydifferentfailuremodethatwetermSuppressedCriteria(SC)--criteriathataresatisfiedbysomerolloutsyetwhoselearningsignalsarelostduringoptimizationbecausescalarrewardaggregationassignsthemnon-positiveaggregateadvantages.OuranalysisrevealsthatSCareremarkablyprevalent:over57%ofsamplesexhibitthisfailuremodethroughouttraining,withanaverageof1.8SCpersample.TosimultaneouslyaddressbothUCandSCwithoutintroducingtraining-inferencemismatch,weproposeCriterion-DistilledPolicyOptimization(CriPO),whichenhancesrubric-basedRLviaon-policyself-distillation.ForUC,CriPOconstructsacriterion-injectionself-teacherandcomputesalocalizedforward-KLlosstoinjectmissingbehaviorsintothepolicy.ForSC,CriPOemploysacounterfactualself-teachertolocatecriterion-relevanttokensinnegative-advantagerolloutsandflipstheirtoken-leveladvantagestopositivevalues,preservingusefulpatternsthatwouldotherwisebesuppressed.ExperimentsonmedicineandsciencebenchmarksdemonstratethatCriPOconsistentlyoutperformsrubric-basedRL,achievingstrongerfinalperformancewithapproximately2timesfeweroptimizationsteps.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.18082

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.18082 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.18082 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.18082 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Rubrics as Privileged Information for Open-Ended Generation

arXiv cs.LG

This paper introduces Rubrics as Privileged Information (RuPI), extending on-policy self-distillation to open-ended generation by conditioning the teacher on rubrics as soft privileged information. The method outperforms rubric-as-reward RL and reference-completion distillation across multiple LLMs and benchmarks like HealthBench.

Rubric-based On-policy Distillation

Hugging Face Daily Papers

This paper introduces ROPD, a rubric-based on-policy distillation framework that achieves superior sample efficiency compared to traditional logit-based methods. It enables model alignment in black-box scenarios by using structured semantic rubrics instead of teacher logits.

RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards

Hugging Face Daily Papers

This paper introduces RubricEM, a reinforcement learning framework that uses rubric-guided policy decomposition and reflection-based meta-policy evolution to train deep research agents for long-form tasks. The resulting RubricEM-8B model demonstrates strong performance on long-form research benchmarks by leveraging stage-aware planning and denser semantic feedback.