CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization

Hugging Face Daily Papers Papers
counterfactual-replay token-level-credit reinforcement-learning language-models grpo rubric-guided-policy-optimization

Summary

CoRT proposes a token-level credit weighting method for GRPO that uses counterfactual replay to compute token-wise log-likelihood contrasts, redistributing the signed advantage across tokens without an auxiliary scorer, achieving average gains of 4.4 percentage points over response-level GRPO.

Rubric-based reinforcement learning enriches language model training by evaluating model outputs against explicit criteria. Yet in GRPO-style pipelines, these structured judgments are reduced to a scalar response-level reward and converted into a response-level advantage, which is broadcast uniformly to all generated tokens. This leaves no explicit mechanism for allocating credit within a response, even when different criteria are grounded in different spans, formatting decisions, or semantic choices. We propose CoRT, a token-level credit weighting method for rubric-conditioned GRPO. Instead of training an auxiliary token scoring model, CoRT uses counterfactual replay to rescore the same sampled response under the original rubric-conditioned prompt and a matched criteria-free prompt. The resulting tokenwise log-likelihood contrasts serve as a proxy for dependence on the rubric context. CoRT maps these contrasts to bounded, response-normalized weights and uses them to redistribute the signed GRPO advantage across tokens, without introducing an auxiliary scorer or changing the response-level reward. Experiments across instruction-tuned models and reward granularities show that CoRT improves over matched response-level GRPO in the vast majority of comparisons, with an average gain of 4.4 percentage points. The method remains competitive with learned token-level credit baselines while avoiding a separate relevance-learning stage. These results suggest that policy-internal counterfactual likelihood contrasts provide an effective training signal for within-response credit allocation while retaining the simplicity and stability of GRPO.
Original Article
View Cached Full Text

Cached at: 07/30/26, 05:46 AM

Paper page - CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization

Source: https://huggingface.co/papers/2607.25659 Published on Jul 28

·

Submitted byhttps://huggingface.co/jwhe

quinnon Jul 30

Abstract

Rubric-basedreinforcementlearningenricheslanguagemodeltrainingbyevaluatingmodeloutputsagainstexplicitcriteria.YetinGRPO-stylepipelines,thesestructuredjudgmentsarereducedtoascalarresponse-levelrewardandconvertedintoaresponse-leveladvantage,whichisbroadcastuniformlytoallgeneratedtokens.Thisleavesnoexplicitmechanismforallocatingcreditwithinaresponse,evenwhendifferentcriteriaaregroundedindifferentspans,formattingdecisions,orsemanticchoices.WeproposeCoRT,atoken-levelcreditweightingmethodforrubric-conditionedGRPO.Insteadoftraininganauxiliarytokenscoringmodel,CoRTusescounterfactualreplaytorescorethesamesampledresponseundertheoriginalrubric-conditionedpromptandamatchedcriteria-freeprompt.Theresultingtokenwiselog-likelihoodcontrastsserveasaproxyfordependenceontherubriccontext.CoRTmapsthesecontraststobounded,response-normalizedweightsandusesthemtoredistributethesignedGRPOadvantageacrosstokens,withoutintroducinganauxiliaryscorerorchangingtheresponse-levelreward.Experimentsacrossinstruction-tunedmodelsandrewardgranularitiesshowthatCoRTimprovesovermatchedresponse-levelGRPOinthevastmajorityofcomparisons,withanaveragegainof4.4percentagepoints.Themethodremainscompetitivewithlearnedtoken-levelcreditbaselineswhileavoidingaseparaterelevance-learningstage.Theseresultssuggestthatpolicy-internalcounterfactuallikelihoodcontrastsprovideaneffectivetrainingsignalforwithin-responsecreditallocationwhileretainingthesimplicityandstabilityofGRPO.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.25659

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.25659 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.25659 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.25659 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization

Hugging Face Daily Papers

This paper introduces Guidance Contrastive Policy Optimization (GCPO), a novel algorithm that enables per-token credit assignment in reinforcement learning by contrasting model predictions under positive and negative prompts, consistently outperforming GRPO and DAPO baselines on text-to-image generation and chain-of-thought reasoning benchmarks.