Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift

Hugging Face Daily Papers Papers
off-policy token-level faithful-generation distribution-shift summarization machine-translation lora

Summary

Proposes Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm for faithful generation that reframes post-training as token-level correctness prediction, achieving strong out-of-distribution generalization across summarization and machine translation tasks.

We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task. Our key intuition is that by training the model to distinguish good and bad tokens in a response, we naturally guide the model towards generating good tokens, while avoiding the pitfalls that come with directly training the model to generate off-policy tokens. Experiments on document summarization tasks show that TOPL achieves strong out-of-distribution generalization across 11 datasets against a diverse set of sequence-level and token-level baselines. We further demonstrate that TOPL transfers effectively to machine translation, suggesting that its benefits generalize across different faithful generation tasks. Through ablation studies, we confirm that our token-level learning signal is critical to good performance; sequence-level analogues do not confer similar benefits. Finally, we show that TOPL induces interpretable model updates: the LoRA adapters learned through TOPL function as linear classification heads and steering vectors.
Original Article
View Cached Full Text

Cached at: 07/21/26, 06:35 AM

Paper page - Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift

Source: https://huggingface.co/papers/2607.17524

Abstract

WeproposeToken-LevelOff-PolicyLabeling(TOPL),anoff-policytrainingparadigmthatreframespost-trainingasatoken-levelcorrectnesspredictiontask.Ourkeyintuitionisthatbytrainingthemodeltodistinguishgoodandbadtokensinaresponse,wenaturallyguidethemodeltowardsgeneratinggoodtokens,whileavoidingthepitfallsthatcomewithdirectlytrainingthemodeltogenerateoff-policytokens.ExperimentsondocumentsummarizationtasksshowthatTOPLachievesstrongout-of-distributiongeneralizationacross11datasetsagainstadiversesetofsequence-levelandtoken-levelbaselines.WefurtherdemonstratethatTOPLtransferseffectivelytomachinetranslation,suggestingthatitsbenefitsgeneralizeacrossdifferentfaithfulgenerationtasks.Throughablationstudies,weconfirmthatourtoken-levellearningsignaliscriticaltogoodperformance;sequence-levelanaloguesdonotconfersimilarbenefits.Finally,weshowthatTOPLinducesinterpretablemodelupdates:theLoRAadapterslearnedthroughTOPLfunctionaslinearclassificationheadsandsteeringvectors.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.17524

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.17524 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.17524 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.17524 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Trust Region On-Policy Distillation

Hugging Face Daily Papers

The paper proposes Trust Region On-Policy Distillation (TrOPD) to stabilize on-policy distillation of large language models by using trust regions, outlier estimation, and off-policy guidance, outperforming existing methods on reasoning and code generation benchmarks.

Latent On-Policy Self-Distillation

Hugging Face Daily Papers

This paper introduces Latent On-Policy Self-Distillation (LOPD), a method that makes the teacher's privileged context learnable end-to-end from experience, providing dense token-level supervision to enhance agent performance and efficiency in agentic tool use and code generation.

Tail-Aware Top-$k$ On-Policy Distillation

arXiv cs.LG

This paper proposes Tail-Aware Top-k On-Policy Distillation (TA-OPD) to address the loss of tail probability in on-policy distillation for language models, improving downstream accuracy on benchmarks.

ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

arXiv cs.LG

ShortOPD proposes a short-to-long on-policy distillation schedule that recovers pruned LLMs for free-form generation by focusing training on effective prefixes, achieving up to 9x improvement over unrecovered models and matching long-horizon distillation with a quarter of the training time.