Smaller Models, Better Rejects: Preference Distillation Scaling

Hugging Face Daily Papers Papers
preference-distillation dpo knowledge-distillation language-models reward-modeling scaling-laws

Summary

This paper finds that smaller frozen models can generate better negative samples (rejects) for preference distillation than self-generated responses, reducing inference cost while training stronger 7B–72B students on code generation and math reasoning. The authors derive a finite-horizon utility bound for DPO in a linearized feature model to explain why task structure and lower-likelihood candidate selection improve reject quality.

Preference distillation typically treats a teacher response as preferred and the student's own response as rejected. This assumes that self-generated failures are the most informative negatives and that rejects must come from a model at least as large as the student, making generation costly at scale. We find neither assumption holds: across students from 7B to 72B, smaller frozen models generate rejects with less inference compute yet train stronger students than self-generated rejects, before and after sequence-level knowledge distillation, on code generation and mathematical reasoning. To explain this result, we derive a finite-horizon utility bound for Direct Preference Optimization in a linearized feature model. The bound characterizes favorable reject distributions and motivates three interventions. First, mixing rejects from smaller and student-scale models improves performance as the smaller model's share increases. Second, reassigning rejects to other prompts and shuffling their code tokens still outperform length-matched gibberish, showing that task structure contributes to reject utility. Third, selecting candidates with lower likelihood under the reference policy improves net transfer when higher-likelihood candidates provide less useful contrast. Lower-likelihood selections outperform higher-likelihood ones for every source. These results suggest that effective rejects preserve task structure while limiting coupling to the reference policy, and that smaller frozen models can provide them at low cost.
Original Article
View Cached Full Text

Cached at: 10/02/26, 08:27 AM

Paper page - Smaller Models, Better Rejects: Preference Distillation Scaling

Source: https://huggingface.co/papers/2609.38987 Authors:

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

,

Abstract

Preferencedistillationtypicallytreatsateacherresponseaspreferredandthestudent’sownresponseasrejected.Thisassumesthatself-generatedfailuresarethemostinformativenegativesandthatrejectsmustcomefromamodelatleastaslargeasthestudent,makinggenerationcostlyatscale.Wefindneitherassumptionholds:acrossstudentsfrom7Bto72B,smallerfrozenmodelsgeneraterejectswithlessinferencecomputeyettrainstrongerstudentsthanself-generatedrejects,beforeandaftersequence-levelknowledgedistillation,oncodegenerationandmathematicalreasoning.Toexplainthisresult,wederiveafinite-horizonutilityboundforDirectPreferenceOptimizationinalinearizedfeaturemodel.Theboundcharacterizesfavorablerejectdistributionsandmotivatesthreeinterventions.First,mixingrejectsfromsmallerandstudent-scalemodelsimprovesperformanceasthesmallermodel’sshareincreases.Second,reassigningrejectstootherpromptsandshufflingtheircodetokensstilloutperformlength-matchedgibberish,showingthattaskstructurecontributestorejectutility.Third,selectingcandidateswithlowerlikelihoodunderthereferencepolicyimprovesnettransferwhenhigher-likelihoodcandidatesprovidelessusefulcontrast.Lower-likelihoodselectionsoutperformhigher-likelihoodonesforeverysource.Theseresultssuggestthateffectiverejectspreservetaskstructurewhilelimitingcouplingtothereferencepolicy,andthatsmallerfrozenmodelscanprovidethematlowcost.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.38987

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.38987 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.38987 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.38987 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

DecisionRank: On-Policy Distillation for Decision-Model Rerankers

arXiv cs.CL

Meta's DecisionRank paper introduces item-space on-policy distillation to transfer a generative LLM teacher's ranking knowledge into a single-token decision-model reranker, achieving state-of-the-art results on TREC DL, BEIR, and BRIGHT while running up to 24.5x faster than its generative counterpart and surpassing its 27B teacher at just 1.7B parameters.

Reasoning Compression with Mixed-Policy Distillation

arXiv cs.AI

This paper proposes Mixed-Policy Distillation (MPD), a framework that transfers concise reasoning behaviors from large teacher models to smaller student models, reducing token usage by up to 27.1% while improving performance.

Smaller Models are Natural Explorers for Policy-Level Diversity in GRPO

Hugging Face Daily Papers

The S2L-PO framework uses smaller models as natural explorers to enhance policy diversity in GRPO for training large language models. It achieves faster convergence and improves accuracy on mathematical reasoning benchmarks while reducing rollout compute.