From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation

Hugging Face Daily Papers Papers
on-policy-distillation knowledge-distillation reinforcement-learning gradient-analysis quantization language-models multi-teacher

Summary

This paper investigates multi-teacher on-policy distillation (MOPD) by analyzing how teacher signals translate into parameter changes when combining RL-trained teachers into Qwen3-1.7B and SmolLM3-3B students. It finds that loss averaging implicitly weights responses by length, Adam's first moment largely erases differences between teacher gradients (cosine similarity 0.83), BF16 rounding hides most weight updates, and top-64 intersection KL gradients can match full-vocabulary gradients depending on the averaging rule used.

Multi-teacher on-policy distillation (MOPD) aims to combine the strengths of RL-trained teachers in a single student, but how teacher signals affect parameter changes remains underexplored. We study Qwen3-1.7B with four domain teachers trained with RL from the same initialization as the student, comparing gradients, optimizer updates, and task learning curves, with additional SmolLM3-3B diagnostics. We find that several factors influence teacher signals. First, loss averaging implicitly weights responses: token averaging favors longer responses, and equalizing domain contributions retains this weighting within domains. Second, Adam's first moment reduces differences in parameter updates: the cosine similarity is 0.83 between teachers and 0.96 between averaging rules, despite differences in raw gradients. Third, BF16 rounding hides small changes: about 97\% of FP32 master weights differ from initialization, but only 7--11\% of BF16 weights do. Finally, the top-64 intersection KL gradient closely matches Qwen's full-vocabulary gradient, but the effect on task performance depends on averaging: mathematics accuracy is 2.6 points higher than with sampled-token policy-gradient (PG) under response averaging and 2.1 points lower under global token averaging.
Original Article
View Cached Full Text

Cached at: 10/05/26, 04:49 PM

Paper page - From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation

Source: https://huggingface.co/papers/2610.02179

Abstract

Multi-teacheron-policydistillation(MOPD)aimstocombinethestrengthsofRL-trainedteachersinasinglestudent,buthowteachersignalsaffectparameterchangesremainsunderexplored.WestudyQwen3-1.7BwithfourdomainteacherstrainedwithRLfromthesameinitializationasthestudent,comparinggradients,optimizerupdates,andtasklearningcurves,withadditionalSmolLM3-3Bdiagnostics.Wefindthatseveralfactorsinfluenceteachersignals.First,lossaveragingimplicitlyweightsresponses:tokenaveragingfavorslongerresponses,andequalizingdomaincontributionsretainsthisweightingwithindomains.Second,Adam’sfirstmomentreducesdifferencesinparameterupdates:thecosinesimilarityis0.83betweenteachersand0.96betweenaveragingrules,despitedifferencesinrawgradients.Third,BF16roundinghidessmallchanges:about97\%ofFP32masterweightsdifferfrominitialization,butonly7--11\%ofBF16weightsdo.Finally,thetop-64intersectionKLgradientcloselymatchesQwen’sfull-vocabularygradient,buttheeffectontaskperformancedependsonaveraging:mathematicsaccuracyis2.6pointshigherthanwithsampled-tokenpolicy-gradient(PG)underresponseaveragingand2.1pointslowerunderglobaltokenaveraging.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.02179

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.02179 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.02179 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.02179 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training

Hugging Face Daily Papers

MOPD proposes a multi-teacher on-policy distillation paradigm for LLM post-training, enabling efficient integration of multiple domain capabilities by distilling specialized RL teachers into a student model using its own rollouts. It outperforms existing methods like Mix-RL and Cascade RL, and has been deployed in industrial-scale models.

Latent-MOPD: Latent Multi-Teacher On-Policy Distillation

arXiv cs.LG

The paper introduces Latent-MOPD, the first representation-level multi-teacher on-policy distillation method for LLMs, which integrates multiple specialist teachers through both their output distributions and hidden states without extra teacher training, outperforming token-only, representation-only, and averaging baselines across nine math, code, and logic benchmarks.