Continually Post-training Small Models via Synthetic Self-distillation

Hugging Face Daily Papers Papers
continual-learning self-distillation small-models post-training knowledge-retention fine-tuning

Summary

该论文提出Synthetic Self-Distillation (SynthSelf)，通过借助同家族稍大的对齐模型进行合成自蒸馏，实现小型模型的持续后训练，在保持原有分布的同时兼顾新能力获取与知识保留，并在多个基准上超越现有方法。

Continual post-training faces a dual challenge of acquisition and retention: models must learn new capabilities while preserving prior knowledge. While Self-Distillation Fine-Tuning (SDFT) mitigates this using self-supervision with in-context demonstrations, it relies on strong in-context learning and instruction-following abilities. Consequently, SDFT fails on smaller models critical for low-latency, on-device use. To enable continual learning at smaller scales, we first identify the root cause of failure. We show that poor acquisition and forgetting correlate linearly with the post-trained model's distribution shift, revealing the necessity of in-distribution learning. We thus introduce Synthetic Self-Distillation (SynthSelf), imitating SDFT's in-distribution rewrite by outsourcing the task to a minimally larger, same-family aligner. Coupled with off-policy supervision and rejection sampling, this provides the high-quality supervision small models require without breaking their original distribution. Evaluations spanning 4 acquisition datasets (including a mixture) and 14 retention benchmarks show SynthSelf consistently outperforms 6 methods (including off- and on-policy distillation) across 3 small model scales. Extensive further analyses confirm SynthSelf paves the way toward effective small-model continual learning.
Original Article
View Cached Full Text

Cached at: 10/12/26, 06:54 AM

Paper page - Continually Post-training Small Models via Synthetic Self-distillation

Source: https://huggingface.co/papers/2610.13113

Abstract

Continualpost-trainingfacesadualchallengeofacquisitionandretention:modelsmustlearnnewcapabilitieswhilepreservingpriorknowledge.WhileSelf-DistillationFine-Tuning(SDFT)mitigatesthisusingself-supervisionwithin-contextdemonstrations,itreliesonstrongin-contextlearningandinstruction-followingabilities.Consequently,SDFTfailsonsmallermodelscriticalforlow-latency,on-deviceuse.Toenablecontinuallearningatsmallerscales,wefirstidentifytherootcauseoffailure.Weshowthatpooracquisitionandforgettingcorrelatelinearlywiththepost-trainedmodel’sdistributionshift,revealingthenecessityofin-distributionlearning.WethusintroduceSyntheticSelf-Distillation(SynthSelf),imitatingSDFT’sin-distributionrewritebyoutsourcingthetasktoaminimallylarger,same-familyaligner.Coupledwithoff-policysupervisionandrejectionsampling,thisprovidesthehigh-qualitysupervisionsmallmodelsrequirewithoutbreakingtheiroriginaldistribution.Evaluationsspanning4acquisitiondatasets(includingamixture)and14retentionbenchmarksshowSynthSelfconsistentlyoutperforms6methods(includingoff-andon-policydistillation)across3smallmodelscales.ExtensivefurtheranalysesconfirmSynthSelfpavesthewaytowardeffectivesmall-modelcontinuallearning.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.13113

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.13113 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.13113 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.13113 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Self-Distillation Enables Continual Learning [pdf]

Hacker News Top

Introduces Self-Distillation Fine-Tuning (SDFT), a method that enables on-policy learning from demonstrations to achieve continual learning without catastrophic forgetting, outperforming supervised fine-tuning.

MIMESIS: Learning User Simulators as Training Environments for Interactive Agents

Hugging Face Daily Papers

本文提出MIMESIS，一个基于真实用户对话与显式推理监督训练的专用用户模拟器（9B模型），在行为保真度上超越前沿模型；研究还将该模拟器冻结后用于多轮强化学习训练智能体，并提出Coached On-Policy Self-Distillation（CSD）方法，在九个未见用户模拟器上实现更强泛化性能。