Self-distillation with an EMA teacher at scale: ZeRO-3 + PEFT [P]

Reddit r/MachineLearning 10/27/26, 06:16 PM Tools

self-distillation lora peft deepspeed zero-3 ema-teacher open-source

Summary

A plug-and-play repo implementing self-distillation with an EMA teacher that scales to large models by combining LoRA adapters with DeepSpeed ZeRO-3, avoiding the memory and latency overhead of holding separate full teacher and student models. Tested on Qwen3.5 and ToolAlpaca, it includes adapter switching, EMA checkpointing, and memory-optimized DeepSpeed configs.

Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc. We can mitigate this with LoRA finetuning: the backbone is shared & frozen, and we need to maintain two light adapters: one for the student (trained) and one for the teacher (EMA of the student adapter). Getting this to work cleanly under TRL and ZeRO-3 took some fixes, so I packaged it as a plug-and-play repo. It supports: -> robust SDFT implementation that runs under ZeRO-3 -> ZeRO-3 + LoRA integration with adapter switching for teacher/student passes -> Saving and resuming EMA teacher checkpoint -> working deepspeed configs that fix memory spikes when GPU memory is tight I tested on Qwen3.5 and ToolAlpaca. Repo: https://github.com/ifiaposto/sdft_zero3 Feedback/ more examples/ issues welcome!

Original Article

Similar Articles

AMRD: Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition

Hugging Face Daily Papers

This paper introduces AMRD, an adaptive multi-teacher relational distillation method for compressing large self-supervised speech emotion recognition models into lightweight student models for edge devices. It addresses teacher reliability variation and relational structure loss, showing improvements on IEMOCAP and CREMA-D datasets.

Post-Trained MoE Can Skip Half Experts via Self-Distillation

Hugging Face Daily Papers

ZEDA is a low-cost framework that converts post-trained static MoE models into dynamic ones by injecting zero-output experts and using self-distillation, achieving over 50% expert FLOP reduction with marginal accuracy loss on benchmarks.

@rohanpaul_ai: A large MoE model may be wasting half its expert compute on tokens that barely need expert help. In this paper 50% of e…

X AI KOLs Timeline

A new method called Zero-Expert Self-Distillation Adaptation (ZEDA) allows MoE models like Qwen3 and GLM to skip half their expert computations on easy tokens with minimal accuracy loss, achieving ~20% inference speedup by adding dummy experts that output nothing.

[Paper] EdgeRazor: A Lightweight Framework for Large Language Models via Mixed-Precision Quantization-Aware Distillation

Reddit r/LocalLLaMA

EdgeRazor is a lightweight framework for compressing large language models using entropy-guided mixed-precision quantization-aware distillation, achieving 1.88 bits per parameter while preserving teacher model competence and requiring no changes to inference implementations like llama.cpp. The method is demonstrated on small models such as MobileLLM and Qwen variants.

MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation

arXiv cs.LG

The paper proposes Muon-RT, a momentum-free variant of the Muon optimizer that reuses the EMA teacher-student weight displacement in LLM self-distillation, eliminating separate momentum buffers while improving accuracy and reducing persistent training memory on Qwen2.5-7B and Olmo-3-7B.