Tag
A plug-and-play repo implementing self-distillation with an EMA teacher that scales to large models by combining LoRA adapters with DeepSpeed ZeRO-3, avoiding the memory and latency overhead of holding separate full teacher and student models. Tested on Qwen3.5 and ToolAlpaca, it includes adapter switching, EMA checkpointing, and memory-optimized DeepSpeed configs.