Tag
KDFP: A first-principles approach to knowledge distillation in large language models Source: https://arxiv.org/html/2610.10854 Ryan Swift & Konstantinos PsounisAffiliation:Thomas Lord Department of Computer ScienceAffiliation:University of Southern CaliforniaAffiliation:Los Angeles, CA 90007, USAEmail:[{ryanswif,kpsounis}@usc.edu](mailto:) ###### Abstract Knowledge distillation is an established technique for improving the capabilities of small, efficient student models by training them with the representations of larger, more capable teacher models. Much of the recent work in the distillation of large language models (LLMs) has focused on distilling abilities learned during post-training, such as instruction following, chain-of-thought reasoning, and tool usage. This has left a large research gap in general knowledge distillation for LLMs, which is essential for developing efficient and private systems suitable for deployment on edge devices. We take a first-principles approach, evaluating previous lessons from prior works and conducting new explorations to develop a distillation methodology suitable for modern LLMs. We present KDFP, a novel methodology for white-box general knowledge distillation in LLMs. We demonstrate that KDFP outperforms existing methods by 1.6% – 4.9% across 9 benchmarks while increasing training efficiency by up to 99.1% through ephemeral parameter reduction.
论文指出权重量化中降低重建损失并不总能提升模型性能，甚至可能因输入激活分布偏移而劣化，并提出分布鲁棒量化（DRQ）后处理优化框架，通过在约束激活分布集上最小化最坏情况重建损失来改进 AWQ、GPTQ、ParoQuant 等六种 PTQ 方法的下游表现，且不增加推理开销。
The paper introduces Trajectory-Anchored Pruning (TAP), the first structural pruning framework for RL-trained long-horizon agents, which couples channel pruning with trajectory-anchored on-policy recovery. With 60% of FFN channels removed, TAP retains 99.2% and 88.0% of dense 7B agents' task success on ALFWorld and WebShop while cutting GPU time per successful task by roughly 22% and 17%.
OnlineQAT 采用两阶段框架：先通过块级 QAT 得到低位初始化，再在学生自生成的轨迹上用冻结的全精度教师进行反向 KL 的在线蒸馏，从而缓解固定补全训练与部署时学生自采样前缀之间的分布错配。在 Qwen3-1.7B 上，OnlineQAT 在 W3A16（57.28）和 W2A16（32.52）设置中取得对比方法中的最佳平均分，较 ReasoningQAT 分别提升 2.90 和 0.44 分。
LRCC proposes token-dependent compute allocation by training lightweight routers over nested low-rank paths in frozen pretrained LLMs, improving perplexity and downstream accuracy over static low-rank compression at matched active-parameter budgets and decoding latency on Llama and Qwen models.
该论文提出 reViT，通过在每个递归深度用一个共享专家库的凸组合来表示 FFN，使单个 Transformer 块以递归方式应用即可在相近推理 FLOPs 下匹配全深度视觉编码器的精度，并支持弹性深度部署与权重量级压缩。
The post discusses how quantization is shifting from fixed bit-widths to adaptive schemes: Unsloth Dynamic already compresses sensitive weights lightly and insensitive ones aggressively, while GSQ + RCO trains a network to use gradients to decide how and at what bit-width each weight is compressed, with IQ3_S performance nearing BF16 on Qwen3.8-Flash-Next-GSQ-RCO-GGUF.
A project demonstrates a 2.78-trillion-parameter model that can run on just 8GB of memory using a pure C99 inference engine with no BLAS, frameworks, or GPU reliance, trading speed for extreme memory efficiency (about 26.5 seconds/token at 8GB, 5.6 seconds/token with 128GB+).
SoloQ提出一种免校准的量化框架，将权重和激活映射到归一化旋转基底，实现数据无关的4比特量化，在扩散语言模型上保持精度并最多降低2.61倍峰值内存、加速2.24倍推理。
FastOPD proposes an on-policy distillation framework that compresses large Vision-Language-Action (VLA) foundation models into lightweight students with few inference steps, retaining 84% of π₀.₅'s LIBERO performance at a 78.1% latency reduction and deploying a compact student distilled from MolmoAct2 on a real robot.
TORQUE 提出一种在旋转前后联合优化保留哪些坐标以高精度的量化框架，在固定比特预算下证明 top-k 保留可最小化误差上界，并在 KV 缓存压缩、激活压缩和近邻检索上取得更好的精度-存储权衡。
The paper introduces HeadGuard, a composable method that protects a small subset of KV heads in high precision while applying low-bit quantization to the rest of a VLM's image KV cache, substantially recovering accuracy lost by aggressive 2-bit quantization across multiple models and benchmarks.
QuantMLA 提出一种功能对齐的双路径量化框架，对 MLA 的内容路径与 RoPE 路径进行低比特 KV 缓存压缩，首次实现 INT4 联合缓存且精度几乎无损，并通过自定义低比特注意力内核在 128K 上下文下获得 3.59 倍压缩和 5.168 倍吞吐提升。
A 1.6 MB model using ternary 3:4 weights runs on WebGPU in the browser, playing Connect Four as well as a 7.8 MB int8 model, demonstrating the effectiveness of ternary formats for lightweight AI deployment.
Released GSQ-RCO quantized and expert-pruned versions of Qwen3.8-Flash-Next, achieving BF16-level performance at reduced bit-widths and enabling deployment on smaller hardware.
ToMoE is a method that converts dense large language models into mixture-of-experts models through dynamic structural pruning, achieving strong performance without weight updates and outperforming state-of-the-art pruning and MoE techniques.
FermionResearch released Phonon-2, the most accurate open English speech recognition model under 900 MB, averaging 5.21% word error on the Open ASR Leaderboard while matching its 2.5 GB teacher through ~2.1-bit quantization.
This paper analyzes how prompt template selection during knowledge distillation affects the safety alignment of student large language models, finding that chat templates lead to greater degradation compared to non-chat templates across multiple models and benchmarks.
PrismQuant introduces a quantizer-aware rotation framework that aligns activation eigenspace with grouped INT4 quantizers, achieving state-of-the-art performance on large language models like Llama and Qwen, with significant speedups in inference.
The paper proposes LT-OPD, an on-policy self-distillation framework for extreme visual token reduction in multimodal large language models, significantly improving performance under low token budgets while reducing KV-cache usage and FLOPs.