distillation

The paper proposes Latent Sparse Attention (LSA), which hierarchically clusters Keys in latent space so that Queries only need to score against cluster centers. This achieves a 72.4x speedup in Prefill attention and a 42.5x speedup in Decode for a single attention layer of Qwen3.8-27B under 10M ultra-long context, reduces end-to-end first-token latency to about 1/9, and keeps accuracy essentially unchanged. The method includes two variants: distillable D-LSA and natively trained N-LSA, with code and distilled weights open-sourced.

TinyAstral evolves a frozen teacher's agent harness via recursive self-improvement with statistical acceptance tests, then distills the gains into a 4B student that matches the teacher's RoboTwin success (62.8% vs 63.6%) using about one fifth of the accelerator time.

Apriel Voice 27B Preview 是一个开源语音-语言模型，基于 Gemma 4 26B-A4B 后训练检查点构建，通过两阶段训练将语音理解、推理与工具调用统一到共享推理骨干中，并在 Big Bench Audio、Speech-BFCL 和 τ-Voice 等基准上取得有竞争力的成绩。

A Zyphra Research paper introduces Zyphra Hierarchical Attention (ZHA), a subquadratic sparse attention method using a hierarchical tree-structured indexer over the KV cache, which can be adapted to pretrained models via short KL distillation and scales dramatically better beyond 1M context.

提出了异步流映射语言模型（aFMLM），通过token级时间调度将预测顺序编码进单步生成的隐藏计算中，在Sudoku和GSM8K代码生成任务上刷新了一步生成的最先进性能。

SkillMem introduces a compact external latent memory for reusable skills in latent multi-agent reasoning, achieving 38× storage reduction and up to 4.23× inference speedup across benchmarks with Qwen3 models from 4B to 14B.

DecisionRank 使用 item-space on-policy distillation 将生成式教师模型的排序知识迁移到高效决策模型，在 TREC DL、BEIR、BRIGHT 等检索基准上达到最先进性能，同时推理速度显著提升（最高快 24.5 倍），1.7B 参数即可超越 27B 生成式教师。

Anyscale is hosting a webinar on reinforcement learning for LLM post-training, covering RLVR with GRPO, on-policy distillation (OPD/MOPD), terminal agent post-training, and a deep dive into SkyRL's async RL capabilities.

UkisAI announces Swift GLM-5.3-Flash and Swift Ornith 9B, two reasoning-efficient models that cut thinking token usage significantly (-61.4% and -33.5% respectively) while preserving accuracy, using GSQ-RCO penalization of pathological overthinking tokens, RL-based adaptation, and on-policy distillation, with multiple quantization formats released.

IDM: Intrinsic Distribution Matching 提出让生成器自身同时提供分数监督和分布匹配表示的方法，在保留完整 DMD 校正的基础上，对生成器 EMA 特征的均值和逐坐标标准差添加显式损失。在 2 NFE 预算下，基于 Z-Image 的 IDM 在 LongText-CN 和 LongText-EN 上取得最佳分数。

OmniBrain-0 is a lightweight unified embodied agent that coordinates manipulation and navigation through a shared high-level decision model, trained via multi-teacher on-policy distillation and a rubric-guided recursive self-improvement data flywheel, achieving strong cross-domain performance with much lower overhead than its closed-source teacher.

This paper combines cryptanalytic weight-direction extraction with label-based distillation to reconstruct fully connected neural networks, showing that Leaky ReLU enables neuron recovery where ReLU fails, and demonstrating high prediction agreement on synthetic and CIFAR-10 victim models.

讨论 DeepSeek 模型在对话中自称是 ChatGPT，引发关于这是模型蒸馏（distillation）的痕迹还是单纯识别错误的争议。

PrunaAI released 6-step LoRA adapters for Qwen-Image-2.1 that cut image generation from 40 steps to 6, with one variant removing the prompt-upsampling step entirely for ~5x faster end-to-end generation.

TAD introduces Trajectory-only Adversarial Distillation, which improves vision-language-action policies using only a fixed bank of teacher trajectories—no teacher action scores or task rewards—via a discriminator providing on-policy rewards, boosting OpenVLA-OFT success rates substantially while trailing white-box on-policy distillation.

NLD-Uni 提出一个统一的离散扩散大语言模型，通过跨范式在策略蒸馏和渐进式自我精炼扩散，将自回归 VLM 的视觉理解能力与连续流模型的视觉生成能力统一起来，在多项视觉理解与生成任务上达到统一 DLLM 中的最先进性能。

Sebastian Raschka released an 11-hour, eight-video course on building reasoning models from scratch in PyTorch, covering evaluation, inference scaling, reinforcement learning with verifiable rewards, and distillation, serving as a sequel to his earlier Coding LLMs from the Ground Up course.

Nanbeige4.3-33B 是南ベージ系列的旗舰教师模型，宣称在约30B参数级别的开源模型中达到最先进水平，专注于复杂编码与专业协作智能体任务，并为后续小模型蒸馏（如 Nanbeige4.3-3B）奠定基础。

A commentary questioning whether Anthropic's new anti-distillation defense, claimed to block 90-99% of distillation attacks, actually slows Chinese labs down given that high-quality expert datasets remain available for sale overseas by US data companies.

论文提出 OnQ 流水线，通过 on-policy 蒸馏与候选级 Dynamic KL 恢复低比特量化 LLM 的推理能力，在 Qwen3 系列 2/3 比特量化上取得最优九基准平均成绩，2 比特下 Qwen3-8B 平均提升 10.26 分。