distillation

Kimi K3 is a 2.8T parameter open model from Moonshot AI, showing strong benchmark performance but likely over-optimized and lagging behind top closed models by months. It is distilled from Claude and its release may precede an IPO.

AlayaWorld is a 15B-parameter interactive video world model that generates 24-fps video at 540p and 720p, using autoregressive latent chunk generation with bounded visual context and distillation to reduce inference steps. It achieves state-of-the-art performance on the iWorld-Bench for long-horizon generation.

Ben Thompson proposes US legislation to explicitly allow training data as fair use and bar terms of service that forbid distillation, arguing this could help US open models compete with Chinese counterparts, and suggests Alibaba's release of Qwen 3.8 Max may have been influenced by Xi Jinping's call for open source collaboration.

This paper investigates the discrepancy between log-likelihood and generation-based evaluation in distilled hybrid sequence models, showing that perplexity alone can be misleading. It introduces GenDistill, a pipeline for distilling Transformers into Hybrid-KDA models, achieving up to 75% KV cache reduction and 2-4x faster time-to-first-token while retaining 86-90% of teacher accuracy.

This paper introduces a framework for when transfer should be expected in continual learning and proposes Transfer-Selective Replay (TSR), which selects replay data predicted to benefit the incoming task rather than indiscriminately replaying past examples. TSR improves forward transfer while maintaining stability, outperforming existing replay baselines.

This tweet discusses OPD, OPSD, RLSD, and OPD^2 distillation techniques for language models, emphasizing the use of log probability differences to isolate post-training improvements.

Jerry Liu suggests that for any task, a generalized model can be distilled into a specialized one for higher accuracy and lower cost, with automation enabling this process by defining goals and rubrics instead of manual workflow coding.

This paper proposes ReOPD, a method for on-policy distillation of LLM agents that reuses pre-collected teacher trajectories as replayed prefixes, achieving improved efficiency and accuracy without new environment interactions.

RESOURCE2SKILL is a framework that distills executable agent skills from multimodal resources like tutorial videos, code repositories, articles, and artifacts into a hierarchical SkillWiki, improving agent performance by 11.9 percentage points over no-skill agents.

Jack Morris corrects his earlier claim about an open-weight model being trained without distillation from OpenAI or Anthropic, acknowledging that it actually did use a small amount of distillation.

A.L.F.R.E.D. proposes a system that distills knowledge from large models into small ones and routes simple tasks to the small models, achieving performance of 35B models with 2B models while reducing inference cost by 4x.

This paper presents a combined proof-of-mechanism study of ontology-amplified distillation for sovereign enterprise language models and a contextuality-audit method, using a Qwen3.6-27B student adapted via supervised fine-tuning and DPO. The results are underpowered and negative, showing no superiority over frontier baselines and zero contextuality in routing.

OvisOCR2 is a 0.8B parameter end-to-end document parsing model that converts document page images to Markdown, achieving state-of-the-art scores on public benchmarks through a combination of supervised fine-tuning, reinforcement learning, and model fusion.

EasyOPD is a new on-policy distillation framework for large language models that separates configuration, supervision logic, and execution, with support for cross-tokenizer, self-distillation, and step-wise OPD.

This paper introduces MCLASH, a multilingual moral decision-making benchmark, and MET, a theory-grounded prompting method for culture-aware moral reasoning, along with MET-D, a self-distillation training method that improves multilingual moral reasoning across different model families and sizes.

A critique of the practice of fine-tuning AI models on summarized or censored chain-of-thought reasoning traces, arguing that distillation on such traces degrades model quality compared to the base model's actual capabilities.

Satya Nadella criticizes the practice of imposing restrictive terms on AI model distillation, arguing it is ironic given that model providers themselves rely on fair use of public data to train their models.

Introduces Parallel-Distill-Refine (PDR), an inference method where LLMs generate drafts in parallel, distill them into a compact workspace, and refine to achieve better accuracy than long chain-of-thought with lower latency. An 8B model trained with RL using PDR shows significant gains on math benchmarks.

Rilasciata un'interfaccia per gestire due piccoli modelli (4B e 1.7B) che girano localmente sullo smartphone. Il 4B funziona bene su telefoni di fascia alta; il 1.7B ha problemi di stabilità con il reasoning, in fase di miglioramento con fine-tuning approfondito usando 130k esempi e distillazione da un teacher 32B.

This paper explores distilling a large reasoning teacher model (8B) into a small student model (0.6B) for on-device structured text enrichment, achieving substantial speedup while recovering significant quality. The study finds that the reasoning nature of the teacher, rather than its scale, drives improvement in summary quality, but a same-size instruction teacher yields more faithful outputs on certain articles.