distillation

This paper proposes scaffold-mediated post-training, a paradigm where procedural scaffolds co-evolve with LLM parameters through discovery, distillation, and dynamic recompilation. On FeatureBench, automatically discovered skills improve pass rate by 8.1pp, with a 27.7% pass rate after distillation.

This paper investigates subliminal learning in language models, showing that biases can transfer from teacher to student via seemingly random synthetic data. The authors find that adding Gaussian noise to weights increases transfer, and that students inherit not just the semantic bias but also the type of intervention used, with implications for training safety and data auditing.

StepReflect reformulates per-step GUI reflection for mobile agents as supervised structured prediction, achieving higher transition accuracy than GPT-5.2 on AndroidWorld while reducing API costs.

Proposes Woodpecker Distillation, a weak-to-strong training framework that uses weak probe models to identify and repair local reasoning bugs in stronger models via contrastive local interventions, improving performance on math reasoning benchmarks.

Jensen Huang was asked whether open-source model companies should be allowed to distill closed-source models. He said distillation is not copying, but a fundamental way of intelligence.

This paper studies On-Policy Delta Distillation (OPD^2) for multilingual math reasoning in English, Korean, and Japanese, showing consistent improvements over standard OPD and narrowing language gaps.

An independent developer updates the AttnRes project: replacing standard residual stream with attention-based routing, distilling from Gemma 4 31b via a weaning schedule and top-K logits, with plans for an Apache 2.0 community model.

Introduces TQLite, a distillation framework that uses a multi-LRM jury to train small language models for real-time MQM-based translation quality evaluation, achieving performance far exceeding off-the-shelf SLMs while remaining cost-effective.

This paper introduces OPTD, an on-policy transition distillation method with consistency-guided adaptive compression for few-step diffusion language models, improving quality-efficiency trade-offs across four reasoning and code-generation benchmarks.

ContextMaster is a unified model for interactive multi-shot video creation that maintains shared history across generation, reference conditioning, and editing while using fixed-budget sparse context routing and two-stage privileged context distillation for efficiency.

Poly-OPD is a framework for distilling complementary strengths from heterogeneous text-to-image flow models into a single compact flow-matching student, using pixel bridges and gradient-compatible adapters. It improves GenEval and DrawBench scores while consolidating multiple teacher capabilities.

The author observes that Chinese AI models like Qwen and Kimi produce better-looking frontend code than OpenAI's and Anthropic's offerings, and wonders whether this is due to distillation or other techniques.

SERL-SQL proposes a selective execution-grounded reinforcement learning framework for multi-turn Text-to-SQL agents, using teacher-student likelihood gaps to reweight GRPO advantages on SQL action tokens. It achieves strong results on BIRD and Spider benchmarks.

This thread introduces β-OPSD, a post-training abstraction that frames RL as a compiler for supervision, using policy optimization to derive targets and distillation for training.

Any-OPD presents the first framework for on-policy distillation between arbitrary latent flow-matching generators, enabling distillation from a 12B FLUX model to a 2.5B SD3.5 model by bridging via a frozen vision representation. It improves the student's PickScore from 0.846 to 0.884, rivaling the teacher at a fifth of its size.

JoyAI-Video-Edit is a 16B-parameter autoregressive diffusion framework for real-time, open-ended video editing, achieving 720p editing at ~30 FPS on a single NVIDIA B200 GPU.

This article argues that the US should sanction Chinese AI labs engaged in fraudulent extraction of American AI trade secrets, while distinguishing legitimate distillation techniques and avoiding bans on open-weight models. It highlights recent incidents involving Moonshot AI and Anthropic's Fable model.

This paper introduces Maga, a method for consolidating domain-specific GUI agents into a single cross-platform policy via structured action distillation, reallocating training signals to focus on erroneous actions. It achieves strong success rates across mobile, web, and desktop benchmarks.

This paper introduces Dual-Anchored Policy Distillation (DAPD), a framework to address privilege illusion in on-policy self-distillation for language models. DAPD improves Qwen3-4B by +2.00 points on average across tasks, with gains persisting at larger scales.

EdgeRazor is a lightweight framework for compressing large language models using entropy-guided mixed-precision quantization-aware distillation, achieving 1.88 bits per parameter while preserving teacher model competence and requiring no changes to inference implementations like llama.cpp. The method is demonstrated on small models such as MobileLLM and Qwen variants.