distillation

This paper proposes amortizing the high token cost of reasoning-mode LLMs by distilling domain-specific skills from existing trajectories into system prompts, recovering most of the reasoning gap on agentic benchmarks while emitting far fewer tokens.

DistilVDR is a compact 524M visual document retriever distilled from an 8B teacher via cosine alignment, achieving near-teacher accuracy on ViDoRe with 15.6x smaller indexes and faster indexing.

A researcher discusses rumors that Chinese labs reverse-engineered reasoning traces from Claude Code and Codex to improve open-weights models, and shares a paper on Trace Inversion models that can reconstruct reasoning traces from outputs.

A podcast about AI model competition, discussing the importance of data, distillation and pre-training innovation, and an interview with Evolvent AI co-founder Meng Fanqing, covering topics such as synthetic data, RSI, and differences in domestic models.

This paper proposes scaffold-mediated post-training, a paradigm where procedural scaffolds co-evolve with LLM parameters through discovery, distillation, and dynamic recompilation. On FeatureBench, automatically discovered skills improve pass rate by 8.1pp, with a 27.7% pass rate after distillation.

This paper investigates subliminal learning in language models, showing that biases can transfer from teacher to student via seemingly random synthetic data. The authors find that adding Gaussian noise to weights increases transfer, and that students inherit not just the semantic bias but also the type of intervention used, with implications for training safety and data auditing.

StepReflect reformulates per-step GUI reflection for mobile agents as supervised structured prediction, achieving higher transition accuracy than GPT-5.2 on AndroidWorld while reducing API costs.

Proposes Woodpecker Distillation, a weak-to-strong training framework that uses weak probe models to identify and repair local reasoning bugs in stronger models via contrastive local interventions, improving performance on math reasoning benchmarks.

Jensen Huang was asked whether open-source model companies should be allowed to distill closed-source models. He said distillation is not copying, but a fundamental way of intelligence.

Proposes Gated Hindsight Distillation (GHD), a method that uses future screenshots as privileged information to recover correct reasoning during training of mobile GUI agents, improving task success on AndroidWorld and AndroidLab across two vision-language models.

This paper studies On-Policy Delta Distillation (OPD^2) for multilingual math reasoning in English, Korean, and Japanese, showing consistent improvements over standard OPD and narrowing language gaps.

An independent developer updates the AttnRes project: replacing standard residual stream with attention-based routing, distilling from Gemma 4 31b via a weaning schedule and top-K logits, with plans for an Apache 2.0 community model.

Introduces TQLite, a distillation framework that uses a multi-LRM jury to train small language models for real-time MQM-based translation quality evaluation, achieving performance far exceeding off-the-shelf SLMs while remaining cost-effective.

This paper introduces OPTD, an on-policy transition distillation method with consistency-guided adaptive compression for few-step diffusion language models, improving quality-efficiency trade-offs across four reasoning and code-generation benchmarks.

ContextMaster is a unified model for interactive multi-shot video creation that maintains shared history across generation, reference conditioning, and editing while using fixed-budget sparse context routing and two-stage privileged context distillation for efficiency.

Poly-OPD is a framework for distilling complementary strengths from heterogeneous text-to-image flow models into a single compact flow-matching student, using pixel bridges and gradient-compatible adapters. It improves GenEval and DrawBench scores while consolidating multiple teacher capabilities.

The author observes that Chinese AI models like Qwen and Kimi produce better-looking frontend code than OpenAI's and Anthropic's offerings, and wonders whether this is due to distillation or other techniques.

SERL-SQL proposes a selective execution-grounded reinforcement learning framework for multi-turn Text-to-SQL agents, using teacher-student likelihood gaps to reweight GRPO advantages on SQL action tokens. It achieves strong results on BIRD and Spider benchmarks.

This thread introduces β-OPSD, a post-training abstraction that frames RL as a compiler for supervision, using policy optimization to derive targets and distillation for training.

Any-OPD presents the first framework for on-policy distillation between arbitrary latent flow-matching generators, enabling distillation from a 12B FLUX model to a 2.5B SD3.5 model by bridging via a frozen vision representation. It improves the student's PickScore from 0.846 to 0.884, rivaling the teacher at a fifth of its size.