distillation

This paper studies strong-to-weak capability transfer at test time, showing that stronger models can build inference-time harnesses that nearly double weaker models' performance without parameter updates.

Researchers have, for the first time, systematically extracted hidden chain-of-thought from closed-source models at OpenAI, Anthropic, and Google. They exploited an API encryption compatibility flaw to have weaker models decrypt the content, thereby bypassing CoT distillation protections, and also studied open-source models such as Kimi K3, GLM-5.2, and DeepSeek.

Researchers discovered a method to extract hidden reasoning traces from frontier AI models via APIs, exposing personal-information leakage risks and enabling large-scale distillation attacks. The findings also suggest possible reasoning distillation from US models into Chinese model Kimi K3, though not conclusively.

This paper proposes an adaptive supervised anchoring framework for on-policy self-distillation, addressing the problem of rollout-conditioned signal degradation in language model training. The method separates rollout-conditioned distribution matching from canonical-context supervision, improving task acquisition while preserving general capabilities.

This paper proposes amortizing the high token cost of reasoning-mode LLMs by distilling domain-specific skills from existing trajectories into system prompts, recovering most of the reasoning gap on agentic benchmarks while emitting far fewer tokens.

DistilVDR is a compact 524M visual document retriever distilled from an 8B teacher via cosine alignment, achieving near-teacher accuracy on ViDoRe with 15.6x smaller indexes and faster indexing.

A researcher discusses rumors that Chinese labs reverse-engineered reasoning traces from Claude Code and Codex to improve open-weights models, and shares a paper on Trace Inversion models that can reconstruct reasoning traces from outputs.

A podcast about AI model competition, discussing the importance of data, distillation and pre-training innovation, and an interview with Evolvent AI co-founder Meng Fanqing, covering topics such as synthetic data, RSI, and differences in domestic models.

This paper proposes scaffold-mediated post-training, a paradigm where procedural scaffolds co-evolve with LLM parameters through discovery, distillation, and dynamic recompilation. On FeatureBench, automatically discovered skills improve pass rate by 8.1pp, with a 27.7% pass rate after distillation.

This paper investigates subliminal learning in language models, showing that biases can transfer from teacher to student via seemingly random synthetic data. The authors find that adding Gaussian noise to weights increases transfer, and that students inherit not just the semantic bias but also the type of intervention used, with implications for training safety and data auditing.

StepReflect reformulates per-step GUI reflection for mobile agents as supervised structured prediction, achieving higher transition accuracy than GPT-5.2 on AndroidWorld while reducing API costs.

Proposes Woodpecker Distillation, a weak-to-strong training framework that uses weak probe models to identify and repair local reasoning bugs in stronger models via contrastive local interventions, improving performance on math reasoning benchmarks.

Jensen Huang was asked whether open-source model companies should be allowed to distill closed-source models. He said distillation is not copying, but a fundamental way of intelligence.

Proposes Gated Hindsight Distillation (GHD), a method that uses future screenshots as privileged information to recover correct reasoning during training of mobile GUI agents, improving task success on AndroidWorld and AndroidLab across two vision-language models.

This paper studies On-Policy Delta Distillation (OPD^2) for multilingual math reasoning in English, Korean, and Japanese, showing consistent improvements over standard OPD and narrowing language gaps.

An independent developer updates the AttnRes project: replacing standard residual stream with attention-based routing, distilling from Gemma 4 31b via a weaning schedule and top-K logits, with plans for an Apache 2.0 community model.

Introduces TQLite, a distillation framework that uses a multi-LRM jury to train small language models for real-time MQM-based translation quality evaluation, achieving performance far exceeding off-the-shelf SLMs while remaining cost-effective.

This paper introduces OPTD, an on-policy transition distillation method with consistency-guided adaptive compression for few-step diffusion language models, improving quality-efficiency trade-offs across four reasoning and code-generation benchmarks.

ContextMaster is a unified model for interactive multi-shot video creation that maintains shared history across generation, reference conditioning, and editing while using fixed-budget sparse context routing and two-stage privileged context distillation for efficiency.

Poly-OPD is a framework for distilling complementary strengths from heterogeneous text-to-image flow models into a single compact flow-matching student, using pixel bridges and gradient-compatible adapters. It improves GenEval and DrawBench scores while consolidating multiple teacher capabilities.