distillation

A software developer discusses how OpenAI and Anthropic's strategy of keeping models internal may influence open weight releases and the replication of models by Chinese companies.

The paper proposes TUP, a method for BoN-style distillation via rank-based classification that truncates low-ranked completions and upweights high-ranked ones to improve alignment efficiency and performance.

SecOPD is a defense method that uses token-level feedback during fine-tuning to mitigate adaptive prompt injection attacks in large language models, achieving significantly lower attack success rates compared to previous approaches.

The U.S. administration accuses Chinese AI firm Moonshot of stealing Anthropic's model through distillation, while maintaining that AI output is not owned by anyone, revealing contradictions in intellectual property law.

The paper explains that agent skills improve performance by turning past experience into clean procedures, with the skill version outperforming workflow memory by 6.06 percentage points, mainly through procedural anchoring.

SMOPD is a loss-only stabilization method for multi-turn on-policy self-distillation that uses selective token-entropy masking to improve accuracy in dirty-history settings, demonstrating improvements with Qwen3 models.

DUET is a dual-teacher on-policy distillation method for ensuring LLMs comply with runtime prohibitions, achieving high violation compliance while preserving utility through token-selective learning.

The article questions how the relatively small Qwen 3.8 27B model achieves high intelligence, raising doubts about current scaling laws and the efficiency of large model parameters.

The paper introduces Consistency Forcing (CForce), a distillation technique for diffusion large language models that improves parallel decoding by aligning early-stage predictions with later stages, enhancing speed-quality trade-offs.

Qwen3.8-9B-Distill is a distilled version of the Qwen3.8 2.4T A95B model into a 9B architecture, trained on curated teacher traces to enhance reasoning in mathematics and code, showing improved benchmark performance.

A Microsoft paper proposes a method to replace expensive test-time reasoning with distilled skills from past agent runs, showing competitive performance on some benchmarks while reducing output token usage.

Allegations suggest Moonshot may have used Anthropic's Claude model to develop Kimi K3, as Kimi K3 reportedly identified itself as Claude during testing, raising questions about model distillation and training methods.

ForgeWM is a progressive framework that distills bidirectional video generators into efficient few-step interactive world models, supporting low-latency interaction and replay-time refinement with demonstrated improvements on Minecraft and FPS gameplay.

This paper introduces LOFA, a framework that enables LLM-based shopping agents to learn from real online user interaction logs using reinforcement learning over purchase outcomes and feedback-aware on-policy distillation, improving recommendation quality and user satisfaction.

The author discusses a paper that demystifies the warm-up process for OPD (likely on-policy distillation), explaining how warm-up enables well-defined student-sampled sequences and educational token-level dense rewards from the teacher.

DreamX-Phi 1.0 is an action-conditioned video world model for robotic manipulation that predicts future observations using geometric attention encoding, depth estimation, and distillation, achieving top results in the WorldArena 2.0 Challenge.

A recent paper demonstrates a technique to retrieve hidden reasoning traces from proprietary LLM APIs like Claude and GPT, with implications for open-source model comparisons, benchmark integrity, and distillation efforts.

MERA introduces a multi-cycle adaptation approach for LLM agents that improves small models by distilling execution-verified demonstrations into a SkillBook and fine-tuning LoRA adapters, with router-based deployment and verifier-backed fallback. Experiments show Qwen2.5-Coder-1.5B improves from 28.7% to 49.7% pass on HumanEval++MBPP while retaining most quality at lower cost.

This paper studies strong-to-weak capability transfer at test time, showing that stronger models can build inference-time harnesses that nearly double weaker models' performance without parameter updates.

Researchers have, for the first time, systematically extracted hidden chain-of-thought from closed-source models at OpenAI, Anthropic, and Google. They exploited an API encryption compatibility flaw to have weaker models decrypt the content, thereby bypassing CoT distillation protections, and also studied open-source models such as Kimi K3, GLM-5.2, and DeepSeek.