distillation

MiMo-V2.6 has been distilled into Qwen 9B, creating a more efficient version of the Qwen model released on Hugging Face.

The article argues that the AI bubble could burst without AI failure due to competition from cheaper models and cost-reduction techniques like distillation, which may erode profits from large industry spending.

The paper introduces GUARD, a method for natural forgetting in large reasoning models that uses guided answer-reasoning distillation to suppress unsafe or private content in chain-of-thought traces while preserving reasoning utility.

The paper presents ACLArena, a framework for evaluating Agent Continual Learning in multi-stage post-training, analyzing forgetting and generalization mechanisms, and proposing an improved ACL recipe using offline replay and LoRA experts.

The paper introduces THAW-VLA, a method that distills world-model representations into Vision-Language-Action models for robotics, enhancing robustness and performance on simulation and real hardware without additional inference overhead.

The article critiques Anthropic's report accusing seven Chinese AI labs of illicitly distilling Claude's reasoning, highlighting weak evidence for Alibaba's Qwen and suggesting political framing in the accusations.

Empero AI releases their first MoE model, Qwen3.8-35B-A3B-Distill, which distills reasoning from Qwen3.8 into a smaller, efficient architecture, showing improved benchmarks on ARC-Challenge.

The paper presents a method to distill reasoning into compact video-language models using synthetic chain-of-thought rationales and difficulty-aware fine-tuning, enabling smaller models to outperform larger ones with minimal compute.

This paper introduces MIFS, a pipeline for synthesizing RL-ready multimodal data to enhance instruction following in MLLMs, achieving an 8.13% average improvement on benchmarks and faster training convergence.

The paper introduces Zing-0.5, a 5B parameter autoregressive world model for generating playable worlds with real-time user interaction through combined keyboard and text controls. It achieves high performance in navigation tasks and demonstrates low-cost real-time inference at 24 FPS.

A discussion on future work exploring how byte-level models respond to quantization for potential improvements in local AI, based on a Meta paper showing byte models outperforming token models as compute scales.

A research paper from Meta demonstrates that byte-level language models, initially inferior to token-based models, can outperform them as computational resources increase, shown through distilled 1B models trained on up to 1 trillion bytes.

A paper from Meta shows that byte-level models start behind token models but surpass them with increasing compute, demonstrated with distilled 1B models trained on up to 1 trillion bytes.

UkisAI has post-trained the Qwen 3.8 27B model to reduce unnecessary thinking tokens by 58% and achieve 1.95x speed up with less than 1% accuracy loss, open-sourcing the model and offering a free research API.

This paper presents a large-scale study comparing distilled byte and token models, finding that byte models achieve higher performance ceilings with more compute and are more data-efficient than token models.

This paper introduces the KuaiRP series of role-playing models, detailing a multi-stage training pipeline that addresses the trade-off between deep domain knowledge injection and general capability preservation, achieving state-of-the-art performance with low deployment costs.

Anthropic revealed that Chinese AI companies Alibaba, Moonshot AI, and DeepSeek executed extensive distillation attacks on its Claude model to extract chain-of-thought reasoning for training their own models.

Mask Forcing mitigates mode collapse in autoregressive video diffusion distillation by injecting masked cleaner signals during self-rollout, improving visual quality without extra training data.

AuK is an open-source foundational model that unifies speech generation and editing through natural-language instructions, achieving leading performance with efficient inference via distillation.

The paper introduces On-Policy Reverse Distillation (OPRD), a method that enables stronger AI models to exceed weaker supervisors by amplifying verifier-supported policy gradients along the teacher's shift direction, achieving higher performance with fewer updates in distillation scenarios.