Tag
NVIDIA introduces Cosmos-H-Dreams, a real-time action-conditioned generative simulator for surgical robotics, distilled from the larger Cosmos-H-Surgical-Simulator. It runs on a single RTX PRO 6000 GPU using the FlashDreams inference library, enabling interactive closed-loop control for training and evaluation.
A retired engineer proposes Blueprint Distillation (BPD) to separate expensive teacher analysis from student compression for large MoE models like Kimi K3, enabling reusable compression blueprints for multiple student sizes.
Introduces Byte-Prefix Marginalization (BPM) for cross-tokenizer on-policy distillation, re-expressing teacher token distributions over student vocabulary in a shared byte space, outperforming baselines on mathematics and programming benchmarks.
AI distillation has become a hot topic from Silicon Valley to Washington, as concerns grow that Chinese labs like Moonshot AI are using the technique to quickly catch up to US frontier models, sparking debates over national security and intellectual property.
Wired reports on the deep divide in Silicon Valley over Chinese open-weight AI models, with some startups opposing a ban while AI giants like Anthropic push for regulation due to concerns about IP theft and lack of safety guardrails.
Argues that accusations of knowledge distillation by China from US AI models are legally baseless and amount to propaganda to protect overvalued US AI companies.
An article simplifying the concept of LLM distillation for a political audience, explaining how smaller models learn from larger ones.
Suhail predicts that restrictions on open weight models will lead to a 10x increase in distillation research as a way to troll the government, potentially making AI more efficient.
Experts are skeptical that Chinese company Moonshot achieved Kimi K3's advanced capabilities by copying Anthropic's Fable model via distillation, citing insufficient time and technical limitations.
A claim suggests that a distilled AI model can outperform its larger original model, which is counterintuitive.
VCSD removes the need for external teachers, privileged answers, or visual evidence in on-policy self-distillation by using content-erased control images to produce contrastive signals, consistently outperforming existing methods on vision-language benchmarks.
The White House is divided on how to respond to China's rapid AI progress, with debates over stricter controls versus workable restrictions, particularly regarding distillation attacks on US models like Anthropic's Claude Fable 5.
The U.S. Treasury threatens sanctions against Chinese AI company Moonshot for allegedly distilling Anthropic's Fable model, escalating tensions over AI intellectual property theft and export control violations.
Session-Adaptive Orthogonal Distillation (SAOD) is a technique that compresses a 744-billion parameter model (1.5TB) to under 100GB, greatly reducing storage and inference costs.
Commentary criticizing Anthropic and OpenAI for claiming their closed models were hacked/distilled by Chinese model Kimi3, questioning the inconsistency with earlier security claims.
The former White House Science Advisor claimed that Kimi K3 was distilled from Anthropic's Fable model.
Moonshot AI allegedly distilled Anthropic's Fable model for its K3 using sophisticated methods to evade detection, raising concerns about large-scale industrial espionage in AI development.
MUX proposes a method for lossless continuous reasoning by distilling discrete reasoning steps into multiplexed latent tokens that encode a superposition of subwords, achieving higher bandwidth and enabling parallel exploration in language model reasoning tasks.
The article argues that AI companies' competitive moat, built on expensive model training, is easily undermined by distillation—replicating models through repeated API queries—as demonstrated by industry practices like xAI training Grok on OpenAI models and Anthropic accusing Chinese labs of mining Claude.
A quote from @enoreyes of @factoryAI suggests that distillation is unstoppable and that model labs may already recognize the shape of intelligence.