@Suhail: There is going to be 10x more distillation research with the threat of open weight model restrictions. It will Streisan…
Summary
Suhail predicts that restrictions on open weight models will lead to a 10x increase in distillation research as a way to troll the government, potentially making AI more efficient.
Similar Articles
@Miles_Brundage: I am not sure I have seen a good analysis of how much distillation reduces this gap - people have very different views …
Miles Brundage comments on the lack of quantitative analysis on how distillation affects the capability gap between open-weight and proprietary AI models, referencing a claim by Epoch AI that open-weight models lag by four months.
@rabois: Good distillation.
A commentator highlights a breakdown of US government options to restrict Chinese AI models, including a FUD approach.
As US weighs response to Chinese AI, industry urges against broad open-weight restrictions
Several major AI companies sign an open letter urging US policymakers against broad restrictions on open-weight AI models, arguing that such measures would harm innovation and that techniques like distillation should not be conflated with intellectual property theft.
@jxmnop: ok sorry everyone apparently they did distill lol. but only a tiny bit
Jack Morris corrects his earlier claim about an open-weight model being trained without distillation from OpenAI or Anthropic, acknowledging that it actually did use a small amount of distillation.
Model "distillation" accusations are getting way overblown at this point
The article argues that recent accusations of model distillation in AI are being exaggerated and overblown.