@datologyai: Why do larger models hold onto rare skills that smaller ones can lose? Jing Huang (Stanford NLP) points to interference…
Summary
The article discusses a talk by Jing Huang from Stanford NLP at the Summer of Data event, explaining that larger AI models retain rare skills better due to task interference and greater capacity.
View Cached Full Text
Cached at: 08/17/26, 08:26 AM
Why do larger models hold onto rare skills that smaller ones can lose?
Jing Huang (Stanford NLP) points to interference between tasks. Every model has limited capacity, and its tasks compete for it during training. In a small model, the rare tasks in the tail get crowded out. A bigger model has room to keep them.
This week’s Summer of Data talk is live. Thanks to Jing for coming by. Full talk in the reply.
Full talk:
Similar Articles
@rohanpaul_ai: Great Stanford + MIT + Harvard + Anthropic paper. Gives a clear training-based reason for why larger models learn abili…
A Stanford, MIT, Harvard, and Anthropic paper explains that larger AI models learn rare skills better because they forget them less during training; their extra capacity protects weak learning signals from being overwritten by common tasks.
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
This paper investigates why larger models outperform smaller ones, attributing it to reduced gradient interference and better resource allocation, allowing them to learn rare and complex tasks even with infinite data. Experiments on synthetic data and OLMo models verify that larger models avoid overwriting rare-task features due to weaker gradient updates for common tasks.
@ChrisGPotts: We take for granted that larger models are better than smaller ones, but why is this so? Our new paper, led by Jing Hua…
This paper investigates why larger models outperform smaller ones, attributing it to data-induced competition for neural resources through formal analysis and experiments.
@_jasonwei: When language models first started using tools well, I was sympathetic to the narrative that instead of scaling up lang…
The author argues that while tool use allows smaller language models to perform tasks effectively, larger models remain crucial for speed, reliability, and internalized knowledge, emphasizing the ongoing need for scaling in AI.
@rohanpaul_ai: Stronger agents will not come only from larger models, but from better systems around them. The problem is that many AI…
This tweet discusses the paper 'From Model Scaling to System Scaling' which argues that stronger AI agents require better system design (harness) including context control, memory, and routing, not just larger models.