@SwayStar123: Crazy paper opening
Summary
A new paper reveals "mode-hopping" in LLM pre-training, where models repeatedly toggle between shallow pattern-matching and genuine generalization even as training loss stays flat — suggesting more pre-training does not always guarantee better generalization.
View Cached Full Text
Cached at: 10/04/26, 01:07 AM
Crazy paper opening https://t.co/A13Iq3mU9a
alphaXiv (@askalphaxiv): More pre-training doesn’t necessarily mean better generalization?
This new paper discovers mode-hopping, where LLMs repeatedly switch between shallow pattern-matching and actual generalization, even while training loss remains stable.
For example, OLMo3-32B went from 81%
Similar Articles
Generalization Dynamics of LM Pre-training (17 minute read)
This paper reveals that during pre-training, language models frequently and suddenly switch between pattern-matching and generalization behaviors, a phenomenon called mode-hopping, and presents a toy evaluation suite to study it.
Generalization Dynamics of LM Pre-training (4 minute read)
该论文提出"模式跳变"(mode-hopping)概念，指出LM预训练中平滑下降的损失与稳定基准分数会掩盖模型答题方式的突变（如OLMo3-32B在算术提示上81%→0%→81.7%的反复），并通过六项廉价行为测试排除了评估指标非线性与局部优化波动等简单解释。
@omarsar0: Great paper for all you pre-training and post-training nerds. The surprising finding is that pre-trained LLMs, equipped…
A paper from Meta Superintelligence Labs shows that pre-trained LLMs equipped with a light inference harness can act as capable agents and even outperform RL post-trained counterparts when given a sufficient test-time sampling budget.
@dair_ai: Banger paper from Meta Superintelligence Labs. They find something super interesting and unexpected. (bookmark it) Base…
Meta Superintelligence Labs finds that RL post-training narrows solution coverage of LLM agents (the 'Sharpening Tax'): base models with a light harness can solve more distinct agentic tasks at large K, while post-trained models win at pass@1. They propose PTGS, an adaptive-temperature RL method that reduces this tax while improving pass@1.
Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages
This paper investigates whether pretraining LLMs on artificial languages (pre-pretraining) consistently improves token efficiency across multiple natural languages, finding that gains are highly dependent on experimental setup and random seed, though stable gains appear for small models with the Llama tokenizer.