Tag
A new paper reveals "mode-hopping" in LLM pre-training, where models repeatedly toggle between shallow pattern-matching and genuine generalization even as training loss stays flat — suggesting more pre-training does not always guarantee better generalization.
该论文提出"模式跳变"(mode-hopping)概念，指出LM预训练中平滑下降的损失与稳定基准分数会掩盖模型答题方式的突变（如OLMo3-32B在算术提示上81%→0%→81.7%的反复），并通过六项廉价行为测试排除了评估指标非线性与局部优化波动等简单解释。
This paper reveals that during pre-training, language models frequently and suddenly switch between pattern-matching and generalization behaviors, a phenomenon called mode-hopping, and presents a toy evaluation suite to study it.