Tag
该论文研究在数据有限但算力充裕场景下的重复数据预训练，发现小批量通过梯度噪声提供隐式正则化，并在固定时间预算下小批量+少GPU可取得更低验证损失，挑战了常见地增大数据并行度以最大化GPU利用率的做法。
The paper introduces practical training recipes that cut looped (recurrent) language model training from 7.7T to 310B tokens, with a 1.4B LoopLM outperforming a parameter-matched dense baseline on all 12 benchmarks, and shows how to convert pretrained dense models like Qwen3-1.7B-Base into looped ones with minimal added parameters.
An open-source end-to-end post-training recipe covering SFT, RL, and test-time scaling with models, datasets, and code.
This paper introduces an auto-research framework using specialist agents to iteratively refine training recipes through an empirical loop of code execution and feedback. The system autonomously improves performance on tasks like Parameter Golf and NanoChat without human intervention by leveraging lineage feedback.