Tag
该论文研究在数据有限但算力充裕场景下的重复数据预训练，发现小批量通过梯度噪声提供隐式正则化，并在固定时间预算下小批量+少GPU可取得更低验证损失，挑战了常见地增大数据并行度以最大化GPU利用率的做法。
This paper shows that no principled batch-size scaling rule for the Muon optimizer works consistently across training settings, and that the best optimizer for language model pretraining changes with batch size even after extensive tuning, arguing that optimizer design and benchmarking must account for batch size.
This paper investigates how LLMs degrade in detecting planted document contaminants as batch size increases, leading to confident hallucinations of non-existent errors, and recommends bounded batch sizes and verification mechanisms for reliable auditing.
A roofline model from the LLM Engineer's Almanac estimates speedups from speculative decoding for different draft lengths across models and hardware, with a note that it may underestimate benefits when overhead is significant.
Proposes a three-term scaling law that decouples model size, training steps, and batch size, enabling robust fitting with fewer runs and deriving scaling laws for suboptimal batch sizes.
OpenAI researchers discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training across a wide range of tasks. They found that more complex tasks and more powerful models tolerate larger batch sizes, suggesting future AI systems can scale further through increased parallelization.