Tag
This paper proposes EMA-FS, an algorithm-level optimization that uses exponential moving averages of per-feature split gains to selectively construct histograms only for top-K features during GBDT training, achieving up to 2.61x speedups while maintaining or improving accuracy through implicit regularization.
Reasoning Arena improves reinforcement learning with verifiable rewards by using trace tournaments and Bradley-Terry models to generate meaningful gradients from non-diverse reward groups, resulting in faster training and better reasoning performance.
This paper investigates the parameter-level mechanisms behind the efficiency of On-Policy Distillation (OPD) for large language models, attributing it to early 'foresight' in module allocation and update direction. It proposes EffOPD, a plug-and-play method that accelerates OPD training by 3x without compromising final performance.
This paper introduces Repr-Align, a method to adapt autoregressive language models into diffusion language models via representation alignment, achieving up to 4x training acceleration without retraining representations from scratch.
The paper introduces Momentum DeltaNet (MDN), a linear attention model that uses stepwise momentum and parallel algorithms to improve training efficiency and performance over models like Mamba2.
OpenAI presents weight normalization, a reparameterization technique that decouples weight vector length from direction to improve neural network training convergence and computational efficiency without introducing minibatch dependencies, making it suitable for RNNs and noise-sensitive applications.