Tag
The paper introduces Modular Norm RandOpt, an architecture-aware perturbation method for efficient ensembling of language models, showing improved performance with fewer candidates across multiple tasks and model scales.
This paper introduces SWARR, a two-stage recipe using supervised fine-tuning and reinforcement learning to adapt sliding-window attention models for mathematical reasoning, showing that RL can narrow the performance gap with self-attention while maintaining efficiency.