Tag
This paper identifies a Rank-1 Subspace phenomenon in LLM pre-training trajectories and proposes Extra-Merge, a training-free strategy that extrapolates along this subspace to minimize loss, achieving consistent zero-shot accuracy gains across GPT-2 and LLaMA families up to 2B parameters.