Tag
This paper analyzes post-training weight updates in LLMs using singular value decomposition, identifying geometric components that drive performance gains, with insights suggesting that reshaping singular values is less critical than rotating and routing changes.
This article explains Truncated SVD and its relation to PCA, demonstrating how it can be used to approximate images by reconstructing them with fewer singular values. It uses the example of a moon image to illustrate the reconstruction process.
A historical account of the development of the singular value decomposition (SVD) up to 1993, tracing its origins and key contributions. The paper is a classic reference for the mathematical underpinnings of many modern data analysis and machine learning techniques.
This blog post explains how to derive the Singular Value Decomposition (SVD) from scratch by focusing on the underlying intuition and the motivation behind the concept, arguing that traditional math books often present formalized conclusions without showing the exploratory path.
This paper identifies a spectral phenomenon called Stability of Singular Distribution (SoSD) in large language model pre-training, where the singular value spectrum stabilizes early while parameters continue to evolve. The authors prove that this stabilization marks the transition to the slow-descent phase of training, and they analyze how training strategies like WSD and Muon affect this behavior.
This paper introduces Rotation-Preserving Supervised Fine-Tuning (RPSFT), a method that improves out-of-domain generalization by preserving projected rotations in pretrained singular subspaces during fine-tuning.