svd

This blog post explains how to derive the Singular Value Decomposition (SVD) from scratch by focusing on the underlying intuition and the motivation behind the concept, arguing that traditional math books often present formalized conclusions without showing the exploratory path.

This paper introduces low-cost High-Order Singular Value Decomposition (lcHOSVD), a tensor-based method for reconstructing high-dimensional environmental fields from sparse sensor measurements. Applied to urban flow and air-quality datasets, it achieves lower reconstruction errors and greater robustness to uneven sensor distributions compared to matrix-based approaches.

This study reveals that LLM text embeddings are hijacked by high-frequency tokens (e.g., periods, articles) and proposes EmbedFilter, which performs SVD on the unembedding matrix and subtracts the projection component to release true semantics, achieving zero-training-cost dimensionality reduction and retrieval efficiency gains.

Sebastian Raschka points out the chain of inspiration from LatentMoE back to eigendecomposition through MLA, LoRA, and SVD.

A post-hoc method reduces spurious correlations in fine-tuned LLMs by truncating the tail of the SVD of the weight update matrix. It reduces the spurious-group gap by up to 5x with less than 2pp accuracy loss, without retraining or group labels.

Introduces SigmaScale, a method that learns auxiliary scaling matrices for SVD-based LLM compression, showing competitive performance on Llama 3.1 8B and Qwen3-8B benchmarks.

This paper proposes DG-Hard, a post-hoc spectral repair method that recovers capabilities damaged by fine-tuning without retraining, using only the pretrained and fine-tuned checkpoints. It applies Donoho-Gavish hard singular-value thresholding to weight updates to remove noise and restore degraded performance.

The paper introduces CERSA, a novel parameter-efficient fine-tuning method that uses singular value decomposition to retain principal components, significantly reducing memory usage while outperforming existing methods like LoRA.