svd

CORAM introduces a coherent orthogonal rotation method for model merging that partitions weight matrices into row slices, uses SVD in the base model's frame, and merges updates on manifolds to improve accuracy over existing techniques.

This paper investigates singular components in LoRA and proposes SCLoRA, a method that uses spectral clipping to improve task adaptation while reducing catastrophic forgetting of pre-trained knowledge.

This paper proposes extracting mechanism mounts directly from linear weight sites via column-tiled SVD, offering an alternative to proxy dictionaries like sparse autoencoders for mechanistic interpretability. Evaluated on Gemma-2-2B, the method passes all 182 site-layer checks.

This paper proposes FedSLM, a parameter-centric framework for federated fine-tuning of foundation models with heterogeneous compressed clients, using SVD-based decomposition and a weak-to-strong elicitation step to handle resource asymmetry. Experiments show it outperforms existing federated baselines while reducing client GPU memory by ~50%.

This paper proposes a method for compressing large language models by combining neuron importance and data-aware low rank approximation, along with an efficient dynamic compression rate allocation algorithm, achieving performance on par with or better than previous state-of-the-art.

LuffyTheFox releases Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V5-GGUF, a modified version of Qwen model using the Genesis post-training algorithm to repair signal purity and reduce noise via custom SVD on GGUF-format tensors.

KQ-SVD is a new method for KV-cache compression that directly approximates the attention matrix via optimal low-rank decomposition, achieving 5-10x lower error than key-only SVD on LLaMA and Mistral models.

LACE-SVD is a novel low-rank compression method for large language models that uses a loss-aware rank allocation strategy and a propagation-aware correction technique to mitigate cumulative error propagation in the residual stream, achieving better perplexity than prior SVD-based methods at high compression ratios.

This blog post explains how to derive the Singular Value Decomposition (SVD) from scratch by focusing on the underlying intuition and the motivation behind the concept, arguing that traditional math books often present formalized conclusions without showing the exploratory path.

This paper introduces low-cost High-Order Singular Value Decomposition (lcHOSVD), a tensor-based method for reconstructing high-dimensional environmental fields from sparse sensor measurements. Applied to urban flow and air-quality datasets, it achieves lower reconstruction errors and greater robustness to uneven sensor distributions compared to matrix-based approaches.

This study reveals that LLM text embeddings are hijacked by high-frequency tokens (e.g., periods, articles) and proposes EmbedFilter, which performs SVD on the unembedding matrix and subtracts the projection component to release true semantics, achieving zero-training-cost dimensionality reduction and retrieval efficiency gains.

Sebastian Raschka points out the chain of inspiration from LatentMoE back to eigendecomposition through MLA, LoRA, and SVD.

A post-hoc method reduces spurious correlations in fine-tuned LLMs by truncating the tail of the SVD of the weight update matrix. It reduces the spurious-group gap by up to 5x with less than 2pp accuracy loss, without retraining or group labels.

Introduces SigmaScale, a method that learns auxiliary scaling matrices for SVD-based LLM compression, showing competitive performance on Llama 3.1 8B and Qwen3-8B benchmarks.

This paper proposes DG-Hard, a post-hoc spectral repair method that recovers capabilities damaged by fine-tuning without retraining, using only the pretrained and fine-tuned checkpoints. It applies Donoho-Gavish hard singular-value thresholding to weight updates to remove noise and restore degraded performance.

The paper introduces CERSA, a novel parameter-efficient fine-tuning method that uses singular value decomposition to retain principal components, significantly reducing memory usage while outperforming existing methods like LoRA.