Spectral Tempering for Embedding Compression in Dense Passage Retrieval

# Spectral Tempering for Embedding Compression in Dense Passage Retrieval
Source: https://arxiv.org/html/2603.19339
(2026)

###### Abstract

Dimensionality reduction is critical for deploying dense retrieval systems at scale, yet mainstream post-hoc methods face a fundamental trade-off: principal component analysis (PCA) preserves dominant variance but underutilizes representational capacity, while whitening enforces isotropy at the cost of amplifying noise in the heavy-tailed eigenspectrum of retrieval embeddings. Intermediate spectral scaling methods unify these extremes by reweighting dimensions with a power coefficient γ, but treat γ as a fixed hyperparameter that requires task-specific tuning. We show that the optimal scaling strength γ is not a global constant: it varies systematically with target dimensionality k and is governed by the signal-to-noise ratio (SNR) of the retained subspace. Based on this insight, we propose Spectral Tempering (SpecTemp), a learning-free method that derives an adaptive γ(k) directly from the corpus eigenspectrum using local SNR analysis and knee-point normalization, requiring no labeled data or validation-based search. Extensive experiments demonstrate that Spectral Tempering consistently achieves near-oracle performance relative to grid-searched γ*(k) while remaining fully learning-free and model-agnostic. Our code is publicly available at https://github.com/liyongkang123/SpecTemp.

Dense Retrieval, Embedding Compression, Principal Component Analysis

††journalyear:2026††copyright:cc††conference:Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia††booktitle:Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '26), July 20–24, 2026, Melbourne, VIC, Australia††doi:10.1145/3805712.3809896††isbn:979-8-4007-2599-9/2026/07††ccs:Information systems Retrieval models and ranking††ccs:Computing methodologies Natural language processing

## 1. Introduction

Refer to captionFigure 1. Consistent spectral structure of dense retrieval embeddings. Eigenvalue distributions from 1M sampled embeddings on MS MARCO and NQ exhibit consistent heavy-tailed decay across diverse retrievers, revealing a head–tail signal-to-noise ratio (SNR) gradient—leading components are signal-dominant while tail dimensions grow noise-prone—motivating dimensionality-adaptive tempering.

Dense retrieval has become the dominant paradigm for first-stage retrieval in modern search systems (Karpukhin et al., 2020; Xiong et al., 2021; Reimers and Gurevych, 2019), where queries and documents are encoded as high-dimensional embeddings and relevance is computed via similarity functions such as cosine similarity. While recent encoders based on Large Language Models (LLMs) (Zhang et al., 2025; Li et al., 2023; Long et al., 2025) achieve state-of-the-art (SOTA) performance, they routinely produce high-dimensional embeddings (e.g., 1024–4096), increasing the memory footprint of vector indexes and the cost of similarity computation in large-scale deployment.

To mitigate these costs, training-based approaches such as learned projections (Zhang et al., 2026), conditional autoencoders (Liu et al., 2022), and knowledge distillation (Lioutas et al., 2020) have been explored, but require retraining infrastructure tied to specific encoders. Consequently, post-hoc compression—reducing dimensionality without parameter updates—offers a more practical alternative, yet its dominant baselines occupy flawed extremes. Principal Component Analysis (PCA) retains maximal variance (Zhang et al., 2024) but leaves the energy distribution highly skewed, allowing head dimensions to overshadow complementary discriminative signals. Conversely, standard whitening (Su et al., 2021) enforces isotropy by normalizing all dimensions to unit variance; yet the eigenspectrum of retrieval embeddings is heavily tailed (Figure 1), and this normalization substantially amplifies noise. Intermediate spectral scaling methods attempt to resolve this dilemma by weighting dimensions with a fractional power λ_i^{-γ/2} (γ ∈ [0,1]) (Su, 2022). However, prior work treats γ as a static hyperparameter that requires per-task tuning, overlooking that optimal tempering varies systematically with the target dimensionality k. For instance, aggressive whitening (γ ≈ 1) benefits compact subspaces (k = 64) but degrades quality at large k by amplifying low SNR tail components.

In this work, we formalize this dimensionality-dependent behavior through a local SNR analysis of the corpus eigenspectrum. By estimating a spectral noise floor, we obtain an SNR profile that reveals a smooth head–tail transition from signal-dominant to noise-prone components—explaining why optimal tempering strength should decrease as target dimensionality k grows to include low-SNR tail directions. Building on this insight, we propose Spectral Tempering (SpecTemp), a learning-free method that analytically derives an adaptive γ(k) directly from the SNR profile, automatically interpolating between variance preservation (PCA) and isotropy (whitening). The resulting linear transform is computed offline from corpus embeddings and applied identically to queries at inference time, requiring no labeled data or validation tuning.

Our contributions are three-fold:

• We characterize the **dimensionality-dependent** optimality of spectral scaling, demonstrating that the ideal γ is intrinsically governed by the subspace SNR rather than being a fixed constant.

• We propose **SpecTemp**, a learning-free method that analytically derives an adaptive γ(k) from the corpus eigenspectrum, requiring no labeled data or validation-based tuning.

• We conduct extensive experiments across multiple LLM-based embedding models and diverse retrieval datasets, demonstrating that SpecTemp consistently achieves near-oracle performance relative to grid-searched γ*(k).

## 2. Related Work

### Dense Retrieval

Dense retrieval has evolved from BERT-based bi-encoders (Devlin et al., 2019; Karpukhin et al., 2020; Xiong et al., 2021; Hofstätter et al., 2021; Lin et al., 2023) with compact 768d representations to massive LLM-based architectures. To capture complex semantics, recent SOTA models like RepLLaMA (Ma et al., 2024), E5-Mistral (Wang et al., 2022), and Qwen3-Embedding (Zhang et al., 2025) employ billion-scale, often decoder-only backbones. While yielding superior generalization, this shift often produces high-dimensional embeddings (e.g., 4096d), creating the storage bottlenecks that motivate our study. Recent work has also examined dense retrievers beyond effectiveness and efficiency, including their robustness under adversarial settings (Li, 2026), such as query perturbations (Penha et al., 2022) and corpus poisoning (Zhong et al., 2023; Liu et al., 2023; Li et al., 2025a,b).

### Embedding Compression

Strategies to mitigate these overheads fall into two broad categories: training-based and post-hoc.

**Training-based methods** optimize compression objectives during or after training-time. Matryoshka Representation Learning (MRL) (Kusupati et al., 2022) has gained widespread adoption for enabling flexible truncation by nesting information in prefix dimensions. Other approaches employ knowledge distillation to transfer capabilities to smaller students (Lioutas et al., 2020), or optimize conditional autoencoders to compress fixed embeddings into latent codes (Liu et al., 2022). While effective, these strategies require additional training data and incur high computational costs for retraining, rendering them impractical for off-the-shelf or API-only models.

**Post-hoc methods**, in contrast, transform pretrained embeddings without parameter updates. Spectral projections dominate this landscape, scaling dimensions based on their eigenvalues. PCA (γ = 0) maximizes variance but leaves the space anisotropic (Zhang et al., 2024; Ma et al., 2021; Zuo and Khashabi, 2026), while Standard Whitening (γ = 1) enforces isotropy but risks amplifying tail noise (Su et al., 2021; Huang et al., 2021). Intermediate strategies employ a fractional exponent γ ∈ [0,1] to interpolate between these extremes (Su, 2022), yet they rely on a static hyperparameter requiring per-task tuning. Alternatively, Random Projection offers dimension-agnostic compression via the Johnson–Lindenstrauss lemma (Johnson et al., 1984) but ignores the learned manifold structure.

A separate line of work targets isotropy via post-processing, such as removing dominant directions (Mu and Viswanath, 2018; Rajaee and Pilehvar, 2021; Raunak et al., 2019) or mapping to uniform distributions (Li et al., 2020), though these focus on quality rather than dimensionality reduction. Similarly, Product Quantization (PQ) (Jégou et al., 2011) and its variants achieve index-level compression via codebooks (Douze et al., 2024); being a downstream operation, this approach is orthogonal to and composable with linear projections like ours.

**SpecTemp** occupies a distinct position in this landscape: it is a **post-hoc, learning-free** linear projection that derives a dimensionality-adaptive tempering strength γ(k) from the local SNR of the retained subspace, requiring no labeled data, retraining, validation-based tuning, or index-level modifications.

## 3. Methodology

We now describe Spectral Tempering (SpecTemp), a post-hoc compression method that derives a dimensionality-adaptive tempering exponent γ(k) directly from the eigenspectrum of corpus embeddings. The method proceeds in three stages: spectral decomposition, SNR-guided exponent derivation, and embedding transformation.

### 3.1. Spectral Decomposition

Given a corpus embedding matrix **X** ∈ ℝ^{n×d}, we first center it by subtracting the column-wise mean **μ**:

(1) **X̄** = **X** - **1μ**^⊤

Centering reduces the influence of a global offset direction and yields a more stable covariance spectrum; we apply the same corpus-derived centering to both queries and documents to preserve geometric consistency. We then compute the eigendecomposition of the covariance matrix:

(2) **C** = (1/(n-1))**X̄**^⊤**X̄** = **UΛU**^⊤

where **Λ** = diag(λ₁, ..., λ_d) with λ₁ ≥ λ₂ ≥ ... ≥ λ_d, and **U** = [**u₁**, ..., **u_d**] are the corresponding eigenvectors.

### 3.2. SNR-Guided Exponent Derivation

The core insight of Spectral Tempering is that the appropriate tempering strength should be governed by the signal quality of the retained subspace. We formalize this through a local SNR analysis.

#### Noise Floor Estimation

We estimate the noise floor σ²_noise as the mean eigenvalue of the spectral tail:

(3) σ²_noise = (1/|𝒯|) ∑_{i∈𝒯} λ_i

where 𝒯 denotes the last 10% of eigenvalue indices. As shown in Figure 1, diverse retrieval encoders exhibit a consistently heavy-tailed eigenspectrum whose tail consistently plateaus into a stable noise floor, making this region a reliable, model-agnostic anchor for noise estimation. We verify in Section 4.2.4 that SpecTemp is insensitive to the exact percentile choice, confirming that this default requires no per-task tuning.

#### Local SNR Computation

The local SNR at rank i measures the excess energy above the noise floor:

(4) SNR(i) = max(0, (λ_i - σ²_noise) / σ²_noise)

We note that this quantity is not intended as a generative statistical estimate in the sense of spiked covariance models, but as a monotonic, spectrum-level proxy for relative signal dominance—sufficient for calibrating the tempering exponent. This quantity is large for head components where the signal dominates, and vanishes in the tail where eigenvalues converge to the noise floor.

#### Anchor Point and Adaptive γ(k)

To derive γ(k) without task-specific tuning, we need a reference point that separates the high-confidence signal regime from the transitional regime. We identify this anchor as the **knee point** of the SNR curve—the rank at which SNR transitions from rapid to gradual decay—detected via the Kneedle algorithm (Satopaa et al., 2011). Let k_knee denote this rank and S_ref = SNR(k_knee) the corresponding SNR value.

Since the k-th component defines the noise bottleneck of the retained subspace, we use its SNR as a conservative proxy for subspace signal quality. This ensures that the tempering strength is constrained by the worst-case noise exposure rather than being overly influenced by optimistic, high-variance directions. The adaptive exponent for target dimensionality k is then:

(5) γ(k) = min(1, SNR(k) / S_ref)