SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum

Hugging Face Daily Papers Papers
representation-learning embedding-compression matryoshka-representation-learning spectral-basis retrieval mteb

Summary

SpecMRL 提出在参数化谱空间中重新组织特征以改进嵌入的前缀读出，在 MTEB 上以更少的嵌入维度超越标准 MRL 及其全维表现。

Matryoshka Representation Learning (MRL) is designed to support flexible representation sizes by jointly optimizing nested prefixes. While effective, this design lacks an explicit constraint to ensure that the leading portion provides a suitable compact encoding. To address this, we propose SpecMRL, which reorganizes the feature space to enable more effective prefix readout. Specifically, we extend MRL with a parameter-derived spectral basis organized with feature importance. SpecMRL otherwise follows standard MRL, with prefix readout performed in the learned spectral space. Experiments on MTEB show that SpecMRL outperforms MRL at all evaluated reduced dimensions, with larger gains under more aggressive compression. These results suggest that the learned spectral basis better prioritizes informative directions, enabling more effective representation compression. Notably, SpecMRL extends the performance frontier of MRL, surpassing full-dimensional MRL with only half the embedding dimensions.
Original Article
View Cached Full Text

Cached at: 10/20/26, 07:33 AM

Paper page - SpecMRL: Redesign Matryoshka Representation Learning within Parametric Spectrum

Source: https://huggingface.co/papers/2610.26824

Abstract

SpecMRL improves compressed embeddings by reorganizing features via a learned spectral basis that prioritizes informative directions, outperforming standard MRL at reduced dimensions.

Matryoshka Representation Learning(MRL) is designed to support flexible representation sizes by jointly optimizingnested prefixes. While effective, this design lacks an explicit constraint to ensure that the leading portion provides a suitable compact encoding. To address this, we propose SpecMRL, which reorganizes the feature space to enable more effectiveprefix readout. Specifically, we extend MRL with a parameter-derivedspectral basisorganized withfeature importance. SpecMRL otherwise follows standard MRL, withprefix readoutperformed in the learned spectral space. Experiments onMTEBshow that SpecMRL outperforms MRL at all evaluated reduced dimensions, with larger gains under more aggressive compression. These results suggest that the learnedspectral basisbetter prioritizes informative directions, enabling more effectiverepresentation compression. Notably, SpecMRL extends the performance frontier of MRL, surpassing full-dimensional MRL with only half the embedding dimensions.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2610\.26824

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.26824 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.26824 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.26824 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Spectral Rewiring for Exploration, Purification, and Model Merging

arXiv cs.LG

The paper introduces Subspace-Aligned Rewiring (SAR), a post-hoc editing method that retains the spectral core of RL updates to preserve reasoning gains, remove interference, and enable model merging across experts, achieving strong performance with minimal parameters.

A Broader Look into MLLM's Retrieval Heads

Hugging Face Daily Papers

该论文研究多模态大模型（MLLM）中的稀疏检索头在新增音频-视觉模态下的分布与共享机制，并通过排序目标优化这些注意力头，将 Qwen3-VL-8B 在 MMDocIR 上的页面 Recall@1 提升 6.03 个点。

Honey, I shrunk the embeddings: Matryoshka vs. PCA

Hacker News Top

Dylan Castillo compares Matryoshka Representation Learning (MRL) with Principal Component Analysis (PCA) for reducing embedding dimensions, running experiments across eight BEIR datasets to evaluate retrieval quality versus vector size reduction.