UEmbed: Unified Sparse and Dense Multimodal Embeddings

Hugging Face Daily Papers Papers
multimodal embeddings sparse-retrieval dense-retrieval decoder-only retrieval-augmented-generation

Summary

UEmbed is a decoder-only multimodal embedding model that produces both sparse and dense representations in a single forward pass, released at 2B, 4B, and 9B scales. It outperforms existing public-data-trained multimodal embedding models on MMEB-v2 and remains competitive on BEIR.

Sparse retrieval underpins modern search systems, from web search to retrieval-augmented generation. Existing work has introduced Learned Sparse Retrieval (LSR) to push beyond exact lexical matching toward richer semantics. Yet LSR has so far remained tied to encoder-style bidirectional architectures, and its extension to multimodal settings still relies heavily on auxiliary cross-modal modules. To address these limitations, we introduce UEmbed (Unified Embedding), a decoder-only multimodal embedding model that produces both sparse lexical and dense representations in one causal forward pass. UEmbed appends N learnable special tokens to the input and partitions the vocabulary into N disjoint subsets. Each token's causal hidden state predicts sparse weights over its assigned subset, and the N subsets are concatenated into the full sparse vector. Trained on public data, we release UEmbed at 2B, 4B, and 9B scales. UEmbed-9B reaches 71.8 (dense) and 71.0 (sparse) on MMEB-v2, outperforming multimodal embedding models trained on publicly available data (e.g., RzenEmbed). On BEIR, UEmbed also remains competitive with strong dense and sparse baselines. Furthermore, we demonstrate the practical utility of UEmbed across three dimensions: effectiveness, efficiency, and agentic applications. Overall, UEmbed offers a new paradigm: it unifies dense and sparse embeddings in one model, while further extending sparse retrieval to unify text and multimodal inputs.
Original Article
View Cached Full Text

Cached at: 08/04/26, 05:37 AM

Paper page - UEmbed: Unified Sparse and Dense Multimodal Embeddings

Source: https://huggingface.co/papers/2608.02583

Abstract

Sparseretrievalunderpinsmodernsearchsystems,fromwebsearchtoretrieval-augmentedgeneration.ExistingworkhasintroducedLearnedSparseRetrieval(LSR)topushbeyondexactlexicalmatchingtowardrichersemantics.YetLSRhassofarremainedtiedtoencoder-stylebidirectionalarchitectures,anditsextensiontomultimodalsettingsstillreliesheavilyonauxiliarycross-modalmodules.Toaddresstheselimitations,weintroduceUEmbed(UnifiedEmbedding),adecoder-onlymultimodalembeddingmodelthatproducesbothsparselexicalanddenserepresentationsinonecausalforwardpass.UEmbedappendsNlearnablespecialtokenstotheinputandpartitionsthevocabularyintoNdisjointsubsets.Eachtoken’scausalhiddenstatepredictssparseweightsoveritsassignedsubset,andtheNsubsetsareconcatenatedintothefullsparsevector.Trainedonpublicdata,wereleaseUEmbedat2B,4B,and9Bscales.UEmbed-9Breaches71.8(dense)and71.0(sparse)onMMEB-v2,outperformingmultimodalembeddingmodelstrainedonpubliclyavailabledata(e.g.,RzenEmbed).OnBEIR,UEmbedalsoremainscompetitivewithstrongdenseandsparsebaselines.Furthermore,wedemonstratethepracticalutilityofUEmbedacrossthreedimensions:effectiveness,efficiency,andagenticapplications.Overall,UEmbedoffersanewparadigm:itunifiesdenseandsparseembeddingsinonemodel,whilefurtherextendingsparseretrievaltounifytextandmultimodalinputs.

View arXiv pageView PDFProject pageGitHub2Add to collection

Get this paper in your agent:

hf papers read 2608\.02583

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper3

#### Alibaba-NLP/UEmbed-9B Feature Extraction• 8B• Updatedabout 2 hours ago • 4 #### Alibaba-NLP/UEmbed-2B Feature Extraction• 2B• Updatedabout 2 hours ago • 2 #### Alibaba-NLP/UEmbed-4B Feature Extraction• 5B• Updatedabout 2 hours ago • 1

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.02583 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.02583 in a Space README.md to link it from this page.

Collections including this paper1

Similar Articles

Douyin Multimodal Embedding Model Technical Report

Hugging Face Daily Papers

This technical report introduces Douyin Multimodal Embedding (DME), a two-stage trained model that combines contrastive pre-training with evidence-grounded latent reasoning and cross-conditional reconstruction, achieving state-of-the-art results on MMEB-v2 and deployment in Douyin search.

MVEB: Massive Video Embedding Benchmark

Hugging Face Daily Papers

This paper introduces MVEB, a large-scale benchmark for evaluating video embeddings across 23 tasks, finding that no single model dominates and that audio's contribution depends on dataset annotation provenance. It integrates into the MTEB ecosystem for unified multimodal evaluation.

Your Embedding Model is SMARTer Than You Think

Hugging Face Daily Papers

SMART is a framework that unlocks latent multi-vector capabilities in single-vector models for multimodal retrieval, improving state-of-the-art performance with reduced computational costs via contrastive training and late-interaction inference.

Gemma 2B multimodal model matches larger models without encoder

Reddit r/singularity

Google's Gemma 4 12B introduces an encoder-free multimodal architecture that competes with larger models, though benchmark comparisons show it trailing Qwen 2.5 9B on most tasks. The article also covers related developments including open-weight model security risks, Uber's Claude Code spending caps, and NeurIPS's misuse of an uncalibrated AI detector.