Learning Multimodal Embeddings with Evidence-Aligned Readout

Hugging Face Daily Papers Papers
multimodal-embeddings retrieval multimodal-llm contrastive-learning readout-strategy

Summary

这篇论文提出 EviAlign，将语义证据生成与边界读出（Boundary Readout）耦合到共享的多模态大语言模型中，联合训练生成与对比检索目标，在 12 个 MMEB 检索任务上以 500K 训练对达到 76.9 的平均 Recall@1，同时保持单向量索引与打分。

Multimodal large language models can expose task-relevant evidence through generation, but producing useful evidence does not by itself determine how it enters a retrieval embedding. We study whether the semantic organization of that evidence can also specify where representations are read. To address this question, we introduce EviAlign, which couples Semantic Evidence Generation with Boundary Readout in a shared multimodal large language model. It organizes evidence into five semantic units, reads the contextualized state at each unit boundary, and aggregates these states into a single normalized embedding. Generation and contrastive retrieval objectives jointly train this shared structure. With the same trailing readout, semantic evidence and free-form CoT yield nearly identical retrieval performance, suggesting that evidence organization alone does not explain the full gain. A controlled 2times3 study compares consistent and permuted evidence organization across three readout strategies, using training targets with matched evidence spans. With five readout states and the same mean pooling, the advantage of consistent semantic organization grows from 0.65 points at length-based training positions to 2.39 at evidence boundaries, yielding a 1.74-point co-design interaction. Across 12 MMEB retrieval tasks, EviAlign achieves 76.9 average Recall@1 with 500K training pairs while retaining single-vector indexing and scoring.
Original Article
View Cached Full Text

Cached at: 10/08/26, 12:20 PM

Paper page - Learning Multimodal Embeddings with Evidence-Aligned Readout

Source: https://huggingface.co/papers/2609.33659 Published on Sep 27

·

Submitted byhttps://huggingface.co/zrchen03

zrchenon Oct 8

Abstract

Multimodallargelanguagemodelscanexposetask-relevantevidencethroughgeneration,butproducingusefulevidencedoesnotbyitselfdeterminehowitentersaretrievalembedding.Westudywhetherthesemanticorganizationofthatevidencecanalsospecifywhererepresentationsareread.Toaddressthisquestion,weintroduceEviAlign,whichcouplesSemanticEvidenceGenerationwithBoundaryReadoutinasharedmultimodallargelanguagemodel.Itorganizesevidenceintofivesemanticunits,readsthecontextualizedstateateachunitboundary,andaggregatesthesestatesintoasinglenormalizedembedding.Generationandcontrastiveretrievalobjectivesjointlytrainthissharedstructure.Withthesametrailingreadout,semanticevidenceandfree-formCoTyieldnearlyidenticalretrievalperformance,suggestingthatevidenceorganizationalonedoesnotexplainthefullgain.Acontrolled2times3studycomparesconsistentandpermutedevidenceorganizationacrossthreereadoutstrategies,usingtrainingtargetswithmatchedevidencespans.Withfivereadoutstatesandthesamemeanpooling,theadvantageofconsistentsemanticorganizationgrowsfrom0.65pointsatlength-basedtrainingpositionsto2.39atevidenceboundaries,yieldinga1.74-pointco-designinteraction.Across12MMEBretrievaltasks,EviAlignachieves76.9averageRecall@1with500Ktrainingpairswhileretainingsingle-vectorindexingandscoring.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.33659

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.33659 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.33659 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.33659 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles