UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval

# UMER: Unifying Embedding and Ranking via Pair-Aware Discriminative Reasoning for Universal Multimodal Retrieval
Source: [https://arxiv.org/html/2608.18504](https://arxiv.org/html/2608.18504)
###### Abstract

Universal multimodal retrieval aims to support diverse instruction\-aware retrieval tasks, demanding both efficient corpus\-scale matching and fine\-grained semantic reasoning\. Recent MLLM\-based embedding methods typically derive representations from hidden states, while Chain\-of\-Thought \(CoT\) reasoning is emerging as a promising strategy for embedding enhancement by encoding intermediate semantic evidence into the representation space\. However, existing CoT methods typically use item\-wise reasoning over queries and candidates in isolation, providing no explicit evidence to distinguish a positive from a semantically confusable hard negative\. Moreover, contrastive embeddings capture global similarity but struggle with meta\-tasks requiring answer verification, category judgment or fine\-grained reasoning\. In this paper, we proposeUMER, aUnifiedMultimodalEmbedding andRanking framework for universal multimodal retrieval\. UMER replaces item\-wise reflection with Pair\-Aware Discriminative Reasoning, which compares query–candidate pairs to identify instruction\-relevant matching and discrepancy evidence\. UMER jointly learns contrastive embeddings for efficient global matching and discriminative ranking for explicit pairwise relevance judgment within a single MLLM\. A complementary mutual distillation strategy further transfers reliable pairwise preferences between the embedding and ranking functions\. On the MMEB\-V2 benchmark, UMER achieves state\-of\-the\-art performance under comparable experimental settings while supporting budget\-adjustable inference\.

## Introduction

![Refer to caption](https://arxiv.org/html/2608.18504v1/intro.png)Figure 1:Two overlooked issues in universal multimodal retrieval\. \(a\) Item\-wise self\-reflective CoT lacks pair\-aware evidence for hard\-negative discrimination\. \(b\) Different meta\- tasks require different capabilities, with embedding and ranking offering complementary strengths\.Universal multimodal retrieval aims to support diverse instruction\-aware tasks across images, texts, videos and documents within a unified framework\. A dominant solution is to encode heterogeneous inputs into a shared embedding space, enabling efficient retrieval over large\-scale multimodal corpora\. Early multimodal representation models, such as CLIP\([Radford et al\. 2021](https://arxiv.org/html/2608.18504#bib.bib17)\), ALIGN\([Jia et al\. 2021](https://arxiv.org/html/2608.18504#bib.bib7)\)and SigLIP\([Zhai et al\. 2023](https://arxiv.org/html/2608.18504#bib.bib24)\), typically use dual encoders to align visual and textual modalities and excel at conventional cross\-modal content retrieval\. Recently, MMEB and MMEB\-V2\([Jiang et al\. 2024b](https://arxiv.org/html/2608.18504#bib.bib10);[Meng et al\. 2025](https://arxiv.org/html/2608.18504#bib.bib15)\)have introduced more challenging unified retrieval benchmarks\. These benchmarks cover a broad range of meta\-tasks, including classification, question answering, visual grounding, image\-level and video\-level retrieval and visual document retrieval, imposing higher requirements on content matching and reasoning\-intensive relevance judgment\. Recent studies apply multimodal large language models \(MLLMs\)\([Wang et al\. 2024](https://arxiv.org/html/2608.18504#bib.bib19)\)to universal multimodal embedding for complex instruction\-aware retrieval\. Most MLLM\-based embedding methods\([Jiang et al\. 2024a](https://arxiv.org/html/2608.18504#bib.bib9);[Zhang et al\. 2025](https://arxiv.org/html/2608.18504#bib.bib25);[Lan et al\. 2025](https://arxiv.org/html/2608.18504#bib.bib11);[Gu et al\. 2025](https://arxiv.org/html/2608.18504#bib.bib4)\)extract a global representation from a special\-token hidden state and optimize it with contrastive learning\. To exploit LLM semantic understanding and reasoning, some works\([Cui et al\. 2025](https://arxiv.org/html/2608.18504#bib.bib3);[Wang et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib20)\)introduce Chain\-of\-Thought \(CoT\)\([Wei et al\. 2022](https://arxiv.org/html/2608.18504#bib.bib21)\)into multimodal retrieval\. These methods generate textual rationales and derive embeddings from the resulting hidden states, allowing representations to incorporate both the original multimodal inputs and intermediate reasoning\.

However, by examining how CoT is currently used in multimodal retrieval and what capabilities complex instruction\-aware retrieval tasks require, we identify two overlooked issues, as illustrated in Fig\.[1](https://arxiv.org/html/2608.18504#Sx1.F1)\.First, existing item\-wise CoT lacks pair\-aware discriminative evidence for representation learning\.Existing CoT\-based embedding methods\([Lan et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib12);[Jiang et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib8);[He et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib6)\)usually adopt an item\-wise self\-reflective reasoning paradigm, where the model independently describes the query or candidate\. Although such CoT enriches item\-level semantics, it does not model query–candidate interactions and cannot explain why a positive matches or a hard negative fails\. This is especially problematic for hard samples, which often share similar visual content, textual semantics or scene structures, but differ only in subtle attributes, relations, answer evidence or task intent\. Moreover, CoT is typically supervised by autoregressive cross\-entropy and only indirectly injected into the embedding token, causing a mismatch between language generation quality and embedding discriminability\.Second, different meta\-tasks in universal multimodal retrieval require different model capabilities\.Specifically, these meta\-tasks range from content\-matching tasks, such as image retrieval, video retrieval, moment retrieval and visual grounding, which primarily rely on global query–candidate similarity, to reasoning\-intensive tasks, such as question answering and classification, which require answer evidence or category\-specific conditions\. A single contrastive embedding objective is therefore insufficient to model task\-specific matching evidence and fine\-grained discrepancies\. In contrast, interaction\-based discriminative ranking\([Lin et al\. 2025](https://arxiv.org/html/2608.18504#bib.bib14);[Gu et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib5);[Li et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib13)\)explicitly models pairwise interactions and task satisfaction, complementing efficient embedding retrieval with stronger semantic judgment\. As further corroborated by Fig\.[4](https://arxiv.org/html/2608.18504#Sx4.F4)\(a\), the task\-dependent complementarity between embedding and ranking suggests that universal multimodal retrieval should combine representation\-based retrieval with explicit discriminative ranking\.

To address these issues, we proposeUMER, aUnifiedMultimodalEmbedding andRanking framework for universal multimodal retrieval\. Specifically, we design a Pair\-Aware Discriminative CoT paradigm that takes positive and hard\-negative query–candidate pairs as reasoning inputs, encouraging the model to explicitly compare matching and discrepancy evidence and to produce a deterministic ranking judgment\. We build a unified multi\-task framework that supports both embedding contrastive learning and ranking discriminative learning\. The embedding branch learns global matching in a metric space for corpus\-scale retrieval, while the ranking branch models query–candidate interactions through pair\-aware reasoning for complex tasks and hard samples\. To co\-optimize these capabilities, we introduce Complementary Mutual Distillation \(CMD\) between embedding and ranking\. The ranking branch refines the embedding space with discriminative knowledge, while the embedding branch stabilizes ranking with global semantic structure\. At inference, UMER supports a budget\-adjustable Embedding\-then\-CoT\-Ranking pipeline\. In efficiency\-oriented settings, it extracts embeddings in one forward pass and performs large\-scale retrieval through vector indexing\. In accuracy\-oriented or hard\-sample settings, it additionally generates pair\-aware discriminative CoT and relevance scores for explicit reasoning and reranking\.

The main contributions are summarized as follows:

- •We propose Pair\-Aware Discriminative Reasoning and construct pair\-aware CoT and ranking data, enabling explicit modeling of matching evidence for positives and discrepancy evidence for hard negatives\.
- •We present UMER, a unified MLLM\-based framework that jointly learns multimodal embeddings and discriminative ranking to address the heterogeneous requirements of universal multimodal retrieval\.
- •We introduce Complementary Mutual Distillation between embedding and ranking: discriminative ranking refines the embedding space, while global embedding structure stabilizes ranking\.
- •UMER achieves state\-of\-the\-art performance on the MMEB\-V2 benchmark under comparable experimental settings, demonstrating efficient vector retrieval and improved accuracy through explicit reasoning and ranking\.

![Refer to caption](https://arxiv.org/html/2608.18504v1/framework.png)Figure 2:Overview of UMER, a unified multimodal embedding and ranking framework\. \(a\) Query and candidate inputs are first encoded independently through masked branch attention to extract their embeddings, and are then jointly used for pair\-aware autoregressive reasoning followed by a ranking token\. \(b\) Multi\-task heads support metric embedding learning, generative CoT learning and discriminative ranking learning\. \(c\) Unified co\-optimization combines multi\-task supervision and complementary mutual distillation to jointly improve embedding and ranking\.
## Related Work

### Multimodal Embedding Models

Universal multimodal embedding has progressed from dual encoders such as CLIP, ALIGN and SigLIP\([Radford et al\. 2021](https://arxiv.org/html/2608.18504#bib.bib17);[Jia et al\. 2021](https://arxiv.org/html/2608.18504#bib.bib7);[Zhai et al\. 2023](https://arxiv.org/html/2608.18504#bib.bib24)\)to MLLM\-based encoders that follow task instructions and fuse multimodal inputs\. VLM2Vec\([Jiang et al\. 2024b](https://arxiv.org/html/2608.18504#bib.bib10)\)and VLM2Vec\-V2\([Meng et al\. 2025](https://arxiv.org/html/2608.18504#bib.bib15)\)convert MLLMs into embedders through contrastive training on the MMEB and MMEB\-V2 benchmarks\. UniME\-V2\([Gu et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib5)\)further improves discriminability via MLLM\-as\-a\-Judge hard\-negative mining and soft\-label alignment, and Qwen3\-VL\-Embedding\([Li et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib13)\)demonstrates strong performance with large\-scale contrastive training and reranker distillation\. Despite these advances, most MLLM\-based embedding models still rely on in\-batch contrastive objectives and provide limited explicit reasoning about why a query should match one candidate over another, especially for hard negatives with subtle semantic differences\.

### Reasoning\-Enhanced Multimodal Retrieval

Reasoning\-enhanced embedding models generate intermediate rationales before extracting representations\. UME\-R1\([Lan et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib12)\)formulates multimodal embedding as reasoning\-driven generation, where the MLLM autoregressively produces a rationale conditioned on the input and then derives the representation from the hidden states after reasoning\. Embed\-RL\([Jiang et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib8)\)further optimizes such rationales with reinforcement learning, using retrieval\-oriented rewards to align the reasoning trace with downstream matching signals\. However, explicit CoT reasoning introduces substantial decoding cost, as a rationale must be generated before each embedding\. To reduce this overhead, PLUME\([He et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib6)\)replaces verbalized CoT with a short rollout of continuous latent states under a progressive curriculum, while LaME\([Wu et al\. 2026b](https://arxiv.org/html/2608.18504#bib.bib23)\)performs latent reasoning through an information bottleneck with learnable reason tokens\. Despite these efficiency gains, these approaches remain item\-wise, enriching query and candidate representations independently rather than explicitly modeling the matching and discrepancy evidence between them\.

### Multimodal Reranking Models

MLLM rerankers jointly encode query–candidate pairs to improve fine\-grained relevance estimation after initial retrieval\. RagVL\([Chen et al\. 2024](https://arxiv.org/html/2608.18504#bib.bib2)\)shows that MLLMs can serve as strong multimodal rerankers in retrieval\-augmented generation by instruction\-tuning the model to filter retrieved images\. UniME\-V2\-Reranker\([Gu et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib5)\)trains a reranking model on mined hard negatives with pairwise and listwise optimization\. Qwen3\-VL\-Reranker\([Li et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib13)\)performs fine\-grained relevance estimation with a cross\-encoder architecture and is designed to complement Qwen3\-VL\-Embedding in a two\-stage retrieval pipeline\. Existing rerankers capture fine\-grained query–candidate interactions, but they are usually trained and deployed as separate second\-stage modules\. Consequently, ranking knowledge can reach the embedding only through discrete offline targets such as soft labels or mined pseudo\-negatives\. UMER instead unifies embedding, Pair\-Aware Discriminative Reasoning and ranking within a single MLLM, letting the two branches share the same backbone and be co\-optimized end\-to\-end\.

## Methodology

Method overview\.Given an instruction\-aware multimodal queryqqand a corpus of multimodal candidates𝒞\\mathcal\{C\}, universal multimodal retrieval ranks candidates in𝒞\\mathcal\{C\}by their relevance to the query instruction\. As illustrated in Fig\.[2](https://arxiv.org/html/2608.18504#Sx1.F2), UMER jointly learns two complementary functions with a single shared MLLM: an independently computable embedding function for corpus\-scale candidate retrieval and a pair\-aware ranking function for explicit verification\. The two functions share the MLLM backbone while using attention masks and task\-specific tokens to preserve distinct information flows\. The following subsections present the unified architecture, pair\-aware CoT data construction, complementary mutual distillation and training and inference procedures\.

### Unified Embedding and Ranking Architecture

Embedding encoding phase\.LetQQandCCdenote the tokenized multimodal query and candidate, respectively\. We denote a relevant candidate byc\+c^\{\+\}and a semantically confusable but irrelevant candidate byc−c^\{\-\}\. We appendMMlearnable embedding tokens to each input,Eq=\{q1,…,qM\}E\_\{q\}=\\\{q\_\{1\},\\ldots,q\_\{M\}\\\}for the query andEc=\{c1,…,cM\}E\_\{c\}=\\\{c\_\{1\},\\ldots,c\_\{M\}\\\}for the candidate, following recent MLLM retrievers that use dedicated tokens as explicit aggregation interfaces\([Sun et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib18)\)\. As shown in Fig\.[2](https://arxiv.org/html/2608.18504#Sx1.F2)\(a\), under branch\-wise attention masks,EqE\_\{q\}attends only toQQandEcE\_\{c\}only toCC\. This enables packed training while preserving the independent query and candidate encodings required for corpus indexing and approximate nearest\-neighbor search\. We mean\-pool andℓ2\\ell\_\{2\}\-normalize theMMembedding\-token states:

ex=Norm⁡\(1M​∑k=1Mhxk\),x∈\{q,c\}\.e\_\{x\}=\\mathrm\{Norm\}\\\!\\left\(\\frac\{1\}\{M\}\\sum\_\{k=1\}^\{M\}h\_\{x\_\{k\}\}\\right\),\\qquad x\\in\\\{q,c\\\}\.\(1\)wherehxkh\_\{x\_\{k\}\}denotes the final hidden state of thekk\-th embedding token forx=qx=q\(query\) orx=cx=c\(candidate\)\. TheMMtokens provide multiple aggregation slots whose pooled state forms a single fixed\-dimensional embedding\. We optimize the independently computable representations using an in\-batch contrastive objective augmented with mined hard negatives:

ℒemb=−1B∑i=1Blogexp⁡\(se​\(qi,ci\+\)\)∑c∈\{ci\+\}∪𝒩iexp⁡\(se​\(qi,c\)\),\\mathcal\{L\}\_\{\\rm emb\}=\-\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\log\\frac\{\\exp\(s^\{e\}\(q\_\{i\},c\_\{i\}^\{\+\}\)\)\}\{\\sum\_\{c\\in\\\{c\_\{i\}^\{\+\}\\\}\\cup\\mathcal\{N\}\_\{i\}\}\\exp\(s^\{e\}\(q\_\{i\},c\)\)\},\(2\)wherese​\(q,c\)=eq⊤​ec/τes^\{e\}\(q,c\)=e\_\{q\}^\{\\top\}e\_\{c\}/\\tau\_\{e\}is the temperature\-scaled cosine similarity,τe\\tau\_\{e\}is a temperature hyperparameter,BBis the batch size and𝒩i\\mathcal\{N\}\_\{i\}contains in\-batch candidates and mined hard negatives forqiq\_\{i\}\. This objective learns a shared metric space that supports independent corpus indexing, while the attention masks ensure that each embedding remains independent of cross\-item information unavailable at retrieval time\.

Pair\-aware reasoning and ranking phase\.Global similarity may fail to distinguish candidates that share visual content or surface semantics but differ in instruction\-critical evidence\. We therefore introduce pair\-aware discriminative reasoning for fine\-grained relevance estimation\. For each positive or hard\-negative pair\(q,ca\)\(q,c^\{a\}\), wherea∈\{\+,−\}a\\in\\\{\+,\-\\\}, we first encode its inputs under branch\-wise attention masks, then append a fixed promptPPand autoregressively generate a pair\-aware reasoning traceRa=\(r1a,…,rTaa\)R^\{a\}=\(r^\{a\}\_\{1\},\\ldots,r^\{a\}\_\{T\_\{a\}\}\)with full pairwise attention\. LetXa=\[Q;Eq;Ca;Ec;P\]X^\{a\}=\[Q;E\_\{q\};C^\{a\};E\_\{c\};P\]denote the resulting input sequence, whereCaC^\{a\}is the tokenized candidate, and letgag^\{a\}be the verified target reasoning trace\. We supervise the generation of reasoning traces with

ℒcot=∑a∈\{\+,−\}CE⁡\(ga,Xa\),\\mathcal\{L\}\_\{\\rm cot\}=\\sum\_\{a\\in\\\{\+,\-\\\}\}\\mathrm\{CE\}\\\!\\left\(g^\{a\};X^\{a\}\\right\),\(3\)whereCE⁡\(ga,Xa\)\\mathrm\{CE\}\(g^\{a\};X^\{a\}\)is the standard token\-level cross\-entropy under the model’s autoregressive distribution\. This objective trains pair\-aware discriminative CoT to ground its reasoning in jointly visible query–candidate evidence, rather than independently describing each item as in item\-wise self\-reflective CoT\. During training, we teacher\-force the target rationale; at inference, we decode it autoregressively\.

The rationale\-conditioned pair representation is then converted into a relevance judgment by a learnable\[𝚁𝙰𝙽𝙺\]\\mathtt\{\[RANK\]\}token followingRaR^\{a\}, which attends to the full pair and produces a logitzq,ca=w⊤​h\[𝚁𝙰𝙽𝙺\]\+bz\_\{q,c^\{a\}\}=w^\{\\top\}h\_\{\\mathtt\{\[RANK\]\}\}\+b, with probabilitypq,ca=σ⁡\(zq,ca\)p\_\{q,c^\{a\}\}=\\sigma\(z\_\{q,c^\{a\}\}\)\. We train the positive and hard\-negative pairs with binary cross\-entropy,

ℒbce=−log⁡pq,c\+−log⁡\(1−pq,c−\),\\mathcal\{L\}\_\{\\rm bce\}=\-\\log p\_\{q,c^\{\+\}\}\-\\log\\left\(1\-p\_\{q,c^\{\-\}\}\\right\),\(4\)and use a logistic pairwise objective to rank the positive above the hard negative:

ℒmargin=−log⁡σ⁡\(zq,c\+−zq,c−\)\.\\mathcal\{L\}\_\{\\rm margin\}=\-\\log\\sigma\\\!\\left\(z\_\{q,c^\{\+\}\}\-z\_\{q,c^\{\-\}\}\\right\)\.\(5\)We average each loss over valid triplets\(q,c\+,c−\)\(q,c^\{\+\},c^\{\-\}\)in the minibatch\. This smooth objective encourageszq,c\+\>zq,c−z\_\{q,c^\{\+\}\}\>z\_\{q,c^\{\-\}\}\.ℒcot\\mathcal\{L\}\_\{\\rm cot\}supervises the pair\-conditioned discriminative evidence,ℒbce\\mathcal\{L\}\_\{\\rm bce\}provides absolute relevance labels, andℒmargin\\mathcal\{L\}\_\{\\rm margin\}encourages the positive logit to exceed the hard\-negative logit\.

### Pair\-Aware Discriminative CoT Data Construction

Item\-wise CoT\([Lan et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib12);[Jiang et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib8);[He et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib6)\)can richly describe individual items while still omitting the evidence needed to distinguish a positive from a semantically confusable negative\. We therefore construct pair\-aware CoT and ranking supervision through three steps: hard\-negative mining, structured CoT generation and evidence\-sufficiency filtering\.

Hard\-negative mining\.Given the training set𝒟=\{\(qi,ci\+\)\}i=1N\\mathcal\{D\}=\\\{\(q\_\{i\},c\_\{i\}^\{\+\}\)\\\}\_\{i=1\}^\{N\}, we use a frozen off\-the\-shelf multimodal embedderf0f\_\{0\}to obtain normalized representations for every query and candidate\. For each queryqiq\_\{i\}, we rank non\-relevant candidates𝒞i−\\mathcal\{C\}\_\{i\}^\{\-\}by cosine similarity and retain the three highest\-scoring candidates:

ℋi=Top3c∈𝒞i−​f0​\(qi\)⊤​f0​\(c\)\.\\mathcal\{H\}\_\{i\}=\\mathrm\{Top3\}\_\{c\\in\\mathcal\{C\}\_\{i\}^\{\-\}\}\\,f\_\{0\}\(q\_\{i\}\)^\{\\top\}f\_\{0\}\(c\)\.\(6\)Each query thus yields one positive pair\(qi,ci\+\)\(q\_\{i\},c\_\{i\}^\{\+\}\)and three hard\-negative pairs\(qi,ci,j−\)\(q\_\{i\},c^\{\-\}\_\{i,j\}\), which serve as candidates for verified triplet construction\.

Structured CoT generation\.For each positive or mined hard\-negative pair, we prompt Qwen3\.5\-9B\([Qwen Team 2026](https://arxiv.org/html/2608.18504#bib.bib16)\)to generate structured pair\-aware CoT\. The CoT contains two fields:Query IntentandTarget Observations\. Query Intent specifies the evidence required by the instruction, whereas Target Observations identifies candidate evidence that supports or contradicts that intent\. This structure instantiates the pair\-aware reasoning paradigm in Fig\.[1](https://arxiv.org/html/2608.18504#Sx1.F1)\(a\) by evaluating a shared query intent against each positive or hard\-negative candidate\. We omit an explicit Yes/No answer so that relevance must be inferred from the CoT evidence rather than copied from a label\.

Evidence\-sufficiency filtering\.A fluent CoT does not necessarily contain sufficient discriminative evidence\. We therefore retain a generated CoT only if a lightweight text\-only verifier \(Qwen3\.5\-0\.8B\) can correctly infer the pair’s ground\-truth Yes/No relevance label from the CoT alone\. For each mined hard negativeci,j−c^\{\-\}\_\{i,j\}, we retain the triplet\(qi,ci\+,ci,j−\)\(q\_\{i\},c\_\{i\}^\{\+\},c^\{\-\}\_\{i,j\}\)only if the CoTs for both the positive pair and its corresponding hard\-negative pair pass this verification\. Applying this generate\-and\-verify pipeline to all training queries yields verified triplets for CoT generation, discriminative ranking and distillation\. More details are provided in the appendix\.

### Complementary Mutual Distillation

Embedding and ranking capture complementary relevance signals: embedding organizes global semantic similarity, whereas pair\-aware ranking resolves instruction\-critical confusions\. Rather than forcing the two functions to agree indiscriminately, we use selective bidirectional CMD to transfer a preference only to the*other*capability and only on meta\-tasks where its source is more reliable\. This preserves their complementary roles while allowing each to address the other’s blind spots\.

For each verified triplet, a frozen reference model produces temperature\-scaled positive–negative preference distributionsπ¯e\\bar\{\\pi\}\_\{e\}andπ¯r\\bar\{\\pi\}\_\{r\}for embedding and ranking, respectively;πe\\pi\_\{e\}andπr\\pi\_\{r\}are the corresponding distributions from the trainable model\. We assign each meta\-task to an embedding\-favored or ranking\-favored set according to its task attribute: Retrieval and Grounding/MR are treated as content\-matching tasks where global embedding structure is the primary teacher, while Classification and QA are treated as reasoning\-intensive tasks where pair\-aware ranking provides the primary teacher\. The gatege→rg\_\{e\\rightarrow r\}is one only when an embedding\-favored triplet is correctly ordered by the reference embedding; analogously,gr→eg\_\{r\\rightarrow e\}is one only when a ranking\-favored triplet is correctly ordered by the reference ranker\. Both gates are zero otherwise, preventing unreliable preferences from propagating\.

ℒcmd=ge→rKL\(sg\[π¯e\]∥πr\)\+gr→eKL\(sg\[π¯r\]∥πe\),\\begin\{array\}\[\]\{rcl\}\\mathcal\{L\}\_\{\\rm cmd\}&=&g\_\{e\\rightarrow r\}\\,\\mathrm\{KL\}\\\!\\left\(\\mathrm\{sg\}\[\\bar\{\\pi\}\_\{e\}\]\\\|\\pi\_\{r\}\\right\)\\\\ &&\+g\_\{r\\rightarrow e\}\\,\\mathrm\{KL\}\\\!\\left\(\\mathrm\{sg\}\[\\bar\{\\pi\}\_\{r\}\]\\\|\\pi\_\{e\}\\right\),\\end\{array\}\(7\)wheresg⁡\[⋅\]\\mathrm\{sg\}\[\\cdot\]stops gradients through the frozen reference, and the loss is averaged over activated triplets\. The first term is exclusively embedding\-to\-ranking distillation, and the second is exclusively ranking\-to\-embedding distillation\. Consequently, reliable embedding preferences stabilize ranking on content\-matching meta\-tasks, while reliable ranking preferences refine the embedding space on reasoning\-intensive meta\-tasks\.

### Progressive Training and Flexible Inference

Three\-stage training\.We train UMER progressively to prevent early generative and ranking objectives from destabilizing the shared backbone, following curriculum learning and staged reasoning\-to\-embedding optimization\([Bengio et al\. 2009](https://arxiv.org/html/2608.18504#bib.bib1);[He et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib6)\)\. The overall objective is

ℒall=ℒemb\+λcot​ℒcot\+λbce​ℒbce\+λmargin​ℒmargin\+λcmd​ℒcmd,\\begin\{array\}\[\]\{rcl\}\\mathcal\{L\}\_\{\\mathrm\{all\}\}&=&\\mathcal\{L\}\_\{\\rm emb\}\+\\lambda\_\{\\rm cot\}\\mathcal\{L\}\_\{\\rm cot\}\+\\lambda\_\{\\rm bce\}\\mathcal\{L\}\_\{\\rm bce\}\\\\ &&\+\\lambda\_\{\\rm margin\}\\mathcal\{L\}\_\{\\rm margin\}\+\\lambda\_\{\\rm cmd\}\\mathcal\{L\}\_\{\\rm cmd\},\\end\{array\}\(8\)where theλ\\lambdaterms weight the CoT, ranking and CMD losses relative toℒemb\\mathcal\{L\}\_\{\\rm emb\}\. Stage 1 learns a stable, indexable embedding space using onlyℒemb\\mathcal\{L\}\_\{\\rm emb\}on the full MMEB training set\. Stage 2 addsℒcot\\mathcal\{L\}\_\{\\rm cot\},ℒbce\\mathcal\{L\}\_\{\\rm bce\}andℒmargin\\mathcal\{L\}\_\{\\rm margin\}on verified pair\-aware triplets to learn discriminative evidence and relevance ordering\. Stage 3 freezes the Stage 2 checkpoint as the reference model and addsℒcmd\\mathcal\{L\}\_\{\\rm cmd\}, transferring reliable preferences across capabilities on complementary meta\-tasks\.

Table 1:Main results on MMEB\-V2\. The best and second\-best scores in each column are inboldandunderlined, respectively\.Flexible inference\.UMER supports budget\-adjustable inference in three modes:UMER\-Efor embedding\-only retrieval,UMER\-Rfor pair\-aware ranking andUMER\-Hfor combining both signals\. We precompute candidate embeddings with Eq\.[1](https://arxiv.org/html/2608.18504#Sx3.E1), build an approximate nearest\-neighbor index and independently encode each query to retrieve the top\-KKcandidates byse​\(q,c\)s^\{e\}\(q,c\);UMER\-Euses this score directly without autoregressive reasoning\. ForUMER\-RandUMER\-H, we apply the pair\-aware prompt only to the top\-KKcandidates to obtainzq,cz\_\{q,c\}, following the standard retrieve\-then\-rerank paradigm\([Chen et al\. 2024](https://arxiv.org/html/2608.18504#bib.bib2);[Li et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib13)\)\.UMER\-Ruseszq,cz\_\{q,c\}for reranking, whereasUMER\-Huses

shyb​\(q,c\)=se​\(q,c\)^\+α​zq,c^,s^\{\\rm hyb\}\(q,c\)=\\widehat\{s^\{e\}\(q,c\)\}\+\\alpha\\,\\widehat\{z\_\{q,c\}\},\(9\)where⋅^\\widehat\{\\cdot\}denotes normalization within the top\-KKlist andα\\alphacontrols the ranking contribution\.

## Experiments and Results

### Experimental Setup

Benchmark and metrics\.We evaluate UMER on MMEB\-V2\([Meng et al\. 2025](https://arxiv.org/html/2608.18504#bib.bib15)\), which comprises 78 tasks across image, video and visual\-document domains, covering classification, question answering, retrieval, grounding and moment retrieval\. We use the corrected versions of ViDoSeek\-page and MMLongBench\-page\. For a fair comparison, our training data are drawn from the same public source datasets as VLM2Vec\-V2\. Following the official protocol, we report Hit@1 for image and video tasks and NDCG@5 for visual\-document retrieval\. Each meta\-task score is the macro average over its constituent datasets, andAllis the macro average over all 78 tasks\. We report detailed results for each task and different model sizes in the supplementary material\.

Baselines\.We compare with eight representative unified multimodal retrieval methods: GME\([Zhang et al\. 2025](https://arxiv.org/html/2608.18504#bib.bib25)\), VLM2Vec\([Jiang et al\. 2024b](https://arxiv.org/html/2608.18504#bib.bib10)\), VLM2Vec\-V2\([Meng et al\. 2025](https://arxiv.org/html/2608.18504#bib.bib15)\), DUME\([Lan et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib12)\), Bottleneck Tokens \(BToks\)\([Sun et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib18)\), UME\-R1\([Lan et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib12)\), PLUME\([He et al\. 2026](https://arxiv.org/html/2608.18504#bib.bib6)\)and RIME\([Wu et al\. 2026a](https://arxiv.org/html/2608.18504#bib.bib22)\)\. All compared methods use Qwen2\-VL\-2B backbones\.

Implementation details\.UMER is initialized from Qwen2\-VL\-2B\-Instruct\. Each input receivesM=4M=4learnable embedding tokens\. Relative to the embedding objective, the effective weights for CoT generation, binary ranking, pairwise ranking and bidirectional distillation are\(λcot,λbce,λmargin,λcmd\)=\(0\.2,0\.1,0\.2,0\.2\)\(\\lambda\_\{\\rm cot\},\\lambda\_\{\\rm bce\},\\lambda\_\{\\rm margin\},\\lambda\_\{\\rm cmd\}\)=\(0\.2,0\.1,0\.2,0\.2\)\. At inference time, pair\-aware reasoning is applied to the topK=5K=5candidates retrieved by the embedding branch, with generation capped at 256 new tokens\. ForUMER\-H, the embedding and ranking scores are separately z\-score normalized over these five candidates, and we setα=2\.0\\alpha=2\.0in Eq\.[9](https://arxiv.org/html/2608.18504#Sx3.E9)\. Additional details are provided in the supplementary material\.

### Main Results on MMEB\-V2

Table[1](https://arxiv.org/html/2608.18504#Sx3.T1)shows thatUMER\-Hachieves the best overall score of 65\.5, and obtains the best modality\-level results on Image \(70\.4\), Video \(45\.0\) and VisDoc \(73\.6\)\. The two scoring functions exhibit different task preferences:UMER\-Rperforms better on I\-QA, V\-CLS, V\-QA and VisDoc\-VR, where relevance requires explicit answer or category verification, whereasUMER\-Eis stronger on I\-RET, I\-GD, VDRv1 and VDRv2, which rely more on global metric\-space matching\. This task\-dependent behavior is consistent with the intended roles of embedding retrieval and pair\-aware ranking described above\. As an inference\-time score fusion,UMER\-Himproves the overall score from 63\.1 forUMER\-Eand 63\.9 forUMER\-Rto 65\.5\.

### Ablation Studies

ConfigurationEmbedding mode \(UMER\-E\)Ranking mode \(UMER\-R\)Hybrid mode \(UMER\-H\)ImageVideoVisDocAllImageVideoVisDocAllImageVideoVisDocAllEmbedding only66\.537\.869\.260\.7––––––––\+\+pair\-aware CoT, w/o ranking losses68\.241\.171\.262\.944\.431\.857\.145\.468\.642\.271\.363\.3\+\+ranking losses, w/o CoT68\.040\.269\.362\.068\.643\.169\.463\.070\.845\.171\.265\.0\+\+pair\-aware CoT and ranking losses67\.939\.671\.162\.467\.244\.272\.163\.470\.344\.973\.465\.4Pair\-aware model, w/o CMD67\.939\.671\.162\.467\.244\.272\.163\.470\.344\.973\.465\.4\+\+ranking→\\rightarrowembedding only67\.940\.471\.562\.768\.543\.370\.963\.470\.644\.772\.765\.3\+\+embedding→\\rightarrowranking only67\.840\.470\.162\.269\.243\.971\.163\.970\.845\.172\.665\.5\+\+always\-on bidirectional CMD68\.040\.271\.162\.668\.543\.671\.063\.670\.745\.373\.065\.5\+\+selective bidirectional CMD \(full\)68\.041\.172\.263\.168\.544\.071\.863\.970\.445\.073\.665\.5Table 2:Controlled ablations of pair\-aware CoT, ranking supervision and complementary mutual distillation \(CMD\)\. The best and second\-best scores in each column are inboldandunderlined, respectively\.Pair\-aware reasoning and ranking\.Table[2](https://arxiv.org/html/2608.18504#Sx4.T2)separates the effects of pair\-aware reasoning and ranking supervision\. Adding pair\-aware CoT alone lifts UMER\-E further to 62\.9, showing that discriminative pair evidence directly benefits the embedding space, but leaves UMER\-R at 45\.4 because the\[𝚁𝙰𝙽𝙺\]\\mathtt\{\[RANK\]\}head is not calibrated\. Adding ranking losses alone brings UMER\-E from 60\.7 to 62\.0 and yields 65\.0 in hybrid mode, confirming that pairwise labels already provide effective supervision\. Combining both signals restores UMER\-R to 63\.4 and reaches 65\.4 in hybrid mode, indicating that the two forms of supervision are complementary rather than substitutable\.

Complementary mutual distillation\.As shown in Table[2](https://arxiv.org/html/2608.18504#Sx4.T2), each one\-way variant improves its intended branch, r→\\rightarrowe lifts UMER\-E from 62\.4 to 62\.7 with UMER\-R unchanged, and e→\\rightarrowr lifts UMER\-R from 63\.4 to 63\.9 with UMER\-E only marginally perturbed to 62\.2\. Applying both directions unconditionally captures only part of the benefit \(62\.6 / 63\.6\) because agreement is enforced even when the teacher is incorrect\. The selective bidirectional gate improves UMER\-E and UMER\-R to 63\.1 and 63\.9, showing that the two branches can co\-evolve without collapsing into identical decision functions\.

### In\-Depth Analysis

Hard\-Negative Separation: Item\-Wise vs\. Pair\-Aware Embeddings\.Both UME\-R1 and UMER\-E yield independently indexable embeddings, but differ in how these representations are learned: UME\-R1 relies on item\-wise reasoning, whereas UMER\-E is optimized within a pair\-aware reasoning and supervision framework\. We compare them on the same 3,600 queries from all 36 image tasks\. For each query, we define the separation margin as the similarity of its highest\-scoring positive minus that of its highest\-scoring non\-positive\. Fig\.[3](https://arxiv.org/html/2608.18504#Sx4.F3)\(a\) shows that UMER\-E increases the task\-macro\-averaged positive\-to\-hard\-negative cosine margin from 5\.24 to 6\.80 points\. While the two models achieve comparable positive\-margin coverage, Fig\.[3](https://arxiv.org/html/2608.18504#Sx4.F3)\(b\) reveals a widening advantage under stricter thresholds: UMER\-E retains 5\.0 and 6\.8 percentage points more queries at margins of 10 and 20, respectively\. These results indicate that pair\-aware embeddings separate successful matches more decisively from their hardest distractors\.

From Capability Specialization to Complementary Transfer\.Fig\.[4](https://arxiv.org/html/2608.18504#Sx4.F4)\(a\) reveals a systematic capability split within the same unified Stage 2 model: embedding outperforms ranking on content\-matching Retrieval and Grounding/MR, whereas ranking outperforms embedding on reasoning\-intensive CLS and QA\. This task\-dependent reversal directly supports the motivation in Fig\.[1](https://arxiv.org/html/2608.18504#Sx1.F1)\(b\): heterogeneous retrieval tasks require distinct inductive biases, and neither metric\-space embedding nor pair\-aware ranking is uniformly optimal\. Unification is therefore not merely an architectural convenience, but a means to retain and coordinate both capabilities within one model\. Building on this specialization, Fig\.[4](https://arxiv.org/html/2608.18504#Sx4.F4)\(b\) examines whether CMD can convert complementary strengths into transferable supervision\. On embedding\-favored Retrieval and Grounding/MR, ranking gains 1\.6 and 0\.8 points, consistent with embedding\-to\-ranking transfer; on ranking\-favored CLS and QA, embedding gains 0\.3 and 0\.5 points, supporting the reverse direction\. Together with the positive gains for both functions across all groups, these targeted improvements indicate that CMD narrows capability gaps without collapsing the specialization that makes the two functions complementary\.

Figure 3:Embedding separation under item\-wise and pair\-aware supervision on 3,600 queries from the 36 image tasks of MMEB\-V2\. \(a\) Bootstrap distribution of the macro\-averaged positive–hard\-negative margin\. \(b\) Fraction of queries exceeding margin thresholds\.Figure 4:Capability specialization and transfer via CMD\. \(a\) Embedding is stronger for content matching, while ranking is stronger for reasoning\-intensive relevance judgment\. \(b\) CMD transfers these complementary strengths between the two functions\.
### Accuracy–Efficiency Tradeoff

ModelKKScore↑\\uparrowReasoning TokensQuery LatencyIndexing Time\(/query\)\(s/query\)↓\\downarrow\(s/candidate\)↓\\downarrowUME\-R1–60\.13529\.96311\.755PLUME–61\.680\.3290\.366UMER\-E–63\.100\.0840\.118UMER\-H364\.841013\.5660\.0\.565\.566519\.8801066\.0130542\.7642066\.1250881\.726Table 3:Accuracy–efficiency on MMEB\-V2 using one NVIDIA A100 GPU \(batch size=1\)\.UMER\-Hreuses theUMER\-Eindex; 0\. denotes no extra indexing cost\.Embedding efficiency\.Table[3](https://arxiv.org/html/2608.18504#Sx4.T3)shows thatUMER\-Eavoids the sequential reasoning required before embedding extraction by UME\-R1 and PLUME\. It achieves the best embedding\-only score of 63\.1 with zero reasoning tokens, yielding query/indexing speedups of118\.6×118\.6\\times/99\.6×99\.6\\timesover UME\-R1 and3\.9×3\.9\\times/3\.1×3\.1\\timesover PLUME\. Since indexing cost scales with corpus size, this advantage is particularly important for large\-scale retrieval\.

Budget\-adjustable inference\.UMER\-Hreuses theUMER\-Eindex and spends additional computation only on online top\-KKreranking\. AsKKincreases from 3 to 20, tokens and latency grow nearly linearly, whereas the score saturates from 64\.8 to 66\.1\. The defaultK=5K=5reaches 65\.5, capturing 80% of the maximum observed gain overUMER\-Eat only 24% of theK=20K=20latency, providing a favorable accuracy–efficiency balance\.

## Conclusion

In this paper, we presented UMER, a unified multimodal embedding and ranking framework for universal multimodal retrieval\. UMER introduces Pair\-Aware Discriminative Reasoning to explicitly compare a query with a candidate and identify matching or discrepancy evidence, overcoming the limited discriminative supervision of item\-wise CoT\. A shared MLLM jointly learns independently indexable embeddings, pair\-aware CoT generation and discriminative ranking, while complementary mutual distillation co\-optimizes global metric matching and fine\-grained relevance judgment\. Extensive experiments on MMEB\-V2 show that UMER establishes a new state of the art under comparable experimental settings, validating the complementary value of efficient embedding retrieval and explicit reasoning\-based ranking for universal multimodal retrieval\.

## References

- Bengio et al\. \(2009\)Bengio, Y\.; Louradour, J\.; Collobert, R\.; and Weston, J\. 2009\.Curriculum Learning\.In*Proceedings of the 26th Annual International Conference on Machine Learning*, 41–48\.
- Chen et al\. \(2024\)Chen, Z\.; Xu, C\.; Qi, Y\.; and Guo, J\. 2024\.MLLM Is a Strong Reranker: Advancing Multimodal Retrieval\-Augmented Generation via Knowledge\-Enhanced Reranking and Noise\-Injected Training\.arXiv:2407\.21439\.
- Cui et al\. \(2025\)Cui, X\.; Cheng, J\.; Chen, H\.\-y\.; Shukla, S\. N\.; Awasthi, A\.; Pan, X\.; Ahuja, C\.; Mishra, S\. K\.; Yang, Y\.; Xiao, J\.; Guo, Q\.; Lim, S\.\-N\.; Singh, A\.; and Fan, X\. 2025\.Think Then Embed: Generative Context Improves Multimodal Embedding\.arXiv:2510\.05014\.
- Gu et al\. \(2025\)Gu, T\.; Yang, K\.; Feng, Z\.; Wang, X\.; Zhang, Y\.; Long, D\.; Chen, Y\.; Cai, W\.; and Deng, J\. 2025\.Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs\.*arXiv preprint arXiv:2504\.17432*\.
- Gu et al\. \(2026\)Gu, T\.; Yang, K\.; Zhang, K\.; An, X\.; Feng, Z\.; Zhang, Y\.; Cai, W\.; Deng, J\.; and Bing, L\. 2026\.UniME\-V2: MLLM\-as\-a\-Judge for Universal Multimodal Embedding Learning\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 40, 21378–21386\.
- He et al\. \(2026\)He, C\.; Hao, X\.; Yang, T\.; Ma, Y\.; Jia, Y\.; Wu, L\.; Zhao, C\.; Guo, H\.; and Wang, J\. 2026\.PLUME: Latent Reasoning Based Universal Multimodal Embedding\.arXiv:2604\.02073\.
- Jia et al\. \(2021\)Jia, C\.; Yang, Y\.; Xia, Y\.; Chen, Y\.\-T\.; Parekh, Z\.; Pham, H\.; Le, Q\. V\.; Sung, Y\.; Li, Z\.; and Duerig, T\. 2021\.Scaling Up Visual and Vision\-Language Representation Learning With Noisy Text Supervision\.In*Proceedings of the 38th International Conference on Machine Learning*, volume 139 of*Proceedings of Machine Learning Research*, 4904–4916\. PMLR\.
- Jiang et al\. \(2026\)Jiang, H\.; Wang, Y\.; Zhu, Y\.; Lu, X\.; Qin, W\.; Wang, M\.; Wan, P\.; and Tang, Y\. 2026\.Embed\-RL: Reinforcement Learning for Reasoning\-Driven Multimodal Embeddings\.arXiv:2602\.13823\.
- Jiang et al\. \(2024a\)Jiang, T\.; Song, M\.; Zhang, Z\.; Huang, H\.; Deng, W\.; Sun, F\.; Zhang, Q\.; Wang, D\.; and Zhuang, F\. 2024a\.E5\-V: Universal Embeddings with Multimodal Large Language Models\.*arXiv preprint arXiv:2407\.12580*\.
- Jiang et al\. \(2024b\)Jiang, Z\.; Meng, R\.; Yang, X\.; Yavuz, S\.; Zhou, Y\.; and Chen, W\. 2024b\.VLM2Vec: Training Vision\-Language Models for Massive Multimodal Embedding Tasks\.arXiv:2410\.05160\.
- Lan et al\. \(2025\)Lan, Z\.; Niu, L\.; Meng, F\.; Zhou, J\.; and Su, J\. 2025\.LLaVE: Large Language and Vision Embedding Models with Hardness\-Weighted Contrastive Learning\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, 13721–13735\. Suzhou, China: Association for Computational Linguistics\.
- Lan et al\. \(2026\)Lan, Z\.; Niu, L\.; Meng, F\.; Zhou, J\.; and Su, J\. 2026\.UME\-R1: Exploring Reasoning\-Driven Generative Multimodal Embeddings\.Accepted by ICLR 2026, arXiv:2511\.00405\.
- Li et al\. \(2026\)Li, M\.; Zhang, Y\.; Long, D\.; Chen, K\.; Song, S\.; Bai, S\.; Yang, Z\.; Xie, P\.; Yang, A\.; Liu, D\.; Zhou, J\.; and Lin, J\. 2026\.Qwen3\-VL\-Embedding and Qwen3\-VL\-Reranker: A Unified Framework for State\-of\-the\-Art Multimodal Retrieval and Ranking\.arXiv:2601\.04720\.
- Lin et al\. \(2025\)Lin, S\.\-C\.; Lee, C\.; Shoeybi, M\.; Lin, J\.; Catanzaro, B\.; and Ping, W\. 2025\.MM\-Embed: Universal Multimodal Retrieval with Multimodal LLMs\.In*International Conference on Learning Representations*\.
- Meng et al\. \(2025\)Meng, R\.; Jiang, Z\.; Liu, Y\.; Su, M\.; Yang, X\.; Fu, Y\.; Qin, C\.; Chen, Z\.; Xu, R\.; Xiong, C\.; Zhou, Y\.; Chen, W\.; and Yavuz, S\. 2025\.VLM2Vec\-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents\.arXiv:2507\.04590\.
- Qwen Team \(2026\)Qwen Team\. 2026\.Qwen3\.5: Towards Native Multimodal Agents\.Qwen Blog\.
- Radford et al\. \(2021\)Radford, A\.; Kim, J\. W\.; Hallacy, C\.; Ramesh, A\.; Goh, G\.; Agarwal, S\.; Sastry, G\.; Askell, A\.; Mishkin, P\.; Clark, J\.; Krueger, G\.; and Sutskever, I\. 2021\.Learning Transferable Visual Models from Natural Language Supervision\.In*Proceedings of the 38th International Conference on Machine Learning*, volume 139 of*Proceedings of Machine Learning Research*, 8748–8763\. PMLR\.
- Sun et al\. \(2026\)Sun, S\.; Ren, J\.; Liao, Z\.; Mao, D\.; Ren, X\.; Zhang, Y\.; Zhao, H\.; Lin, W\.; Jiang, S\.; Zhang, L\.; and Zheng, Y\. 2026\.Bottleneck Tokens for Unified Multimodal Retrieval\.arXiv:2604\.11095\.
- Wang et al\. \(2024\)Wang, P\.; Bai, S\.; Tan, S\.; Wang, S\.; Fan, Z\.; Bai, J\.; Chen, K\.; Liu, X\.; Wang, J\.; Ge, W\.; Fan, Y\.; Dang, K\.; Du, M\.; Ren, X\.; Men, R\.; Liu, D\.; Zhou, C\.; Zhou, J\.; and Lin, J\. 2024\.Qwen2\-VL: Enhancing Vision\-Language Model’s Perception of the World at Any Resolution\.arXiv:2409\.12191\.
- Wang et al\. \(2026\)Wang, Y\.; Yu, H\.; Bian, W\.; Long, J\.; Liang, X\.; Feng, C\.; and Li, H\. 2026\.MMEmb\-R1: Reasoning\-Enhanced Multimodal Embedding with Pair\-Aware Selection and Adaptive Control\.arXiv:2604\.06156\.
- Wei et al\. \(2022\)Wei, J\.; Wang, X\.; Schuurmans, D\.; Bosma, M\.; Ichter, B\.; Xia, F\.; Chi, E\. H\.; Le, Q\. V\.; and Zhou, D\. 2022\.Chain\-of\-Thought Prompting Elicits Reasoning in Large Language Models\.In*Advances in Neural Information Processing Systems*, volume 35, 24824–24837\.
- Wu et al\. \(2026a\)Wu, P\.; Mei, K\.; Ma, F\.; Chai, B\.; Lan, Z\.; Zhao, C\.; Yan, S\.; Chen, J\.; Hu, Z\.; Peng, Y\.; Lin, B\.; Zhou, J\.; Yin, D\.; Wang, T\.; Rao, F\.; Lv, J\.; Li, H\.; and Sun, X\. 2026a\.Beyond Chain\-of\-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings\.arXiv:2604\.22280\.
- Wu et al\. \(2026b\)Wu, P\.; Yang, B\.; Ma, F\.; Chai, B\.; Lin, B\.; Yuan, W\.; Yang, F\.; Gao, T\.; Li, H\.; and Sun, X\. 2026b\.LaME: Learning to Think in Latent Space for Multimodal Embedding via Information Bottleneck\.arXiv:2606\.13061\.
- Zhai et al\. \(2023\)Zhai, X\.; Mustafa, B\.; Kolesnikov, A\.; and Beyer, L\. 2023\.Sigmoid Loss for Language Image Pre\-Training\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, 11975–11986\.
- Zhang et al\. \(2025\)Zhang, X\.; Zhang, Y\.; Xie, W\.; Li, M\.; Dai, Z\.; Long, D\.; Xie, P\.; Zhang, M\.; Li, W\.; and Zhang, M\. 2025\.Bridging Modalities: Improving Universal Multimodal Retrieval by Multimodal Large Language Models\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 9274–9285\.

## Supplementary Material

## Appendix AImplementation and Evaluation Details

Table[1](https://arxiv.org/html/2608.18504#A1.T1)consolidates the complete implementation and evaluation settings for UMER\. The training mixture spans image, video, and visual\-document sources, with curated positives replacing raw targets when they are available\. Each training instance is paired with a retrieved hard negative for contrastive and pairwise supervision\. The pair\-aware CoT targets in this mixture are generated and audited offline before training, as detailed below\.

Table 1:Implementation and evaluation settings for UMER\.
## Appendix BPair\-Aware CoT Data Construction

We construct pair\-aware rationales offline in three stages\. The pipeline starts with a source\-specific query–positive pair, then obtains a semantically close but non\-relevant candidate, and finally creates a separate rationale for each of the positive and negative pairs\. This makes the supervision suitable for learning which details distinguish the labeled target from a plausible distractor, rather than simply describing a query in isolation\.

Stage 1: hard\-negative mining\.We embed queries and source\-local galleries with a frozen Qwen3\-VL\-Embedding\-8B model\. For every query, the miner masks all annotated positives and searches the remaining candidates by cosine similarity\. To avoid treating likely missing positives as negatives, it discards any candidate whose similarity is at least0\.750\.75times the similarity of the annotated positive, and ranks only the remaining candidates\. We retain a pool of at most three candidates\. For retrieval sources that provide an independently reviewed candidate list, the reviewed non\-relevant candidate is preferred, with the embedding\-mined pool used as a fallback\. The final training configuration takes one offline hard negative per query: the first candidate for the embedding branch and the first candidate with an available CoT record for pairwise reranking\.

Stage 2: structured CoT generation\.Qwen3\.5\-9B independently annotates every positive and selected hard\-negative pair\. It receives the query side and the target side, including any associated image or video, but not the pair label\. The output is constrained to two fields:*query intent*, a concrete statement of the query\-specific retrieval need, and*target observations*, a list of target\-side facts relevant to that need\. The prompt explicitly asks for salient overlaps as well as missing or conflicting details, while forbidding a final match/non\-match decision\. We sample three candidate rationales per pair for image and retrieval sources and one for video and visual\-document sources, using temperature0\.70\.7and top\-pp0\.90\.9\. The generation limit is 384 tokens for image/retrieval data and 192 tokens for video/document data\. During training, a canonical CoT record is retrieved by its normalized query–target pair key; a reranking example is formed only when both its positive and selected negative have such a record\.

Stage 2: image/retrieval CoT generation\. *You will see a retrieval query and a target\. Produce objective notes only\.* Return valid JSON with two keys:query\_intent, one concrete sentence stating the query’s retrieval need and constraints; andtarget\_observations, three to eight short target\-side facts relevant to those constraints\. Include relevant entities, attributes, actions, relations, or text cues\. Use neutral, non\-generic wording and do not state a final match/non\-match decision\.

Stage 3: text\-only evidence verification\.A Qwen3\.5\-0\.8B auditor sees only the two generated fields—not the original query, target, media, source dataset, or relevance label—and returnsmatch,not\_match, orunknown\. A strict audit passes a trace only when its prediction agrees with the pair label, the schema is valid, and no final decision leaks into the rationale\. Four source sidecars contain 6,632,206 canonical pair records; the released configuration uses the first schema\-valid image/retrieval trace and one video/document trace per pair\.

Stage 3 verifier prompt\.*Use only the generated notes, not the original query, target, media, labels, priors, or outside facts\.*Returnmatchonly for concrete support,not\_matchonly for concrete conflict, andunknownotherwise\. An asserted decision without observations is insufficient\.

## Appendix CComplete MMEB\-V2 Results

### Qwen2\-VL\-2B Results

Table[2](https://arxiv.org/html/2608.18504#A3.T2)reports every MMEB\-V2 task\.UMER\-Hattains the best overall, image, video, and visual\-document averages \(65\.5, 70\.4, 45\.0, and 73\.6\), surpassing RIME by 1\.4 points overall; detailed task\-level outcomes are listed below\.

Table 2:Complete 78\-task MMEB\-V2 comparison of Qwen2\-VL\-2B models\. Only baselines with published 2B task\-level results are included\. E, R and H denote embedding retrieval, pair\-aware ranking and hybrid inference; image/video use Hit@1 and visual documents use NDCG@5\. Best and second\-best values areboldandunderlined\.
### Qwen2\-VL\-7B Results

Table[3](https://arxiv.org/html/2608.18504#A3.T3)compares Qwen2\-VL\-7B models with published 78\-task breakdowns\. With the sameK=5K=5hybrid configuration,UMER\-Hreaches 70\.6 overall and 74\.3/50\.5/80\.0 on image/video/visual\-document tasks\.

TaskGME\-7BVLM2Vec\-7BDUME\-7BUME\-R1\-7BRIME\-7BUMER\-EUMER\-RUMER\-HAvg \- All \(78 tasks\)57\.852\.355\.964\.568\.667\.069\.670\.6Avg \- Image \(36 tasks\)56\.065\.566\.471\.373\.471\.272\.874\.3Avg \- Video \(18 tasks\)38\.433\.729\.447\.549\.445\.249\.950\.5Avg \- VisDoc \(24 tasks\)75\.246\.460\.367\.175\.676\.879\.680\.0I\-CLS \(10\)57\.762\.764\.267\.170\.366\.267\.568\.0I\-QA \(10\)34\.756\.957\.069\.271\.769\.773\.173\.7I\-RET \(12\)71\.269\.470\.871\.973\.272\.374\.575\.6I\-GD \(4\)59\.382\.281\.884\.986\.384\.680\.087\.7V\-CLS \(5\)37\.439\.132\.948\.652\.648\.056\.555\.2V\-QA \(5\)50\.430\.047\.460\.762\.054\.760\.560\.7V\-RET \(5\)28\.429\.08\.638\.238\.435\.639\.040\.5V\-MR \(3\)37\.038\.928\.039\.341\.640\.739\.542\.5VD\-ViDoRe\-V1 \(10\)89\.456\.967\.175\.780\.981\.384\.485\.0VD\-ViDoRe\-V2 \(4\)55\.69\.435\.250\.555\.657\.861\.862\.1VD\-VisRAG \(6\)85\.059\.182\.683\.785\.885\.187\.587\.9VD\-OOD \(4\)44\.438\.134\.937\.666\.972\.573\.673\.8ImageNet\-1K64\.680\.176\.680\.480\.982\.081\.482\.8N24News50\.579\.777\.282\.382\.780\.874\.578\.5HatefulMemes53\.669\.779\.679\.076\.254\.160\.260\.2VOC200780\.380\.785\.590\.891\.074\.175\.676\.5SUN39769\.577\.474\.680\.380\.677\.079\.079\.9Place36539\.137\.441\.946\.845\.543\.844\.144\.1ImageNet\-A41\.258\.148\.653\.957\.458\.061\.160\.9ImageNet\-R83\.973\.988\.890\.189\.990\.893\.292\.2ObjectNet69\.040\.144\.842\.372\.774\.579\.178\.0Country21124\.829\.824\.725\.026\.426\.727\.327\.3OK\-VQA33\.256\.861\.671\.774\.167\.670\.071\.9A\-OKVQA21\.047\.351\.458\.761\.859\.158\.561\.4DocVQA41\.489\.786\.393\.894\.494\.796\.095\.7InfographicsVQA20\.360\.062\.379\.279\.176\.676\.779\.2ChartQA17\.856\.949\.875\.177\.464\.262\.164\.9Visual7W22\.252\.752\.155\.254\.964\.967\.369\.1ScienceQA28\.038\.545\.553\.759\.050\.762\.858\.9VizWiz39\.039\.944\.351\.655\.356\.260\.960\.9GQA76\.955\.146\.969\.373\.676\.187\.986\.5TextVQA46\.871\.669\.983\.587\.087\.088\.888\.8VisDial60\.881\.975\.780\.782\.683\.188\.188\.4CIRR54\.951\.151\.655\.360\.054\.060\.059\.0VisualNews\_t2i79\.780\.576\.976\.879\.878\.880\.981\.4VisualNews\_i2t83\.681\.282\.382\.083\.582\.184\.184\.5MSCOCO\_t2i71\.277\.277\.178\.377\.878\.580\.481\.6MSCOCO\_i2t57\.773\.971\.271\.472\.673\.676\.476\.2NIGHTS67\.667\.669\.668\.168\.664\.966\.067\.8WebQA91\.488\.390\.390\.990\.789\.889\.391\.4FashionIQ37\.817\.120\.523\.426\.027\.126\.629\.6Wiki\-SS\-NQ78\.262\.370\.672\.576\.373\.679\.379\.7OVEN75\.166\.570\.571\.468\.672\.168\.073\.0EDIS96\.085\.792\.892\.091\.689\.994\.894\.5MSCOCO31\.475\.772\.372\.772\.072\.573\.675\.2RefCOCO60\.987\.686\.891\.491\.792\.394\.894\.2RefCOCO\-Matching78\.484\.685\.191\.193\.792\.860\.490\.8Visual7W\-Pointing66\.581\.083\.184\.287\.980\.691\.390\.4K70039\.735\.527\.342\.855\.051\.556\.055\.8SmthSmthV230\.632\.125\.150\.455\.149\.760\.960\.3HMDB5147\.942\.242\.658\.358\.950\.359\.558\.9UCF10154\.761\.848\.870\.068\.564\.275\.672\.7Breakfast14\.323\.820\.821\.525\.424\.230\.728\.4MVBench46\.628\.547\.458\.259\.952\.855\.257\.5Video\-MME39\.227\.840\.247\.349\.644\.145\.746\.4NExTQA53\.620\.348\.669\.669\.755\.371\.869\.1EgoSchema46\.821\.850\.452\.455\.650\.257\.458\.0ActivityNetQA65\.651\.450\.276\.075\.170\.972\.272\.3DiDeMo26\.429\.30\.140\.038\.436\.237\.440\.4MSR\-VTT31\.834\.50\.138\.941\.536\.942\.943\.7MSVD49\.746\.728\.860\.859\.458\.162\.463\.9VATEX24\.925\.513\.832\.632\.731\.233\.935\.2YouCook29\.19\.00\.018\.519\.915\.918\.219\.2QVHighlight59\.557\.729\.454\.956\.956\.149\.256\.6Charades\-STA14\.019\.815\.821\.921\.619\.920\.220\.5MomentSeeker37\.439\.338\.841\.146\.245\.949\.050\.4ViDoRe\_arxivqa86\.960\.266\.673\.684\.183\.885\.286\.0ViDoRe\_docvqa57\.534\.735\.841\.146\.948\.051\.651\.5ViDoRe\_infovqa91\.670\.472\.880\.885\.486\.889\.990\.3ViDoRe\_tabfquad94\.678\.289\.290\.295\.394\.795\.496\.1ViDoRe\_tatdqa74\.127\.638\.546\.752\.654\.661\.360\.9ViDoRe\_shiftproject96\.838\.661\.965\.073\.778\.478\.580\.9ViDoRe\_syntheticDocQA\_artificial\_intelligence99\.667\.769\.389\.596\.191\.797\.096\.4ViDoRe\_syntheticDocQA\_energy95\.360\.468\.485\.788\.588\.992\.492\.3ViDoRe\_syntheticDocQA\_government\_reports98\.861\.883\.189\.891\.791\.994\.896\.4ViDoRe\_syntheticDocQA\_healthcare\_industry99\.369\.984\.994\.394\.993\.998\.098\.8ViDoRe\_esg\_reports\_human\_labeled\_v263\.46\.840\.450\.460\.263\.968\.169\.9ViDoRe\_biomedical\_lectures\_v2\_multilingual49\.55\.137\.450\.751\.556\.560\.960\.4ViDoRe\_economics\_reports\_v2\_multilingual54\.213\.929\.657\.859\.254\.656\.956\.5ViDoRe\_esg\_reports\_v2\_multilingual55\.411\.933\.543\.251\.656\.161\.161\.5VisRAG\_ArxivQA87\.452\.677\.380\.584\.083\.385\.185\.5VisRAG\_ChartQA81\.970\.283\.485\.085\.485\.887\.989\.1VisRAG\_MP\-DocVQA89\.252\.883\.883\.487\.485\.088\.588\.5VisRAG\_SlideVQA94\.572\.891\.591\.594\.194\.095\.595\.8VisRAG\_InfoVQA93\.572\.088\.289\.291\.890\.994\.193\.9VisRAG\_PlotQA63\.434\.471\.372\.772\.171\.674\.074\.3ViDoSeek\-page23\.222\.320\.221\.385\.685\.689\.588\.9ViDoSeek\-doc83\.977\.873\.275\.380\.993\.093\.293\.1MMLongBench\-page16\.211\.810\.312\.352\.453\.653\.855\.1MMLongBench\-doc54\.340\.536\.041\.348\.857\.757\.958\.2
Table 3:Complete 78\-task MMEB\-V2 comparison of Qwen2\-VL\-7B models\. Only baselines with published 7B task\-level results are included\. E, R and H denote embedding retrieval, pair\-aware ranking and hybrid inference; image/video use Hit@1 and visual documents use NDCG@5\. Best and second\-best values areboldandunderlined\.

## Appendix DFurther Inference Analysis

### Fusion Configuration and Sensitivity

The default hybrid mode standardizes embedding and ranking scores within the retrieved candidate set and assigns the ranking\-score weight 2\.0\. Table[4](https://arxiv.org/html/2608.18504#A4.T4)varies only this weight\.UMER\-Hremains robust across the broad displayed range of 0\.5–5\.0, with all scores within 0\.8 points of the peak \(65\.5\); the small decline at larger weights indicates that neither branch should dominate every query\. All settings use identical candidates, normalization, and decoding, and introduce no retrained model\.

Table 4:Ranking\-score\-weight sensitivity ofUMER\-HatK=5K=5, with candidates, decoding configuration, and normalization fixed\.
### Embedding–Ranking Complementarity

Hybrid inference is useful only when the embedding and ranking branches contribute correct decisions on different queries\. If the ranking branch merely replaced embedding retrieval, nearly every query would fall into either “both correct” or “ranking only”; conversely, if ranking added no information, the embedding\-only set would be negligible\. We therefore compare the top decision made by the two branches on the same 83,530 MMEB\-V2 queries, without changing the retrieved candidates or evaluation labels\.

For each query, Table[5](https://arxiv.org/html/2608.18504#A4.T5)assigns the two decisions to one of four mutually exclusive outcomes:*Both*when both branches are correct,*E only*when only embedding retrieval is correct,*R only*when only pair\-aware ranking is correct, and*Neither*when both are incorrect\. We report percentages separately for reasoning/semantic and content\-matching query families so that the overlap is not obscured by their different failure profiles\.

Table 5:Decision overlap of embedding retrieval \(E\) and pair\-aware ranking \(R\) over 83,530 MMEB\-V2 queries\. Both, E only, R only, and Neither indicate which branch produces the correct top decision\. Each cell is a within\-family percentage\.Across all queries, 9\.0% are correct only with embedding retrieval and 10\.3% only with pair\-aware ranking, while 47\.2% are solved by both branches\. The two exclusive sets are comparable in size, showing that neither branch is uniformly dominant\. Ranking\-only successes are more frequent for reasoning/semantic queries \(12\.1% versus 9\.0% embedding\-only\), consistent with the benefit of comparing candidate\-specific evidence; content\-matching queries exhibit nearly symmetric exclusive sets \(8\.9% and 8\.3%\)\. This analysis does not claim that fusion corrects every exclusive error\. Instead, it establishes the complementary decision signal that hybrid inference can exploit; the controlled weight sweep in Table[4](https://arxiv.org/html/2608.18504#A4.T4)then measures how effectively the reported fusion configuration uses that signal\.

## Appendix EQualitative Results and Failure Cases

Each case fixes one query and contrasts the positive candidate with the top\-1 non\-positive candidate responsible for a correction or regression\. Both columns reproduce the independently generated pair\-aware CoT for that exact query–candidate pair; candidate ranks are shown before and after pair\-aware reranking\.

### Cross\-Case Analysis

Cases 1–3 isolate embedding\-to\-ranking corrections: the relevant candidate is already retrieved, but embedding similarity overweights a broad category or topic\. Pair\-aware ranking instead promotes evidence for the exact compositional, visual, or document\-level constraint\. Case 4 is a complementary failure case: embedding retrieval selects the requested background motorcycle, whereas pair\-aware ranking favors the visually salient foreground motorcycle and loses the required relative\-position constraint\. The complete query–candidate–CoT triplets below permit each interpretation to be checked against the corresponding saved model output\.Case 1: fine\-grained composition\.*Why embedding is wrong:*the top result retains the dog category and a compatible coat pattern, but it is a running outdoor image rather than the requested head\-focused portrait\.*Why ranking is correct:*the promoted candidate is a centred close\-up in which the facial attributes and portrait composition are visible, moving the positive from embedding rank 5 to ranking rank 1\.

Case 2: conjunctive attribute constraints\.*Why embedding is wrong:*it retrieves the visually similar underwater reference itself, preserving the scene while violating two explicit requirements—two animals of another species and no person\.*Why ranking is correct:*the promoted image shows two ray\-like animals and no human, raising the labelled positive from rank 4 to rank 1\.

Case 3: visual\-metaphor retrieval\.*Why embedding is wrong:*the video slide shares the broad topic of data and consequences, but its visual evidence is a video player rather than the requested sinking\-car metaphor\.*Why ranking is correct:*the promoted slide explicitly combines the car\-in\-water icon with the statement about blindly following data, moving the positive from rank 3 to rank 1\.

Case 4: spatial\-relation ranking regression\.*Why ranking is wrong:*embedding retrieval correctly selects the motorcycle behind the red foreground bike\. Pair\-aware ranking instead promotes the red motorcycle closest to the camera\. Its candidate\-specific CoT focuses on foreground salience and fails to preserve the query’s*behind/on\-the\-right*relation, even though the correct candidate remains in the top\-2 set\.

Case 1: Fine\-grained image composition\.The embedding top result preserves the dog category but fails the requested close\-up composition; pair\-aware ranking promotes the positive target\.Query\.Find an everyday image of the same breed dog, specifically a Cavalier King Charles Spaniel, focusing on its head\.The reference image, embedding top\-1 hard negative, and positive target are shown from left to right\.Embedding top\-1 hard\-negative candidate\.Candidate image shown in the centre panel \(embedding rank 1, pair\-aware rank 3\)\.Pair\-aware CoT\.Query intent\.Find an everyday image of a dog that is thesame breedas the one in the query image, specificallyfocusing on its head\.Target observations\.∙\\bulletThe target image shows a dogin mid\-motion, running on grass\.∙\\bulletThe dog has a black, white, and tan coat pattern\.∙\\bulletThe dog’s head is visible, showing a tri\-color coat pattern\.∙\\bulletThe background is a blurred green field, indicating an outdoor setting\.∙\\bulletThe dog’s tail is raised and appears to be in motion\.∙\\bulletThe image is a dynamic, action shot,not a static head shot\.∙\\bulletThe dog’s facial features arenot clearly visible due to motion blur\.Positive candidate\.Candidate image shown in the right panel \(embedding rank 5, pair\-aware rank 1\)\.Pair\-aware CoT\.Query intent\.Find an everyday image of the same breed dog, specifically aCavalier King Charles Spaniel,focusing on its head\.Target observations\.∙\\bulletThe image shows aCavalier King Charles Spanielwith long, floppy ears\.∙\\bulletThe dog has a white and tan coat with adistinct white blazeon its face\.∙\\bulletThe dog is positioned in front of a purple background\.∙\\bulletThe dog’s head is centered in the frame, showing its facial features\.∙\\bulletThe image is aclose\-up portrait, focusing on the dog’sheadand upper body\.∙\\bulletThe dog’s eyes are visible and appear to be dark\-colored\.∙\\bulletThe background is plain and out of focus, emphasizing the dog\.

Case 2: Attribute\-sensitive composed retrieval\.The embedding top result is visually plausible but violates the requested composition; pair\-aware ranking promotes the positive candidate\.Query\.Find an everyday image showing two animals that are of a different species from the first one, and no human appearing\.The query image, embedding top\-1 hard negative, and positive target are shown from left to right\. The embedding branch retrieves the unmodified query image itself\.Embedding top\-1 hard\-negative candidate\.Candidate image shown in the centre panel \(embedding rank 1, pair\-aware rank 2\)\.Pair\-aware CoT\.Query intent\.Find an everyday image that showstwo animals of a different speciesfrom the first one, andno humanappears\.Target observations\.∙\\bulletThe image containsonelarge, orange, jellyfish\-like creature\.∙\\bulletA human is visibleunderwater, wearing a scuba suit and helmet\.∙\\bulletThe human is positioned near the bottom of the frame, facing away from the camera\.∙\\bulletThe background consists of underwater coral and rocks\.∙\\bulletThe image includes a watermark in the bottom\-left corner\.∙\\bulletThe lighting appears natural, suggesting an underwater setting\.∙\\bulletThe human is not the primary subject of the image\.Positive candidate\.Candidate image shown in the right panel \(embedding rank 4, pair\-aware rank 1\)\.Pair\-aware CoT\.Query intent\.Find an everyday image showingtwo animalsthat are of adifferent speciesfrom the first one, andno humanappearing\.Target observations\.∙\\bulletThe image containstwolarge, black, ray\-like marine animals swimming together\.∙\\bulletThe animals have long, slender tails and are positioned in a natural underwater environment\.∙\\bulletThe background consists of a coral reef and blue water\.∙\\bulletThere areno human figuresvisible in the image\.∙\\bulletThe animals are the only visible marine life in the scene\.∙\\bulletThe image depicts a natural, wild habitat with no artificial elements\.∙\\bulletThe animals arenot of the same speciesas the first one in the query image\.

Case 3: Visual\-metaphor document retrieval\.The embedding branch selects a semantically related video slide, whereas pair\-aware ranking promotes the slide containing the requested sinking\-car metaphor\.Query\.The slides use a visual metaphor of a car sinking in water to illustrate the consequence of blindly following data\.The text query card is followed by the embedding top\-1 hard\-negative slide and the positive slide promoted by ranking\.Embedding top\-1 hard\-negative candidate\.Document shown in the centre panel \(embedding rank 1, pair\-aware rank 2\)\.Pair\-aware CoT\.Query intent\.The slides use aYouTube videotitled ’Why Data Matters: Context of Data Reveals Answers’ to show the consequence of blindly following data\.Target observations\.∙\\bulletThe target text explicitly states the video title is ’Why Data Matters: Context of Data Reveals Answers’\.∙\\bulletThe target text confirms the video is a YouTube video\.∙\\bulletThe target textdoes not mention any visual elementslike a video player interface or video thumbnails\.∙\\bulletThe target text does not reference any specific YouTube channel or channel name\.Positive candidate\.Document shown in the right panel \(embedding rank 3, pair\-aware rank 1\)\.Pair\-aware CoT\.Query intent\.The slides use a visual metaphor of acar sinking in waterto illustrate the consequence ofblindly following data\.Target observations\.∙\\bulletThe target text describes the metaphor as’blindly following data’, which aligns with the visual of acar sinking\.∙\\bulletThe target text explicitly states that this metaphor implies ’augment decision makers with objective, trustworthy, and relevant data’, which is not visually represented in the slide\.∙\\bulletThe slide includes a small icon of acar sinking, but the text does not explicitly state that this is the consequence of blind data following\.∙\\bulletThe target text does not mention the visual metaphor or the specific consequence it implies, which is the focus of the query\.

Case 4: Spatial\-relation ranking regression\.Embedding retrieval correctly selects the motorcycle behind the red foreground bike, but pair\-aware ranking promotes the motorcycle closest to the camera\.Query\.Identify the motorcycle behind the red motorcycle on the right\. The green outline marks the requested background motorcycle and the red outline marks the foreground distractor for visualization only\.The image context, salient foreground distractor, and requested background motorcycle are shown from left to right\.Pair\-aware ranking top\-1 hard\-negative candidate\.*motorcycle closest to camera*\(embedding rank 2, pair\-aware rank 1\)\.Pair\-aware CoT\.Query intent\.Select the portion of the image that shows themotorcycle closest to the camera, based on its position and visual prominence in the scene\.Target observations\.∙\\bulletThe motorcycle is the central subject, positioned prominently in theforeground\.∙\\bulletIt is the only motorcycle clearly visible in the image, with no other motorcycles overlapping it\.∙\\bulletThe motorcycle’sfront wheel and handlebarsare clearly in focus, indicating it is the closest to the camera\.∙\\bulletThe motorcycle’s red and white color scheme is distinct and contrasts with the gravel ground and surrounding objects\.∙\\bulletThe motorcycle’s branding, including ’Marlboro’ and ’PLAYBOY’, is clearly visible on its side fairing\.∙\\bulletThe motorcycle is angled slightly to the right, with its front wheel pointing toward the viewer\.∙\\bulletThe motorcycle is stationary and appears to be the primary subject of the image\.Positive candidate\.*Cycle behind red cycle on right*\(embedding rank 1, pair\-aware rank 2\)\.Pair\-aware CoT\.Query intent\.Identify and extract the portion of the image that corresponds to the’Cycle behind red cycle on right’description, specifically focusing on the motorcycle visible in thebackground behind the red motorcycle in the foreground\.Target observations\.∙\\bulletThe target text refers to acycle behind the red cycle on the right, indicating a motorcycle positioned to the right of the red motorcycle in the foreground\.∙\\bulletThe target text specifieson right, matching the orientation of the red motorcycle in the image\.∙\\bulletThe target text includes the wordCycle, a general term for a motorcycle, matching the query’s focus on a vehicle behind the red motorcycle\.∙\\bulletThe target text does not mention any specific brand or model, which is not required by the query but is present in the image\.∙\\bulletThe target text does not include any additional context or attributes beyond the motorcycle’s position relative to the red motorcycle\.∙\\bulletThe target text is a direct description of the image content, matching the query’s request for a specific portion of the image\.∙\\bulletThe target text does not include any visual cues such as branding, logos, or text overlays that are present in the image\.