Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval

Hugging Face Daily Papers Papers
multimodal-retrieval chain-of-thought hard-negatives retrieval-augmented lvlm reranking

Summary

This paper introduces UniME-R1, an embedder-adviser framework for unified multimodal retrieval that generates Retrieval-Centric Chain-of-Thought (RC-CoT) conditioned on retrieval feedback, improving retrieval performance by learning from hard negatives.

Unified multimodal retrieval aims to identify candidates that satisfy complex user intent expressed through heterogeneous inputs. Although Large Vision-Language Model (LVLM)-based retrievers are efficient and scalable, directly encoding raw multimodal inputs often misses fine-grained discriminative cues, leading to confusion among semantically similar candidates. Recent methods mitigate this limitation by generating Chain-of-Thought (CoT) rationales to enrich the query representation. However, such reasoning is typically derived from the query alone: it explains what the query describes, but not what the retriever misunderstands. We argue that effective retrieval reasoning should instead be conditioned on retrieval feedback. Based on this insight, we introduce UniME-R1, an embedder-adviser framework that learns to reason over initially retrieved candidates and generate Retrieval-Centric Chain-of-Thought (RC-CoT). The adviser analyzes candidates individually to identify the discriminative cues confused by the embedder. If the target appears in the initial top-k set, UniME-R1 directly reranks the candidates; otherwise, it generates RC-CoT to refine the retrieval direction and performs full-corpus re-retrieval with a dual-mode embedder. To train the framework, we mine hard negatives to simulate realistic retrieval failures, jointly optimize direct retrieval and RC-CoT-augmented retrieval, and align the adviser with retrieval outcomes through supervised learning and retrieval-oriented reinforcement learning. Extensive experiments on MMEB-V2 and a diverse set of general multimodal retrieval benchmarks demonstrate that UniME-R1 consistently improves retrieval performance over strong baselines.
Original Article
View Cached Full Text

Cached at: 08/07/26, 05:56 AM

Paper page - Learning from Failures: Retrieval-Centric CoT via Hard Negatives for Unified Multimodal Retrieval

Source: https://huggingface.co/papers/2608.06060

Abstract

Unifiedmultimodalretrievalaimstoidentifycandidatesthatsatisfycomplexuserintentexpressedthroughheterogeneousinputs.AlthoughLargeVision-LanguageModel(LVLM)-basedretrieversareefficientandscalable,directlyencodingrawmultimodalinputsoftenmissesfine-graineddiscriminativecues,leadingtoconfusionamongsemanticallysimilarcandidates.RecentmethodsmitigatethislimitationbygeneratingChain-of-Thought(CoT)rationalestoenrichthequeryrepresentation.However,suchreasoningistypicallyderivedfromthequeryalone:itexplainswhatthequerydescribes,butnotwhattheretrievermisunderstands.Wearguethateffectiveretrievalreasoningshouldinsteadbeconditionedonretrievalfeedback.Basedonthisinsight,weintroduceUniME-R1,anembedder-adviserframeworkthatlearnstoreasonoverinitiallyretrievedcandidatesandgenerateRetrieval-CentricChain-of-Thought(RC-CoT).Theadviseranalyzescandidatesindividuallytoidentifythediscriminativecuesconfusedbytheembedder.Ifthetargetappearsintheinitialtop-kset,UniME-R1directlyreranksthecandidates;otherwise,itgeneratesRC-CoTtorefinetheretrievaldirectionandperformsfull-corpusre-retrievalwithadual-modeembedder.Totraintheframework,weminehardnegativestosimulaterealisticretrievalfailures,jointlyoptimizedirectretrievalandRC-CoT-augmentedretrieval,andaligntheadviserwithretrievaloutcomesthroughsupervisedlearningandretrieval-orientedreinforcementlearning.ExtensiveexperimentsonMMEB-V2andadiversesetofgeneralmultimodalretrievalbenchmarksdemonstratethatUniME-R1consistentlyimprovesretrievalperformanceoverstrongbaselines.

View arXiv pageView PDFGitHub3Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.06060 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.06060 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.06060 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Beyond Retrieval: Analytic Memory for Multimodal Agents

arXiv cs.AI

This paper introduces AdaMM, a framework that complements retrieval-based multimodal memory with analytic memory, enabling filtering, aggregation, ranking, and temporal comparison over accumulated observations. Experiments on MemEye and MemGallery benchmarks show improvements of up to 11.3% and 7.3% respectively.

The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation

arXiv cs.CL

This paper identifies and formalizes 'recorruption' in multimodal RAG, where adding accurate context causes models to abandon correct predictions due to attentional collapse (visual blindness and positional bias). The authors propose BAIR, a parameter-free inference-time framework that restores visual saliency and penalizes textual distractors, improving reliability across medical, fairness, and geospatial benchmarks.