Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering

# Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering
Source: [https://arxiv.org/html/2608.02907](https://arxiv.org/html/2608.02907)
Jingchen Sun1,Shaobo Han2,Ruiyi Zhang3,Naresh Kumar Devulapally1,Ming Liu4, Yitao Long5,Vishnu Suresh Lokhande1,Changyou Chen1

1University at Buffalo,2NEC Laboratories America,3Adobe Research, 4Iowa State University,5New York University

###### Abstract

Multimodal retrievers are essential for knowledge\-based visual question answering, where they retrieve external evidence for image–question pairs\. However, existing contrastive training methods typically treat all unmatched query–document pairs as equally informative negatives, which is problematic because many unmatched documents may still be semantically relevant or partially useful\. We proposeBayesian Data Reweighting, a probabilistic framework that models query–document importance as latent variables and adaptively infers posterior weights to downweight likely false negatives\. With closed\-form posterior updates under conjugate priors and stochastic EM optimization, our method consistently improves retrieval accuracy across three retrievers and seven knowledge\-based VQA benchmarks\.

Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge\-Based Visual Question Answering

Jingchen Sun1, Shaobo Han2, Ruiyi Zhang3, Naresh Kumar Devulapally1, Ming Liu4,Yitao Long5,Vishnu Suresh Lokhande1,Changyou Chen11University at Buffalo,2NEC Laboratories America,3Adobe Research,4Iowa State University,5New York University

## 1Introduction

Knowledge\-based Visual Question Answering \(KB\-VQA\)Marinoet al\.\([2019](https://arxiv.org/html/2608.02907#bib.bib62)\); Schwenket al\.\([2022](https://arxiv.org/html/2608.02907#bib.bib63)\)extends the traditional VQA task by requiring models to incorporate external knowledge sources, such as structured knowledge graphsSpeeret al\.\([2017](https://arxiv.org/html/2608.02907#bib.bib14)\), unstructured textual corporaVrandečić and Krötzsch \([2014](https://arxiv.org/html/2608.02907#bib.bib18)\), or large\-scale encyclopedic documentsMensinket al\.\([2023](https://arxiv.org/html/2608.02907#bib.bib17)\)to answer questions\. These questions often involve commonsense reasoningZellerset al\.\([2019](https://arxiv.org/html/2608.02907#bib.bib68)\), fine\-grained factual knowledgeChenet al\.\([2023](https://arxiv.org/html/2608.02907#bib.bib16)\), or entity disambiguationJianet al\.\([2024](https://arxiv.org/html/2608.02907#bib.bib67)\), which is often absent from raw visual or linguistic input\. As such, KB\-VQA serves as a key benchmark for evaluating a model’s ability to integrate perception with world knowledgeCaffagniet al\.\([2024](https://arxiv.org/html/2608.02907#bib.bib65)\); Yan and Xie \([2024](https://arxiv.org/html/2608.02907#bib.bib66)\), and has significant implications for downstream applications in education, healthcare, and open\-domain dialog systems\.

![Refer to caption](https://arxiv.org/html/2608.02907v1/x1.png)Figure 1:Examples of false negative reweighting\.We select representative FN samples from EVQA to illustrate the false\-negative issue in KB\-VQA, where unpaired documents may still describe related entities or partially relevant evidence\. Unlike standard InfoNCE, which uniformly treats these samples as negatives and pushes them away from the query, our method BDR adaptively infers weights for FN passages, suppressing noisy contrastive signals and making the learning process more robust\.Recent advances in knowledge\-based visual question answering \(KB\-VQA\) have largely focused on improving the modeling of multimodal retrievers, including late\-interaction architecturesLinet al\.\([2023](https://arxiv.org/html/2608.02907#bib.bib20),[2024](https://arxiv.org/html/2608.02907#bib.bib21)\), unified embedding modelsJianget al\.\([2025](https://arxiv.org/html/2608.02907#bib.bib23)\); Weiet al\.\([2024](https://arxiv.org/html/2608.02907#bib.bib22)\); Linet al\.\([2025](https://arxiv.org/html/2608.02907#bib.bib61)\), and retrieval\-augmented frameworks with stronger generatorsLin and Byrne \([2022](https://arxiv.org/html/2608.02907#bib.bib19)\); Huet al\.\([2023](https://arxiv.org/html/2608.02907#bib.bib71)\)\. Despite these architectural progress, most existing retrieversLinet al\.\([2023](https://arxiv.org/html/2608.02907#bib.bib20),[2024](https://arxiv.org/html/2608.02907#bib.bib21)\); Caffagniet al\.\([2025](https://arxiv.org/html/2608.02907#bib.bib25)\); Jianget al\.\([2025](https://arxiv.org/html/2608.02907#bib.bib23)\)are still trained with the standard InfoNCE objectiveOordet al\.\([2018](https://arxiv.org/html/2608.02907#bib.bib26)\), which treats all non\-positive documents in a batch as equally informative negatives\.

However, this assumption is particularly problematic in KB\-VQA\. Unlike conventional image\-text retrieval, KB\-VQA relies on open\-domain external knowledge, where documents are often weakly annotated, semantically overlapping, and only partially aligned with the image\-question pair\. As illustrated in Figure[1](https://arxiv.org/html/2608.02907#S1.F1), an unpaired document is not necessarily irrelevant: it may describe a closely related entity, provide partially useful evidence, or even support the correct answer but remain unlabeled\. Such samples becomefalse negativeswhen they are incorrectly treated as negatives during contrastive training\. Pushing these semantically relevant documents away from the query introduces noisy gradients and can degrade retrieval performance, especially in knowledge\-intensive settings where entity ambiguity and incomplete evidence annotations are common\.

Prior studies have attempted to address false negatives through negative reweighting or debiased learning methodsWuet al\.\([2017](https://arxiv.org/html/2608.02907#bib.bib94)\); Schroffet al\.\([2015](https://arxiv.org/html/2608.02907#bib.bib55)\); Yehet al\.\([2022](https://arxiv.org/html/2608.02907#bib.bib93)\); Robinsonet al\.\([2020](https://arxiv.org/html/2608.02907#bib.bib59)\); Zhenget al\.\([2019](https://arxiv.org/html/2608.02907#bib.bib58)\); Chuanget al\.\([2020](https://arxiv.org/html/2608.02907#bib.bib28)\)\. However, these methods remain limited for knowledge\-intensive retrieval\. Sampling or reweighting\-based approaches often rely on heuristic scores or predefined rules, which may still assign large weights to semantically relevant but unlabeled documents and amplify noisy supervision\. Debiased contrastive learning methods usually apply global correction terms, making it difficult to capture the query\-dependent nature of false negatives in KB\-VQA\. Moreover, existing methods rarely model the uncertainty of whether an unmatched document is truly irrelevant or incorrectly labeled as negative\. As a result, they may still push potentially useful evidence away from the query, leading to suboptimal retriever training under incomplete supervision\.

To address this limitation, we propose a novelBayesian Data Reweighting \(BDR\)framework for noise\-aware multimodal retrieval\. Instead of assuming all unmatched query–document pairs contribute equally, BDR introduces latent importance weights to model the reliability and usefulness of each pair during training\. We formulate contrastive learning as a Bayesian reweighting problem, where sample\-wise weights are inferred from posterior distributions under selected priors\. This allows the model to adaptively downweight likely false negatives, without explicitly identifying their labels\. To make inference tractable and efficient, we derive closed\-form posterior updates via auxiliary variable augmentation and optimize the retriever with a stochastic approximation expectation\-maximization algorithm\. In this way, BDR provides a principled, instance\-adaptive training objective that mitigates noisy supervision and improves the robustness of KB\-VQA retrieval\.

Extensive experiments show that BDR consistently improves multimodal retrieval across seven knowledge\-based VQA benchmarks and three different LLM backbones\. Compared with standard InfoNCE and existing reweighting baselines, BDR achieves stronger retrieval accuracy and further improves downstream answer generation on InfoSeek and EVQA datasets when paired with frozen generators\. Detailed analyses show that BDR learns better\-separated query–document embeddings, improves Recall@K across retrieval budgets, and effectively assigns lower weights to annotated false negatives, validating its ability to mitigate noisy supervision\. Moreover, BDR incurs negligible additional training cost, making it a practical and scalable objective for robust KB\-VQA retrieval\.

Our main contributions are summarized as follows:First, we identify the false\-negative issue in KB\-VQA retrieval, where semantically relevant but unannotated documents are incorrectly treated as negatives under standard contrastive learning\.Second,we propose Bayesian Data Reweighting \(BDR\), a probabilistic framework that models query–document importance as latent variables and adaptively infers posterior weights to suppress noisy false negatives\.Third, we conduct extensive experiments across seven KB\-VQA benchmarks and three multimodal LLM backbones, demonstrating consistent retrieval improvements, stronger downstream VQA performance, meaningful false\-negative\-aware weighting, and negligible additional training overhead\.

## 2Related Work

#### Knowledge\-based Visual Question Answering\.

KB\-VQA extends traditional VQA by adopting a retrieval\-augmented generation \(RAG\) paradigmMarinoet al\.\([2019](https://arxiv.org/html/2608.02907#bib.bib62)\); Schwenket al\.\([2022](https://arxiv.org/html/2608.02907#bib.bib63)\)\. Recent advances in multimodal RAG have demonstrated strong effectiveness for KB\-VQA by integrating external textual knowledge into large multimodal language modelsCaffagniet al\.\([2024](https://arxiv.org/html/2608.02907#bib.bib65)\); Yan and Xie \([2024](https://arxiv.org/html/2608.02907#bib.bib66)\); Longet al\.\([2025](https://arxiv.org/html/2608.02907#bib.bib64)\)\. Despite these advances, KB\-VQA performance remains highly sensitive to retrieval qualityLin and Byrne \([2022](https://arxiv.org/html/2608.02907#bib.bib19)\); Linet al\.\([2022](https://arxiv.org/html/2608.02907#bib.bib44)\)\. Existing pipelines often retrieve redundant or weakly relevant passagesHaoet al\.\([2024](https://arxiv.org/html/2608.02907#bib.bib70)\), or fail to capture fine\-grained entities and attributes grounded in the visual sceneJianet al\.\([2024](https://arxiv.org/html/2608.02907#bib.bib67)\)\. These limitations motivate us to develop a more robust and effective retriever\.

#### Negative Reweighting in Contrastive Learning\.

False negatives are a common challenge in contrastive learning, where semantically relevant samples are incorrectly treated as negatives\. Existing methods typically mitigate this issue by correcting or reducing the contribution of suspicious negatives\. For example, debiased contrastive learningChuanget al\.\([2020](https://arxiv.org/html/2608.02907#bib.bib28)\)estimates the true\-negative contribution at the distribution level to reduce the bias introduced by false negatives\. Other methods identify potential false negatives using sample\-level similarity signals and then remove, cancel, or downweight them during trainingHuynhet al\.\([2022](https://arxiv.org/html/2608.02907#bib.bib60)\); Chenet al\.\([2021](https://arxiv.org/html/2608.02907#bib.bib86)\)\. Beyond false\-negative mitigation, several studies further combine false\-negative debiasing with hard\-negative reweighting\. These methods aim to suppress harmful false negatives while emphasizing informative hard negatives, often by using similarity scores, hardness\-aware sampling distributions, margin\-based objectivesSchroffet al\.\([2015](https://arxiv.org/html/2608.02907#bib.bib55)\); Robinsonet al\.\([2020](https://arxiv.org/html/2608.02907#bib.bib59)\); Kalantidiset al\.\([2020](https://arxiv.org/html/2608.02907#bib.bib84)\); Zhenget al\.\([2019](https://arxiv.org/html/2608.02907#bib.bib58)\); Xionget al\.\([2021](https://arxiv.org/html/2608.02907#bib.bib85)\)\.

However, most existing methods still rely on heuristic similarity thresholds, predefined sampling rules, or global correction terms\. As a result, they may be sensitive to hyperparameter choices and lack a principled mechanism to model the uncertainty of whether an unmatched sample is truly irrelevant or a false negative\. This limitation is particularly important in KB\-VQA retrieval, where external documents are weakly annotated, semantically dense, and often only partially aligned with the image–question pair\. Motivated by this, we propose Bayesian Data Reweighting, which treats sample importance as latent variables and infers posterior weights in an instance\-adaptive manner\.

## 3Bayesian Data Reweighting

Most prior retrievers are trained with the InfoNCE objectiveOordet al\.\([2018](https://arxiv.org/html/2608.02907#bib.bib26)\)\. Given a training set𝒳=\{\(𝐯i,𝐝i\)\}i=1M\\mathcal\{X\}=\\\{\(\\mathbf\{v\}\_\{i\},\\mathbf\{d\}\_\{i\}\)\\\}\_\{i=1\}^\{M\}, each pair\(𝐯i,𝐝i\)\(\\mathbf\{v\}\_\{i\},\\mathbf\{d\}\_\{i\}\)is treated as a positive pair, while\(𝐯i,𝐝k\)\(\\mathbf\{v\}\_\{i\},\\mathbf\{d\}\_\{k\}\)withk≠ik\\neq iare treated as negatives\. Let𝐪i\\mathbf\{q\}\_\{i\}and𝐳k\\mathbf\{z\}\_\{k\}denote the query and document embeddings\. We define the exponential cosine similarity assi​k=exp⁡\(sim​\(𝐪i,𝐳k\)/τ\),s\_\{ik\}=\\exp\\left\(\\mathrm\{sim\}\(\\mathbf\{q\}\_\{i\},\\mathbf\{z\}\_\{k\}\)/\\tau\\right\),whereτ\>0\\tau\>0is the temperature\. For a mini\-batchℬ\\mathcal\{B\}, the standard InfoNCE loss is

ℒ​\(ℬ;𝜽\)=−1B​∑i=1Blog⁡si\+si\+\+∑k=1Ksi​k−\.\\displaystyle\\mathcal\{L\}\(\\mathcal\{B\};\\boldsymbol\{\\theta\}\)=\-\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\log\\frac\{s\_\{i^\{\+\}\}\}\{s\_\{i^\{\+\}\}\+\\sum\_\{k=1\}^\{K\}s\_\{ik^\{\-\}\}\}\.
In KB\-VQA retrieval, unmatched documents are not always true negatives, since they may still contain related or partially useful evidence\. Inspired by classical importance samplingKatharopoulos and Fleuret \([2018](https://arxiv.org/html/2608.02907#bib.bib10)\)and reweighted probabilistic modelsWanget al\.\([2017](https://arxiv.org/html/2608.02907#bib.bib89)\), we introduce a probabilistic variablewi​k≥0w\_\{ik\}\\geq 0to model the importance of each negative pair\(𝐯i,𝐝i​k−\)\(\\mathbf\{v\}\_\{i\},\\mathbf\{d\}\_\{ik^\{\-\}\}\)\. This leads to the following weighted contrastive objective:

ℒBDR​\(ℬ;𝜽\)=−1B​∑i=1Blog⁡si\+si\+\+∑k=1Kwi​k​si​k−,\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{BDR\}\}\(\\mathcal\{B\};\\boldsymbol\{\\theta\}\)=\-\\frac\{1\}\{B\}\\sum\_\{i=1\}^\{B\}\\log\\frac\{s\_\{i^\{\+\}\}\}\{s\_\{i^\{\+\}\}\+\\sum\_\{k=1\}^\{K\}w\_\{ik\}s\_\{ik^\{\-\}\}\},wherewi​kw\_\{ik\}controls the contribution of the negative pair\(𝐯i,𝐝i​k−\)\(\\mathbf\{v\}\_\{i\},\\mathbf\{d\}\_\{ik^\{\-\}\}\)\. Whenwi​k=1w\_\{ik\}=1for all negatives, this objective reduces to the standard InfoNCE loss\.

#### Bayesian Modeling of Local Weights

We treat the negative weights𝐰=\{wi​k\}\\mathbf\{w\}=\\\{w\_\{ik\}\\\}as latent random variables and infer them jointly with the retriever parameters𝜽\\boldsymbol\{\\theta\}during training, without assuming access to a clean validation setRenet al\.\([2018](https://arxiv.org/html/2608.02907#bib.bib12)\)or relying on gradient\-based importance estimatesKatharopoulos and Fleuret \([2018](https://arxiv.org/html/2608.02907#bib.bib10)\)\. We interpret the weighted contrastive objective as the following sample\-wise pseudo\-likelihood:

pi​\(𝐰i,𝜽\)∝si\+si\+\+∑k=1Kwi​k​si​k−\.p\_\{i\}\(\\mathbf\{w\}\_\{i\},\\boldsymbol\{\\theta\}\)\\propto\\frac\{s\_\{i^\{\+\}\}\}\{s\_\{i^\{\+\}\}\+\\sum\_\{k=1\}^\{K\}w\_\{ik\}s\_\{ik^\{\-\}\}\}\.
The joint posterior distribution is

p​\(𝐰,𝜽∣𝒳\)∝∏i=1\|𝒳\|pi​\(𝐰i,𝜽\)​∏i,kp​\(wi​k\)​p​\(𝜽\)\.p\(\\mathbf\{w\},\\boldsymbol\{\\theta\}\\mid\\mathcal\{X\}\)\\propto\\prod\_\{i=1\}^\{\|\\mathcal\{X\}\|\}p\_\{i\}\(\\mathbf\{w\}\_\{i\},\\boldsymbol\{\\theta\}\)\\prod\_\{i,k\}p\(w\_\{ik\}\)p\(\\boldsymbol\{\\theta\}\)\.
Our goal is thus to infer the posterior distributionp​\(𝐰,𝜽∣𝒳\)p\(\\mathbf\{w\},\\boldsymbol\{\\theta\}\\mid\\mathcal\{X\}\)in a principled Bayesian manner\. However, direct Bayesian inference is intractable\. The normalization termλi=si\+\+∑k=1Kwi​k​si​k−\\lambda\_\{i\}=s\_\{i^\{\+\}\}\+\\sum\_\{k=1\}^\{K\}w\_\{ik\}s\_\{ik^\{\-\}\}couples all negative weights through a summation\. As a result, the conditional posterior distributions do not admit closed\-form solutions under common prior choices\. This breaks the standard conditional conjugacy and prevents direct Gibbs sampling\.

### 3\.1Laplace\-Augmented Factorization

To address the non\-conjugacy caused by the normalization term, we introduce an auxiliary variableuiu\_\{i\}using a Laplace\-transform\-based augmentation techniqueTanner and Wong \([1987](https://arxiv.org/html/2608.02907#bib.bib82)\)\. Denote the normalization term byλi=si\+\+∑k=1Kwi​k​si​k−\\lambda\_\{i\}=s\_\{i^\{\+\}\}\+\\sum\_\{k=1\}^\{K\}w\_\{ik\}s\_\{ik^\{\-\}\}\. Using the identityλi−1=∫0∞exp⁡\(−λi​ui\)​dui\\lambda\_\{i\}^\{\-1\}=\\int\_\{0\}^\{\\infty\}\\exp\(\-\\lambda\_\{i\}u\_\{i\}\)\\,\\mathrm\{d\}u\_\{i\}forλi\>0\\lambda\_\{i\}\>0, the sample\-wise pseudo\-likelihood admits the following marginally equivalent augmented form:

si\+λi=∫0∞si\+​e−λi​ui​dui\.\\displaystyle\\frac\{s\_\{i^\{\+\}\}\}\{\\lambda\_\{i\}\}=\\int\_\{0\}^\{\\infty\}s\_\{i^\{\+\}\}e^\{\-\\lambda\_\{i\}u\_\{i\}\}\\,\\mathrm\{d\}u\_\{i\}\.Conditioned onuiu\_\{i\}, substitutingλi\\lambda\_\{i\}yields the following factorized augmented pseudo\-likelihood:

p~i​\(ui,𝐰i,𝜽\)∝si\+​e−ui​si\+​∏k=1Ke−ui​wi​k​si​k−\.\\displaystyle\\widetilde\{p\}\_\{i\}\(u\_\{i\},\\mathbf\{w\}\_\{i\},\\boldsymbol\{\\theta\}\)\\propto s\_\{i^\{\+\}\}e^\{\-u\_\{i\}s\_\{i^\{\+\}\}\}\\prod\_\{k=1\}^\{K\}e^\{\-u\_\{i\}w\_\{ik\}s\_\{ik^\{\-\}\}\}\.Thus, the original normalized pseudo\-likelihood is converted into a conditionally factorized exponential\-family form, enabling closed\-form conditional posterior updates under conjugate priors\.

#### Posterior Inference via Gibbs Sampling\.

After Laplace augmentation, the latent variables admit closed\-form conditional posteriors and can be efficiently sampled by alternating

ui∼p​\(ui∣𝐰i,𝜽,𝒳\),wi​k∼p​\(wi​k∣ui,𝜽,𝒳\)\.u\_\{i\}\\sim p\(u\_\{i\}\\mid\\mathbf\{w\}\_\{i\},\\boldsymbol\{\\theta\},\\mathcal\{X\}\),\\quad w\_\{ik\}\\sim p\(w\_\{ik\}\\mid u\_\{i\},\\boldsymbol\{\\theta\},\\mathcal\{X\}\)\.Here,uiu\_\{i\}acts as an auxiliary normalization variable that decouples the negative weights, making eachwi​kw\_\{ik\}conditionally independent givenuiu\_\{i\}\.

### 3\.2Prior Selection and Posterior Inference

We place a Gamma prior on the auxiliary variable,ui∼Gamma⁡\(au,bu\)u\_\{i\}\\sim\\operatorname\{Gamma\}\(a\_\{u\},b\_\{u\}\), sinceui\>0u\_\{i\}\>0and is conjugate to the exponential factore−ui​λie^\{\-u\_\{i\}\\lambda\_\{i\}\}in the augmented pseudo\-likelihood\. Its conditional posterior is

ui∣−∼Gamma\(au,bu\+si\+\+∑k=1Kwi​ksi​k−\)\.u\_\{i\}\\mid\-\\sim\\operatorname\{Gamma\}\\left\(a\_\{u\},\\;b\_\{u\}\+s\_\{i^\{\+\}\}\+\\sum\_\{k=1\}^\{K\}w\_\{ik\}s\_\{ik^\{\-\}\}\\right\)\.For the negative pair weightswi​kw\_\{ik\}, we consider two prior choices\.

#### Gamma Prior\.

For smooth continuous reweighting, we use a Gamma priorwi​k∼Gamma⁡\(aw,bw\)w\_\{ik\}\\sim\\operatorname\{Gamma\}\(a\_\{w\},b\_\{w\}\), whereawa\_\{w\}andbwb\_\{w\}are the shape and rate parameters\. Combining it with the augmented pseudo\-likelihood gives

p​\(wi​k∣−\)∝wi​kaw−1​exp⁡\[−\(bw\+ui​si​k−\)​wi​k\]\.p\(w\_\{ik\}\\mid\-\)\\propto w\_\{ik\}^\{a\_\{w\}\-1\}\\exp\\\!\\left\[\-\(b\_\{w\}\+u\_\{i\}s\_\{ik^\{\-\}\}\)w\_\{ik\}\\right\]\.and hence

wi​k∣−∼Gamma\(aw,bw\+uisi​k−\)\.w\_\{ik\}\\mid\-\\sim\\operatorname\{Gamma\}\\left\(a\_\{w\},\\;b\_\{w\}\+u\_\{i\}s\_\{ik^\{\-\}\}\\right\)\.The expectation of this posterior distribution is

𝔼​\[wi​k∣−\]=awbw\+ui​si​k−\.\\mathbb\{E\}\[w\_\{ik\}\\mid\-\]=\\frac\{a\_\{w\}\}\{b\_\{w\}\+u\_\{i\}s\_\{ik^\{\-\}\}\}\.By inspecting this posterior expectation, we can observe that: when a negative sample has high similarity to the query,si​k−s\_\{ik^\{\-\}\}increases the posterior rate and decreases the expected weight\. Thus, the Gamma prior enables continuous false\-negative\-aware reweighting\.

#### Bernoulli Prior\.

As an alternative, we use a Bernoulli priorwi​k∼Bernoulli⁡\(pw\)w\_\{ik\}\\sim\\operatorname\{Bernoulli\}\(p\_\{w\}\)withwi​k∈\{0,1\}w\_\{ik\}\\in\\\{0,1\\\}for binary negative gating, wherewi​k=1w\_\{ik\}=1keeps the negative andwi​k=0w\_\{ik\}=0removes it\. The unnormalized posterior probabilities are

p​\(wi​k=1∣−\)\\displaystyle p\(w\_\{ik\}=1\\mid\-\)∝pw​e−ui​si​k−,\\displaystyle\\propto p\_\{w\}e^\{\-u\_\{i\}s\_\{ik^\{\-\}\}\},p​\(wi​k=0∣−\)\\displaystyle p\(w\_\{ik\}=0\\mid\-\)∝1−pw\.\\displaystyle\\propto 1\-p\_\{w\}\.After normalization,

wi​k∣−∼Bernoulli\(pw​e−ui​si​k−1−pw\+pw​e−ui​si​k−\)\.w\_\{ik\}\\mid\-\\sim\\operatorname\{Bernoulli\}\\left\(\\frac\{p\_\{w\}e^\{\-u\_\{i\}s\_\{ik^\{\-\}\}\}\}\{1\-p\_\{w\}\+p\_\{w\}e^\{\-u\_\{i\}s\_\{ik^\{\-\}\}\}\}\\right\)\.The posterior expectation is

𝔼​\[wi​k∣−\]=pw​e−ui​si​k−1−pw\+pw​e−ui​si​k−\.\\mathbb\{E\}\[w\_\{ik\}\\mid\-\]=\\frac\{p\_\{w\}e^\{\-u\_\{i\}s\_\{ik^\{\-\}\}\}\}\{1\-p\_\{w\}\+p\_\{w\}e^\{\-u\_\{i\}s\_\{ik^\{\-\}\}\}\}\.By inspecting this expectation, we can see that the Bernoulli posterior performs more aggressive false\-negative filtering: highly similar negatives have smaller retaining probabilities becausee−ui​si​k−e^\{\-u\_\{i\}s\_\{ik^\{\-\}\}\}decreases assi​k−s\_\{ik^\{\-\}\}grows\. Compared with the Gamma prior, the Bernoulli prior provides hard keep\-or\-drop gating for likely false negatives\.

### 3\.3Optimization

We optimize the proposedBDRframework with stochastic approximation EM \(SAEM\)Delyonet al\.\([1999](https://arxiv.org/html/2608.02907#bib.bib1)\), shown in Algorithm[1](https://arxiv.org/html/2608.02907#alg1)\. At each iteration, BDR samples a mini\-batch, computes positive and negative similarities with the current retriever, and draws the auxiliary variableuiu\_\{i\}and latent weightswi​kw\_\{ik\}from their closed\-form conditional posteriors\. The sampled weights are then used to form a weighted contrastive loss, which updates the stochastic surrogate objective\. Finally, the retriever parameters are optimized by stochastic gradient descent on this surrogate\. By alternating posterior inference and parameter updates, BDR adaptively reweights noisy negatives during training\.

Algorithm 1BDROptimization via SAEMInput:training set

𝒳\\mathcal\{X\}, parameters

𝜽0\\boldsymbol\{\\theta\}\_\{0\}, step sizes

\{ρt\}\\\{\\rho\_\{t\}\\\}, learning rates

\{ηt\}\\\{\\eta\_\{t\}\\\}
Initialize

Q0​\(𝜽\)←0Q\_\{0\}\(\\boldsymbol\{\\theta\}\)\\leftarrow 0
for

t=0,1,…t=0,1,\\ldotsdo

Sample mini\-batch

ℬt=\{\(𝐯i,𝐝i\)\}i=1B\\mathcal\{B\}\_\{t\}=\\\{\(\\mathbf\{v\}\_\{i\},\\mathbf\{d\}\_\{i\}\)\\\}\_\{i=1\}^\{B\}
Compute

si\+s\_\{i^\{\+\}\}and

\{si​k−\}k=1K\\\{s\_\{ik^\{\-\}\}\\\}\_\{k=1\}^\{K\}using

𝜽t\\boldsymbol\{\\theta\}\_\{t\}
E\-step:

foreach

i∈ℬti\\in\\mathcal\{B\}\_\{t\}do

Sample

ui∼p​\(ui∣𝐰i,𝜽t,𝒳\)u\_\{i\}\\sim p\(u\_\{i\}\\mid\\mathbf\{w\}\_\{i\},\\boldsymbol\{\\theta\}\_\{t\},\\mathcal\{X\}\)
Sample

wi​k∼p​\(wi​k∣ui,𝜽t,𝒳\)w\_\{ik\}\\sim p\(w\_\{ik\}\\mid u\_\{i\},\\boldsymbol\{\\theta\}\_\{t\},\\mathcal\{X\}\), for

n=1,…,Nn=1,\\ldots,N
endfor

Q^t\+1​\(𝜽\)←ℒBDR​\(ℬt;𝜽\)\\widehat\{Q\}\_\{t\+1\}\(\\boldsymbol\{\\theta\}\)\\leftarrow\\mathcal\{L\}\_\{\\mathrm\{BDR\}\}\(\\mathcal\{B\}\_\{t\};\\boldsymbol\{\\theta\}\)

M\-step:

𝜽t\+1←𝜽t−ηt​∇𝜽Qt\+1​\(𝜽\)\|𝜽=𝜽t\\boldsymbol\{\\theta\}\_\{t\+1\}\\leftarrow\\boldsymbol\{\\theta\}\_\{t\}\-\\eta\_\{t\}\\nabla\_\{\\boldsymbol\{\\theta\}\}Q\_\{t\+1\}\(\\boldsymbol\{\\theta\}\)\|\_\{\\boldsymbol\{\\theta\}=\\boldsymbol\{\\theta\}\_\{t\}\}

endfor

Table 1:Retrieval performance comparison on individual dataset\.BDR consistently outperforms uniform sampling and existing reweighting baselines across both Recall@K and Pseudo Recall@K, with the Gamma prior achieving the best overall performance\.MethodOKVQAEVQAR@1R@5PR@1PR@5R@1R@5PR@1PR@5Uniform Negative Sampling23\.045\.839\.861\.728\.057\.035\.563\.6Random Negative Reweighting22\.846\.837\.459\.527\.255\.234\.162\.4Similarity\-based Reweighting24\.647\.639\.060\.127\.658\.134\.664\.8Margin\-based Reweighting Loss25\.449\.340\.761\.428\.859\.835\.565\.7Decoupled Positive Reweighting25\.649\.140\.560\.929\.960\.336\.967\.0Debiased Negative Reweighting25\.950\.542\.062\.630\.660\.937\.467\.4Hardness Negative Reweighting26\.549\.842\.662\.831\.761\.137\.666\.9Bayesian Reweighting \(Bernoulli Prior\)28\.9±0\.7\\textbf\{28\.9\}\{\\scriptstyle\\pm 0\.7\}53\.8±0\.7\\textbf\{53\.8\}\{\\scriptstyle\\pm 0\.7\}44\.4±1\.0\\textbf\{44\.4\}\{\\scriptstyle\\pm 1\.0\}63\.5±0\.4\\textbf\{63\.5\}\{\\scriptstyle\\pm 0\.4\}33\.5±1\.3\\textbf\{33\.5\}\{\\scriptstyle\\pm 1\.3\}63\.9±1\.3\\textbf\{63\.9\}\{\\scriptstyle\\pm 1\.3\}40\.6±1\.3\\textbf\{40\.6\}\{\\scriptstyle\\pm 1\.3\}69\.7±1\.0\\textbf\{69\.7\}\{\\scriptstyle\\pm 1\.0\}Bayesian Reweighting \(Gamma Prior\)30\.1±0\.4\\textbf\{30\.1\}\{\\scriptstyle\\pm 0\.4\}54\.8±0\.8\\textbf\{54\.8\}\{\\scriptstyle\\pm 0\.8\}45\.9±0\.9\\textbf\{45\.9\}\{\\scriptstyle\\pm 0\.9\}66\.0±0\.6\\textbf\{66\.0\}\{\\scriptstyle\\pm 0\.6\}35\.1±0\.9\\textbf\{35\.1\}\{\\scriptstyle\\pm 0\.9\}65\.6±1\.6\\textbf\{65\.6\}\{\\scriptstyle\\pm 1\.6\}42\.2±0\.9\\textbf\{42\.2\}\{\\scriptstyle\\pm 0\.9\}71\.5±1\.4\\textbf\{71\.5\}\{\\scriptstyle\\pm 1\.4\}Table 2:Retrieval performance on M2KR benchmarks\.BDR consistently improves retrieval performance across seven knowledge\-intensive VQA datasets, with the Gamma\-prior variant achieving the best average Recall\.MethodEVQAOKVQAKVQAInfoSeekWITOVENLLaVAAvg\.R@1R@5R@1R@5R@1R@5R@1R@5R@1R@5R@1R@5R@1R@5Uniform Negative Sampling27\.755\.824\.947\.026\.152\.021\.143\.323\.043\.326\.252\.677\.292\.743\.8Debiased Negative Reweighting26\.956\.723\.145\.427\.153\.627\.847\.735\.658\.538\.764\.265\.387\.247\.0Hardness Negative Reweighting30\.259\.219\.639\.125\.751\.936\.459\.633\.956\.542\.066\.858\.684\.647\.4Bayesian Reweighting \(Bernoulli Prior\)34\.065\.123\.544\.031\.959\.140\.266\.141\.765\.147\.471\.576\.093\.554\.2Bayesian Reweighting \(Gamma Prior\)37\.868\.826\.050\.035\.961\.639\.362\.842\.967\.048\.372\.776\.693\.155\.9

Table 3:Retrieval performance with different LLM backbones on M2KR\.For OVEN and KVQA, we only report R@5, and for LLaVA, we only report R@1, to ensure comparability with previous baselines: PreFLMRLinet al\.\([2024](https://arxiv.org/html/2608.02907#bib.bib21)\)and ReTCaffagniet al\.\([2025](https://arxiv.org/html/2608.02907#bib.bib25)\)\. BDR consistently improves Qwen2\-VL\-2B, Phi\-3\.5\-V\-3\.8B, and Qwen2\-VL\-7B across knowledge\-intensive VQA benchmarks\.Table 4:Answer generation performance comparison on InfoSeek and EVQA\.We report VQA Accuracy, Exact Match \(EM\), BLEU\-1, andBERT Matching \(BEM\)\. The Oracle Retriever retrieves all ground\-truth documents\. Our BDR retriever consistently outperforms PreFLMRLinet al\.\([2024](https://arxiv.org/html/2608.02907#bib.bib21)\)and ReTCaffagniet al\.\([2025](https://arxiv.org/html/2608.02907#bib.bib25)\), approaching the Oracle upper bound\.InfoSeekEVQAGenerator \(Frozen\)RetrieverR@5VQA\_AccEMBLEU\_1R@5VQA\_AccBLEU\_1BEMLLaVA\-1\.6\-13B✗\-5\.45\.311\.9\-2\.78\.969\.8LLaVA\-1\.6\-13BPreFLMR39\.212\.912\.421\.262\.08\.726\.274\.3LLaVA\-1\.6\-13BReT52\.017\.317\.228\.948\.66\.519\.273\.2LLaVA\-1\.6\-13BQwen2\-VL\-7B \(Ours\)66\.720\.820\.934\.064\.39\.126\.977\.2LLaVA\-1\.6\-13BOracle Retriever\-37\.539\.556\.4\-16\.146\.186\.7Qwen2\.5\-VL\-7B✗\-14\.414\.525\.2\-4\.614\.365\.2Qwen2\.5\-VL\-7BPreFLMR39\.221\.516\.124\.162\.011\.534\.768\.3Qwen2\.5\-VL\-7BReT52\.025\.921\.532\.248\.610\.828\.167\.9Qwen2\.5\-VL\-7BQwen2\-VL\-7B \(Ours\)66\.732\.127\.541\.364\.314\.437\.171\.2Qwen2\.5\-VL\-7BOracle Retriever\-46\.241\.361\.9\-23\.357\.889\.1

## 4Experiments

In the experimental section, we investigate five key research questions:RQ1:How does Bayesian Data Reweighting compare with existing reweighting methods?RQ2:How does the reweighting improve downstream VQA performance?RQ3:How does Bayesian Data Reweighting adjust sample importance?RQ4:How should the Bayesian prior hyperparameters be determined?RQ5:Does it introduce additional computational overhead?

### 4\.1Comparison with Existing Baselines

We consider the following reweighting\-based methods as baselines\.Random Negative Reweightingassigns stochastic weights on negative samples randomly;Similarity\-based ReweightingWuet al\.\([2017](https://arxiv.org/html/2608.02907#bib.bib94)\)emphasizes high\-similarity negatives usingwi​k−∝exp⁡\(γ​si​k−\)w\_\{ik\}^\{\-\}\\propto\\exp\(\\gamma s\_\{ik\}^\{\-\}\);Margin\-based ReweightingSchroffet al\.\([2015](https://arxiv.org/html/2608.02907#bib.bib55)\)focuses on a certain margin gap by enforcingℒmargin,i=\[α\+si​k−−si\+\]\+\\mathcal\{L\}\_\{\\mathrm\{margin\},i\}=\[\\alpha\+s\_\{ik\}^\{\-\}\-s\_\{i\}^\{\+\}\]\_\{\+\};Decoupled Positive ReweightingYehet al\.\([2022](https://arxiv.org/html/2608.02907#bib.bib93)\)removes the positive pair from the denominator and upweights hard positives,ℒDCLW,i=−wi​si\+/τ\+log​∑kexp⁡\(si​k−/τ\)\\mathcal\{L\}\_\{\\mathrm\{DCLW\},i\}=\-w\_\{i\}s\_\{i\}^\{\+\}/\\tau\+\\log\\sum\_\{k\}\\exp\(s\_\{ik\}^\{\-\}/\\tau\);Debiased Negative ReweightingChuanget al\.\([2020](https://arxiv.org/html/2608.02907#bib.bib28)\)corrects false\-negative bias by estimating the true\-negative contribution; andHardness Negative ReweightingRobinsonet al\.\([2020](https://arxiv.org/html/2608.02907#bib.bib59)\)assigns hardness\-aware weightsqi​k−∝exp⁡\(β​si​k−\)q\_\{ik\}^\{\-\}\\propto\\exp\(\\beta s\_\{ik\}^\{\-\}\)and combines them with the false\-negative debiasing\.

#### Experimental Setup\.

To compare with existing reweighting methods, we conduct experiments on OKVQAMarinoet al\.\([2019](https://arxiv.org/html/2608.02907#bib.bib62)\)and EVQAZellerset al\.\([2019](https://arxiv.org/html/2608.02907#bib.bib68)\)\. For each dataset, we sample 10K training examples to fine\-tune a multimodal LLM\-based embedding model and evaluate retrieval performance on the test set\. Note the original evaluation corpus contains millions of documents, so we filter the test corpus to retain only ground\-truth\-contained documents for faster retrieval evaluation\. We report Recall@K \(R@K\), which measures whether the target document appears in the top\-KKretrieved results, and Pseudo Recall@K \(PR@K\), which measures whether any top\-KKdocument contains the correct answer, following prior workLin and Byrne \([2022](https://arxiv.org/html/2608.02907#bib.bib19)\); Linet al\.\([2024](https://arxiv.org/html/2608.02907#bib.bib21)\)\.

We use Qwen2\-VL\-2BWanget al\.\([2024](https://arxiv.org/html/2608.02907#bib.bib47)\)as the backbone and apply LoRA\-based fine\-tuning with different reweighting strategies\. Following VLM2VecJianget al\.\([2025](https://arxiv.org/html/2608.02907#bib.bib23)\), the model uses the last token as the input embedding for contrastive learning\. The main hyperparameters are: LoRA rankr=4r=4, LoRA alpha=64=64, batch size=64=64, learning rate=1×10−4=1\\times 10^\{\-4\}, dropout=0\.1=0\.1, and temperatureτ=0\.02\\tau=0\.02\. All models are trained on 4 NVIDIA A100 GPUs under the same optimization setup\. Results are reported in Table[1](https://arxiv.org/html/2608.02907#S3.T1)\.

#### Reweighting Results Analysis\.

As shown in Table[1](https://arxiv.org/html/2608.02907#S3.T1), Bayesian reweighting achieves the best performance on both OKVQA and EVQA across all metrics\. Among the baselines, hardness negative reweighting and debiased negative reweighting perform competitively, suggesting that false\-negative mitigation is important for KB\-VQA retrieval\. In contrast, heuristic or deterministic methods, such as random, similarity\-based, margin\-based, provide limited gains because they rely on fixed rules or sensitive hyperparameters, therefore limiting their generalization ability\. BDR instead models sample importance as latent variables and adaptively infers posterior weights, yielding more consistent improvements\. The Gamma\-prior variant further outperforms the Bernoulli\-prior variant, indicating that continuous uncertainty\-aware reweighting is more effective than binary selection for diverse negative samples\.

#### Broader M2KR Benchmarks\.

We further evaluate BDR on seven M2KR datasetsLinet al\.\([2024](https://arxiv.org/html/2608.02907#bib.bib21)\)using Qwen2\-VL\-2B as the backbone, with 1K training samples per task trained jointly\. As shown in Table[2](https://arxiv.org/html/2608.02907#S3.T2), BDR achieves the best average retrieval performance, with the Gamma\-prior variant performing best overall and the Bernoulli\-prior variant achieving strong results on InfoSeekChenet al\.\([2023](https://arxiv.org/html/2608.02907#bib.bib16)\)\. These results demonstrate that BDR generalizes well across diverse knowledge\-intensive VQA benchmarks and consistently improves over existing reweighting baselines\.

#### Broader Retriever Backbones\.

To evaluate the general applicability of BDR, we apply the Gamma\-prior variant to three LLM backbones, including Qwen2\-VL\-2B, Phi\-3\.5\-V\-3\.8B, and Qwen2\-VL\-7B, using the complete M2KR training corpus and evaluating on the full test set\. As shown in Table[3](https://arxiv.org/html/2608.02907#S3.T3), BDR achieves strong performance across different model scales and surpasses previous state\-of\-the\-art retrievers on this benchmark, including Pre\-FLMRLinet al\.\([2024](https://arxiv.org/html/2608.02907#bib.bib21)\)and ReTCaffagniet al\.\([2025](https://arxiv.org/html/2608.02907#bib.bib25)\)\. Notably, Qwen2\-VL\-7B w/ BDR obtains the best overall average performance, demonstrating that BDR is broadly effective across diverse backbone architectures for knowledge\-intensive VQA retrieval\.

### 4\.2Impact on Downstream VQA Task

Answer generation results\.To evaluate downstream VQA performance, we integrate the trained retriever with two frozen multimodal LLMs, LLaVA\-1\.6\-13BLiuet al\.\([2023](https://arxiv.org/html/2608.02907#bib.bib48)\)and Qwen2\.5\-VL\-7BTeam \([2025](https://arxiv.org/html/2608.02907#bib.bib46)\), to build a complete retrieval\-augmented VQA pipeline\. We evaluate the generated answers on InfoSeek and EVQA using VQA ScoreLin and Byrne \([2022](https://arxiv.org/html/2608.02907#bib.bib19)\)and related generation metrics\. As shown in Table[4](https://arxiv.org/html/2608.02907#S3.T4), the BDR\-based retriever consistently achieves the best answer generation performance across both datasets and generators\. Compared with existing strong retrievers, BDR improves VQA accuracy, exact match, BLEU\-1, and BEM in most cases, indicating that better retrieval quality effectively translates into stronger downstream reasoning\. Moreover, its performance is closer to the Oracle Retriever, suggesting that BDR retrieves more relevant evidence and better narrows the gap between practical retrieval and oracle\-level generation\.

![Refer to caption](https://arxiv.org/html/2608.02907v1/x2.png)Figure 2:Distribution of query–passage cosine similarities on EVQA\.The dashed lines denote the mean of similarities, and the annotated gap measures the positive–negative separation\.![Refer to caption](https://arxiv.org/html/2608.02907v1/x3.png)Figure 3:Recall@K comparison on EVQA\.We evaluate retrieval performance by varyingKKand reporting whether the ground\-truth document appears in the top\-KKresults\.
### 4\.3How does Bayesian Reweighting work?

Decision Boundary Visualization\.We analyze the cosine similarity distributions of positive and negative query–passage pairs to examine the effect of Bayesian Reweighting on the decision boundary\. As shown in Figure[2](https://arxiv.org/html/2608.02907#S4.F2), our method achieves the largest positive–negative similarity gap, improving it from0\.3030\.303with InfoNCE to0\.3480\.348\. This gap is also larger than those of debiased negative reweighting and hardness negative reweighting, indicating that Bayesian Reweighting learns more discriminative embeddings by better separating relevant and irrelevant passages\.

Recall@K analysis\.We analyze how the performance of retrievers trained with different methods changes by varying the number of top\-KKretrieved documents\. The results are shown in Figure[3](https://arxiv.org/html/2608.02907#S4.F3)\. Compared with retrievers trained using other reweighting methods, the two retrievers trained with BDR achieve consistently higher Recall@KKacross different top\-KKsettings, clearly outperforming existing baselines\. The gains are especially pronounced at smallerKK, indicating that BDR improves the ranking of relevant documents near the top of the retrieved list\. This experiment demonstrates that BDR yields more robust retrieval quality across different retrieval budgets\.

Weight Calibration Analysis\.To evaluate whether the learned Bayesian weights align with sample quality, we conduct a calibration analysis on EVQA\. We first use an external LLM embedding model to measure passage\-level semantic similarity, followed by manual verification to identify false negatives \(FNs\)\. Among 1,000 sampled query–passage pairs, 54 are identified as FNs\. We then record the weights inferred by BDR, divide negative samples into ten equal\-frequency weight deciles, and compute the FN proportion in each bin\. As shown in the right panel of Figure[4](https://arxiv.org/html/2608.02907#S4.F4), the FN proportion decreases as the weight decile increases, indicating that lower\-weight bins contain more FNs while higher\-weight bins are dominated by true negatives \(TNs\)\. We also visualize the weight distributions of TNs and FNs, where FN samples receive lower weights on average than TN samples, suggesting that BDR effectively suppresses potentially mislabeled negatives\. These results demonstrate that the Bayesian weighting mechanism is well calibrated with respect to sample quality\.

![Refer to caption](https://arxiv.org/html/2608.02907v1/x4.png)Figure 4:Weight calibration analysis\.Left: FN samples receive lower Bayesian weights than TN samples on average\. Right: the FN fraction generally decreases across increasing weight deciles, suggesting that lower\-weight samples are more likely to be false negatives\.
### 4\.4Sensitivity to Bayesian Priors

We study the sensitivity of BDR to different prior hyperparameters on OKVQA\. For the Gamma prior, we fixaw=au=1a\_\{w\}=a\_\{u\}=1and varybwb\_\{w\}andbub\_\{u\}to examine the effect of prior strength\. As shown in Tables[5](https://arxiv.org/html/2608.02907#S4.T5), BDR remains stable across most settings, while an overly largebub\_\{u\}degrades performance, indicating that too strong a prior may limit adaptive weight inference\. For the Bernoulli prior, we vary the retaining probabilityppand observe only mild performance changes, Table[6](https://arxiv.org/html/2608.02907#S4.T6)suggesting that binary gating is also reasonably robust\. We further test the number of posterior sampling iterationsNNunder the Gamma\-prior setting\. Performance remains stable fromN=1N=1to88, showing that BDR can achieve reliable posterior estimates with only a few sampling steps\.

Table 5:Sensitivity analysis ofbwb\_\{w\}andbub\_\{u\}on OKVQA\.Table 6:Sensitivity analysis ofppandNNon OKVQA\.
### 4\.5Training Efficiency Comparison

Bayesian Reweighting is computationally efficient because its EM updates only require closed\-form, element\-wise sampling with a lightweight cost ofO​\(M​B​K\)O\(MBK\)\. Since the dominant cost comes from the multimodal encoder forward and backward updates, the additional reweighting overhead is negligible in practice\. As shown in Table[7](https://arxiv.org/html/2608.02907#S4.T7), BDR achieves nearly identical GPU memory usage and training time compared with standard InfoNCE across different backbones and batch sizes\.

Table 7:Model training efficiency comparison\.

## 5Conclusion

We propose Bayesian Data Reweighting \(BDR\), a principled and efficient framework for uncertainty\-aware contrastive learning in knowledge\-intensive VQA retrieval\. BDR adaptively reweights training pairs, consistently improving retrieval performance across M2KR datasets and multiple LLM backbones, which further benefits downstream answer generation\. Analysis shows that BDR learns more discriminative embeddings, suppresses harmful false negatives, and adds negligible computational overhead\. Future work will extend BDR beyond retrieval\-based VQA to broader multimodal RAG and reasoning tasks\.

## Limitations

This work focuses on knowledge\-based VQA retrieval, although BDR is evaluated across multiple KB\-VQA benchmarks and retriever backbones, it has several limitations\. First, the evaluation is limited to the datasets, models, and metrics considered in this paper, and BDR is not directly designed for multi\-hop retrieval\. Second, from a strict Bayesian perspective, the posterior is formulated using a pseudo\-likelihood for tractable derivation\. Therefore, it is not a fully normalized posterior distribution, but serve as uncertainty\-aware importance weights\. Future work will validate BDR under full open\-corpus retrieval and explore joint retriever\-generator optimization for end\-to\-end multimodal RAG systems\.

## Ethical Considerations

This work aims to improve multimodal retrieval for knowledge\-based VQA and does not introduce new datasets or collect personal information\. However, retrieval\-augmented systems may still retrieve biased, outdated, or misleading evidence, which can lead to incorrect downstream answers\. Therefore, BDR should not be viewed as a guarantee of factuality or safety, especially in high\-stakes applications where human verification remains necessary\.

## References

- Wiki\-llava: hierarchical retrieval\-augmented generation for multimodal llms\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 1818–1826\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p1.1),[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Caffagni, S\. Sarto, M\. Cornia, L\. Baraldi, and R\. Cucchiara \(2025\)Recurrence\-enhanced vision\-and\-language transformers for robust multimodal document retrieval\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p2.1),[Table 3](https://arxiv.org/html/2608.02907#S3.T3),[Table 3](https://arxiv.org/html/2608.02907#S3.T3.3.10.7.1),[Table 4](https://arxiv.org/html/2608.02907#S3.T4),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.SSS0.Px4.p1.1)\.
- T\. Chen, W\. Hung, H\. Tseng, S\. Chien, and M\. Yang \(2021\)Incremental false negative detection for contrastive learning\.arXiv preprint arXiv:2106\.03719\.Cited by:[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Chen, H\. Hu, Y\. Luan, H\. Sun, S\. Changpinyo, A\. Ritter, and M\. Chang \(2023\)Can pre\-trained vision and language models answer visual information\-seeking questions?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.SSS0.Px3.p1.1)\.
- C\. Chuang, J\. Robinson, Y\. Lin, A\. Torralba, and S\. Jegelka \(2020\)Debiased contrastive learning\.Advances in Neural Information Processing Systems33,pp\. 8765–8775\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p4.1),[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.p1.4)\.
- B\. Delyon, M\. Lavielle, and E\. Moulines \(1999\)Convergence of a stochastic approximation version of the em algorithm\.Annals of statistics,pp\. 94–128\.Cited by:[§3\.3](https://arxiv.org/html/2608.02907#S3.SS3.p1.2)\.
- D\. Hao, Q\. Wang, L\. Guo, J\. Jiang, and J\. Liu \(2024\)Self\-bootstrapped visual\-language model for knowledge selection and question answering\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Hu, A\. Iscen, C\. Sun, Z\. Wang, K\. Chang, Y\. Sun, C\. Schmid, D\. A\. Ross, and A\. Fathi \(2023\)Reveal: retrieval\-augmented visual\-language pre\-training with multi\-source multimodal knowledge memory\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 23369–23379\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p2.1)\.
- T\. Huynh, S\. Kornblith, M\. R\. Walter, M\. Maire, and M\. Khademi \(2022\)Boosting contrastive self\-supervised learning with false negative cancellation\.InProceedings of the IEEE/CVF winter conference on applications of computer vision,pp\. 2785–2795\.Cited by:[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px2.p1.1)\.
- P\. Jian, D\. Yu, and J\. Zhang \(2024\)Large language models know what is key visual entity: an llm\-assisted multimodal retrieval for vqa\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p1.1),[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Jiang, R\. Meng, X\. Yang, S\. Yavuz, Y\. Zhou, and W\. Chen \(2025\)VLM2Vec: training vision\-language models for massive multimodal embedding tasks\.InProceedings of the Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p2.1),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.SSS0.Px1.p2.6)\.
- Y\. Kalantidis, M\. B\. Sariyildiz, N\. Pion, P\. Weinzaepfel, and D\. Larlus \(2020\)Hard negative mixing for contrastive learning\.Advances in neural information processing systems33,pp\. 21798–21809\.Cited by:[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Katharopoulos and F\. Fleuret \(2018\)Not all samples are created equal: deep learning with importance sampling\.InInternational Conference on Machine Learning,pp\. 2525–2534\.Cited by:[§3](https://arxiv.org/html/2608.02907#S3.SS0.SSS0.Px1.p1.2),[§3](https://arxiv.org/html/2608.02907#S3.p2.2)\.
- S\. Lin, C\. Lee, M\. Shoeybi, J\. Lin, B\. Catanzaro, and W\. Ping \(2025\)Mm\-embed: universal multimodal retrieval with multimodal llms\.International Conference on Learning Representations\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p2.1)\.
- W\. Lin and B\. Byrne \(2022\)Retrieval augmented visual question answering with outside knowledge\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p2.1),[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.SSS0.Px1.p1.2),[§4\.2](https://arxiv.org/html/2608.02907#S4.SS2.p1.1)\.
- W\. Lin, J\. Chen, J\. Mei, A\. Coca, and B\. Byrne \(2023\)Fine\-grained late\-interaction multi\-modal retrieval for retrieval augmented visual question answering\.Advances in Neural Information Processing Systems36,pp\. 22820–22840\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p2.1)\.
- W\. Lin, J\. Mei, J\. Chen, and B\. Byrne \(2024\)PreFLMR: scaling up fine\-grained late\-interaction multi\-modal retrievers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p2.1),[Table 3](https://arxiv.org/html/2608.02907#S3.T3),[Table 3](https://arxiv.org/html/2608.02907#S3.T3.3.9.6.1),[Table 4](https://arxiv.org/html/2608.02907#S3.T4),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.SSS0.Px1.p1.2),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.SSS0.Px4.p1.1)\.
- Y\. Lin, Y\. Xie, D\. Chen, Y\. Xu, C\. Zhu, and L\. Yuan \(2022\)Revive: regional visual representation matters in knowledge\-based visual question answering\.Advances in Neural Information Processing Systems35,pp\. 10560–10571\.Cited by:[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee \(2023\)Improved baselines with visual instruction tuning\.External Links:2310\.03744Cited by:[§4\.2](https://arxiv.org/html/2608.02907#S4.SS2.p1.1)\.
- X\. Long, Z\. Ma, E\. Hua, K\. Zhang, B\. Qi, and B\. Zhou \(2025\)Retrieval\-augmented visual question answering via built\-in autoregressive search engines\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 24723–24731\.Cited by:[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Marino, M\. Rastegari, A\. Farhadi, and A\. Schwing \(2019\)OK\-vqa: a visual question answering benchmark requiring external knowledge\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p1.1),[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.SSS0.Px1.p1.2)\.
- T\. Mensink, J\. Uijlings, L\. Castrejon, A\. Goel, F\. Cadar, H\. Zhou, F\. Sha, A\. Araujo, and V\. Ferrari \(2023\)Encyclopedic vqa: visual questions about detailed properties of fine\-grained categories\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 3113–3124\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p1.1)\.
- A\. v\. d\. Oord, Y\. Li, and O\. Vinyals \(2018\)Representation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p2.1),[§3](https://arxiv.org/html/2608.02907#S3.p1.9)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.\(2021\)Learning transferable visual models from natural language supervision\.InInternational Conference on Machine Learning,pp\. 8748–8763\.Cited by:[Table 3](https://arxiv.org/html/2608.02907#S3.T3.3.7.4.1),[Table 3](https://arxiv.org/html/2608.02907#S3.T3.3.8.5.1)\.
- M\. Ren, W\. Zeng, B\. Yang, and R\. Urtasun \(2018\)Learning to reweight examples for robust deep learning\.InInternational Conference on Machine Learning,pp\. 4334–4343\.Cited by:[§3](https://arxiv.org/html/2608.02907#S3.SS0.SSS0.Px1.p1.2)\.
- J\. Robinson, C\. Chuang, S\. Sra, and S\. Jegelka \(2020\)Contrastive learning with hard negative samples\.arXiv preprint arXiv:2010\.04592\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p4.1),[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.p1.4)\.
- F\. Schroff, D\. Kalenichenko, and J\. Philbin \(2015\)Facenet: a unified embedding for face recognition and clustering\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 815–823\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p4.1),[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.p1.4)\.
- D\. Schwenk, A\. Khandelwal, C\. Clark, K\. Marino, and R\. Mottaghi \(2022\)A\-okvqa: a benchmark for visual question answering using world knowledge\.InEuropean Conference on Computer Vision,pp\. 146–162\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p1.1),[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px1.p1.1)\.
- R\. Speer, J\. Chin, and C\. Havasi \(2017\)ConceptNet 5\.5: an open multilingual graph of general knowledge\.InProceedings of the AAAI Conference on Artificial Intelligence,Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p1.1)\.
- M\. A\. Tanner and W\. H\. Wong \(1987\)The calculation of posterior distributions by data augmentation\.Journal of the American statistical Association82\(398\),pp\. 528–540\.Cited by:[§3\.1](https://arxiv.org/html/2608.02907#S3.SS1.p1.4)\.
- Q\. Team \(2025\)Qwen2\.5\-vl\.External Links:[Link](https://qwenlm.github.io/blog/qwen2.5-vl/)Cited by:[§4\.2](https://arxiv.org/html/2608.02907#S4.SS2.p1.1)\.
- D\. Vrandečić and M\. Krötzsch \(2014\)Wikidata: a free collaborative knowledgebase\.Communications of the ACM57\(10\),pp\. 78–85\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p1.1)\.
- P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, Y\. Fan, K\. Dang, M\. Du, X\. Ren, R\. Men, D\. Liu, C\. Zhou, J\. Zhou, and J\. Lin \(2024\)Qwen2\-vl: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.SSS0.Px1.p2.6)\.
- Y\. Wang, A\. Kucukelbir, and D\. M\. Blei \(2017\)Robust probabilistic modeling with bayesian data reweighting\.InInternational conference on machine learning,pp\. 3646–3655\.Cited by:[§3](https://arxiv.org/html/2608.02907#S3.p2.2)\.
- C\. Wei, Y\. Chen, H\. Chen, H\. Hu, G\. Zhang, J\. Fu, A\. Ritter, and W\. Chen \(2024\)Uniir: training and benchmarking universal multimodal information retrievers\.InEuropean Conference on Computer Vision,pp\. 387–404\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p2.1),[Table 3](https://arxiv.org/html/2608.02907#S3.T3.3.6.3.1)\.
- C\. Wu, R\. Manmatha, A\. J\. Smola, and P\. Krahenbuhl \(2017\)Sampling matters in deep embedding learning\.InProceedings of the IEEE international conference on computer vision,pp\. 2840–2848\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p4.1),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.p1.4)\.
- L\. Xiong, C\. Xiong, Y\. Li, K\. Tang, J\. Liu, P\. Bennett, J\. Ahmed, and A\. Overwijk \(2021\)Approximate nearest neighbor negative contrastive learning for dense text retrieval\.International Conference on Learning Representations\.Cited by:[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Yan and W\. Xie \(2024\)EchoSight: advancing visual\-language models with wiki knowledge\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p1.1),[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Yeh, C\. Hong, Y\. Hsu, T\. Liu, Y\. Chen, and Y\. LeCun \(2022\)Decoupled contrastive learning\.InEuropean conference on computer vision,pp\. 668–684\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p4.1),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.p1.4)\.
- R\. Zellers, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)From recognition to cognition: visual commonsense reasoning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 6720–6731\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.02907#S4.SS1.SSS0.Px1.p1.2)\.
- W\. Zheng, Z\. Chen, J\. Lu, and J\. Zhou \(2019\)Hardness\-aware deep metric learning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 72–81\.Cited by:[§1](https://arxiv.org/html/2608.02907#S1.p4.1),[§2](https://arxiv.org/html/2608.02907#S2.SS0.SSS0.Px2.p1.1)\.