Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering
Summary
KBMR is an MLLM-based retriever for knowledge-based visual question answering that uses semantic alignment to improve retrieval and VQA accuracy, showing significant gains over CLIP baselines.
View Cached Full Text
Cached at: 09/03/26, 11:52 AM
Paper page - Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering
Source: https://huggingface.co/papers/2608.21450
Abstract
KBMR uses a multimodal language model to embed images by semantic identity rather than surface appearance, improving retrieval and visual question answering via continuous distillation and hard negative sampling.
Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employCLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the firstMLLM-based embedding retrievertailored forKB-VQA. Leveraging the robustautoregressive capabilitiesof MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-basedsemantic discriminatorthat generatescontinuous entity-consistency weights. These weights guide a novelcontinuous semantic distillationobjective, enabling effectivehard negative samplingand soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.21450
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.21450 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.21450 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.21450 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Ground Then Rank: Revisiting Knowledge-Based VQA with Training-Free Entity Identification
This paper proposes a training-free 'identify-before-answer' (IBA) framework for Knowledge-Based Visual Question Answering (KB-VQA) that decouples entity identification from evidence ranking, outperforming fine-tuned multi-modal retrieval-augmented generation baselines while reducing complexity.
VisKG-LM: Compiling Knowledge Graphs into Visual Memory for Multiple-Choice Question Answering
This paper proposes VisKG-LM, a method that compiles knowledge graphs into visual memory for efficient multiple-choice question answering, achieving performance gains over baselines by decoupling graph encoding from language reasoning.
ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
Proposes ProMSA, a progressive multimodal search agent for knowledge-based visual question answering that adaptively selects search strategies and optimizes through sequence-level reinforcement learning, achieving consistent gains on E-VQA and InfoSeek.
Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering
This paper introduces Bayesian Data Reweighting, a probabilistic framework that adaptively weights query-document pairs to mitigate false negatives in contrastive training, improving multimodal retrieval across multiple knowledge-based VQA benchmarks.
VISPATH: Visual-Intent-Guided Path Reasoning for Multimodal Knowledge Graph Question Answering
VisPath introduces a visual-intent-guided path reasoning framework for multimodal knowledge graph question answering, achieving significant improvements over baselines on a new benchmark, VisPath-Bench, and existing datasets.