Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering
Summary
KBMR is an MLLM-based retriever for knowledge-based visual question answering that uses semantic alignment to improve retrieval and VQA accuracy, showing significant gains over CLIP baselines.
View Cached Full Text
Cached at: 09/03/26, 11:52 AM
Paper page - Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering
Source: https://huggingface.co/papers/2608.21450
Abstract
KBMR uses a multimodal language model to embed images by semantic identity rather than surface appearance, improving retrieval and visual question answering via continuous distillation and hard negative sampling.
Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employCLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the firstMLLM-based embedding retrievertailored forKB-VQA. Leveraging the robustautoregressive capabilitiesof MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-basedsemantic discriminatorthat generatescontinuous entity-consistency weights. These weights guide a novelcontinuous semantic distillationobjective, enabling effectivehard negative samplingand soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2608\.21450
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.21450 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.21450 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.21450 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Ground Then Rank: Revisiting Knowledge-Based VQA with Training-Free Entity Identification
This paper proposes a training-free 'identify-before-answer' (IBA) framework for Knowledge-Based Visual Question Answering (KB-VQA) that decouples entity identification from evidence ranking, outperforming fine-tuned multi-modal retrieval-augmented generation baselines while reducing complexity.
ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
Proposes ProMSA, a progressive multimodal search agent for knowledge-based visual question answering that adaptively selects search strategies and optimizes through sequence-level reinforcement learning, achieving consistent gains on E-VQA and InfoSeek.
Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering
This paper introduces Bayesian Data Reweighting, a probabilistic framework that adaptively weights query-document pairs to mitigate false negatives in contrastive training, improving multimodal retrieval across multiple knowledge-based VQA benchmarks.
Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models
This paper introduces Act2Answer, a protocol to evaluate knowledge retention in Vision-Language-Action (VLA) models by requiring agents to answer questions through physical actions. It finds that VLAs retain basic knowledge but show gaps on richer semantic categories, and that VQA co-training helps.
KARMA: Knowledge graph-based Automated Reasoning Materialization and Alignment
KARMA proposes a knowledge graph-based approach to generate slot-aligned contrastive candidates and uses Slot-Parallel Alignment (SPA) to apply preference optimization at the entity-slot level, addressing the Resolution Mismatch Problem in LLM reasoning supervision.