Evidence-Backed Video Question Answering

Hugging Face Daily Papers Papers
video-qa visual-grounding explainability spatio-temporal benchmark fine-tuning

Summary

This paper introduces Evidence-Backed Video Question Answering (E-VQA), a new task requiring models to output both semantic answers and precise spatio-temporal evidence like tracked object segmentation masklets. The authors create a human-verified benchmark and a scalable training dataset, showing significant improvements over baselines.

Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone fails to bridge. To address this, we develop scalable, automated generation pipelines to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data yields substantial gains over the corresponding size-matched UniPixel baselines (e.g., +27.2 t-mean and +13.8 J&F on a 7B model), establishing a robust baseline for explainable, evidence-backed video understanding. Code and data are available at https://github.com/SalesforceAIResearch/EVQA.
Original Article
View Cached Full Text

Cached at: 07/15/26, 12:18 AM

Paper page - Evidence-Backed Video Question Answering

Source: https://huggingface.co/papers/2607.11862

Abstract

CurrentVideoLargeLanguageModels(VideoLLMs)excelinquestionanswering(QA)butlargelyoperateasblackboxes,providingtextualanswerswithoutverifiablevisualgrounding.Existingexplainabilityeffortsrelyontextualrationalesorsparseboundingboxes,whichstruggletocapturecomplexvideodynamicssuchasocclusionsandnon-rigiddeformations.WeproposeEvidence-BackedVideoQuestionAnswering(E-VQA),anoveltaskrequiringmodelstojointlyoutputasemanticanswerandprecisespatio-temporalevidence:temporalsegmentsanddense,trackedobjectsegmentationmasklets.Tosupportthis,weintroduceST-Evidence,thefirsthuman-verifiedbenchmarkforbothdiscriminativeandgenerativepixel-levelgrounding.Evaluationsofstate-of-the-artmodelsrevealacriticaldecouplingbetweenQAaccuracyandtruevisualperceptionthatscalingalonefailstobridge.Toaddressthis,wedevelopscalable,automatedgenerationpipelinestocreateST-Evidence-Instruct,a160k-scaledatasetbridginghigh-levelreasoningwithfine-grainedgrounding.Fine-tuninggroundedVideoLLMsonthisdatayieldssubstantialgainsoverthecorrespondingsize-matchedUniPixelbaselines(e.g.,+27.2t-meanand+13.8J&Fona7Bmodel),establishingarobustbaselineforexplainable,evidence-backedvideounderstanding.Codeanddataareavailableathttps://github.com/SalesforceAIResearch/EVQA.

View arXiv pageView PDFAdd to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.11862 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.11862 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.11862 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory

Hugging Face Daily Papers

SuperMemory-VQA is a new egocentric VQA benchmark featuring 52.9 hours of AI-glasses footage and 4,853 QA pairs designed to evaluate AI assistants on long-horizon memory tasks spanning object recall, intent, timelines, and conversations. Benchmarking reveals existing agentic frameworks and LLMs remain far from reliable on these real-world memory challenges.

Brain-IT-VQA: From Brain Signals to Answers

Hugging Face Daily Papers

Brain-IT-VQA framework decodes visual content from fMRI signals using transformer architecture, outperforming previous methods. The authors also introduce NSD-VQA, a new dataset with richer annotations for evaluating fMRI-based visual question answering.

EVE-Agent: Evidence-Verifiable Self-Evolving Agents

arXiv cs.AI

EVE-Agent introduces a framework for self-evolving search agents that ensure evidence verifiability by generating questions, answers, and evidence spans, and training on marginal accuracy gain of evidence. This improves grounded correctness without human annotations.