SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Models

Hugging Face Daily Papers Papers
benchmark multimodal large-language-models text-vision-conflict geolocation evaluation

Summary

Introduces SIGNPOST-Bench, a benchmark for evaluating how multimodal large language models resolve conflicts between text and visual cues, using counterfactual image variants and geolocation as a diagnostic.

Multimodal large language models (MLLMs) make grounded predictions in real-world scenes by combining visual and textual cues, yet existing benchmarks rarely reveal how they arbitrate between these evidence sources when they conflict. We introduce SIGNPOST-Bench, a controlled counterfactual benchmark for evaluating text-vision conflict resolution. Each source image is transformed into a counterfactual quintuplet of Original, Blank, Similar, Random, and Adversarial variants. Synthetic, localized scene-text interventions are designed to preserve non-textual content, enabling paired measurements of changes in localization performance and directed shifts toward geographic targets introduced by conflicting text. SIGNPOST-Bench contains 5,111 counterfactual groups and 25,555 image variants from four datasets. We evaluate 20 MLLMs from seven providers. Compared with Original images, Adversarial variants raise median localization error from 282 km to 1,347 km, a 4.8-fold increase. Among geocodable adversarial samples, 6.5-20.1% of predictions lie less than 50 km from the injected target across models, and every evaluated model exhibits a positive mean paired reduction in target distance from Blank to Adversarial. Compatible, unrelated, and conflicting text replacements produce distinct effects on model predictions, while clean-input localization performance does not fully predict robustness to conflicting text. These results establish visual geolocation as a continuous diagnostic of scene-text arbitration and provide a controlled framework for evaluating how MLLMs resolve conflicting multimodal evidence.
Original Article
View Cached Full Text

Cached at: 08/06/26, 05:52 PM

Paper page - SIGNPOST-Bench: Benchmarking Text-Vision Conflict Resolution in Multimodal Large Language Models

Source: https://huggingface.co/papers/2608.04244

Abstract

Multimodallargelanguagemodels(MLLMs)makegroundedpredictionsinreal-worldscenesbycombiningvisualandtextualcues,yetexistingbenchmarksrarelyrevealhowtheyarbitratebetweentheseevidencesourceswhentheyconflict.WeintroduceSIGNPOST-Bench,acontrolledcounterfactualbenchmarkforevaluatingtext-visionconflictresolution.EachsourceimageistransformedintoacounterfactualquintupletofOriginal,Blank,Similar,Random,andAdversarialvariants.Synthetic,localizedscene-textinterventionsaredesignedtopreservenon-textualcontent,enablingpairedmeasurementsofchangesinlocalizationperformanceanddirectedshiftstowardgeographictargetsintroducedbyconflictingtext.SIGNPOST-Benchcontains5,111counterfactualgroupsand25,555imagevariantsfromfourdatasets.Weevaluate20MLLMsfromsevenproviders.ComparedwithOriginalimages,Adversarialvariantsraisemedianlocalizationerrorfrom282kmto1,347km,a4.8-foldincrease.Amonggeocodableadversarialsamples,6.5-20.1%ofpredictionslielessthan50kmfromtheinjectedtargetacrossmodels,andeveryevaluatedmodelexhibitsapositivemeanpairedreductionintargetdistancefromBlanktoAdversarial.Compatible,unrelated,andconflictingtextreplacementsproducedistincteffectsonmodelpredictions,whileclean-inputlocalizationperformancedoesnotfullypredictrobustnesstoconflictingtext.Theseresultsestablishvisualgeolocationasacontinuousdiagnosticofscene-textarbitrationandprovideacontrolledframeworkforevaluatinghowMLLMsresolveconflictingmultimodalevidence.

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2608\.04244

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.04244 in a model README.md to link it from this page.

Datasets citing this paper1

#### inorganicwriter/SIGNPOST-Bench Updatedabout 18 hours ago • 56

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.04244 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

PlanBench-V: A Spatial Planning Map Benchmark for Vision-Language Models

arXiv cs.CL

This paper introduces PlanBench-V, the first comprehensive benchmark for evaluating Vision-Language Models on spatial planning map interpretation, including an expert-annotated dataset and a four-dimension evaluation framework. Experiments show significant progress but highlight persistent challenges in implementation-oriented tasks.

Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Models

arXiv cs.AI

Introduces Blind-Spots-Bench, a benchmark designed to expose persistent failures in modern multimodal AI models on tasks that are trivial for humans. Evaluates a range of models, revealing performance gaps and that no single model dominates across all task types.

Benchmarking Visual State Tracking in Multimodal Video Understanding

Hugging Face Daily Papers

Introduces VSTAT, a benchmark for evaluating visual state tracking in multimodal large language models (MLLMs) using 834 clips and 1,500 questions. Current MLLMs perform poorly compared to humans, failing at visual perception rather than reasoning.