It's Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them

Hugging Face Daily Papers Papers
vision-language-models evaluation benchmark robustness multimodal vlm-judges

Summary

论文提出 MIST（Misleading Image Stress Test）基准，测试 13 个 VLM judge 在存在对齐、误导或缺失图像时标签的稳定性，发现图像是否出现而非图像内容才是改变判断的主要因素，说明可替代性结论对评测配置同样敏感。

Vision-language models (VLMs) are increasingly used in place of human annotators, making it important that substitutability tests reflect the model rather than incidental evaluation conditions. We introduce MIST, the Misleading-Image Stress Test: 200 English sentences, each built around a phrase readable either figuratively or literally and shown with an aligned image depicting its reading, a misleading image depicting the opposite, or no image at all. The guidelines require the label to be decided from the sentence alone, so no image should change any answer. We expected each image to pull a judge's labels toward the sense it depicts, and neither kind did. Across thirteen VLM judges, an aligned image changed 20.5% of labels and a misleading one 19.4%, close for every judge and both above the 11.6% produced by deleting the ignore-the-image instruction with the image left in place. Yet only 37% of the labels that differ between the two images moved toward the sense shown, and agreement with our human annotators is unchanged whether the image is absent, aligned or misleading. The effect is smaller in the seven judges that pass the alt-test than in the six that never do, but present in all of them: what moves a judge is that an image is there, not which of the two it is, so a substitutability verdict describes a configuration as much as a model.
Original Article
View Cached Full Text

Cached at: 10/01/26, 12:24 PM

Paper page - It’s Not What the Image Shows: Irrelevant Context Destabilises VLM Judges Without Informing Them

Source: https://huggingface.co/papers/2609.37863

Abstract

Vision-languagemodels(VLMs)areincreasinglyusedinplaceofhumanannotators,makingitimportantthatsubstitutabilitytestsreflectthemodelratherthanincidentalevaluationconditions.WeintroduceMIST,theMisleading-ImageStressTest:200Englishsentences,eachbuiltaroundaphrasereadableeitherfigurativelyorliterallyandshownwithanalignedimagedepictingitsreading,amisleadingimagedepictingtheopposite,ornoimageatall.Theguidelinesrequirethelabeltobedecidedfromthesentencealone,sonoimageshouldchangeanyanswer.Weexpectedeachimagetopullajudge’slabelstowardthesenseitdepicts,andneitherkinddid.AcrossthirteenVLMjudges,analignedimagechanged20.5%oflabelsandamisleadingone19.4%,closeforeveryjudgeandbothabovethe11.6%producedbydeletingtheignore-the-imageinstructionwiththeimageleftinplace.Yetonly37%ofthelabelsthatdifferbetweenthetwoimagesmovedtowardthesenseshown,andagreementwithourhumanannotatorsisunchangedwhethertheimageisabsent,alignedormisleading.Theeffectissmallerinthesevenjudgesthatpassthealt-testthaninthesixthatneverdo,butpresentinallofthem:whatmovesajudgeisthatanimageisthere,notwhichofthetwoitis,soasubstitutabilityverdictdescribesaconfigurationasmuchasamodel.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.37863

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.37863 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.37863 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.37863 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Frozen Scenes, Shifting Winners: Configuration Fragility in Text-to-3D Evaluation

arXiv cs.AI

This paper audits configuration fragility in text-to-3D evaluation by holding 300 generated scenes fixed while varying render and caption settings, finding that configuration variance often exceeds between-generator variance and that VLM-based evaluators frequently shift their reported winner without any change in the underlying scenes.

A Plausible Explanation Is Enough to Mislead Multimodal Judges

arXiv cs.CL

该论文提出了一个名为JUPE的视觉问答基准，发现普通的看似合理的解释就足以误导14个多模态大模型评判者，使错误答案的拒绝率下降约12.7-13.8个百分点，研究者将此现象称为“plausibility capture”。