The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

Hugging Face Daily Papers Papers
causal-audit visual-tool-use multimodal-llm llm-evaluation causal-inference perception-benchmarks

Summary

This paper audits visual tool-use in multimodal LLMs via causal interventions, revealing that returned observations often lack causal effect despite aggregate accuracy gains. It identifies failure modes like 'Calling Without Looking' and 'Looking Without Planning', introducing the concept of the 'illusion of visual tool-use'.

The "thinking-with-images" paradigm equips multimodal LLMs with active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulate visual tool-use as a causal graph that separates observation-mediated paths from action-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand, Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncover policy miscalibration with two failure modes. In Calling Without Looking, returned observations have no causal effect on the answer. In Looking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy the illusion of visual tool-use: despite aggregate accuracy gains, visual tool-use is not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.
Original Article
View Cached Full Text

Cached at: 08/13/26, 03:33 PM

Paper page - The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

Source: https://huggingface.co/papers/2608.06270

Abstract

Visual tool-use in multimodal LLMs often lacks causal effectiveness, with returned observations frequently failing to influence answers or being used incoherently despite aggregate accuracy improvements.

The “thinking-with-images” paradigm equipsmultimodal LLMswith active visual operations such as crop-and-zoom. However, models using these operations often achieve only marginal or negative gains over direct inference at substantially higher token cost. They may also repeatedly crop irrelevant regions and fail on questions that direct inference answers correctly. We ask whether the returned visual evidence causally affects the answer. To answer this question, we formulatevisual tool-useas acausal graphthat separatesobservation-mediated pathsfromaction-induced shortcuts. We then audit it through interventions at the three levels: policy (comparing tool-use with direct inference), trajectory (corrupting all observations during rollout), and step (counterfactually replacing one individual observation under a fixed prefix). Our step-level estimand,Visual Evidence Gain, isolates the contribution of each returned observation. Across six representative models and five fine-grained perception benchmarks, we uncoverpolicy miscalibrationwith two failure modes. InCalling Without Looking, returned observations have no causal effect on the answer. InLooking Without Planning, observations are informative but the call schedule is incoherent. A trajectory-level diagnostic decomposes the policy-level accuracy gain and shows that the gain is concentrated in a Calibrated minority. We term this discrepancy theillusion of visual tool-use: despite aggregate accuracy gains,visual tool-useis not causally effective across a broad range of rollouts. The code is available at https://github.com/OpenCausaLab/CauAudit.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2608\.06270

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.06270 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.06270 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.06270 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles