See2Think: Do Multimodal Models Really Use Intermediate Visual States?

Hugging Face Daily Papers Papers
multimodal reasoning benchmark evaluation visual-states large-language-models

Summary

This paper introduces See2Think, a unified evaluation framework to test whether multimodal models genuinely rely on intermediate visual states during reasoning. It finds that visual reasoning is highly model- and environment-dependent, with faithful rendering being the main bottleneck.

Multimodal large language models increasingly use sketches, annotations, tools, and intermediate images during reasoning, but it remains unclear whether they truly rely on these visual states. Existing benchmarks are limited both by task collections with narrow coverage or partially text-solvable samples and by evaluations that emphasize final answers without diagnosing how intermediate visual states are generated, rendered, and used. We introduce See2Think, a unified evaluation framework comprising See2ThinkBench and Visual Action-of-Thought (VAoT). See2ThinkBench contains 1,200 open-ended, visually dependent problems across 12 task categories spanning 2D structured, 3D scene, and real-world reasoning. VAoT records textual thoughts, visual actions, rendered states, and subsequent reasoning under four controlled inference settings. Evaluating representative proprietary and open-source multimodal models, we find that visual reasoning is strongly model- and environment-dependent, with no single setting consistently dominating across tasks. Process analysis further shows that models usually select relevant visual operations, while faithful rendering remains the clearest bottleneck and high feedback uptake does not necessarily translate into accuracy gains. Under task-relevant corrupted feedback, models exhibit behavioral dependence on visual states, with accuracy dropping by over 10 percentage points in controlled interventions.
Original Article
View Cached Full Text

Cached at: 07/31/26, 09:53 AM

Paper page - See2Think: Do Multimodal Models Really Use Intermediate Visual States?

Source: https://huggingface.co/papers/2607.26769 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Multimodallargelanguagemodelsincreasinglyusesketches,annotations,tools,andintermediateimagesduringreasoning,butitremainsunclearwhethertheytrulyrelyonthesevisualstates.Existingbenchmarksarelimitedbothbytaskcollectionswithnarrowcoverageorpartiallytext-solvablesamplesandbyevaluationsthatemphasizefinalanswerswithoutdiagnosinghowintermediatevisualstatesaregenerated,rendered,andused.WeintroduceSee2Think,aunifiedevaluationframeworkcomprisingSee2ThinkBenchandVisualAction-of-Thought(VAoT).See2ThinkBenchcontains1,200open-ended,visuallydependentproblemsacross12taskcategoriesspanning2Dstructured,3Dscene,andreal-worldreasoning.VAoTrecordstextualthoughts,visualactions,renderedstates,andsubsequentreasoningunderfourcontrolledinferencesettings.Evaluatingrepresentativeproprietaryandopen-sourcemultimodalmodels,wefindthatvisualreasoningisstronglymodel-andenvironment-dependent,withnosinglesettingconsistentlydominatingacrosstasks.Processanalysisfurthershowsthatmodelsusuallyselectrelevantvisualoperations,whilefaithfulrenderingremainstheclearestbottleneckandhighfeedbackuptakedoesnotnecessarilytranslateintoaccuracygains.Undertask-relevantcorruptedfeedback,modelsexhibitbehavioraldependenceonvisualstates,withaccuracydroppingbyover10percentagepointsincontrolledinterventions.

View arXiv pageView PDFProject pageGitHub3Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.26769 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.26769 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.26769 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap

arXiv cs.CL

This paper introduces CrossMath, a controlled multimodal reasoning benchmark that reveals a critical limitation in current vision-language models: they perform reasoning primarily in textual space rather than genuine vision-grounded reasoning, with visual input often degrading performance compared to text-only baselines. The authors propose fine-tuning approaches to mitigate this modality gap and improve multimodal reasoning capabilities.

More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models

Papers with Code Trending

This paper uncovers that prolonged reasoning in vision-language models can impair perceptual grounding, causing recognition failures on basic visual questions. It proposes Vision-Anchored Policy Optimization (VAPO) to steer reasoning toward visually grounded trajectories, achieving state-of-the-art performance with the VAPO-Thinker-7B model.