More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models

Papers with Code Trending Papers
vision-language-models multimodal-reasoning reinforcement-learning perceptual-grounding visual-forgetting vapo

Summary

This paper uncovers that prolonged reasoning in vision-language models can impair perceptual grounding, causing recognition failures on basic visual questions. It proposes Vision-Anchored Policy Optimization (VAPO) to steer reasoning toward visually grounded trajectories, achieving state-of-the-art performance with the VAPO-Thinker-7B model.

Reasoning has emerged as a pivotal capability in Large Language Models (LLMs). Through Reinforcement Learning (RL), typically Group Relative Policy Optimization (GRPO), these models are able to solve complex tasks such as mathematics and code generation. Building on these advances, recent research has sought to extend reasoning to Vision-Language Models (VLMs), yielding promising results across diverse visual tasks. Despite this progress, our study uncovers the dual nature of multimodal reasoning: while it substantially enhances logical inference and facilitates performance on challenging problems, it may gradually impair perceptual grounding, leading to recognition failures on otherwise basic visual questions. Through further analysis, we attribute this phenomenon to visual forgetting, wherein prolonged reasoning causes the model to increasingly disregard visual input. To address this, we propose Vision-Anchored Policy Optimization (VAPO), a simple yet effective method that explicitly steers the reasoning process toward visually grounded trajectories. Our result model, VAPO-Thinker-7B, significantly strengthens the model's reliance on visual information and achieves new state-of-the-art results on a wide range of established benchmarks. Project page: https://xytian1008.github.io/VAPO/
Original Article
View Cached Full Text

Cached at: 05/26/26, 06:35 AM

Paper page - More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models

Source: https://huggingface.co/papers/2509.25848 Published on Sep 30, 2025

Abstract

VAPO-Thinker-7B enhances multimodal reasoning by anchoring the process to visual information, improving performance on visual tasks while maintaining logical inference.

Reasoning has emerged as a pivotal capability in Large Language Models (LLMs). ThroughReinforcement Learning(RL), typically Group Relative Policy Optimization (GRPO), these models are able to solve complex tasks such as mathematics and code generation. Building on these advances, recent research has sought to extend reasoning toVision-Language Models(VLMs), yielding promising results across diverse visual tasks. Despite this progress, our study uncovers the dual nature ofmultimodal reasoning: while it substantially enhanceslogical inferenceand facilitates performance on challenging problems, it may gradually impairperceptual grounding, leading to recognition failures on otherwise basic visual questions. Through further analysis, we attribute this phenomenon tovisual forgetting, wherein prolonged reasoning causes the model to increasingly disregard visual input. To address this, we proposeVision-Anchored Policy Optimization(VAPO), a simple yet effective method that explicitly steers the reasoning process toward visually grounded trajectories. Our result model, VAPO-Thinker-7B, significantly strengthens the model’s reliance on visual information and achieves new state-of-the-art results on a wide range of established benchmarks. Project page: https://xytian1008.github.io/VAPO/

View arXiv pageView PDFProject pageGitHub344Add to collection

Get this paper in your agent:

hf papers read 2509\.25848

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2509.25848 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2509.25848 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2509.25848 in a Space README.md to link it from this page.

Collections including this paper7

Browse 7 collections that include this paper

Similar Articles

Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap

arXiv cs.CL

This paper introduces CrossMath, a controlled multimodal reasoning benchmark that reveals a critical limitation in current vision-language models: they perform reasoning primarily in textual space rather than genuine vision-grounded reasoning, with visual input often degrading performance compared to text-only baselines. The authors propose fine-tuning approaches to mitigate this modality gap and improve multimodal reasoning capabilities.

Thinking with Visual Grounding

Hugging Face Daily Papers

This paper introduces visually grounded thinking, a method for vision-language models to interleave natural-language reasoning with explicit visual evidence grounding using points or boxes. A scalable synthesis pipeline and grounding-aware reinforcement learning improve reasoning accuracy, enabling a 4B model to match or surpass a 27B model on spatial and counting benchmarks.