Visual Access Boundaries in Vision-Language Model Reasoning

# Visual Access Boundaries in Vision-Language Model Reasoning
Source: [https://arxiv.org/html/2607.12815](https://arxiv.org/html/2607.12815)
Hiroto Osaka The University of Tokyo hiroto\.osaka@weblab\.t\.u\-tokyo\.ac\.jp&Shohei Taniguchi The University of Tokyo taniguchi@weblab\.t\.u\-tokyo\.ac\.jpGouki Minegishi The University of Tokyo minegishi@weblab\.t\.u\-tokyo\.ac\.jp&Kai Yamashita The University of Tokyo kai\.yamashita@weblab\.t\.u\-tokyo\.ac\.jpMasahiro Suzuki The University of Tokyo masa@weblab\.t\.u\-tokyo\.ac\.jp&Yutaka Matsuo The University of Tokyo matsuo@weblab\.t\.u\-tokyo\.ac\.jp

###### Abstract

Chain\-of\-Thought \(CoT\) prompting is widely used as a test\-time scaling strategy for Vision\-Language Models \(VLMs\), but it remains unclear what is extended when VLMs generate longer reasoning traces\. We ask whether CoT requires continued access to image tokens, or whether it mainly operates over visual information already made available earlier in the forward pass\. We introduceVisual Access Sweep, a causal intervention that masks attention from generated\-token queries to image\-token keys along layer depth and generation time, and define theVisual Access Boundary\(VAB\) as the minimal access region that preserves task accuracy\. Across six model configurations from Qwen2\.5\-VL and InternVL3, both no\-CoT direct answering and CoT prompting exhibit finite VABs\. In Qwen2\.5\-VL\-32B and InternVL3 at 14B and 38B scales, when CoT is evaluated against the no\-CoT full\-access target, its VAB layer differs from the no\-CoT boundary by at most two layers, despite substantially longer generations\. This suggests that CoT does not primarily improve performance by prolonging direct image\-token access throughout the reasoning trace, but by extending language\-side computation over image\-derived hidden\-state information\. We further show that CoT gains are constrained by perceptual readout\. CoT helps when the queried visual attribute can be reliably read out by the model, but not when that readout is unreliable\. A symbolic\-attribute oracle shows that CoT can improve counting once ground\-truth attributes are supplied as text, while a single\-object probe\-vs\-decode check shows that hard attributes can be linearly recoverable from hidden states yet difficult for the model itself to output\. Together, these analyses place the bottleneck at readout rather than counting\.

## 1Introduction

CoT prompting is a standard test\-time scaling tool in LLMs\(Kojimaet al\.,[2022](https://arxiv.org/html/2607.12815#bib.bib10); Weiet al\.,[2022](https://arxiv.org/html/2607.12815#bib.bib3); Wanget al\.,[2023](https://arxiv.org/html/2607.12815#bib.bib4)\)and has also been adopted in VLMs\(Zhanget al\.,[2024](https://arxiv.org/html/2607.12815#bib.bib22); Xuet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib32); Linet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib18)\)\. However, its benefits on visual tasks are not uniform: some studies report gains from explicit intermediate reasoning, whereas others find limited or degraded gains on perception\-heavy visual QA settings\(Luet al\.,[2024](https://arxiv.org/html/2607.12815#bib.bib28); Liuet al\.,[2024b](https://arxiv.org/html/2607.12815#bib.bib11)\)\. What mechanism underlies CoT’s contribution to VLM performance? Does CoT improve performance by prolonging direct access to image tokens during generation, or by enabling additional language\-side computation over visual information already made available earlier?

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/overview_vab_framework-3.png)Figure 1:Overview of Visual Access Sweep and the Visual Access Boundary\.\(a\)Visual Access Sweep\.At each sweep point\(tcutoff,ℓcutoff\)\(t\_\{\\text\{cutoff\}\},\\ell\_\{\\text\{cutoff\}\}\), we allow direct image\-token access only for generated\-token queries satisfyingt≤tcutofft\\leq t\_\{\\text\{cutoff\}\}and layers satisfyingℓ≤ℓcutoff\\ell\\leq\\ell\_\{\\text\{cutoff\}\}\. Outside this allowed rectangle, attention from generated\-token queries to image\-token keys is masked\. TheVisual Access Boundary\(VAB\) is the minimal access pair\(ℓ∗,τ∗\)\(\\ell^\{\*\},\\tau^\{\*\}\)that preserves accuracy within toleranceϵ\\epsilon\. \(b\)Key result\.Under direct prompting, knocking out later layers barely affects accuracy, identifying a depthℓ∗\\ell^\{\*\}beyond which continued direct image\-token access is functionally redundant\. Under CoT prompting, the VAB region is preserved despite a much larger generated\-token count, indicating that extended language\-side reasoning does not require extended direct image\-token access\.Existing evidence on this question is largely observational: studies document that VLM attention to image tokens decays as generation proceeds\(Tonget al\.,[2024](https://arxiv.org/html/2607.12815#bib.bib24); Rahmanzadehgerviet al\.,[2024](https://arxiv.org/html/2607.12815#bib.bib15); Kanget al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib7); Yuanet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib19); Yanget al\.,[2025a](https://arxiv.org/html/2607.12815#bib.bib25)\), but attention magnitude does not establish necessity\. Low attention can mean that continued access is no longer used, or that access is failing; the two are indistinguishable from attention values alone\. We therefore test necessity by intervention rather than by reading off attention\.

To answer this question, we introduce theVisual Access Boundary\(VAB\)\. We intervene by masking direct attention from generated\-token queries to image\-token keys across layer depth and generation time \(Figure[1](https://arxiv.org/html/2607.12815#S1.F1)\(a\)\)\. We call this two\-axis interventionVisual Access Sweep\. This intervention turns an otherwise observational question into a causal one\. If performance is preserved after removing later generated\-token access to image tokens, then those direct visual accesses are not necessary for the target behavior\. Instead, when CoT helps, it appears to operate over image\-derived information already carried in hidden states\.

To isolate visual grounding from downstream reasoning, we design controlled attribute\-counting tasks in which object attributes and spatial layouts are fully parameterized, following the spirit of controlled visual reasoning benchmarks\. This avoids cases where a model can answer from parametric knowledge or dataset bias rather than from the image itself, while keeping the reasoning operation fixed as counting and varying only the queried visual attribute\. We compare no\-CoT direct answering, which we refer to as Direct, with CoT prompting\. We further test whether the same structure appears beyond synthetic images using a GQA\-derived real\-image yes/no task\(Hudson and Manning,[2019](https://arxiv.org/html/2607.12815#bib.bib2)\)\.

Applied across six model configurations from Qwen2\.5\-VL and InternVL3, Visual Access Sweep yields two main findings\. First, as schematized in Figure[1](https://arxiv.org/html/2607.12815#S1.F1)\(b\), a Visual Access Boundary exists clearly under both Direct and CoT prompting\. In every main model\-task setting we test, accuracy is preserved while a broad upper\-layer region is blocked, then drops sharply once the intervention reaches earlier layers\. Taken together, these results adjudicate between the two mechanisms above\. In Direct answering, there is a depth after which generated tokens no longer need direct attention to image tokens\. Under CoT, generation becomes much longer, but the necessary visual\-access region does not expand proportionally along the generation\-time axis\. Thus, in the regimes we test, CoT does not primarily improve performance by continuing to re\-read the image throughout the reasoning trace\. Instead, when CoT helps, it appears to operate over image\-derived information already carried in hidden states\. The same qualitative VAB structure also appears on the GQA\-derived real\-image task\. Second, on Qwen2\.5\-VL across three scales, per\-attribute CoT gains are predicted by multi\-object perceptual readout accuracy at 3B and 7B and saturate at 32B\. An oracle bypass that supplies the attribute symbolically restores CoT gains uniformly across attributes, and a probe\-vs\-decode gap on hard attributes provides separate single\-object evidence that the bottleneck sits at perceptual readout rather than the counting operation itself \(Section[6](https://arxiv.org/html/2607.12815#S6)\)\.

Our contributions are as follows\. \(i\) We introduce Visual Access Sweep and the Visual Access Boundary, a causal intervention and an operational measure that together map the functional necessity of direct image\-token access along both layer depth and generation time\. \(ii\) We show that finite VABs appear under both Direct and CoT prompting across the main Qwen2\.5\-VL and InternVL3 settings\. In larger models, the CoT VAB layer remains within two layers of the no\-CoT boundary when evaluated against the no\-CoT full\-access target, despite much longer generations\. The same qualitative VAB structure also appears on a GQA\-derived real\-image yes/no task\. \(iii\) On Qwen2\.5\-VL, per\-attribute CoT gain tracks multi\-object perceptual readout accuracy across three scales and is restored uniformly by an oracle bypass on the same checkpoint without image input\. A separate single\-object probe\-vs\-decode diagnostic on hard attributes provides supporting evidence that the bottleneck sits at perceptual readout rather than the counting operation itself\.

## 2Related Work

#### Visual attention decay in VLMs\.

A growing body of work shows that VLM attention to image tokens often weakens, becomes noisy, or is misallocated during generation, and several methods improve performance by strengthening, redistributing, or re\-injecting visual signals during inference\(Tonget al\.,[2024](https://arxiv.org/html/2607.12815#bib.bib24); Rahmanzadehgerviet al\.,[2024](https://arxiv.org/html/2607.12815#bib.bib15); Kanget al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib7); Yuanet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib19); Yanget al\.,[2025a](https://arxiv.org/html/2607.12815#bib.bib25)\)\. These studies establish that modifying visual attention can be beneficial\. Our question is complementary\. In an unmodified prefix\-based decoder\-only VLM, when is continued direct image\-token access functionally necessary during generation? Rather than enhancing visual attention, Visual Access Sweep removes generated\-token access to image tokens along layer depth and generation time, thereby testing necessity rather than utility of attention amplification\.

#### CoT and test\-time scaling in VLMs\.

CoT and test\-time scaling are well established in LLMs\(Weiet al\.,[2022](https://arxiv.org/html/2607.12815#bib.bib3); Kojimaet al\.,[2022](https://arxiv.org/html/2607.12815#bib.bib10); Wanget al\.,[2023](https://arxiv.org/html/2607.12815#bib.bib4); Liuet al\.,[2025a](https://arxiv.org/html/2607.12815#bib.bib26)\)and have been extended to VLMs, with some studies reporting improved multimodal reasoning\(Zhanget al\.,[2024](https://arxiv.org/html/2607.12815#bib.bib22); Xuet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib32); Linet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib18); Wuet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib23)\)and others showing limited or degraded gains in perception\-heavy settings\(Luet al\.,[2024](https://arxiv.org/html/2607.12815#bib.bib28); Liuet al\.,[2024b](https://arxiv.org/html/2607.12815#bib.bib11)\)\. Rather than searching for the best prompting recipe, we ask what is being extended when CoT does help\. Because generated rationales need not faithfully reveal the computation that produced the answer\(Turpinet al\.,[2023](https://arxiv.org/html/2607.12815#bib.bib40); Lanhamet al\.,[2023](https://arxiv.org/html/2607.12815#bib.bib41)\), we do not infer visual use from the CoT text itself; we instead test the functional necessity of image\-token access by intervention\.

#### Active perception and visual re\-access\.

Several recent approaches modify VLM inference by allowing the model to acquire additional visual evidence during reasoning, for example through visual memory, region refocusing, or explicit perception actions\(Zouet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib35); Yanget al\.,[2025b](https://arxiv.org/html/2607.12815#bib.bib36); Yuet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib37)\)\. These methods study how to improve models by adding a perception\-reasoning feedback loop\. Our work asks a complementary diagnostic question\. We test whether ordinary CoT already makes continued direct access from generated tokens to the initial image\-token prefix functionally necessary in prefix\-based decoder\-only VLMs\.

#### Mechanistic interpretability and probing in VLMs\.

Causal\-intervention methods on attention or activations\(Viget al\.,[2020](https://arxiv.org/html/2607.12815#bib.bib21); Geigeret al\.,[2021](https://arxiv.org/html/2607.12815#bib.bib27); Menget al\.,[2022](https://arxiv.org/html/2607.12815#bib.bib5); Schwettmannet al\.,[2023](https://arxiv.org/html/2607.12815#bib.bib17); Gandelsmanet al\.,[2024](https://arxiv.org/html/2607.12815#bib.bib16)\)and probing studies showing VLM hidden states encode visual information the model fails to verbalize\(Alain and Bengio,[2016](https://arxiv.org/html/2607.12815#bib.bib9); Belroseet al\.,[2023](https://arxiv.org/html/2607.12815#bib.bib8); Gevaet al\.,[2023](https://arxiv.org/html/2607.12815#bib.bib29); Fuet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib33); Liuet al\.,[2025b](https://arxiv.org/html/2607.12815#bib.bib34)\)both inform our approach\. Unlike tuned\-lens\-style observational decoding, the VAB measures via causal masking where direct image\-token access stops being*necessary*, and we go beyond generic readout\-bottleneck observations by relating the probe\-vs\-decode gap to CoT gain through oracle bypass\.

## 3Experimental Design

### 3\.1VLM formulation

We focus on decoder\-only VLMs in which a frozen vision encoder followed by a projection layer produces image tokens that are concatenated as a prefix to text tokens, and outputs are generated autoregressively\(Vaswaniet al\.,[2017](https://arxiv.org/html/2607.12815#bib.bib30); Liuet al\.,[2023](https://arxiv.org/html/2607.12815#bib.bib1),[2024a](https://arxiv.org/html/2607.12815#bib.bib20); Baiet al\.,[2024](https://arxiv.org/html/2607.12815#bib.bib31),[2025](https://arxiv.org/html/2607.12815#bib.bib6)\)\. Throughout this paper, when we refer to attention from generated\-token queries to image\-token keys, we mean the corresponding entries of the standard self\-attention matrix in this prefix\-based decoder\-only setup, rather than a separate cross\-attention module\. Let𝐭img=\(timg,1,…,timg,K\)\\mathbf\{t\}\_\{\\mathrm\{img\}\}=\(t\_\{\\mathrm\{img\},1\},\\ldots,t\_\{\\mathrm\{img\},K\}\)denote theKKimage tokens,𝐭text\\mathbf\{t\}\_\{\\mathrm\{text\}\}the input text tokens, and𝐭gen,1:t−1\\mathbf\{t\}\_\{\\mathrm\{gen\},1:t\-1\}the previously generated tokens; at stepttthe model conditions on𝐓t=\[𝐭img,𝐭text,𝐭gen,1:t−1\]\\mathbf\{T\}\_\{t\}=\[\\mathbf\{t\}\_\{\\mathrm\{img\}\},\\,\\mathbf\{t\}\_\{\\mathrm\{text\}\},\\,\\mathbf\{t\}\_\{\\mathrm\{gen\},1:t\-1\}\]and predictstgen,tt\_\{\\mathrm\{gen\},t\}\. The operational definition ofvisual accessthat our intervention manipulates is given in Section[4\.1](https://arxiv.org/html/2607.12815#S4.SS1)\.

### 3\.2Tasks and datasets

We use two task families, a controlled synthetic attribute\-counting task family that allows precise factor isolation and a real\-image scene\-graph QA task derived from GQA that probes external validity beyond synthetic stimuli\.

#### Controlled attribute\-counting tasks\.

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/controlled_task_design-2.png)Figure 2:Controlled attribute\-counting task\.The reasoning operation \(count\) is fixed while the queried attribute varies across \{color, shape, location, angle, size\}, isolating the effect of perceptual difficulty on a single, fixed reasoning program\.The controlled tasks vary the queried visual attribute while holding the reasoning operation fixed \(counting\)\. Each scene is procedurally generated with parameterized objects whose attributes \(color, shape, location, angle, size, with 4 classes each\) span a range of perceptual difficulty \(Appendix[A\.1](https://arxiv.org/html/2607.12815#A1.SS1)\)\. For each scene we issue five counting queries of the form “How manyattribute=valueobjects are in the image?” \(Figure[2](https://arxiv.org/html/2607.12815#S3.F2)\), one per attribute\.

The role of these tasks is diagnostic, not benchmark coverage\. Holding the reasoning operation fixed across attributes makes it harder to attribute differences in CoT gain to differences in reasoning difficulty\. What varies is the difficulty of obtaining the relevant attribute in a form usable by counting\. Section[6](https://arxiv.org/html/2607.12815#S6)uses two diagnostics on the same attributes, namely multi\-object readout on the counting scenes and a single\-object probe\-vs\-decode check in Appendix[D\.1](https://arxiv.org/html/2607.12815#A4.SS1)\. Both test whether the queried attribute is available as a usable symbol\. A concrete counting example is in Appendix[A\.2](https://arxiv.org/html/2607.12815#A1.SS2)\.

#### Real\-image scene\-graph QA\.

To test whether the VAB structure is specific to synthetic stimuli, we additionally construct a real\-image extension from the balanced validation split of GQA\(Hudson and Manning,[2019](https://arxiv.org/html/2607.12815#bib.bib2)\)\.111[https://huggingface\.co/datasets/vikhyatk/gqa](https://huggingface.co/datasets/vikhyatk/gqa)GQA contains natural photographs paired with scene\-graph\-derived questions, and its balanced split is designed to reduce answer\-distribution bias\. We use the yes/no subset to keep the answer space controlled and to reduce answer\-parsing ambiguity\. Full preprocessing, prompting, answer extraction, and license details are in Appendix[C\.6](https://arxiv.org/html/2607.12815#A3.SS6)\.

For both task families, we compare Direct prompting, which asks for a final answer with minimal intermediate output, against CoT prompting, which asks the model to produce intermediate reasoning before the final answer\. Controlled counting is evaluated by constrained option\-key accuracy, and the GQA\-derived task by binary yes/no accuracy\. Full prompt templates, answer extraction rules, decoding settings, and compute details are provided in Appendix[A\.3](https://arxiv.org/html/2607.12815#A1.SS3)and Appendix[A\.4](https://arxiv.org/html/2607.12815#A1.SS4)\.

## 4Visual Access Sweep

### 4\.1Intervention

We operationally define visual access at\(ℓ,t\)\(\\ell,t\)as direct attention from the generated\-token query at generation stepttand layerℓ\\ellto image\-token keys \(Figure[1](https://arxiv.org/html/2607.12815#S1.F1)\(a\)\)\. Visual Access Sweep restricts this channel by two cutoffs\. Image\-token access is allowed only inside the rectangleℓ≤ℓcutoff\\ell\\leq\\ell\_\{\\text\{cutoff\}\}andt≤tcutofft\\leq t\_\{\\text\{cutoff\}\}, and is masked outside this region\. Sweeping\(ℓcutoff,tcutoff\)\(\\ell\_\{\\text\{cutoff\}\},t\_\{\\text\{cutoff\}\}\)thus traces how much direct image\-token access is needed to preserve task performance\.

Throughout this paper, visual access denotes this specific computational channel\. Visual information that has already been incorporated into hidden states through earlier computation can still persist through the residual stream and is not removed by this intervention\. Concretely, for the additive attention mask𝐌ℓ\\mathbf\{M\}^\{\\ell\}at layerℓ\\ell, we set𝐌i​jℓ=−∞\\mathbf\{M\}^\{\\ell\}\_\{ij\}=\-\\inftyfor every \(generated\-query positionii, image\-token key positionjj\) pair within the masked region, so that the corresponding post\-softmax attention weight is exactly zero:

Attention​\(𝐐ℓ,𝐊ℓ,𝐕ℓ\)=softmax​\(𝐐ℓ​𝐊ℓ⊤d\+𝐌ℓ\)​𝐕ℓ,𝐌i​jℓ=\{−∞if intervention active,0otherwise\.\\mathrm\{Attention\}\(\\mathbf\{Q\}^\{\\ell\},\\mathbf\{K\}^\{\\ell\},\\mathbf\{V\}^\{\\ell\}\)=\\mathrm\{softmax\}\\\!\\left\(\\frac\{\\mathbf\{Q\}^\{\\ell\}\{\\mathbf\{K\}^\{\\ell\}\}^\{\\top\}\}\{\\sqrt\{d\}\}\+\\mathbf\{M\}^\{\\ell\}\\right\)\\mathbf\{V\}^\{\\ell\},\\;\\mathbf\{M\}^\{\\ell\}\_\{ij\}=\\begin\{cases\}\-\\infty&\\text\{if intervention active\},\\\\ 0&\\text\{otherwise\}\.\\end\{cases\}\(1\)Thelayer cutoffℓcutoff\\ell\_\{\\text\{cutoff\}\}keeps direct image\-token access in layers1,…,ℓcutoff1,\\ldots,\\ell\_\{\\text\{cutoff\}\}and removes it in all subsequent layers\. Thetoken cutofftcutofft\_\{\\text\{cutoff\}\}keeps access for the firsttcutofft\_\{\\text\{cutoff\}\}generated tokens and removes it thereafter\. Because direct prompting and CoT prompting produce sequences of very different lengths, we use the effective windowteff=min⁡\(tcutoff,Tgen\)t\_\{\\text\{eff\}\}=\\min\(t\_\{\\text\{cutoff\}\},T\_\{\\text\{gen\}\}\)and treat samples withTgen<tcutoffT\_\{\\text\{gen\}\}<t\_\{\\text\{cutoff\}\}as having full token\-axis access\. The full\-access baseline corresponds to the sweep point with the largest layer and token cutoffs\. To verify that the observed boundary is not an artifact of the masking operation, Appendix[B\.1](https://arxiv.org/html/2607.12815#A2.SS1)reports null\-sink and late\-layer query\-text\-block controls\. These controls test for attention redistribution and query\-mediated re\-reading, respectively, and both preserve the qualitative VAB structure on Qwen2\.5\-VL\-32B\.

### 4\.2Visual Access Boundary

Figure[1](https://arxiv.org/html/2607.12815#S1.F1)\(b\) illustrates how the sweep is read\. If accuracy remains within tolerance after masking a broad late\-layer or late\-token region, then continued direct image\-token access in that region is not functionally necessary for the task\.

For compact notation, we writeτ\\taufor the effective token windowtefft\_\{\\text\{eff\}\}\. LetAfullpA^\{p\}\_\{\\mathrm\{full\}\}denote the full\-access accuracy under prompt conditionp∈\{D,CoT\}p\\in\\\{\\mathrm\{D\},\\mathrm\{CoT\}\\\}, and letAp​\(ℓ,τ\)A^\{p\}\(\\ell,\\tau\)denote the accuracy after applying the visual\-access cutoff\(ℓ,τ\)\(\\ell,\\tau\)under the same prompt condition\.

For a toleranceϵ\\epsilon, each boundary is defined as the minimal access region on the sweep grid whose criterion\-specific accuracy drop is at mostϵ\\epsilon\. We useϵ=0\.05\\epsilon=0\.05throughout, corresponding to a 5\-percentage\-point accuracy budget\. The three drops we compare againstϵ\\epsilonare

ΔD​\(ℓ,τ\)\\displaystyle\\Delta\_\{\\mathrm\{D\}\}\(\\ell,\\tau\)=AfullD−AD​\(ℓ,τ\),\\displaystyle=A^\{\\mathrm\{D\}\}\_\{\\mathrm\{full\}\}\-A^\{\\mathrm\{D\}\}\(\\ell,\\tau\),\(2\)ΔCoT​\(ℓ,τ\)\\displaystyle\\Delta\_\{\\mathrm\{CoT\}\}\(\\ell,\\tau\)=AfullCoT−ACoT​\(ℓ,τ\),\\displaystyle=A^\{\\mathrm\{CoT\}\}\_\{\\mathrm\{full\}\}\-A^\{\\mathrm\{CoT\}\}\(\\ell,\\tau\),ΔDA​\(ℓ,τ\)\\displaystyle\\Delta\_\{\\mathrm\{DA\}\}\(\\ell,\\tau\)=AfullD−ACoT​\(ℓ,τ\)\.\\displaystyle=A^\{\\mathrm\{D\}\}\_\{\\mathrm\{full\}\}\-A^\{\\mathrm\{CoT\}\}\(\\ell,\\tau\)\.The Direct boundaryℓD∗\\ell^\{\*\}\_\{\\mathrm\{D\}\}is computed fromΔD\\Delta\_\{\\mathrm\{D\}\}\. The CoT\-own\-max boundaryℓCoT∗\\ell^\{\*\}\_\{\\mathrm\{CoT\}\}is computed fromΔCoT\\Delta\_\{\\mathrm\{CoT\}\}, asking when CoT under intervention preserves CoT’s own full\-access ceiling\. The direct\-anchored CoT boundaryℓDA∗\\ell^\{\*\}\_\{\\mathrm\{DA\}\}is computed fromΔDA\\Delta\_\{\\mathrm\{DA\}\}, asking when CoT under intervention reaches the Direct full\-access target\.

We report two layer shifts,

Δ​ℓown∗=ℓCoT∗−ℓD∗,Δ​ℓDA∗=ℓDA∗−ℓD∗\.\\Delta\\ell^\{\*\}\_\{\\mathrm\{own\}\}=\\ell^\{\*\}\_\{\\mathrm\{CoT\}\}\-\\ell^\{\*\}\_\{\\mathrm\{D\}\},\\qquad\\Delta\\ell^\{\*\}\_\{\\mathrm\{DA\}\}=\\ell^\{\*\}\_\{\\mathrm\{DA\}\}\-\\ell^\{\*\}\_\{\\mathrm\{D\}\}\.\(3\)Per\-setting numbers are summarized in Section[5\.2](https://arxiv.org/html/2607.12815#S5.SS2)and reported in Appendix[C\.1](https://arxiv.org/html/2607.12815#A3.SS1)\. We verify in Appendix[B\.2](https://arxiv.org/html/2607.12815#A2.SS2)that the central CoT\-vs\-Direct comparison is stable underϵ∈\{0\.03,0\.05,0\.07,0\.10\}\\epsilon\\in\\\{0\.03,0\.05,0\.07,0\.10\\\}\(\|Δ​ℓDA∗\|≤1\|\\Delta\\ell^\{\*\}\_\{\\mathrm\{DA\}\}\|\\leq 1under the direct\-anchored criterion at allϵ\\epsilontested\)\. By construction, the VAB measures the functional necessity of continued direct image\-token access\. It does not erase image\-derived information that has already propagated into hidden states\.

## 5Visual Access Boundary in Direct and CoT Inference

Across model families, scales, and stimulus domains, Visual Access Sweep reveals a consistent structure\. Performance is preserved after masking a broad late region of direct image\-token access, and drops sharply only when the intervention reaches earlier layers or earlier generated tokens\. This structure appears in Direct inference, persists under CoT prompting, and also appears on a GQA\-derived real\-image QA task\. The main empirical result is therefore not a particular numerical layer shift, but the repeated emergence of a finite Visual Access Boundary\. Extended generation does not imply continued direct image\-token access throughout the reasoning trace\.

### 5\.1Visual Access Boundary in direct inference

We first establish that direct inference exhibits a clear functional boundary on continued image\-token access\. Figure[3](https://arxiv.org/html/2607.12815#S5.F3)\(a\) shows the Direct layer sweep on Qwen2\.5\-VL\-32B for color and shape counting\. Accuracy remains near the full\-access baseline while a broad upper\-layer region is blocked, then drops sharply once the intervention reaches earlier layers\. Exactℓ∗\\ell^\{\*\}values are reported in the figure and in Appendix[C\.1](https://arxiv.org/html/2607.12815#A3.SS1)\. We interpret this boundary operationally\. Beyondℓ∗\\ell^\{\*\}, additional direct attention from generated tokens to image tokens is no longer needed for this task, although visual information already carried in hidden states may still be used\.

### 5\.2Visual Access Boundary in CoT inference

We now ask whether the much longer generation produced by CoT requires a correspondingly larger region of direct image\-token access\. Figure[3](https://arxiv.org/html/2607.12815#S5.F3)\(b\) shows the joint sweep over layer and effective token cutoffs under CoT on Qwen2\.5\-VL\-32B\. The region whereΔCoT​\(ℓ,τ\)≤ϵ=0\.05\\Delta\_\{\\mathrm\{CoT\}\}\(\\ell,\\tau\)\\leq\\epsilon=0\.05occupies the upper portion of the heatmap, and the star marks the empirical\(ℓ∗,τ∗\)\(\\ell^\{\*\},\\tau^\{\*\}\)\.

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_direct_qwen25_32b_appendix-fixed.png)\(a\)Direct: layer sweep for color and shape counting\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_cot_qwen25_32b_appendix-fixed.png)\(b\)CoT: layer\-token sweep\. Stars mark the empirical\(ℓ∗,τ∗\)\(\\ell^\{\*\},\\tau^\{\*\}\)\.

Figure 3:Visual Access Boundary on Qwen2\.5\-VL\-32B\.\(a\)Direct inference shows a sharp layer\-wise transition: accuracy is preserved while a broad upper\-layer region is blocked, but drops once the intervention reaches earlier layers\.\(b\)Under CoT, the high\-accuracy region remains concentrated in a similar depth range despite much longer generations\. Stars mark the empirical VAB underϵ=0\.05\\epsilon=0\.05\. Exact boundary values are reported in Appendix[C\.1](https://arxiv.org/html/2607.12815#A3.SS1)\.We use the CoT\-own\-max and direct\-anchored criteria defined in Section[4\.2](https://arxiv.org/html/2607.12815#S4.SS2)\. The structural observation is that, across the 12 \(family, scale, task\) settings we test,\|Δ​ℓown∗\|\|\\Delta\\ell^\{\*\}\_\{\\mathrm\{own\}\}\|stays within 9 layers while generation length increases by approximately50×50\\times\. The direct\-anchored CoT boundary is generally close to the Direct VAB on the larger models for which the target is attainable \(Qwen2\.5\-VL\-32B and the 14B and 38B variants of InternVL3 yield\|Δ​ℓDA∗\|≤2\|\\Delta\\ell^\{\*\}\_\{\\mathrm\{DA\}\}\|\\leq 2\)\. Settings where the direct\-anchored target is not reached are reported separately\. The consistent pattern is the existence of a finite VAB under CoT, not a monotonic deepening of the boundary\. We discuss possible mechanisms for this decoupling in Section[7](https://arxiv.org/html/2607.12815#S7)\.

### 5\.3Cross\-family and cross\-scale VAB sweeps

A natural concern is that the VAB structure could be specific to a single family, since differences in vision encoder, projection, and training data could in principle create an architecture\-specific late\-layer redundancy\. We evaluate the same Visual Access Sweep on Qwen2\.5\-VL\(Baiet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib6)\)at three scales \(3B, 7B, 32B\) and on InternVL3\(Zhuet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib39)\)at three scales \(8B, 14B, 38B\)\. A layer\-wise VAB is observed in every case \(per\-model figures in Appendix[C\.4](https://arxiv.org/html/2607.12815#A3.SS4), architectural details in Appendix[C\.2](https://arxiv.org/html/2607.12815#A3.SS2)\)\. The shift is not consistently positive\. Qwen2\.5\-VL scales tend to show non\-negative CoT\-own\-max shifts, whereas InternVL3 does not show a systematic deepening\. In some larger InternVL3 settings, the CoT boundary is comparable to or slightly shallower than the Direct boundary\. What is constant across architectures is therefore the absence of generation\-proportional expansion, not a single\-direction shift\.

#### Cross\-attention fusion architectures\.

We further apply the same intervention principle to a cross\-attention\-based VLM \(Llama\-3\.2\-11B\-Vision\-Instruct\(Grattafioriet al\.,[2024](https://arxiv.org/html/2607.12815#bib.bib14)\)\), where visual access is implemented by masking the generated\-token queries’ cross\-attention to visual keys rather than self\-attention to an image\-token prefix \(Appendix[C\.5](https://arxiv.org/html/2607.12815#A3.SS5)\)\. The CoT\-side sweep again exhibits a finite VAB\. This pilot check suggests that the intervention principle can be applied beyond prefix\-fusion models, with explicit iterative re\-perception modules remaining the main untested regime\.

### 5\.4Real\-image scene\-graph QA extension

A second concern is that the VAB structure might be a property of the synthetic CLEVR\-style stimuli\(Johnsonet al\.,[2017](https://arxiv.org/html/2607.12815#bib.bib38)\)rather than of decoder\-only VLMs\. We apply Visual Access Sweep to the GQA\-derived real\-image scene\-graph QA task on Qwen2\.5\-VL\-32B \(Section[3\.2](https://arxiv.org/html/2607.12815#S3.SS2.SSS0.Px2)\)\. At full access, Direct accuracy is0\.840\.84and CoT accuracy is0\.800\.80\(Appendix[C\.6](https://arxiv.org/html/2607.12815#A3.SS6)\)\. Figure[4](https://arxiv.org/html/2607.12815#S5.F4)shows the joint sweep for both prompting modes\.

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_direct_qwen25_32b_gqa.png)\(a\)Direct: layer sweep on GQA yes/no\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_cot_qwen25_32b_gqa.png)\(b\)CoT: layer\-token sweep\. Star marks\(ℓ∗,τ∗\)\(\\ell^\{\*\},\\tau^\{\*\}\)\.

Figure 4:Real\-image extension: VAB on GQA yes/no \(Qwen2\.5\-VL\-32B\)\.\(a\)Direct inference reproduces the qualitative pattern of the controlled tasks\. A broad upper\-layer region can be blocked with little accuracy loss before a sharp transition\.\(b\)Under CoT, the high\-accuracy region is concentrated within a similar depth range despite much longer generations\. Both prompting modes therefore exhibit a finite VAB on real images\. Exact boundary values are reported in Appendix[C\.6](https://arxiv.org/html/2607.12815#A3.SS6)\.Although CoT does not improve this binary GQA setting, this aligns with recent evidence that longer reasoning in VLMs can weaken perceptual grounding rather than monotonically improve visual QA\(Tianet al\.,[2026](https://arxiv.org/html/2607.12815#bib.bib12); Kanchetiet al\.,[2026](https://arxiv.org/html/2607.12815#bib.bib13)\)\. Both Direct and CoT sweeps still exhibit a finite VAB, with the redundant late\-layer region comparable in size to synthetic counting on the same model\. The GQA result thus extends the VAB phenomenon to real images while separating the existence of a visual\-access boundary from whether CoT improves final accuracy\.

## 6Perceptual Readout Restricts CoT Gains

The Visual Access Sweep results say that, in the regime we study, CoT gains do not come from extending direct image\-token access\. They do not yet say what determines whether CoT gains exist at all on a given task\. In this section, we show that within Qwen2\.5\-VL, the magnitude of CoT gains is constrained by whether the queried visual attribute can be read out into a form usable by the model’s own reasoning chain\.

### 6\.1Defining perceptual readout

We distinguish two questions\. The first is whether an attribute is encoded somewhere in the model’s hidden states\. The second is whether the model can turn that attribute into a symbol that its own answer\-generation process can use\. This section focuses on the second question\.

We use two related readout diagnostics\. The main predictor of CoT gain is a*multi\-object perceptual readout*task, in which the model is given a scene with multiple objects and must fill in the queried attribute value for every object, while the remaining attributes are provided as textual scaffolding\. We use this multi\-object perceptual readout accuracy as the x\-axis in Figure[5](https://arxiv.org/html/2607.12815#S6.F5)\. Separately, Appendix[D\.1](https://arxiv.org/html/2607.12815#A4.SS1)reports a single\-object probe\-vs\-decode diagnostic, which tests whether attribute information is linearly decodable from hidden states even when the model cannot reliably express it in its own output\. Neither is a parallel main task\. Both are evaluation devices for whether the queried visual attribute is available as a usable symbol\.

This distinction is necessary because latent encoding and decoded availability can diverge\. On Qwen2\.5\-VL\-32B, hard attributes such as angle, location, and size show large probe\-vs\-decode gaps\. A linear probe on hidden states recovers them with high accuracy while the model’s own direct decoding is substantially lower \(Appendix[D\.1](https://arxiv.org/html/2607.12815#A4.SS1), Table[12](https://arxiv.org/html/2607.12815#A4.T12)\)\. The bottleneck is therefore not whether visual information exists internally, but whether the model can read it out into a usable symbolic form\.

### 6\.2Perceptual readout and CoT gain correlation

We hold the reasoning operation fixed \(counting\) and vary only the queried attribute, then plot per\-attribute CoT gain \(AccCoT−AccDirect\\mathrm\{Acc\}\_\{\\text\{CoT\}\}\-\\mathrm\{Acc\}\_\{\\text\{Direct\}\}\) against multi\-object perceptual readout accuracy on Qwen2\.5\-VL at three scales \(3B, 7B, 32B\) in Figure[5](https://arxiv.org/html/2607.12815#S6.F5)\.

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/perception_cot_scatter_cross_scale.png)Figure 5:Perceptual readout predicts CoT gain across Qwen2\.5\-VL scales\.Each point is one queried attribute \(color, shape, location, size, angle\)\. The x\-axis is multi\-object perceptual readout accuracy: the model is asked to fill in the queried attribute for every object in a multi\-object scene, and accuracy is computed on the queried attribute values\. The y\-axis is the CoT gain on the corresponding counting task\. CoT gains are largest when the queried attribute can be reliably read out\. Per\-attribute values and per\-scale correlations are in Appendix[D\.2](https://arxiv.org/html/2607.12815#A4.SS2)\.The relationship is positive at every scale\. Attributes with relatively higher readout accuracy within each scale tend to show larger CoT gains, whereas attributes with weaker readout \(angle and size at every scale\) show essentially no CoT gain or slightly negative gain\. At 32B, three of five attributes saturate above readout accuracy0\.770\.77, compressing residual variance\. We therefore readρ\\rhoas the more stable summary at high scale and the linear coefficient as a within\-readout\-range slope\. The angle limiting case \(chance\-level readout, VAB ill\-defined\) is in Appendix[C\.7](https://arxiv.org/html/2607.12815#A3.SS7)\.

Because the reasoning operation is held fixed across attributes, the variation in CoT gain is difficult to explain by reasoning difficulty alone\. CoT improves counting only after the queried visual attribute is available as a usable symbol, complementing Section[5](https://arxiv.org/html/2607.12815#S5)by identifying what limits CoT once extended visual access is not the source of improvement\.

### 6\.3Symbolic\-attribute oracle bypass

The correlation in Figure[5](https://arxiv.org/html/2607.12815#S6.F5)is observational\. Hard attributes may show small CoT gains either because the model cannot read out the relevant visual attribute, or because the corresponding counting question is intrinsically difficult to solve\. To separate these possibilities, we run an oracle\-bypass experiment\.

In this setting, we use the same Qwen2\.5\-VL\-3B checkpoint without image tokens\. No image is provided, the visual\-token pathway is bypassed, and ground\-truth object attributes are serialized directly into the prompt\. The model is then asked the same multiple\-choice counting questions under Direct and CoT prompting\. The experiment therefore does not test visual perception\. It tests whether the language\-side counting operation can benefit from CoT once the relevant attributes are already available as symbols\.

We consider two oracle inputs\. The*minimal oracle*provides only the queried attribute for each object \(e\.g\., color values when the question asks about color counting\)\. This is the cleanest test because it removes the visual readout step while preserving the counting operation\. The*full oracle*provides all object attributes, adding irrelevant information and serving as a robustness condition rather than the primary evidence\.

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/oracle_reasoning_qwen25_3b-fixed.png)Figure 6:Oracle bypass with the Qwen2\.5\-VL\-3B checkpoint without image input\.Ground\-truth attributes are serialized as text\.\(a\)Minimal oracle provides only the queried attribute for each object\.\(b\)Full oracle provides all object attributes\.Figure[6](https://arxiv.org/html/2607.12815#S6.F6)\(a\) shows that under the minimal oracle, CoT gains become consistent across attributes, including angle, size, and location, which showed weak gains under image input \(Figure[5](https://arxiv.org/html/2607.12815#S6.F5)\)\. The counting operation is therefore not the limiting factor once the relevant attributes are supplied as symbols\. The limiting step is making the queried visual attribute available in a usable symbolic form\. Figure[6](https://arxiv.org/html/2607.12815#S6.F6)\(b\) reports the full\-oracle condition, where the larger prompt changes the amount of irrelevant information and the headroom for CoT gains\. We treat it as a supplementary check\.

The probe\-decode gap and the oracle bypass together separate three quantities that are easy to conflate\. Linear probing asks whether the attribute is present in hidden states \(Section[6\.1](https://arxiv.org/html/2607.12815#S6.SS1)\)\. Direct decoding asks whether the model can read out the attribute as a usable symbol \(Section[6\.2](https://arxiv.org/html/2607.12815#S6.SS2)\)\. Oracle bypass asks whether the counting operation can benefit from CoT once that symbol is supplied \(this section\)\. The results point to the middle step as the bottleneck\. CoT can improve reasoning over available symbols, but it does not reliably recover a visual attribute that the model cannot itself read out\.

## 7Discussion

#### Interpreting the VAB\.

This work provides a causal test of what is extended when VLMs generate longer CoT traces\. Across the regimes tested here, Visual Access Sweep reveals a finite Visual Access Boundary under both Direct and CoT prompting\. Extended generation does not require continued direct image\-token access throughout the reasoning trace\. Operationally, the VAB marks when additional direct attention from generated\-token queries to image\-token keys becomes functionally unnecessary\. It does not imply that visual evidence is absent from the model\. Such evidence may already have been transformed into hidden\-state representations and carried forward through the residual stream\. This reframes visual\-attention decay and visual forgetting\. The key question is not whether attention to image tokens decreases, but whether later direct access is still necessary for the task\.

#### Implications for CoT and active perception\.

The perceptual\-readout results suggest that CoT helps when the needed visual attribute has first been made available in a form the model’s reasoning trace can use\. When readout succeeds, longer reasoning can improve the downstream counting operation\. When readout fails, CoT does not reliably recover the missing symbol, even if latent visual information may be present\. This gives a practical rule for test\-time scaling: before spending tokens on longer reasoning, test whether the relevant visual attribute can be read out reliably\. Within Qwen2\.5\-VL, a cheap multi\-object readout probe supports such routing, preserving most of the always\-CoT accuracy at substantially lower token cost \(Appendix[E](https://arxiv.org/html/2607.12815#A5)\)\. The same result clarifies the role of active perception methods such as MemVR\(Zouet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib35)\), Look\-Back\(Yanget al\.,[2025b](https://arxiv.org/html/2607.12815#bib.bib36)\), and Visual Perception Token\(Yuet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib37)\)\. These methods are complementary: ordinary CoT in the tested regime does not by itself close the perception\-reasoning loop, whereas active perception methods explicitly try to acquire new visual evidence conditioned on the reasoning state\.

#### Scope and limitations\.

The evidence in this paper covers controlled attribute\-counting tasks across families and scales, a GQA\-derived real\-image setting, and an additional cross\-attention fusion setting \(Appendix[C\.5](https://arxiv.org/html/2607.12815#A3.SS5)\)\. Tasks deliberately designed to require iterative visual inspection remain outside the current evaluation\. The sweep result also does not by itself identify the internal mechanism that produces the boundary\. The same VAB pattern could arise because visual attributes are bound into hidden\-state representations early, because late layers receive sufficient image\-derived information through the residual stream, or because CoT prompting front\-loads visual extraction into early generated tokens\. Visual Access Sweep isolates the necessity of one channel \(direct generated\-token\-to\-image\-token attention\), so distinguishing these accounts will require complementary interventions on the residual stream and on the temporal distribution of perceptual extraction\.

## Acknowledgments and Disclosure of Funding

We thank our colleagues for their valuable discussions and feedback throughout the development of this work\.

## References

- Understanding intermediate layers using linear classifier probes\.External Links:1610\.01644,[Document](https://dx.doi.org/10.48550/arXiv.1610.01644),[Link](https://arxiv.org/abs/1610.01644)Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px4.p1.1)\.
- J\. Bai, S\. Bai, S\. Yang, S\. Wang, S\. Tan, P\. Wang, J\. Lin, C\. Zhou, and J\. Zhou \(2024\)Qwen\-VL: a versatile vision\-language model for understanding, localization, text reading, and beyond\.External Links:[Link](https://openreview.net/forum?id=qrGjFJVl3m)Cited by:[§3\.1](https://arxiv.org/html/2607.12815#S3.SS1.p1.7)\.
- S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. Lin \(2025\)Qwen2\.5\-VL technical report\.arXiv preprint arXiv:2502\.13923\.External Links:2502\.13923,[Link](https://arxiv.org/abs/2502.13923),[Document](https://dx.doi.org/10.48550/arXiv.2502.13923)Cited by:[Table 9](https://arxiv.org/html/2607.12815#A3.T9.4.2.1.2.1.2.1),[§3\.1](https://arxiv.org/html/2607.12815#S3.SS1.p1.7),[§5\.3](https://arxiv.org/html/2607.12815#S5.SS3.p1.1)\.
- N\. Belrose, Z\. Furman, L\. Smith, D\. Halawi, I\. Ostrovsky, L\. McKinney, S\. Biderman, and J\. Steinhardt \(2023\)Eliciting latent predictions from transformers with the tuned lens\.External Links:2303\.08112,[Link](https://arxiv.org/abs/2303.08112)Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px4.p1.1)\.
- S\. Fu, T\. Bonnen, D\. Guillory, and T\. Darrell \(2025\)Hidden in plain sight: VLMs overlook their visual representations\.External Links:2506\.08008,[Link](https://arxiv.org/abs/2506.08008)Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px4.p1.1)\.
- Y\. Gandelsman, A\. A\. Efros, and J\. Steinhardt \(2024\)Interpreting CLIP’s image representation via text\-based decomposition\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px4.p1.1)\.
- A\. Geiger, H\. Lu, T\. Icard, and C\. Potts \(2021\)Causal abstractions of neural networks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.34,pp\. 9574–9586\.Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px4.p1.1)\.
- M\. Geva, J\. Bastings, K\. Filippova, and A\. Globerson \(2023\)Dissecting recall of factual associations in auto\-regressive language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px4.p1.1)\.
- A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.External Links:[Link](https://arxiv.org/abs/2407.21783)Cited by:[Table 9](https://arxiv.org/html/2607.12815#A3.T9.4.4.1.2.1.2.1),[§5\.3](https://arxiv.org/html/2607.12815#S5.SS3.SSS0.Px1.p1.1)\.
- D\. A\. Hudson and C\. D\. Manning \(2019\)GQA: a new dataset for real\-world visual reasoning and compositional question answering\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p4.1),[§3\.2](https://arxiv.org/html/2607.12815#S3.SS2.SSS0.Px2.p1.1)\.
- J\. Johnson, B\. Hariharan, L\. van der Maaten, J\. Hoffman, L\. Fei\-Fei, C\. L\. Zitnick, and R\. Girshick \(2017\)CLEVR: a diagnostic dataset for compositional language and elementary visual reasoning\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:1612\.06890,[Link](https://arxiv.org/abs/1612.06890)Cited by:[§5\.4](https://arxiv.org/html/2607.12815#S5.SS4.p1.2)\.
- S\. S\. Kancheti, A\. S\. Kanade, V\. N\. Balasubramanian, and T\. Ganu \(2026\)Chain\-of\-thought degrades visual spatial reasoning capabilities of multimodal LLMs\.arXiv preprint arXiv:2604\.16060\.External Links:[Link](https://arxiv.org/abs/2604.16060)Cited by:[§5\.4](https://arxiv.org/html/2607.12815#S5.SS4.p2.1)\.
- S\. Kang, J\. Kim, J\. Kim, and S\. J\. Hwang \(2025\)See what you are told: visual attention sink in large multimodal models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=7uDI7w5RQA)Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p2.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px1.p1.1)\.
- T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. Iwasawa \(2022\)Large language models are zero\-shot reasoners\.Advances in neural information processing systems35,pp\. 22199–22213\.Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p1.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Lanham, A\. Chen, A\. Radhakrishnan, B\. Steiner, C\. Denison, D\. Hernandez, D\. Li, E\. Durmus, E\. Hubinger, J\. Kernion, K\. Lukošiūtė, K\. Nguyen, N\. Cheng, N\. Joseph, N\. Schiefer, O\. Rausch, R\. Larson, S\. McCandlish, S\. Kundu, S\. Kadavath, S\. Yang, T\. Henighan, T\. Maxwell, T\. Telleen\-Lawton, T\. Hume, Z\. Hatfield\-Dodds, J\. Kaplan, J\. Brauner, S\. R\. Bowman, and E\. Perez \(2023\)Measuring faithfulness in chain\-of\-thought reasoning\.External Links:2307\.13702,[Link](https://arxiv.org/abs/2307.13702)Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Lin, A\. Wang, M\. Chen, J\. Liu, H\. Liu, J\. Su, and X\. Xiao \(2025\)Investigating inference\-time scaling for chain of multi\-modal thought: a preliminary study\.External Links:2502\.11514,[Link](https://arxiv.org/abs/2502.11514)Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p1.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Liu, C\. Li, Y\. Li, and Y\. J\. Lee \(2024a\)Improved baselines with visual instruction tuning\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§3\.1](https://arxiv.org/html/2607.12815#S3.SS1.p1.7)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§3\.1](https://arxiv.org/html/2607.12815#S3.SS1.p1.7)\.
- R\. Liu, S\. Bai, N\. Saunshi, and S\. Kakade \(2024b\)Chain\-of\-thought can reduce performance on tasks where thinking makes humans worse\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=rpbzBXdo4x)Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p1.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Liu, Z\. Li, Z\. Fang, N\. Xu, R\. He, and T\. Tan \(2025a\)Rethinking the role of prompting strategies in LLM test\-time scaling: a perspective of probability theory\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:2505\.10981,[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1356),[Link](https://arxiv.org/abs/2505.10981)Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Liu, Z\. Chen, H\. Liu, C\. Luo, X\. Tang, S\. Wang, J\. Zeng, Z\. Dai, Z\. Shi, T\. Wei, B\. Dumoulin, and H\. Tong \(2025b\)Seeing but not believing: probing the disconnect between visual attention and answer correctness in VLMs\.External Links:2510\.17771,[Link](https://arxiv.org/abs/2510.17771)Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px4.p1.1)\.
- P\. Lu, H\. Banber, L\. Xia, J\. Zhu, C\. Li, H\. Hajishirzi, H\. Wang, Y\. Choi, and F\. Poursabzi\-Sangdeh \(2024\)MathVista: evaluating mathematical reasoning of foundation models in visual contexts\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p1.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px2.p1.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022\)Locating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems,Note:NeurIPS 2022External Links:[Link](https://arxiv.org/abs/2202.05262)Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px4.p1.1)\.
- P\. Rahmanzadehgervi, L\. Bolton, M\. R\. Taesiri, and A\. T\. Nguyen \(2024\)Vision language models are blind\.InProceedings of the Asian Conference on Computer Vision \(ACCV\),Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p2.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Schwettmann, N\. Chowdhury, and A\. Torralba \(2023\)Multimodal neurons in pretrained text\-only transformers\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 2859–2869\.Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px4.p1.1)\.
- X\. Tian, S\. Zou, Z\. Yang, M\. He, F\. Waschkowski, L\. Wesemann, P\. Tu, and J\. Zhang \(2026\)More thought, less accuracy? on the dual nature of reasoning in vision\-language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2509.25848)Cited by:[§5\.4](https://arxiv.org/html/2607.12815#S5.SS4.p2.1)\.
- S\. Tong, Z\. Liu, Y\. Zhai, Y\. Ma, Y\. LeCun, and S\. Xie \(2024\)Eyes wide shut? exploring the visual shortcomings of multimodal LLMs\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 9568–9578\.Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p2.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Turpin, J\. Michael, E\. Perez, and S\. R\. Bowman \(2023\)Language models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.Note:NeurIPS 2023External Links:2305\.04388,[Link](https://arxiv.org/abs/2305.04388)Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)Cited by:[§3\.1](https://arxiv.org/html/2607.12815#S3.SS1.p1.7)\.
- J\. Vig, S\. Gehrmann, Y\. Belinkov, S\. Qian, D\. Nevo, Y\. Singer, and S\. Shieber \(2020\)Investigating gender bias in language models using causal mediation analysis\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.33,pp\. 12388–12401\.Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px4.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, and D\. Zhou \(2023\)Self\-consistency improves chain of thought reasoning in language models\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p1.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p1.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px2.p1.1)\.
- Q\. Wu, X\. Yang, Y\. Zhou, C\. Fang, B\. Song, X\. Sun, and R\. Ji \(2025\)Grounded chain\-of\-thought for multimodal large language models\.External Links:2503\.12799,[Link](https://arxiv.org/abs/2503.12799)Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px2.p1.1)\.
- G\. Xu, P\. Jin, Z\. Wu, H\. Li, Y\. Song, L\. Sun, and L\. Yuan \(2025\)LLaVA\-CoT: let vision language models reason step\-by\-step\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),pp\. 2087–2098\.Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p1.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Yang, C\. Cui, Y\. Zhou, Y\. Chen, P\. Xia, Y\. Wei, T\. Yu, Y\. Huang, and L\. Wang \(2025a\)IKOD: mitigating visual attention degradation in large vision\-language models\.External Links:2508\.03469,[Link](https://arxiv.org/abs/2508.03469)Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p2.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Yang, Y\. Niu, Y\. Liu, Y\. Ye, B\. Lin, and L\. Yuan \(2025b\)Look\-back: implicit visual re\-focusing in MLLM reasoning\.External Links:2507\.03019,[Link](https://arxiv.org/abs/2507.03019)Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px3.p1.1),[§7](https://arxiv.org/html/2607.12815#S7.SS0.SSS0.Px2.p1.1)\.
- R\. Yu, X\. Ma, and X\. Wang \(2025\)Introducing visual perception token into multimodal large language model\.External Links:2502\.17425,[Link](https://arxiv.org/abs/2502.17425)Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px3.p1.1),[§7](https://arxiv.org/html/2607.12815#S7.SS0.SSS0.Px2.p1.1)\.
- Q\. Yuan, Q\. Zhang, Y\. Liu, J\. Chen, Y\. Lu, H\. Lin, J\. Zheng, X\. Han, and L\. Sun \(2025\)ShortV: efficient multimodal large language models by freezing visual tokens in ineffective layers\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:2504\.00502,[Link](https://arxiv.org/abs/2504.00502)Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p2.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Zhang, A\. Zhang, M\. Li, H\. Zhao, G\. Karypis, and A\. Smola \(2024\)Multimodal chain\-of\-thought reasoning in language models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2607.12815#S1.p1.1),[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Zhu, W\. Wang, Z\. Chen, Z\. Liu, S\. Ye, L\. Gu, H\. Tian, Y\. Duan, W\. Su, J\. Shao, Z\. Gao, E\. Cui, X\. Wang, Y\. Cao, Y\. Liu, X\. Wei, H\. Zhang, H\. Wang, W\. Xu, H\. Li, J\. Wang, N\. Deng, S\. Li, Y\. He, T\. Jiang, J\. Luo, Y\. Wang, C\. He, B\. Shi, X\. Zhang, W\. Shao, J\. He, Y\. Xiong, W\. Qu, P\. Sun, P\. Jiao, H\. Lv, L\. Wu, K\. Zhang, H\. Deng, J\. Ge, K\. Chen, L\. Wang, M\. Dou, L\. Lu, X\. Zhu, T\. Lu, D\. Lin, Y\. Qiao, J\. Dai, and W\. Wang \(2025\)InternVL3: exploring advanced training and test\-time recipes for open\-source multimodal models\.External Links:2504\.10479,[Link](https://arxiv.org/abs/2504.10479)Cited by:[Table 9](https://arxiv.org/html/2607.12815#A3.T9.4.3.1.2.1.2.1),[§5\.3](https://arxiv.org/html/2607.12815#S5.SS3.p1.1)\.
- X\. Zou, Y\. Wang, Y\. Yan, Y\. Lyu, K\. Zheng, S\. Huang, J\. Chen, P\. Jiang, J\. Liu, C\. Tang, and X\. Hu \(2025\)Look twice before you answer: memory\-space visual retracing for hallucination mitigation in multimodal large language models\.External Links:2410\.03577,[Link](https://arxiv.org/abs/2410.03577)Cited by:[§2](https://arxiv.org/html/2607.12815#S2.SS0.SSS0.Px3.p1.1),[§7](https://arxiv.org/html/2607.12815#S7.SS0.SSS0.Px2.p1.1)\.

###### Appendix Contents

1. [1Introduction](https://arxiv.org/html/2607.12815#S1)
2. [2Related Work](https://arxiv.org/html/2607.12815#S2)
3. [3Experimental Design](https://arxiv.org/html/2607.12815#S3)1. [3\.1VLM formulation](https://arxiv.org/html/2607.12815#S3.SS1) 2. [3\.2Tasks and datasets](https://arxiv.org/html/2607.12815#S3.SS2)
4. [4Visual Access Sweep](https://arxiv.org/html/2607.12815#S4)1. [4\.1Intervention](https://arxiv.org/html/2607.12815#S4.SS1) 2. [4\.2Visual Access Boundary](https://arxiv.org/html/2607.12815#S4.SS2)
5. [5Visual Access Boundary in Direct and CoT Inference](https://arxiv.org/html/2607.12815#S5)1. [5\.1Visual Access Boundary in direct inference](https://arxiv.org/html/2607.12815#S5.SS1) 2. [5\.2Visual Access Boundary in CoT inference](https://arxiv.org/html/2607.12815#S5.SS2) 3. [5\.3Cross\-family and cross\-scale VAB sweeps](https://arxiv.org/html/2607.12815#S5.SS3) 4. [5\.4Real\-image scene\-graph QA extension](https://arxiv.org/html/2607.12815#S5.SS4)
6. [6Perceptual Readout Restricts CoT Gains](https://arxiv.org/html/2607.12815#S6)1. [6\.1Defining perceptual readout](https://arxiv.org/html/2607.12815#S6.SS1) 2. [6\.2Perceptual readout and CoT gain correlation](https://arxiv.org/html/2607.12815#S6.SS2) 3. [6\.3Symbolic\-attribute oracle bypass](https://arxiv.org/html/2607.12815#S6.SS3)
7. [7Discussion](https://arxiv.org/html/2607.12815#S7)
8. [References](https://arxiv.org/html/2607.12815#bib)
9. [AExperimental Details](https://arxiv.org/html/2607.12815#A1)1. [A\.1Dataset Generation](https://arxiv.org/html/2607.12815#A1.SS1) 2. [A\.2Task Examples](https://arxiv.org/html/2607.12815#A1.SS2) 3. [A\.3Prompting, decoding, and answer extraction](https://arxiv.org/html/2607.12815#A1.SS3) 4. [A\.4Compute Resources](https://arxiv.org/html/2607.12815#A1.SS4)
10. [BVisual Access Sweep: Methodology Details](https://arxiv.org/html/2607.12815#A2)1. [B\.1Control Experiments: Ruling Out Intervention Artifacts](https://arxiv.org/html/2607.12815#A2.SS1) 2. [B\.2VAB Sensitivity to the Tolerance Parameterϵ\\epsilon](https://arxiv.org/html/2607.12815#A2.SS2)
11. [CVisual Access Boundary: Detailed Results](https://arxiv.org/html/2607.12815#A3)1. [C\.1Boundary Summary Tables](https://arxiv.org/html/2607.12815#A3.SS1) 2. [C\.2Model Architectures Compared](https://arxiv.org/html/2607.12815#A3.SS2) 3. [C\.3Qwen2\.5\-VL Family Sweeps](https://arxiv.org/html/2607.12815#A3.SS3) 4. [C\.4InternVL3 Family Sweeps](https://arxiv.org/html/2607.12815#A3.SS4) 5. [C\.5Cross\-Attention Fusion Pilot](https://arxiv.org/html/2607.12815#A3.SS5) 6. [C\.6GQA Real\-Image Extension](https://arxiv.org/html/2607.12815#A3.SS6) 7. [C\.7Limiting Case: Angle Counting](https://arxiv.org/html/2607.12815#A3.SS7) 8. [C\.8Qualitative Error Breakdown](https://arxiv.org/html/2607.12815#A3.SS8)
12. [DPerceptual Readout: Detailed Results](https://arxiv.org/html/2607.12815#A4)1. [D\.1Probe\-vs\-Decode Gap](https://arxiv.org/html/2607.12815#A4.SS1) 2. [D\.2Per\-Attribute CoT Gain](https://arxiv.org/html/2607.12815#A4.SS2)
13. [EPractical Implication: Selective CoT Routing](https://arxiv.org/html/2607.12815#A5)

## Appendix AExperimental Details

### A\.1Dataset Generation

We construct controlled synthetic tasks in which the queried visual attribute varies while the counting operation is held fixed\. Images are 256×\\times256 px RGB generated withmatplotlibunder a fixed random seed\. Tables[1](https://arxiv.org/html/2607.12815#A1.T1)and[2](https://arxiv.org/html/2607.12815#A1.T2)summarize the controlled attributes and dataset sizes\.

Table 1:Controlled visual attributes\. Each attribute has four classes\.AttributeClassesColorred / green / yellow / blueShapestar / circle / triangle / squareSizexs \(18 px\) / s \(24 px\) / l \(30 px\) / xl \(36 px\)Locationfour image quadrantsAngle0∘/ 45∘/ 90∘/ 135∘Table 2:Dataset sizes for the two task families used in the controlled experiments\.TaskImagesObjectsQueriesSingle\-object readout7501one attribute query per imageCounting1506–7five queries per image
### A\.2Task Examples

Figure[7](https://arxiv.org/html/2607.12815#A1.F7)shows one multi\-object counting instance with its per\-object attributes and the five corresponding queries\.

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/example_scene.png)

IDColorShapeSizeLocationAngle0redtriangleslower\_right45∘1yellowsquaresupper\_right90∘2yellowsquareslower\_left0∘3yellowcirclexsupper\_left45∘4redsquarexslower\_right45∘5greensquaresupper\_right90∘6yellowsquaresupper\_left135∘AttributeQuestion targetCorrect answerColorgreen objectsD: 1Shapesquare objectsC: 5Angle135∘orientationD: 1Locationupper\_right regionC: 2Sizes \(small\) objectsA: 5

Figure 7:Example controlled counting instance\. Left: example image\. Right: per\-object attributes \(top\) and the five corresponding queries \(bottom\)\. The same image is paired with five queries, one per attribute, so the counting operation is fixed while the queried attribute changes\. Option keys are randomly permuted per query\.
### A\.3Prompting, decoding, and answer extraction

We use a unified prompt template across all models with greedy decoding on frozen public checkpoints\. Table[3](https://arxiv.org/html/2607.12815#A1.T3)summarizes the prompt instructions and answer\-extraction rules across settings, and Table[4](https://arxiv.org/html/2607.12815#A1.T4)lists the verbatim prompt strings\. For Direct, CoT, and Oracle counting, option keys are randomly permuted per query and extracted from the model output \(no constrained decoding\)\. GQA dataset construction details are in Appendix[C\.6](https://arxiv.org/html/2607.12815#A3.SS6)\.

Table 3:Prompting and answer\-extraction summary across settings\.SettingPrompt instructionAnswer extractionDirect countinganswer with option key onlyoption\-key extraction \(A–E\)CoT countingthink step by step, then answer in bracesfinal braced option key \(A–E\)Single\-object readoutmultiple\-choice attribute question \(Table[4](https://arxiv.org/html/2607.12815#A1.T4)\)option\-key extraction \(A–E\)Multi\-object readoutfill missing target attribute per object, JSON output \(Table[4](https://arxiv.org/html/2607.12815#A1.T4)\)per\-object value parse from JSONGQA Directshort\-answer format with curly\-brace final linelast “yes” / “no” \(yn\_last\)GQA CoTthink step by step, finish with curly\-brace final linelast “yes” / “no” \(yn\_last\)Oracle \(counting\)serialized attributes as text \(minimal or full\)option\-key extraction \(A–E\)Table 4:Verbatim prompt strings used in our experiments\. Mustache\-style placeholders such as\{attribute\_value\}and\{question\}are filled in per query\. Image input is concatenated to the user message via the model’s default chat template\.SettingPrompt bodyCounting / DirectHow many \{attribute\_value\} objects are in the image?
options: \{A: 2, B: 4, C: 1, D: 3, E: 5\}
Answer with the option key only \(no ’:’ or value\)\.
Respond with a single final line in the curly braces format: \{A\}Counting / CoTHow many \{attribute\_value\} objects are in the image?
options: \{A: 2, B: 4, C: 1, D: 3, E: 5\}
Think step by step\. You must write intermediate steps\.
Respond in curly braces format: \{A\}Single\-object readoutWhich color is this object in the image?
options: \{A: blue, B: red, C: green, D: yellow\}
Answer with the option key only \(no ’:’ or value\)\.
Respond with a single final line in the curly braces format: \{A\}Multi\-object readoutIn the image, there are \{N\} objects\.
Fill in the missing color for each object\.
Valid color values: red, green, yellow, blue
Object 0: color=\_\_, shape=triangle, size=l, location=upper\_left, angle=135
Object 1: color=\_\_, shape=circle, size=s, location=lower\_right, angle=0
…
Output in following format:
\{"objects": \[\{"id": 0, "color": "red"\}, \{"id": 1, "color": "green"\}, …\]\}GQA / Direct\{question\}
Answer with a short phrase only\.
Respond with a single final line in the curly braces format:
Answer: \{your short answer\}GQA / CoT\{question\}
Think step by step\. You must write intermediate steps\.
Finish with a single final line in the curly braces format after your reasoning:
Answer: \{your short answer\}Oracle \(minimal\)In the image, there are \{N\} objects\.
Object 0: color=blue
Object 1: color=yellow
…
How many \{attribute\_value\} objects are there in the image?
Answer with the option key only \(no ’:’ or value\)\.
Respond with a single final line in the curly braces format: \{A\}Oracle \(full\)In the image, there are \{N\} objects\.
Object 0: color=blue, shape=star, size=l, location=lower\_right, angle=0
Object 1: color=yellow, shape=circle, size=s, location=upper\_left, angle=135
…
How many \{attribute\_value\} objects are there in the image?
Answer with the option key only \(no ’:’ or value\)\.
Respond with a single final line in the curly braces format: \{A\}
### A\.4Compute Resources

All experiments in this paper are inference\-time interventions on frozen public VLM checkpoints\. No VLM is fine\-tuned\. Only lightweight linear probes are trained for the probe\-vs\-decode analysis \(Appendix[D\.1](https://arxiv.org/html/2607.12815#A4.SS1)\)\. Hardware and runtime are summarized in Table[5](https://arxiv.org/html/2607.12815#A1.T5)\. Aggregate compute across the main results reported in this paper is on the order of∼1,100\\sim 1\{,\}100GPU\-hours, and preliminary or discarded experiments \(alternative sweep grids, additional probe configurations, control runs not reported\) account for a comparable additional amount\.

Table 5:Compute resources used in this paper\. All sweeps were run on3×3\\timesNVIDIA H100 NVL 96 GB nodes; runtime is wall\-clock×\\times3 GPUs\.Experiment familyHardwareApprox\. runtimeSmall\-model sweeps\(Qwen2\.5\-VL\-3B/7B, InternVL3\-8B, Llama\-3\.2\-11B\-Vision\)3×3\\timesH100 NVL 96 GB∼600\\sim 600GPU\-hoursLarge\-model sweeps\(Qwen2\.5\-VL\-32B, InternVL3\-14B/38B\)3×3\\timesH100 NVL 96 GB∼500\\sim 500GPU\-hoursQwen2\.5\-VL\-32B 2D layer\-×\\times\-token sweep3×3\\timesH100 NVL 96 GB45–60 GPU\-hoursPerceptual\-readout analysis\(probe, oracle, controls\)3×3\\timesH100 NVL 96 GB∼21\\sim 21GPU\-hoursTotal reported main results—∼1,100\\sim 1\{,\}100GPU\-hoursPeak GPU memory is bounded by full\-attention forward\-pass memory at the largest scale \(Qwen2\.5\-VL\-32B\) with the image\-token prefix, and fits within3×3\\timesH100 NVL 96 GB with standard tensor\-parallel sharding\.

## Appendix BVisual Access Sweep: Methodology Details

This section reports two intervention controls and anϵ\\epsilon\-sensitivity analysis for Visual Access Sweep\.

### B\.1Control Experiments: Ruling Out Intervention Artifacts

We run two controls targeting possible artifacts of Visual Access Sweep\.

#### Late\-layer query\-text block\.

The first concern is a query\-mediated re\-reading loophole\. Even if generated tokens cannot directly attend to image tokens in late layers, prompt or query\-text tokens might still attend to the image in those layers and pass image\-derived information to generated tokens\. To test this, we apply a stricter intervention that additionally blocks query\-text→\\toimage\-token attention from the same cutoff layer upward\. Importantly, early\-layer query\-text access is left unchanged\. The control only removes the late\-layer path that could bypass the generated\-token intervention\. Figure[8](https://arxiv.org/html/2607.12815#A2.F8)shows that the qualitative VAB pattern is preserved under this stricter intervention\.

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_direct_qwen25_32b_color_ab_control.png)\(a\)Direct layer sweep\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_cot_qwen25_32b_color_ab_control.png)\(b\)CoT layer\-token sweep\.

Figure 8:Late\-layer query\-text block control \(Color Counting, Qwen2\.5\-VL\-32B\)\.In addition to blocking generated\-token→\\toimage\-token attention, this control blocks query\-text→\\toimage\-token attention from the same cutoff layer upward\. Early\-layer query\-text access is left intact\. The qualitative VAB pattern is preserved, weakening the query\-mediated late\-rereading explanation\.
#### Null\-sink control\.

The second concern is an attention\-redistribution artifact\. Setting image\-token logits to−∞\-\\inftychanges the softmax normalization over the remaining real tokens, which could in principle distort the model’s behavior beyond simply removing image access\. To test whether the observed boundary is caused by this redistribution rather than by removing image access, we redirect the removed image\-token attention mass to a dummy non\-visual sink slot\. Figure[9](https://arxiv.org/html/2607.12815#A2.F9)shows that the qualitative VAB pattern is again preserved\.

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_direct_qwen25_32b_color_null_sink.png)\(a\)Direct layer sweep\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_cot_qwen25_32b_color_null_sink.png)\(b\)CoT layer\-token sweep\.

Figure 9:Null\-sink control \(Color Counting, Qwen2\.5\-VL\-32B\)\.Removed image\-token attention mass is redirected to a dummy non\-visual sink instead of being redistributed over real non\-image tokens\. The qualitative VAB pattern is preserved, weakening the attention\-redistribution artifact explanation\.
#### Takeaway\.

Together, these controls support the interpretation that the reported boundary is not primarily caused by either query\-mediated late re\-reading or attention\-mass redistribution\. They do not claim that all visual information flow stops after the boundary\. Rather, they support the narrower claim used throughout the paper\. Additional late direct image\-token access is not functionally necessary in the tested setting\.

### B\.2VAB Sensitivity to the Tolerance Parameterϵ\\epsilon

The VAB definition \(Section[4\.2](https://arxiv.org/html/2607.12815#S4.SS2)\) depends on a toleranceϵ\\epsilon, which we fix toϵ=0\.05\\epsilon=0\.05throughout the main text\. A natural concern is that the CoT\-vs\-Direct boundary comparison in Section[5\.2](https://arxiv.org/html/2607.12815#S5.SS2)could rest on this specific choice ofϵ\\epsilon\. To check this, we recomputeℓ∗\\ell^\{\*\}on Qwen2\.5\-VL\-32B for color and shape counting under both criteria acrossϵ∈\{0\.03,0\.05,0\.07,0\.10\}\\epsilon\\in\\\{0\.03,0\.05,0\.07,0\.10\\\}\(Table[6](https://arxiv.org/html/2607.12815#A2.T6)\)\.

Table 6:VABϵ\\epsilon\-sensitivity on Qwen2\.5\-VL\-32B\.Δ​ℓ∗\\Delta\\ell^\{\*\}across the two criteria and four tolerance values\. “Direct\-anchored” usesΔ​ℓDA∗=ℓDA∗−ℓD∗\\Delta\\ell^\{\*\}\_\{\\mathrm\{DA\}\}=\\ell^\{\*\}\_\{\\mathrm\{DA\}\}\-\\ell^\{\*\}\_\{\\mathrm\{D\}\}\. “CoT\-own\-max” usesΔ​ℓown∗=ℓCoT∗−ℓD∗\\Delta\\ell^\{\*\}\_\{\\mathrm\{own\}\}=\\ell^\{\*\}\_\{\\mathrm\{CoT\}\}\-\\ell^\{\*\}\_\{\\mathrm\{D\}\}\. Shaded columns \(ϵ=0\.05\\epsilon=0\.05\) correspond to the tolerance used in the main text, with values shown in bold\.Direct\-anchored criterionCoT\-own\-max criterionTaskϵ=0\.03\\epsilon\{=\}0\.03ϵ=0\.05\\epsilon\{=\}0\.05ϵ=0\.07\\epsilon\{=\}0\.07ϵ=0\.10\\epsilon\{=\}0\.10ϵ=0\.03\\epsilon\{=\}0\.03ϵ=0\.05\\epsilon\{=\}0\.05ϵ=0\.07\\epsilon\{=\}0\.07ϵ=0\.10\\epsilon\{=\}0\.10colorℓD∗\\ell^\{\*\}\_\{\\mathrm\{D\}\}4443434244434342ℓDA∗\\ell^\{\*\}\_\{\\mathrm\{DA\}\}/ℓCoT∗\\ell^\{\*\}\_\{\\mathrm\{CoT\}\}4343424252525251Δ​ℓDA∗\\Delta\\ell^\{\*\}\_\{\\mathrm\{DA\}\}−1\-1𝟎\\boldsymbol\{0\}−1\-10————Δ​ℓown∗\\Delta\\ell^\{\*\}\_\{\\mathrm\{own\}\}————\+8\+8\+𝟗\\boldsymbol\{\+9\}\+9\+9\+9\+9shapeℓD∗\\ell^\{\*\}\_\{\\mathrm\{D\}\}4242424242424242ℓDA∗\\ell^\{\*\}\_\{\\mathrm\{DA\}\}/ℓCoT∗\\ell^\{\*\}\_\{\\mathrm\{CoT\}\}4242424144434342Δ​ℓDA∗\\Delta\\ell^\{\*\}\_\{\\mathrm\{DA\}\}0𝟎\\boldsymbol\{0\}0−1\-1————Δ​ℓown∗\\Delta\\ell^\{\*\}\_\{\\mathrm\{own\}\}————\+2\+2\+𝟏\\boldsymbol\{\+1\}\+1\+10#### Takeaway\.

Two points are stable across tolerance choices\. First, under the direct\-anchored criterion, the CoT boundary remains close to the Direct boundary across all testedϵ\\epsilonvalues \(\|Δ​ℓDA∗\|≤1\|\\Delta\\ell^\{\*\}\_\{\\mathrm\{DA\}\}\|\\leq 1\), so the main CoT\-Direct comparison is not an artifact of choosingϵ=0\.05\\epsilon=0\.05\. Second, under the CoT\-own\-max criterion the boundary can be deeper because the target accuracy is higher\. This reflects the stricter accuracy target rather than a proportional expansion of direct image\-token access with generation length\.

## Appendix CVisual Access Boundary: Detailed Results

This section reports per\-family, per\-scale, and per\-task VAB results supporting Section[5](https://arxiv.org/html/2607.12815#S5): boundary summary tables, model architectures compared, family\-level sweeps for Qwen2\.5\-VL and InternVL3, a cross\-attention fusion pilot, the GQA real\-image extension, an angle\-attribute limiting case, and a qualitative error breakdown\.

### C\.1Boundary Summary Tables

Tables[7](https://arxiv.org/html/2607.12815#A3.T7)and[8](https://arxiv.org/html/2607.12815#A3.T8)report the per\-setting VAB layers used in Section[5](https://arxiv.org/html/2607.12815#S5)\. The CoT\-own\-max criterion asks when CoT preserves its own full\-access accuracy, whereas the direct\-anchored criterion asks when CoT reaches Direct’s full\-access accuracy\. We denote the corresponding CoT\-side boundaries byℓCoT∗\\ell^\{\*\}\_\{\\mathrm\{CoT\}\}andℓDA∗\\ell^\{\*\}\_\{\\mathrm\{DA\}\}, respectively, and report shifts relative to the Direct boundaryℓD∗\\ell^\{\*\}\_\{\\mathrm\{D\}\}following Eq\. \([3](https://arxiv.org/html/2607.12815#S4.E3)\)\. We report both criteria because they answer different questions\. The former measures the access region needed for CoT’s own ceiling, while the latter controls for Direct–CoT ceiling differences\. N/A in Table[8](https://arxiv.org/html/2607.12815#A3.T8)indicates that CoT never reaches the direct\-anchored target in that setting, so the boundary is undefined under this criterion\.

Table 7:CoT\-own\-max Visual Access Boundary across families and scales \(ϵ=0\.05\\epsilon=0\.05\)\.Color CountingShape CountingModel𝑳\\boldsymbol\{L\}ℓ𝐃∗\\boldsymbol\{\\ell^\{\*\}\_\{\\mathrm\{D\}\}\}ℓ𝐂𝐨𝐓∗\\boldsymbol\{\\ell^\{\*\}\_\{\\mathrm\{CoT\}\}\}𝚫​ℓ𝐨𝐰𝐧∗\\boldsymbol\{\\Delta\\ell^\{\*\}\_\{\\mathrm\{own\}\}\}𝔼​\[𝑻gen\]\\mathbb\{E\}\[\\boldsymbol\{T\_\{\\text\{gen\}\}\}\]ℓ𝐃∗\\boldsymbol\{\\ell^\{\*\}\_\{\\mathrm\{D\}\}\}ℓ𝐂𝐨𝐓∗\\boldsymbol\{\\ell^\{\*\}\_\{\\mathrm\{CoT\}\}\}𝚫​ℓ𝐨𝐰𝐧∗\\boldsymbol\{\\Delta\\ell^\{\*\}\_\{\\mathrm\{own\}\}\}𝔼​\[𝑻gen\]\\mathbb\{E\}\[\\boldsymbol\{T\_\{\\text\{gen\}\}\}\]Qwen2\.5\-VL\-3B362534\+9\+9128±32128\\pm 322229\+7\+7142±29142\\pm 29Qwen2\.5\-VL\-7B282021\+1\+185±3385\\pm 332019−1\-176±1876\\pm 18Qwen2\.5\-VL\-32B644352\+9\+9183±33183\\pm 334244\+2\+2143±30143\\pm 30InternVL3\-8B282026\+6\+6105±20105\\pm 202023\+3\+3103±27103\\pm 27InternVL3\-14B483636075±1775\\pm 173537\+2\+286±1786\\pm 17InternVL3\-38B644544−1\-166±1166\\pm 114647\+1\+192±2192\\pm 21Table 8:Direct\-anchored Visual Access Boundary across families and scales \(ϵ=0\.05\\epsilon=0\.05\)\. N/A indicates that the direct\-anchored target is not reached\.Color CountingShape CountingModel𝑳\\boldsymbol\{L\}ℓ𝐃∗\\boldsymbol\{\\ell^\{\*\}\_\{\\mathrm\{D\}\}\}ℓ𝐃𝐀∗\\boldsymbol\{\\ell^\{\*\}\_\{\\mathrm\{DA\}\}\}𝚫​ℓ𝐃𝐀∗\\boldsymbol\{\\Delta\\ell^\{\*\}\_\{\\mathrm\{DA\}\}\}ℓ𝐃∗\\boldsymbol\{\\ell^\{\*\}\_\{\\mathrm\{D\}\}\}ℓ𝐃𝐀∗\\boldsymbol\{\\ell^\{\*\}\_\{\\mathrm\{DA\}\}\}𝚫​ℓ𝐃𝐀∗\\boldsymbol\{\\Delta\\ell^\{\*\}\_\{\\mathrm\{DA\}\}\}Qwen2\.5\-VL\-3B362533\+8\+822N/AN/AQwen2\.5\-VL\-7B2820200205−15\-15Qwen2\.5\-VL\-32B644343042420InternVL3\-8B2820N/AN/A20N/AN/AInternVL3\-14B483634−2\-23537\+2\+2InternVL3\-38B644546\+1\+14647\+1\+1
### C\.2Model Architectures Compared

Table 9:Architectural summary of VLMs evaluated by Visual Access Sweep\. “Fusion” indicates how the visual stream is exposed to the language decoder\.FamilyScaleDecoder layersVision encoder / connectorFusionQwen2\.5\-VL\[Baiet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib6)\]3B / 7B / 32B36 / 28 / 64Qwen ViT \+ MLPPrefixInternVL3\[Zhuet al\.,[2025](https://arxiv.org/html/2607.12815#bib.bib39)\]8B / 14B / 38B28 / 48 / 64InternViT \+ MLPPrefixLlama\-3\.2\-Vision\-Instruct\[Grattafioriet al\.,[2024](https://arxiv.org/html/2607.12815#bib.bib14)\]11B40CLIP ViT \+ cross\-attnCross\-attention
### C\.3Qwen2\.5\-VL Family Sweeps

Figure[10](https://arxiv.org/html/2607.12815#A3.F10)shows per\-scale Direct VAB curves and CoT VAB heatmaps for the Qwen2\.5\-VL family\.

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_direct_qwen25_3b.png)\(a\)3B Direct\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_cot_qwen25_3b.png)\(b\)3B CoT\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_direct_qwen25_7b.png)\(c\)7B Direct\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_cot_qwen25_7b.png)\(d\)7B CoT\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_direct_qwen25_32b_appendix-fixed.png)\(e\)32B Direct\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_cot_qwen25_32b_appendix-fixed.png)\(f\)32B CoT\.

Figure 10:Qwen2\.5\-VL family sweeps\. Rows correspond to model scale; left panels show Direct layer sweeps and right panels show CoT layer\-token sweeps\. Exact boundary values are reported in Tables[7](https://arxiv.org/html/2607.12815#A3.T7)and[8](https://arxiv.org/html/2607.12815#A3.T8)\.
### C\.4InternVL3 Family Sweeps

To assess whether the VAB phenomenon generalizes across model families, we additionally evaluated InternVL3 models \(8B, 14B, 38B\)\. These models use a different vision encoder and projection architecture compared to Qwen2\.5\-VL\.

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_direct_internvl3_8b-fixed.png)\(a\)8B Direct\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_cot_internvl3_8b-fixed.png)\(b\)8B CoT\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_direct_internvl3_14b-fixed.png)\(c\)14B Direct\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_cot_internvl3_14b-fixed.png)\(d\)14B CoT\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_direct_internvl3_38b-fixed.png)\(e\)38B Direct\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_cot_internvl3_38b-fixed.png)\(f\)38B CoT\.

Figure 11:InternVL3 family sweeps\. Rows correspond to model scale; left panels show Direct layer sweeps and right panels show CoT layer\-token sweeps\. Exact boundary values are reported in Tables[7](https://arxiv.org/html/2607.12815#A3.T7)and[8](https://arxiv.org/html/2607.12815#A3.T8)\.
### C\.5Cross\-Attention Fusion Pilot

The main experiments use prefix\-fusion VLMs, where image tokens are concatenated with text tokens and direct visual access is realized through self\-attention from generated\-token queries to image\-token keys\. As a pilot check, we apply the same intervention principle to Llama\-3\.2\-11B\-Vision\-Instruct, where visual features are exposed through cross\-attention modules rather than through an image\-token prefix\. In this case, the sweep masks generated\-token queries’ cross\-attention to visual keys over layer depth and generation time\.

We report the CoT\-side sweep only\. Direct prompting on this model did not reliably follow the option\-key answer format under our extraction protocol, so the Direct layer sweep was not a clean comparison\. This result should therefore be read as an intervention\-transfer check for cross\-attention fusion, not as a full family\-level comparison\.

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/cot_vab_llama11b_crossattn.png)Figure 12:CoT\-side Visual Access Sweep on a cross\-attention fusion VLM \(Llama\-3\.2\-11B\-Vision\-Instruct, color counting,ϵ=0\.05\\epsilon=0\.05\)\. The intervention masks generated\-token cross\-attention to visual keys\. Reported as a pilot check that the sweep can be applied beyond prefix\-fusion models\.
### C\.6GQA Real\-Image Extension

Table[10](https://arxiv.org/html/2607.12815#A3.T10)summarizes the construction and evaluation protocol for the GQA yes/no extension\. This experiment is used as a real\-image check of the VAB structure, not as a claim that CoT improves GQA accuracy\. We extract the final occurrence of\\b\(yes\|no\)\\b\(case\-insensitive\) as the predicted label\.

Table 10:GQA yes/no extension setup and boundary summary\.ItemValueSource splitval\_balancedFilteranswer∈\\in\{yes, no\} \(case\-insensitive\)Sample size100, shuffled with seed0Label balance53 no / 47 yes \(chance level0\.50\.5\)ModelQwen2\.5\-VL\-32BImage preprocessingRGB conversion, resize to model native inputExtraction rulelast yes/no token \(yn\_last\)Direct full accuracy /ℓ∗\\ell^\{\*\}0\.840\.84/4040\(N∗=24N^\{\*\}=24\)CoT full accuracy /\(ℓ∗,τ∗\)\(\\ell^\{\*\},\\tau^\{\*\}\)0\.800\.80/\(28,≈32\)\(28,\\,\\approx\\\!32\)A finite VAB appears in both Direct and CoT sweeps, even though CoT does not improve over Direct on this binary\-answer subset\. This supports the main\-text claim that the VAB phenomenon extends beyond synthetic stimuli \(Section[5\.4](https://arxiv.org/html/2607.12815#S5.SS4)\), independent of whether CoT itself improves task accuracy\.

### C\.7Limiting Case: Angle Counting

Figure[13](https://arxiv.org/html/2607.12815#A3.F13)shows angle counting on Qwen2\.5\-VL\-32B\. We include this as a limiting case rather than as main VAB evidence\. When full\-access accuracy is near the task floor, the sweep cannot cleanly localize a boundary, because preserving low accuracy is not informative about successful visual grounding\. This case supports the interpretation used in Section[6](https://arxiv.org/html/2607.12815#S6): extended reasoning cannot compensate when the queried visual attribute is not reliably read out into a usable symbol\.

![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_direct_qwen25_32b_angle.png)\(a\)Direct layer sweep\.
![Refer to caption](https://arxiv.org/html/2607.12815v1/figures/vab_cot_qwen25_32b_angle.png)\(b\)CoT layer\-token sweep\.

Figure 13:Angle counting as a limiting case \(Qwen2\.5\-VL\-32B\)\. Full\-access performance is near the task floor, so the sweep does not yield an interpretable boundary in the same sense as color or shape counting\.
### C\.8Qualitative Error Breakdown

Table[11](https://arxiv.org/html/2607.12815#A3.T11)provides a small qualitative breakdown of representative errors\. We use it only to characterize failure modes, not as a main quantitative result\. Errors are grouped into four categories:*alt\-attribute\-consistent*, where the predicted count matches the count of a non\-target attribute value in the same scene;*tally\-inconsistent*, where the target attribute is identified but the written count disagrees with the visible tally;*format*errors, where outputs fail extraction \(no valid option key or yes/no token\); and*other*errors that match none of the above\.

#### Takeaway\.

When direct visual access is blocked before the VAB, errors are mostly alt\-attribute\-consistent, suggesting that the intervention corrupts which visual attribute or value is available for counting\. Under full\-access CoT, the remaining errors shift toward tally inconsistency, suggesting that once the visual attribute is available, residual failures more often occur in the symbolic counting step\.

Table 11:Qualitative error breakdown over incorrect examples only\. Categories follow the definitions above\.TaskConditionAccAlt\-attr\. cons\.Tally incon\.OtherFormatcolorFull access \(Direct\)0\.8020000colorBeyond VAB \(ℓ=36\\ell\{=\}36\)0\.3565000colorStrong block \(ℓ=16\\ell\{=\}16\)0\.3565000colorCoT full access0\.950410shapeFull access \(Direct\)0\.8416000shapeBeyond VAB \(ℓ=34\\ell\{=\}34\)0\.4951000shapeStrong block \(ℓ=16\\ell\{=\}16\)0\.2080000shapeCoT full access0\.890920angleFull access \(Direct\)0\.2277001angleCoT full access0\.3706102

## Appendix DPerceptual Readout: Detailed Results

This section reports two pieces of supporting evidence for Section[6](https://arxiv.org/html/2607.12815#S6): a probe\-vs\-decode comparison that separates latent visual grounding from decoded readout, and a per\-attribute breakdown of the readout–CoT\-gain relationship underlying Figure[5](https://arxiv.org/html/2607.12815#S6.F5)\.

### D\.1Probe\-vs\-Decode Gap

This appendix uses a single\-object setting only for the probe\-vs\-decode diagnostic\. This is separate from the multi\-object perceptual readout accuracy used as the Figure[5](https://arxiv.org/html/2607.12815#S6.F5)predictor in Section[6](https://arxiv.org/html/2607.12815#S6)\. The goal here is to test whether attribute information is present in hidden states even when the model’s own decoded answer is unreliable\. Table[12](https://arxiv.org/html/2607.12815#A4.T12)reports this comparison on Qwen2\.5\-VL\-32B\. An attribute can be linearly recoverable from hidden states \(Probing Acc\) while still being unavailable to the model as a decoded output symbol \(Decoding Acc\)\.

Table 12:Probe\-vs\-decode comparison on Qwen2\.5\-VL\-32B\. Probing Acc measures linear recoverability from hidden states; Decoding Acc measures whether the model can output the attribute value itself\. Gap \(highlighted\) is Probe minus Decode\.AttributeProbing AccDecoding AccGap \(Probe−\-Decode\)Color1\.0001\.0000\.000Shape1\.0001\.0000\.000Angle0\.9540\.383\+\+0\.571Location0\.9710\.312\+\+0\.658Size0\.8920\.283\+\+0\.608Table 13:Linear\-probe protocol used in Table[12](https://arxiv.org/html/2607.12815#A4.T12)\.ItemValueModelQwen2\.5\-VL\-32BLayerFinal decoder layer \(after the last Transformer block\)Token positionFinal image\-token positionData500 single\-object images, 400 / 100 train\-validation splitProbeℓ2\\ell\_\{2\}\-regularized multinomial logistic regression \(λ=1\.0\\lambda=1\.0, scikit\-learn\)EvaluationHeld\-out accuracy on the 100\-image validation split
### D\.2Per\-Attribute CoT Gain

Table[14](https://arxiv.org/html/2607.12815#A4.T14)reports the per\-attribute values underlying Figure[5](https://arxiv.org/html/2607.12815#S6.F5)\. Each block corresponds to one Qwen2\.5\-VL scale\. Within each block, attributes are ordered by multi\-object perceptual readout accuracy\. Pearsonrrand Spearmanρ\\rhoare descriptive summaries over five attributes per scale, not independent evidence of a universal scaling law\.

#### Takeaway\.

Across Qwen2\.5\-VL scales, attributes with stronger multi\-object readout generally show larger CoT gains, while attributes with weak readout show small or negative gains\. Together with the oracle result in Figure[6](https://arxiv.org/html/2607.12815#S6.F6), this supports the interpretation in Section[6](https://arxiv.org/html/2607.12815#S6): CoT gain is constrained by whether the queried visual attribute is available as a usable symbol\.

Table 14:Per\-attribute readout and CoT\-gain values underlying Figure[5](https://arxiv.org/html/2607.12815#S6.F5)\.Δ\\DeltaCoT is CoT minus Direct; shading indicates the sign ofΔ\\DeltaCoT \(green positive, red negative\)\. Correlations are computed over the five attributes within each scale\.AttributeReadoutDirectCoT𝚫\\boldsymbol\{\\Delta\}CoTQwen2\.5\-VL\-3Bcolor0\.7130\.5800\.830\+\+0\.250shape0\.6050\.6100\.770\+\+0\.160location0\.6010\.3600\.470\+\+0\.110size0\.3470\.1400\.190\+\+0\.050angle0\.2450\.2900\.320\+\+0\.030Pearsonr=\+0\.92r=\+0\.92, Spearmanρ=\+1\.00\\rho=\+1\.00Qwen2\.5\-VL\-7Bcolor0\.4980\.7900\.890\+\+0\.100shape0\.3540\.7700\.850\+\+0\.080location0\.2220\.3700\.420\+\+0\.050size0\.1310\.3100\.340\+\+0\.030angle0\.0050\.4400\.320−\-0\.120Pearsonr=\+0\.88r=\+0\.88, Spearmanρ=\+1\.00\\rho=\+1\.00Qwen2\.5\-VL\-32Bcolor0\.9200\.8400\.950\+\+0\.110shape0\.8650\.8500\.890\+\+0\.040location0\.7710\.4100\.570\+\+0\.160size0\.4790\.2000\.270\+\+0\.070angle0\.2540\.3700\.400\+\+0\.030Pearsonr=\+0\.51r=\+0\.51, Spearmanρ=\+0\.50\\rho=\+0\.50

## Appendix EPractical Implication: Selective CoT Routing

The perceptual\-readout analysis suggests a simple routing heuristic, namely using CoT only when the queried attribute can be reliably read out, and otherwise falling back to Direct inference\. For each queried attribute, we route to CoT if the corresponding multi\-object readout accuracy on the readout validation split exceeds0\.50\.5\. This threshold is not tuned for optimal deployment\. It is used as a simple illustrative rule\. Table[15](https://arxiv.org/html/2607.12815#A5.T15)shows that this rule preserves most of the accuracy gain of always using CoT while substantially reducing output\-token cost\.

Table 15:Selective CoT routing as an efficiency\-oriented heuristic\. The adaptive policy uses CoT only when the queried attribute’s multi\-object readout accuracy exceeds0\.50\.5\.*Gain recovered*is the fraction of the Always\-Direct→\\toAlways\-CoT improvement preserved by the adaptive policy\.*Token cost vs CoT*is average output tokens normalized by the Always\-CoT row\. The adaptive row is highlighted\.ModelPolicyAcc\.Gain vs DirectGain recoveredAvg\. tokensTokens vs CoTQwen2\.5\-VL\-3BAlways Direct0\.396——2\.61\.6%Qwen2\.5\-VL\-3BAlways CoT0\.516\+\+0\.120100%166\.7100%Qwen2\.5\-VL\-3BAdaptive \(thr==0\.5\)0\.500\+\+0\.10486\.7%82\.749\.6%Qwen2\.5\-VL\-32BAlways Direct0\.534——3\.01\.3%Qwen2\.5\-VL\-32BAlways CoT0\.616\+\+0\.082100%224\.3100%Qwen2\.5\-VL\-32BAdaptive \(thr==0\.5\)0\.596\+\+0\.06275\.6%102\.945\.9%#### Takeaway\.

A cheap attribute\-level readout check can recover most of the benefit of always using CoT while avoiding roughly half of the output\-token cost in these two representative Qwen2\.5\-VL settings\.