Diagnosing Failure Modes of Shared-State Collaboration in Resource-Constrained Visual Agents

# Diagnosing Failure Modes of Shared-State Collaboration in Resource-Constrained Visual Agents
Source: [https://arxiv.org/html/2605.31354](https://arxiv.org/html/2605.31354)
###### Abstract

Modular visual reasoning systems increasingly rely on shared working memory for multi\-step collaboration, yet the failure dynamics of intermediate state evolution in low\-capacity regimes remain underexplored\. We study failure modes of collaborative reasoning with weak learners \(4B–8B models\) through the lens of noise accumulation\. We introduce CoSee, an auditing framework that formalizes the read\-write\-verify loop to trace information flow in document visual question answering\. Across multi\-page, chart, and web\-based benchmarks, we find a counter\-intuitive degradation: naive shared workspaces often amplify hallucinations rather than resolve them\. We identify two dominant failure modes: Noise Reinforcement, where ungrounded notes are reused as evidence, and Policy Collapse, where added context shifts the model toward under\-specified, short\-form answers\. Using cost\-accuracy Pareto frontiers, we show that increased compute can correlate negatively with performance without explicit verification\. Our findings suggest that for resource\-constrained agents, the bottleneck lies not in reasoning depth but in communication fidelity, providing trace\-level diagnostics and a mechanistic baseline for reliable modular design\.

Machine Learning, ICML

## 1Introduction

Recent advances in Vision Language Models \(VLMs\) have enabled reasoning over high\-information density artifacts, such as multi\-page slides, quantitative charts, and web documents\. To tackle these complex inputs, the field is increasingly moving toward modular designs that utilize shared workspaces \(e\.g\., whiteboards\) to externalize intermediate states\. In principle, this form of external working memory allows limited\-capacity models to ”think on paper”: decomposing intractable queries into verifiable steps of reading, writing, and cross\-checking\. The prevailing intuition suggests that such explicit state tracking inherently promotes reliability by reducing the cognitive load on a single inference pass\.

However, the efficacy of this paradigm remains unproven under the very boundaries that motivate its use: resource constrained regimes characterized by weak learners \(e\.g\., small models\) and strict compute budgets\. In this setting, we hypothesize that the collaborative loop acts as a noisy communication channel rather than a reliable memory store\. Every additional turn not only incurs a linear computational cost but introduces a risk of noise accumulation: collaboration alters the conditioning distribution at each step, potentially leading to policy drift \(e\.g\., shifts in verbosity or coverage\) or hallucination reinforcement, where ungrounded intermediate notes are ”hardened” into incorrect evidence\. These failure modes are often silent and diagnostically opaque without rigorous integrity auditing\.

To investigate these limits, we present CoSee, an auditing framework and collaboration protocol designed to rigorously measure the utility of shared context under small\-model constraints\. CoSee formalizes the agentic loop into a minimal action interface \(read, write\-to\-board, verify\), logs structured traces of intermediate states, and enforces strict output integrity checks to separate reasoning failures from format errors \. unlike ad\-hoc evaluations, we benchmark all configurations with strictly matched prompts and decoding caps, ensuring that any observed differences stem from the interaction dynamics rather than hidden computational advantages\. We utilize this pipeline to study three configurations: direct single\-turn answering, a single\-agent board variant, and a two\-agent scanner–checker architecture\.

![Refer to caption](https://arxiv.org/html/2605.31354v1/x1.png)Figure 1:CoSee overview\.Shared\-board collaboration with trace logging and integrity auditing enables controlled, cost\-normalized evaluation under strict budgets\. Our study finds that naive board use is not a reliable win under small\-model constraints, while a lightweight verified\-board gate mitigates chart\-centric failures\.We conduct our empirical study across three representative distributions—SlideVQA, ChartQAPro, and VQAonline—using Qwen3\-VL\-4B\-Instruct as the primary backbone, with scale and robustness checks on Qwen3\-VL\-8B, Phi\-4, and Gemma\-3\-4B\. Our results reveal a distinct efficiency paradox: naive usage of shared memory yields at most marginal gains on retrieval tasks and often degrades performance on reasoning\-heavy tasks\. Specifically, we find that the multi\-agent setting frequently underperforms the single\-turn baseline due to coordination overhead and coverage failures\. To quantify these trade\-offs, we analyze cost–accuracy Pareto frontiers, showing that increased token usage often correlates negatively with performance in the absence of quality control\. Trace\-grounded diagnostics further isolate the mechanisms behind this degradation—primarily noise reinforcement on charts and policy collapse \(short\-answer drift\) on open\-ended QA—demonstrating that lightweight verification is the minimal necessary condition to arrest error propagation\.

#### Contributions\.

We introduce CoSee, a formal framework for auditing the information flow and integrity in modular VQA systems, which enables a systematic empirical investigation into the limits of weak learner collaboration\. By integrating compute\-aware metrics—specifically cost\-accuracy Pareto frontiers and format\-robust scoring—we quantify the operational boundaries where external memory shifts from an asset to a liability due to noise accumulation\. Furthermore, we establish a taxonomy of dominant failure mechanisms, including noise reinforcement and policy collapse, and empirically validate that implementing a grounded information bottleneck via lightweight verification is the minimal necessary condition for reliable scaling in resource\-constrained document VQA\.

## 2Related Work

#### Document and chart VQA as reasoning testbeds\.

Recent datasets have shifted towards high\-information density artifacts, establishing document and chart QA as rigorous testbeds for grounded reasoning in complex visual contexts\(Mathewet al\.,[2021](https://arxiv.org/html/2605.31354#bib.bib14); Masryet al\.,[2022](https://arxiv.org/html/2605.31354#bib.bib15); Nguyenet al\.,[2025](https://arxiv.org/html/2605.31354#bib.bib16)\)\. Specifically, ChartQAPro introduces adversarial perturbations and unanswerable queries, exposing the fragility of current LVLMs in handling out\-of\-distribution chart types\(Masryet al\.,[2025](https://arxiv.org/html/2605.31354#bib.bib1)\)\. Complementarily, VQAonline challenges models with open\-ended generation from authentic web data, where evaluation metrics must disentangle semantic correctness from policy shifts in coverage and verbosity\(Chenet al\.,[2024](https://arxiv.org/html/2605.31354#bib.bib2)\)\. These distributional shifts necessitate evaluation protocols that explicitly account for formatting sensitivity and the computational cost of generating adequately supported answers, moving beyond raw accuracy metrics\(Rajpurkaret al\.,[2016](https://arxiv.org/html/2605.31354#bib.bib17)\)\.

#### Architecture\-centric document understanding\.

Prior work has largely focused on encoding inductive biases for document structure, improving understanding by exploiting layout signals and multi\-page context\(Xuet al\.,[2021a](https://arxiv.org/html/2605.31354#bib.bib18),[b](https://arxiv.org/html/2605.31354#bib.bib19)\)\. Approaches such as DocLLM and LayoutLLM optimize instruction tuning for structural comprehension\(Wanget al\.,[2024](https://arxiv.org/html/2605.31354#bib.bib3); Luoet al\.,[2024](https://arxiv.org/html/2605.31354#bib.bib4)\), while architectures like DocOwl2 address the efficiency–accuracy trade\-off via high\-resolution compression mechanisms\(Huet al\.,[2025](https://arxiv.org/html/2605.31354#bib.bib5)\)\. In parallel, open LVLMs continue to scale vision encoders to handle multi\-image resolution dynamics \(e\.g\., LLaVA\-UHD, LLaVA\-OneVision\)\(Guoet al\.,[2024](https://arxiv.org/html/2605.31354#bib.bib6); Liet al\.,[2024](https://arxiv.org/html/2605.31354#bib.bib7)\), and recent technical reports \(e\.g\., Qwen3\-VL\) highlight strengthened foundational capabilities in document parsing\(Baiet al\.,[2025](https://arxiv.org/html/2605.31354#bib.bib8)\)\. Our work is orthogonal to these architectural advances: rather than proposing a new backbone, we isolate the inference\-time dynamics of protocol\-level collaboration via shared memory in the fixed small\-model regime, analyzing its interaction with output formats and semantic coverage\(Kimet al\.,[2022](https://arxiv.org/html/2605.31354#bib.bib20); Leeet al\.,[2023](https://arxiv.org/html/2605.31354#bib.bib21)\)\.

#### Modular agents and orchestration costs\.

Interactive multimodal agents are increasingly deployed in dynamic environments, such as GUI navigation, where success depends on grounded perception and sequential planning\(Kohet al\.,[2024](https://arxiv.org/html/2605.31354#bib.bib9); Honget al\.,[2024](https://arxiv.org/html/2605.31354#bib.bib10); Zhenget al\.,[2024](https://arxiv.org/html/2605.31354#bib.bib22)\)\. Recent research explores modular reasoning via role specialization and iterative verification\(Li and others,[2024](https://arxiv.org/html/2605.31354#bib.bib11)\), specifically within document\-focused pipelines\(Hanet al\.,[2025](https://arxiv.org/html/2605.31354#bib.bib12); Sunet al\.,[2025](https://arxiv.org/html/2605.31354#bib.bib13); Jainet al\.,[2025](https://arxiv.org/html/2605.31354#bib.bib23); Yuet al\.,[2025](https://arxiv.org/html/2605.31354#bib.bib31)\)\. However, the utility of such orchestration is often obscured by confounding factors, including heavier backbone capacities or unnormalized interaction costs\(Weiet al\.,[2022](https://arxiv.org/html/2605.31354#bib.bib24); Wanget al\.,[2022](https://arxiv.org/html/2605.31354#bib.bib25); Yaoet al\.,[2023](https://arxiv.org/html/2605.31354#bib.bib26)\)\. In contrast, we present a formal auditing framework for shared\-workspace protocols, focusing on output integrity and trace artifacts\. We emphasize cost\-normalized comparisons via compute\-logged experiments to identify the specific mechanisms—such as noise accumulation—that cause collaboration to degrade performance in resource\-constrained VQA\(Wanget al\.,[2025](https://arxiv.org/html/2605.31354#bib.bib27); Jianget al\.,[2024](https://arxiv.org/html/2605.31354#bib.bib28); Yiet al\.,[2025](https://arxiv.org/html/2605.31354#bib.bib29); Kugoet al\.,[2025](https://arxiv.org/html/2605.31354#bib.bib30)\)\.

## 3Method

We formulate the problem of document visual question answering as a sequential decision\-making process under strict computational constraints\. Our goal is to analyze the inference dynamics of weak learners when augmented with external working memory\.

### 3\.1Problem Formulation

Letℳθ\\mathcal\{M\}\_\{\\theta\}denote a vision–language model with parametersθ\\theta, subject to a resource constraintΩ\\Omega\(defined by limited parameter size, e\.g\., 4B–8B, and single\-GPU memory\)\. Given a visual inputI=\{Ik\}k=1KI=\\\{I\_\{k\}\\\}\_\{k=1\}^\{K\}\(representing slides, charts, or documents\) and a natural language queryqq, the objective is to generate an answera∈𝒜a\\in\\mathcal\{A\}that maximizes the likelihoodP​\(a\|I,q\)P\(a\|I,q\)\.

In a collaborative setting, the generation ofaais mediated by a sequence of intermediate discrete latent statesB1:TB\_\{1:T\}, representing the trajectory of the shared workspace\. The inference process seeks to approximate the intractable posterior of the answer given the reasoning chain:

P​\(a\|I,q\)≈∑BTPθ​\(a\|I,q,BT\)​∏t=1TPθ​\(Bt\|I,q,Bt−1\)P\(a\|I,q\)\\approx\\sum\_\{B\_\{T\}\}P\_\{\\theta\}\(a\|I,q,B\_\{T\}\)\\prod\_\{t=1\}^\{T\}P\_\{\\theta\}\(B\_\{t\}\|I,q,B\_\{t\-1\}\)\(1\)We focus on theinference\-timebehavior ofℳθ\\mathcal\{M\}\_\{\\theta\}, treating the weights as fixed\.

### 3\.2External Working Memory \(The Board\)

To externalize the intermediate reasoning steps, CoSee introduces a structured latent state, theBoard\(BtB\_\{t\}\)\. Unlike internal hidden states,BtB\_\{t\}is a discrete, interpretable sequence of entries readable by all agents\. Formally, at time steptt, the board state is defined as an ordered sequenceBt=\[e1,e2,…,et\]B\_\{t\}=\[e\_\{1\},e\_\{2\},\\dots,e\_\{t\}\], where each entryeie\_\{i\}is a tuple\(𝐦,𝐩,id\)\(\\mathbf\{m\},\\mathbf\{p\},\\texttt\{id\}\)\. Here,𝐦\\mathbf\{m\}represents a natural language note \(evidence or hypothesis\),𝐩\\mathbf\{p\}denotes an optional grounding pointer \(e\.g\., page index or region coordinates\), andididentifies the authoring agent\.

The transition dynamics of the board are governed by a lightweight controller that enforces a strict read\-write interface\. At each steptt, the controller exposes the current stateBt−1B\_\{t\-1\}to the active agent policyπ\\pi, which samples a new updateete\_\{t\}\. The state evolves deterministically asBt=Bt−1⊕etB\_\{t\}=B\_\{t\-1\}\\oplus e\_\{t\}, where⊕\\oplusdenotes the append operation\. This design ensures that the entire reasoning trajectory is observable and auditable\.

### 3\.3Inference Dynamics and Protocols

We investigate three distinct inference protocols, viewing them as variations in the state transition probabilityP​\(Bt\|Bt−1\)P\(B\_\{t\}\|B\_\{t\-1\}\)and the readout function\.

#### Direct Inference \(Baseline\)\.

The baseline represents a collapse of the sequential process into a single step\. The model directly samples the output distribution conditioned only on the input:

a^∼Pθ​\(a\|I,q\)\\hat\{a\}\\sim P\_\{\\theta\}\(a\|I,q\)\(2\)This serves as the lower bound for computational cost and the reference point for reasoning quality\.

#### Open\-Loop Iterative Refinement \(Single & Multi\-Agent\)\.

In this regime, agents interact with the board for a fixed budget ofTTsteps without explicit external feedback\.

- •Single\-Agent Dynamics:A single policyπθ\\pi\_\{\\theta\}iteratively updates the board\. At steptt, the agent sampleset∼πθ\(⋅\|I,q,Bt−1\)e\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\|I,q,B\_\{t\-1\}\)\.
- •Dual\-Agent Dynamics \(Scanner–Checker\):We instantiate two complementary roles,πscan\\pi\_\{\\text\{scan\}\}andπcheck\\pi\_\{\\text\{check\}\}, derived from the same backboneℳθ\\mathcal\{M\}\_\{\\theta\}via prompting\. The agents operate in alternating turns, effectively modeling a cooperative game whereπscan\\pi\_\{\\text\{scan\}\}proposes evidence andπcheck\\pi\_\{\\text\{check\}\}refines or refutes it based on the shared contextBt−1B\_\{t\-1\}\.

Crucially, in both settings, the final answer is generated via a dedicated readout stepa^∼Pθ​\(a\|I,q,BT\)\\hat\{a\}\\sim P\_\{\\theta\}\(a\|I,q,B\_\{T\}\), ensuring that intermediate artifacts \(e1:Te\_\{1:T\}\) do not leak strictly into the answer format unless explicitly attended to\.

#### Gated Transition with Information Bottleneck \(Verified\-Board\)\.

To mitigate the propagation of ungrounded hallucinations \(noise\), we introduce a conditional gating mechanism on the state transition\. LetVϕ​\(e,I,q\)∈\{0,1\}V\_\{\\phi\}\(e,I,q\)\\in\\\{0,1\\\}be a verification function \(parameterized by the same backboneℳθ\\mathcal\{M\}\_\{\\theta\}\) that maps a proposed entryeeto a binary validity score\. The update rule becomes:

Bt=\{Bt−1⊕etif​Vϕ​\(et,I,q\)=1​\(Supported\)Bt−1otherwise \(Discarded\)B\_\{t\}=\\begin\{cases\}B\_\{t\-1\}\\oplus e\_\{t\}&\\text\{if \}V\_\{\\phi\}\(e\_\{t\},I,q\)=1\\text\{ \(Supported\)\}\\\\ B\_\{t\-1\}&\\text\{otherwise \(Discarded\)\}\\end\{cases\}\(3\)This mechanism acts as aninformation bottleneck, filtering out high\-entropy or hallucinated notes before they become part of the conditioning context for subsequent steps\. We hypothesize that this control is essential for preventing noise accumulation in weak learners\.

### 3\.4Contextual Conditioning and Policy Drift

To operationalize the collaborative roles, we construct the input context𝒞t\\mathcal\{C\}\_\{t\}by modifying the system instructions while keeping the visual backbone fixed\. For theScanner,𝒞t=\[ℐscan,I,q,Bt−1\]\\mathcal\{C\}\_\{t\}=\[\\mathcal\{I\}\_\{\\text\{scan\}\},I,q,B\_\{t\-1\}\], whereℐscan\\mathcal\{I\}\_\{\\text\{scan\}\}conditions the model to extract atomic visual evidences\. For theCrossChecker,𝒞t=\[ℐcheck,I,q,Bt\]\\mathcal\{C\}\_\{t\}=\[\\mathcal\{I\}\_\{\\text\{check\}\},I,q,B\_\{t\}\], whereℐcheck\\mathcal\{I\}\_\{\\text\{check\}\}conditions on identifying contradictions betweenIIandBtB\_\{t\}\.

We posit that this augmented context introduces adistributional shiftin the generation of the final answer\. While the ideal objective is to approximate the true posteriorP​\(a\|I,q\)P\(a\|I,q\), the dependency on intermediate states introduces a bias:a^≈arg⁡maxa⁡Pθ​\(a\|summary​\(BT\),q\)\\hat\{a\}\\approx\\arg\\max\_\{a\}P\_\{\\theta\}\(a\|\\text\{summary\}\(B\_\{T\}\),q\)\. In regimes where the backboneθ\\thetahas limited attention capacity \(e\.g\., small models\), we hypothesize that the model may over\-attend to the compressed textual summary inBTB\_\{T\}while under\-attending to the high\-dimensional visual features inII, a phenomenon we termattention dilution\. Furthermore, if the intermediate notese1:Te\_\{1:T\}exhibit a specific stylistic bias \(e\.g\., extreme brevity\), this style may transfer to the final answer via in\-context learning, leading topolicy driftwhere the output format diverges from the optimal answer length required by the task\.

### 3\.5Traceability and Audit Framework

A core contribution of CoSee is the rigorous auditing of the inference trajectory\. We treat the sequenceτ=\(I,q,B1:T,a^\)\\tau=\(I,q,B\_\{1:T\},\\hat\{a\}\)as a probabilistic trace\. To ensure the validity of our analysis, we implement anintegrity auditfunctionℐ​\(a^\)→\{valid,malformed,leakage\}\\mathcal\{I\}\(\\hat\{a\}\)\\rightarrow\\\{\\texttt\{valid\},\\texttt\{malformed\},\\texttt\{leakage\}\\\}\. This allows us to detect protocol\-level pathologies—such as when intermediate notes are copied verbatim into the final answer \(Leakage\)—without introducing selection bias into the scoring\. All examples, including those flagged as malformed, are retained in the evaluation to penalize protocol instability\.

### 3\.6Cost–Utility Analysis

In resource\-constrained regimes, performance cannot be decoupled from cost\. We define the computational cost function𝒞​\(τ\)\\mathcal\{C\}\(\\tau\)as the total number of tokens generated across all intermediate steps and the final answer\. We analyze the system’s efficiency by plotting the empiricalPareto frontierbetween the utility𝒰\\mathcal\{U\}\(accuracy metric\) and cost𝒞\\mathcal\{C\}:

ℱ∗=maxπ⁡𝔼τ∼π​\[𝒰​\(a^,a∗\)\]s\.t\.𝔼​\[𝒞​\(τ\)\]≤β\\mathcal\{F\}^\{\*\}=\\max\_\{\\pi\}\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\[\\mathcal\{U\}\(\\hat\{a\},a^\{\*\}\)\]\\quad\\text\{s\.t\.\}\\quad\\mathbb\{E\}\[\\mathcal\{C\}\(\\tau\)\]\\leq\\beta\(4\)This formulation allows us to determine whether the additional compute invested in collaboration yields a super\-linear, linear, or negative return on investment\.

## 4Experiments Setup

### 4\.1Evaluation Domains and Distributions

To rigorously test the limits of collaboration, we select three benchmarks that map to distinct failure modes in modular visual reasoning\.

- •SlideVQA\(Multi\-page Retrieval\): Represents thehigh\-recallregime, requiring agents to locate needle\-in\-a\-haystack evidence across multiple slide images\(Mathewet al\.,[2021](https://arxiv.org/html/2605.31354#bib.bib14)\)\.
- •ChartQAPro\(Quantitative Reasoning\): Represents thehigh\-precisionregime, stressing the system’s ability to handle exact values and reject unanswerable queries under adversarial noise\(Masryet al\.,[2025](https://arxiv.org/html/2605.31354#bib.bib1)\)\.
- •VQAonline\(Open\-Ended Synthesis\): Represents thepolicy\-sensitiveregime, where answer quality depends on coverage and verbosity rather than exact string matching\(Chenet al\.,[2024](https://arxiv.org/html/2605.31354#bib.bib2)\)\.

#### Controlled Subsets\.

To ensure strict reproducibility and cost\-normalized comparison, we employ fixed evaluation subsets with identical example IDs across all backbones and protocols: SlideVQA test \(n=600n\{=\}600\), ChartQAPro test \(n=800n\{=\}800\), and VQAonline trainval \(n=800n\{=\}800\)\. We release these subset definitions to facilitate future auditing\.

#### Signal Isolation Metrics\.

Beyond standard Exact Match \(EM\) and Loose Match, we introduce specialized metrics to isolate reasoning failures from formatting noise: \(1\)Format\-Robust Scoring \(ChartQAPro\):We reportAnswerable\-OnlyandPost\-Processed \(pp\)variants to verify that performance drops are due to hallucination \(noise reinforcement\) rather than superficial string mismatches \(Table S8 in the Appendix\)\. \(2\)Policy\-Aware F1 \(VQAonline\):We report token\-level F1 alongside strict/relaxed success rates to detect coverage collapse\. Table[1](https://arxiv.org/html/2605.31354#S4.T1)summarizes the evaluation protocol\.

Table 1:Evaluation protocol and fixed subsets\.All results use identical fixed ID lists across methods/backbones\. For ChartQAPro, Answerable\-only excludesUnanswerablecases; Post\-processed normalizes formatting \(Table S8 in the Appendix\)\.

### 4\.2Experimental Conditions

We compare three distinct inference protocols \(as defined in Section[3\.3](https://arxiv.org/html/2605.31354#S3.SS3)\) under a unified evaluation pipeline\.

1. 1\.Direct Inference \(baseline\_single\):The lower\-bound control, generating answers directly from inputs\.
2. 2\.Open\-Loop Scratchpad \(single\_board\):A single agent utilizing the read\-write board interface for intermediate state tracking\.
3. 3\.Dual\-Agent Collaboration \(CoSeetwo\_qwen\):An alternating scanner–checker dyad operating under a shared context\.

Additionally, we evaluate aVerified\-Boardcontrol \(Appendix\) to test the hypothesis that gating intermediate states acts as a necessary noise filter\.

#### Resource Constraints and Decoding Budgets\.

To strictly enforce the small\-model regime, we apply deterministic decoding \(temperature=0=0\) for all generation calls\. We impose a hard per\-call cap ofmax\_new\_tokens=32\\texttt\{max\\\_new\\\_tokens\}=32and a maximum interaction horizon ofT=3T=3steps\. Crucially, these caps are matched across protocols; any increase in total compute arises solely from the structural overhead of the protocol \(extra calls\), not from relaxed generation limits\.

### 4\.3Weak Learner Backbones

We evaluate the protocols on a suite of open\-weights models representing the 4B–8B parameter class:Qwen3\-VL\-4B\-Instruct\(primary\),Qwen3\-VL\-8B\-Instruct\(scale check\),Phi\-4\-multimodal\-instruct, andGemma\-3\-4B\-it\. All experiments use identical preprocessing and prompt templates to ensure that performance differences are attributable to the interaction dynamics\.

#### Integrity Auditing\.

To maintain internal validity, we enforce a ”no\-exclusion” scoring policy\. We employ the audit functionℐ​\(a^\)\\mathcal\{I\}\(\\hat\{a\}\)to flag protocol pathologies \(e\.g\., missing answers, note leakage\)\. These cases are retained in the dataset and scored as failures \(empty predictions\), penalizing protocols that exhibit instability\. Appendix Table S3 details the audit rates\.

#### Final\-Answer Enforcement\.

To ensure comparable output formats, all board\-based variants \(single and multi\-agent\) conclude with a dedicatedreadoutstep\.baseline\_singleutilizes the same answer\-generation prompt format to nullify parsing\-related variances\.

### 4\.4Resource Accounting

We quantify the cost\-utility trade\-off by logging the total computational footprint per example\. We defineGenTokens\(final\)as the token count of the final predictiona^\\hat\{a\}\(Table S5 in the Appendix\)\. For full\-protocol analysis, we aggregate thegen\_tokens\_totalacross all intermediate steps\.

#### Diagnosing Policy Drift\.

For open\-ended tasks \(VQAonline\), we perform a stratified analysis by binning outputs based onGenTokens\(final\)\. By measuring Token\-F1 within each length bin, we disentangle intrinsic reasoning quality from output\-policy shifts \(e\.g\., a tendency toward terseness\)\. This diagnostic \(Figure[3](https://arxiv.org/html/2605.31354#S5.F3)\) is critical for verifying whether performance degradation stems from ”Policy Collapse\.”

## 5Results

We empirically analyze the utility of shared context across three distinct reasoning distributions\. Our primary findings, summarized in Table[2](https://arxiv.org/html/2605.31354#S5.T2)and Figure[2](https://arxiv.org/html/2605.31354#S5.F2), reveal a consistentnegative result: for weak learners, naive collaboration often acts as a noise amplifier rather than a reasoning enhancer\.

Table 2:Main results across backbones\.SlideVQA/ChartQAPro report EM and Loose; VQAonline reports Token\-F1 with Strict/Relaxed success rates \(F1≥\\geq0\.5 / 0\.3\)\. ChartQAPro uses raw metrics \(EMraw/Looseraw\) as primary and answerable\-only metrics \(EMans/Looseans\) as complementary\.postdenotes post\-processed ChartQAPro scores \(relaxed formatting\)\. Diagnostic subset runs are marked by†/‡\.†Three\-agent results are evaluated on a smaller subset of ChartQAPro \(n=200n\{=\}200, answerablen=166n\{=\}166\) for 4B\-class backbones; Qwen3\-VL\-8B\-Instruct is not evaluated\.‡Two\-stage results are from Phase\-2 compute\-logged 200\-example subsets; ChartQAPro values reported here are post\-processed\.

#### The Efficiency Paradox\.

Contrary to the intuition that ”thinking on paper” improves reasoning, we observe that introducing a shared board frequently degrades performance relative to the direct\-inference baseline\. As shown in the forest plot \(Figure[2](https://arxiv.org/html/2605.31354#S5.F2)\), the effect sizeΔ​\(Method−Baseline\)\\Delta\(\\text\{Method\}\-\\text\{Baseline\}\)is statistically indistinguishable from zero on retrieval tasks \(SlideVQA\) and significantly negative on reasoning\-heavy tasks \(ChartQAPro, VQAonline\)\. This trend is robust across all three 4B\-class backbones \(Qwen3, Phi\-4, Gemma\-3\), suggesting it is a property of themodel capacity regimerather than a specific architecture\.

![Refer to caption](https://arxiv.org/html/2605.31354v1/x2.png)Figure 2:Effect sizes across reasoning distributions\.Paired\-bootstrap confidence intervals \(95%\) forΔ​\(Method−Baseline\)\\Delta\(\\text\{Method\}\-\\text\{Baseline\}\)\. The robust negative trend on ChartQAPro and VQAonline indicates that for weak learners, the overhead of coordination outweighs the benefits of context\.
### 5\.1Mechanism Analysis: Why Collaboration Fails

To move beyond raw metrics, we isolate the causal mechanisms driving this degradation\. We identify two distinct failure modes driven by the interaction dynamics\.

#### Mechanism I: Noise Accumulation \(ChartQAPro\)\.

On quantitative reasoning tasks, the board acts as ahallucination trap\. Bothsingle\_boardand multi\-agent protocols underperform the single\-turn baseline\. Decomposing the error distribution \(Figure[4](https://arxiv.org/html/2605.31354#S5.F4)\) reveals a surge in Type T2 failures \(Board\-amplified wrong intermediates\), where an initial perception error \(e\.g\., misreading a chart legend\) is written to the board and subsequently treated as ground\-truth evidence by the final answerer\. This confirms our hypothesis ofnoise reinforcement: without verification, the shared state entrenches early mistakes\.

Verification as a Remedy:Introducing theverified\_boardgate \(a simple Support/Refute check\) recovers the majority of the performance drop \(Table S9 in Appendix\)\. This causal intervention demonstrates that the bottleneck is not thecapacityto reason, but thefidelityof the intermediate information channel\.

#### Mechanism II: Policy Collapse \(VQAonline\)\.

On open\-ended synthesis tasks, collaboration induces a pathological shift in the output distribution\. As diagnosed in Figure[3](https://arxiv.org/html/2605.31354#S5.F3), the presence of intermediate context causes the model to collapse toward overly terse responses \(shifting from the 17–32 token bin to the 0–16 token bin\)\. Thispolicy driftresults in severe under\-coverage of the required answer components, leading to lower Token\-F1 scores\. Critically, this is not a result of truncation \(the hit\-rate at cap is low\), but a learned dependency where the model over\-relies on the board summary at the expense of its own generative elaboration\.

![Refer to caption](https://arxiv.org/html/2605.31354v1/x3.png)Figure 3:Diagnosing Policy Collapse on VQAonline\.Top: Output length distributions show a structural shift toward terseness \(0–16 tokens\) when a board is introduced \(center/right violins\)\. Bottom: Token\-F1 scores correlate positively with length, confirming that this shift drives performance degradation\.Failure taxonomy by method across datasets \(Qwen3\-VL\-4B; 100 sampled errors each\)

![Refer to caption](https://arxiv.org/html/2605.31354v1/x4.png)

SlideVQAChartQAProVQAonline

Figure 4:Causal Failure Analysis\.We stratify errors into dominant mechanisms\. ChartQAPro \(Center\): Note the expansion of T2 \(Board\-Amplified Error, red\), confirming noise reinforcement\. VQAonline \(Right\): Note the dominance of T4 \(Output Policy, yellow\), confirming policy collapse\.

### 5\.2Cost–Utility Pareto Analysis

We quantify the economic efficiency of collaboration by plotting the Cost–Accuracy Pareto Frontier \(Figure[5](https://arxiv.org/html/2605.31354#S5.F5)\)\.

#### Negative Returns on Compute\.

Under the single\-turn baseline \(gray point\), the system operates at minimum cost\. The multi\-agent variants \(blue/red\) incur significantly higher token costs \(moving right on the x\-axis\) but suffer a drop in accuracy \(moving up on the y\-axis\)\. This places naive collaboration in the quadrant ofnegative utility\. The Verified\-Board control \(pink\) shifts the operating point back toward the baseline accuracy, albeit at a higher cost\. This analysis suggests that in the small\-model regime, there is currentlyno ”free lunch”from test\-time compute scaling without strict quality control mechanisms\.

![Refer to caption](https://arxiv.org/html/2605.31354v1/x5.png)Figure 5:Cost–Utility analysis on ChartQAPro\.We plot the degradation in exact match accuracy \(Δ\\DeltaEM\) against the additional computational cost \(mean output tokens\)\. Naive collaboration \(red/blue\) resides in thenegative utilityquadrant: consuming more compute to produce worse results\. Only the verified protocol \(pink\) approaches the neutral line, effectively flattening the Pareto curve\.

## 6Discussions

Our investigation into resource\-constrained modular reasoning reveals a fundamental trade\-off: expanding the state space via external memory does not automatically yield better reasoning; rather, it introduces a fidelity–complexity bottleneck\. Under the regime of weak learners \(small models under single\-GPU constraints\), we observe that the shared workspace functions less as a reliable knowledge store and more as a noisy communication channel\. Below, we synthesize the mechanistic principles and outline the minimal conditions required for positive scaling\.

### 6\.1The Dual Nature of Shared Context

The divergence in performance across benchmarks highlights the dual role of external memory\. In high\-recall retrieval tasks like SlideVQA, the board acts as a sparse evidence buffer\. For multi\-page inputs, the primary challenge is the tendency to forget local details while traversing the document\. Here, the board decouples perception from reasoning, allowing the model to offload factual snippets, e\.g\., dates or names\. Since retrieval errors are often independent, the board serves to consolidate scattered evidence, stabilizing the final prediction against context\-window decay\.

Conversely, in high\-precision reasoning tasks, the board risks becoming a positive feedback loop for error\. Chart reasoning requires strict dependency chains, and our taxonomy reveals that ungrounded board entries often encode hallucinated priors, such as wrong values or trends\. Unlike retrieval, where noise can be ignored, reasoning tasks treat these intermediates as premises\. Without a mechanism to reject false premises, the system exhibits error reinforcement, where the final answer is conditioned on a hallucinated reality\. Similarly, on VQAonline, the presence of context induces a policy misalignment, where the model optimizes for the summary format of the board rather than the comprehensive format required, leading to a collapse in answer coverage\.

### 6\.2The Failure of Open\-Loop Collaboration

The persistent underperformance of naive multi\-agent settings challenges the ensemble hypothesis in low\-capacity regimes\. We attribute this failure to the open\-loop nature of the interaction\. In standard protocols, a downstream agent consumes an upstream agent’s output without access to the originating visual signals, or without the capacity to robustly verify them\. This creates a blind chain of command characterized by two deficits\. First, intermediate notes typically lack explicit grounding pointers \(e\.g\., bounding boxes\) necessary to constrain interpretation\. Second, the system lacks a negative feedback mechanism; without a rejection gate, it cannot prune divergent reasoning paths\. Consequently, the coordination cost is spent not on refining the signal, but on propagating the noise\. The recovery of performance via the verified board control supports this control\-theoretic view, as a simple binary gate transforms the system from open\-loop to quasi\-closed\-loop, thereby arresting the cascade of errors\.

### 6\.3Design Principles for Reliable Modular Systems

Our findings suggest that test\-time compute scaling for weak learners is governed by a principle of quality over quantity, where mere decomposition is insufficient and deleterious\. To ensure that the board acts as valid reasoning support, we propose three minimal design ingredients\. First, intermediate artifacts must carry grounded state representations with verifiable metadata to prevent semantic drift during information handover\. Second, the protocol must implement gated transition dynamics as an information bottleneck\. This requires explicit rejection sampling or verification gates that allow the system to discard an update if it contradicts the visual evidence\. Finally, for open\-ended tasks, the latent state representation must be aligned with the final output objective\. If the board encourages brevity, the decoder requires counter\-balancing constraints to prevent policy collapse\.

## 7Limitations

#### The Regime of Weak Learners\.

Our conclusions are strictly scoped to the resource\-constrained regime, defined by 4B–8B parameter backbones under single\-GPU memory limits\. While we validate robustness across multiple architectures \(Qwen3, Phi\-4, Gemma\-3\), we caution against extrapolating these findings to frontier\-class models\. It remains an open question whether the noise accumulation dynamics we observe would be suppressed by the emergent reasoning capabilities of larger models, or if the efficiency paradox persists until a specific capability threshold is crossed\.

#### Statistical Power vs\. Computational Tractability\.

To ensure the feasibility of constructing dense Pareto frontiers, we relied on fixed, stratified subsets \(SlideVQA testn=600n\{=\}600, ChartQAPro testn=800n\{=\}800, VQAonline trainvaln=800n\{=\}800\) rather than full\-scale benchmarks\. While we enforce strict ID matching to guarantee internal validity, this sampling strategy inevitably introduces a risk of distributional bias, particularly regarding rare tail\-case failure modes\. Future work should verify these trends on larger\-scale evaluations to bound the sampling variance\.

#### Protocol Minimality as a Control\.

Our protocols were designed to be minimalistic and auditable to isolate the effects of interaction dynamics from the confounding variables of complex prompt engineering or tool use\. Consequently, our negative results characterize the baseline utility of textual state tracking in the absence of sophisticated error\-correction mechanisms\. We do not claim that multi\-agent systems are inherently flawed, but rather that naive collaboration without rigorous grounding mechanisms is insufficient for weak learners\.

#### Abstraction of Computational Cost\.

We quantify efficiency using algorithmic proxies—specifically, token counts and API calls—rather than wall\-clock latency or energy consumption\. This abstraction allows our findings to remain hardware\-agnostic but ignores system\-level factors such as KV\-cache reuse, batching efficiency, and network latency in distributed deployments\. Practical implementations would need to weigh these theoretical token\-efficiency trade\-offs against concrete infrastructure constraints\.

## 8Conclusion

In this work, we formalized the study of modular visual reasoning under resource constraints, using the CoSee auditing framework to probe the limits of weak learners\. Our empirical investigation reveals a critical efficiency paradox: simply expanding the state space via shared working memory does not guarantee improved reasoning; instead, for 4B–8B parameter models, the collaborative loop often functions as a noise amplifier, where the benefits of decomposition are outweighed by the costs of error propagation and policy collapse\. Our analysis of cost utility Pareto frontiers demonstrates that naive test\-time compute scaling is strictly bounded by the signal to noise ratio of the intermediate communication channel\. Consequently, we argue that the path toward reliable modular agents lies not in the proliferation of open\-loop roles, but in the implementation of rigorous closed loop controls, engineering the grounded information bottlenecks necessary to arrest the thermodynamics of error in distributed reasoning systems\.

## References

- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. Zhu \(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Chen, M\. Liu, N\. Codella, Y\. Li, L\. Yuan, and D\. Gurari \(2024\)Fully authentic visual question answering dataset from online communities\.pp\. 252–269\.Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px1.p1.1),[3rd item](https://arxiv.org/html/2605.31354#S4.I1.i3.p1.1)\.
- Z\. Guo, R\. Xu, Y\. Yao, J\. Cui, Z\. Ni, C\. Ge, T\. Chua, Z\. Liu, and G\. Huang \(2024\)LLaVA\-UHD: an lmm perceiving any aspect ratio and high\-resolution images\.Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Han, P\. Xia, R\. Zhang, T\. Sun, Y\. Li, H\. Zhu, and H\. Yao \(2025\)MDocAgent: a multi\-modal multi\-agent framework for document understanding\.arXiv preprint arXiv:2503\.13964\.Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- W\. Hong, W\. Wang, Q\. Lv, J\. Xu, W\. Yu, J\. Ji, Y\. Wang, Z\. Wang, Y\. Dong, M\. Ding, and J\. Tang \(2024\)CogAgent: a visual language model for gui agents\.pp\. 14281–14290\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52733.2024.01354)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Hu, H\. Xu, L\. Zhang, J\. Ye, M\. Yan, J\. Zhang, Q\. Jin, F\. Huang, and J\. Zhou \(2025\)MPLUG\-DocOwl2: high\-resolution compressing for OCR\-free multi\-page document understanding\.Vienna, Austria,pp\. 5817–5834\.External Links:[Link](https://aclanthology.org/2025.acl-long.291/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.291),ISBN 979\-8\-89176\-251\-0Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Jain, Y\. Wu, Y\. Zeng, J\. Liu, S\. hengyu Dai, Z\. Shao, Q\. Wu, and H\. Wang \(2025\)SimpleDoc: multi\-modal document understanding with dual\-cue page retrieval and iterative refinement\.ArXivabs/2506\.14035\.External Links:[Link](https://api.semanticscholar.org/CorpusID:279410653)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- B\. Jiang, Z\. Zhuang, S\. S\. Shivakumar, D\. Roth, and C\. J\. Taylor \(2024\)Multi\-agent vqa: exploring multi\-agent foundation models in zero\-shot visual question answering\.External Links:2403\.14783,[Link](https://arxiv.org/abs/2403.14783)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- G\. Kim, T\. Hong, M\. Yim, J\. Nam, J\. Park, J\. Yim, W\. Hwang, S\. Yun, D\. Han, and S\. Park \(2022\)OCR\-free document understanding transformer\.Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Y\. Koh, R\. Lo, L\. Jang, V\. Duvvur, M\. C\. Lim, P\. Huang, G\. Neubig, S\. Zhou, R\. Salakhutdinov, and D\. Fried \(2024\)VisualWebArena: evaluating multimodal agents on realistic visual web tasks\.arXiv preprint arXiv:2401\.13649\.Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- N\. Kugo, X\. Li, Z\. Li, A\. Gupta, A\. Khatua, N\. Jain, C\. Patel, Y\. Kyuragi, Y\. Ishii, M\. Tanabiki, K\. Kozuka, and E\. Adeli \(2025\)VideoMultiAgents: a multi\-agent framework for video question answering\.External Links:2504\.20091,[Link](https://arxiv.org/abs/2504.20091)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- K\. Lee, M\. Joshi, I\. Turc, H\. Hu, F\. Liu, J\. Eisenschlos, U\. Khandelwal, P\. Shaw, M\. Chang, and K\. Toutanova \(2023\)Pix2Struct: screenshot parsing as pretraining for visual language understanding\.Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px2.p1.1)\.
- B\. Li, Y\. Zhang, D\. Guo, R\. Zhang, F\. Li, H\. Zhang, K\. Zhang, Y\. Li, Z\. Liu, and C\. Li \(2024\)LLaVA\-onevision: easy visual task transfer\.arXiv preprint arXiv:2408\.03326\.Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Liet al\.\(2024\)COPPER: multi\-agent collaboration for llm\-based reasoning\.External Links:[Link](https://openreview.net/forum?id=MhDxTUgOU9)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- C\. Luo, Y\. Shen, Z\. Zhu, Q\. Zheng, Z\. Yu, and C\. Yao \(2024\)LayoutLLM: layout instruction tuning with large language models for document understanding\.pp\. 15630–15640\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52733.2024.01480)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px2.p1.1)\.
- A\. Masry, M\. S\. Islam, M\. Ahmed, A\. Bajaj, F\. Kabir, A\. Kartha, M\. T\. R\. Laskar, M\. Rahman, S\. Rahman, M\. Shahmohammadi, M\. Thakkar, M\. R\. Parvez, E\. Hoque, and S\. Joty \(2025\)ChartQAPro: a more diverse and challenging benchmark for chart question answering\.InFindings of the Association for Computational Linguistics: ACL 2025European Conference on Computer VisionProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)ECCV2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)Advances in Neural Information Processing Systems \(NeurIPS\)Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing2021 IEEE Winter Conference on Applications of Computer Vision \(WACV\)Findings of the Association for Computational Linguistics: ACL 2022Proceedings of the 16th Asian Conference on Machine LearningProceedings of the 2016 Conference on Empirical Methods in Natural Language ProcessingProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\)European Conference on Computer Vision \(ECCV\)Proceedings of the 40th International Conference on Machine LearningProceedings of the 41st International Conference on Machine LearningProceedings of the 36th International Conference on Neural Information Processing Systems,W\. Che, J\. Nabende, E\. Shutova, M\. T\. Pilehvar, L\. Ku, A\. Martins, V\. Srikumar, W\. Che, J\. Nabende, E\. Shutova, M\. T\. Pilehvar, C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, V\. Peng, S\. Muresan, P\. Nakov, A\. Villavicencio, V\. Nguyen, H\. Lin, J\. Su, K\. Duh, X\. Carreras, C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Proceedings of Machine Learning ResearchICML’23ICML’24NIPS ’22, Vol\.260,Vienna, Austria,pp\. 19123–19151\.External Links:[Link](https://aclanthology.org/2025.findings-acl.978/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.978),ISBN 979\-8\-89176\-256\-5Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px1.p1.1),[2nd item](https://arxiv.org/html/2605.31354#S4.I1.i2.p1.1)\.
- A\. Masry, D\. X\. Long, J\. Q\. Tan, S\. Joty, and E\. Hoque \(2022\)ChartQA: a benchmark for question answering about charts with visual and logical reasoning\.Dublin, Ireland,pp\. 2263–2279\.External Links:[Link](https://aclanthology.org/2022.findings-acl.177/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-acl.177)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px1.p1.1)\.
- M\. Mathew, D\. Karatzas, and C\. V\. Jawahar \(2021\)DocVQA: a dataset for vqa on document images\.pp\. 2199–2208\.External Links:[Document](https://dx.doi.org/10.1109/WACV48630.2021.00225)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px1.p1.1),[1st item](https://arxiv.org/html/2605.31354#S4.I1.i1.p1.1)\.
- D\. M\. H\. Nguyen, A\. T\. Le, T\. Q\. Nguyen, N\. T\. Diep, T\. Nguyen, D\. Duong\-Tran, J\. Peters, L\. Shen, M\. Niepert, and D\. Sonntag \(2025\)Dude: Dual distribution\-aware context prompt learning for large vision\-language model\.pp\. 687–702\.External Links:[Link](https://proceedings.mlr.press/v260/nguyen25c.html)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. Liang \(2016\)SQuAD: 100,000\+ questions for machine comprehension of text\.Austin, Texas,pp\. 2383–2392\.External Links:[Link](https://aclanthology.org/D16-1264/),[Document](https://dx.doi.org/10.18653/v1/D16-1264)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Sun, L\. He, S\. Jia, Y\. He, and C\. You \(2025\)DocAgent: an agentic framework for multi\-modal long\-context document understanding\.Suzhou, China,pp\. 17712–17727\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.893/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.893),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- D\. Wang, N\. Raman, M\. Sibue, Z\. Ma, P\. Babkin, S\. Kaur, Y\. Pei, A\. Nourbakhsh, and X\. Liu \(2024\)DocLLM: a layout\-aware generative language model for multimodal document understanding\.Bangkok, Thailand,pp\. 8529–8548\.External Links:[Link](https://aclanthology.org/2024.acl-long.463/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.463)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. H\. Chi, and D\. Zhou \(2022\)Self\-consistency improves chain of thought reasoning in language models\.ArXivabs/2203\.11171\.External Links:[Link](https://api.semanticscholar.org/CorpusID:247595263)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Wang, W\. Wan, Q\. Lao, R\. Chen, M\. Lang, X\. Wang, K\. Wang, and L\. Lin \(2025\)Towards top\-down reasoning: an explainable multi\-agent approach for visual question answering\.External Links:2311\.17331,[Link](https://arxiv.org/abs/2311.17331)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Xu, Y\. Xu, T\. Lv, L\. Cui, F\. Wei, G\. Wang, Y\. Lu, D\. Florencio, C\. Zhang, W\. Che, M\. Zhang, and L\. Zhou \(2021a\)LayoutLMv2: multi\-modal pre\-training for visually\-rich document understanding\.Online,pp\. 2579–2591\.External Links:[Link](https://aclanthology.org/2021.acl-long.201/),[Document](https://dx.doi.org/10.18653/v1/2021.acl-long.201)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px2.p1.1)\.
- Y\. Xu, T\. Lv, L\. Cui, G\. Wang, Y\. Lu, D\. A\. F\. Florêncio, C\. Zhang, and F\. Wei \(2021b\)LayoutXLM: multimodal pre\-training for multilingual visually\-rich document understanding\.ArXivabs/2104\.08836\.External Links:[Link](https://api.semanticscholar.org/CorpusID:233296696)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px2.p1.1)\.
- S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. Narasimhan \(2023\)Tree of thoughts: deliberate problem solving with large language models\.External Links:2305\.10601,[Link](https://arxiv.org/abs/2305.10601)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Yi, J\. Liu, T\. Xiao, and M\. V\. Albert \(2025\)A multi\-agent system for complex reasoning in radiology visual question answering\.External Links:2508\.02841,[Link](https://arxiv.org/abs/2508.02841)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- X\. Yu, C\. Xu, Z\. Chen, Y\. Zhang, S\. Lu, C\. Yang, J\. Zhang, S\. Yan, and X\. Hu \(2025\)Visual document understanding and reasoning: a multi\-agent collaboration framework with agent\-wise adaptive test\-time scaling\.External Links:2508\.03404,[Link](https://arxiv.org/abs/2508.03404)Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.
- B\. Zheng, B\. Gou, J\. Kil, H\. Sun, and Y\. Su \(2024\)GPT\-4v\(ision\) is a generalist web agent, if grounded\.Cited by:[§2](https://arxiv.org/html/2605.31354#S2.SS0.SSS0.Px3.p1.1)\.

Supplementary Material

## ASupplementary Tables

### A\.1ChartQAPro postprocessing robustness

We report raw vs postprocessed matching for ChartQAPro in Table[S1](https://arxiv.org/html/2605.31354#S1.T1)\.

Table S1:ChartQAPro robustness to answer\-format variation\.Postprocessing canonicalizes outputs before applying normalization\. We report postprocessed scores as a diagnostic check\.
### A\.2ChartQAPro subset composition

See Table[S2](https://arxiv.org/html/2605.31354#S1.T2)for the subset composition used in this paper\.

Table S2:ChartQAPro subset composition used in this paper\.

## BOutput Integrity Audit

#### Goal\.

Multi\-stage and multi\-agent settings may produce intermediate notes or trace\-like artifacts\. To ensure that all reported metrics are computed from valid final predictions, we audit every per\-example output and report invalid\-output statistics for each run\.

#### Answer extraction\.

Given a raw model outputyy, we extract a final answer stringa^\\hat\{a\}as follows: \(i\) if a Final Answer: field is present, we take the span following it until the end of the message; \(ii\) otherwise, we fall back to the last non\-empty line ofyy\(baseline compatibility\)\. We then strip leading/trailing whitespace and surrounding punctuation\.

#### Audit labels\.

We assign one of three audit labels based ona^\\hat\{a\}:okifa^\\hat\{a\}is non\-empty and answer\-like;missingif no parsable final answer is found ora^\\hat\{a\}is empty after stripping;note\-likeifa^\\hat\{a\}exhibits board\-note patterns rather than an answer\.

#### No\-exclusion scoring\.

All reported results include every example in the fixed subset\. We do not discard any examples during evaluation: if an output is labeled missing or note\-like, we seta^=∅\\hat\{a\}=\\emptysetand score it as an incorrect prediction under the corresponding metric\. Table[S3](https://arxiv.org/html/2605.31354#S2.T3)summarizes the audit statistics for all reported runs\.

Result file \(final\_raw\)𝒏nmissingnote\-likeshort\(≤\\leq2\)long\(≥\\geq200\)Dataset: SlideVQA \(Official Test Split\)baseline\_qwen\_single\_slidevqa\_test600001180single\_qwen\_board\_slidevqa\_test600001120cosee\_two\_qwen\_slidevqa\_test60000980baseline\_gemma\_single\_slidevqa\_test600002550single\_gemma\_board\_slidevqa\_test600002380cosee\_two\_gemma\_slidevqa\_test600002380baseline\_phi\_single\_slidevqa\_test60000720single\_phi\_board\_slidevqa\_test60000682cosee\_two\_phi\_slidevqa\_test60000545Dataset: ChartQAPro \(Test Split\)baseline\_qwen\_single\_chartqapro\_test800001350single\_qwen\_board\_chartqapro\_test\.merged80000560cosee\_two\_qwen\_chartqapro\_test80000710baseline\_gemma\_single\_chartqapro\_test800003120single\_gemma\_board\_chartqapro\_test\.merged800001420cosee\_two\_gemma\_chartqapro\_test800001850baseline\_phi\_single\_chartqapro\_test80000983single\_phi\_board\_chartqapro\_test\.merged80000425cosee\_two\_phi\_chartqapro\_test80000652Dataset: ChartQAPro \(200\-Subset for Three\-Agent CoSee\)cosee\_three\_qwen\_chartqapro\_test\_20020000220cosee\_three\_gemma\_chartqapro\_test\_20020000680cosee\_three\_phi\_chartqapro\_test\_20020000191Dataset: VQAonline \(Trainval\)baseline\_qwen\_single\_vqaonline\_trainval80000198single\_qwen\_board\_vqaonline\_trainval800003587cosee\_two\_qwen\_vqaonline\_trainval8000095baseline\_gemma\_single\_vqaonline\_trainval800002151single\_gemma\_board\_vqaonline\_trainval800001684cosee\_two\_gemma\_vqaonline\_trainval800001922baseline\_phi\_single\_vqaonline\_trainval800001245single\_phi\_board\_vqaonline\_trainval8000028156cosee\_two\_phi\_vqaonline\_trainval80000896Table S3:Output integrity audit for all reported runs\.This table covers every experimental setting reported in Table 2 \(main text\), including baselines, the single\_board pipeline, and multi\-agent variations\. We confirm output integrity for all runs \(missing=0, note\-like=0\), i\.e\., every example has a non\-empty final prediction and none is a board\-trace artifact\. Short/long counts are computed on the final prediction string \(pred\_answer\) using the model tokenizer\. Gemma\-3\-4B\-it indicates a tendency toward terse outputs in these settings \(higher counts in≤\\leq2\), particularly in difficult tasks\. Phi\-4\-multimodal\-instruct exhibits higher verbosity in open\-ended scenarios like VQAonline\. We observe a small number of unusually long generations in Phi\-4 on SlideVQA \(long≥\\geq200\); manual inspection confirms these are natural\-language answers rather than board traces\.

## CStatistical Stability and Counts

### C\.1Paired bootstrap confidence intervals

To assess statistical stability under fixed subsets, we report paired bootstrap confidence intervals for key comparisons in Table[S4](https://arxiv.org/html/2605.31354#S3.T4)\.

Table S4:Paired bootstrap confidence intervals for key comparisons \(Qwen3\-VL\-4B\)\.#### SlideVQA evaluation counts\.

To ensure strict accounting of the evaluation set, we report the exact numerator/denominator counts for our SlideVQA \(test,n=600n\{=\}600\) runs under the primary backbone Qwen3\-VL\-4B\. As reflected in the accuracy metrics in Table 2 \(main text\):

- •baseline single: 408/600 EM \(0\.680\)
- •single board: 411/600 EM \(0\.685\)
- •cosee two qwen: 342/600 EM \(0\.570\)

These counts confirm that all SlideVQA reported scores are computed on the fixed 600\-sample test subset under a fixed evaluation script and seed\.

## DOutput Length Analysis

### D\.1Final\-answer length statistics

We report final\-answer token length statistics in Table[S5](https://arxiv.org/html/2605.31354#S4.T5)\(GenTokens\(final\), computed under the Qwen tokenizer\)\.

Table S5:Observed final\-answer length statistics measured by GenTokens\(final\)\.Defined as the number of tokens in the extracted final answer string \(pred answer\) under the Qwen tokenizer\. Lengths may concentrate near the per\-call max\_new\_tokens due to truncation\. Importantly, all configurations use the same per\-call decoding cap \(max\_new\_tokens=32\)\.#### Observed final\-answer length statistics \(GenTokens\(final\)\)\.

Table[S5](https://arxiv.org/html/2605.31354#S4.T5)reports summary statistics of the extracted final\-answer length, measured byGenTokens​\(final\)\\mathrm\{GenTokens\(final\)\}, i\.e\., the number of Qwen\-tokenizer tokens in the final answer string \(pred answer\) under the same per\-call decoding cap \(max new tokens=32\) for all methods\. We report the mean, median \(P50\), 90th percentile \(P90\), and maximum \(Max\) to characterize both central tendency and tail behavior\. Because the per\-call cap is fixed across configurations, systematic shifts in these statistics primarily reflect protocol\-induced changes in output policy \(e\.g\., verbosity or coverage\), rather than differences in decoding budgets\. At the same time, concentration near the maximum value \(e\.g\., P50/P90 close to 32\) can indicate frequent truncation or a tendency to saturate the cap\.

Across SlideVQA and ChartQAPro, final answers are generally short \(P50≈\\approx3–5 tokens; P90≈\\approx8–16 tokens\), consistent with factoid\-style responses\. On ChartQAPro, ”Single\+Board” increases the typical answer length \(Mean: 5\.535→\\rightarrow9\.474; P50: 4→\\rightarrow8\), while ”CoSee” remains closer to the baseline range\. In contrast, VQAonline exhibits strong cap\-saturation under the single\-turn baseline \(P50=P90=32; Mean=27\.349\), whereas ”Single\+Board” shifts outputs toward substantially shorter answers \(Mean=19\.777; P50=21\), with ”CoSee” \(two qwen\) lying in between \(Mean=23\.480; P50=29\)\. These length\-distribution shifts motivate our length–quality diagnostics in Figure 3 \(main text\), which disentangle truncation effects from protocol\-induced output\-policy drift\.

### D\.2Holistic Cost Accounting: Visual vs\. Textual Compute

Our primary cost analysis \(Figure 4 in main text\) tracks generated tokens\. However, multi\-turn protocols also incur repeated visual encoding or cross\-attention costs for each new call\. LetCvisC\_\{\\text\{vis\}\}be the visual encoding cost andCtokC\_\{\\text\{tok\}\}be the per\-token generation cost\.

- •Baseline:Cost≈Cvis\+L⋅Ctok\\approx C\_\{\\text\{vis\}\}\+L\\cdot C\_\{\\text\{tok\}\}\.
- •Multi\-Agent \(NNturns\):Cost≈N⋅Cvis\+\(Lintermediate\+Lfinal\)⋅Ctok\\approx N\\cdot C\_\{\\text\{vis\}\}\+\(L\_\{\\text\{intermediate\}\}\+L\_\{\\text\{final\}\}\)\\cdot C\_\{\\text\{tok\}\}\.

SinceN≥2N\\geq 2for all collaborative protocols, including the fixed visual overhead significantly increases the total FLOPs for multi\-agent methods relative to the single\-turn baseline\. Consequently, if we plotted FLOPs\-based Pareto frontiers instead of token\-based ones, the collaborative methods would shift further to the right \(higher cost\), making the Efficiency Paradox even more pronounced\. The token\-only view thus serves as a conservative lower bound on the inefficiency of open\-loop collaboration\.

## ERobustness to Decoding Constraints

### E\.1Cap sensitivity on VQAonline \(max new tokens=128\)

Table S6:VQAonline results under larger per\-call caps\.Strt\./Rlxd\. denote Strict/Relaxed F1 \(≥\\geq0\.5/0\.3\)\. HitC\. reports the HitCap probability\.#### Summary\.

Table[S6](https://arxiv.org/html/2605.31354#S5.T6)shows that relaxing the per\-call cap \(32→\\rightarrow128\) modestly improves Token\-F1 for all protocols and greatly reduces truncation pressure \(HitCap drops to 3–6%\)\. However, the ordering remains similar: single\_board still underperforms baseline\_single, suggesting the degradation is not explained by truncation alone\.

### E\.2Prompt\-level mitigation for output\-policy drift on VQAonline

Table S7:Sanity check: length\-aware final\-answer prompting for VQAonline\.Strt\./Rlxd\. denote Strict/Relaxed F1 \(≥\\geq0\.5/0\.3\)\. HitC\. reports the HitCap probability\.#### Prompt delta \(final\-answer call only\)\.

We append the following constraint to the final\-answer instruction: Write a complete answer with sufficient coverage\. Use at least 6 bullet points \(or 80\+ tokens\) unless the question is inherently short\.

#### Takeaway\.

A minimal length/coverage constraint increases realized answer length and partially restores Token\-F1, consistent with the interpretation that single\_board degradation on VQAonline is largely driven by an output\-policy shift toward under\-coverage rather than truncation\.

### E\.3Stochastic decoding and majority voting

To address whether stochastic test\-time strategies can mitigate the fragility of weak learners, we conducted a targeted ablation on ChartQAPro \(Qwen3\-VL\-4B\) using two\_qwen with temperatureT=0\.7T=0\.7and majority voting overN=5N=5reasoning paths\.

Findings\.While voting yields a modest accuracy gain \(\+1\.2pp EM\), it incurs a linear increase in computational cost \(5×5\\timescalls\)\. Crucially, when plotted on the Cost–Accuracy Pareto frontier, this strategy is strictly dominated by the Verified\-Board control\. The stochastic ensemble consumes∼\\sim200 tokens/example to achieve parity with the deterministic baseline, whereas the Verifier achieves baseline parity with only∼\\sim25 additional tokens\. This reinforces our ”Efficiency Paradox” conclusion: for weak learners, blind over\-generation \(quantity\) is less effective than targeted gating \(quality\)\.

## FChartQAPro Analysis and Verification

### F\.1ChartQAPro answerability and formatting controls

We report Answerable\-Only and post\-processed metrics for ChartQAPro in Table[S8](https://arxiv.org/html/2605.31354#S6.T8)\.

Table S8:Additional ChartQAPro metrics\.We report Answerable\-Only \(raw\) and Post\-Processed \(relaxed formatting\) metrics to reduce formatting noise\. Bold values denote the best\-performing protocol for each backbone within each metric block\. The relative drop of single\-board and two\-agent protocols versus the single\-turn baseline persists at both 4B and 8B scales \(e\.g\., on 8B, single\-board drops from 0\.225→\\rightarrow0\.150 EM even under post\-processing\), suggesting the degradation is not solely attributable to formatting strictness\.#### Additional ChartQAPro metrics and robustness to formatting\.

Table[S8](https://arxiv.org/html/2605.31354#S6.T8)provides two complementary evaluation views on ChartQAPro for both backbones \(Qwen3\-VL\-8B\-Instruct and Qwen3\-VL\-4B\-Instruct\)\. First, we report*Answerable\-Only \(Raw\)*scores computed on the answerable subset \(N=650N\{=\}650\), which reduces confounds from unanswerable questions and isolates performance when a chart\-grounded numeric/textual answer is expected\. Second, we report*Post\-Processed \(Relaxed Formatting\)*scores on the full test subset \(N=800N\{=\}800\), where predictions and references are normalized to mitigate superficial formatting mismatches\. Within each metric block, boldface indicates the best\-performing protocol for a given backbone\.

Across both backbones, the single\-turn baseline remains the strongest configuration\. The degradation underSingle Boardpersists even after controlling for answerability and formatting noise\. Overall, these breakdowns strengthen the main claim that naive board usage can amplify chart mis\-grounding and harm end\-task accuracy, and that the effect persists at both 4B and 8B scales under relaxed evaluation\.

### F\.2ChartQAPro verified\-board control

We report a minimal verified\-board control on ChartQAPro in Table[S9](https://arxiv.org/html/2605.31354#S6.T9)\.

Table S9:Verified\-board control on ChartQAPro \(Qwen3\-VL\-4B\)\.A minimal verification step filters unsupported board notes before final answer generation\. The verified control recovers most of the naive single\-board drop in EM across raw/answerable/post\-processed metrics, suggesting that the degradation is largely driven by ungrounded intermediates rather than the board mechanism itself\.#### Verified\-board control\.

To test whether the ChartQAPro degradation is driven by ungrounded intermediate notes rather than the board mechanism itself, we implement a minimal verified\-board variant for the single\-agent setting\. Concretely, the model first proposes a short board note, and then runs an additional verification call that checks whether the proposed note is directly supported by the input chart image and question\.

Verifier Calibration Analysis\.A critical question is why the verifier helps despite being the same weak learner \(Qwen3\-VL\-4B\)\. We analyzed the verifier’s decision matrix on a manually annotated subset of 100 note/chart pairs:

- •Precision \(Safety\):The verifier achieves high precision \(92%92\\%\) for theSUPPORTEDclass\. This implies that when a note is accepted, it is highly likely to be grounded\.
- •Recall \(Conservatism\):The recall is moderate \(64%64\\%\)\. The verifier tends to reject ambiguous or complex reasoning steps \(false negatives\)\.

This conservative calibration acts as ahigh\-pass filterfor grounding\. In the context of noise accumulation, a False Negative \(rejecting a true note\) forces the model to fall back to the safe baseline behavior \(direct answering\), whereas a False Positive \(accepting a hallucination\) leads to error reinforcement\. Thus, the verifier succeeds by effectively closing the ”open loop” of error propagation\.

Protocol Details\.The verifier outputs one label fromSUPPORTED/UNSUPPORTED/UNCERTAIN\. We apply a deterministic gate:keep​\(n\)=𝟙​\[verifier​\(I,q,n\)=SUPPORTED\]\\text\{keep\}\(n\)=\\mathbb\{1\}\[\\texttt\{verifier\}\(I,q,n\)=\\texttt\{SUPPORTED\}\]\. This choice intentionally biases toward precision\. As shown in Table[S9](https://arxiv.org/html/2605.31354#S6.T9), this minimal verification largely recovers the naive single\-board drop across raw, answerable\-only, and post\-processed metrics\.

#### Scope of Verification \(Why ChartQAPro?\)\.

We targeted the verified\_board intervention specifically at ChartQAPro because our failure taxonomy identified Type T2 \(Noise Reinforcement\) as the dominant failure mode there\. Verification is well\-defined for charts \(checking a specific value against an image\)\. In contrast:

- •VQAonline: The primary failure is Type T4 \(Policy Collapse\), which is a deficiency in output style/coverage rather than factual correctness\. A simple ”supported/unsupported” verifier does not address coverage issues\.
- •SlideVQA: This is a high\-recall retrieval task\. Effective verification would require re\-scanning multiple images to confirm a negative, which is computationally prohibitive and functionally redundant with the retrieval step itself\.

Thus, while verification is a powerful tool for arresting hallucination \(ChartQAPro\), different mechanisms \(e\.g\., constrained decoding\) are required for policy\-drift failures \(VQAonline\)\.

## GTwo\-stage Baseline \(Diagnostic\)

### G\.1Two\-stage evidence→\\rightarrowanswer baseline

This appendix specifies the two\-stage single\-agent baseline reported in Table 2 \(marked‡\)\. The goal is to isolate the effect of multi\-step test\-time prompting from shared\-board collaboration: the baseline uses no board and no inter\-agent communication, but introduces an additional evidence\-oriented call before the final answer\. All evaluation uses the same output parsing and no\-exclusion scoring policy as the main runs\.

#### Protocol\.

Given an instance\(I,q\)\(I,q\), the two\-stage baseline makes exactly two model calls: \(1\) anevidence stagethat extracts a compact, checkable evidence summary; and \(2\) afinal\-answer stagethat produces the scored prediction conditioned on\(I,q\)\(I,q\)and the evidence summary\.

#### Backbone and decoding\.

We use the same backbone as the corresponding main setting\. Both stages use deterministic decoding \(temperature=0=0\) and the same per\-call decoding cap \(max\_new\_tokens=32=32\)\.

#### Prompts\.

Stage 1: Evidence extraction prompt \(no answer\)\.

```
Task: extract evidence only.
Do NOT answer the question.
Output format (strict):
EVIDENCE:
- <bullet 1>
- <bullet 2>
```

Stage 2: Final answer prompt \(answer only\)\.

```
EVIDENCE SUMMARY: {EVIDENCE_FROM_STAGE_1}
Answer the question using ONLY the evidence
summary and what is visible in the image.
Output format (strict):
FINAL: <answer>
```

## HAdditional Qualitative Evidence

### H\.1Case Studies

#### Case studies\.

Figure[S1](https://arxiv.org/html/2605.31354#S8.F1)provides representative examples for the two dominant failure mechanisms surfaced by our taxonomy\. On VQAonline, thesingle\_boardsetting often collapses to overly terse final outputs \(T4\), consistent with the strong coupling between answer length and token\-F1\. On ChartQAPro, intermediate notes can be mis\-grounded \(T2\): the shared board records a salient statistic but misinterprets the variable \(e\.g\., rate vs\. count\), hardening the error\.

![Refer to caption](https://arxiv.org/html/2605.31354v1/x6.png)

Case A: VQAonline \(T4\) Failure: Output\-policy drift Query:Solutions ofx2=x\+1x^\{2\}=x\+1inℤ/n​ℤ\\mathbb\{Z\}/n\\mathbb\{Z\}\.Gold:Long\-form derivation/characterization\.Pred:B\(near\-empty\)\.Mechanism:Intermediate notes do not translate into final coverage\.

![Refer to caption](https://arxiv.org/html/2605.31354v1/)

Case B: ChartQAPro \(T2\) Failure: Mis\-grounded intermediate Query:*Approx\. \# people who strongly favor the death penalty \(round down\)\.*Gold:1379Pred:27\.Board:“27% strongly favor” \(percentage\)\.Mechanism:The board captures the wrong variable \(rate vs\. count\), and the final answer inherits it\.

Figure S1:Qualitative analysis of dominant failure mechanisms\.Top \(Case A\):On VQAonline, the presence of board context induces a policy collapse, shifting the model toward overly terse responses \(Type T4\)\.Bottom \(Case B\):On ChartQAPro, the model hallucinates a unit \(treating percentage as count\) and writes it to the board, reinforcing the error in the final step \(Type T2\)\.

## IFailure Taxonomy

### I\.1Sampling and annotation protocol

For each dataset and method, we construct an error pool from the fixed evaluation subset\. From each pool, we uniformly sample100100error examples*without replacement*to form the taxonomy set\. Each case includes the input image, question, final prediction, and full trace artifacts\.

#### Labeling procedure\.

Each case is assigned exactly one dominant failure type from T0–T6 using a deterministic rubric, prioritizing the*earliest causal mechanism*: \(1\) parsing/format failure \(T0\); \(2\) artifact leakage \(T1\); \(3\) board\-amplified wrong note \(T2\); \(4\) attention dilution \(T3\); \(5\) output\-policy drift/under\-coverage \(T4\); \(6\) decision failure \(T5\); \(7\) residual perception error \(T6\)\.

### I\.2Failure type definitions

#### T0: Missing or invalid final answer\.

The pipeline fails to extract a valid final prediction string \(e\.g\., empty string\)\. Scored as incorrect\.

#### T1: Note/trace leakage\.

The final answer is dominated by intermediate artifacts \(e\.g\.,Board:,Note:\) rather than an actual response\.

#### T2: Board\-amplified wrong intermediate \(Noise Reinforcement\)\.

A wrong intermediate note is written to the board and “hardened” into the final answer\. Example: Chart shows 50 \(count\), Note says ”50%”, Final Answer says ”50%”\.

#### T3: Attention dilution\.

The trace shows agents attending to irrelevant evidence or conflicting notes without convergence, resulting in inconsistent reasoning\.

#### T4: Output\-policy drift \(Policy Collapse\)\.

The final answer is materially under\-specified relative to the question, driven by a shift toward overly short responses \(common on VQAonline\)\.

#### T5: Decision failure\.

Mishandling of answerability \(e\.g\., answering an unanswerable question or vice versa\)\.

#### T6: Residual perception error\.

Standard OCR or grounding failures not specific to the protocol \(no note leakage or policy drift\)\.

## JImplementation Details and Prompts

### J\.1Reproducibility Statement

\(Response to reviewer inquiry on replication\)To enable full third\-party replication and auditing of weak\-learner dynamics, we will release the following artifacts upon publication:

- •Codebase:Full PyTorch/HuggingFace implementation of theCoSeecontroller, agent roles, and evaluation pipeline\.
- •Trace Logs:The complete JSONL logs for all main experiments \(approx\. 200MB\), including every board state, intermediate note, and audit flag\. This allows future work to analyze error dynamics without re\-running inference\.
- •Verified Exemplars:The specific subset of board notes accepted vs\. rejected by the verifier, serving as a dataset for studying hallucination detection in charts\.
- •Subset IDs:The exact example IDs used for the fixed 600/800\-sample evaluation subsets\.

### J\.2Prompt Templates

We use a unified prompt structure across all backbones \(Qwen3, Phi\-4, Gemma\-3\)\. The system instruction is prepended to the user query\.

#### Board\-Writing Prompt \(Scanner / Single\-Agent\)\.

Used for generating intermediate notes\.

```
You are a helpful assistant.
IMAGE: [Image Features]
BOARD STATE: [Previous Notes (if any)]
QUESTION: {question}
INSTRUCTION: Identify specific visual
evidence relevant to the question.
Write a concise note. Do not answer
the question yet.
FORMAT: "Note: <content>"
```

#### Verification Prompt \(Verified\-Board\)\.

Used for the S6\.2 ChartQAPro control\.

```
Task: Verify if the claim is directly
supported by the image.
CLAIM: {note_content}
IMAGE: [Image Features]
QUESTION: {question}
INSTRUCTION: Look at the image.
Is the claim supported by visible
evidence?
Output strictly one token: SUPPORTED,
UNSUPPORTED, or UNCERTAIN.
```

#### Final Answer Prompt\.

Used for all protocols \(Single\-Board, CoSee, Baseline\) to generate the final prediction\.

```
You are a helpful assistant.
IMAGE: [Image Features]
BOARD HISTORY:
{board_summary}
QUESTION: {question}
INSTRUCTION: Answer the question based
on the image and the board history.
If the board contains errors, prioritize
the image.
FORMAT: "Final Answer: <answer>"
```

### J\.3Analysis of Grounding Pointers

\(Response to reviewer inquiry on pointer usage\)A structural limitation of the weak learners evaluated \(4B–8B parameters\) is their struggle to produce precise visual coordinates \(bounding boxes\) even when explicitly prompted\. In our qualitative audit of thesingle\_boardtraces on ChartQAPro:

- •Pointer Scarcity:Valid grounding pointers \(e\.g\., box coordinates matching a chart element\) appeared in less than5%5\\%of intermediate notes\.
- •Hallucinated Precision:When forced to output coordinates \(via a ”pointer\-required” ablation prompt\), the models frequently generated boxes covering the entire image or irrelevant regions, which did not mitigate T2 errors\.

This observation explains the high prevalence ofType T2 \(Board\-Amplified Error\)failures: without reliable geometric grounding, the board relies solely on textual semantic matching, which is prone to hallucination\. This negative result motivates our specific design of theverified\_boardcontrol \(Section[F\.2](https://arxiv.org/html/2605.31354#S6.SS2.SSS0.Px1)\), which uses an image\-to\-text verification step as a proxy for missing geometric grounding\.