TestHallVQA: Exploring LVLMs' Document-Level Reasoning under Redundant Contexts from Scientific Exams

# TestHallVQA: Exploring LVLMs’ Document-Level Reasoning under Redundant Contexts from Scientific Exams
Source: [https://arxiv.org/html/2609.13158](https://arxiv.org/html/2609.13158)
Yongqi Yu Harbin Institute of Technology Harbin, China yqyu@ir\.hit\.edu\.cn [ORCID: 0009\-0005\-0674\-788X](https://orcid.org/0009-0005-0674-788X)&Yu Zhang Harbin Institute of Technology Harbin, China zhangyu@ir\.hit\.edu\.cn [ORCID: 0000\-0003\-3090\-7431](https://orcid.org/0000-0003-3090-7431)

###### Abstract

Large Vision–Language Models \(LVLMs\) are increasingly expected to perform visual question answering \(VQA\) over planar media\. However, existing planar VQA benchmarks typically emphasize isolated challenges: some emphasize long\-document understanding with limited reasoning depth, while others require complex visual reasoning but remain restricted to single\-page, noise\-free settings\. Moreover, through theoretical analysis, we identify the impact of irrelevant visual tokens, which leads to measurable performance degradation but has received little attention with respect to systematic quantification\. To address these limitations, we introduce TestHallVQA, a multi\-image VQA benchmark that simultaneously embodies document\-level scale and the difficulty of human examinations, while providing comprehensive task coverage\. Leveraging TestHallVQA’s ability to controllably inject multi\-level contextual redundancy, we further propose a novel metric, F1\-R2, which jointly quantifies LVLMs’ computational reasoning capability and their evidence retrieval robustness against document\-level redundancy\. Extensive experiments and analyses on mainstream LVLMs reveal their latent deficiencies across multiple dimensions, offering concrete insights and directions for future research\. The associated datasets, code, and complete theoretical derivations are available at[https://github\.com/yqyu2317/TestHallVQA\-benchmark](https://github.com/yqyu2317/TestHallVQA-benchmark)\.

*K*eywordsDocument Visual Question Answering⋅\\cdotMultimodal Reasoning⋅\\cdotLarge Vision\-Language Models

## 1Introduction

With the maturation of cross\-modal alignment techniques, large vision\-language models \(LVLMs\) have emerged as a dominant paradigm for extending large language models \(LLMs\)\[[47](https://arxiv.org/html/2609.13158#bib.bib72)\]with visual perception capabilities for multimodal tasks\. Meanwhile, planar media—including books, academic papers, and reports—have long served as fundamental carriers of human knowledge dissemination\[[17](https://arxiv.org/html/2609.13158#bib.bib71)\]\. Accordingly, enabling LVLMs to process planar visual content efficiently has become an important research direction\[[39](https://arxiv.org/html/2609.13158#bib.bib54),[16](https://arxiv.org/html/2609.13158#bib.bib67),[52](https://arxiv.org/html/2609.13158#bib.bib70),[60](https://arxiv.org/html/2609.13158#bib.bib64),[33](https://arxiv.org/html/2609.13158#bib.bib63)\], aiming to reduce the repetitive manual effort required for document understanding and management\.

Unlike natural scene images, planar data constitute a prototypical form ofmultimodal media, in which textual and visual elements—including illustrations, charts, spatial layouts, and other structural components—are tightly intertwined\. This intrinsic heterogeneity places substantial demands on LVLMs’ fine\-grained visual perception and textual understanding\.

Current planar visual question answering \(VQA\) benchmarks can be broadly categorized into two groups:

\(1\) Multi\-Page Document VQA:These datasets are constructed from text\-centric sources, including documents\[[52](https://arxiv.org/html/2609.13158#bib.bib70),[50](https://arxiv.org/html/2609.13158#bib.bib69),[35](https://arxiv.org/html/2609.13158#bib.bib68),[16](https://arxiv.org/html/2609.13158#bib.bib67)\], academic papers\[[15](https://arxiv.org/html/2609.13158#bib.bib66)\], and web pages\[[49](https://arxiv.org/html/2609.13158#bib.bib65)\]\. As illustrated in Figure[1](https://arxiv.org/html/2609.13158#S1.F1)a, they pose dual challenges in contextual scale and layout complexity, requiring cross\-page and cross\-element interactions\. However, they remain limited in capturing the depth of complex reasoning\. Consequently, they primarily evaluate a model’s*retrieval*capability\.

\(2\) Single\-Question Scientific VQA:In contrast, this category focuses on highly abstract and challenging planar visuals, such as geometric diagrams and chemical structures\[[60](https://arxiv.org/html/2609.13158#bib.bib64),[33](https://arxiv.org/html/2609.13158#bib.bib63),[4](https://arxiv.org/html/2609.13158#bib.bib61)\]\. As illustrated in Figure[1](https://arxiv.org/html/2609.13158#S1.F1)b, each sample typically consists of a single small image in which the relevant evidence is presented explicitly\. Without the information density and contextual complexity of document\-level inputs, these samples constitute a self\-contained and relatively noise\-free setting\. Consequently, such benchmarks evaluate only a model’s*reasoning*capability\.

However, in real\-world deployments, critical information is often buried in long and highly redundant contexts\. For example, complex reasoning over dozens of pages of documents is a common practical requirement\.Models cannot know*a priori*which parts of the context will be relevant, and users likewise cannot feasibly identify all supporting evidence manually\.Existing benchmarks, however, either lack sufficient reasoning depth or assume an overly clean context\. These two settings are largely orthogonal, each capturing only a limited aspect of real\-world complexity\.

On the other hand, we theoretically analyze the effect of redundant visual tokens on information flow in LVLM decoders, showing that such redundant tokens contaminate relevant tokens with an exponentially increasing effect across layers\. We term this phenomenonredundant visual contamination\.

![Refer to caption](https://arxiv.org/html/2609.13158v1/x1.png)Figure 1:Illustration of the siloed status of existing planar\-media VQA benchmarks\. TestHallVQA unifies the strengths of both sides, providing a comprehensive challenge\.Motivated by the foregoing practical and theoretical considerations, we seek a data source that simultaneously embodies broad contextual scope and substantial reasoning depth, and identify*illustrated examination papers*as an ideal candidate\. To this end, we curate a large collection of exam papers and develop a semi\-automated annotation pipeline to construct a document\-level scientific examination benchmark comprising over 10,000 question–answer pairs and 7,000 paper images\. We term this datasetTestHallVQA\(Figure[1](https://arxiv.org/html/2609.13158#S1.F1)c\), as it places LVLMs in an environment analogous to a real\-world examination hall, targets more realistic evaluation settings, and introduces challenges for document\-level deep reasoning\. Furthermore, to investigate how LVLM performance degrades with increasing redundant context, we equip TestHallVQA with a controlled mechanism for injecting multi\-level, highly confounding redundant image pages, thereby inducing smoothly varying contextual*regimes*for each question\.

Building on this design, we introduceF1\-R2\(F1score onRetrieval–Reasoning\), a unified evaluation metric that jointly integrates the assessment of visual reasoning ability and robustness to redundant visual distractions, rather than decoupling them as in conventional benchmarks\. The contributions of this work are threefold:

- •We provide preliminary theoretical evidence for the existence of“redundant visual contamination”in LVLMs—a phenomenon that is not captured or diagnosed by mainstream VQA benchmarks\.
- •We introduceTestHallVQA, a multi\-image VQA benchmark that not only poses substantial visual and computational challenges, but also supports the controlled injection of document\-level visual redundancy, thereby more faithfully reflecting real\-world application scenarios\.
- •We propose a novel metric,F1\-R2, which jointly quantifies LVLMs’ problem\-solving capability \(Reasoning\) and their robustness to redundant visual contamination \(Retrieval\), moving beyond single\-dimensional evaluation\.

## 2Related Work

Existing VQA tasks can be broadly grouped into real\-world scenes, 3D simulated scenes, and planar\-media scenes\. The first two, as projections or simulations of the physical world, have abundant annotations and are well\-established in prior work\[[2](https://arxiv.org/html/2609.13158#bib.bib58),[36](https://arxiv.org/html/2609.13158#bib.bib57),[20](https://arxiv.org/html/2609.13158#bib.bib56),[24](https://arxiv.org/html/2609.13158#bib.bib55),[3](https://arxiv.org/html/2609.13158#bib.bib21)\]\.

Beyond real scenes, planar\-media VQA has gained increasing attention\. Originating from DocVQA\[[51](https://arxiv.org/html/2609.13158#bib.bib51),[40](https://arxiv.org/html/2609.13158#bib.bib50)\], exemplified by SingleDocVQA\[[39](https://arxiv.org/html/2609.13158#bib.bib54)\], this line introduced document\-page datasets as a new planar medium\. VisualMRC\[[49](https://arxiv.org/html/2609.13158#bib.bib65)\]scaled this direction with web\-based images, while InfographicsVQA\[[38](https://arxiv.org/html/2609.13158#bib.bib49)\]and ChartQA\[[37](https://arxiv.org/html/2609.13158#bib.bib53)\]expanded to infographics and chart images requiring structured visual\-semantic reasoning, yet early work was limited to single\-page contexts until DUDE\[[52](https://arxiv.org/html/2609.13158#bib.bib70)\]and MP\-DocVQA\[[50](https://arxiv.org/html/2609.13158#bib.bib69)\]introduced multi\-page inputs demanding cross\-page reasoning\. MMLongBench\[[35](https://arxiv.org/html/2609.13158#bib.bib68)\]further extended the context length \(average 47\.5 pages\), while LongDocURL\[[16](https://arxiv.org/html/2609.13158#bib.bib67)\]advanced this to approximately 85 pages with denser cross\-element dependencies\. Nevertheless, they remain text\-dominant, emphasizing information redundancy over genuine reasoning depth, with most answers attainable through shallow inference rather than true chain\-of\-thought \(CoT\)\[[56](https://arxiv.org/html/2609.13158#bib.bib22)\]reasoning\.

Another line of planar\-media VQA centers on deep reasoning\. Early studies addressed purely textual corpora \(e\.g\., MATH\[[21](https://arxiv.org/html/2609.13158#bib.bib48)\], GSM8K\[[11](https://arxiv.org/html/2609.13158#bib.bib47)\]\), later extending to visually grounded reasoning\. GeoQA\[[10](https://arxiv.org/html/2609.13158#bib.bib46)\], Geometry3K\[[34](https://arxiv.org/html/2609.13158#bib.bib44)\], and UniGeo\[[9](https://arxiv.org/html/2609.13158#bib.bib45)\]targeted geometric problem solving; MathVista\[[33](https://arxiv.org/html/2609.13158#bib.bib63)\]broadened scope to multimodal mathematics\.Zhanget al\.\[[63](https://arxiv.org/html/2609.13158#bib.bib62)\], Liuet al\.\[[32](https://arxiv.org/html/2609.13158#bib.bib43)\]identified redundancy between textual and visual content\. MATH\-Vision\[[53](https://arxiv.org/html/2609.13158#bib.bib42)\]enriched task and domain diversity, while WeMath\[[46](https://arxiv.org/html/2609.13158#bib.bib41)\]introduced process\-level evaluation\. Other mathematical datasets\[[48](https://arxiv.org/html/2609.13158#bib.bib40),[30](https://arxiv.org/html/2609.13158#bib.bib39),[65](https://arxiv.org/html/2609.13158#bib.bib37),[14](https://arxiv.org/html/2609.13158#bib.bib36)\]exhibit complementary strengths\. Beyond mathematics, LogicVista\[[57](https://arxiv.org/html/2609.13158#bib.bib38)\]examined logical inference over varied planar imagery, and MMMU\[[60](https://arxiv.org/html/2609.13158#bib.bib64)\], MMMU\-Pro\[[61](https://arxiv.org/html/2609.13158#bib.bib35)\], SceMQA\[[31](https://arxiv.org/html/2609.13158#bib.bib59)\]expanded to broader scientific reasoning\. Despite their diversity, these benchmarks assume a noise\-free setting with one question per image and no redundant or distracting information, which differs substantially from real\-world redundant and complex scenarios\.

## 3TestHallVQA Benchmark

In this section, we provide a comprehensive overview of the proposed TestHallVQA benchmark\.

### 3\.1Theoretical Motivation: The Phenomenon of Redundant Visual Contamination

Due to attention interactions during autoregressive decoding in LVLMs, the introduction of redundant images causes irrelevant token contributions to be mixed into the hidden states of relevant tokens, thereby leading to representation contamination\. To characterize how redundant information perturbs the reasoning process of LVLMs, we employa symbolization\-based approximate algorithmto provide a preliminary theoretical investigation of its overall trend\. Here, we focus exclusively on the most common self\-attention decoder architecture as our analytical model; the derivations for other architectural variants are largely analogous and can be transferred straightforwardly\.

Let the model receiveMMrelevant tokens in the clean setting, including the question text and evidence images, andM\+NM\+Ntokens whenNNirrelevant visual tokens are introduced\. For notational convenience, we index the firstMMtokens as relevant and the remainingNNtokens as irrelevant\. We consider the attention layer within the decoder block, which serves as the primary module for token interactions across different positions\. Such redundancy will disperse attention from relevant to irrelevant tokens and dilute effective information\. Lethxh\_\{x\}denote the hidden\-state representation of thexx\-th token in the clean setting\. WhenNNirrelevant visual tokens are introduced, the corresponding hidden state is denoted byhx′h^\{\\prime\}\_\{x\}:

hx=\\displaystyle h\_\{x\}=h^x\+α1​V1\+α2​V2\+…\+αM​VM=h^x\+∑i=1Mαi​Vi,\\displaystyle\\hat\{h\}\_\{x\}\+\\alpha\_\{1\}V\_\{1\}\+\\alpha\_\{2\}V\_\{2\}\+\\ldots\+\\alpha\_\{M\}V\_\{M\}=\\hat\{h\}\_\{x\}\+\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V\_\{i\},\(1\)hx′=\\displaystyle h^\{\\prime\}\_\{x\}=h^x′\+β1​V1′\+β2​V2′\+…\+βM\+N​VM\+N′=h^x′\+∑j=1M\+Nβj​Vj′,\\displaystyle\\hat\{h\}^\{\\prime\}\_\{x\}\+\\beta\_\{1\}V^\{\\prime\}\_\{1\}\+\\beta\_\{2\}V^\{\\prime\}\_\{2\}\+\\ldots\+\\beta\_\{M\+N\}V^\{\\prime\}\_\{M\+N\}=\\hat\{h\}^\{\\prime\}\_\{x\}\+\\sum\_\{j=1\}^\{M\+N\}\\beta\_\{j\}V^\{\\prime\}\_\{j\},\(2\)
whereh^x\\hat\{h\}\_\{x\}andh^x′\\hat\{h\}^\{\\prime\}\_\{x\}denote the outputs from the previous layer \(due to the presence of residual connections\),αi\\alpha\_\{i\}andβi\\beta\_\{i\}denote the attention weights, andVi,Vi′∈ℝdV\_\{i\},V^\{\\prime\}\_\{i\}\\in\\mathbb\{R\}^\{d\}are the*value*vectors at theii\-th position\.

After redundant tokens are introduced, the attention mass originally allocated to theMMrelevant tokens is diluted and redistributed across theNNirrelevant tokens\. The average degree of attention dispersion can be expressed as

βi=δ​MM\+N​αi,\(i=1,2,…,M\),\\beta\_\{i\}=\\frac\{\\delta M\}\{M\+N\}\\alpha\_\{i\},\\quad\(i=1,2,\\ldots,M\),\(3\)whereδ\\deltadenotes the harmonization coefficient used in our approximate algorithm\. Here, the dispersion ratio is positively correlated withMM\+N\\frac\{M\}\{M\+N\}\. However, attention generally remains concentrated on the relevant tokens, so the actual degree of dispersion is typically smaller thanMM\+N\\frac\{M\}\{M\+N\}, implying thatδ\>1\\delta\>1\. Therefore, Eq\. 2 can be rewritten as follows:

hx′=h^x′\+δ​MM\+N​∑i=1Mαi​Vi\+∑j=M\+1M\+Nβj​Vj′\.h^\{\\prime\}\_\{x\}=\\hat\{h\}^\{\\prime\}\_\{x\}\+\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V\_\{i\}\+\\sum^\{M\+N\}\_\{j=M\+1\}\\beta\_\{j\}V^\{\\prime\}\_\{j\}\.\(4\)
We next compute theInformation Purityη\\etaof the hidden state of the token at thexx\-th position, defined as the proportion of its values attributable to relevant content, in order to quantify the contamination introduced by redundant tokens\. The purity is formulated as:

η=hx′−∑j=M\+1M\+Nβj​Vj′hx′=h^x′\+δ​MM\+N​∑i=1Mαi​Vi′h^x′\+δ​MM\+N​∑i=1Mαi​Vi′\+∑j=M\+1M\+Nβj​Vj′=1\+h^x′δ​MM\+N​∑i=1Mαi​Vi′1\+h^x′δ​MM\+N​∑i=1Mαi​Vi′\+M\+Nδ​M⋅∑j=M\+1M\+Nβj​Vj′∑i=1Mαi​Vi′\.\\begin\{split\}\\eta&=\\frac\{h^\{\\prime\}\_\{x\}\-\\sum^\{M\+N\}\_\{j=M\+1\}\\beta\_\{j\}V^\{\\prime\}\_\{j\}\}\{h^\{\\prime\}\_\{x\}\}=\\frac\{\\hat\{h\}^\{\\prime\}\_\{x\}\+\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\}\{\\hat\{h\}^\{\\prime\}\_\{x\}\+\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\+\\sum^\{M\+N\}\_\{j=M\+1\}\\beta\_\{j\}V^\{\\prime\}\_\{j\}\}\\\\ &=\\frac\{1\+\\frac\{\\hat\{h\}^\{\\prime\}\_\{x\}\}\{\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\}\}\{1\+\\frac\{\\hat\{h\}^\{\\prime\}\_\{x\}\}\{\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\}\+\\frac\{M\+N\}\{\\delta M\}\\cdot\\frac\{\\sum^\{M\+N\}\_\{j=M\+1\}\\beta\_\{j\}V^\{\\prime\}\_\{j\}\}\{\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\}\}\.\\\\ \\end\{split\}\(5\)
Here,∑j=M\+1M\+Nβj​Vj′∑i=1Mαi​Vi′\\frac\{\\sum\_\{j=M\+1\}^\{M\+N\}\\beta\_\{j\}V^\{\\prime\}\_\{j\}\}\{\\sum\_\{i=1\}^\{M\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\}denotes the ratio between the weighted value sum of theNNredundant tokens and that of theMMrelevant tokens\. In the extreme case, this ratio approachesNM\\frac\{N\}\{M\}\. However, since relevant tokens typically receive larger attention weights than irrelevant ones, it cannot fully attain this upper bound\. We therefore introduce a harmonization coefficientλ∈\(0,1\)\\lambda\\in\(0,1\)to capture this bias, yieldingλ​NM\\frac\{\\lambda N\}\{M\}\. Accordingly, Eq\. 5 can be simplified as:

η=1\+h^x∑i=1Mβi​Vi′1\+h^x∑i=1Mβi​Vi′\+M\+Nδ​M⋅λ​NM∈\(0,1\),δ\>1,λ∈\(0,1\)\.\\begin\{split\}\\eta=\\frac\{1\+\\frac\{\\hat\{h\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}\}\{1\+\\frac\{\\hat\{h\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}\+\\frac\{M\+N\}\{\\delta M\}\\cdot\\frac\{\\lambda N\}\{M\}\}\\in\(0,1\),\\quad\\delta\>1,\\lambda\\in\(0,1\)\.\\end\{split\}\(6\)
At this point, we have derived the attenuation factor of information purity\. However, due to space limitations, the discussion above presents only the central intuition of the analysis\. For interested readers, the complete derivations beyond this point are provided in Appendix[A](https://arxiv.org/html/2609.13158#A1)\. Accordingly, we omit the technical details in the main text and state the final conclusion directly:

Information purity, denoted byη\\eta, is recursively compositional\. Specifically, after each block, the information purity is scaled by a factor ofη\\etarelative to that of the preceding layer\. Consequently, for anLL\-layer decoder, the information purity of the output tokens decreases toηL\\eta^\{L\}, implying an exponential compression of informative content\. Moreover, as further verified in Appendix[A](https://arxiv.org/html/2609.13158#A1), a larger ratioNM\\frac\{N\}\{M\}results in lower information purity\.

This layer\-wise proliferative contamination implies that, even whenη\\etais very close to11, the resulting exponential decay can still cause substantial degradation\. Such contamination weakens the model’s ability to capture informative signals and may ultimately impair vocabulary decoding and text generation\. The experiments presented in this paper further demonstrate the significant impact of this phenomenon on model performance\.

Given the inevitability of redundancy in real\-world data, we argue that greater emphasis should be placed on an LVLM’s ability to sustain reasoning performance under redundant conditions, rather than under unrealistically clean settings, as an extended evaluation criterion\. Our benchmark is motivated precisely by this perspective\.

### 3\.2Data Collection and Annotation

![Refer to caption](https://arxiv.org/html/2609.13158v1/x2.png)Figure 2:Data annotation process\. An approach combining automation and manual efforts is used to balance efficiency and reliability\.We seek a data source that combines reasoning complexity with large\-scale document context, for which multi\-page examination papers are a natural choice\. Such documents feature diverse textual layouts and planar visual elements, requiring models to possess strong retrieval capabilities to identify relevant evidence, while the questions themselves demand nontrivial, human\-level reasoning\.

Focusing on three reasoning\-intensive disciplines—mathematics, physics, and chemistry—we collected 995 exam PDFs from a publicly accessible website111[https://www\.exam\-mate\.com](https://www.exam-mate.com/)\. After excluding exams with limited visual content or only elementary\-level difficulty, we retained 382 exams and their answer booklets\. We then adopted a hybrid annotation strategy that combines manual annotation with LLM\-assisted annotation to balance efficiency and quality control, as illustrated in Figure[2](https://arxiv.org/html/2609.13158#S3.F2)\.

Specifically, we first convert each PDF file into images to preserve complete visual information\. We then use GPT\-5\[[44](https://arxiv.org/html/2609.13158#bib.bib16)\]to automatically remove blank or irrelevant pages, retaining only the core examination content and answer tables\. Next, each answer\-table page is processed by GPT\-5 with layout\-specific restrictive prompts, which instruct it to ignore distracting elements and generate a JSON object containing question indices and their corresponding answer text\. We then manually complete missing cross\-page question indices, decompose multi\-part answers into structured lists, separate solution procedures from final answers, restore omitted text, and normalize all annotations into a standardized LaTeX format\. At this stage, GPT\-5 is used under strict constraints solely for text extraction as a format converter, thereby minimizing the influence of LLM preferences on dataset quality\.

We categorized exam questions into six types:Normal, with fixed unique answers but flexible formatting;Statement, requiring free\-form descriptive responses;Calculate, involving numerical computation with tolerance\-based evaluation;Choice, selecting from predefined options;Proof, requiring step\-by\-step justification; andBinary, involving justified binary classification\. Because automatic classification with GPT\-5 yielded unsatisfactory results, we manually completed the annotation\.

For each question, we annotate its relevant page range as a noise\-free context \(the “Oracle” setting\)\. A page is considered relevant if it falls within the span from the beginning of the parent question stem to the end of the page on which the subquestion is fully presented\. The total span is constrained to at most four pages\.

Ultimately, we construct a large\-scale dataset comprising10,242 QA pairsand7,155 high\-resolution exam images\.

### 3\.3Overview of Benchmark Key Highlights

![Refer to caption](https://arxiv.org/html/2609.13158v1/x3.png)Figure 3:Example of nested question numbering in TestHallVQA, where subquestions are indicated by indentation and may span multiple pages\.DatasetsData SizeQuestion TypesMulti\-PageDeep\-ReasoningEvaluatorPVCQA PairsImagesDocVQA TasksSingleDocVQA\[[39](https://arxiv.org/html/2609.13158#bib.bib54)\]12,76750,0001\(short\)✗✗ANLS\[[8](https://arxiv.org/html/2609.13158#bib.bib33)\]✗VisualMRC\[[49](https://arxiv.org/html/2609.13158#bib.bib65)\]10,19730,5621\(short\)✗✗BERTScore\[[64](https://arxiv.org/html/2609.13158#bib.bib32)\]✗ChartQA\[[37](https://arxiv.org/html/2609.13158#bib.bib53)\]32,71921,9481\(short\)✗✗EM \(Exact Match\)✗MP\-DocVQA\[[50](https://arxiv.org/html/2609.13158#bib.bib69)\]46,00048,0001\(short\)✓✗ANLS✗DUDE\[[52](https://arxiv.org/html/2609.13158#bib.bib70)\]28,00041,5411\(short\)✓✗ANLS✗MMLongBench\-Doc\[[35](https://arxiv.org/html/2609.13158#bib.bib68)\]1,0826,4121\(short\)✓✗regex\-based matching✗LongDocURL\[[16](https://arxiv.org/html/2609.13158#bib.bib67)\]2,32533,8981\(short\)✓✗regex\-based matching✗Scientific VQA TasksMathVista\[[33](https://arxiv.org/html/2609.13158#bib.bib63)\]6,1415,4872\(short45%, choice55%\)✗✓regex\-based matching✗MathVerse\[[63](https://arxiv.org/html/2609.13158#bib.bib62)\]2,6122,4202\(short38%, choice62%\)✗✓GPT\-4V\[[43](https://arxiv.org/html/2609.13158#bib.bib31)\]✗MMMU\[[60](https://arxiv.org/html/2609.13158#bib.bib64)\]11,55011,2642\(short6%, choice94%\)✗✓regex\-based matching✗MMMU\-Pro\[[61](https://arxiv.org/html/2609.13158#bib.bib35)\]3,460\-1\(choice\)✗✓regex\-based matching✗50%✓50%SceMQA\[[31](https://arxiv.org/html/2609.13158#bib.bib59)\]1,045\-2\(short20%, choice80%\)✗✓EM \(Exact Match\)✗HC\-M3D\[[32](https://arxiv.org/html/2609.13158#bib.bib43)\]1,8511,8511\(choice\)✗✓\-✗TestHallVQA \(ours\)10,2427,1556 \(Details in Caption\)✓✓LLMs \(open\-source friendly\)✓

Table 1:Comparison of TestHallVQA with existing related benchmarks\. “PVC” denotes Pure Visual Cues\. For question types, “short” refers to short\-answer questions, and “choice” refers to multiple\-choice questions\. The six question types in TestHallVQA are: Normal, Statement, Choice, Calculate, Proof, and Binary\.Cross\-Page Challenges\.TestHallVQA exam papers adopt a hierarchical question–subquestion structure, in which a substantial portion further employs nested numbering and indentation, distributing complete question identifiers across multiple pages \(as shown in Figure[3](https://arxiv.org/html/2609.13158#S3.F3)\)\. This structure renders page\-level RAG methods\[[1](https://arxiv.org/html/2609.13158#bib.bib26),[59](https://arxiv.org/html/2609.13158#bib.bib24)\]ineffective and requires LVLMs to infer inter\-page dependencies and integrate dispersed evidence through global reasoning\.

Pure Visual Cues\.Following\[[61](https://arxiv.org/html/2609.13158#bib.bib35)\], humans acquire both textual and graphical information through vision, suggesting that LVLMs should likewise rely solely on“pure visual cues”rather than explicit textual input\. Accordingly, TestHallVQA adopts a question\-index–guided prompting strategy \(e\.g\., “Please answer question 3\(b\)ii\.”\), where no semantically relevant text is provided\. LVLMs must therefore gather all evidence through visual perception alone\. This end\-to\-end VQA setting evaluates their intrinsic visual analysis and text reasoning abilities while avoiding confounding factors from system\-level pipelines\.

Open\-source\-friendly LLMs as judges\.Most existing planar VQA benchmarks rely on heuristic evaluation metrics \(e\.g\., rule\-based matching and ANLS\[[8](https://arxiv.org/html/2609.13158#bib.bib33)\]\), which are brittle to semantically equivalent variations that preserve the target format\. This limitation may systematically underestimate model performance and, in turn, overstate the difficulty of the benchmark\. Following prior work\[[18](https://arxiv.org/html/2609.13158#bib.bib30),[42](https://arxiv.org/html/2609.13158#bib.bib29),[28](https://arxiv.org/html/2609.13158#bib.bib28)\], we adopt an LLM\-as\-a\-Judge evaluation framework with format\-specific judging protocols tailored to the six task types in TestHallVQA\. Subsequent experiments show that this setup enables LLMs, including open\-source models, to achieve strong agreement with human preferences\. Complete evaluation templates and illustrative examples are provided in Appendix[G](https://arxiv.org/html/2609.13158#A7)\.

We compare TestHallVQA with related benchmarks in Table[1](https://arxiv.org/html/2609.13158#S3.T1)\.

### 3\.4Multi\-Level Redundant Context Injection

The objective of the TestHallVQA benchmark is to unify the evaluation of problem\-solving capacity \(Reasoning\) and robustness to redundant visual contamination \(Retrieval\) into a coupled capability\.

Reasoning capability can be evaluated directly through answer accuracy\. In contrast, assessing retrieval capability requires more careful design\. One intuitive approach is to ask the model to identify the relevant pages and use page prediction accuracy as the retrieval metric\[[52](https://arxiv.org/html/2609.13158#bib.bib70),[50](https://arxiv.org/html/2609.13158#bib.bib69)\]\. However, this formulation effectively reduces the benchmark to a decoupled retrieval\-and\-reasoning setting, where the two capabilities are evaluated independently\. Instead, retrieval capability should be measured by performance under increasing visual redundancy: models that remain accurate in highly redundant contexts exhibit stronger retrieval capability\.

To introduce such progressively increasing visual redundancy, we propose a controllableredundant context injectionstrategy\. Specifically, each question is paired with multiple context scales, termed“regimes”\. The first regime includes only the relevant pages \(i\.e\., the “Oracle” setting\), and each subsequent regime adds ten redundant pages until the full exam document is reached\. To maximize confusability, redundant pages are sampled from the same exam paper\. Given that the maximum page count in our collected exams is 44, the number of regimes is capped at five\. Detailed pseudocode is provided in Appendix[B](https://arxiv.org/html/2609.13158#A2)\.

![Refer to caption](https://arxiv.org/html/2609.13158v1/x4.png)Figure 4:Pyramid\-shaped distribution in TestHallVQA\. The boxed region denotes a subset in which the question set remains identical across all regimes within the subset\.However, due to the varying lengths of the documents, TestHallVQA’s overall distribution follows a stepped pattern, as shown in Figure[4](https://arxiv.org/html/2609.13158#S3.F4)\. We partition TestHallVQA into five test subsets using the rectangular selection method illustrated in Figure[4](https://arxiv.org/html/2609.13158#S3.F4)\. This design maximizes data utilization while ensuring that the questions in each test subset remain consistent across regime transitions\.

### 3\.5F1\-R2Metric

After constructing the multi\-regime evaluation subsets described above, we now detail the computation of ourF1\-R2metric \(F1score forRetrieval andReasoning\)\. For clarity, we take the computation of F1\-R2on the subset “Regime 1–3” as an example; the other cases are defined analogously\.

Each question in the subset “Regime 1–3” is evaluated under different regimes using the LLM\-as\-a\-Judge protocol, in which the LLM assigns a correctness score in the range of\[0,1\]\[0,1\]to each model output\. The average accuracy scores across the three regimes are denoted bye1,e2,e3∈\[0,100\]e\_\{1\},e\_\{2\},e\_\{3\}\\in\[0,100\], expressed as percentages\. Letp¯1,p¯2,p¯3\\bar\{p\}\_\{1\},\\bar\{p\}\_\{2\},\\bar\{p\}\_\{3\}denote the average numbers of pages under the corresponding regimes\. TheF1\-R2metric is defined as follows:

F1​\-​R2=2⋅SR​T⋅SR​NSR​T\+SR​N,\\mathrm\{F1\\text\{\-\}R\}^\{2\}=\\frac\{2\\cdot S\_\{RT\}\\cdot S\_\{RN\}\}\{S\_\{RT\}\+S\_\{RN\}\},\(7\)
whereSR​TS\_\{RT\}denotes the retrieval score, andSR​NS\_\{RN\}denotes the reasoning score\.SR​N∈\[0,1\]S\_\{RN\}\\in\[0,1\]denotes the accuracy under the last regime \(Regime 3 in this case\), reflecting the model’s true reasoning ability in document\-level, realistically redundant settings:

SR​N=e3%=e3100∈\[0,1\]\.S\_\{RN\}=e\_\{3\}\\%=\\frac\{e\_\{3\}\}\{100\}\\in\[0,1\]\.\(8\)
![Refer to caption](https://arxiv.org/html/2609.13158v1/x5.png)Figure 5:Performance decay of an LVLM across the three contextual regimes in TestHallVQA\.ForSR​TS\_\{RT\}, we define it as the slope of performance degradation induced by the introduction of redundant pages\. To ensure thatSR​TS\_\{RT\}is sensitive to performance degradation and possesses sufficient discriminative power for model comparison, we derive it from therotation angle, which serves as a measure of the extent of decay\. Subsequent ablation studies further validate its superior discriminative capacity\.

Specifically, as shown in Figure[5](https://arxiv.org/html/2609.13158#S3.F5), when the first segment of the decay polyline reaches its maximum drop\(e2=0\)\(e\_\{2\}=0\), it removes an area represented by the triangles1\+2\\hbox to11\.37pt\{\\vbox to11\.37pt\{\\pgfpicture\\makeatletter\\hbox\{\\quad\\lower\-5\.68657pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\{\}\{\{\{\}\}\}\{\{\}\}\{\}\{\}\{\}\{\}\{\}\{\}\{\}\{\}\{\}\{\{\}\\pgfsys@moveto\{5\.48657pt\}\{0\.0pt\}\\pgfsys@curveto\{5\.48657pt\}\{3\.03018pt\}\{3\.03018pt\}\{5\.48657pt\}\{0\.0pt\}\{5\.48657pt\}\\pgfsys@curveto\{\-3\.03018pt\}\{5\.48657pt\}\{\-5\.48657pt\}\{3\.03018pt\}\{\-5\.48657pt\}\{0\.0pt\}\\pgfsys@curveto\{\-5\.48657pt\}\{\-3\.03018pt\}\{\-3\.03018pt\}\{\-5\.48657pt\}\{0\.0pt\}\{\-5\.48657pt\}\\pgfsys@curveto\{3\.03018pt\}\{\-5\.48657pt\}\{5\.48657pt\}\{\-3\.03018pt\}\{5\.48657pt\}\{0\.0pt\}\\pgfsys@closepath\\pgfsys@moveto\{0\.0pt\}\{0\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-2\.5pt\}\{\-3\.22221pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{1\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\{\{\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\+\\hbox to11\.37pt\{\\vbox to11\.37pt\{\\pgfpicture\\makeatletter\\hbox\{\\quad\\lower\-5\.68657pt\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@setlinewidth\{\\the\\pgflinewidth\}\\pgfsys@invoke\{ \}\\nullfont\\hbox to0\.0pt\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\{\{\}\}\\hbox\{\\hbox\{\{\\pgfsys@beginscope\\pgfsys@invoke\{ \}\{\{\}\{\{\{\}\}\}\{\{\}\}\{\}\{\}\{\}\{\}\{\}\{\}\{\}\{\}\{\}\{\{\}\\pgfsys@moveto\{5\.48657pt\}\{0\.0pt\}\\pgfsys@curveto\{5\.48657pt\}\{3\.03018pt\}\{3\.03018pt\}\{5\.48657pt\}\{0\.0pt\}\{5\.48657pt\}\\pgfsys@curveto\{\-3\.03018pt\}\{5\.48657pt\}\{\-5\.48657pt\}\{3\.03018pt\}\{\-5\.48657pt\}\{0\.0pt\}\\pgfsys@curveto\{\-5\.48657pt\}\{\-3\.03018pt\}\{\-3\.03018pt\}\{\-5\.48657pt\}\{0\.0pt\}\{\-5\.48657pt\}\\pgfsys@curveto\{3\.03018pt\}\{\-5\.48657pt\}\{5\.48657pt\}\{\-3\.03018pt\}\{5\.48657pt\}\{0\.0pt\}\\pgfsys@closepath\\pgfsys@moveto\{0\.0pt\}\{0\.0pt\}\\pgfsys@stroke\\pgfsys@invoke\{ \} \}\{\{\{\{\}\}\\pgfsys@beginscope\\pgfsys@invoke\{ \}\\pgfsys@transformcm\{1\.0\}\{0\.0\}\{0\.0\}\{1\.0\}\{\-2\.5pt\}\{\-3\.22221pt\}\\pgfsys@invoke\{ \}\\hbox\{\{\\definecolor\{pgfstrokecolor\}\{rgb\}\{0,0,0\}\\pgfsys@color@rgb@stroke\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\pgfsys@color@rgb@fill\{0\}\{0\}\{0\}\\pgfsys@invoke\{ \}\\hbox\{\{2\}\} \}\}\\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \\pgfsys@invoke\{ \}\\pgfsys@endscope\}\}\} \} \\pgfsys@invoke\{ \}\\pgfsys@endscope\{\{\{\}\}\}\{\}\{\}\\hss\}\\pgfsys@discardpath\\pgfsys@invoke\{ \}\\pgfsys@endscope\\hss\}\}\\endpgfpicture\}\}\. At this point, the rotation angle of the score polyline is the sum of∠​1\+∠​2\\angle 1\+\\angle 2\. In contrast, when retrieval performance is perfect, the polyline remains flat, and no area is removed\. Using this bounded interval, the normalized retrieval performance is defined as:

k1=∠​2∠​1\+∠​2=1−∠​1∠​1\+∠​2=1−arctan⁡\(e1−e2p¯2−p¯1\)arctan⁡\(e1p¯2−p¯1\)∈\[0,1\]\.k\_\{1\}=\\frac\{\\angle\{2\}\}\{\\angle\{1\}\+\\angle\{2\}\}=1\-\\frac\{\\angle\{1\}\}\{\\angle\{1\}\+\\angle\{2\}\}=1\-\\frac\{\\arctan\(\\frac\{e\_\{1\}\-e\_\{2\}\}\{\\bar\{p\}\_\{2\}\-\\bar\{p\}\_\{1\}\}\)\}\{\\arctan\(\\frac\{e\_\{1\}\}\{\\bar\{p\}\_\{2\}\-\\bar\{p\}\_\{1\}\}\)\}\\in\[0,1\]\.\(9\)
Similarly, the decay factor from Regime 2 to 3 is:

k2=1−arctan⁡\(e2−e3p¯3−p¯2\)arctan⁡\(e2p¯3−p¯2\)∈\[0,1\]\.k\_\{2\}=1\-\\frac\{\\arctan\(\\frac\{e\_\{2\}\-e\_\{3\}\}\{\\bar\{p\}\_\{3\}\-\\bar\{p\}\_\{2\}\}\)\}\{\\arctan\(\\frac\{e\_\{2\}\}\{\\bar\{p\}\_\{3\}\-\\bar\{p\}\_\{2\}\}\)\}\\in\[0,1\]\.\(10\)
The retrieval scoreSR​TS\_\{RT\}is then computed as a weighted combination ofk1,k2k\_\{1\},k\_\{2\}, where the weights correspond to the average page span between the two regimes:

SR​T=\(p¯2−p¯1\)​k1\+\(p¯3−p¯2\)​k2p¯3−p¯1∈\[0,1\]\.S\_\{RT\}=\\frac\{\(\\bar\{p\}\_\{2\}\-\\bar\{p\}\_\{1\}\)k\_\{1\}\+\(\\bar\{p\}\_\{3\}\-\\bar\{p\}\_\{2\}\)k\_\{2\}\}\{\\bar\{p\}\_\{3\}\-\\bar\{p\}\_\{1\}\}\\in\[0,1\]\.\(11\)

## 4Experiments

### 4\.1Evaluation Protocols

We evaluate a broad range of LVLMs officially released by January 2026 on each test subset of TestHallVQA\. We also include OCR\-based baselines as supplementary references\. Specifically,PaddleOCR\-v5\[[13](https://arxiv.org/html/2609.13158#bib.bib3)\]is used to extract text from the original images and organize it in natural reading order\. The extracted text is then provided as contextual input in place of the images\.

We useQwen3\-30B\-A3B\-Instruct\-2507\[[58](https://arxiv.org/html/2609.13158#bib.bib20)\]as the judge LLM, employing the standard prompt templates described in Appendix[G](https://arxiv.org/html/2609.13158#A7)\. We also gratefully utilize the vLLM framework\[[26](https://arxiv.org/html/2609.13158#bib.bib19)\]to enhance inference efficiency\. We adopt greedy decoding for reproducibility of results and perform all experiments on 8 A100\-SXM4\-80GB GPUs\.

Given the large evaluation scale and broad model coverage, we evaluate API\-based models and models with more than 70B parameters on a*mini*test set that matches the full test\-set distribution \(see Appendix[C](https://arxiv.org/html/2609.13158#A3)\), balancing fidelity and computational cost\.

Some models are unable to accommodate the context length introduced by multi\-page image inputs under certain evaluation settings\. Their results on the corresponding subsets are therefore marked asnull, indicating insufficient long\-context capability\. Nevertheless, to improve the completeness of the experimental evaluation, we additionally report results at lower image resolutions to fill in thesenullentries wherever reasonable\. Specifically, when a subset exceeds a model’s context budget, we uniformly downsample all images in that subset until all samples become processable, using scaling factors of0\.7070\.707\(50% pixel reduction\) and0\.50\.5\(75% pixel reduction\)\. If neither setting suffices, the subset is excluded\. Importantly,only results obtained at the same image resolution are horizontally comparable, since varying resolutions introduce confounding effects from differences in visual fidelity\.

### 4\.2Main Results

ModelSizeOracleRegime 1–2 \(pmax=10p\_\{\\max\}=10\)Regime 1–3 \(pmax=20p\_\{\\max\}=20\)Regime 1–4 \(pmax=30p\_\{\\max\}=30\)Regime 1–5 \(pmax=40p\_\{\\max\}=40\)SR​NS\_\{RN\}SR​TS\_\{RT\}SR​NS\_\{RN\}F​1−R2F1\-R^\{2\}SR​TS\_\{RT\}SR​NS\_\{RN\}F​1−R2F1\-R^\{2\}SR​TS\_\{RT\}SR​NS\_\{RN\}F​1−R2F1\-R^\{2\}SR​TS\_\{RT\}SR​NS\_\{RN\}F​1−R2F1\-R^\{2\}\\rowcolorgray\!15LLMs/LVLMs with OCR\-processed text input\\rowcolorgray\!15 GPT\-5[2025](https://arxiv.org/html/2609.13158#bib.bib16)\-46\.6868\.8542\.6252\.6574\.3741\.3453\.1470\.6239\.7450\.8671\.4238\.4049\.95\\rowcolorgray\!15 Qwen2\.5\-VL[2025b](https://arxiv.org/html/2609.13158#bib.bib12)7B28\.7469\.1424\.9636\.6872\.1523\.0734\.9664\.8720\.4931\.1455\.4117\.9227\.08\\rowcolorgray\!15 InternVL3\.5[2025](https://arxiv.org/html/2609.13158#bib.bib14)8B39\.0878\.0236\.2449\.4970\.1932\.1644\.1167\.5527\.4239\.0769\.4525\.1836\.96\\rowcolorgray\!15 Mistral\-Small\-3\.1[2025](https://arxiv.org/html/2609.13158#bib.bib7)24B40\.2480\.6238\.8552\.4370\.2834\.3346\.1371\.3430\.4742\.6572\.8628\.0440\.50\\rowcolorgray\!15 DeepSeek\-R1[2025](https://arxiv.org/html/2609.13158#bib.bib2)32B42\.8580\.1140\.4153\.7278\.4437\.0450\.3264\.5733\.6444\.2370\.8328\.4440\.58Closed\-Source LVLMsQwen\-VL\-max∗[2023](https://arxiv.org/html/2609.13158#bib.bib18)\-65\.7670\.2961\.6265\.6762\.3055\.3458\.6161\.9051\.4656\.1958\.3244\.7050\.60GPT\-4o∗[2024](https://arxiv.org/html/2609.13158#bib.bib23)\-78\.9159\.4572\.8765\.4760\.0567\.3063\.4658\.4962\.3860\.3758\.8355\.3454\.01GPT\-5∗[2025](https://arxiv.org/html/2609.13158#bib.bib16)\-85\.7763\.5780\.4371\.0167\.9676\.5972\.0163\.2570\.8166\.8161\.4565\.5963\.45Gemini\-2\.5\-Pro∗[2025](https://arxiv.org/html/2609.13158#bib.bib15)\-84\.5073\.9580\.8277\.2368\.0175\.5971\.5967\.9371\.6669\.7461\.6464\.3662\.97Large\-Scale Open\-Source LVLMsLLaVA\-OneVision∗[2024a](https://arxiv.org/html/2609.13158#bib.bib10)72B32\.4443\.36‡\\ddagger15\.69‡\\ddagger23\.04‡\\ddagger\-\-\-\-\-\-\-\-\-LLaVA\-Next∗[2024b](https://arxiv.org/html/2609.13158#bib.bib9)72B29\.2974\.6026\.1438\.7167\.80†\\dagger22\.67†\\dagger33\.97†\\dagger67\.13‡\\ddagger14\.98‡\\ddagger24\.49‡\\ddagger53\.66‡\\ddagger8\.84‡\\ddagger15\.17‡\\ddaggerInternVL3∗[2025](https://arxiv.org/html/2609.13158#bib.bib13)78B64\.5465\.8459\.7762\.6560\.66†\\dagger53\.64†\\dagger56\.93†\\dagger55\.59†\\dagger47\.78†\\dagger51\.38†\\dagger54\.58‡\\ddagger37\.24‡\\ddagger44\.27‡\\ddaggerQwen2\.5\-VL∗[2025b](https://arxiv.org/html/2609.13158#bib.bib12)72B63\.2054\.8456\.4255\.6150\.1148\.6849\.3847\.0242\.4344\.6852\.4536\.8143\.26Mistral\-Small\-3\.1[2025](https://arxiv.org/html/2609.13158#bib.bib7)24B65\.2074\.1561\.6367\.3164\.2454\.8259\.1565\.1149\.5556\.2757\.2543\.9249\.70InternVL3\.5[2025](https://arxiv.org/html/2609.13158#bib.bib14)38B72\.3077\.8269\.2373\.2171\.3365\.0668\.0568\.72†\\dagger57\.47†\\dagger62\.59†\\dagger60\.77†\\dagger51\.85†\\dagger55\.96†\\daggerQwen3\-VL[2025a](https://arxiv.org/html/2609.13158#bib.bib11)30B80\.6558\.2274\.4265\.3364\.1370\.3667\.1061\.7965\.5363\.6162\.5660\.1261\.37Small\-Scale Open\-Source LVLMsInternLM\-XC2\.5[2024a](https://arxiv.org/html/2609.13158#bib.bib17)7B25\.7542\.1417\.9025\.1229\.25†\\dagger10\.52†\\dagger15\.47†\\dagger37\.76‡\\ddagger6\.61‡\\ddagger11\.25‡\\ddagger40\.50‡\\ddagger3\.20‡\\ddagger5\.93‡\\ddaggerLLaVA\-OneVision[2024a](https://arxiv.org/html/2609.13158#bib.bib10)7B23\.6752\.77‡\\ddagger13\.12‡\\ddagger21\.01‡\\ddagger\-\-\-\-\-\-\-\-\-LLaVA\-Next[2024b](https://arxiv.org/html/2609.13158#bib.bib9)8B17\.5728\.63‡\\ddagger8\.36‡\\ddagger12\.94‡\\ddagger\-\-\-\-\-\-\-\-\-DeepSeek\-OCR[2025](https://arxiv.org/html/2609.13158#bib.bib8)3B18\.4936\.22†\\dagger10\.62†\\dagger16\.42†\\dagger16\.97‡\\ddagger3\.14‡\\ddagger5\.29‡\\ddagger14\.50‡\\ddagger1\.24‡\\ddagger2\.28‡\\ddagger\-\-\-Qwen2\.5\-VL[2025b](https://arxiv.org/html/2609.13158#bib.bib12)7B56\.0438\.4145\.7341\.7543\.6738\.6841\.0247\.4931\.8038\.0954\.2124\.6633\.90InternVL3[2025](https://arxiv.org/html/2609.13158#bib.bib13)8B60\.4170\.5556\.3862\.6768\.33†\\dagger47\.50†\\dagger56\.04†\\dagger60\.82†\\dagger39\.77†\\dagger48\.09†\\dagger62\.28‡\\ddagger26\.12‡\\ddagger36\.80‡\\ddaggerGLM\-4\.1V\-Thinking[2025](https://arxiv.org/html/2609.13158#bib.bib6)9B79\.9463\.2174\.7568\.4960\.2765\.8062\.9167\.31†\\dagger54\.18†\\dagger60\.03†\\dagger60\.83†\\dagger49\.26†\\dagger54\.44†\\daggerInternVL3\.5[2025](https://arxiv.org/html/2609.13158#bib.bib14)8B69\.4378\.6066\.4972\.0474\.9462\.6668\.2570\.58†\\dagger54\.42†\\dagger61\.45†\\dagger66\.29†\\dagger49\.70†\\dagger56\.81†\\daggerQwen3\-VL[2025a](https://arxiv.org/html/2609.13158#bib.bib11)8B78\.2861\.7272\.2366\.5666\.7168\.8667\.7767\.4162\.8865\.0767\.1655\.5560\.81

Table 2:Main results on TestHallVQA\. “Oracle” denotes Regime 1, which represents the clean case with relevant pages as context\. “Regime 1–N” denotes the test subset comprising Regime 1 through Regime N\. “pmaxp\_\{\\max\}” represents the maximum number of images across the selected regimes\. “∗\\ast” indicates evaluation on the*mini\-test*set\. “†\\dagger” and “‡\\ddagger” indicate image side\-length scaling by 0\.707 \(50% pixels\) and 0\.5 \(25% pixels\), respectively\. Bold values indicate the best result among models of comparable scale, while bold and underlined values denote the global SOTA \(excluding cases with OCR processing and resolution reduction\)\.Table[4\.2](https://arxiv.org/html/2609.13158#S4.SS2)summarizes the evaluation results of all models across the five test subsets of TestHallVQA\. Based on a detailed analysis of the experimental results, we report the following key observations:

Reducing image resolution leads to performance degradation\.For certain LVLMs, such as the LLaVA series, input\-length constraints remain prohibitive even after the image resolution is reduced to one quarter of the original pixel count\. Moreover, when comparing the resolution\-reduced subset with the immediately preceding subset evaluated at the original resolution, we observe a sharp drop in bothSR​TS\_\{RT\}andSR​NS\_\{RN\}\. Several failure cases indicate that this degradation stems from the fact that fine\-grained visual elements, such as subscripts and other small typographic markers, become indistinguishable after resolution reduction\.

Pure\-text inputs are insufficient\.As shown in the gray rows of Table[4\.2](https://arxiv.org/html/2609.13158#S4.SS2), OCR\-based conversion discards graphical elements, charts, and other visual cues, leading to substantial evidence loss and a lowerSR​NS\_\{RN\}score\. This result underscores the strong visual dependence of TestHallVQA\. Although converting the input into pure text reduces the token count and alleviates context\-length pressure, thereby yielding a highSR​TS\_\{RT\}, this advantage is of limited practical value given the extremely low answer accuracy\.

Reasoning performance deteriorates sharply under multi\-page redundancy\.Focusing on theSR​NS\_\{RN\}metric, we observe that models perform strongly when conditioned solely on the “Oracle” regime, representing a highly idealized and noise\-free setting \(e\.g\., GPT\-5 achieves 85\.77%\)\. However, extending the context to the document level results in a sharp and monotonic decline inSR​NS\_\{RN\}across all models \(e\.g\., GPT\-5 drops to 63\.45% on the “Regime 1–5” subset\)\. This trend is fully consistent with the theoretical prediction in Section[3\.1](https://arxiv.org/html/2609.13158#S3.SS1), which states that information purity decreases asNM\\frac\{N\}\{M\}increases\. The pronounced left–right contrast in the table underscores that existing LVLMs experience substantial degradation in reasoning performance under complex contextual settings and demonstrate limited robustness to redundancy\.

Retrieval capacity saturates as the number of pages increases\.As shown throughout the table, robustness to redundant context, as measured bySR​TS\_\{RT\}, remains consistently low across all models and further deteriorates as the evaluation regimes scale up, revealing a fundamental retrieval bottleneck\. These findings suggest that, for complex visual tasks, the effective context length of LLMs is substantially smaller than that implied by needle\-in\-a\-haystack evaluations\[[25](https://arxiv.org/html/2609.13158#bib.bib5)\]\. Therefore, metrics derived from the severity of performance degradation, such asSR​TS\_\{RT\}, are better suited to assessing context capacity\.

Counterintuitive finding: scaling does not mitigate contamination\.Intuitively, larger models are expected to be more robust to redundant\-token contamination and to better preserve the accuracy achieved on the oracle set\. However, our results suggest that this expectation does not hold in practice: larger models often attain comparable or even lowerSR​TS\_\{RT\}than their smaller counterparts \(e\.g\., InternVL and Qwen across different parameter scales\), and even advanced closed\-source models rarely achieve the best overallSR​TS\_\{RT\}\. This phenomenon can be interpreted through the information\-purity formulationηL\\eta^\{L\}introduced in Section[3\.1](https://arxiv.org/html/2609.13158#S3.SS1)\. Specifically, although increasing the number of layers may allow each layer to capture more fine\-grained features and partially suppress contamination, deeper models with largerLLtend to yield lowerηL\\eta^\{L\}, thereby exacerbating the propagation of noise introduced by redundant tokens\.

### 4\.3Extended Analysis and Ablations

EvaluatorModelGemini\-2\.5\-ProInternVL3\.5\-38BGLM\-4\.1V\-9BInternVL3\.5\-8BHuman Avg\.0\.6310\.5140\.5930\.547GPT\-50\.634 \(\+0\.003\)0\.520 \(\+0\.006\)0\.596 \(\+0\.003\)0\.551 \(\+0\.004\)GPT\-5\-nano0\.629 \(\-0\.002\)0\.524 \(\+0\.01\)0\.595 \(\+0\.002\)0\.544 \(\-0\.003\)Qwen3\-30B0\.638 \(\+0\.007\)0\.517 \(\+0\.003\)0\.585 \(\-0\.008\)0\.543 \(\-0\.004\)Qwen3\-8B0\.609 \(\-0\.022\)0\.499 \(\-0\.015\)0\.587 \(\-0\.006\)0\.537 \(\-0\.01\)Table 3:LLM\-as\-a\-Judge evaluation results compared with human annotations on 1K randomly sampled instances from different regimes of the mini\-test set\.![Refer to caption](https://arxiv.org/html/2609.13158v1/x6.png)Figure 6:Classification accuracy radar plots, divided into two figures according to score ranking\.First, we validate our LLM\-as\-a\-Judge paradigm by comparing the accuracy scores assigned by human evaluators and LLM\-based judges, including two human annotators and four representative LLMs\. As shown in Table[3](https://arxiv.org/html/2609.13158#S4.T3), aside from the weaker performance of the smaller Qwen3\-8B, models at other scales demonstrate strong agreement with human judgments\. This indicates that our prompt design \(Appendix[G](https://arxiv.org/html/2609.13158#A7)\) provides clear criteria for fine\-grained LLM scoring while mitigating the influence of self\-preference\.

Figure[6](https://arxiv.org/html/2609.13158#S4.F6)reports the accuracy of selected LVLMs across different question categories under Regime 1, the broadest test subset of the benchmark\. Detailed statistics for each question category are provided in Appendix[C](https://arxiv.org/html/2609.13158#A3)\. The results indicate that the difficulty levels across the various disciplines and question types are relatively well balanced within the benchmark\.

![Refer to caption](https://arxiv.org/html/2609.13158v1/x7.png)Figure 7:Box plots of the score distributions ofSR​TS\_\{RT\}under the rotation\-angle method and baseline methods\.To demonstrate the advantages of the rotation angle\-basedSR​TS\_\{RT\}metric, we compare it against several baseline metrics, including step retention ratio \(SRR\), end\-to\-end retention \(E2E\), normalized AUC \(N\-AUC\), and log\-based decay \(LD\); their formal definitions are provided in Appendix[D](https://arxiv.org/html/2609.13158#A4)\. Figure[7](https://arxiv.org/html/2609.13158#S4.F7)shows the distributions ofSR​TS\_\{RT\}scores across all models and subsets under these metrics, excluding the OCR and image\-resolution experiments\. As illustrated, even after normalization, the baseline metrics compress models with substantially different capabilities into a narrow score range\. In contrast, our metric demonstrates the highest sensitivity, producing the widest score distribution \(from 0\.38 to 0\.79\) and thereby offering the strongest discriminative power for the evaluation pipeline\.

![Refer to caption](https://arxiv.org/html/2609.13158v1/x8.png)Figure 8:Performance degradation of selected LVLMs on the “Regime 1–5” Subset\.Figure[8](https://arxiv.org/html/2609.13158#S4.F8)shows the degradation curves of selected LVLMs under different regimes on the most comprehensive test subset, “Regime 1–5\.” The steep declines provide an intuitive illustration of the models’ substantial performance deterioration, highlighting the detrimental impact of redundant visual contamination\.

We provide more detailed supplementary experiments and analyses for interested readers\. Appendix[E](https://arxiv.org/html/2609.13158#A5)presents extended ablation studies on the effects of the number of pages involved in each question and the degree of visual confusability\. In Appendix[F](https://arxiv.org/html/2609.13158#A6), we further show that existing visual encoders are unable to capture the sequential temporal order across multiple images; this structural limitation may further aggravate redundant visual contamination\.

## 5Error Analysis and Case Studies

![Refer to caption](https://arxiv.org/html/2609.13158v1/x9.png)Figure 9:Examples of six representative error types\.Through a comprehensive error analysis, we identify several representative failure modes and present typical examples in Figure[9](https://arxiv.org/html/2609.13158#S5.F9)\. For consistency, all cases are drawn from Qwen3\-VL\-8B under the relevant\-page setting\. The red boxes provide detailed diagnoses of the underlying causes\. Specifically, casea\.shows difficulty in chart coordinate localization; caseb\.illustrates hallucinations caused by complex statistical visualizations; casec\.reveals insufficient domain\-specific knowledge; cased\.highlights query\-grounding drift in cross\-page retrieval; casee\.indicates difficulty in capturing abstract layout information; and casef\.shows that complex geometric diagrams can induce cyclic stalling in CoT reasoning\.

These errors suggest that TestHallVQA operates not only at a variable\-length, document\-level scale, but also poses rich and multidimensional visual challenges\.

## 6Conclusion

In this paper, we identify the phenomenon of redundant visual contamination, provide a symbolic theoretical characterization of its underlying mechanism, and introduce TestHallVQA, a novel multi\-page VQA benchmark that combines document\-level redundant context with deep visual reasoning, thereby bridging the gap between existing planar VQA benchmarks\. We further propose F1\-R2, a two\-dimensional metric for jointly evaluating the retrieval and reasoning capabilities of LVLMs under multiple regimes of redundant visual contamination\. Extensive experiments and analyses validate our theoretical findings and reveal that current LVLMs still perform unsatisfactorily in document\-level settings with redundant visual inputs\. We hope that our benchmark and findings will draw broader attention from the research community to document\-level deep\-reasoning VQA and inspire future advances in LVLMs\.

## 7Acknowledgements

We would like to thank the anonymous reviewers for their helpful comments\. This work was supported by the National Natural Science Foundation of China \(No\. 62476066\)\.

## References

- \[1\]O\. Abramovich, N\. Nayman, S\. Fogel, I\. Lavi, R\. Litman, S\. Tsiper, R\. Tichauer, S\. Appalaraju, S\. Mazor, and R\. Manmatha\(2024\)VisFocus: prompt\-guided vision encoders for ocr\-free dense document understanding\.InEuropean Conference on Computer Vision,pp\. 241–259\.Cited by:[§3\.3](https://arxiv.org/html/2609.13158#S3.SS3.p1.1)\.
- \[2\]S\. Antol, A\. Agrawal, J\. Lu, M\. Mitchell, D\. Batra, C\. L\. Zitnick, and D\. Parikh\(2015\)Vqa: visual question answering\.InProceedings of the IEEE international conference on computer vision,pp\. 2425–2433\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p1.1)\.
- \[3\]D\. Azuma, T\. Miyanishi, S\. Kurita, and M\. Kawanabe\(2022\)Scanqa: 3d question answering for spatial scene understanding\.Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 19129–19139\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p1.1)\.
- \[4\]S\. Bae, D\. Kyung, J\. Ryu, E\. Cho, G\. Lee, S\. Kweon, J\. Oh, L\. Ji, E\. Chang, T\. Kim,et al\.\(2023\)Ehrxqa: a multi\-modal question answering dataset for electronic health records with chest x\-ray images\.Advances in Neural Information Processing Systems36,pp\. 3867–3880\.Cited by:[§1](https://arxiv.org/html/2609.13158#S1.p5.1)\.
- \[5\]J\. Bai, S\. Bai, S\. Yang, S\. Wang, S\. Tan, P\. Wang, J\. Lin, C\. Zhou, and J\. Zhou\(2023\)Qwen\-vl: a versatile vision\-language model for understanding, localization, text reading, and beyond\.arXiv preprint arXiv:2308\.12966\.Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.18.18.18.18.1)\.
- \[6\]S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. Zhu\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§F\.2](https://arxiv.org/html/2609.13158#A6.SS2.p4.1),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.97.97.97.107.1),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.97.97.97.110.1)\.
- \[7\]S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang,et al\.\(2025\)Qwen2\. 5\-vl technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[§F\.2](https://arxiv.org/html/2609.13158#A6.SS2.p4.1),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.46.46.46.46.1),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.97.97.97.100.1),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.97.97.97.109.1)\.
- \[8\]A\. F\. Biten, R\. Tito, A\. Mafla, L\. Gomez, M\. Rusinol, E\. Valveny, C\. Jawahar, and D\. Karatzas\(2019\)Scene text visual question answering\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 4291–4301\.Cited by:[§3\.3](https://arxiv.org/html/2609.13158#S3.SS3.p3.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.4.7)\.
- \[9\]J\. Chen, T\. Li, J\. Qin, P\. Lu, L\. Lin, C\. Chen, and X\. Liang\(2022\)UniGeo: unifying geometry logical reasoning via reformulating mathematical expression\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp\. 3313–3323\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1)\.
- \[10\]J\. Chen, J\. Tang, J\. Qin, X\. Liang, L\. Liu, E\. Xing, and L\. Lin\(2021\)Geoqa: a geometric question answering benchmark towards multimodal numerical reasoning\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,pp\. 513–523\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1)\.
- \[11\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1)\.
- \[12\]G\. Comanici, E\. Bieber, M\. Schaekermann, I\. Pasupat, N\. Sachdeva, I\. Dhillon, M\. Blistein, O\. Ram, D\. Zhang, E\. Rosen,et al\.\(2025\)Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.arXiv preprint arXiv:2507\.06261\.Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.21.21.21.21.1)\.
- \[13\]C\. Cui, T\. Sun, M\. Lin, T\. Gao, Y\. Zhang, J\. Liu, X\. Wang, Z\. Zhang, C\. Zhou, H\. Liu,et al\.\(2025\)Paddleocr 3\.0 technical report\.arXiv preprint arXiv:2507\.05595\.Cited by:[§4\.1](https://arxiv.org/html/2609.13158#S4.SS1.p1.1)\.
- \[14\]R\. Das, S\. Hristov, H\. Li, D\. Dimitrov, I\. Koychev, and P\. Nakov\(2024\)Exams\-v: a multi\-discipline fmahmultilingual multimodal exam benchmark for evaluating vision language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 7768–7791\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1)\.
- \[15\]P\. Dasigi, K\. Lo, I\. Beltagy, A\. Cohan, N\. A\. Smith, and M\. Gardner\(2021\)A dataset of information\-seeking questions and answers anchored in research papers\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 4599–4610\.Cited by:[§1](https://arxiv.org/html/2609.13158#S1.p4.1)\.
- \[16\]C\. Deng, J\. Yuan, P\. Bu, P\. Wang, Z\. Li, J\. Xu, X\. Li, Y\. Gao, J\. Song, B\. Zheng,et al\.\(2025\)Longdocurl: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1135–1159\.Cited by:[§1](https://arxiv.org/html/2609.13158#S1.p1.1),[§1](https://arxiv.org/html/2609.13158#S1.p4.1),[§2](https://arxiv.org/html/2609.13158#S2.p2.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.10.1)\.
- \[17\]J\. E\. Dittmar\(2011\)Information technology and economic change: the impact of the printing press\.The Quarterly Journal of Economics126\(3\),pp\. 1133–1172\.Cited by:[§1](https://arxiv.org/html/2609.13158#S1.p1.1)\.
- \[18\]M\. Gao, X\. Hu, X\. Yin, J\. Ruan, X\. Pu, and X\. Wan\(2025\)Llm\-based nlg evaluation: current status and challenges\.Computational Linguistics,pp\. 1–27\.Cited by:[§3\.3](https://arxiv.org/html/2609.13158#S3.SS3.p3.1)\.
- \[19\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.97.97.97.103.1)\.
- \[20\]D\. Gurari, Q\. Li, A\. J\. Stangl, A\. Guo, C\. Lin, K\. Grauman, J\. Luo, and J\. P\. Bigham\(2018\)Vizwiz grand challenge: answering visual questions from blind people\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 3608–3617\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p1.1)\.
- \[21\]D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt\(2021\)Measuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1)\.
- \[22\]W\. Hong, W\. Yu, X\. Gu, G\. Wang, G\. Gan, H\. Tang, J\. Cheng, J\. Qi, J\. Ji, L\. Pan,et al\.\(2025\)Glm\-4\.5 v and glm\-4\.1 v\-thinking: towards versatile multimodal reasoning with scalable reinforcement learning\.arXiv preprint arXiv:2507\.01006\.Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.91.91.91.91.7)\.
- \[23\]A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.19.19.19.19.1)\.
- \[24\]K\. Kafle and C\. Kanan\(2017\)An analysis of visual question answering algorithms\.InProceedings of the IEEE international conference on computer vision,pp\. 1965–1973\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p1.1)\.
- \[25\]G\. Kamradt\(2023\)LLMTest: needle in a haystack\.Note:[https://github\.com/gkamradt/LLMTest\_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.114.114)\.
- \[26\]W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica\(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§4\.1](https://arxiv.org/html/2609.13158#S4.SS1.p2.1)\.
- \[27\]B\. Li, Y\. Zhang, D\. Guo, R\. Zhang, F\. Li, H\. Zhang, K\. Zhang, P\. Zhang, Y\. Li, Z\. Liu,et al\.\(2024\)Llava\-onevision: easy visual task transfer\.arXiv preprint arXiv:2408\.03326\.Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.22.22.22.22.1),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.64.64.64.64.4)\.
- \[28\]D\. Li, B\. Jiang, L\. Huang, A\. Beigi, C\. Zhao, Z\. Tan, A\. Bhattacharjee, Y\. Jiang, C\. Chen, T\. Wu,et al\.\(2025\)From generation to judgment: opportunities and challenges of llm\-as\-a\-judge\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 2757–2791\.Cited by:[§3\.3](https://arxiv.org/html/2609.13158#S3.SS3.p3.1)\.
- \[29\]F\. Li, R\. Zhang, H\. Zhang, Y\. Zhang, B\. Li, W\. Li, Z\. Ma, and C\. Li\(2024\)Llava\-next\-interleave: tackling multi\-image, video, and 3d in large multimodal models\.arXiv preprint arXiv:2407\.07895\.Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.26.26.26.26.1),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.67.67.67.67.4)\.
- \[30\]H\. Liang, L\. Sun, z\. zhouminxuan, Z\. Chen, M\. Qiang, M\. Lin, T\. Li, F\. Yang, Z\. Zhou, and W\. Zhang\(2025\)MathScape: benchmarking multimodal large language models in real\-world mathematical contexts\.InProceedings of the 33rd ACM International Conference on Multimedia,pp\. 12942–12948\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1)\.
- \[31\]Z\. Liang, K\. Guo, G\. Liu, T\. Guo, Y\. Zhou, T\. Yang, J\. Jiao, R\. Pi, J\. Zhang, and X\. Zhang\(2024\)SceMQA: a scientific college entrance level multimodal question answering benchmark\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 109–119\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.16.1)\.
- \[32\]Y\. Liu, Y\. Du, T\. Ji, J\. Wang, Y\. Liu, Y\. Wu, A\. Zhou, M\. Zhang, and X\. Cai\(2025\)The role of visual modality in multimodal mathematical reasoning: challenges and insights\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 22596–22611\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.17.1)\.
- \[33\]P\. Lu, H\. Bansal, T\. Xia, J\. Liu, C\. Li, H\. Hajishirzi, H\. Cheng, K\. Chang, M\. Galley, and J\. Gao\(2024\)MathVista: evaluating mathematical reasoning of foundation models in visual contexts\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.13158#S1.p1.1),[§1](https://arxiv.org/html/2609.13158#S1.p5.1),[§2](https://arxiv.org/html/2609.13158#S2.p3.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.12.1)\.
- \[34\]P\. Lu, R\. Gong, S\. Jiang, L\. Qiu, S\. Huang, X\. Liang, and S\. Zhu\(2021\)Inter\-gps: interpretable geometry problem solving with formal language and symbolic reasoning\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 6774–6786\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1)\.
- \[35\]Y\. Ma, Y\. Zang, L\. Chen, M\. Chen, Y\. Jiao, X\. Li, X\. Lu, Z\. Liu, Y\. Ma, X\. Dong,et al\.\(2024\)Mmlongbench\-doc: benchmarking long\-context document understanding with visualizations\.Advances in Neural Information Processing Systems37,pp\. 95963–96010\.Cited by:[§1](https://arxiv.org/html/2609.13158#S1.p4.1),[§2](https://arxiv.org/html/2609.13158#S2.p2.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.9.1)\.
- \[36\]K\. Marino, M\. Rastegari, A\. Farhadi, and R\. Mottaghi\(2019\)Ok\-vqa: a visual question answering benchmark requiring external knowledge\.InProceedings of the IEEE/cvf conference on computer vision and pattern recognition,pp\. 3195–3204\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p1.1)\.
- \[37\]A\. Masry, X\. L\. Do, J\. Q\. Tan, S\. Joty, and E\. Hoque\(2022\)Chartqa: a benchmark for question answering about charts with visual and logical reasoning\.InFindings of the association for computational linguistics: ACL 2022,pp\. 2263–2279\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p2.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.6.1)\.
- \[38\]M\. Mathew, V\. Bagal, R\. Tito, D\. Karatzas, E\. Valveny, and C\. Jawahar\(2022\)Infographicvqa\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pp\. 1697–1706\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p2.1)\.
- \[39\]M\. Mathew, D\. Karatzas, and C\. Jawahar\(2021\)Docvqa: a dataset for vqa on document images\.InProceedings of the IEEE/CVF winter conference on applications of computer vision,pp\. 2200–2209\.Cited by:[§E\.2](https://arxiv.org/html/2609.13158#A5.SS2.p1.1),[§1](https://arxiv.org/html/2609.13158#S1.p1.1),[§2](https://arxiv.org/html/2609.13158#S2.p2.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.4.1)\.
- \[40\]M\. Mathew, R\. Tito, D\. Karatzas, R\. Manmatha, and C\. Jawahar\(2020\)Document visual question answering challenge 2020\.arXiv preprint arXiv:2008\.08899\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p2.1)\.
- \[41\]Mistral AI\(2025\)Mistral\-Small\-3\.1\-24B\-Instruct\-2503\.Note:[https://huggingface\.co/mistralai/Mistral\-Small\-3\.1\-24B\-Instruct\-2503](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503)Instruction\-tuned version of Mistral Small 3\.1 with 24B parameters, enhanced vision and long\-context capabilities \(128k tokens\)External Links:[Link](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503)Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.97.97.97.102.1),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.97.97.97.106.1)\.
- \[42\]B\. Murugadoss, C\. Poelitz, I\. Drosos, V\. Le, N\. McKenna, C\. S\. Negreanu, C\. Parnin, and A\. Sarkar\(2025\)Evaluating the evaluator: measuring llms’ adherence to task evaluation instructions\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 19589–19597\.Cited by:[§3\.3](https://arxiv.org/html/2609.13158#S3.SS3.p3.1)\.
- \[43\]OpenAI\(2023\-09\-25\)GPT\-4v\(ision\) system card\.Technical reportOpenAI,San Francisco, CA\.External Links:[Link](https://cdn.openai.com/papers/GPTV_System_Card.pdf)Cited by:[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.13.7)\.
- \[44\]OpenAI\(2025\)Introducing gpt\-5\.Note:[https://openai\.com/zh\-Hans\-CN/index/introducing\-gpt\-5/](https://openai.com/zh-Hans-CN/index/introducing-gpt-5/)Cited by:[§3\.2](https://arxiv.org/html/2609.13158#S3.SS2.p3.1),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.20.20.20.20.1),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.97.97.97.99.1)\.
- \[45\]J\. Qi, J\. Liu, H\. Tang, and Z\. Zhu\(2025\)Beyond semantics: rediscovering spatial awareness in vision\-language models\.arXiv preprint arXiv:2503\.17349\.Cited by:[§F\.1](https://arxiv.org/html/2609.13158#A6.SS1.p3.1)\.
- \[46\]R\. Qiao, Q\. Tan, G\. Dong, M\. MinhuiWu, C\. Sun, X\. Song, J\. Wang, Z\. Gongque, S\. Lei, Y\. Zhang,et al\.\(2025\)We\-math: does your large multimodal model achieve human\-like mathematical reasoning?\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 20023–20070\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1)\.
- \[47\]A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§1](https://arxiv.org/html/2609.13158#S1.p1.1)\.
- \[48\]K\. Sun, Y\. Bai, J\. Qi, L\. Hou, and J\. Li\(2024\)MM\-math: advancing multimodal math evaluation with process evaluation and fine\-grained classification\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 1358–1375\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1)\.
- \[49\]R\. Tanaka, K\. Nishida, and S\. Yoshida\(2021\)Visualmrc: machine reading comprehension on document images\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 13878–13888\.Cited by:[§1](https://arxiv.org/html/2609.13158#S1.p4.1),[§2](https://arxiv.org/html/2609.13158#S2.p2.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.5.1)\.
- \[50\]R\. Tito, D\. Karatzas, and E\. Valveny\(2023\)Hierarchical multimodal transformers for multipage docvqa\.Pattern Recognition144,pp\. 109834\.Cited by:[§1](https://arxiv.org/html/2609.13158#S1.p4.1),[§2](https://arxiv.org/html/2609.13158#S2.p2.1),[§3\.4](https://arxiv.org/html/2609.13158#S3.SS4.p2.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.7.1)\.
- \[51\]R\. Tito, M\. Mathew, C\. Jawahar, E\. Valveny, and D\. Karatzas\(2021\)ICDAR 2021 competition on document visual question answering\.InInternational Conference on Document Analysis and Recognition,pp\. 635–649\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p2.1)\.
- \[52\]J\. Van Landeghem, R\. Tito, Ł\. Borchmann, M\. Pietruszka, P\. Joziak, R\. Powalski, D\. Jurkiewicz, M\. Coustaty, B\. Anckaert, E\. Valveny,et al\.\(2023\)Document understanding dataset and evaluation \(dude\)\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 19528–19540\.Cited by:[§1](https://arxiv.org/html/2609.13158#S1.p1.1),[§1](https://arxiv.org/html/2609.13158#S1.p4.1),[§2](https://arxiv.org/html/2609.13158#S2.p2.1),[§3\.4](https://arxiv.org/html/2609.13158#S3.SS4.p2.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.8.1)\.
- \[53\]K\. Wang, J\. Pan, W\. Shi, Z\. Lu, H\. Ren, A\. Zhou, M\. Zhan, and H\. Li\(2024\)Measuring multimodal mathematical reasoning with math\-vision dataset\.Advances in Neural Information Processing Systems37,pp\. 95095–95169\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1)\.
- \[54\]W\. Wang, Z\. Gao, L\. Gu, H\. Pu, L\. Cui, X\. Wei, Z\. Liu, L\. Jing, S\. Ye, J\. Shao,et al\.\(2025\)Internvl3\. 5: advancing open\-source multimodal models in versatility, reasoning, and efficiency\.arXiv preprint arXiv:2508\.18265\.Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.52.52.52.52.7),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.97.97.97.101.1),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.97.97.97.97.7)\.
- \[55\]H\. Wei, Y\. Sun, and Y\. Li\(2025\)Deepseek\-ocr: contexts optical compression\.arXiv preprint arXiv:2510\.18234\.Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.76.76.76.76.10)\.
- \[56\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p2.1)\.
- \[57\]Y\. Xiao, E\. Sun, T\. Liu, and W\. Wang\(2024\)Logicvista: multimodal llm logical reasoning benchmark in visual contexts\.arXiv preprint arXiv:2407\.04973\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1)\.
- \[58\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2609.13158#S4.SS1.p2.1)\.
- \[59\]S\. Yu, C\. Tang, B\. Xu, J\. Cui, J\. Ran, Y\. Yan, Z\. Liu, S\. Wang, X\. Han, Z\. Liu,et al\.\(2025\)VisRAG: vision\-based retrieval\-augmented generation on multi\-modality documents\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§3\.3](https://arxiv.org/html/2609.13158#S3.SS3.p1.1)\.
- \[60\]X\. Yue, Y\. Ni, K\. Zhang, T\. Zheng, R\. Liu, G\. Zhang, S\. Stevens, D\. Jiang, W\. Ren, Y\. Sun,et al\.\(2024\)Mmmu: a massive multi\-discipline multimodal understanding and reasoning benchmark for expert agi\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 9556–9567\.Cited by:[§1](https://arxiv.org/html/2609.13158#S1.p1.1),[§1](https://arxiv.org/html/2609.13158#S1.p5.1),[§2](https://arxiv.org/html/2609.13158#S2.p3.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.14.1)\.
- \[61\]X\. Yue, T\. Zheng, Y\. Ni, Y\. Wang, K\. Zhang, S\. Tong, Y\. Sun, B\. Yu, G\. Zhang, H\. Sun,et al\.\(2025\)Mmmu\-pro: a more robust multi\-discipline multimodal understanding benchmark\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15134–15186\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1),[§3\.3](https://arxiv.org/html/2609.13158#S3.SS3.p2.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.15.1)\.
- \[62\]P\. Zhang, X\. Dong, Y\. Zang, Y\. Cao, R\. Qian, L\. Chen, Q\. Guo, H\. Duan, B\. Wang, L\. Ouyang,et al\.\(2024\)Internlm\-xcomposer\-2\.5: a versatile large vision language model supporting long\-contextual input and output\.arXiv preprint arXiv:2407\.03320\.Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.61.61.61.61.10)\.
- \[63\]R\. Zhang, D\. Jiang, Y\. Zhang, H\. Lin, Z\. Guo, P\. Qiu, A\. Zhou, P\. Lu, K\. Chang, Y\. Qiao,et al\.\(2024\)Mathverse: does your multi\-modal llm truly see the diagrams in visual math problems?\.InEuropean Conference on Computer Vision,pp\. 169–186\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1),[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.13.1)\.
- \[64\]T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi\(2020\)BERTScore: evaluating text generation with bert\.InInternational Conference on Learning Representations,Cited by:[Table 1](https://arxiv.org/html/2609.13158#S3.T1.1.1.5.7)\.
- \[65\]W\. Zhang, M\. Aljunied, C\. Gao, Y\. K\. Chia, and L\. Bing\(2023\)M3exam: a multilingual, multimodal, multilevel benchmark for examining large language models\.Advances in Neural Information Processing Systems36,pp\. 5484–5505\.Cited by:[§2](https://arxiv.org/html/2609.13158#S2.p3.1)\.
- \[66\]J\. Zhu, W\. Wang, Z\. Chen, Z\. Liu, S\. Ye, L\. Gu, H\. Tian, Y\. Duan, W\. Su, J\. Shao,et al\.\(2025\)Internvl3: exploring advanced training and test\-time recipes for open\-source multimodal models\.arXiv preprint arXiv:2504\.10479\.Cited by:[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.36.36.36.36.1),[§4\.2](https://arxiv.org/html/2609.13158#S4.SS2.85.85.85.85.10)\.

## Appendix

## Contents

## Appendix ATheoretical Derivation about the impact of redundant information

To characterize how redundant information disrupts LLM reasoning, we quantify this via symbolic derivation\.

We consider two distinct scenarios\. In the first, LVLMs perform multimodal reasoning and analysis within a context devoid of interference, where both the input images and textual content are solely intended to provide evidence for problem\-solving\. In contrast, real\-world scenarios are considerably more complex: beyond useful cues, the input often contains a substantial amount of irrelevant content\. At the visual level, this manifests as numerous unrelated images, where relevant images are interspersed among all images and jointly provided as input to the LLM\.

LetMMdenote the number of relevant tokens received by the model in the idealized setting, where the input contains only the question string and the visually relevant pages associated with the question, collectively covering all evidence required for question answering; this corresponds to a clean\-case scenario\. In the presence ofNNirrelevant visual tokens, the total number of tokens becomesM\+NM\+N\. This redundancy serves to divert the model’s attention from the relevant tokens to the irrelevant ones, thereby contaminating the hidden representations and diminishing the effectiveness of the information\.

However, it is important to emphasize that such redundant information must exhibit a certain degree of confusability\. Specifically, the visual features of the irrelevant images should bear strong similarity to the target problem, thereby inducing a high level of distraction and semantic ambiguity\. Only under this condition do the redundant tokens possess substantial semantic attraction; otherwise, the model will allocate negligible attention to these tokens\.

We start fromthe first layer\. Let the attention computation for thexx\-th token in the clean case be represented ashx1h^\{1\}\_\{x\}\. Upon the introduction ofNNirrelevant visual tokens, the hidden state for thexx\-th token after the attention layer evolves tohx′⁣1h^\{\\prime 1\}\_\{x\}:

hx1=h^x0\+\\displaystyle h^\{1\}\_\{x\}=\\hat\{h\}^\{0\}\_\{x\}\+α1​V1\+α2​V2\+…\+αM​VM,\\displaystyle\\alpha\_\{1\}V\_\{1\}\+\\alpha\_\{2\}V\_\{2\}\+\\ldots\+\\alpha\_\{M\}V\_\{M\},\(12\)hx′⁣1=h^x′⁣0\+\\displaystyle h^\{\\prime 1\}\_\{x\}=\\hat\{h\}^\{\\prime 0\}\_\{x\}\+β1​V1′\+β2​V2′\+…\+βM\+N​VM\+N′,\\displaystyle\\beta\_\{1\}V^\{\\prime\}\_\{1\}\+\\beta\_\{2\}V^\{\\prime\}\_\{2\}\+\\ldots\+\\beta\_\{M\+N\}V^\{\\prime\}\_\{M\+N\},\(13\)whereh^x0\\hat\{h\}^\{0\}\_\{x\}andh^x′⁣0\\hat\{h\}^\{\\prime 0\}\_\{x\}denote the output states of the previous decoder block \(here, the initial embeddings\), representing the effect of the residual connection,αi,βi\\alpha\_\{i\},\\beta\_\{i\}represent the attention weights, andVi,Vi′∈ℝdV\_\{i\},V^\{\\prime\}\_\{i\}\\in\\mathbb\{R\}^\{d\}are the*value*vectors at theii\-th position\. The order of the tokens does not affect the derivation; for simplicity, we assume that the firstMMtokens are relevant, while the subsequentNNtokens are irrelevant\.

Due to the normalization of attention scores, both∑i=1Mαi\\sum\_\{i=1\}^\{M\}\\alpha\_\{i\}and∑j=1M\+Nβj\\sum\_\{j=1\}^\{M\+N\}\\beta\_\{j\}are equal to 1\. After the addition of redundant tokens, the attention scores for theMMrelevant tokens are diluted and redistributed across theNNirrelevant tokens\. To standardize the notation, let the transformation of the average scores before and after the redistribution in relevant tokens be expressed as:

βi=MM\+N⋅δ​αi,i=1,2,…,M,\\beta\_\{i\}=\\frac\{M\}\{M\+N\}\\cdot\\delta\\alpha\_\{i\},\\quad i=1,2,\\ldots,M,\(14\)whereδ\\deltais a harmonic coefficient\. For convenience, we assume that theδ\\deltavalue is equal across all tokens after averaging\. Given that the model inherently exhibits a strong bias towards relevant information, the degree of attenuation of the attention scores for theMMrelevant tokens is not as severe asMM\+N\\frac\{M\}\{M\+N\}compared to the clean case, henceδ\>1\\delta\>1\.

Therefore, the expression forhx′h^\{\\prime\}\_\{x\}in the first layer of the decoder block is given by:

hx′⁣1=h^x′⁣0\+β1​V1′\+β2​V2′\+…\+βM\+N​VM\+N′=h^x′⁣0\+δ​MM\+N​\(α1​V1′\+α2​V2′\+…\+αM​VM′\)\+βM\+1​VM\+1′\+…\+βM\+N​VM\+N′=h^x′⁣0\+δ​MM\+N​∑i=1Mαi​Vi′\+∑j=M\+1M\+Nβj​Vj′,δ\>1\.\\begin\{split\}h^\{\\prime 1\}\_\{x\}=&\\hat\{h\}^\{\\prime 0\}\_\{x\}\+\\beta\_\{1\}V^\{\\prime\}\_\{1\}\+\\beta\_\{2\}V^\{\\prime\}\_\{2\}\+\\ldots\+\\beta\_\{M\+N\}V^\{\\prime\}\_\{M\+N\}\\\\ =&\\hat\{h\}^\{\\prime 0\}\_\{x\}\+\\frac\{\\delta M\}\{M\+N\}\(\\alpha\_\{1\}V^\{\\prime\}\_\{1\}\+\\alpha\_\{2\}V^\{\\prime\}\_\{2\}\+\\ldots\+\\alpha\_\{M\}V^\{\\prime\}\_\{M\}\)\\\\ &\+\\beta\_\{M\+1\}V^\{\\prime\}\_\{M\+1\}\+\\ldots\+\\beta\_\{M\+N\}V^\{\\prime\}\_\{M\+N\}\\\\ =&\\hat\{h\}^\{\\prime 0\}\_\{x\}\+\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\+\\sum^\{M\+N\}\_\{j=M\+1\}\\beta\_\{j\}V^\{\\prime\}\_\{j\},\\quad\\delta\>1\.\\end\{split\}\(15\)
The computation ofhx′h^\{\\prime\}\_\{x\}above reflects the fact that, in the presence of redundant context, after the first layer of attention, the hidden state at each token position will be mixed with information derived from the irrelevant tokens\. As a result, the information purity of the hidden state is diminished\.

We quantify theinformation purityof thexx\-th token after the first layer asη\\eta, which represents the proportion of information from relevant tokens in thexx\-th hidden state after the first layer of attention, relative to the total value of thexx\-th hidden state:

η=h^x′⁣0\+δ​MM\+N​∑i=1Mαi​Vi′h^x′⁣0\+δ​MM\+N​∑i=1Mαi​Vi′\+∑j=M\+1M\+Nβj​Vj′=1\+h^x′⁣0δ​MM\+N​∑i=1Mαi​Vi′1\+h^x′⁣0δ​MM\+N​∑i=1Mαi​Vi′\+M\+Nδ​M⋅∑j=M\+1M\+Nβj​Vj′∑i=1Mαi​Vi′=1\+h^x′⁣0∑i=1Mβi​Vi′1\+h^x′⁣0∑i=1Mβi​Vi′\+M\+Nδ​M⋅∑j=M\+1M\+Nβj​Vj′∑i=1Mαi​Vi′∈\(0,1\),δ\>1\.\\begin\{split\}\\eta=&\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\+\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\}\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\+\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\+\\sum^\{M\+N\}\_\{j=M\+1\}\\beta\_\{j\}V^\{\\prime\}\_\{j\}\}\\\\ =&\\frac\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\}\}\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\}\+\\frac\{M\+N\}\{\\delta M\}\\cdot\\frac\{\\sum^\{M\+N\}\_\{j=M\+1\}\\beta\_\{j\}V^\{\\prime\}\_\{j\}\}\{\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\}\}\\\\ =&\\frac\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}\}\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}\+\\frac\{M\+N\}\{\\delta M\}\\cdot\\frac\{\\sum^\{M\+N\}\_\{j=M\+1\}\\beta\_\{j\}V^\{\\prime\}\_\{j\}\}\{\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\}\}\\\\ \\in&\(0,1\),\\quad\\delta\>1\.\\\\ \\end\{split\}\(16\)At this point, we have obtained a dilution factorη∈\(0,1\)\\eta\\in\(0,1\)\. Since∑j=M\+1M\+Nβj​Vj′∑i=1Mαi​Vi′\\frac\{\\sum^\{M\+N\}\_\{j=M\+1\}\\beta\_\{j\}V^\{\\prime\}\_\{j\}\}\{\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\}represents the weighted value of theNNredundant tokens divided by the weighted value of theMMrelevant tokens, it is primarily influenced by the ratio ofMMtoNN\. In the extreme case, this ratio approachesNM\\frac\{N\}\{M\}\. However, due to the model’s inherent resistance to redundant information, the attention weightsβ\\betaallocated to the redundant tokens will be lower than those of the relevant tokens\. As a result, the numerator will be smaller than the ideal case\. Moreover, the weights in the denominator correspond to the original attention weightsα\\alphaof theMMrelevant tokens, rather than the diluted weightsβ\\beta, which further increases the denominator\. Consequently, the ratio cannot fully reach the value ofNM\\frac\{N\}\{M\}\. To account for this, we introduce a harmonic coefficientλ\\lambda, yielding a ratio ofλ​NM\\frac\{\\lambda N\}\{M\}, where0<λ<10<\\lambda<1\.

Therefore, the original equation \([16](https://arxiv.org/html/2609.13158#A1.E16)\) can be expressed as:

η=1\+h^x′⁣0∑i=1Mβi​Vi′1\+h^x′⁣0∑i=1Mβi​Vi′\+M\+Nδ​M⋅∑j=M\+1M\+Nβj​Vj′∑i=1Mαi​Vi′⇒1\+h^x′⁣0∑i=1Mβi​Vi′1\+h^x′⁣0∑i=1Mβi​Vi′\+M\+Nδ​M⋅λ​NM∈\(0,1\),δ\>1,λ∈\(0,1\)\.\\begin\{split\}\\eta&=\\frac\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}\}\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}\+\\frac\{M\+N\}\{\\delta M\}\\cdot\\frac\{\\sum^\{M\+N\}\_\{j=M\+1\}\\beta\_\{j\}V^\{\\prime\}\_\{j\}\}\{\\sum^\{M\}\_\{i=1\}\\alpha\_\{i\}V^\{\\prime\}\_\{i\}\}\}\\\\ &\\Rightarrow\\frac\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}\}\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}\+\\frac\{M\+N\}\{\\delta M\}\\cdot\\frac\{\\lambda N\}\{M\}\}\\\\ &\\in\(0,1\),\\quad\\delta\>1,\\lambda\\in\(0,1\)\.\\end\{split\}\(17\)
By analogy, referring to Eqs\. \([16](https://arxiv.org/html/2609.13158#A1.E16)\), on thexx\-th token, after reaching the second layer, the ratio of the value originating from relevant information in the hidden state to the total value is given by:

η2=\(h^x′⁣1\+δ​MM\+N​∑i=1Mαi2​Vi′⁣2\)⋅ηh^x′⁣1\+δ​MM\+N​∑i=1Mαi2​Vi′⁣2\+∑j=M\+1M\+Nβj2​Vj′⁣2\.\\eta\_\{2\}=\\frac\{\(\\hat\{h\}^\{\\prime 1\}\_\{x\}\+\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha^\{2\}\_\{i\}V^\{\\prime 2\}\_\{i\}\)\\cdot\\eta\}\{\\hat\{h\}^\{\\prime 1\}\_\{x\}\+\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha^\{2\}\_\{i\}V^\{\\prime 2\}\_\{i\}\+\\sum^\{M\+N\}\_\{j=M\+1\}\\beta^\{2\}\_\{j\}V^\{\\prime 2\}\_\{j\}\}\.\(18\)
In the formula, “22” denotes “the second layer”, not “square”\. Here, we explain why each value in the numerator must be multiplied by the dilution factorη\\etafrom the first layer: At this point, even the hidden states at the positions of the relevant tokens are contaminated by redundant information from the previous layer\. Not all the information at these positions is relevant, and the proportion of relevant information has already been computed asη\\eta\. Therefore, the actual relevant information at theii\-th position in this layer ish^i′⁣1​WV2⋅η=Vi′⁣2⋅η\\hat\{h\}^\{\\prime 1\}\_\{i\}W^\{2\}\_\{V\}\\cdot\\eta=V^\{\\prime 2\}\_\{i\}\\cdot\\eta, which is why we multiply each value in the numerator byη\\eta\.

Finally, analogous to Eqs\. \([16](https://arxiv.org/html/2609.13158#A1.E16)\) and \([17](https://arxiv.org/html/2609.13158#A1.E17)\) in the first layer, simplifying Eq\. \([18](https://arxiv.org/html/2609.13158#A1.E18)\) yields the information purity after the second layer:

η2=\(h^x′⁣1\+δ​MM\+N​∑i=1Mαi2​Vi′⁣2\)⋅ηh^x′⁣1\+δ​MM\+N​∑i=1Mαi2​Vi′⁣2\+∑j=M\+1M\+Nβj2​Vj′⁣2⇒1\+h^x′⁣1∑i=1Mβi2​Vi′⁣21\+h^x′⁣1∑i=1Mβi2​Vi′⁣2\+M\+Nδ​M⋅λ​NM⋅η=1\+h^x′⁣0⋅η∑i=1Mβi​Vi′⋅η1\+h^x′⁣0⋅η∑i=1Mβi​Vi′⋅η\+M\+Nδ​M⋅λ​NM⋅η=1\+h^x′⁣0∑i=1Mβi​Vi′1\+h^x′⁣0∑i=1Mβi​Vi′\+M\+Nδ​M⋅λ​NM⋅η=η2∈\(0,1\)\.\\begin\{split\}\\eta\_\{2\}=&\\frac\{\(\\hat\{h\}^\{\\prime 1\}\_\{x\}\+\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha^\{2\}\_\{i\}V^\{\\prime 2\}\_\{i\}\)\\cdot\\eta\}\{\\hat\{h\}^\{\\prime 1\}\_\{x\}\+\\frac\{\\delta M\}\{M\+N\}\\sum^\{M\}\_\{i=1\}\\alpha^\{2\}\_\{i\}V^\{\\prime 2\}\_\{i\}\+\\sum^\{M\+N\}\_\{j=M\+1\}\\beta^\{2\}\_\{j\}V^\{\\prime 2\}\_\{j\}\}\\\\ \\Rightarrow&\\frac\{1\+\\frac\{\\hat\{h\}^\{\\prime 1\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta^\{2\}\_\{i\}V^\{\\prime 2\}\_\{i\}\}\}\{1\+\\frac\{\\hat\{h\}^\{\\prime 1\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta^\{2\}\_\{i\}V^\{\\prime 2\}\_\{i\}\}\+\\frac\{M\+N\}\{\\delta M\}\\cdot\\frac\{\\lambda N\}\{M\}\}\\cdot\\eta\\\\ =&\\frac\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\\cdot\\eta\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\\cdot\\eta\}\}\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\\cdot\\eta\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\\cdot\\eta\}\+\\frac\{M\+N\}\{\\delta M\}\\cdot\\frac\{\\lambda N\}\{M\}\}\\cdot\\eta\\\\ =&\\frac\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}\}\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}\+\\frac\{M\+N\}\{\\delta M\}\\cdot\\frac\{\\lambda N\}\{M\}\}\\cdot\\eta=\\eta^\{2\}\\in\(0,1\)\.\\end\{split\}\(19\)In other words, after each decoder block, the proportion of relevant information in the hidden state \(the information purity\) is scaled byη\\eta\. Consequently, for anLL\-layer decoder, the effective information retained in the final hidden state is onlyηL\\eta^\{L\}of its original value, leading to an exponential compression of the informative content:

ηL=ηL,L∈ℤ\.\\eta\_\{\{\}\_\{L\}\}=\\eta^\{L\},\\quad L\\in\\mathbb\{Z\}\.\(20\)
Furthermore, we analyze the influence ofMMandNNonη\\eta:

From the result in Eq\. \([17](https://arxiv.org/html/2609.13158#A1.E17)\), we can observe that the term1\+h^x′⁣0∑i=1Mβi​Vi′1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\sum\_\{i=1\}^\{M\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}appears entirely in the denominator and constitutes a constant greater than11\. For notational simplicity, we denote this constant bybb\. We focus on the remaining denominator termM\+Nδ​M⋅λ​NM\\frac\{M\+N\}\{\\delta M\}\\cdot\\frac\{\\lambda N\}\{M\}:

η⇒1\+h^x′⁣0∑i=1Mβi​Vi′1\+h^x′⁣0∑i=1Mβi​Vi′\+M\+Nδ​M⋅λ​NM=bb\+M\+Nδ​M⋅λ​NM=bb\+λ​M​N\+λ​N2δ​M2=bb\+λ​NM\+λ​\(NM\)2δ=bb\+λδ​\(NM\+\(NM\)2\)\.\\begin\{split\}\\eta&\\Rightarrow\\frac\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}\}\{1\+\\frac\{\\hat\{h\}^\{\\prime 0\}\_\{x\}\}\{\\sum^\{M\}\_\{i=1\}\\beta\_\{i\}V^\{\\prime\}\_\{i\}\}\+\\frac\{M\+N\}\{\\delta M\}\\cdot\\frac\{\\lambda N\}\{M\}\}\\\\ &=\\frac\{b\}\{b\+\\frac\{M\+N\}\{\\delta M\}\\cdot\\frac\{\\lambda N\}\{M\}\}=\\frac\{b\}\{b\+\\frac\{\\lambda MN\+\\lambda N^\{2\}\}\{\\delta M^\{2\}\}\}=\\frac\{b\}\{b\+\\frac\{\\lambda\\frac\{N\}\{M\}\+\\lambda\\left\(\\frac\{N\}\{M\}\\right\)^\{2\}\}\{\\delta\}\}\\\\ &=\\frac\{b\}\{b\+\\frac\{\\lambda\}\{\\delta\}\\left\(\\frac\{N\}\{M\}\+\\left\(\\frac\{N\}\{M\}\\right\)^\{2\}\\right\)\}\.\\end\{split\}\(21\)
As shown above, the denominator reduces to a univariate quadratic formx2\+xx^\{2\}\+x, wherex=NMx=\\frac\{N\}\{M\}\. SinceNM\>0\\frac\{N\}\{M\}\>0and the functionx2\+xx^\{2\}\+xis monotonically increasing forx\>0x\>0, the denominator increases monotonically asNM\\frac\{N\}\{M\}grows, i\.e\., as redundant tokens constitute a larger proportion of the input\. Consequently,η\\etadecreases monotonically, indicating a degradation in information purity\.

We therefore conclude that as the ratio of redundant tokens to relevant tokens increases, the purity of information in the hidden states decreases, which aligns with intuitive expectations\.

Even ifη\\etais a value close to11, its impact under exponentiation can still be substantial\. Given the increasing number of layers in modern LLMs and vision encoders, this phenomenon warrants more attention\. The contamination of the hidden state with redundant information as additional tokens are incorporated hampers the ability to capture useful signals, ultimately disrupting both vocabulary decoding and output generation\.

Given the inevitability of redundancy in real\-world data, we argue that an LVLM’s capacity to maintain reasoning performance under redundant conditions—rather than under an unrealistically clean setting—should be given greater emphasis as an extended evaluation criterion\. Our benchmark is designed precisely for this setting\.

## Appendix BMulti\-Level Redundant Context Injection Procedure

In TestHallVQA, each question is associated with multiple levels of contextual redundancy, which we refer to asregimes\. Under our design, regimes are primarily distinguished by the number of pages they include, with an approximate interval of ten pages between adjacent regimes\. Based on dataset statistics, the maximum length of exam documents in our data source is 44 pages\. Accordingly, we define up to five regimes \(*Regime 1*through*Regime 5*\)\. The annotation procedure is described as Algorithm[1](https://arxiv.org/html/2609.13158#alg1):

Algorithm 1Multi\-Level Redundant Context Generation1:Document total pages

NN, Relevant page set

RR\(

\|R\|≤4\|R\|\\leq 4\)

2:

R≤NR\\leq N
3:

N^←10×round​\(N/10\)\\hat\{N\}\\leftarrow 10\\times\\text\{round\}\(N/10\)⊳\\trianglerightRound maximum pages to nearest multiple of 10

4:

K←N^/10\+1K\\leftarrow\\hat\{N\}/10\+1⊳\\trianglerightTotal number of regimes

5:Initialize an array

PPof size

KKto store page counts for each regime

6:

P1←\|R\|P\_\{1\}\\leftarrow\|R\|⊳\\trianglerightRegime 1 page count is the size of the relevant page set

7:for

i=2i=2to

KKdo

8:

Pi←10×\(i−1\)P\_\{i\}\\leftarrow 10\\times\(i\-1\)⊳\\trianglerightRegime pages: 10, 20, 30, …

9:endfor

10:if

N<N^N<\\hat\{N\}then

11:

PK←NP\_\{K\}\\leftarrow N⊳\\trianglerightRestore the last regime’s page count toNN

12:endif

13:

𝒮←∅\\mathcal\{S\}\\leftarrow\\emptyset⊳\\trianglerightThe set of redundant page collections \(Regimes\)

14:for

i=1i=1to

KKdo

15:

Si←RS\_\{i\}\\leftarrow R
16:// Step 1: Fill internal gaps between min and max pages

17:

pm​i​n←min⁡\(R\)p\_\{min\}\\leftarrow\\min\(R\)
18:

pmax←max⁡\(R\)p\_\{\\max\}\\leftarrow\\max\(R\)
19:

C←\{p∣pm​i​n≤p≤pmax\}∖SiC\\leftarrow\\\{p\\mid p\_\{min\}\\leq p\\leq p\_\{\\max\}\\\}\\setminus S\_\{i\}
20:while

\|Si\|<Pi\|S\_\{i\}\|<P\_\{i\}and

C≠∅C\\neq\\emptysetdo

21:Extract a page

ppfrom

CCand

Si←Si∪\{p\}S\_\{i\}\\leftarrow S\_\{i\}\\cup\\\{p\\\}
22:endwhile

23:// Step 2: Automated left\-right expansion strategy

24:

l​e​f​t←pm​i​n−1left\\leftarrow p\_\{min\}\-1
25:

r​i​g​h​t←pmax\+1right\\leftarrow p\_\{\\max\}\+1
26:while

\|Si\|<Pi\|S\_\{i\}\|<P\_\{i\}and\(

l​e​f​t≥1left\\geq 1or

r​i​g​h​t≤Nright\\leq N\)do

27:

d​i​r←RandomChoice​\(\{left,right\}\)dir\\leftarrow\\text\{RandomChoice\}\(\\\{\\text\{left\},\\text\{right\}\\\}\)
28:if

d​i​r=leftdir=\\text\{left\}and

l​e​f​t≥1left\\geq 1then

29:

Si←Si∪\{l​e​f​t\}S\_\{i\}\\leftarrow S\_\{i\}\\cup\\\{left\\\}
30:

l​e​f​t←l​e​f​t−1left\\leftarrow left\-1
31:elseif

d​i​r=rightdir=\\text\{right\}and

r​i​g​h​t≤Nright\\leq Nthen

32:

Si←Si∪\{r​i​g​h​t\}S\_\{i\}\\leftarrow S\_\{i\}\\cup\\\{right\\\}
33:

r​i​g​h​t←r​i​g​h​t\+1right\\leftarrow right\+1
34:endif

35:endwhile

36:

𝒮←𝒮∪\{Si\}\\mathcal\{S\}\\leftarrow\\mathcal\{S\}\\cup\\\{S\_\{i\}\\\}
37:endfor

38:return

𝒮\\mathcal\{S\}

Through this annotation process, the difference in page count between adjacent redundant regimes is at least five pages \(except for*Regime 1*, which represents strictly relevant pages\)\. This design ensures clear separability between redundancy levels while maximizing the utilization of available exam pages\.

## Appendix CTestHallVQA Dataset Details

Table[4](https://arxiv.org/html/2609.13158#A3.T4)presents key statistical information about TestHallVQA, enabling researchers to gain an intuitive overview of the dataset’s composition\. Table[5](https://arxiv.org/html/2609.13158#A3.T5)reports results on a*mini\-test*set, which is obtained via class\-wise random sampling from the full TestHallVQA dataset\. This subset preserves the distribution of problem types while significantly reducing evaluation cost, enabling efficient comparison and ablation analysis without affecting the overall evaluation trends\.

StatisticNumberTotalTotal Question ID Samples10,242Total Subquestion Samples11,317Total Images7,155SubjectsMathematics3,302Physics3,231Chemistry3,709Question TypesNormal4,189Calculate2,755Statement1,940Choice1,923Proof410Binary100Context RegimesRegime 1 \(relevant pages, 1–4 pages\)10,242Regime 2 \(5–10 pages\)9,327Regime 3 \(15–20 pages\)7,977Regime 4 \(25–30 pages\)3,772Regime 5 \(35–40 pages\)859Maximum, Minimum, and AverageMaximum Exam Document Length44Minimum Exam Document Length1Average Exam Document Length18\.73Average Regime 1 Pages1\.55Average Regime 2 Pages9\.92Average Regime 3 Pages19\.22Average Regime 4 Pages28\.62Average Regime 5 Pages37\.13
Table 4:Detailed statistics of the full TestHallVQA test set\. Some question IDs contain multiple subquestions, each of which requires a separate answer\.StatisticNumberTotalTotal Question ID Samples1,000Total Subquestion Samples1,133Total Images734SubjectsMathematics297Physics345Chemistry358Question TypesNormal410Calculate269Statement198Choice201Proof43Binary12Context RegimesRegime 1 \(relevant pages, 1–4 pages\)1,000Regime 2 \(5–10 pages\)926Regime 3 \(15–20 pages\)781Regime 4 \(25–30 pages\)373Regime 5 \(35–40 pages\)88Maximum, Minimum, and AverageMaximum Exam Document Length41Minimum Exam Document Length1Average Exam Document Length19\.56Average Regime 1 Pages1\.41Average Regime 2 Pages9\.35Average Regime 3 Pages19\.72Average Regime 4 Pages29\.03Average Regime 5 Pages36\.50
Table 5:Detailed statistics of the TestHallVQA mini\-test set\.

## Appendix DAblation Setup for theSR​TS\_\{RT\}Metric

As mentioned in the main text, to demonstrate the advantage ofSR​TS\_\{RT\}in F1\-R2, we compare it with several baseline metrics, including step retention ratio \(SRR\), end\-to\-end retention \(E2E\), normalized AUC \(N\-AUC\), and log\-based decay \(LD\)\. In this section, we present the computation of each baseline metric in detail\.

We continue to use the illustrative example from Section[3\.5](https://arxiv.org/html/2609.13158#S3.SS5)\. Specifically, for a sub\-test set with three regimes, let the average accuracies under Regime 1 to Regime 3 be denoted bye1,e2,e3∈\[0,100\]e\_\{1\},e\_\{2\},e\_\{3\}\\in\[0,100\], respectively\.

Step retention ratio \(SRR\)directly characterizes performance degradation using relative decay, namely, the score ratio between two consecutive regimes, i\.e\.,e2e1\\frac\{e\_\{2\}\}\{e\_\{1\}\}ande3e2\\frac\{e\_\{3\}\}\{e\_\{2\}\}\. After normalization, it is defined as:

SR​T​\-​S​R​R=Δ​p1⋅e2e1\+Δ​p2⋅e3e2Δ​p1\+Δ​p2,S\_\{RT\\text\{\-\}SRR\}=\\frac\{\\Delta p\_\{1\}\\cdot\\frac\{e\_\{2\}\}\{e\_\{1\}\}\+\\Delta p\_\{2\}\\cdot\\frac\{e\_\{3\}\}\{e\_\{2\}\}\}\{\\Delta p\_\{1\}\+\\Delta p\_\{2\}\},\(22\)whereΔ​p1=p¯2−p¯1\\Delta p\_\{1\}=\\bar\{p\}\_\{2\}\-\\bar\{p\}\_\{1\}andΔ​p2=p¯3−p¯2\\Delta p\_\{2\}=\\bar\{p\}\_\{3\}\-\\bar\{p\}\_\{2\}\. The interpretation of this metric is straightforward: it measures how much performance is retained for each additional block of redundant pages\.

End\-to\-end retention \(E2E\)measures how much performance remains from the oracle setting to the most challenging regime\. It can be viewed as a simplified version of step retention ratio, with particularly strong interpretability\. Instead of considering the intermediate trend of the performance curve, it directly compares the initial and final regimes:

SR​T​\-​E​2​E=e3e1\.S\_\{RT\\text\{\-\}E2E\}=\\frac\{e\_\{3\}\}\{e\_\{1\}\}\.\(23\)
Normalized AUC \(N\-AUC\)is closely related to the rotation\-angle formulation, but removes thearctan\\arctantransformation\. It is therefore well suited to addressing a central question: does the angle transformation truly provide additional discriminative power, or is the area\-based characterization alone already sufficient?

Since the rotation\-angle metric in the main paper is itself motivated by the interpretation of the “removed triangular area,” the most natural baseline is to directly use the ratio between the trapezoidal area remaining after removing the triangle and the area of the original rectangle:

SRT\-AUC=e1\+e22​Δ​p1e1​Δ​p1⋅Δ​p1Δ​p1\+Δ​p2\+e2\+e32​Δ​p2e2​Δ​p2⋅Δ​p2Δ​p1\+Δ​p2,S\_\{\\text\{RT\-AUC\}\}=\\frac\{\\frac\{e\_\{1\}\+e\_\{2\}\}\{2\}\\Delta p\_\{1\}\}\{e\_\{1\}\\Delta p\_\{1\}\}\\cdot\\frac\{\\Delta p\_\{1\}\}\{\\Delta p\_\{1\}\+\\Delta p\_\{2\}\}\+\\frac\{\\frac\{e\_\{2\}\+e\_\{3\}\}\{2\}\\Delta p\_\{2\}\}\{e\_\{2\}\\Delta p\_\{2\}\}\\cdot\\frac\{\\Delta p\_\{2\}\}\{\\Delta p\_\{1\}\+\\Delta p\_\{2\}\},\(24\)whereΔ​p1=p¯2−p¯1\\Delta p\_\{1\}=\\bar\{p\}\_\{2\}\-\\bar\{p\}\_\{1\}andΔ​p2=p¯3−p¯2\\Delta p\_\{2\}=\\bar\{p\}\_\{3\}\-\\bar\{p\}\_\{2\}\. This formulation likewise yields a score in the range\[0,1\]\[0,1\]\.

Log\-Based Decay \(LD\)quantifies performance degradation using a logarithmic transformation\. Sincee2e1∈\[0,1\]\\frac\{e\_\{2\}\}\{e\_\{1\}\}\\in\[0,1\]represents the proportion of performance retained,1−e2e1∈\[0,1\]1\-\\frac\{e\_\{2\}\}\{e\_\{1\}\}\\in\[0,1\]correspondingly measures the proportion of performance lost\. A larger loss should lead to a lower final score\. The negative logarithm, i\.e\.,f​\(x\)=−log⁡\(x\)f\(x\)=\-\\log\(x\), is well suited to this purpose over the interval\[0,1\]\[0,1\]\. Moreover, it is more sensitive to values approaching zero, thereby improving sensitivity to mild degradation:

SRT\-LD1′=−log⁡\(1−e2e1\+ε\),S\_\{\\text\{RT\-LD\}\_\{1\}\}^\{\\prime\}=\-\\log\\left\(1\-\\frac\{e\_\{2\}\}\{e\_\{1\}\}\+\\varepsilon\\right\),\(25\)whereε\\varepsilonis a small constant introduced to avoid numerical overflow at zero; in our experiments, we setε=10−3\\varepsilon=10^\{\-3\}\.

We then normalize this quantity by considering its boundary cases\. Whene2=0e\_\{2\}=0, corresponding to the worst possible performance, the value becomes−log⁡\(1\+ε\)\-\\log\(1\+\\varepsilon\)\. Whene2=e1e\_\{2\}=e\_\{1\}, corresponding to perfect retention, the value becomes−log⁡\(ε\)\-\\log\(\\varepsilon\)\. Therefore, the normalized score for the first segment is:

SRT\-LD1=−log⁡\(1−e2e1\+ε\)−\(−log⁡\(1\+ε\)\)−log⁡\(ε\)−\(−log⁡\(1\+ε\)\)=log⁡\(1\+ε\)−log⁡\(1−e2e1\+ε\)log⁡\(1\+ε\)−log⁡\(ε\)\.S\_\{\\text\{RT\-LD\}\_\{1\}\}=\\frac\{\-\\log\\left\(1\-\\frac\{e\_\{2\}\}\{e\_\{1\}\}\+\\varepsilon\\right\)\-\\left\(\-\\log\(1\+\\varepsilon\)\\right\)\}\{\-\\log\(\\varepsilon\)\-\\left\(\-\\log\(1\+\\varepsilon\)\\right\)\}=\\frac\{\\log\(1\+\\varepsilon\)\-\\log\\left\(1\-\\frac\{e\_\{2\}\}\{e\_\{1\}\}\+\\varepsilon\\right\)\}\{\\log\(1\+\\varepsilon\)\-\\log\(\\varepsilon\)\}\.\(26\)
Similarly, the score for the second segment is:

SRT\-LD2=log⁡\(1\+ε\)−log⁡\(1−e3e2\+ε\)log⁡\(1\+ε\)−log⁡\(ε\)\.S\_\{\\text\{RT\-LD\}\_\{2\}\}=\\frac\{\\log\(1\+\\varepsilon\)\-\\log\\left\(1\-\\frac\{e\_\{3\}\}\{e\_\{2\}\}\+\\varepsilon\\right\)\}\{\\log\(1\+\\varepsilon\)\-\\log\(\\varepsilon\)\}\.\(27\)
The final normalized metric is computed as:

SRT\-LD=SRT\-LD1⋅Δ​p1Δ​p1\+Δ​p2\+SRT\-LD2⋅Δ​p2Δ​p1\+Δ​p2\.S\_\{\\text\{RT\-LD\}\}=S\_\{\\text\{RT\-LD\}\_\{1\}\}\\cdot\\frac\{\\Delta p\_\{1\}\}\{\\Delta p\_\{1\}\+\\Delta p\_\{2\}\}\+S\_\{\\text\{RT\-LD\}\_\{2\}\}\\cdot\\frac\{\\Delta p\_\{2\}\}\{\\Delta p\_\{1\}\+\\Delta p\_\{2\}\}\.\(28\)

## Appendix EExtended Ablation Analyses

### E\.1Cross\-Page Question Distractiveness Analysis

As TestHallVQA involves cross\-page evidence aggregation \(as shown in Figure[3](https://arxiv.org/html/2609.13158#S3.F3)\), we report the accuracy on questions with different page spans in the Regime 1 subset\. Figure[10](https://arxiv.org/html/2609.13158#A5.F10)shows that accuracy generally declines as the number of spanned pages increases, although the trend is not strictly monotonic\. This is because, while cross\-page reasoning introduces greater visual challenges, the intrinsic difficulty of the questions also plays a substantial role, and the reasoning difficulty is only weakly correlated with the number of pages spanned\.

![Refer to caption](https://arxiv.org/html/2609.13158v1/x10.png)Figure 10:Performance differences for questions spanning different numbers of relevant pages\.
### E\.2Redundant Image Distractiveness Analysis

![Refer to caption](https://arxiv.org/html/2609.13158v1/x11.png)Figure 11:Comparison of Model Performance Perturbations under Different Redundant Image Selections\.To examine how the distractiveness of redundant information in TestHallVQA affects model performance, we replace the original redundant images, which are drawn from the same exam paper as the relevant image, with two alternative sources: \(1\) exam papers from different subjects within TestHallVQA, and \(2\) DocVQA\[[39](https://arxiv.org/html/2609.13158#bib.bib54)\]\.

For cross\-subject redundancy, we ensure that question index ranges do not overlap\. Specifically, for each question ID, we randomly select an exam paper containing the same ID, identify the page range of its relevant context, and sample redundant pages only from outside this range via random left–right expansion\. For DocVQA\-based redundancy, pages are randomly sampled from DocVQA until the target number of redundant images is reached\. Compared with exam paper images, DocVQA pages are much denser in text and visually distinct from the original exam\-style documents\.

In all settings, redundant images are resized to match the resolution of the corresponding relevant images, and the number of images is kept identical to the original annotations, ensuring consistent token length across configurations\.

We annotate these settings on the mini\-test split of TestHallVQA and evaluate several stable models from Table[4\.2](https://arxiv.org/html/2609.13158#S4.SS2)\. To isolate model behavior under different context scales, we report accuracy for each regime separately, without computing cross\-regime F1\-R2\. As shown in the figure[11](https://arxiv.org/html/2609.13158#A5.F11), unstructured redundant images are less misleading to models\. This suggests that existing datasets cannot achieve controlled inclusion of highly misleading redundant visual context through data augmentation alone, further underscoring the unique value of TestHallVQA\.

## Appendix FLimits of LVLMs in Multi\-Image Visual Encoding

![Refer to caption](https://arxiv.org/html/2609.13158v1/x12.png)\(a\)Visual encoding pipeline of conventional LVLMs\.
![Refer to caption](https://arxiv.org/html/2609.13158v1/x13.png)\(b\)Visual encoding pipeline of the Qwen2\.5/3\-VL series\.

Figure 12:Visual encoding pipelines\.Admittedly, the fundamental cause of the degradation in LVLMs’ reasoning performance under document\-level contexts lies in the phenomenon of redundant visual contamination\. However, through an analysis of failure cases, we observe that models frequently exhibit errors in locating the correct question identifier, corresponding to typed\.in Figure[9](https://arxiv.org/html/2609.13158#S5.F9)\. We attribute this issue to the fact that question identifiers in documents are often distributed across multiple pages, with different parts of a complete identifier scattered over several pages and organized in a sequential order\. Based on this observation, we infer that LVLMs perform poorly when processing document images with inherent temporal dependencies\.

Through a careful examination of both the source code and the corresponding papers, we indeed identified evidence supporting this hypothesis\. We find that, when processing multiple images, the visual encoder in LVLMs is unable to capture the sequential dependencies across images; that is, throughout the visual encoding stage, different images are either treated as order\-agnostic or are effectively isolated from one another\. Only when the visual tokens are first linearly serialized and fed into the LLM decoder do the cross\-page temporal dependencies begin to be modeled, by which point the opportunity to encode such relationships at the visual stage has already been missed\.

Specifically, we analyze this issue from the perspective of LVLM architecture and summarize two of the most prevalent visual encoder paradigms adopted by contemporary LVLMs\.

### F\.1Conventional LVLM Architectures

As illustrated in Figure[12\(a\)](https://arxiv.org/html/2609.13158#A6.F12.sf1), conventional LVLMs, such as InternVL and LLaVA, are built upon the standard Vision Transformer \(ViT\) architecture\. Input images are resized to a fixed resolution \(e\.g\., 448\) and fed into the ViT to obtain the corresponding visual tokens\. When the input resolution is excessively high, a*tiling*strategy is adopted: the image is divided into multiple sub\-images, which are processed as a batch by the ViT, while a resized version of the original image is additionally retained as a global\-context image\. This procedure naturally extends to the multi\-image setting, where each image is processed independently in the same manner, resulting in a larger batch of sub\-images\.

The limitation of this design is that it implicitly treats each sub\-image as an independent image, even though each of them represents only a local region of the original image\. The 2D rotary positional encoding \(2D\-RoPE\) in the ViT is applied only within each individual sub\-image and does not explicitly encode the spatial relationships among sub\-images\. As a result, sub\-images are encoded in a mutually isolated manner during visual encoding and remain unaware of one another\. Different sub\-images do not interact until they are passed to the LLM\. At that stage, the LLM applies one\-dimensional RoPE to both visual and textual tokens to impose a sequential ordering\.

However,\[[45](https://arxiv.org/html/2609.13158#bib.bib4)\]show that the norm of visual embeddings is typically one to three orders of magnitude larger than that of textual embeddings\. Consequently, when positional information is introduced only at the LLM stage, the resulting positional signal is inherently weak, as it is easily dominated by the much stronger visual features\.

Moreover, because inter\-sub\-image relationships are not modeled during visual encoding, the LLM is forced to reconstruct such dependencies from scratch\. This additional burden interferes with semantic understanding and reasoning, thereby distracting the LLM from the downstream task\.

### F\.2Emerging Architectures: Qwen2\.5/3\-VL

The Qwen2\.5/3\-VL family adopts a more recent visual\-encoder design, as illustrated in Figure[12\(b\)](https://arxiv.org/html/2609.13158#A6.F12.sf2)\. Unlike approaches that partition an image into multiple sub\-images and encode them separately, Qwen2\.5/3\-VL processes all visual tokens belonging to the same image jointly within the vision transformer\. This allows tokens from different spatial regions of an individual image to interact directly during visual encoding\.

For multi\-image inputs, however, the images remain isolated from one another inside the visual encoder\. Although the token sequences of multiple images may be packed together for efficient implementation, an attention mask prevents visual tokens from one image from attending to tokens belonging to another image\. Therefore, the vision encoder does not establish cross\-image correspondences or model the sequential relationships among images, and each image constitutes an independent attention unit within the visual\-encoding stage\.

The positional encodings used by the visual encoder are likewise defined independently for each image according to its two\-dimensional spatial structure\. They encode the height and width coordinates of tokens within an image, but do not represent the relative order of different images\. For example, tokens located at the same spatial coordinates in two identically sized images may receive the same two\-dimensional positional indices\. This does not cause them to interact within the vision encoder, because cross\-image attention is masked, but it also means that the visual encoder itself cannot distinguish or exploit the temporal ordering of the images\.

After visual encoding, the resulting image\-token sequences are projected and inserted into the input sequence of the LLM decoder\. At this stage, Qwen\-VL applies M\-RoPE\[[7](https://arxiv.org/html/2609.13158#bib.bib12),[6](https://arxiv.org/html/2609.13158#bib.bib11)\], which incorporates temporal, height, and width positional components\. The temporal component assigns distinct positional indices to different images or video frames, thereby introducing their sequential order into the language\-modeling stage\. Consequently, cross\-image relationships are modeled primarily by the LLM decoder rather than by the visual encoder\. This design preserves efficient per\-image visual encoding, but places the burden of integrating evidence and reasoning across multiple images on the LLM\.

### F\.3Summary

In summary, existing LVLMs areunable to perform joint multi\-image analysisat an early stage \(i\.e\., during visual encoding\)\. As a result, inter\-image relationships areinsufficiently modeled, and visual content is passed to the LLM before being adequately understood\. Consequently, the LLM decoder must devote additional capacity to processing and inferring temporal relationships among cross\-image tokens from scratch, thereby distributing attention more diffusely and furtherexacerbating the phenomenon of redundant visual contamination\.

This limitation is reflected in common error cases observed in TestHallVQA, such as misalignment in multi\-level question indexing and incomplete extraction of visual elements and regions\. Consequently, we expect future work to address this limitation and propose corresponding improvements\.

## Appendix GLLM\-as\-a\-Judge Scoring Details

We iteratively designed and refined a set of prompts—guided by continuous feedback from LLM outputs—to ultimately obtain a robust, fully instructed prompting framework\. This framework provides comprehensive reasoning guidance and complete logical coverage, ensuring strong generalization across the TestHallVQA benchmark\. It is used to elicit the analytical capabilities of an LLM acting as an evaluator, enabling it to assess the outputs of various LVLMs\.

For the six problem types, we design specialized prompts tailored to the unique reasoning requirements of each category\. In this section, we present the exact prompt templates and provide several representative interaction examples, offering an intuitive illustration of the evaluation process\.

The detailed content is shown in Figures[13\(a\)](https://arxiv.org/html/2609.13158#A7.F13.sf1)and[13\(b\)](https://arxiv.org/html/2609.13158#A7.F13.sf2), Figures[14\(a\)](https://arxiv.org/html/2609.13158#A7.F14.sf1)and[14\(b\)](https://arxiv.org/html/2609.13158#A7.F14.sf2), and Figures[15\(a\)](https://arxiv.org/html/2609.13158#A7.F15.sf1)and[15\(b\)](https://arxiv.org/html/2609.13158#A7.F15.sf2)\.

![Refer to caption](https://arxiv.org/html/2609.13158v1/x14.png)\(a\)Scoring process forNormal\-type questions\.
![Refer to caption](https://arxiv.org/html/2609.13158v1/x15.png)\(b\)Scoring process forCalculate\-type questions\.

Figure 13:LLM\-as\-a\-Judge scoring processes for Normal and Calculate questions\.![Refer to caption](https://arxiv.org/html/2609.13158v1/x16.png)\(a\)Scoring process forStatement\-type questions\.
![Refer to caption](https://arxiv.org/html/2609.13158v1/x17.png)\(b\)Scoring process forChoice\-type questions\.

Figure 14:LLM\-as\-a\-Judge scoring processes for Statement and Choice questions\.![Refer to caption](https://arxiv.org/html/2609.13158v1/x18.png)\(a\)Scoring process forProof\-type questions\.
![Refer to caption](https://arxiv.org/html/2609.13158v1/x19.png)\(b\)Scoring process forBinary\-type questions\.

Figure 15:LLM\-as\-a\-Judge scoring processes for Proof and Binary questions\.