Layers, Sinks, and Scaling: Adaptive Evidence Selection for Multimodal Large Language Models

# Layers, Sinks, and Scaling: Adaptive Evidence Selection for Multimodal Large Language Models
Source: [https://arxiv.org/html/2609.16795](https://arxiv.org/html/2609.16795)
###### Abstract

Multimodal large language models \(MLLMs\) can answer knowledge\-intensive visual questions by combining visual evidence from images with facts retrieved from external sources\. However, MLLMs may overlook relevant evidence in both modalities, attending weakly to the textual sentences or visual regions needed for the correct answer\. Recent efforts address this by highlighting retrieved text and marking visual regions before generation, but apply a fixed, one\-shot policy that cannot adapt to three sources of variation: whether highlighting is necessary, how much evidence different examples require, and when different textual evidence becomes relevant as the answer unfolds\. We introduce Adaptive Relevance\-guided Evidence Allocation \(AREA\), a training\-free inference\-time method that formulates evidence highlighting as adaptive allocation\. AREA generates a single probe token to read visual and textual relevance from fixed backbone layers, then makes three decisions:i\)whether to intervene \(controlled by natural attention coverage and visual sink contamination\),ii\)how much evidence to expose \(determined by relevance entropy\), andiii\)when to refresh text during generation \(triggered by causal context\-attention peaks\)\. Across four KB\-VQA and seven standard multimodal benchmarks with nine frozen MLLM checkpoints, establishes the best performance among training\-free highlighting methods\.

Sichuan University

wangzhenbin@stu\.scu\.edu\.cn

11footnotetext:Lei Zhang is the corresponding author
Code:https://github\.com/wongzbb/AREA## 1Introduction

Knowledge\-based visual question answering \(KB\-VQA\) requires multimodal large language models \(MLLMs\) to combine localized visual cues with facts retrieved from external knowledge sources\([Chen et al\. 2023](https://arxiv.org/html/2609.16795#bib.bib12);[Mensink et al\. 2023](https://arxiv.org/html/2609.16795#bib.bib14)\)\. Supplying both modalities, however, does not solve the task: the model must still determine which textual statements and image regions are jointly needed to answer the question\. Retrieved passages mix critical facts with irrelevant or weakly related content\([Caffagni et al\. 2024](https://arxiv.org/html/2609.16795#bib.bib2)\), while images contain many salient objects and regions beyond the answer\-bearing area\([Kang et al\. 2025](https://arxiv.org/html/2609.16795#bib.bib11)\)\. Consequently, the model can overlook a required sentence, focus on the wrong visual region, or fail to connect complementary evidence across modalities, producing an incorrect answer even when all required evidence is available\.

The challenge therefore lies not only in obtaining relevant evidence, but also in ensuring that the generator uses evidence already present in its input\. While retrieval and filtering methods improve which external content reaches the model\([Hong et al\. 2025](https://arxiv.org/html/2609.16795#bib.bib5);[Yan and Xie 2024](https://arxiv.org/html/2609.16795#bib.bib6);[Yang et al\. 2025](https://arxiv.org/html/2609.16795#bib.bib7);[Ye et al\. 2026](https://arxiv.org/html/2609.16795#bib.bib8)\), a complementary line of work targets evidence utilization at inference time\. Recent training\-free methods improve evidence utilization through explicit highlighting: SelfElicit highlights relevant context sentences, while Look Twice \(LoT\) extends the intervention to both retrieved text and visual regions\([Liu et al\. 2025](https://arxiv.org/html/2609.16795#bib.bib10);[Morini et al\. 2026](https://arxiv.org/html/2609.16795#bib.bib1)\)\. By converting selected evidence into sentence markers and visual crops, such methods make latent evidence explicit without updating model parameters\. However, they instantiate highlighting as a fixed, one\-shot policy: the same number of sentences and the same visual extent are selected once before answer generation\.

This design overlooks variation across examples and generation steps, as illustrated in Figure[1](https://arxiv.org/html/2609.16795#S1.F1)\. In some cases, the model already uses the relevant evidence, so additional highlighting is unnecessary\. The required evidence also varies in scope: one sentence or a compact image region may suffice for some questions, whereas others depend on multiple sentences or broader visual context\. As the answer unfolds, the relevant textual evidence can change, with later tokens depending on context that was not selected before decoding\. Static evidence allocation therefore cannot match heterogeneous and time\-varying demand\. An effective policy must decide whether to intervene, how much evidence to expose, and when to refresh textual evidence\.

![Refer to caption](https://arxiv.org/html/2609.16795v1/adalot_figure1.png)Figure 1:Schematic illustration of three mismatches between fixed, one\-shot evidence highlighting and evidence demand\. \(a\) Highlighting is redundant when the model already uses the relevant evidence\. \(b\) A fixed sentence budget can omit part of a multi\-sentence evidence chain\. \(c\) Evidence selected before decoding can become stale when later answer tokens require different textual evidence\. Blue marks the fixed selection, gold marks evidence already used or required at the corresponding stage, and red labels the resulting mismatch\.We introduce Adaptive Relevance\-guided Evidence Allocation \(AREA\), a training\-free inference\-time method that turns evidence highlighting into adaptive allocation\. A single probe token reads visual and textual relevance from fixed, backbone\-specific layer groups\. AREA uses the entropy of the textual and visual relevance distributions to set the number of selected sentences and the crop scale\. It then gates the two modalities independently: the text gate measures how much attention the selected sentences already receive, while the visual gate measures the fraction of the unfiltered visual relevance assigned to detected sink tokens\. During generation, AREA monitors attention over the original retrieved\-context tokens\. When its entropy exceeds a threshold computed from earlier decoding steps, the method reselects relevant sentences and appends them at the current decoding position, subject to a fixed refresh budget\. Together, these operations adapt whether to intervene, how much evidence to expose, and when to refresh textual evidence, while keeping all model parameters frozen\.

The main contributions of this work are summarized as follows:

- •We formalize adaptive evidence allocation for frozen MLLMs through three decisions: whether to mark selected context sentences and whether to add a visual crop, how many sentences to mark and what spatial extent the crop should cover, and at which decoding steps to reselect and append marked sentences\.
- •We develop AREA, a training\-free method for frozen MLLMs that obtains layer\-resolved visual and textual relevance in one probe pass, uses it to set sentence count and crop extent, gates text and vision with selected\-sentence attention surprisal and raw visual sink mass, and triggers text refresh from decoding\-time context attention under a fixed budget\.
- •We evaluate AREA with nine frozen MLLM checkpoints on four KB\-VQA and seven standard multimodal benchmarks, where it consistently establishes state\-of\-the\-art performance across both knowledge\-intensive and standard multimodal settings under a frozen\-backbone inference protocol\.

## 2Related Work

### 2\.1KB\-VQA and Multimodal Retrieval

KB\-VQA combines image understanding with facts from external sources\. Benchmarks like Encyclopedic VQA, InfoSeek, and ViQuAE test whether models can answer questions about fine\-grained entities, unseen knowledge, or facts requiring multi\-step reasoning\([Mensink et al\. 2023](https://arxiv.org/html/2609.16795#bib.bib14);[Chen et al\. 2023](https://arxiv.org/html/2609.16795#bib.bib12);[Lerner et al\. 2022](https://arxiv.org/html/2609.16795#bib.bib13)\)\. Systems that perform well on these tasks typically improve the upstream evidence pipeline: hierarchical retrieval over structured knowledge\([Caffagni et al\. 2024](https://arxiv.org/html/2609.16795#bib.bib2)\), multimodal reranking to prioritize relevant passages\([Yan and Xie 2024](https://arxiv.org/html/2609.16795#bib.bib6);[Yang et al\. 2025](https://arxiv.org/html/2609.16795#bib.bib7)\), learned retrieval\-relevance decisions\([Cocchi et al\. 2025](https://arxiv.org/html/2609.16795#bib.bib3)\), reasoning\-augmented retrieval\([Compagnoni et al\. 2026](https://arxiv.org/html/2609.16795#bib.bib4)\), question\-focused filtering\([Ye et al\. 2026](https://arxiv.org/html/2609.16795#bib.bib8)\), and multimodal knowledge graph integration\([Yuan et al\. 2026](https://arxiv.org/html/2609.16795#bib.bib9)\)\. These methods determine which external content reaches the model\. AREA addresses the complementary question: given that relevant evidence is available in the input, how should it be presented to ensure the frozen generator uses it? This distinction permits controlled comparisons under identical retrieval, isolating evidence utilization from retrieval quality\.

### 2\.2Inference\-Time Evidence Highlighting

Training\-free evidence highlighting provides a direct approach to improving evidence utilization without changing the retrieval pipeline or updating model parameters\. SelfElicit uses model\-derived relevance to select and mark context sentences, while LoT extends this approach to MLLMs by combining textual highlighting with visual localization\([Liu et al\. 2025](https://arxiv.org/html/2609.16795#bib.bib10);[Morini et al\. 2026](https://arxiv.org/html/2609.16795#bib.bib1)\)\. Together, these methods show that relevance signals from a frozen model can guide how available evidence is presented during inference\. Existing approaches, however, make the intervention in a single pre\-decoding step under a predetermined allocation policy\. AREA advances this direction by independently adapting the intervention for text and vision, scaling the exposed evidence to each example, and refreshing textual evidence as generation unfolds\.

## 3Method

### 3\.1Problem Setup and Overview

Given an imageII, a questionQQ, and a candidate contextCCofNCN\_\{C\}tokens, AREA generates an answer with a frozen MLLMℱθ\\mathcal\{F\}\_\{\\theta\}; vision\-only tasks haveNC=0N\_\{C\}=0\. The visual front end mapsIItoNV=G2N\_\{V\}=G^\{2\}spatial patch tokens arranged on a squareG×GG\\times Ggrid\. The decoder hasLLlayers,NHN\_\{H\}attention heads per layer, and hidden widthdd\. From the unhighlighted tokenized prompt,ℱθ\\mathcal\{F\}\_\{\\theta\}generates exactly one probe token\. LetSpS\_\{p\}denote the resulting sequence length andipi\_\{\\rm p\}the final position occupied by that token\. For layerℓ∈\{1,…,L\}\\ell\\in\\\{1,\\ldots,L\\\}and headh∈\{1,…,NH\}h\\in\\\{1,\\ldots,N\_\{H\}\\\}, the probe pass exposes post\-softmax causal attention𝐀pℓ,h∈\[0,1\]Sp×Sp\\mathbf\{A\}\_\{p\}^\{\\ell,h\}\\in\[0,1\]^\{S\_\{p\}\\times S\_\{p\}\}and hidden states𝐇pℓ∈ℝSp×d\\mathbf\{H\}\_\{p\}^\{\\ell\}\\in\\mathbb\{R\}^\{S\_\{p\}\\times d\}\. An entry𝐀pℓ,h​\[r,s\]\\mathbf\{A\}\_\{p\}^\{\\ell,h\}\[r,s\]gives attention from query positionrrto key positionss\. Tokenization maps visual patchvvto its absolute prompt positionχvis​\(v\)\\chi\_\{\\rm vis\}\(v\)and, whenNC\>0N\_\{C\}\>0, original context tokenjjtoχctx​\(j\)\\chi\_\{\\rm ctx\}\(j\)\.

Our objective is to allocate the available visual and textual evidence according to the evidence demand of each example and generation step\. AREA therefore decideswhetherto intervene in each modality,how muchevidence to expose through the number of selected sentences and the visual crop extent, andwhento refresh textual evidence during decoding, while keepingℱθ\\mathcal\{F\}\_\{\\theta\}frozen\.

Figure[2](https://arxiv.org/html/2609.16795#S3.F2)connects Layer\-Resolved Evidence Readout to Entropy\-Calibrated Evidence Scaling and Modality\-Specific Intervention Gating, followed by Causal Text\-Evidence Refresh during generation\. Algorithm[1](https://arxiv.org/html/2609.16795#alg1)presents the complete inference sequence\.

![Refer to caption](https://arxiv.org/html/2609.16795v1/framework.png)Figure 2:Overview of AREA\. One probe token reads visual and textual relevance from fixed, backbone\-specific layer groups of a frozen MLLM\. Independent gates use selected\-sentence attention coverage and raw visual sink mass to decide whether to intervene, while the sentence\-level and sink\-filtered spatial relevance distributions set the sentence count and crop extent\. After warmup, both gates use prior\-sample medians and update their histories only after deciding\. During the same autoregressive stream, strictly causal peaks in context\-attention entropy trigger bounded textual refresh; visual evidence and the initial gates remain fixed\.
### 3\.2Layer\-Resolved Evidence Readout

We use nonempty, fixed, backbone\-specific layer groupsℒvis,ℒtxt⊆\{1,…,L\}\\mathcal\{L\}\_\{\\rm vis\},\\mathcal\{L\}\_\{\\rm txt\}\\subseteq\\\{1,\\ldots,L\\\}for visual and textual relevance readout, together with nonempty fixed sink dimensions𝒟sink⊆\{1,…,d\}\\mathcal\{D\}\_\{\\rm sink\}\\subseteq\\\{1,\\ldots,d\\\}for activation\-based visual sink detection\([Kang et al\. 2025](https://arxiv.org/html/2609.16795#bib.bib11)\)\. The fixed question parser identifies the target\-object phrase; let𝒪\\mathcal\{O\}denote the nonempty set of its absolute prompt positions\. The same probe pass produces raw visual relevance for each patchvvand, when context is present, textual relevance for each original context tokenjj:

avis,vraw\\displaystyle a\_\{\{\\rm vis\},v\}^\{\\rm raw\}=∑o∈𝒪∑ℓ∈ℒvis∑h=1NH𝐀pℓ,h​\[o,χvis​\(v\)\]\|𝒪\|​\|ℒvis\|​NH,\\displaystyle=\\frac\{\\displaystyle\\sum\_\{o\\in\\mathcal\{O\}\}\\sum\_\{\\ell\\in\\mathcal\{L\}\_\{\\rm vis\}\}\\sum\_\{h=1\}^\{N\_\{H\}\}\\mathbf\{A\}\_\{p\}^\{\\ell,h\}\[o,\\chi\_\{\\rm vis\}\(v\)\]\}\{\|\\mathcal\{O\}\|\\,\|\\mathcal\{L\}\_\{\\rm vis\}\|N\_\{H\}\},\(1\)atxt,j\\displaystyle a\_\{\{\\rm txt\},j\}=∑ℓ∈ℒtxt∑h=1NH𝐀pℓ,h​\[ip,χctx​\(j\)\]\|ℒtxt\|​NH\.\\displaystyle=\\frac\{\\displaystyle\\sum\_\{\\ell\\in\\mathcal\{L\}\_\{\\rm txt\}\}\\sum\_\{h=1\}^\{N\_\{H\}\}\\mathbf\{A\}\_\{p\}^\{\\ell,h\}\[i\_\{\\rm p\},\\chi\_\{\\rm ctx\}\(j\)\]\}\{\|\\mathcal\{L\}\_\{\\rm txt\}\|N\_\{H\}\}\.Collecting these scores yields the raw visual relevance vector𝐚visraw∈ℝ≥0NV\\mathbf\{a\}\_\{\\rm vis\}^\{\\rm raw\}\\in\\mathbb\{R\}\_\{\\geq 0\}^\{N\_\{V\}\}and textual relevance vector𝐚txt∈ℝ≥0NC\\mathbf\{a\}\_\{\\rm txt\}\\in\\mathbb\{R\}\_\{\\geq 0\}^\{N\_\{C\}\}\. Thus the visual branch reads object\-to\-patch attention, while the textual branch reads probe\-to\-context attention\. Both directly average the original post\-softmax weights over their fixed layers and all heads\.

#### Visual sink filtering\.

Visual sink tokens concentrate activation in a small set of hidden dimensions and can distort the relevance map used to localize the target object\. We score this concentration by normalizing the strongest activation in𝒟sink\\mathcal\{D\}\_\{\\rm sink\}by the root mean square across allddhidden dimensions:

ssink,v=1\|ℒvis\|​∑ℓ∈ℒvismaxr∈𝒟sink⁡\|𝐇pℓ​\[χvis​\(v\),r\]\|1d​∑q=1d\(𝐇pℓ​\[χvis​\(v\),q\]\)2\.s\_\{\{\\rm sink\},v\}=\\frac\{1\}\{\|\\mathcal\{L\}\_\{\\rm vis\}\|\}\\sum\_\{\\ell\\in\\mathcal\{L\}\_\{\\rm vis\}\}\\frac\{\\displaystyle\\max\_\{r\\in\\mathcal\{D\}\_\{\\rm sink\}\}\|\\mathbf\{H\}\_\{p\}^\{\\ell\}\[\\chi\_\{\\rm vis\}\(v\),r\]\|\}\{\\displaystyle\\sqrt\{\\frac\{1\}\{d\}\\sum\_\{q=1\}^\{d\}\\bigl\(\\mathbf\{H\}\_\{p\}^\{\\ell\}\[\\chi\_\{\\rm vis\}\(v\),q\]\\bigr\)^\{2\}\}\}\.\(2\)For visual sink thresholdτ\\tau, tokens satisfyingssink,v\>τs\_\{\{\\rm sink\},v\}\>\\tauformℐsink\\mathcal\{I\}\_\{\\rm sink\}\. We define𝐚vis=\(avis,1,…,avis,NV\)∈ℝ≥0NV\\mathbf\{a\}\_\{\\rm vis\}=\(a\_\{\{\\rm vis\},1\},\\ldots,a\_\{\{\\rm vis\},N\_\{V\}\}\)\\in\\mathbb\{R\}\_\{\\geq 0\}^\{N\_\{V\}\}by settingavis,v=0a\_\{\{\\rm vis\},v\}=0insideℐsink\\mathcal\{I\}\_\{\\rm sink\}andavis,v=avis,vrawa\_\{\{\\rm vis\},v\}=a\_\{\{\\rm vis\},v\}^\{\\rm raw\}otherwise\. This filtered vector determines crop location and scale, and the corresponding raw sink mass controls the visual intervention gate\.

#### Spatial localization\.

We reshape𝐚vis\\mathbf\{a\}\_\{\\rm vis\}in the visual processor’s patch order and divide each cell by the sum over allG2G^\{2\}cells, obtaining the spatial distributionM~u,w\\widetilde\{M\}\_\{u,w\}foru,w∈\{0,…,G−1\}u,w\\in\\\{0,\\ldots,G\-1\\\}\. Its center and spread are

μx\\displaystyle\\mu\_\{x\}=∑u,wwM~u,w,σx=\(∑u,w\(w−μx\)2​M~u,w\),\\displaystyle\{\\displaystyle=\}\\sum\_\{u,w\}w\\widetilde\{M\}\_\{u,w\},\\quad\\sigma\_\{x\}\{=\}\\sqrt\{\(\\sum\_\{u,w\}\(w\-\\mu\_\{x\}\)^\{2\}\\widetilde\{M\}\_\{u,w\}\)\},\(3\)μy\\displaystyle\\mu\_\{y\}=∑u,wuM~u,w,σy=\(∑u,w\(u−μy\)2​M~u,w\)\.\\displaystyle\{\\displaystyle=\}\\sum\_\{u,w\}u\\widetilde\{M\}\_\{u,w\},\\quad\\sigma\_\{y\}\{=\}\\sqrt\{\(\\sum\_\{u,w\}\(u\-\\mu\_\{y\}\)^\{2\}\\widetilde\{M\}\_\{u,w\}\)\}\.These moments locate the crop and define its horizontal and vertical extent before adaptive scaling\.

### 3\.3Entropy\-Calibrated Evidence Scaling

#### Text scaling\.

ForNC\>0N\_\{C\}\>0, a sentence segmenter maps the original context to token sets\{𝒮m\}m=1NS\\\{\\mathcal\{S\}\_\{m\}\\\}\_\{m=1\}^\{N\_\{S\}\}\. We form the sentence distribution𝐩=\(p1,…,pNS\)\\mathbf\{p\}=\(p\_\{1\},\\ldots,p\_\{N\_\{S\}\}\)by settingpm∝\|𝒮m\|−1​∑j∈𝒮matxt,jp\_\{m\}\\propto\|\\mathcal\{S\}\_\{m\}\|^\{\-1\}\\sum\_\{j\\in\\mathcal\{S\}\_\{m\}\}a\_\{\{\\rm txt\},j\}and normalizing it so that∑m=1NSpm=1\\sum\_\{m=1\}^\{N\_\{S\}\}p\_\{m\}=1\. LetH\(𝐩\)=−∑m=1NSpmlog2pmH\(\\mathbf\{p\}\)=\-\\sum\_\{m=1\}^\{N\_\{S\}\}p\_\{m\}\\log\_\{2\}p\_\{m\}\. We select the sentence indices

𝒥txt=TopK⁡\(𝐩,min⁡\(⌈2H⁡\(𝐩\)⌉,kmax\)\)\.\\mathcal\{J\}\_\{\\rm txt\}=\\operatorname\{TopK\}\\\!\\left\(\\mathbf\{p\},\\min\\\!\\left\(\\left\\lceil 2^\{H\(\\mathbf\{p\}\)\}\\right\\rceil,k\_\{\\max\}\\right\)\\right\)\.\(4\)Eq\. \([4](https://arxiv.org/html/2609.16795#S3.E4)\) converts relevance dispersion into an evidence budget: low entropy indicates concentrated evidence and selects fewer sentences, whereas high entropy indicates distributed evidence and selects more\. The effective count2H⁡\(𝐩\)2^\{H\(\\mathbf\{p\}\)\}, capped atkmaxk\_\{\\max\}, sets this budget beforeTopK\\operatorname\{TopK\}retains the most relevant sentences\.

#### Visual scaling\.

The spatial entropy controls crop extent:

ℋvis=−∑u,wM~u,wlog2M~u,w,\\displaystyle\\mathcal\{H\}\_\{\\rm vis\}=\-\\sum\_\{u,w\}\\widetilde\{M\}\_\{u,w\}\\log\_\{2\}\\widetilde\{M\}\_\{u,w\},\(5\)β=clip⁡\(12​2ℋvisσx​σy,βmin,βmax\),\\displaystyle\\beta=\\operatorname\{clip\}\\\!\\left\(\\frac\{1\}\{2\}\\sqrt\{\\frac\{2^\{\\mathcal\{H\}\_\{\\rm vis\}\}\}\{\\sigma\_\{x\}\\sigma\_\{y\}\}\},\\beta\_\{\\min\},\\beta\_\{\\max\}\\right\),𝐛pix=Πpix​\(μx−β​σx,μy−β​σy,μx\+β​σx,μy\+β​σy\)\.\\displaystyle\\mathbf\{b\}\_\{\\rm pix\}=\\Pi\_\{\\rm pix\}\\\!\\left\(\\mu\_\{x\}\{\-\}\\beta\\sigma\_\{x\},\\mu\_\{y\}\{\-\}\\beta\\sigma\_\{y\},\\mu\_\{x\}\{\+\}\\beta\\sigma\_\{x\},\\mu\_\{y\}\{\+\}\\beta\\sigma\_\{y\}\\right\)\.The fixed mapΠpix\\Pi\_\{\\rm pix\}converts grid coordinates to pixels and clips the box to the image boundary\. Because2ℋvis2^\{\\mathcal\{H\}\_\{\\rm vis\}\}is the effective number of grid cells carrying relevance, the unconstrained scale satisfies\(2​β​σx\)​\(2​β​σy\)=2ℋvis\(2\\beta\\sigma\_\{x\}\)\(2\\beta\\sigma\_\{y\}\)=2^\{\\mathcal\{H\}\_\{\\rm vis\}\}, tying crop area to spatial relevance dispersion\. The centroid and axis\-wise spreads retain the map’s location and shape, so concentrated maps yield tight crops while diffuse maps yield broader ones\.

### 3\.4Modality\-Specific Intervention Gating

The preceding scaling stage determines the candidate text and crop, but presenting them unconditionally would ignore whether intervention is needed\. Because the model may already use the relevant evidence and intervention demand can differ by modality, AREA gates text and vision independently\. The text score measures how little natural attention reaches the selected sentences, while the visual score is the fraction of raw relevance assigned to sink\-flagged patches:

utxt\\displaystyle u\_\{\\rm txt\}=−log2⁡\(∑m∈𝒥txt∑j∈𝒮matxt,j∑j=1NCatxt,j\+ε\),\\displaystyle=\-\\log\_\{2\}\\\!\\left\(\\frac\{\\sum\_\{m\\in\\mathcal\{J\}\_\{\\rm txt\}\}\\sum\_\{j\\in\\mathcal\{S\}\_\{m\}\}a\_\{\{\\rm txt\},j\}\}\{\\sum\_\{j=1\}^\{N\_\{C\}\}a\_\{\{\\rm txt\},j\}\}\+\\varepsilon\\right\),\(6\)uvis\\displaystyle u\_\{\\rm vis\}=∑v∈ℐsinkavis,vraw∑v=1NVavis,vraw\.\\displaystyle=\\frac\{\\sum\_\{v\\in\\mathcal\{I\}\_\{\\rm sink\}\}a\_\{\{\\rm vis\},v\}^\{\\rm raw\}\}\{\\sum\_\{v=1\}^\{N\_\{V\}\}a\_\{\{\\rm vis\},v\}^\{\\rm raw\}\}\.A highutxtu\_\{\\rm txt\}indicates that the selected sentences are underused, and a highuvisu\_\{\\rm vis\}indicates stronger sink contamination in the raw localization signal\.

For samplennand modalityx∈\{txt,vis\}x\\in\\\{\{\\rm txt\},\{\\rm vis\}\\\}, letmx\(<n\)m\_\{x\}^\{\(<n\)\}be the median of duplicate\-retaining scores from earlier eligible samples in the current dataset–backbone stream\. Vision is always eligible, whereas text is eligible only whenNC\>0N\_\{C\}\>0; thusztxt\(n\)=0z\_\{\\rm txt\}^\{\(n\)\}=0whenNC=0N\_\{C\}=0\. For each eligible modality, we setzx\(n\)=1z\_\{x\}^\{\(n\)\}=1during the firstWwarmW\_\{\\rm warm\}samples andzx\(n\)=𝟏\[ux≥mx\(<n\)\]z\_\{x\}^\{\(n\)\}=\\mathbf\{1\}\[u\_\{x\}\\geq m\_\{x\}^\{\(<n\)\}\]thereafter\. The two decisions are independent and use only preceding samples\. After both decisions, we append the current eligible scores to their histories; histories reset between datasets and backbones\. Whenztxt=1z\_\{\\rm txt\}=1,C⋆C^\{\\star\}isCCwith the sentences in𝒥txt\\mathcal\{J\}\_\{\\rm txt\}wrapped by text\-evidence markers, otherwiseC⋆=CC^\{\\star\}=C\. Whenzvis=1z\_\{\\rm vis\}=1, the image payloadℐ⋆\\mathcal\{I\}^\{\\star\}is the marker\-wrapped crop defined by𝐛pix\\mathbf\{b\}\_\{\\rm pix\}alone, otherwiseℐ⋆\\mathcal\{I\}^\{\\star\}contains the original image\.

### 3\.5Causal Text\-Evidence Refresh

As generation progresses, evidence needs can shift beyond the initial selection\. AREA therefore monitors context\-attention dispersion and reselects and appends textual evidence when the current value exceeds a threshold determined only by earlier decoding steps, subject to a fixed refresh budget\.

The initial generation prompt𝒫1\\mathcal\{P\}\_\{1\}is tokenized over\(ℐ⋆,Q,C⋆\)\(\\mathcal\{I\}^\{\\star\},Q,C^\{\\star\}\)\. WhenNC\>0N\_\{C\}\>0, tokenization supplies the order\-preserving mapj↦ι⁡\(j\)j\\mapsto\\iota\(j\)from each original context token to its prompt position; evidence\-marker tokens are excluded\.RmaxR\_\{\\max\}caps refreshes, andrtr\_\{t\}counts those completed through steptt, withr0=0r\_\{0\}=0\.

Fort≥1t\\geq 1,𝒫t\\mathcal\{P\}\_\{t\}is the cached prefix before decodingyty\_\{t\}\. Once committed and cached,yty\_\{t\}occupiesit=\|𝒫t\|\+1i\_\{t\}=\|\\mathcal\{P\}\_\{t\}\|\+1; let𝐀tℓ,h\[it,:\]∈\[0,1\]it\\mathbf\{A\}\_\{t\}^\{\\ell,h\}\[i\_\{t\},:\]\\in\[0,1\]^\{i\_\{t\}\}denote its post\-softmax causal\-attention row\. Applying the textual readout of Eq\. \([1](https://arxiv.org/html/2609.16795#S3.E1)\) to this row at keysι⁡\(j\)\\iota\(j\)yields𝐚txt\(t\)∈ℝ≥0NC\\mathbf\{a\}\_\{\\rm txt\}^\{\(t\)\}\\in\\mathbb\{R\}\_\{\\geq 0\}^\{N\_\{C\}\}\. We normalize these scores and compute their entropy:

πj\(t\)=atxt,j\(t\)∑q=1NCatxt,q\(t\),et=−∑j=1NCπj\(t\)log2πj\(t\)\.\\pi\_\{j\}^\{\(t\)\}=\\frac\{a\_\{\{\\rm txt\},j\}^\{\(t\)\}\}\{\\sum\_\{q=1\}^\{N\_\{C\}\}a\_\{\{\\rm txt\},q\}^\{\(t\)\}\},\\quad e\_\{t\}=\-\\sum\\nolimits\_\{j=1\}^\{N\_\{C\}\}\\pi\_\{j\}^\{\(t\)\}\\log\_\{2\}\\pi\_\{j\}^\{\(t\)\}\.\(7\)The weightsπj\(t\)\\pi\_\{j\}^\{\(t\)\}cover only original context tokens; evidence markers and appended reminders are excluded\. Their entropyete\_\{t\}is low for concentrated attention and high for dispersed attention, serving as a step\-wise evidence\-demand proxy\.

Fort\>1t\>1,e¯t−1\\bar\{e\}\_\{t\-1\}andσe,t−1\\sigma\_\{e,t\-1\}are the mean and population standard deviation of\(e1,…,et−1\)\(e\_\{1\},\\ldots,e\_\{t\-1\}\);κ≥0\\kappa\\geq 0controls peak sensitivity\. The binary trigger is

γt=\{𝟏\[et\>e¯t−1\+κσe,t−1\],t\>1∧rt−1<Rmax,0,otherwise\.\\gamma\_\{t\}\{=\}\\begin\{cases\}\\mathbf\{1\}\[e\_\{t\}\>\\bar\{e\}\_\{t\{\-\}1\}\{\+\}\\kappa\\sigma\_\{e,t\{\-\}1\}\],&t\{\>\}1\\ \\land\\ r\_\{t\{\-\}1\}\{<\}R\_\{\\max\},\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(8\)The strict test excludesete\_\{t\}from its own threshold\. The first eligible trigger is thereforet=2t=2, where the one\-value population standard deviation is zero\.

Whenγt=1\\gamma\_\{t\}=1, substituting𝐚txt\(t\)\\mathbf\{a\}\_\{\\rm txt\}^\{\(t\)\}for𝐚txt\\mathbf\{a\}\_\{\\rm txt\}in the sentence aggregation and Eq\. \([4](https://arxiv.org/html/2609.16795#S3.E4)\) yields𝒥txt\(t\)\\mathcal\{J\}\_\{\\rm txt\}^\{\(t\)\}\. We serialize these sentences in context order and defineRtR\_\{t\}as the resulting token block, enclosed once by the text\-evidence markers\. With brackets denoting token\-sequence concatenation, the cached prefix and refresh count evolve as

𝒫t\+1\\displaystyle\\mathcal\{P\}\_\{t\+1\}=\{\[𝒫t;yt;Rt\],γt=1,\[𝒫t;yt\],γt=0,\\displaystyle=\\begin\{cases\}\[\\mathcal\{P\}\_\{t\};y\_\{t\};R\_\{t\}\],&\\gamma\_\{t\}=1,\\\\ \[\\mathcal\{P\}\_\{t\};y\_\{t\}\],&\\gamma\_\{t\}=0,\\end\{cases\}rt\\displaystyle r\_\{t\}=rt−1\+γt\.\\displaystyle=r\_\{t\-1\}\+\\gamma\_\{t\}\.\(9\)AppendingRtR\_\{t\}at the decoding frontier preservesι\\iotaand existing key–value cache entries;ete\_\{t\}then enters the history used at stept\+1t\+1\. The mechanism refreshes only textual evidence and remains active whenztxt=0z\_\{\\rm txt\}=0\.

### 3\.6Inference Protocol

Algorithm 1AREA Inference0:

I,Q,CI,Q,C; frozen

ℱθ\\mathcal\{F\}\_\{\\theta\}; state

𝒰txt,𝒰vis,nseen\\mathcal\{U\}\_\{\\rm txt\},\\mathcal\{U\}\_\{\\rm vis\},n\_\{\\rm seen\}
0:Layer\-Resolved Evidence Readout

1:Generate exactly one probe token; expose

𝐀pℓ,h,𝐇pℓ\\mathbf\{A\}\_\{p\}^\{\\ell,h\},\\mathbf\{H\}\_\{p\}^\{\\ell\}
2:Resolve

𝒪,χvis\\mathcal\{O\},\\chi\_\{\\rm vis\}and, when

NC\>0N\_\{C\}\>0,

χctx\\chi\_\{\\rm ctx\}
3:Compute

𝐚visraw\\mathbf\{a\}\_\{\\rm vis\}^\{\\rm raw\}and, when

NC\>0N\_\{C\}\>0,

𝐚txt\\mathbf\{a\}\_\{\\rm txt\}
4:Compute sink scores and

ℐsink=\{v:ssink,v\>τ\}\\mathcal\{I\}\_\{\\rm sink\}=\\\{v:s\_\{\{\\rm sink\},v\}\>\\tau\\\}
5:Form

𝐚vis\\mathbf\{a\}\_\{\\rm vis\},

M~\\widetilde\{M\}, and its spatial moments

5:Entropy\-Calibrated Evidence Scaling

6:if

NC\>0N\_\{C\}\>0then

7:Form

𝐩\\mathbf\{p\}and select

𝒥txt\\mathcal\{J\}\_\{\\rm txt\}by Eq\. \([4](https://arxiv.org/html/2609.16795#S3.E4)\)

8:endif

9:Compute

ℋvis,β,𝐛pix\\mathcal\{H\}\_\{\\rm vis\},\\beta,\\mathbf\{b\}\_\{\\rm pix\}by Eq\. \([5](https://arxiv.org/html/2609.16795#S3.E5)\)

9:Modality\-Specific Intervention Gating

10:Compute

uvisu\_\{\\rm vis\}and, if

NC\>0N\_\{C\}\>0,

utxtu\_\{\\rm txt\}
11:Set eligible

zxz\_\{x\}from warmup or prior medians⊳\\trianglerightprior only

12:Append each eligible

uxu\_\{x\}; increment

nseenn\_\{\\rm seen\}⊳\\trianglerightafter both gates

13:Form the conditional payloads

ℐ⋆\\mathcal\{I\}^\{\\star\}and

C⋆C^\{\\star\}
14:Tokenize

𝒫1\\mathcal\{P\}\_\{1\}and, if

NC\>0N\_\{C\}\>0, obtain

ι\\iota
14:Causal Text\-Evidence Refresh

15:Set

r0←0r\_\{0\}\\leftarrow 0and initialize an empty entropy history

16:for

t=1,2,…t=1,2,\\ldotsuntil generation terminatesdo

17:Decode, commit, and cache

yty\_\{t\}
18:if

NC\>0N\_\{C\}\>0then

19:Read

𝐚txt\(t\)\\mathbf\{a\}\_\{\\rm txt\}^\{\(t\)\}through

ι\\iota; compute

πj\(t\)\\pi\_\{j\}^\{\(t\)\}and

ete\_\{t\}
20:Set

γt\\gamma\_\{t\}by the strict prior\-history test in Eq\. \([8](https://arxiv.org/html/2609.16795#S3.E8)\)

21:if

γt=1\\gamma\_\{t\}=1then

22:Reselect

𝒥txt\(t\)\\mathcal\{J\}\_\{\\rm txt\}^\{\(t\)\}and form

RtR\_\{t\}⊳\\trianglerighttext only

23:endif

24:Update

\(𝒫t\+1,rt\)\(\\mathcal\{P\}\_\{t\+1\},r\_\{t\}\)by Eq\. \([9](https://arxiv.org/html/2609.16795#S3.E9)\)

25:Append

ete\_\{t\}to the entropy history⊳\\trianglerightafter decision

26:endif

27:endfor

28:returngenerated answer tokens

## 4Experiments

### 4\.1Experimental Setup

BackboneMethodE\-VQAInfoSeekOVENViQuAEAvgSHAllU\-QU\-EAllAllAllSmallQwen2\-VL\-2BBase17\.516\.05\.35\.65\.41\.215\.010\.2\+LoT18\.216\.210\.510\.110\.32\.818\.411\.9\(\+1\.7\)\+AREA \(Ours\)18\.716\.711\.410\.010\.93\.419\.212\.6\(\+2\.4\)Qwen2\.5\-VL\-3BBase30\.127\.822\.622\.222\.411\.522\.921\.2\+LoT32\.930\.425\.425\.125\.218\.327\.825\.5\(\+4\.3\)\+AREA \(Ours\)33\.431\.025\.924\.925\.619\.028\.526\.0\(\+4\.8\)Qwen3\-VL\-4BBase35\.032\.827\.928\.628\.325\.134\.530\.2\+LoT37\.334\.830\.830\.230\.524\.536\.331\.5\(\+1\.3\)\+AREA \(Ours\)37\.935\.431\.430\.030\.925\.137\.032\.1\(\+1\.9\)InternVL3\.5\-4BBase29\.126\.228\.829\.028\.910\.836\.425\.6\+LoT31\.728\.733\.333\.133\.211\.545\.629\.8\(\+4\.2\)\+AREA \(Ours\)32\.329\.333\.932\.933\.612\.246\.530\.4\(\+4\.8\)MediumQwen2\-VL\-7BBase25\.622\.924\.224\.724\.411\.133\.022\.9\+LoT28\.725\.629\.830\.129\.916\.640\.828\.2\(\+5\.3\)\+AREA \(Ours\)29\.326\.230\.529\.930\.217\.241\.628\.8\(\+5\.9\)Qwen2\.5\-VL\-7BBase32\.130\.123\.925\.224\.521\.136\.328\.0\+LoT33\.631\.424\.925\.425\.121\.238\.629\.1\(\+1\.1\)\+AREA \(Ours\)34\.031\.825\.525\.225\.421\.839\.429\.6\(\+1\.6\)Qwen3\-VL\-8BBase36\.635\.029\.130\.429\.717\.743\.731\.5\+LoT38\.636\.433\.232\.532\.819\.651\.035\.0\(\+3\.5\)\+AREA \(Ours\)39\.136\.933\.832\.333\.120\.251\.835\.5\(\+4\.0\)InternVL3\.5\-8BBase31\.429\.029\.529\.929\.717\.744\.230\.2\+LoT33\.230\.732\.132\.532\.317\.654\.233\.7\(\+3\.5\)\+AREA \(Ours\)33\.831\.332\.832\.332\.618\.355\.034\.3\(\+4\.1\)LargeQwen2\.5\-VL\-32BBase35\.333\.726\.726\.126\.413\.337\.927\.8\+LoT37\.736\.031\.226\.229\.614\.246\.031\.5\(\+3\.7\)\+AREA \(Ours\)38\.236\.531\.826\.030\.114\.946\.832\.1\(\+4\.3\)Table 1:Performance on KB\-VQA benchmarks\. Base denotes the same checkpoint without evidence highlighting\. SH, U\-Q, and U\-E denote Single\-Hop, Unseen\-Q, and Unseen\-E; Avg is computed over the four All columns\. Colored values in parentheses indicate absolute improvements over Base\.![Refer to caption](https://arxiv.org/html/2609.16795v1/exparea.png)Figure 3:Qualitative examples of AREA on E\-VQA and ViQuAE\. Each row shows the original image with the AREA\-induced visual bounding box and the retrieved textual context with AREA\-selected sentences highlighted\. The examples illustrate how AREA jointly localizes visual evidence and emphasizes relevant textual evidence for knowledge\-intensive visual question answering\. Best viewed when zoomed in\.#### Datasets and metrics\.

We follow the evaluation protocol of LoT\([Morini et al\. 2026](https://arxiv.org/html/2609.16795#bib.bib1)\)\. The knowledge\-based visual question answering \(KB\-VQA\) suite comprises Encyclopedic VQA \(E\-VQA\)\([Mensink et al\. 2023](https://arxiv.org/html/2609.16795#bib.bib14)\), InfoSeek\([Chen et al\. 2023](https://arxiv.org/html/2609.16795#bib.bib12)\), Open\-domain Visual Entity Recognition \(OVEN\)\([Hu et al\. 2023](https://arxiv.org/html/2609.16795#bib.bib21)\), and ViQuAE\([Lerner et al\. 2022](https://arxiv.org/html/2609.16795#bib.bib13)\)\. We report Single\-Hop and All accuracy on the E\-VQA test set; Unseen\-Q, Unseen\-E, and All accuracy on the InfoSeek validation set; validation accuracy on OVEN; and exact match on the ViQuAE test set\. The standard suite contains RealWorldQA\([xAI 2024](https://arxiv.org/html/2609.16795#bib.bib23)\), Vstar\([Wu and Xie 2024](https://arxiv.org/html/2609.16795#bib.bib19)\), TextVQA\([Singh et al\. 2019](https://arxiv.org/html/2609.16795#bib.bib17)\), ChartQA\([Masry et al\. 2022](https://arxiv.org/html/2609.16795#bib.bib16)\), OCRBench\([Liu et al\. 2024](https://arxiv.org/html/2609.16795#bib.bib15)\), POPE\([Li et al\. 2023](https://arxiv.org/html/2609.16795#bib.bib18)\), and AMBER\-D\([Wang et al\. 2023](https://arxiv.org/html/2609.16795#bib.bib22)\)\. We use the Cambrian\-1 evaluation suite\([Tong et al\. 2024](https://arxiv.org/html/2609.16795#bib.bib29)\)except for AMBER\-D, which uses its official discriminative\-accuracy pipeline\.

#### Backbones and protocol\.

We evaluate nine frozen MLLM checkpoints: Qwen2\-VL\-2B and Qwen2\-VL\-7B\([Wang et al\. 2024](https://arxiv.org/html/2609.16795#bib.bib24)\); Qwen2\.5\-VL\-3B, Qwen2\.5\-VL\-7B, and Qwen2\.5\-VL\-32B\([Bai et al\. 2025b](https://arxiv.org/html/2609.16795#bib.bib20)\); Qwen3\-VL\-4B and Qwen3\-VL\-8B\([Bai et al\. 2025a](https://arxiv.org/html/2609.16795#bib.bib25)\); and InternVL3\.5\-4B and InternVL3\.5\-8B\([Wang et al\. 2025](https://arxiv.org/html/2609.16795#bib.bib26)\)\. For KB\-VQA, we retrieve three Wikipedia entities with the same EVA\-CLIP and FAISS pipeline used by LoT\([Sun et al\. 2024](https://arxiv.org/html/2609.16795#bib.bib27);[Johnson et al\. 2019](https://arxiv.org/html/2609.16795#bib.bib28)\)\. Each LoT–AREA pair uses the same checkpoint, retrieved entities, task instruction, decoding configuration, and evaluator\. LoT selects one sentence, fixes the visual scale toβ=2\\beta=2, and always applies its intervention\. AREA useskmax=3k\_\{\\max\}=3,\(βmin,βmax\)=\(1\.5,2\.5\)\(\\beta\_\{\\min\},\\beta\_\{\\max\}\)=\(1\.5,2\.5\),κ=1\\kappa=1,Rmax=4R\_\{\\max\}=4andWwarm=50W\_\{\\rm warm\}=50\.

### 4\.2Results on KB\-VQA

Table 1 reports the results on four KB\-VQA benchmarks\. AREA outperforms LoT for all nine checkpoints, improving the four\-dataset average by approximately 0\.6 points across backbones\. This consistency across model families and scales suggests that the gains arise from better matching the intervention to each example, rather than from a particular architecture\. Unlike LoT, which always exposes a fixed crop and a single sentence, AREA avoids redundant interventions when the relevant evidence is already well covered and expands the evidence scope when relevance is more dispersed\.

The gains are particularly clear on OVEN and ViQuAE, where answering often requires identifying a visual entity and connecting it to a specific retrieved fact\. Adaptive localization can suppress unrelated visual regions, while flexible sentence allocation reduces the risk of omitting supporting information\. On InfoSeek, the improvements are mainly observed on Unseen\-Q and the overall score, whereas Unseen\-E changes only marginally and occasionally decreases\. We conjecture that unseen\-entity questions are more strongly limited by entity recognition and retrieval quality; when the correct entity or passage is absent, improving evidence presentation alone provides limited benefit\.

BackboneMethodVision\-Centric/OCR and ChartHallucinationRealWorldQAV\-StarTextVQAChartQAOCRBenchPOPEAMBER\-DSmallQwen2\-VL\-2BBase54\.846\.672\.073\.474\.688\.442\.1\+LoT56\.7\(\+1\.9\)53\.9\(\+7\.3\)74\.9\(\+2\.9\)73\.2\(\-0\.2\)74\.0\(\-0\.6\)88\.6\(\+0\.2\)75\.0\(\+32\.9\)\+AREA \(Ours\)57\.4\(\+2\.6\)54\.8\(\+8\.2\)75\.4\(\+3\.4\)73\.0\(\-0\.4\)74\.7\(\+0\.1\)89\.0\(\+0\.6\)75\.8\(\+33\.7\)Qwen2\.5\-VL\-3BBase59\.159\.762\.579\.176\.188\.217\.2\+LoT61\.7\(\+2\.6\)61\.8\(\+2\.1\)66\.4\(\+3\.9\)79\.5\(\+0\.4\)79\.7\(\+3\.6\)89\.0\(\+0\.8\)40\.5\(\+23\.3\)\+AREA \(Ours\)62\.4\(\+3\.3\)62\.6\(\+2\.9\)67\.0\(\+4\.5\)79\.3\(\+0\.2\)80\.2\(\+4\.1\)89\.4\(\+1\.2\)41\.7\(\+24\.5\)Qwen3\-VL\-4BBase66\.756\.074\.780\.776\.090\.181\.9\+LoT71\.6\(\+4\.9\)67\.0\(\+11\.0\)76\.5\(\+1\.8\)82\.3\(\+1\.6\)77\.4\(\+1\.4\)90\.2\(\+0\.1\)81\.2\(\-0\.7\)\+AREA \(Ours\)72\.2\(\+5\.5\)67\.8\(\+11\.8\)77\.1\(\+2\.4\)82\.0\(\+1\.3\)78\.1\(\+2\.1\)90\.5\(\+0\.4\)82\.0\(\+0\.1\)InternVL3\.5\-4BBase66\.773\.875\.585\.982\.989\.080\.1\+LoT67\.3\(\+0\.6\)76\.4\(\+2\.6\)76\.0\(\+0\.5\)85\.4\(\-0\.5\)82\.5\(\-0\.4\)86\.4\(\-2\.6\)73\.0\(\-7\.1\)\+AREA \(Ours\)68\.0\(\+1\.3\)77\.1\(\+3\.3\)76\.6\(\+1\.1\)85\.2\(\-0\.7\)83\.1\(\+0\.2\)87\.0\(\-2\.0\)74\.2\(\-5\.9\)MediumQwen2\-VL\-7BBase62\.752\.478\.081\.680\.070\.333\.7\+LoT64\.8\(\+2\.1\)56\.0\(\+3\.6\)79\.3\(\+1\.3\)82\.0\(\+0\.4\)81\.3\(\+1\.3\)89\.1\(\+18\.8\)47\.3\(\+13\.6\)\+AREA \(Ours\)65\.5\(\+2\.8\)56\.8\(\+4\.4\)79\.8\(\+1\.8\)81\.8\(\+0\.2\)81\.9\(\+1\.9\)89\.4\(\+19\.1\)48\.6\(\+14\.9\)Qwen2\.5\-VL\-7BBase65\.057\.175\.777\.084\.987\.462\.0\+LoT67\.5\(\+2\.5\)61\.3\(\+4\.2\)77\.9\(\+2\.2\)79\.4\(\+2\.4\)84\.3\(\-0\.6\)87\.4\(\+0\.0\)62\.6\(\+0\.6\)\+AREA \(Ours\)68\.1\(\+3\.1\)62\.0\(\+4\.9\)78\.4\(\+2\.7\)79\.1\(\+2\.1\)85\.0\(\+0\.1\)87\.8\(\+0\.4\)63\.4\(\+1\.4\)Qwen3\-VL\-8BBase66\.860\.776\.982\.079\.589\.275\.9\+LoT67\.5\(\+0\.7\)61\.3\(\+0\.6\)77\.9\(\+1\.0\)79\.4\(\-2\.6\)84\.3\(\+4\.8\)89\.2\(\+0\.0\)62\.6\(\-13\.3\)\+AREA \(Ours\)68\.2\(\+1\.4\)62\.0\(\+1\.3\)78\.5\(\+1\.6\)79\.1\(\-2\.9\)84\.9\(\+5\.4\)89\.5\(\+0\.3\)63\.8\(\-12\.1\)InternVL3\.5\-8BBase64\.773\.377\.284\.384\.386\.380\.4\+LoT65\.9\(\+1\.2\)69\.6\(\-3\.7\)78\.0\(\+0\.8\)79\.7\(\-4\.6\)83\.0\(\-1\.3\)87\.3\(\+1\.0\)80\.8\(\+0\.4\)\+AREA \(Ours\)66\.5\(\+1\.8\)70\.4\(\-2\.9\)78\.6\(\+1\.4\)79\.5\(\-4\.8\)83\.6\(\-0\.7\)87\.7\(\+1\.4\)81\.4\(\+1\.0\)LargeQwen2\.5\-VL\-32BBase65\.655\.572\.435\.778\.788\.889\.4\+LoT67\.1\(\+1\.5\)65\.4\(\+9\.9\)74\.0\(\+1\.6\)48\.2\(\+12\.5\)79\.0\(\+0\.3\)89\.0\(\+0\.2\)89\.0\(\-0\.4\)\+AREA \(Ours\)67\.8\(\+2\.2\)66\.1\(\+10\.6\)74\.6\(\+2\.2\)50\.0\(\+14\.3\)78\.8\(\+0\.1\)89\.3\(\+0\.5\)89\.6\(\+0\.2\)Table 2:Performance on standard MLLM benchmarks using visual evidence only\. Base denotes the same checkpoint without evidence highlighting\. Colored values in parentheses indicate absolute changes relative to Base\.Figure 3 further illustrates how AREA allocates evidence on representative KB\-VQA examples\. Across questions involving biological, historical, and entity\-specific knowledge, AREA localizes the query\-relevant visual content while highlighting the retrieved sentences that directly support the answer\. The selected evidence is compact and varies across examples, rather than following a uniform sentence budget or crop scale\. This behavior is important for KB\-VQA, where retrieved passages commonly contain facts that are topically related to the question but insufficient for deriving the answer\.

The examples also show that visual and textual evidence need not contribute equally to every prediction\. In some cases, the image primarily identifies the queried entity, while the retrieved context supplies the answer\-specific fact; in others, accurate visual localization is necessary to disambiguate among several plausible entities before consulting the context\. By allocating the two modalities independently, AREA can emphasize the modality that is currently underused without unnecessarily modifying the other\. These qualitative results complement the quantitative improvements in Table 1 and suggest that AREA benefits KB\-VQA by making short cross\-modal evidence chains easier for the frozen model to follow\.

### 4\.3Generalization to Standard Multimodal Benchmarks

Table 2 evaluates AREA on standard multimodal benchmarks without retrieved textual context\. In this setting, text scaling, text gating, and causal evidence refresh are disabled, while sink filtering, entropy\-calibrated crop scaling, and visual intervention gating remain active\. Across the 63 checkpoint–benchmark pairs, AREA improves over LoT by approximately 0\.55 points on average\. The improvement is observed across different backbone families and parameter scales, indicating that adaptive visual allocation is not limited to retrieval\-augmented settings or a particular model architecture\.

The most consistent gains appear on V\-Star and RealWorldQA\. V\-Star frequently requires recognizing small objects, subtle attributes, or fine\-grained spatial cues, for which a question\-conditioned crop can increase the relative prominence of the answer\-bearing region\. RealWorldQA contains more diverse scenes, but its questions often still identify a particular object or local relationship\. In such cases, the visual gate is useful because it can retain the original image when broader context is already necessary, rather than applying a compact crop unconditionally\. This may explain why AREA improves consistently over LoT on both locally focused and more heterogeneous real\-world questions\.

AREA also produces stable, although more moderate, gains on TextVQA and OCRBench\. Visual text may occupy only a small part of an image, making localization beneficial, but the required words can also be distributed across multiple signs, lines, or document regions\. Entropy\-calibrated scaling provides a compromise: concentrated relevance leads to a tighter crop, whereas dispersed relevance preserves a larger visual extent\. Nevertheless, evidence allocation alone cannot resolve recognition errors caused by small fonts, low resolution, or difficult layouts, which likely limits the magnitude of the improvement on these benchmarks\.

The clearest exception is ChartQA, where AREA remains approximately comparable to LoT and occasionally performs slightly worse\. Chart questions often require jointly interpreting axes, legends, categories, and spatially separated numerical values\. Cropping around the most salient region may therefore remove structural information needed to relate these components, even when the localized content itself is relevant\. The improvement obtained with Qwen2\.5\-VL\-32B suggests that stronger models may recover missing global relationships more effectively, but the overall pattern indicates that localized evidence highlighting is less suitable for tasks whose reasoning depends on the complete visual layout\.

On the hallucination\-oriented benchmarks, AREA consistently improves over LoT, with the largest additional gains observed on AMBER\-D\. Sink filtering can prevent highly activated but semantically uninformative visual tokens from dominating localization, while intervention gating avoids altering inputs for which the original visual evidence is already sufficiently represented\. However, highlighting still degrades some strong backbones relative to their Base performance on AMBER\-D, suggesting that adaptive allocation mitigates rather than fully eliminates the risks of visual intervention\. POPE shows smaller gains, likely because several checkpoints already achieve high accuracy and leave limited room for improvement\. Overall, these results support the central premise of AREA: visual highlighting is most effective when its necessity and spatial extent are determined by the evidence distribution of each input, rather than fixed in advance\.

## 5Conclusion

We investigated evidence utilization in frozen multimodal large language models and showed that fixed highlighting policies cannot adapt to diverse evidence demands\. We proposed Adaptive Relevance\-guided Evidence Allocation \(AREA\), a training\-free framework that dynamically determines whether to intervene, how much evidence to expose, and when to refresh textual evidence during generation\. By leveraging relevance signals from a single probe pass, AREA performs adaptive evidence scaling and modality\-specific intervention without modifying model parameters\. Experiments across KB\-VQA and general multimodal benchmarks demonstrate that AREA consistently improves over fixed highlighting strategies, especially when answers rely on compact cross\-modal evidence\. These results highlight the importance of adaptive evidence presentation for improving MLLM inference\.

## References

- Baiet al\.\(2025a\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao,et al\.Qwen3\-VL technical report\.External Links:2511\.21631Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px2.p1.1)\.
- Baiet al\.\(2025b\)S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong,et al\.Qwen2\.5\-VL technical report\.External Links:2502\.13923Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px2.p1.1)\.
- Caffagniet al\.\(2024\)D\. Caffagni, F\. Cocchi, N\. Moratelli, S\. Sarto, M\. Cornia, L\. Baraldi, and R\. CucchiaraWiki\-LLaVA: hierarchical retrieval\-augmented generation for multimodal LLMs\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops,pp\. 1818–1826\.External Links:[Document](https://dx.doi.org/10.1109/CVPRW63382.2024.00188),2404\.15406Cited by:[§1](https://arxiv.org/html/2609.16795#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.16795#S2.SS1.p1.1)\.
- Chenet al\.\(2023\)Y\. Chen, H\. Hu, Y\. Luan, H\. Sun, S\. Changpinyo, A\. Ritter, and M\. ChangCan pre\-trained vision and language models answer visual information\-seeking questions?\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Cited by:[§1](https://arxiv.org/html/2609.16795#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.16795#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- Cocchiet al\.\(2025\)F\. Cocchi, N\. Moratelli, M\. Cornia, L\. Baraldi, and R\. CucchiaraAugmenting multimodal LLMs with self\-reflective tokens for knowledge\-based visual question answering\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 9199–9209\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52734.2025.00859),2411\.16863Cited by:[§2\.1](https://arxiv.org/html/2609.16795#S2.SS1.p1.1)\.
- Compagnoniet al\.\(2026\)A\. Compagnoni, M\. Morini, S\. Sarto, F\. Cocchi, D\. Caffagni, M\. Cornia, L\. Baraldi, and R\. CucchiaraReAG: reasoning\-augmented generation for knowledge\-based visual question answering\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 11901–11911\.External Links:2511\.22715Cited by:[§2\.1](https://arxiv.org/html/2609.16795#S2.SS1.p1.1)\.
- Honget al\.\(2025\)Y\. Hong, J\. Gu, Q\. Yang, L\. Fan, Y\. Wu, Y\. Wang, K\. Ding, S\. Xiang, and J\. YeKnowledge\-based visual question answer with multimodal processing, retrieval and filtering\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38,pp\. 119251–119282\.Cited by:[§1](https://arxiv.org/html/2609.16795#S1.p2.1)\.
- Huet al\.\(2023\)H\. Hu, Y\. Luan, Y\. Chen, U\. Khandelwal, M\. Joshi, K\. Lee, K\. Toutanova, and M\. W\. ChangOpen\-domain visual entity recognition: towards recognizing millions of Wikipedia entities\.InCVPR,Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- Johnsonet al\.\(2019\)J\. Johnson, M\. Douze, and H\. JégouBillion\-scale similarity search with GPUs\.IEEE Trans\. on Big Data7\(3\),pp\. 535–547\.Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px2.p1.1)\.
- Kanget al\.\(2025\)S\. Kang, J\. Kim, J\. Kim, and S\. J\. HwangSee what you are told: visual attention sink in large multimodal models\.InInternational Conference on Learning Representations,External Links:2503\.03321Cited by:[§1](https://arxiv.org/html/2609.16795#S1.p1.1),[§3\.2](https://arxiv.org/html/2609.16795#S3.SS2.p1.2)\.
- Lerneret al\.\(2022\)P\. Lerner, O\. Ferret, C\. Guinaudeau, H\. Le Borgne, R\. Besancon, J\. G\. Moreno, and J\. Lovon MelgarejoViQuAE, a dataset for knowledge\-based visual question answering about named entities\.InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval,Cited by:[§2\.1](https://arxiv.org/html/2609.16795#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- Liet al\.\(2023\)Y\. Li, Y\. Du, K\. Zhou, J\. Wang, W\. X\. Zhao, and J\. WenEvaluating object hallucination in large vision\-language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- Liuet al\.\(2024\)Y\. Liu, Z\. Li, M\. Huang, B\. Yang, W\. Yu, C\. Li, X\. Yin, C\. Liu, L\. Jin, and X\. BaiOCRBench: on the hidden mystery of OCR in large multimodal models\.Science China Information Sciences67\(12\),pp\. 220102\.Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, R\. A\. Amjad, R\. Adkathimar, T\. Wei, and H\. TongSelfElicit: your language model secretly knows where is the relevant evidence\.External Links:2502\.08767Cited by:[§1](https://arxiv.org/html/2609.16795#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.16795#S2.SS2.p1.1)\.
- Masryet al\.\(2022\)A\. Masry, X\. L\. Do, J\. Q\. Tan, S\. Joty, and E\. HoqueChartQA: a benchmark for question answering about charts with visual and logical reasoning\.InFindings of the Association for Computational Linguistics: ACL 2022,Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- Mensinket al\.\(2023\)T\. Mensink, J\. Uijlings, L\. Castrejon, A\. Goel, F\. Cadar, H\. Zhou, F\. Sha, A\. Araujo, and V\. FerrariEncyclopedic VQA: visual questions about detailed properties of fine\-grained categories\.InProceedings of the IEEE/CVF International Conference on Computer Vision,Cited by:[§1](https://arxiv.org/html/2609.16795#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.16795#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- Moriniet al\.\(2026\)M\. Morini, S\. Sarto, M\. Cornia, and L\. BaraldiLook twice: training\-free evidence highlighting in multimodal large language models\.External Links:2604\.01280,[Link](https://arxiv.org/abs/2604.01280)Cited by:[§1](https://arxiv.org/html/2609.16795#S1.p2.1),[§2\.2](https://arxiv.org/html/2609.16795#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- Singhet al\.\(2019\)A\. Singh, V\. Natarajan, M\. Shah, Y\. Jiang, X\. Chen, D\. Batra, D\. Parikh, and M\. RohrbachTowards VQA models that can read\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- Sunet al\.\(2024\)Q\. Sun, J\. Wang, Q\. Yu, Y\. Cui, F\. Zhang, X\. Zhang, and X\. WangEVA\-CLIP\-18B: scaling CLIP to 18 billion parameters\.External Links:2402\.04252Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px2.p1.1)\.
- Tonget al\.\(2024\)P\. Tong, E\. Brown, P\. Wu, S\. Woo, A\. J\. V\. IYER, S\. C\. Akula, S\. Yang, J\. Yang, M\. Middepogu, Z\. Wang,et al\.Cambrian\-1: a fully open, vision\-centric exploration of multimodal LLMs\.InNeurIPS,Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- Wanget al\.\(2023\)J\. Wang, Y\. Wang, G\. Xu, J\. Zhang, Y\. Gu, H\. Jia, J\. Wang, H\. Xu, M\. Yan, J\. Zhang,et al\.AMBER: an LLM\-free multi\-dimensional benchmark for MLLMs hallucination evaluation\.External Links:2311\.07397Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge,et al\.Qwen2\-VL: enhancing vision\-language model’s perception of the world at any resolution\.External Links:2409\.12191Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px2.p1.1)\.
- Wanget al\.\(2025\)W\. Wang, Z\. Gao, L\. Gu, H\. Pu, L\. Cui, X\. Wei, Z\. Liu, L\. Jing, S\. Ye, J\. Shao,et al\.InternVL3\.5: advancing open\-source multimodal models in versatility, reasoning, and efficiency\.External Links:2508\.18265Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px2.p1.1)\.
- Wu and Xie \(2024\)P\. Wu and S\. XieV\*: guided visual search as a core mechanism in multimodal LLMs\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- xAI \(2024\)xAIGrok\.External Links:[Link](https://x.ai/blog/grok-1.5v)Cited by:[§4\.1](https://arxiv.org/html/2609.16795#S4.SS1.SSS0.Px1.p1.1)\.
- Yan and Xie \(2024\)Y\. Yan and W\. XieEchoSight: advancing visual\-language models with Wiki knowledge\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 1538–1551\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.83)Cited by:[§1](https://arxiv.org/html/2609.16795#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.16795#S2.SS1.p1.1)\.
- Yanget al\.\(2025\)W\. Yang, J\. Fu, R\. Wang, J\. Wang, L\. Song, and J\. BianOMGM: orchestrate multiple granularities and modalities for efficient multimodal retrieval\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 24545–24563\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1198)Cited by:[§1](https://arxiv.org/html/2609.16795#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.16795#S2.SS1.p1.1)\.
- Yeet al\.\(2026\)W\. Ye, Y\. Su, Y\. Chen, L\. Gao, J\. Li, R\. Li, and R\. ZhangQKVQA: question\-focused filtering for knowledge\-based VQA\.InProceedings of the ACM International Conference on Multimedia,Rio de Janeiro, Brazil\.External Links:2601\.13856Cited by:[§1](https://arxiv.org/html/2609.16795#S1.p2.1),[§2\.1](https://arxiv.org/html/2609.16795#S2.SS1.p1.1)\.
- Yuanet al\.\(2026\)X\. Yuan, L\. Ning, Q\. Ye, W\. Fan, and Q\. LimKG\-RAG: leveraging multimodal knowledge graphs in retrieval\-augmented generation for knowledge\-intensive VQA\.InProceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval,New York, NY, USA,pp\. 2274–2285\.External Links:[Document](https://dx.doi.org/10.1145/3805712.3809680),2508\.05318Cited by:[§2\.1](https://arxiv.org/html/2609.16795#S2.SS1.p1.1)\.

This standalone appendix accompaniesLayers, Sinks, and Scaling: Adaptive Evidence Selection for Multimodal Large Language Models\. It contains three complementary parts\. Part I reports the runtime comparison between LoT and AREA under a common model and GPU configuration\. Part II isolates the contributions of AREA’s evidence scaling, intervention gating, and textual refresh components\. Part III presents qualitative comparisons across ten benchmarks, exposing the predictions and the visual and textual evidence selected by each method\. Together, these results supplement the main paper with efficiency details, component\-level analysis, and instance\-level evidence\.

## Part I: Runtime Efficiency

The table below reports average end\-to\-end latency per sample\. Per\-sample time is used so that the comparison is not confounded by the different sharding layouts of the recorded full\-dataset runs\.

Table 1:Per\-sample inference runtime of LoT and AREA using Qwen2\.5\-VL\-3B\-Instruct in BF16 with batch size 1 on an NVIDIA A30 24 GB GPU per process\. Each time is the recorded sum of per\-sample elapsed times divided by the number of evaluated samples\. The timed region includes retrieval, evidence processing, and answer generation, but excludes model/dataset initialization and metric evaluation\. Runtime reduction is\(tLoT−tAREA\)/tLoT×100%\(t\_\{\\mathrm\{LoT\}\}\-t\_\{\\mathrm\{AREA\}\}\)/t\_\{\\mathrm\{LoT\}\}\\times 100\\%\. Positive values mean AREA is faster\.
## Part II: Algorithm Ablation

AREA makes three complementary evidence\-allocation decisions: how much evidence to expose through Entropy\-Calibrated Evidence Scaling, whether to intervene through Modality\-Specific Intervention Gating, and when to revisit textual evidence through Causal Text\-Evidence Refresh\. Table[2](https://arxiv.org/html/2609.16795#Sx2.T2)isolates these decisions using one\-factor\-at\-a\-time ablations\. All variants use the same Qwen2\.5\-VL\-3B\-Instruct checkpoint, retrieved entities \(when applicable\), dataset splits, prompts, decoding settings, and evaluators; only the component named by each column is removed\. Causal Text\-Evidence Refresh is not applicable to vision\-only benchmarks because they provide no retrieved textual context\.

Table 2:Component ablation of AREA\. The full AREA column reproduces the Qwen2\.5\-VL\-3B\-Instruct results reported in the main tables\.w/o Scalingfixes the textual evidence budget to one sentence and the visual crop scale toβ=2\\beta=2;w/o Gatingapplies the candidate intervention to every eligible modality; andw/o Refreshdisables generation\-time textual evidence refresh\. All other settings are held fixed\. Higher is better for every metric, and the best value in each row is bold\.The full AREA column serves as the matched reference\. Differences from it quantify the contribution of each decision without changing the underlying model or retrieval inputs\. For vision\-only tasks, the scaling and gating ablations operate on visual evidence alone, while the refresh entry is marked as not applicable\.

Part III: Qualitative Visualizations

The following pages present three instance\-level comparisons for each of ten benchmarks\. Each panel reports the question, reference answer, both predictions, localized visual evidence, and the surrounding retrieved context with selected sentences highlighted\. Vision\-only benchmarks explicitly state that no external retrieval is used\.

E\-VQA: AREA vs\. LoT

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/evqa_example_01.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/evqa_example_02.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/evqa_example_03.png)

InfoSeek: AREA vs\. LoT

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/infoseek_example_01.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/infoseek_example_02.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/infoseek_example_03.png)

ViQuAE: AREA vs\. LoT

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/viquae_example_01.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/viquae_example_02.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/viquae_example_03.png)

RealWorldQA: AREA vs\. LoT

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/real_world_qa_example_01.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/real_world_qa_example_02.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/real_world_qa_example_03.png)

VstarBench: AREA vs\. LoT

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/vstar_example_01.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/vstar_example_02.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/vstar_example_03.png)

TextVQA: AREA vs\. LoT

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/textvqa_example_01.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/textvqa_example_02.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/textvqa_example_03.png)

ChartQA: AREA vs\. LoT

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/chartqa_example_01.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/chartqa_example_02.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/chartqa_example_03.png)

OCRBench: AREA vs\. LoT

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/ocrbench_example_01.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/ocrbench_example_02.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/ocrbench_example_03.png)

POPE: AREA vs\. LoT

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/pope_example_01.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/pope_example_02.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/pope_example_03.png)

AMBER\-D: AREA vs\. LoT

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/amber_example_01.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/amber_example_02.png)

![[Uncaptioned image]](https://arxiv.org/html/2609.16795v1/Figures/appendix/amber_example_03.png)