DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents

# DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents
Source: [https://arxiv.org/html/2609.02059](https://arxiv.org/html/2609.02059)
Le Thien Phuc NguyenAffiliation:University of Wisconsin\-MadisonJaden ParkAffiliation:University of Wisconsin\-MadisonXinyi GuAffiliation:Massachusetts Institute of TechnologyZexue HeAffiliation:Stanford UniversitySoochahn LeeAffiliation:Kookmin UniversityRogerio FerisAffiliation:MIT\-IBM Watson AI Lab, IBM ResearchYong Jae LeeAffiliation:University of Wisconsin\-Madison

###### Abstract

Multimodal Large Language Models \(MLLMs\) have achieved strong performance on structured visual understanding tasks such as chart and document question answering\. However, existing benchmarks typically evaluate these domains in isolation, leaving underexplored a key capability: whether models can use textual context to determine how chart evidence should be selected, interpreted, and aggregated\. We introduceDocHop, a benchmark for integrated chart–context reasoning in document\-style images\. InDocHop, the document narrative specifies multi\-step compositional constraints, while charts provide the corresponding data values\. Questions are grounded on a semantic reference label defined in the narrative, requiring models to resolve target entities from context before aggregating evidence across multiple charts\. To enable systematic evaluation, we constructDocHopvia a stochastic logic\-first generation pipeline with controllable reasoning depth and visual density, covering 2,074 examples across six task categories\. Experiments on a wide range of proprietary and open\-source MLLMs show a substantial gap to human performance: annotators achieve over 90% accuracy, while the best model reaches only 62\.83%\. Reasoning\-enhanced models consistently show improved results, but performance degrades as reasoning complexity increases\. Overall,DocHopprovides a controlled testbed for challenging multi\-hop document reasoning\.

###### Keywords:

Machine Learning, ICML

## 1Introduction

Recent Multimodal Large Language Models \(MLLMs\)\([OpenAI, 2025b](https://arxiv.org/html/2609.02059#bib.bib2);[Liu et al\., 2024b](https://arxiv.org/html/2609.02059#bib.bib17);[Bai et al\., 2025a](https://arxiv.org/html/2609.02059#bib.bib21);[Bai et al\., 2025b](https://arxiv.org/html/2609.02059#bib.bib20);[Comanici et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib3);[Liu et al\., 2023](https://arxiv.org/html/2609.02059#bib.bib26);[Dai et al\., 2023](https://arxiv.org/html/2609.02059#bib.bib27)\)have shown promising results in structured image understanding, spanning domains such as charts\([Masry et al\., 2022](https://arxiv.org/html/2609.02059#bib.bib9);[Masry et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib16);[Xu et al\., 2024](https://arxiv.org/html/2609.02059#bib.bib10);[Wang et al\., 2024](https://arxiv.org/html/2609.02059#bib.bib13);[Xia et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib12)\), documents\([Mathew et al\., 2021](https://arxiv.org/html/2609.02059#bib.bib15)\), and webpage\-based interfaces\([Liu et al\., 2024d](https://arxiv.org/html/2609.02059#bib.bib28);[Liu et al\., 2024c](https://arxiv.org/html/2609.02059#bib.bib29)\)\. However, existing benchmarks predominantly evaluate these domains in isolation\. For example, chart\-centric and document\-centric datasets are typically constructed and tested separately, each focusing on a particular form of visual information\. This separation overlooks a key aspect of real\-world presentation: the interplay between visual data and textual context\. In many practical scenarios, charts and text function together, where the surrounding narrative provides definitions, contextual grounding, or reasoning instructions that are essential for correctly interpreting the accompanying visual evidence\. These scenarios highlight an important but underexplored capability: models must bind textual constraints to visual evidence and then use the grounded evidence for downstream reasoning\.

In this paper, we introduceDocHop111[https://www\.zhuoranyu\.com/dochop\-page/](https://www.zhuoranyu.com/dochop-page/), a benchmark designed to evaluate MLLMs on integrated document understanding that combines chart\-based evidence with surrounding narrative context\. The core intuition is to separate reasoning logic from numerical evidence: the document context specifies the multi\-step constraints, while the charts provide the corresponding data values\. Models must jointly interpret the document narrative and the chart values, using the described reasoning trace to locate and aggregate visual evidence across multiple charts\. This design enables a rigorous evaluation of visual\-intensive multi\-hop reasoning, rather than simple single\-hop lookup\. In the following paragraphs, we highlight the key properties ofDocHop\.

![Refer to caption](https://arxiv.org/html/2609.02059v1/dochop_teaser.png)Figure 1:Illustration of an example in DocHop\. Each document instance corresponds to a stochastically\-sampled multi\-hop reasoning trace, which is transformed into a narrative context interleaved with multiple charts\. The narrative specifies the compositional constraints of the trace and concludes by assigning a semantic reference label to the entities implied by its execution\. Questions are then grounded on this label rather than explicit entity names, requiring models to resolve targets from the document context and aggregate numerical evidence from the charts\. The colored arrows indicate where the constraints are presented in the document; the black arrow indicates the specific chart the model needs to look at to check if the input entities satisfy the constraint\.Integrated Chart\-Context Reasoning\.DocHopconsists of document\-style images in which multiple charts are interleaved with explanatory narrative context\. The narrative does not merely provide background text; it specifies the multi\-step reasoning constraints that govern how candidate entities should be selected across charts\. Crucially, it concludes by assigning a semantic reference label to the entities obtained by executing this reasoning trace under its logical composition\. Questions are then formulated to refer to this label rather than explicit entity names, requiring models to first resolve the target entities from the document context and subsequently retrieve and aggregate the corresponding numerical evidence from the charts\. This design ensures that correct answers depend on joint understanding of both the narrative constraints and the visual data\. An illustrative example can be found in Figure[1](https://arxiv.org/html/2609.02059#S1.F1)\.

Controllable Reasoning Complexity\. To achieve a rigorous evaluation, we employ a stochastic logic\-generation pipeline rather than relying on noisy web\-scraped data\. For each instance, we first sample a reasoning trace—a structured chain of conditional checks that necessitates aggregating information from multiple distinct charts\. We then prompt an LLM to generate both the underlying numerical data and the accompanying textual document to strictly satisfy this pre\-defined trace\. Thisreverse\-engineeringapproach ensures semantic consistency between the text and the charts\. Crucially, it allows us to explicitly control difficulty along two axes: reasoning depth \(the number of logical hops\) and visual density \(the number of charts\)\. This results in a diverse difficulty distribution, enabling a systematic diagnostic of model limitations across the entire complexity spectrum—from lower\-depth chart–context reasoning to high\-load multi\-step aggregation—as detailed in Section[3](https://arxiv.org/html/2609.02059#S3)\.

Diverse Reasoning Contexts\. Our stochastic generation process ensures high variance across logical, visual, and thematic dimensions\. The reasoning traces are randomly generated to produce varied structures, ranging from linear chains to multi\-branch compositions\. Together, these variations broaden the coverage of chart–context reasoning patterns evaluated by the benchmark\. Across semantic domains, the benchmark spans 480 distinct topics, and its documents include 7 different chart types\. In total, DocHop comprises 2,074 examples covering 6 distinct tasks: Value Retrieval, Counting, Numeric Reasoning, Ranking, Hypothetical Reasoning, and Fact Checking\. These are instantiated via 41 unique question templates, ensuring a comprehensive evaluation of model generalizability\.

Multi\-Stage Quality Verification\. To ensure reliability, we implement a multi\-stage verification protocol\. First, leveraging the structured nature of our generation pipeline, we programmatically validate the numerical data in every example\. Since the underlying data tables and reasoning traces are fully accessible, we execute the trace against the data tables to deterministically verify that the generated chart values satisfy the instantiated constraints and that the ground\-truth answers are mathematically correct\. Second, we conduct human verification of the document context to ensure that the generated narrative faithfully verbalizes the underlying reasoning trace, including the textual constraints, logical compositions, and semantic reference label\. Third, we manually inspect the rendered document images to ensure visual readability and completeness, checking for issues such as truncated context, overlapping layout elements, occluded charts, or illegible labels\. This combination of programmatic validation and human review supports high\-fidelity evaluation standards\.

SinceDocHopis synthetically constructed rather than sampled from naturally occurring documents, we position it as an out\-of\-domain multi\-hop reasoning benchmark for chart–document integration\. Its goal is not to model a specific real\-world document distribution, but to test whether models can use narrative context to identify the relevant chart evidence and aggregate it across multiple charts\.

We evaluate DocHop on a wide range of proprietary and open\-source MLLMs\. The results show that DocHop remains challenging but not intractable: human annotators achieve over 90% accuracy, while the best\-performing model \(GPT\-5\.2 Reasoning\([OpenAI, 2025a](https://arxiv.org/html/2609.02059#bib.bib25)\)\) reaches 62\.83% overall\. Reasoning\-enhanced variants consistently outperform their non\-reasoning counterparts across both GPT and Gemini families\. Proprietary models remain stronger overall, with open\-source baselines typically demonstrating 9–24% accuracy in our evaluation\.

Moreover, DocHop enables controlled analysis along reasoning depth and chart\-count axes, where performance degrades steadily as either complexity factor increases \(Figure[5](https://arxiv.org/html/2609.02059#S4.F5)\)\. Finally, we provide qualitative failure case studies to better understand common breakdowns in narrative grounding and cross\-chart evidence aggregation\.

## 2Related Work

### 2\.1Multimodal Large Language Models

Recent Multimodal Large Language Models \(MLLMs\) extend large language models with visual encoders to jointly process images and text\. Through large\-scale pretraining and instruction tuning, both proprietary and open\-source MLLMs have demonstrated strong performance across a wide range of multimodal understanding tasks\. Representative proprietary systems include the GPT series\([OpenAI, 2025b](https://arxiv.org/html/2609.02059#bib.bib2)\), Gemini\([Comanici et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib3)\), and Claude models\([Anthropic, 2025a](https://arxiv.org/html/2609.02059#bib.bib24);[Anthropic, 2025b](https://arxiv.org/html/2609.02059#bib.bib23)\)\. In parallel, open\-source models such as LLaVA\-style architectures\([Liu et al\., 2024b](https://arxiv.org/html/2609.02059#bib.bib17);[Liu et al\., 2023](https://arxiv.org/html/2609.02059#bib.bib26)\), IDEFICS\([Laurençon et al\., 2024b](https://arxiv.org/html/2609.02059#bib.bib18);[Laurençon et al\., 2024a](https://arxiv.org/html/2609.02059#bib.bib19)\), Qwen\-VL\([Bai et al\., 2025b](https://arxiv.org/html/2609.02059#bib.bib20);[Bai et al\., 2025a](https://arxiv.org/html/2609.02059#bib.bib21)\), Molmo\([Deitke et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib22)\), and InternVL\([Wang et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib4)\)provide competitive performance and have enabled reproducible research across diverse benchmarks\. These general\-purpose MLLMs adopt unified architectures that integrate visual perception and language reasoning, enabling zero\-shot and few\-shot generalization across tasks involving images, documents, and structured layouts\. As a result, they have become standard baselines for evaluating multimodal understanding capabilities in contemporary benchmarks\. However, their capabilities in complex multi\-modal multi\-hop reasoning tasks remain underexplored\.

### 2\.2Benchmarks for Structured Multimodal Reasoning

A large body of benchmarks has been proposed to evaluate visual reasoning over structured images—particularly in charts, documents, and webpages\. Early chart question answering datasets such as FigureQA\([Kahou et al\., 2017](https://arxiv.org/html/2609.02059#bib.bib5)\), DVQA\([Kafle et al\., 2018](https://arxiv.org/html/2609.02059#bib.bib6)\), LEAF\-QA\([Chaudhry et al\., 2020](https://arxiv.org/html/2609.02059#bib.bib7)\), and PlotQA\([Methani et al\., 2020](https://arxiv.org/html/2609.02059#bib.bib8)\)rely heavily on questions generated from limited templates\.

ChartQA\([Masry et al\., 2022](https://arxiv.org/html/2609.02059#bib.bib9)\)introduces human\-authored questions on real\-world charts and improves linguistic diversity, but still focuses on single\-chart reasoning without requiring external textual context\. More recent benchmarks on chart and document question answering, including ChartBench\([Xu et al\., 2024](https://arxiv.org/html/2609.02059#bib.bib10)\), MMC\([Liu et al\., 2024a](https://arxiv.org/html/2609.02059#bib.bib11)\), ChartX\([Xia et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib12)\), ChartXiv\([Wang et al\., 2024](https://arxiv.org/html/2609.02059#bib.bib13)\), ChartMuseum\([Tang et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib14)\), ChartQAPro\([Masry et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib16)\), and DocVQA\([Mathew et al\., 2021](https://arxiv.org/html/2609.02059#bib.bib15)\)increase image diversity and question difficulty by incorporating scientific figures, multi\-chart inputs, or open\-ended answers\. Recently, DocHop\-QA\([Park et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib31)\)studies multimodal multi\-hop question answering over document collections\. However, these benchmarks largely follow a single\-hop or limited multi\-chart paradigm, where the reasoning logic is implicit in the question rather than explicitly provided and followed across modalities\. In contrast to prior benchmarks, DocHop is designed to evaluate multi\-round cross\-modal reasoning within a single document\.

![Refer to caption](https://arxiv.org/html/2609.02059v1/dochop_pipeline.png)Figure 2:Illustration of the DocHop data generation pipeline\.\(1\) Metadata Pool: We first prompt an LLM to curate a diverse pool of semantic metadata, including topics, entity names, candidate metrics, and timestamps that may appear in an instance\.\(2\) Chart Configuration Sampling: From this pool, we sample a set of chart schemas by selecting chart types \(e\.g\., bar, line\) and instantiating their required structural fields from the metadata\.\(3\) Reasoning Trace Generation: Conditioned on the sampled chart configurations, we stochastically construct a multi\-hop reasoning trace, where each node corresponds to an instantiated constraint parameterized by a specific chart schema\.\(4\) Chart and Document Context Generation: Given the symbolic trace, we prompt an LLM to generate constraint\-satisfying chart tables and an accompanying narrative context that verbalizes the trace, while keeping the two processes independent\.\(5\) QA Curation and Document Rendering: Finally, we assign a semantic reference label in the narrative, instantiate task\-specific question–answer pairs grounded on this label, and render the charts and text into a single\-page document image for evaluation\.

## 3DocHop

DocHop is constructed with a logic\-first, reverse\-engineered framework\. The key design is to separate*reasoning specification*from*numerical evidence*: a symbolic reasoning trace defines the multi\-step constraints, the document narrative verbalizes these constraints, and the charts provide the corresponding data values\. To enforce joint chart\-context reasoning, the narrative concludes by assigning a*semantic reference label*to the entities implied by executing the trace under its logical composition, and all questions are grounded on this label rather than explicit entity names\. This prevents direct chart\-only lookup and requires models to resolve targets from the narrative before aggregating evidence from charts\. We organize the construction process into four main components: \(1\) Symbolic Reasoning Trace Generation, \(2\) Chart and Document Context Generation, \(3\) Document Composition and QA Curation, and \(4\) Quality Verification\.

### 3\.1Symbolic Reasoning Trace Generation

This subsection describes the first three stages of the pipeline: metadata pool curation, chart configuration sampling, and symbolic reasoning trace generation\. The goal is to construct an abstract reasoning problem before any concrete chart values are generated\.

We first establish a semantic context by prompting a large language model \(LLM\) to generate a diverse metadata pool\. Each pool defines the global semantic space of an instance, including the document topic, candidate entities, measurable metrics, semantic groups that subdivide metric values, value units, and available timestamps\. From this pool, we sample a set ofkkchart schemas, wherek∈\{2,3,4,6\}k\\in\\\{2,3,4,6\\\}\. Each schema instantiates a chart structure by selecting metadata fields appropriate for a particular chart type, such as tracking a metric over time or comparing metric values across groups\. At this stage, the schemas specify only the structural components of the charts, without any concrete numerical values; the values are generated later conditioned on the reasoning trace in the chart and document context generation stage\.

Conditioned on the sampled chart schemas, we construct a symbolic reasoning trace modeled as a logical tree with a target reasoning depthD∈\{2,3,4,5\}D\\in\\\{2,3,4,5\\\}\. The trace contains two types of nodes\. Constraint nodes represent parameterized rules instantiated from templates and filter candidate entities based on chart\-grounded conditions\. Logical composition nodes combine entity sets from their input nodes using Boolean operations\. Directed edges govern how candidate entity sets are passed through the tree\. For a child constraint node, the node inherits the candidate entity pool from its parent and applies its rule to further filter this set\. Consequently, the entities satisfying a child constraint node form a subset of those satisfying its parent\. The trace is generated recursively through the following mechanisms:

Node Parameterization\.To promote constraint diversity, we employ a library of 27 distinct rule templates \(see Appendix[C](https://arxiv.org/html/2609.02059#A3)\) designed by human annotators\. Each template is formulated as a parameterized function \(e\.g\.,<Timestamp\>:<Metric\><Operator\><Threshold\>\) that must be instantiated with concrete values when sampled\. During generation, the pipeline first samples a target subset of entities from the valid pool passed down by the parent node, and then populates the remaining slots \(e\.g\., operator and threshold\) to form a definite constraint \(e\.g\.,Products A and B satisfy the constraint “2024\-06: Items Sold≥\\geq45”\)\. This ensures that the constraint is both structurally valid and logically consistent with the preceding context\.

Logical Composition\.Logical composition nodes aggregate constraints by combining entity sets from their input nodes\. These nodes are parameterized by sampling a Boolean operator, specificallyANDorOR\. Unlike constraint nodes that filter entities through attribute\-based conditions, the output entity set of a logical composition node is deterministic and logic\-driven: it is computed by directly applying the sampled Boolean operator to its input entity sets\.

The algorithm iteratively samples and composes these nodes until the target depthDDis reached\. Due to the stochastic nature of this recursive generation, the process yields diverse reasoning skeletons, ranging from simple linear chains to complex multi\-branch trees\. The resulting logical structure specifies how entities should be filtered and combined across the sampled chart schemas, and serves as the blueprint for subsequent data synthesis\. Examples of these diverse tree structures are provided in Appendix[B](https://arxiv.org/html/2609.02059#A2)\.

\(a\)Distribution by Task![Refer to caption](https://arxiv.org/html/2609.02059v1/dochop_depth_chartnum_heatmap.png)\(b\)Distribution over Reasoning Depth and Number of Charts

Figure 3:Dataset distribution ofDocHop\. We report the distribution across task categories \(left\) and the distribution over reasoning depths and numbers of charts per document \(right\)\.
### 3\.2Chart and Document Context Generation

Given a sampled reasoning trace and its associated chart schemas, we synthesize the two complementary sources of information in eachDocHopinstance: chart tables that provide numerical evidence, and document narratives that verbalize the reasoning specification\. We generate these two components independently so that the document context describes the constraints without leaking the underlying chart values or the final entity assignments\. We use Gemini\-2\.5\-Pro for all LLM\-based synthesis steps, and full prompt details are provided in Appendix[A](https://arxiv.org/html/2609.02059#A1)\.

Chart Data Generation\. We generate numerical tables that satisfy the instantiated constraints in the reasoning trace\. A naive approach would be to prompt the model to produce all charts jointly conditioned on the full multi\-hop trace; however, as reasoning depth and chart count grow, it becomes difficult to reliably ensure that the generated tables satisfy all specified constraints simultaneously\. To improve reliability, we adopt a per\-chart synthesis strategy: we traverse the reasoning tree and aggregate all regular\-node constraints associated with the same chart into a single requirement set\. The prompt explicitly specifies which entities must satisfy or violate each condition, and the LLM generates the corresponding chart table independently for each chart\. We then verify correctness by executing the reasoning trace programmatically\. When converting these tables into chart images, we randomly sample a compatible chart type and color scheme for each chart, introducing diversification in both graphical form and appearance\.

Document Context Generation\. In parallel, we verbalize the symbolic reasoning trace into a coherent document narrative\. Context generation is performed independently from chart synthesis: the LLM has no access to chart values or to the final entity set produced by executing the trace\. Instead, it is provided only with the instantiated constraint structure, including the parameterized conditions at each node and their logical composition\. This information is sufficient to describe the reasoning procedure in natural language, while preventing the narrative from revealing numerical evidence or directly naming the resulting target entities\.

### 3\.3Question and Answer Curation and Document Rendering

Semantic Reference Label\. After generating the document narrative, we assign a semantic reference label \(e\.g\.,Platinum Tier Night Servicein Figure[1](https://arxiv.org/html/2609.02059#S1.F1)\) to the entities determined by executing the reasoning trace under its logical composition\. Questions are grounded on this label rather than explicit entity names, preventing direct chart\-only lookup and requiring models to first identify the relevant entity set from the narrative before aggregating numerical evidence from the charts\. For instance, instead of asking for the average metric value of specific hotels \(e\.g\., HTH, SCS, WG in Figure[1](https://arxiv.org/html/2609.02059#S1.F1)\), a question is phrased as: “What is the averageLinens ReplacedunderNight ShiftinAugust 2025of the hotel\(s\) receiving thePlatinum Tier Night Service Commendation?”

Task Definitions\. Following prior chart question answering benchmarks, we formulateDocHopquestions into six reasoning categories:

- •*Value Retrieval*: Retrieve a metric of entities assigned the semantic reference label, or directly ask which entities are assigned the semantic reference label\.
- •*Counting*: Count how many entities are assigned the semantic reference label, optionally under an additional chart\-grounded condition specified in the question\.
- •*Numeric Reasoning*: Perform arithmetic aggregation \(e\.g\., sum or average\) over chart entries of entities assigned the semantic reference label\.
- •*Ranking*: Identify extrema or order entities assigned the semantic reference label based on their chart values\.
- •*Hypothetical Reasoning*: Answer counterfactual queries that modify either chart values while keeping the semantic reference label fixed, or the reasoning trace itself by introducing additional constraints\.
- •*Fact Checking*: Verify whether a statement is entailed by jointly applying the textual constraints and chart evidence over entities assigned the semantic reference label\.

Question and Answer Construction\. For each task type, we start from a corresponding question template and instantiate it using the synthesized document metadata\. These templates \(41 in total\) are designed by human annotators to cover diverse reasoning patterns across tasks\. Questions are grounded on the semantic reference label defined in the narrative rather than explicit entity names in the charts, ensuring that answering requires integrating information from the narrative context and the associated charts\. For example, a numeric reasoning question may ask: “What is the totalTransaction Costs PaidinDecember 2024of all investment funds receiving theCertified Prudent Operator designation?”\. Since the reasoning trace deterministically defines the target entity set, ground\-truth answers are computed exactly by executing the corresponding operations over the generated chart tables\. We provide the full list of question templates under each task in Appendix[D](https://arxiv.org/html/2609.02059#A4)\.

Document Rendering\. We render the synthesized document narrative and chart images into a single\-page document using ReportLab with a fixed A4\-style layout; the question and answer are kept separately and are not included in the rendered document\. To ensure that each instance fits on one page, we start from font size 10 and progressively decrease the font size until the narrative fits\. We also adjust chart scaling across different subplot layouts so that individual subplots remain roughly comparable in visual size\. The final document is rendered at 300 DPI and paired with its corresponding question and verified answer to form oneDocHopevaluation instance\. The entire data curation pipeline is illustrated in Figure[2](https://arxiv.org/html/2609.02059#S2.F2)\.

### 3\.4Quality Verification

We apply a multi\-stage verification process to ensure the correctness and readability ofDocHopinstances\.

Programmatic Verification of Chart Data\. Since the symbolic reasoning trace and generated chart tables are fully accessible, we verify each instance by executing the trace against the chart data\. This allows us to deterministically check whether the generated numerical values satisfy all instantiated constraints and whether the resulting target entity set and ground\-truth answer are correct\. Instances that fail this verification are discarded and regenerated\.

Human Verification of Document Context\. We then manually inspect whether the generated document narrative faithfully verbalizes the underlying reasoning trace\. This step checks whether the textual constraints, logical compositions, and semantic reference label are expressed clearly and consistently with the symbolic trace\. Instances with missing, ambiguous, or incorrect descriptions of the reasoning process are rejected\.

Final Rendering Verification\. Finally, we inspect the rendered document images to ensure that the document context and charts are fully visible and readable\. This includes checking for truncated text, missing context, overlapping elements, occluded charts, illegible labels, or other layout artifacts introduced during PDF rendering\. Instances that fail the final rendering check are regenerated\.

### 3\.5Dataset Statistics

DocHop contains 2,074 document instances, each paired with a single question\-answer example\. We construct the benchmark with controllable reasoning complexity along two axes: reasoning depthD∈\{2,3,4,5\}D\\in\\\{2,3,4,5\\\}and the number of charts per documentk∈\{2,3,4,6\}k\\in\\\{2,3,4,6\\\}\. Each example is assigned to one of six task categories \(Value Retrieval, Counting, Numeric Reasoning, Ranking, Hypothetical Reasoning, and Fact Checking\), instantiated from a library of question templates\.

During generation, we aim to balance the sampling across task types as well as across different depth and chart\-count configurations, enabling systematic evaluation over a broad range of reasoning difficulty\. The example distributions of DocHop can be found in Figure[3](https://arxiv.org/html/2609.02059#S3.F3)\.

Table 1:Evaluation results on DocHop across task categories\. We report both per\-task accuracy and the overall accuracy of DocHop\.
### 3\.6Evaluation Metrics

We evaluate model performance using normalized answer accuracy\. For non\-numeric questions, we use case\-insensitive exact match\. For numeric questions, we normalize outputs to a canonical format and apply tolerance\-based matching under the required precision\. For binary fact\-checking questions, accuracy is computed over Yes/No predictions\. Additional evaluation details, such as prompting instructions, answer parsing rules, and numeric precision handling, are provided in the Appendix[E](https://arxiv.org/html/2609.02059#A5)\.

## 4Experiments

### 4\.1Evaluation Setup

![Refer to caption](https://arxiv.org/html/2609.02059v1/qualitative-updated.png)Figure 4:Qualitative examples of reasoning outputs\. \(a\) An instance where GPT\-5\.2 Reasoning answers correctly, while Gemini\-2\.5\-Pro fails by attributing chart values to the wrong entity, leading to an incorrect aggregation\. \(b\) A challenging case where both models make incorrect predictions but for different reasons: GPT identifies the global maximum instead of restricting to entities receiving the semantic label, whereas Gemini resolves the correct entity set from the narrative but makes an OCR mistake in the final value extraction\.Evaluated Models\. We evaluate a broad range of state\-of\-the\-art MLLMs, including both proprietary and open\-source systems\. Our proprietary model set includes GPT\-5\.2\([OpenAI, 2025a](https://arxiv.org/html/2609.02059#bib.bib25)\), GPT\-5\-mini\([OpenAI, 2025b](https://arxiv.org/html/2609.02059#bib.bib2)\), Gemini\-2\.5\-Pro/Flash\([Comanici et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib3)\), Claude 4\.5 Haiku\([Anthropic, 2025a](https://arxiv.org/html/2609.02059#bib.bib24)\), and Claude 4\.5 Sonnet\([Anthropic, 2025b](https://arxiv.org/html/2609.02059#bib.bib23)\)\. We additionally report results for their reasoning\-enhanced variants when available \(e\.g\., GPT\-5\.2\-Reasoning, Gemini\-2\.5\-Pro\-Reasoning, Gemini\-2\.5\-Flash\-Reasoning\)\.

For open\-source baselines, we consider representative vision–language models across multiple families and scales, including LLaVA\-Next\-LLaMA3\-8B\([Liu et al\., 2024b](https://arxiv.org/html/2609.02059#bib.bib17)\), IDEFICS2\-8B\([Laurençon et al\., 2024b](https://arxiv.org/html/2609.02059#bib.bib18)\), IDEFICS3\-LLaMA3\-8B\([Laurençon et al\., 2024a](https://arxiv.org/html/2609.02059#bib.bib19)\), Qwen\-2\.5\-VL \(7B/32B/72B\)\([Bai et al\., 2025b](https://arxiv.org/html/2609.02059#bib.bib20)\), Qwen3\-VL\-8B\([Bai et al\., 2025a](https://arxiv.org/html/2609.02059#bib.bib21)\), Molmo\-7B \(O/D\)\([Deitke et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib22)\), Ovis1\.6\-Gemma2\-9B\([Lu et al\., 2024](https://arxiv.org/html/2609.02059#bib.bib30)\)and InternVL\-3\.5 \(8B/30B\-A3B/38B\)\([Wang et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib4)\)\. Full model configuration settings are provided in Appendix[F](https://arxiv.org/html/2609.02059#A6)\.

Human and Random Guessing Baselines\. We obtain human performance by hiring graduate student annotators, who are presented with the full document image and asked to provide free\-form answers\. Following ChartXiv\([Wang et al\., 2024](https://arxiv.org/html/2609.02059#bib.bib13)\), we also obtain a*question\-only*guessing baseline by prompting GPT\-5\.2\([OpenAI, 2025a](https://arxiv.org/html/2609.02059#bib.bib25)\)and Gemini\-2\.5\-Pro\([Comanici et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib3)\)with only the question text, without providing the document image\. This serves as a language\-prior baseline\.

### 4\.2Main Results

Table[1](https://arxiv.org/html/2609.02059#S3.T1)summarizes the full evaluation results\. Here, we summarize the main takeaways\.

Overall Gap to Human Performance\. All evaluated models, including both proprietary and open\-source ones as well as reasoning variants, remain far below human accuracy\. While human annotators achieve over 90% accuracy, the best\-performing model \(GPT\-5\.2 Reasoning\) reaches only 62\.83% overall, indicating a substantial gap in integrated chart\-context reasoning\. It is also worth noting that it takes approximately five minutes per question on average for human annotators to solve one question, suggesting that DocHop requires careful multi\-step reasoning rather than quick visual lookup\. The large discrepancy between human and model performance highlights the difficulty of document\-level multi\-hop reasoning for current MLLMs\.

Reasoning Models Consistently Outperform Their Non\-Reasoning Counterparts\. As shown in Table[1](https://arxiv.org/html/2609.02059#S3.T1), models with explicit reasoning mechanisms achieve substantial gains over their base variants across both GPT and Gemini families\. For example, GPT\-5\.2 improves from 40\.36% to 62\.83% overall when switching to its reasoning variant, and Gemini\-2\.5\-Flash increases from 24\.88% to 32\.02%\. Gemini\-2\.5\-Pro with a dynamic reasoning mechanism further reaches 40\.60% overall accuracy, outperforming the Gemini\-2\.5\-Flash variants\. Overall, these results indicate that explicit reasoning remains an important factor for stronger performance on DocHop\.

Proprietary Models Remain Stronger Overall\. Among all evaluated models, GPT\-5\.2 Reasoning achieves the best overall performance, reaching 62\.83% accuracy on DocHop\. Gemini\-2\.5\-Pro Reasoning is the next strongest proprietary baseline at 40\.60%, though a substantial gap remains compared to GPT\. More broadly, proprietary models generally outperform the open\-source baselines in our current evaluation\. Most open\-source models achieve overall accuracies in the range of roughly 9–24%, indicating that integrated chart–context reasoning remains challenging for current open vision–language models\.

### 4\.3Analysis

Model performance degrades with more charts and higher reasoning depth\. Figure[5](https://arxiv.org/html/2609.02059#S4.F5)shows that the performance of both GPT\-5\.2 Reasoning and Gemini\-2\.5\-Pro Reasoning consistently decreases as the number of charts in the document and the reasoning depth increase\. Accuracy is highest in simpler settings with fewer charts \(k=2k=2\) and shallow traces \(D=2D=2\), and drops steadily as the document becomes visually denser and the reasoning trace requires more hops\. This trend suggests that scaling DocHop along either axis imposes additional difficulty for current MLLMs in aggregating evidence across multiple charts under narrative constraints\.

\(a\)Performance by Reasoning Depth\(b\)Performance by Number of Charts
Figure 5:Performance of GPT\-5\.2 and Gemini\-2\.5\-Pro under controlled DocHop complexity\. We report accuracy of GPT\-5\.2 Reasoning\([OpenAI, 2025a](https://arxiv.org/html/2609.02059#bib.bib25)\)and Gemini\-2\.5\-Pro Reasoning\([Comanici et al\., 2025](https://arxiv.org/html/2609.02059#bib.bib3)\), both operating with explicit reasoning enabled, as reasoning depth and number of charts increase\. Performance generally degrades for both models as either the reasoning trace becomes deeper or the document becomes visually denser, highlighting the increasing difficulty of cross\-chart evidence aggregation under narrative constraints\.Qualitative Examples\.Figure[4](https://arxiv.org/html/2609.02059#S4.F4)shows two representative DocHop instances\.\(a\) Attribution Error\.In Figure[4](https://arxiv.org/html/2609.02059#S4.F4)\(a\), GPT\-5\.2 Reasoning model answers correctly by grounding the reasoning steps on the right entity throughout\. Gemini\-2\.5\-Pro, however, makes an attribution mistake: it confuses the chart values of two entities, leading to an incorrect aggregation despite otherwise plausible intermediate reasoning\.\(b\) Different Failure Modes on the Same Instance\.Figure[4](https://arxiv.org/html/2609.02059#S4.F4)\(b\) illustrates a case where both models fail, but for different reasons\. GPT\-5\.2 Reasoning loses track of the narrative constraint and reports the global maximum rather than the maximum among entities receiving the SLN Designation\. Gemini\-2\.5\-Pro correctly identifies the qualified entity set, but produces an incorrect final answer due to a misread chart value\. These examples suggest that DocHop remains challenging even for strong proprietary models, with errors in both contextual reasoning and chart understanding\.

## Limitations and Future Work

DocHopis designed as a controlled out\-of\-domain reasoning benchmark rather than a dataset that reproduces the distribution of naturally occurring documents\. This design allows us to isolate chart–document integration under controllable reasoning depth and visual density, but it also means that performance onDocHopshould be interpreted as a diagnostic measure of a specific reasoning capability rather than as a direct estimate of performance on real\-world document collections\. Future work can extend this logic\-grounded evaluation framework to naturally occurring reports, scientific articles, business dashboards, and other document domains\.

In addition,DocHopfocuses on chart\-based numerical evidence paired with narrative context\. Real documents often contain other structured visual elements, such as tables, diagrams, forms, equations, and user\-interface screenshots\. Extending the logic\-grounded construction framework to these document elements would enable broader evaluation over heterogeneous document structures\. Our current rendering pipeline also uses controlled single\-page A4\-style layouts with readable charts and narrative text\. This design reduces confounding factors from severe OCR noise, document parsing failures, and layout artifacts, allowing the benchmark to focus on chart–document reasoning; however, it does not capture the full visual variability of naturally occurring documents\. Future work can introduce multi\-page documents, noisier scans, and more diverse layout styles\. Finally, although our construction pipeline combines programmatic verification with human review, synthetic benchmark generation may still leave occasional annotation, wording, or rendering issues\. We plan to maintain the released benchmark with versioned updates and incorporate validated community feedback in future releases\.

## 5Conclusion

We introducedDocHop, a benchmark for integrated chart–context reasoning in document\-style images, where models must resolve narrative constraints and aggregate numerical evidence across multiple charts\. A key design is the use of semantic reference labels grounded in the document narrative, which prevents direct chart\-only lookup and enforces joint reasoning over text and visual data\. Experiments across both proprietary and open\-source MLLMs reveal a substantial gap to human performance, while reasoning\-enhanced variants consistently improve accuracy over their base counterparts\. We further observe a steady degradation as reasoning depth and chart count increase, highlighting persistent challenges in multi\-hop evidence aggregation under dense visual contexts\. Future work can extend this logic\-grounded evaluation to other structured visual elements \(e\.g\., tables, diagrams, and UI documents\) to better characterize multimodal reasoning in realistic settings\.

## Impact Statement

This paper presents work whose goal is to advance the field of multimodal machine learning through the development ofDocHop, a benchmark for evaluating integrated chart–context reasoning in document images\. Our work is primarily intended to support more reliable assessment of multimodal reasoning capabilities, and does not introduce new deployment\-facing model components\. Potential societal impacts are therefore indirect, such as improving the robustness and transparency of evaluation for systems used in document understanding\. As with other synthetic benchmark datasets, misuse could arise if models are over\-optimized for benchmark performance without corresponding real\-world generalization\. We encourage future work to consider broader coverage of document domains and to complement benchmark\-driven progress with responsible evaluation practices\.

## Acknowledgment

This work was supported in part by NSF IIS2404180, IBM, Institute of Information & communications Technology Planning& Evaluation \(IITP\) grant funded by the Korea government \(MSIT\) \(No\. 2022\-0\-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration\), and Electronics and Telecommunications Research Institute \(ETRI\) grant \(26CB1200, Development and Application of Science\-Specialized Multimodal Foundation Models\)\.

## References

- Anthropic \(2025a\)AnthropicClaude haiku 4\.5\.Note:Accessed: 2026\-01\-28External Links:[Link](https://www.anthropic.com/news/claude-haiku-4-5)Cited by:[§2\.1](https://arxiv.org/html/2609.02059#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.13.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p1.1)\.
- Anthropic \(2025b\)AnthropicIntroducing claude sonnet 4\.5\.Note:Accessed: 2026\-01\-28External Links:[Link](https://www.anthropic.com/news/claude-sonnet-4-5)Cited by:[§2\.1](https://arxiv.org/html/2609.02059#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.14.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p1.1)\.
- Baiet al\.\(2025a\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. ZhuQwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.02059#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.20.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p2.1)\.
- Baiet al\.\(2025b\)S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang,et al\.Qwen2\. 5\-vl technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.02059#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.19.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.26.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.28.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p2.1)\.
- Chaudhryet al\.\(2020\)R\. Chaudhry, S\. Shekhar, U\. Gupta, P\. Maneriker, P\. Bansal, and A\. JoshiLeaf\-qa: locate, encode & attend for figure question answering\.InProceedings of the IEEE/CVF winter conference on applications of computer vision,pp\. 3512–3521\.Cited by:[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p1.1)\.
- Comaniciet al\.\(2025\)G\. Comanici, E\. Bieber, and et al\.Gemini 2\.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities\.External Links:2507\.06261,[Link](https://arxiv.org/abs/2507.06261)Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.02059#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.12.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.3.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.8.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.9.1),[Figure 5](https://arxiv.org/html/2609.02059#S4.F5),[Figure 5](https://arxiv.org/html/2609.02059#S4.F5.5.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p3.1)\.
- Daiet al\.\(2023\)W\. Dai, J\. Li, D\. Li, A\. Tiong, J\. Zhao, W\. Wang, B\. Li, P\. N\. Fung, and S\. HoiInstructblip: towards general\-purpose vision\-language models with instruction tuning\.Advances in neural information processing systems36,pp\. 49250–49267\.Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1)\.
- Deitkeet al\.\(2025\)M\. Deitke, C\. Clark, S\. Lee, R\. Tripathi, Y\. Yang, J\. S\. Park, M\. Salehi, N\. Muennighoff, K\. Lo, L\. Soldaini,et al\.Molmo and pixmo: open weights and open data for state\-of\-the\-art vision\-language models\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 91–104\.Cited by:[§2\.1](https://arxiv.org/html/2609.02059#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.21.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.22.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p2.1)\.
- Kafleet al\.\(2018\)K\. Kafle, B\. Price, S\. Cohen, and C\. KananDvqa: understanding data visualizations via question answering\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 5648–5656\.Cited by:[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p1.1)\.
- Kahouet al\.\(2017\)S\. E\. Kahou, V\. Michalski, A\. Atkinson, Á\. Kádár, A\. Trischler, and Y\. BengioFigureqa: an annotated figure dataset for visual reasoning\.arXiv preprint arXiv:1710\.07300\.Cited by:[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p1.1)\.
- Langley \(2000\)P\. LangleyCrafting papers on machine learning\.InProceedings of the 17th International Conference on Machine Learning \(ICML 2000\),P\. Langley \(Ed\.\),Stanford, CA,pp\. 1207–1216\.Cited by:[Appendix F](https://arxiv.org/html/2609.02059#A6.p4.1)\.
- Laurençonet al\.\(2024a\)H\. Laurençon, A\. Marafioti, V\. Sanh, and L\. TronchonBuilding and better understanding vision\-language models: insights and future directions\.\.External Links:2408\.12637Cited by:[§2\.1](https://arxiv.org/html/2609.02059#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.18.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p2.1)\.
- Laurençonet al\.\(2024b\)H\. Laurençon, L\. Tronchon, M\. Cord, and V\. SanhWhat matters when building vision\-language models?\.Advances in Neural Information Processing Systems37,pp\. 87874–87907\.Cited by:[§2\.1](https://arxiv.org/html/2609.02059#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.17.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p2.1)\.
- Liuet al\.\(2024a\)F\. Liu, X\. Wang, W\. Yao, J\. Chen, K\. Song, S\. Cho, Y\. Yacoob, and D\. YuMMC: advancing multimodal chart understanding with large\-scale instruction tuning\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 1287–1310\.Cited by:[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p2.1)\.
- Liuet al\.\(2024b\)H\. Liu, C\. Li, Y\. Li, B\. Li, Y\. Zhang, S\. Shen, and Y\. J\. LeeLLaVA\-next: improved reasoning, ocr, and world knowledge\.External Links:[Link](https://llava-vl.github.io/blog/2024-01-30-llava-next/)Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.02059#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.16.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p2.1)\.
- Liuet al\.\(2023\)H\. Liu, C\. Li, Q\. Wu, and Y\. J\. LeeVisual instruction tuning\.Advances in neural information processing systems36,pp\. 34892–34916\.Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.02059#S2.SS1.p1.1)\.
- Liuet al\.\(2024c\)J\. Liu, T\. Ou, Y\. Song, Y\. Qu, W\. Lam, C\. Xiong, W\. Chen, G\. Neubig, and X\. YueHarnessing webpage uis for text\-rich visual understanding\.arXiv preprint arXiv:2410\.13824\.Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1)\.
- Liuet al\.\(2024d\)J\. Liu, Y\. Song, B\. Y\. Lin, W\. Lam, G\. Neubig, Y\. Li, and X\. YueVisualwebbench: how far have multimodal llms evolved in web page understanding and grounding?\.arXiv preprint arXiv:2404\.05955\.Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1)\.
- Luet al\.\(2024\)S\. Lu, Y\. Li, Q\. Chen, Z\. Xu, W\. Luo, K\. Zhang, and H\. YeOvis: structural embedding alignment for multimodal large language model\.arXiv preprint arXiv:2405\.20797\.Cited by:[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.23.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p2.1)\.
- Masryet al\.\(2022\)A\. Masry, X\. L\. Do, J\. Q\. Tan, S\. Joty, and E\. HoqueChartQA: a benchmark for question answering about charts with visual and logical reasoning\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 2263–2279\.Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p2.1)\.
- Masryet al\.\(2025\)A\. Masry, M\. S\. Islam, M\. Ahmed, A\. Bajaj, F\. Kabir, A\. Kartha, M\. T\. R\. Laskar, M\. Rahman, S\. Rahman, M\. Shahmohammadi, M\. Thakkar, M\. R\. Parvez, E\. Hoque, and S\. JotyChartQAPro: a more diverse and challenging benchmark for chart question answering\.External Links:2504\.05506,[Link](https://arxiv.org/abs/2504.05506)Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p2.1)\.
- Mathewet al\.\(2021\)M\. Mathew, D\. Karatzas, and C\. JawaharDocvqa: a dataset for vqa on document images\.InProceedings of the IEEE/CVF winter conference on applications of computer vision,pp\. 2200–2209\.Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p2.1)\.
- Methaniet al\.\(2020\)N\. Methani, P\. Ganguly, M\. M\. Khapra, and P\. KumarPlotqa: reasoning over scientific plots\.InProceedings of the ieee/cvf winter conference on applications of computer vision,pp\. 1527–1536\.Cited by:[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p1.1)\.
- OpenAI \(2025a\)OpenAIIntroducing gpt\-5\.2\.Note:Accessed: 2026\-01\-28External Links:[Link](https://openai.com/index/introducing-gpt-5-2/)Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p8.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.11.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.4.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.6.1),[Figure 5](https://arxiv.org/html/2609.02059#S4.F5),[Figure 5](https://arxiv.org/html/2609.02059#S4.F5.5.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p3.1)\.
- OpenAI \(2025b\)OpenAIOpenAI gpt\-5 system card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.02059#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.7.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p1.1)\.
- Parket al\.\(2025\)J\. Park, S\. Pyeon, J\. Kim, R\. C\. Cabal, Y\. Ding, and S\. C\. HanDocHop\-qa: towards multi\-hop reasoning over multimodal document collections\.arXiv preprint arXiv:2508\.15851\.Cited by:[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p2.1)\.
- Tanget al\.\(2025\)L\. Tang, G\. Kim, X\. Zhao, T\. Lake, W\. Ding, F\. Yin, P\. Singhal, M\. Wadhwa, Z\. L\. Liu, Z\. Sprague, R\. Namuduri, B\. Hu, J\. D\. Rodriguez, P\. Peng, and G\. DurrettChartMuseum: testing visual reasoning capabilities of large vision\-language models\.External Links:2505\.13444,[Link](https://arxiv.org/abs/2505.13444)Cited by:[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p2.1)\.
- Wanget al\.\(2025\)W\. Wang, Z\. Gao, L\. Gu, H\. Pu, L\. Cui, X\. Wei, Z\. Liu, L\. Jing, S\. Ye, J\. Shao, Z\. Wang, Z\. Chen, H\. Zhang, G\. Yang, H\. Wang, Q\. Wei, J\. Yin, W\. Li, E\. Cui, G\. Chen, Z\. Ding, C\. Tian, Z\. Wu, J\. Xie, Z\. Li, B\. Yang, Y\. Duan, X\. Wang, Z\. Hou, H\. Hao, T\. Zhang, S\. Li, X\. Zhao, H\. Duan, N\. Deng, B\. Fu, Y\. He, Y\. Wang, C\. He, B\. Shi, J\. He, Y\. Xiong, H\. Lv, L\. Wu, W\. Shao, K\. Zhang, H\. Deng, B\. Qi, J\. Ge, Q\. Guo, W\. Zhang, S\. Zhang, M\. Cao, J\. Lin, K\. Tang, J\. Gao, H\. Huang, Y\. Gu, C\. Lyu, H\. Tang, R\. Wang, H\. Lv, W\. Ouyang, L\. Wang, M\. Dou, X\. Zhu, T\. Lu, D\. Lin, J\. Dai, W\. Su, B\. Zhou, K\. Chen, Y\. Qiao, W\. Wang, and G\. LuoInternVL3\.5: advancing open\-source multimodal models in versatility, reasoning, and efficiency\.External Links:2508\.18265,[Link](https://arxiv.org/abs/2508.18265)Cited by:[§2\.1](https://arxiv.org/html/2609.02059#S2.SS1.p1.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.24.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.25.1),[Table 1](https://arxiv.org/html/2609.02059#S3.T1.3.1.27.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p2.1)\.
- Wanget al\.\(2024\)Z\. Wang, M\. Xia, L\. He, H\. Chen, Y\. Liu, R\. Zhu, K\. Liang, X\. Wu, H\. Liu, S\. Malladi,et al\.Charxiv: charting gaps in realistic chart understanding in multimodal llms\.Advances in Neural Information Processing Systems37,pp\. 113569–113697\.Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p2.1),[§4\.1](https://arxiv.org/html/2609.02059#S4.SS1.p3.1)\.
- Xiaet al\.\(2025\)R\. Xia, B\. Zhang, H\. Ye, X\. Yan, Q\. Liu, H\. Zhou, Z\. Chen, P\. Ye, M\. Dou, B\. Shi, J\. Yan, and Y\. QiaoChartX & chartvlm: a versatile benchmark and foundation model for complicated chart reasoning\.External Links:2402\.12185,[Link](https://arxiv.org/abs/2402.12185)Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p2.1)\.
- Xuet al\.\(2024\)Z\. Xu, S\. Du, Y\. Qi, C\. Xu, C\. Yuan, and J\. GuoChartBench: a benchmark for complex visual reasoning in charts\.External Links:2312\.15915,[Link](https://arxiv.org/abs/2312.15915)Cited by:[§1](https://arxiv.org/html/2609.02059#S1.p1.1),[§2\.2](https://arxiv.org/html/2609.02059#S2.SS2.p2.1)\.

## Appendix ALLM Prompts for Data Curation

We provide the prompt templates used in the construction of DocHop\. Given a sampled symbolic reasoning trace \(Section[3](https://arxiv.org/html/2609.02059#S3)\), we employ an LLM to synthesize both the underlying chart tables and the accompanying document narrative\. To ensure reliability and prevent information leakage, chart generation and context generation are performed independently: chart prompts include explicit constraint satisfaction requirements, while narrative prompts only receive the instantiated constraint structure without access to chart values or entity assignments\.

Across the pipeline, we use three main classes of prompts: \(i\)Metadata Pool Curation, which generates diverse topics, entities, metrics, and timestamps; \(ii\)Per\-chart Table Synthesis, which produces numerical tables satisfying aggregated chart\-specific constraints; and \(iii\)Narrative Context Generation, which verbalizes the reasoning trace into policy\-style document text and introduces the semantic reference label used for question grounding\. All prompts are executed with Gemini\-2\.5\-Pro\.

Metadata Pool Prompt\.

YouareMetaSetWriter\.ReturnONLYasingleJSONobject\.

TOPIC:"\{topic\}"

WHATTORETURN

\{

"entity\_type":"<singulartypelabel\>",

"entities":\["<Enames\>"\],

"groups":\["<Gorthogonalsubgroups\>"\],

"metrics":\["<MindependentmetricsinSAMEfamily\>"\],

"unit":"<Aconcretepluralnounrepresentingthemeasurementunit\>",

"data\_type":"<’integer’OR’float’\>",

"timestamps":\["<Tstrictlyincreasingstamps\(e\.g\.,Years’2020’orMonths’2022\-01’\)\>"\]

\}

LOGICRULES\(CRITICAL\)

1\.NOMATHEMATICALREDUNDANCY\(INDEPENDENCE\):

\-Metricsmustbeindependentvariablessothatrandomvaluesdonotcontradicteachother\.

\-THE"PARTvs\.WHOLE"RULE:

\-CONFLICT:DoNOTincludebotha"Total"andits"Parts"inthesamelist\.

\-Bad:\["TotalRevenue","ServiceRevenue","ProductRevenue"\]

\(BecauseProduct\+ServicemightnotequalTotalinrandomdata\)\.

\-SolutionA\(ComponentsOnly\):\["ServiceRevenue","ProductRevenue","ConsultingRevenue"\]\.

\-SolutionB\(AggregatesOnly\):\["TotalRevenue","TotalCost","TotalTax"\]\.

\-THE"FORMULA"RULE:

\-CONFLICT:DoNOTincludevariablesthataresimplemathfunctionsofothers\.

\-Bad:\["Revenue","Cost","Profit"\]\(BecauseProfit=Revenue\-Cost\)\.

\-Fix:Pickalevelofabstraction\.E\.g\.,just\["Revenue","Cost","MarketingSpend"\]

ORjust\["NetProfit","EBITDA"\]\.

2\.DESCRIPTIVEMETRICNAMES\(NATURALLANGUAGEFRIENDLY\):

\-Choosemetricnamesthatsoundnaturalinasentence\(e\.g\.,"The<Metric\>washigh"\)\.

\-GOOD:"BottlesProduced","PatientsAdmitted","ParcelsDelivered"\.

\-BAD:"TotalProduced","Count","Number"\.

\-Note:ItisOKiftheMetricnamerepeatstheUnitname

\(e\.g\.,Metric:"BottlesProduced",Unit:"Bottles"\)ifitimprovesclarity\.

3\.UNITSCALING&SPECIFICITY:

\-NOgeneric"counts"or"values"\.Usespecificnouns:"Parcels","Patients","Tons","Joules"\.

\-SCALING:Usescaleprefixes\(Thousand,Million,Billion\)forANYunitiftheentityscalejustifiesit\.

\-"MillionUSD"\(forCorporations\)

\-"ThousandLiters"\(forWaterPlants\)

\-"MetricTons"\(forMining\)

\-"MillionUsers"\(forTechPlatforms\)

\-NORATES:NO"percent"or"ratio"\.Metricsmustbesummable\.

4\.GROUPSMUSTBEUNIVERSALCATEGORIES\(ORTHOGONAL\):

\-THEGOLDENRULE:EverysingleMetricmustmakesenseforEverysingleGroup\.

\-GOOD:Groups=\["North","South"\]\(Appliestoalmostanything\)\.

\-BAD:Metric="VirusCases",Groups=\["Viral","Bacterial"\]

\(AViruscannotbeBacterial\-\>LogicFail\)\.

\-Groupsmustactasbucketsthatcategorizethemetrics,regardlessofwhatthemetricis\.

SAFETY&CONTENTGUIDELINES\(STRICT\)

1\.NOBIOLOGICALFLUIDS/GORE:

\-Ban:"Blood","Organs","BodilyFluids","Kills"\.

\-Alt:"Essence","Samples","Elixir","Units"\.

2\.NOVIOLENCE/HARM:

\-Ban:"Fatalities","Casualties","Deaths"\.

\-Alt:"Incidents","Transfers","Losses","Expirations"\.

3\.NOSENSITIVEREAL\-WORLDTOPICS:

\-Avoidrealreligiousfigures,politicalhatespeech,orillicitcontraband\.

\-Keepfantasy/culttopicsPG\-13\(e\.g\.,"Mana","Relics"\)\.

4\.NOREAL\-WORLDENTITIES\(USEFICTIONALREALISM\):

\-Ban:Realcountrynames\(e\.g\.,USA,China\),realcities\(e\.g\.,Paris,Tokyo\),

realfamouspeople/politicians,orrealcompanynames\.

\-Ban:Lazyplaceholders\(e\.g\.,"RegionA","PersonX","Country1"\)\.

\-Do:Inventrealistic\-soundingbutfictionalnames

\(e\.g\.,"Westhaven","PortMeridian","DirectorVance","SilverCreek","TheNordicUnion"\)\.

\{entity\_type\_clause\}

IN\-CONTEXTEXAMPLEA\(Financial/Annual\-NoMathConflicts\)

\{

"entity\_type":"conglomerate",

"entities":\["AetherCorp","NebulaLogistics","TerraFirm"\],

"groups":\["NorthAmerica","EMEA","APAC"\],

"metrics":\["GlobalRevenue","LiquidityAssets","MarketCap"\],

"unit":"MillionUSD",

"data\_type":"float",

"timestamps":\["2018","2019","2020","2021","2022","2023"\]

\}

IN\-CONTEXTEXAMPLEB\(Operational/Monthly\-ScaledPhysicalUnit\)

\{

"entity\_type":"bottling\_plant",

"entities":\["VortexSprings","AlpinePure","CrystalCove"\],

"groups":\["GlassBottles","PETPlastic","AluminumCans"\],

"metrics":\["BottlesProduced","BottlesScrapped","BottlesShipped"\],

"unit":"ThousandBottles",

"data\_type":"integer",

"timestamps":\["2023\-01","2023\-02","2023\-03","2023\-04","2023\-05","2023\-06"\]

\}

Data Synthesis Prompt–Line Plot as Example\.

Youmustgeneratethechartbelow\.Thechartmuststrictlyfollowthecharttype,metrics,timestamps,groups,andrulesassignedtoit\.Allentity\-levelconstraintsmustbesatisfiedexactly\.

IMPORTANTCONSTRAINTINTERPRETATION:

\-WhenarulespecifiesthatcertainentitiesMUSTsatisfyaconstraint,thoseentitiesmuststrictlysatisfyit\.

\-EntitiesNOTmentionedinaconstraintMUSTVIOLATEthatconstraint\.

\-Forexample:’ForentitiesA,B:value\>=70’meansAandBmusthavevalue\>=70,whileallotherentitiesmusthavevalue<70\.

YoumustreturnthefinalanswerinthisJSONformat:

\{

"\{chart\_id\}":"<csvstring\>"

\}

Wherethe<csvstring\>isvalidCSV,with:

\-Firstcolumn=entityname

\-\*\*Allnumericalvaluesarein:\{unit\}\*\*\.

\-Notethescale’\{unit\}’:Avalueof50means50\{unit\}\.

\-ColumnheadersMUSTbeexactlythemetric/groupnamesprovided\.DONOTappendtheyear\(e\.g\.use’Revenue’,NOT’Revenue\(2017\)’\)\.

\-Nocommentary,nomarkdown,nochain\-of\-thought,noexplanation\.

\-YouMUSTNOTleavethechartempty\.

\-CSVstringmaybelong;thatisallowed\.

\-YouMUSToutputrealvaluesforthechart\.

\-Eachvalueshouldbeafloatroundedtotwodecimalplacesbetween20\.00and99\.99\.

\-Donotmakeallvaluesendwith\.00or\.50toavoidlookingartificial\.

\-Exampleendingscaninclude\.23,\.87,\.14,\.69,etc\.

Badexamples\(DONOToutput\):

20\.00,21\.00,22\.00,23\.50,24\.50

Goodexamples\(followthesepatterns\):

20\.13,21\.47,22\.84,23\.26,24\.79

CRITICALVISUALIZATIONREQUIREMENTFORLINECHARTS:

\-Whenplotted,thelinesmustbevisuallydistinguishable\.

\-Tryyourbesttocuratevaluessothatateachtimestamp,notwoentitieswillhaveclosevalues\.

\-i\.e,trytoavoidtwopointsbeingtooclosewhenplottingthedata\.

ExampleOutput1:

\{

"\{chart\_id\}":"Entity,2017,2018,2019,2020,2021,2022,2023,2024,2025\\n

A,85\.23,87\.91,86\.45,88\.17,87\.29,89\.73,88\.42,86\.84,87\.56\\n

B,52\.84,58\.39,49\.17,55\.73,61\.29,47\.84,54\.18,59\.62,51\.47\\n

C,38\.47,42\.91,46\.23,50\.18,53\.84,57\.29,60\.73,64\.18,67\.92\\n

D,23\.91,26\.47,29\.18,31\.84,28\.73,32\.19,35\.42,33\.76,36\.84"

\}

NOTE:A\(high80s,stable\),B\(mid50s,highlyvolatile\),

C\(risingfrom38to68\),D\(low30s,slowrise\)

ExampleOutput2:

\{

"\{chart\_id\}":"Entity,2017,2018,2019,2020,2021,2022,2023,2024,2025\\n

A,89\.23,84\.67,79\.41,74\.85,70\.29,66\.73,63\.18,59\.84,56\.47\\n

B,44\.81,41\.29,38\.73,36\.92,39\.47,43\.18,47\.84,52\.29,56\.73\\n

C,67\.34,71\.92,76\.18,79\.84,75\.29,70\.47,65\.91,61\.28,57\.73\\n

D,28\.47,24\.91,21\.73,25\.29,29\.84,34\.18,38\.73,43\.29,47\.91"

\}

NOTE:A\(high70s,steadydecline\),B\(mid40s,U\-shaped\),

C\(high60s\-70s,inverted\-U\),D\(low30s,V\-shaped\)

Bothexamplesshowdiversepatterns\-yourdatashouldalsohavevariedtrends\.

\-\-\-

\#\#CHART\{chart\_id\}

Type:LINE

Entities:\{entity1,entity2,entity3,\.\.\.\}

Unit:\{unit\}

Metric:\{metric\_name\}

Timestamps:\{2017,2018,2019,2020,2021,2022,2023,2024,2025\}

Rules:

\{constraint\_1\}

\{constraint\_2\}

\{constraint\_3\}

\.\.\.

Context Curation Prompt\.

YouareanexpertPolicyDrafter\.Yourgoalistotranslateadecisiontree\(providedinDOTformat\)intoaprofessional,naturallanguagepolicydocument\.

ReturnasingleJSONobject:

\{

"title":"Aprofessional,bureaucratictitleforthepolicy",

"final\_label":"Thespecificnameofthefinalstatus/award",

"text":"ThefullpolicytextinMarkdownformat\.\.\."

\}

I\.GenericGraphGrammar&NarrativeRules

1\.SerialTopology\(TheFilteringPipeline\)

\*Structure:‘NodeA\-\>NodeB‘\(Directarrowconnection\)\.

\*Interpretation:Astrictdependency\.NodeBisnotanindependentstep;itoperates

onlyonthesubsetofentitiesthathavesuccessfullypassedNodeA\.

\*NarrativeRequirement:YoumustexplicitlysignalInheritedEligibilityatthestart

ofthesecondnode’sdescription\.Clarifythatthesubsequentcriteriaapply

exclusivelytothespecificpoolofcandidatesthatsurvivedthepreviousfilter\.

2\.ConvergentTopology\(CombinatorialLogic\)

\*Structure:Multiplearrowsconvergingintoasingle‘COMBO‘node\.

\*EdgeOrderRule:Theorderofincomingedgesinthe‘dot\_string‘definesthe

operandroles\(CrucialforSubtraction\):

\-1stEdge:LeftOperand\(InputA\)\.

\-2ndEdge:RightOperand\(InputB\)\.

3\.LogicDefinitions&NarrativeRequirements

\*AND\(Intersection\)

\-Meaning:Simultaneoussatisfactionofallinputs\.

\-Req:Describeasaunified,mandatorystandardwherenocriterionisoptional\.

\*OR\(Union\)

\-Meaning:Anyinputsuffices\.

\-Req:Describeasalternativepathwaysorflexiblequalificationroutes\.

II\.NARRATIVEARCHITECTURE\(Randomlyassigned\)

Youmustfollowthespecificstructuredirectedintheuserprompt:

1\.STRUCTURE\_A\(Conclusion\-Led\):Statethe\{final\_label\}inthefirstparagraphas

thepolicy’sprimarygoal\.Then,detailtheprerequisiterequirements\.

2\.STRUCTURE\_B\(Criteria\-Led\):Detailtherequirementsphase\-by\-phase,introducing

the\{final\_label\}onlyintheconcludingsummary\.

III\.WRITINGGUIDELINES\(FORMULA\-TO\-POLICYTRANSLATION\)

\-STORYTELLER,NOTCALCULATOR:Donotsimplylistdatapoints\.Writethepolicyas

acohesivenarrativeaboutthe\{entity\_type\}’sperformancehistoryandcompliance

expectations\.

\-FORMULAISTRUTH:Theuserprovideslogicnodesinmathematicalnotation

\(e\.g\.,A/B\>=50%\)\.Thesearetheabsolutegroundtruth\.Yourjobistotranslate

thesemathematicalrelationshipsintoprofessionalpolicyprose\.

\-DE\-MECHANIZATION:Donotsimplyreadtheformulaaloud\.

\*BAD:"ThevalueofXdividedbythesumofYmustbegreaterthan0\.5\."

\*BAD:Ifanode’sformulaterequirescomparingwithotherentities,DONOTWRITE:

"calculatedacrossallentitiesinthedataset"\.USE"allparticipating

\{entity\_type\}"instead\.

\-PROFESSIONALPROSE:Youhavefullcreativefreedomtorephrasethetechnicallabels

fromtheDOTnodesintosmooth,authoritativebureaucraticlanguage\.

\-TERMINOLOGY:Replacegeneric"entity"withthespecific‘entity\_type‘provided

\(e\.g\.,"contract","farm"\)\.

\-HANDLINGEXTREMEVALUES\(MAX/MINConstraints\):

\*Context:Ifanoderulesays"value==themaxvalue"or"value==theminvalue"\.

\*RequirementA\(CompetitiveRanking\):YouMUSTinterpretthisasaranking\.

Explicitlystatethattheentityrecordedthehighest\(orlowest\)absolutefigure

inthecomparativeset\.Usekeywordslike"Unrivaled,""Market\-leading,"

"Highestrecordedvalue\."

\*RequirementB\(ScopeAbstraction\-CRITICAL\):Iftherulelistsspecificentity

names\(e\.g\.,"amongEntityA,EntityB"\),DONOTtranscribethesenames\.

\-Youmustinterpretthislistas"thespecificgroupof\{entity\_type\}sthat

qualifiedthroughthepreviousstep"\.

\-Phrasing:Insteadoflistingnames,usephraseslike"amongtheremaining

eligiblecandidates,""withinthisrefinedselection,"or"relativetothe

qualifyingpeergroup\."

IV\.AGENCYIDENTITY

\-Inventaplausiblegoverningauthoritybasedonthecontext

\(e\.g\.,"BoardofAgriculturalStandards"\)\.

## Appendix BExamples of Symbolic Reasoning Traces

To illustrate the diversity of reasoning structures inDocHop, we provide additional examples of the sampled symbolic reasoning traces in this appendix\. Each trace is represented as a logical tree, where nodes correspond to parameterized constraints instantiated from rule templates, and edges propagate the valid entity set through successive filtering steps\. In addition to simple linear chains, the generation process produces multi\-branch compositions with Boolean operators \(AND/OR\), resulting in varied multi\-hop reasoning patterns across documents\. These traces serve as the underlying blueprint that is later verbalized into document narratives and grounded through the semantic reference label\.

Figure[B\.1](https://arxiv.org/html/2609.02059#A2.F1)shows representative reasoning traces with different depths and branching structures\.

![Refer to caption](https://arxiv.org/html/2609.02059v1/reasoning_trace.png)Figure B\.1:Examples of sampled symbolic reasoning traces\.Each trace forms a logical tree of instantiated constraints, ranging from linear multi\-hop chains to branched compositions withAND/ORoperators\.
## Appendix CAtomic Constraint Rule Templates

#### Atomic Constraint Templates\.

To construct diverse multi\-hop reasoning traces, we define a library of atomic constraint templates \(“atoms”\) that operate over different chart structures\. Each atom specifies a local, parameterized condition grounded in a single chart, such as threshold checks, within\-entity comparisons, temporal trends, or extremal selection\. During trace generation, these templates are instantiated by sampling concrete timestamps, metrics, groups, operators, and numeric thresholds from the chart schema, yielding fully specified symbolic constraints\.

Importantly, we distinguish between*non\-unique*atoms and*unique*atoms\. Non\-unique constraints may be satisfied by multiple entities simultaneously \(e\.g\., “Revenue≥\\geq70”\), enabling gradual filtering across hops\. Unique constraints instead enforce a deterministic extremal selection \(e\.g\., “the entity with the highest value”\), ensuring that the reasoning trace can resolve to a well\-defined target set\. Tables[C\.1](https://arxiv.org/html/2609.02059#A3.T1)–[C\.3](https://arxiv.org/html/2609.02059#A3.T3)summarize all atomic templates used in DocHop\.

Table C\.1:Atomic constraint templates for line charts\.These rules impose temporal thresholds, trends, and extremal conditions over time\-series values\.Table C\.2:Atomic constraint templates for multi\-metric bar charts\.These rules capture within\-entity metric comparisons and cross\-entity extrema\.Table C\.3:Atomic constraint templates for group bar charts\.These rules impose within\-entity group structure and cross\-entity extremal selection\.

## Appendix DQuestion Templates

Given a synthesized document instance, we construct the final evaluation question by instantiating a task\-specific template\. Each template defines a canonical chart question\-answering format \(e\.g\., retrieval, counting, aggregation, ranking, hypothetical updates, or fact checking\), while leaving key slots—such as the target metric, timestamp, aggregation operator, or additional condition—to be filled from the instance metadata\.

Crucially, all templates are grounded on the*semantic reference label*introduced at the end of the document narrative, rather than directly naming entities from the charts\. As a result, questions do not explicitly specify which rows in the chart tables should be queried\. Instead, models must first resolve the labeled entity set implied by the narrative constraints, and then retrieve or aggregate the corresponding numerical evidence from the charts\.

Across the six task categories, we design a total of 41 diverse templates\. These templates cover both direct queries over the labeled entities \(e\.g\., retrieving a metric value or counting the number of qualifying entities\) and more compositional variants that introduce additional chart\-grounded conditions or counterfactual modifications\. This template\-based construction enables systematic control over question forms while ensuring that all answers can be computed exactly from the underlying chart tables\. The complete template list, grouped by task type, is provided in the following pages\.

Template IDTaskQuestion PatternTask 1: Value Retrieval1\.1Value QueryWhat is the \{metric\_desc\} of the \{entity\_type\} receiving the \{semantic\_label\}?1\.2Entity QueryWhich \{entity\_type\} receives the \{semantic\_label\}?Task 2: Counting2\.1Count AllHow many \{entity\_type\}\(s\) receive the \{semantic\_label\}?2\.2Count w/ ThresholdAmong the \{entity\_type\}\(s\) receiving the \{semantic\_label\}, how many have \{metric\_desc\} \{comparison\} \{threshold\}?Task 3: Numeric Reasoning3\.1SumWhat is the total \{metric\_desc\} of the \{entity\_type\}\(s\) receiving the \{semantic\_label\}?3\.2RangeWhat is the difference between the highest and lowest \{metric\_desc\} among the \{entity\_type\}\(s\) receiving the \{semantic\_label\}?3\.3MaxWhat is the highest \{metric\_desc\} among the \{entity\_type\}\(s\) receiving the \{semantic\_label\}?3\.4MinWhat is the lowest \{metric\_desc\} among the \{entity\_type\}\(s\) receiving the \{semantic\_label\}?3\.5AverageWhat is the average \{metric\_desc\} of the \{entity\_type\}\(s\) receiving the \{semantic\_label\}?3\.6Time Range SumWhat is the total \{metric\} from \{start\_time\} to \{end\_time\} of all the \{entity\_type\}s receiving the \{semantic\_label\}?3\.7Time DiffWhat is the difference in the total \{metric\} between \{time1\} and \{time2\} of all the \{entity\_type\}s receiving the \{semantic\_label\}?3\.8Multi\-Metric SumWhat is the total of \{metric1\} and \{metric2\} in \{time\} for all the \{entity\_type\}s receiving the \{semantic\_label\}?3\.9Multi\-Metric DiffWhat is the difference between the total \{metric1\} and total \{metric2\} in \{time\} of all the \{entity\_type\}s receiving the \{semantic\_label\}?3\.10Group SumWhat is the combined total \{metric\} across \{group1\} and \{group2\} in \{time\} for all the \{entity\_type\}s receiving the \{semantic\_label\}?3\.11Group DiffWhat is the difference in total \{metric\} between \{group1\} and \{group2\} in \{time\} of all the \{entity\_type\}s receiving the \{semantic\_label\}?Task 4: Ranking4\.1ArgmaxAmong the \{entity\_type\}\(s\) receiving the \{semantic\_label\}, which one has the highest \{metric\_desc\}?4\.2ArgminAmong the \{entity\_type\}\(s\) receiving the \{semantic\_label\}, which one has the lowest \{metric\_desc\}?4\.3K\-th BestAmong the \{entity\_type\}\(s\) receiving the \{semantic\_label\}, which one has the \{k\-th\} highest \{metric\_desc\}?Table D\.1:Question Templates for Chart\-based QATable D\.2:Question Templates for Chart\-based QA \(Continued \- Hypothetical\)Template IDSubtypeQuestion PatternTask 6: Fact Checking \(Yes/No\)6\.1 \- Entity Verification6\.1\.1Entity MembershipIs \{entity\} listed as a recipient of the \{semantic\_label\}? \(Yes/No\)6\.1\.2Value MatchDid the \{entity\_type\} receiving the \{semantic\_label\} record a \{metric\_desc\} of \{value\}? \(Yes/No\)6\.2 \- Count Verification6\.2\.1Exact CountAre there exactly \{count\} \{entity\_type\}\(s\) receiving the \{semantic\_label\}? \(Yes/No\)6\.3 \- Numeric Verification6\.3\.1Sum CheckIs the total \{metric\_desc\} of the \{entity\_type\}\(s\) receiving the \{semantic\_label\} equal to \{value\}? \(Yes/No\)6\.3\.2Range CheckIs the difference between the highest and lowest \{metric\_desc\} among the \{entity\_type\}\(s\) receiving the \{semantic\_label\} equal to \{value\}? \(Yes/No\)6\.3\.3Max CheckIs the highest \{metric\_desc\} among the \{entity\_type\}\(s\) receiving the \{semantic\_label\} equal to \{value\}? \(Yes/No\)6\.3\.4Min CheckIs the lowest \{metric\_desc\} among the \{entity\_type\}\(s\) receiving the \{semantic\_label\} equal to \{value\}? \(Yes/No\)6\.3\.5Average CheckIs the average \{metric\_desc\} of the \{entity\_type\}\(s\) receiving the \{semantic\_label\} equal to \{value\}? \(Yes/No\)6\.3\.6Time Range Sum CheckIs the total \{metric\} from \{start\_time\} to \{end\_time\} for all the \{entity\_type\}s receiving the \{semantic\_label\} equal to \{value\}? \(Yes/No\)6\.3\.7Time Diff CheckIs the difference in the total \{metric\} between \{time1\} and \{time2\} for all the \{entity\_type\}s receiving the \{semantic\_label\} equal to \{value\}? \(Yes/No\)6\.3\.8Multi\-Metric Sum CheckIs the total of \{metric1\} and \{metric2\} in \{time\} for all the \{entity\_type\}s receiving the \{semantic\_label\} equal to \{value\}? \(Yes/No\)6\.3\.9Multi\-Metric Diff CheckIs the difference between the total \{metric1\} and total \{metric2\} in \{time\} of all the \{entity\_type\}s receiving the \{semantic\_label\} equal to \{value\}? \(Yes/No\)6\.3\.10Group Sum CheckIs the combined total \{metric\} across \{group1\} and \{group2\} in \{time\} for all the \{entity\_type\}s receiving the \{semantic\_label\} equal to \{value\}? \(Yes/No\)6\.3\.11Group Diff CheckIs the difference in total \{metric\} between \{group1\} and \{group2\} in \{time\} of all the \{entity\_type\}s receiving the \{semantic\_label\} equal to \{value\}? \(Yes/No\)6\.4 \- Ranking Verification6\.4\.1Argmax CheckDoes \{entity\} have the highest \{metric\_desc\} among the \{entity\_type\}\(s\) receiving the \{semantic\_label\}? \(Yes/No\)6\.4\.2Argmin CheckDoes \{entity\} have the lowest \{metric\_desc\} among the \{entity\_type\}\(s\) receiving the \{semantic\_label\}? \(Yes/No\)6\.4\.3K\-th Best CheckDoes \{entity\} have the \{k\-th\} highest \{metric\_desc\} among the \{entity\_type\}\(s\) receiving the \{semantic\_label\}? \(Yes/No\)Table D\.3:Question Templates for Chart\-based QA \(Continued \- Fact Checking\)
## Appendix EEvaluation Details

We evaluate model performance using normalized answer accuracy\. To reduce formatting variance, we append a shared instruction block to every query, requiring models to return the final answer in a dedicated sentence of the form‘‘The answer is: <\.\.\.\>’’and to omit units, currency symbols, and percentage signs\. When the question specifies a rounding scheme, models are instructed to follow it strictly; otherwise, integer answers are expected as integers, and non\-integer answers are rounded to two decimal places\.

Prompting Setup \(System Instruction\)\.All models are evaluated in a single\-image VQA setting where the input is a document\-page image containing both narrative text and one or more charts\. We use a unified system\-style instruction \(SHARED\_INSTRUCTION\) appended to the question turn, which enforces \(i\) a single final answer span, \(ii\) a canonical numeric formatting policy, and \(iii\) a fixed output marker \(The answer is:\) to facilitate robust parsing\. The instruction begins with a short preamble describing the input format \(e\.g\., “read the text within the image and analyze the chart\(s\)”\), followed by the shared answer\-format constraint described above\.

Answer Parsing and Normalization\.Given a model response, we extract the predicted answer by searching for the final occurrence of the patternthe answer is:\(case\-insensitive\) and taking the subsequent span; if the marker is missing, we fall back to the last non\-empty line\. We further normalize extracted strings by stripping leadinganswer:prefixes, whitespace, and lightweight formatting tokens \(e\.g\.,\*, backticks\)\.

Correctness Criteria\.We apply the same two\-step procedure to every question, regardless of task type\. We first check for a case\-insensitive exact match between the normalized prediction and ground truth \(e\.g\., for Yes/No fact\-checking answers or exact entity names\)\. If this fails, we fall back to a numeric comparison: we extract the last numeric token from both prediction and ground truth \(allowing an optional minus sign and decimals, and ignoring comma separators\) and mark the prediction correct if the two values are close undermath\.isclosewith relative tolerance0\.010\.01and absolute tolerance10−310^\{\-3\}\.

```
The image provided is a document page containing both text and chart(s).
Please read the text within the image and analyze the chart(s) to answer
the question.

Your final answer should be a single pure value:
- If the question asks for an entity, use its complete name exactly as shown
  in the chart
- If the answer is a number: output integers as integers, decimals rounded to
  two places
  (unless the question specifies otherwise)
- Otherwise, follow the question’s instructions for the expected format

Do not include units, currency symbols, or percentage signs in your final answer.

At the end of your response, format the final answer in a separate sentence
like this:
The answer is: <your_final_answer>
```

## Appendix FModel Configuration

Model NameCategoryHugging Face Checkpoint / APIReasoning ModelsGPT\-5\.2\-ReasoningProprietarygpt\-5\.2\-2025\-12\-11GPT\-5\-mini\-ReasoningProprietarygpt\-5\-mini\-2025\-08\-07Gemini\-2\.5\-Pro\-ReasoningProprietarygemini\-2\.5\-proGemini\-2\.5\-Flash\-ReasoningProprietarygemini\-2\.5\-flashProprietary ModelsGPT\-5\.2Proprietarygpt\-5\.2\-2025\-12\-11Gemini\-2\.5\-FlashProprietarygemini\-2\.5\-flashClaude\-4\.5\-HaikuProprietaryclaude\-haiku\-4\-5\-20251001Claude\-4\.5\-SonnetProprietaryclaude\-sonnet\-4\-5\-20250929Open\-Source ModelsLLaVA\-Next\-LLaMA3\-8BOpen\-Sourcelmms\-lab/llama3\-llava\-next\-8bIDEFICS2\-8BOpen\-SourceHuggingFaceM4/idefics2\-8bIDEFICS3\-LLaMA3\-8BOpen\-SourceHuggingFaceM4/Idefics3\-8B\-Llama3Qwen\-2\.5\-VL\-7BOpen\-SourceQwen/Qwen2\.5\-VL\-7B\-InstructQwen\-2\.5\-VL\-32BOpen\-SourceQwen/Qwen2\.5\-VL\-32B\-InstructQwen\-2\.5\-VL\-72BOpen\-SourceQwen/Qwen2\.5\-VL\-72B\-InstructQwen3\-VL\-8BOpen\-SourceQwen/Qwen3\-VL\-8B\-InstructMolmo\-7B\-O\-0924Open\-Sourceallenai/Molmo\-7B\-O\-0924Molmo\-7B\-D\-0924Open\-Sourceallenai/Molmo\-7B\-D\-0924InternVL\-3\.5\-8BOpen\-SourceOpenGVLab/InternVL3\_5\-8BInternVL\-3\.5\-30B\-A3BOpen\-SourceOpenGVLab/InternVL3\_5\-30B\-A3BInternVL\-3\.5\-38BOpen\-SourceOpenGVLab/InternVL3\_5\-38BOvis1\.6\-Gemma2\-9BOpen\-SourceAIDC\-AI/Ovis1\.6\-Gemma2\-9BTable F\.1:Model List and CheckpointsWe report the evaluation configurations of all models considered in our experiments\. For proprietary APIs, we specify the exact model versions used at evaluation time\. For open\-sourced baselines, we list the corresponding HuggingFace checkpoints\.

For Qwen\-2\.5\-VL and Qwen3\-VL models, we follow the recommended visual tokenization settings, using a minimum pixel resolution of1280×28×281280\\times 28\\times 28and a maximum resolution of16384×28×2816384\\times 28\\times 28\.

Since DocHop is reasoning\-intensive and often requires multi\-step aggregation, we set the maximum generation length for all models to2142^\{14\}tokens whenever supported, avoiding premature truncation due to insufficient output budgets\.