XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding

# XL-DocBench: Benchmarking Evidence-Grounded Extra-Long Document Understanding
Source: [https://arxiv.org/html/2608.00036](https://arxiv.org/html/2608.00036)
Hongchen Wei1,†,‡Yuanzhe Wang2,†,‡Bei Liu2,∗Yifan Yang2 Qi Dai2Ruichun Ma2Kai Qiu2Yunsheng Li2Dongdong Chen2 Chong Luo2Zhenzhong Chen1Baining Guo2 1Wuhan University2Microsoft

###### Abstract

Real\-world document tasks often ask professionals to answer questions from annual reports, regulations, clinical guidelines, and technical manuals that span hundreds or thousands of pages\. Some questions also require comparing related reports\. Reliable long\-document understanding is therefore a prerequisite for using LLMs in compliance, clinical, financial, and engineering workflows, where decisions must be traceable to specific evidence pages and the cost of an unsupported answer is high – yet most existing benchmarks still measure short\-context or single\-page QA\. We introduceXL\-DocBench, a fully human\-verified benchmark for extra\-long document understanding, with 1,519 retained questions from six professional domains and contexts up to 2,303 pages\. XL\-DocBench goes beyond page\-level lookup\. 1,103 examples \(72\.6%\) use multiple evidence pages\. The final set also includes 556 questions \(36\.6%\) that use tables, charts, or figures, and 165 questions \(10\.9%\) that require evidence from multiple documents\. Each question has one of twelve reasoning labels, expert\-annotated evidence pages, a typed verification rule, and an answer format, including 218None\-answer cases\. We build the benchmark with a tree\-guided synthesis pipeline followed by artifact filters and full verification by 194 human experts\. By coupling extra\-long professional contexts with page\-level evidence and typed rules, XL\-DocBench fills a gap left by prior single\-page, short multi\-page, or text\-only long\-context benchmarks, and lets future work attribute system failures to retrieval, evidence use, or rule following rather than to a single leaderboard score\. The results show that current systems still struggle with long contexts, multi\-page evidence, and structured reasoning over professional documents\. Project homepage:[https://officeintelligence.github.io/xl-docbench/](https://officeintelligence.github.io/xl-docbench/)\.

††footnotetext:†Equal contribution\.‡Work done during an internship at MSRA\.∗Project leader\.## 1Introduction

Extra\-long professional documents are common in real document work\. Annual reports, regulations, clinical guidelines, standards, and technical manuals often span hundreds or thousands of pages, and some questions require comparing related files\. In practice, users rarely ask for isolated snippets\. They ask whether a condition is satisfied, which entities should be included in a set, how values change across reports, or whether the documents provide enough support to answer at all\. These questions are common in professional workflows because decisions often depend on policies, exceptions, definitions, footnotes, and repeated measurements that are scattered across long files\. Answering such questions is not a single\-page lookup\. A system must find the right parts of the document, read text together with tables and charts, combine evidence from different places, and decide when the documents do not support an answer\[[8](https://arxiv.org/html/2608.00036#bib.bib1)\]\.

Table 1:Comparison between our benchmark and previous DU datasets\.Unans\.: unanswerable question\.TXT/L/C/TAB/I: pure text/generalized layout/chart/table/image\.Cross\-doc\.: whether answering requires aggregating evidence across multiple documents\.Avg\. Evidence Pages: average number of evidence pages needed to answer the question\.N/A \(text\-only\): text\-only benchmark without native page concept\.As shown in Table[1](https://arxiv.org/html/2608.00036#S1.T1.36), existing document\-understanding benchmarks cover parts of this setting: layout parsing, single\-page QA, short multi\-page reasoning, and text\-only long\-context tasks\. These datasets have helped the field, but they often do not require a model to justify an answer with exact evidence pages across very long professional files\. They also leave open how systems behave when the correct response is to abstain because the needed support is missing\. A model can therefore score well on average while still failing at the step users need most: finding the right pages and using them under an explicit rule\. A useful benchmark should use realistic document lengths and labels tied to evidence pages\. It should also include multimodal cases, cross\-document cases, and cases where support is missing, while filtering out questions answerable without the documents\.

We introduce XL\-DocBench, a fully human\-verified benchmark for extra\-long document understanding\. It contains 1,519 retained questions over long professional documents from six domains\. The median document context is 211 pages, and the longest context spans 2,303 pages\. XL\-DocBench includes both local and multi\-page evidence\. 1,103 examples \(72\.6%\) use multiple evidence pages\. Another 556 questions \(36\.6%\) involve tables, charts, or figures, 165 questions \(10\.9%\) are cross\-document, and 218 examples useNoneas the answer format\. Questions are grouped into twelve reasoning types across three tiers, so we can see which reasoning steps fail instead of relying only on aggregate accuracy\. These labels help separate errors in finding evidence, tracking sets, comparing values, aggregating information, and recognizing missing support\.

XL\-DocBench is built with a tree\-guided synthesis pipeline followed by artifact filters and full verification by 194 human experts\. The automatic checks operate at the chapter or section level; experts then locate the exact supporting pages and quotes, verify the answer, and repair or remove ambiguous items\. This design separates generation at scale from final evidence checking: the pipeline proposes difficult candidates, but the released labels come from human review over the full documents\. Each retained example includes expert\-annotated evidence pages, a typed verification rule, an answer format, and checks for support, completeness, and ambiguity\. Evaluations of proprietary VLMs, open\-weight VLMs, and retrieval\-augmented agent systems show that current systems still make many errors on this task\.

Our main contributions are:

- •We present XL\-DocBench, a fully human\-verified benchmark for extra\-long document understanding: 1,519 retained questions, six professional domains, twelve reasoning types, and diagnostic multimodal,None\-answer, and cross\-document slices\.
- •We introduce a tree\-guided construction protocol with chapter/section\-level path\-dependency checks, typed verification rules, artifact filters, and page\-level evidence annotation by 194 human experts\.
- •We provide evaluation results and breakdowns by document length, evidence span, reasoning type, document type, modality, and cross\-document scope\.

## 2Related Work

Document Understanding Models and Methods\.Document understanding methods have moved from layout\-aware text models to multimodal readers and retrieval\-augmented agents\. LayoutLM\-style models combine OCR with two\-dimensional layout embeddings\[[34](https://arxiv.org/html/2608.00036#bib.bib33),[33](https://arxiv.org/html/2608.00036#bib.bib34),[13](https://arxiv.org/html/2608.00036#bib.bib35),[30](https://arxiv.org/html/2608.00036#bib.bib36),[3](https://arxiv.org/html/2608.00036#bib.bib37)\], while end\-to\-end document readers such as Donut\[[15](https://arxiv.org/html/2608.00036#bib.bib38)\]remove explicit OCR by decoding from page images\. Document\-specialized and general\-purpose VLMs now support higher resolution and longer context, including mPLUG\-DocOwl\[[35](https://arxiv.org/html/2608.00036#bib.bib39),[12](https://arxiv.org/html/2608.00036#bib.bib40)\], TextMonkey\[[18](https://arxiv.org/html/2608.00036#bib.bib41)\], Gemini\[[10](https://arxiv.org/html/2608.00036#bib.bib26)\], Qwen3\.5\[[24](https://arxiv.org/html/2608.00036#bib.bib27)\], InternVL3\.5\[[31](https://arxiv.org/html/2608.00036#bib.bib42)\], and Pixtral\[[1](https://arxiv.org/html/2608.00036#bib.bib43)\]\. For hundreds\-page documents, however, single\-pass reading often fails or exceeds the model context window\. Recent methods use visual page retrieval\[[9](https://arxiv.org/html/2608.00036#bib.bib44),[19](https://arxiv.org/html/2608.00036#bib.bib45),[36](https://arxiv.org/html/2608.00036#bib.bib46)\], document RAG systems\[[6](https://arxiv.org/html/2608.00036#bib.bib47),[26](https://arxiv.org/html/2608.00036#bib.bib48)\], and agents that iteratively search, read, and accumulate evidence\[[11](https://arxiv.org/html/2608.00036#bib.bib31),[14](https://arxiv.org/html/2608.00036#bib.bib30),[16](https://arxiv.org/html/2608.00036#bib.bib32)\]\. These systems need evaluation on navigation and evidence use across long documents, not just on page reading\.

Document Understanding Benchmarks\.Benchmarks have also moved from single\-page visual QA to multi\-page and long\-context tasks\. DocVQA\[[23](https://arxiv.org/html/2608.00036#bib.bib6)\], ChartQA\[[21](https://arxiv.org/html/2608.00036#bib.bib7)\], InfographicsVQA\[[22](https://arxiv.org/html/2608.00036#bib.bib8)\], and TAT\-QA\[[38](https://arxiv.org/html/2608.00036#bib.bib9)\]stress page\-level visual, chart, infographic, and table\-text reasoning; MP\-DocVQA\[[28](https://arxiv.org/html/2608.00036#bib.bib10)\], DUDE\[[29](https://arxiv.org/html/2608.00036#bib.bib13)\], and SlideVQA\[[27](https://arxiv.org/html/2608.00036#bib.bib14)\]extend to short multi\-page inputs\. Recent long\-document benchmarks such as MMLongBench\-Doc\[[20](https://arxiv.org/html/2608.00036#bib.bib15)\], LongDocURL\[[7](https://arxiv.org/html/2608.00036#bib.bib12)\], DocBench\[[40](https://arxiv.org/html/2608.00036#bib.bib17)\], FinRAGBench\-V\[[37](https://arxiv.org/html/2608.00036#bib.bib20)\], M\-LongDoc\[[5](https://arxiv.org/html/2608.00036#bib.bib19)\], MMDocBench\[[39](https://arxiv.org/html/2608.00036#bib.bib18)\], and DocGenome\[[32](https://arxiv.org/html/2608.00036#bib.bib2)\]increase length, domain specialization, or document formats\. Text\-only long\-context benchmarks\[[25](https://arxiv.org/html/2608.00036#bib.bib4),[4](https://arxiv.org/html/2608.00036#bib.bib3)\]test long\-range reasoning without page\-level visual structure\. What is still missing is a benchmark with realistic extra\-long professional documents, evidence spread across pages or related documents, fine\-grained reasoning labels, and construction checks that remove no\-context or weakly supported questions\. XL\-DocBench provides this combination with contexts up to 2,303 pages, 1,103 multi\-page human\-evidenced examples \(72\.6% of the final set\), a cross\-document subset, twelve reasoning types, and full verification by 194 human experts\.

## 3XL\-DocBench

This section describes how we build XL\-DocBench from long professional documents\. The construction pipeline creates questions that need evidence from more than one local snippet while keeping each answer verifiable\. We then summarize the dataset composition, reasoning labels, and validation protocol\.

### 3\.1Tree\-Guided Coarse\-to\-Fine Construction

Building XL\-DocBench cannot be done by simply asking annotators or models to write questions over long PDFs\. Manual annotation alone is hard to scale because experts must inspect hundreds or thousands of pages, understand domain terminology, find the relevant evidence, and check that the answer is well supported\. Fully automatic generation has the opposite problem: it often produces local lookup questions, vague document\-level questions, or answers without clear evidence\. We use*human\-guided, model\-assisted construction*\. Models propose candidates from selected document branches, automatic filters remove common errors, and human experts verify every retained example\. The pipeline is a data construction protocol, not a model architecture\. It makes question generation easier to inspect and scale\.

![Refer to caption](https://arxiv.org/html/2608.00036v1/figures/benchmark_pipeline.png)Figure 1:XL\-DocBench construction pipeline\.Long documents and document series are parsed into hierarchical trees\. The pipeline generates candidate questions from multiple chapter/section branches, refines them with concrete content, adds candidate answers and verification rules, and filters artifacts\. Full verification by 194 human experts identifies the final supporting pages and quotes, checks answer support and ambiguity, and removes or repairs invalid items\.The pipeline starts with document structure and ends with human\-verified examples\. Each PDF is parsed into page images, Markdown, tables, and figure metadata, then organized into a document tree\. We summarize tree nodes and sample chapter or section branches under domain and reasoning\-type quotas\. We then generate candidates from the branch summaries, test branch\-level path\-dependency, refine the questions with concrete content, add provisional answers and verification rules, filter artifacts, and ask human experts to annotate the exact supporting pages and quotes\.

Tree Construction\.Inspired by PageIndex111[https://github\.com/VectifyAI/PageIndex](https://github.com/VectifyAI/PageIndex), we use a tree so the pipeline can move from broad sections to specific pages without prompting over the whole document at once\. Each PDF is represented as a hierarchy: internal nodes contain section summaries, and leaves correspond to pages\. When headings are unreliable, we group pages using formatting cues\. For document series, we attach per\-document trees to a shared root so cross\-document candidates can be generated through the same interface\.

Exploration with Path\-Dependency\.A tree alone does not guarantee that a question needs evidence from multiple places\. A question can look broad but still be answerable from one local excerpt\. We start at the coarsest levelℓ=0\\ell\{=\}0and check path\-dependency over chapter\- or section\-level branches\. Given selected branches𝒰\\mathcal\{U\}, the model proposes a candidateqqthat should need information from each branch\. We use a leave\-one\-branch\-out criterion:

∀u∈𝒰,q​is not answerable from​𝒞​\(𝒰∖\{u\}\)\.\\forall u\\in\\mathcal\{U\},\\quad q\\text\{ is not answerable from \}\\mathcal\{C\}\(\\mathcal\{U\}\\setminus\\\{u\\\}\)\.\(1\)This test does not prove that the final evidence set is minimal, and it does not identify the final evidence pages\. It only filters out candidates that can be answered from one selected chapter or section\. An LLM judge applies the check to each leave\-one\-branch\-out projection, and we keep candidates only when the missing branch removes necessary evidence\. Human experts later annotate the final page\-level evidence\. We also use quotas over twelve reasoning types so rare types such as counterfactual reasoning and reference chains are not lost during generation\.

Iterative Refinement\.Candidates generated from summaries often involve the right branches, but they are too abstract to verify directly\. We expand selected branches one level at a time and rewrite the candidate with newly visible content:

q\(ℓ\+1\)=ℛℓ​\(q\(ℓ\),\{ch​\(u\):u∈𝒰\(ℓ\)\}\)\.q^\{\(\\ell\+1\)\}=\\mathcal\{R\}\_\{\\ell\}\\\!\\left\(q^\{\(\\ell\)\},\\\{\\mathrm\{ch\}\(u\):u\\in\\mathcal\{U\}^\{\(\\ell\)\}\\\}\\right\)\.\(2\)where𝒰\(ℓ\)\\mathcal\{U\}^\{\(\\ell\)\}is the current support frontier andch​\(u\)\\mathrm\{ch\}\(u\)denotes its children\. After each step, a judge scores answerability, ambiguity, and reasoning depth\. We remove low\-quality candidates, keep strong candidates early, and use a novelty penalty to avoid repeating the same support combinations\.

Evidence Grounding\.Refinement turns broad ideas into concrete questions, but the answer still needs to be checkable\. The automated stage proposes a self\-contained questionq⋆q^\{\\star\}, a candidate answera⋆a^\{\\star\}, candidate support, and a typed verification rule; these remain provisional until expert verification\. Reasoning\-type prompts handle cases such as counterfactual recalculation\. A rewriting step then removes internal tree references\. Human experts then inspect the full source documents to mark the concrete evidence pages and supporting quotes that make the answer verifiable; this page\-level annotation is the released ground truth\.

Quality Filtering\.Even concrete examples can be invalid\. Some are answerable from world knowledge, rely on unsupported or ambiguous evidence, use meaningless arithmetic, or contain internal tree references\. We filter these cases before and after expert verification\. The final released subset retains 1,519 of the 3,550 synthetic candidates, and the no\-context filter is the largest single rejection source\. The no\-context filter presents each candidate question to a judge without any document content; candidates are rejected if the answer can be inferred from world knowledge, metadata, or wording leakage alone\. Filtering is not the final quality guarantee: every retained sample is verified by human experts for answer support, page\-level evidence completeness, and ambiguity before release\. Additional filter details are provided in Appendix[A](https://arxiv.org/html/2608.00036#A1)\.

### 3\.2Dataset Composition and Analysis

The pipeline in §[3\.1](https://arxiv.org/html/2608.00036#S3.SS1)produces XL\-DocBench through several filtering stages\. It starts with 3,550 synthetic candidates\. Of these, 2,104 pass expert verification, and 1,519 remain in the final fully human\-verified benchmark after additional difficulty\-oriented filtering\. Table[2](https://arxiv.org/html/2608.00036#S3.T2)summarizes the final retained set\. These statistics define the main test conditions: long professional contexts, expert\-annotated evidence, fine\-grained reasoning labels, and subsets for multimodal,None\-answer, long\-span, and cross\-document questions\.

Table 2:Dataset statistics\. Funnel rows report annotation and filtering counts; other rows describe the final evaluation subset\. Domain order: legal/finance/technical/medical/scientific/narrative; answer\-format order: string/integer/float/none\. Evidence\-page rows distinguish old synthetic candidate pages from human\-annotated evidence pages\.StatisticValueStatisticValue*Construction funnel*All synthetic data3,550Human\-verified candidates2,104 \(59\.3%\)Final filtered subset1,519Candidate\-to\-final retention72\.2%*Final subset composition*Documents331Pages \(total/avg/med/max\)98,342/297\.1/211/2,062Mean document tokens227,463\.2Domains / reasoning types6/12Domain counts311/333/274/294/218/89Cross\-doc Qs165 \(10\.9%\)Multimodal evidence556 \(36\.6%\)Noneanswer218 \(14\.4%\)Answer formats945/238/118/218Human experts194*Final evidence\-page statistics*Old synth\. evidence\. pages \(avg/max\)9\.13/45Human\-annotated evidence\. pages \(avg/max\)2\.30/24Multi\-page human evidence\.1,103 \(76\.4%\)Human evidence\. span \(med/max\)7/1,002

Corpus coverage\.The retained benchmark covers six professional domains and long document contexts from authoritative public sources\. The median context is 211 pages and the mean is 297\.1 pages; 51 examples exceed 1,000 pages\. Cross\-document or series examples can span up to 2,303 pages, so they directly test retrieval across related documents\. Table[2](https://arxiv.org/html/2608.00036#S3.T2)gives the source and domain counts\.

Diagnostic reasoning taxonomy\.For experiments, aggregate accuracy is not enough; we also need to know which reasoning steps fail\. XL\-DocBench assigns each question one of twelve mutually exclusive reasoning types, arranged into three tiers \(Figure[2](https://arxiv.org/html/2608.00036#S3.F2)\)\. The taxonomy covers comparison, reference chain, ranking, coverage, reconciliation, set difference, unanswerable, temporal, compliance \(including rule\-special\-case/exception checks\), counterfactual, aggregation, and consistency\. The quotas in §[3\.1](https://arxiv.org/html/2608.00036#S3.SS1)keep rare but important types instead of merging them into an “other” bucket\. Worked examples are given in Appendix[I](https://arxiv.org/html/2608.00036#A9)\.

![Refer to caption](https://arxiv.org/html/2608.00036v1/x1.png)Figure 2:Reasoning labels and evidence structure in the human\-verified set\.\(a\)Distribution of the twelve reasoning types, colored by tier\.\(b\)Mean number of expert\-annotated evidence pages and related PDFs per reasoning type\.Evidence grounding and span\.The final human annotations show that the benchmark mixes local cases with many questions that need evidence from distant pages\. 1,103 examples \(72\.6%\) require multiple evidence pages, while the remaining examples use a single annotated evidence page\. The median evidence span is 7 pages and the P95 span is 255 pages, so evidence localization over long contexts is a major part of the task\. Figure[3](https://arxiv.org/html/2608.00036#S3.F3)summarizes context length, evidence span, evidence modality, and answer format\.

![Refer to caption](https://arxiv.org/html/2608.00036v1/x2.png)Figure 3:Composition of XL\-DocBench in the final human\-verified set: \(a\) context length per example, summing PDFs for series questions; \(b\) evidence span from expert\-annotated evidence pages; \(c\) evidence page count, separating single\-page, within\-document multi\-page, and strict multi\-PDF cases; \(d\) evidence modality and answer format\.Diagnostic evaluation slices\.Beyond aggregate statistics, XL\-DocBench includes three subsets that match common professional document use: evidence modality, document scope, and answer support\. The multimodal subset has 556 questions \(36\.6%\) that require tables, charts, or figures, so systems cannot rely only on plain OCR text\. The cross\-document subset has 165 questions \(10\.9%\) that require evidence from related PDFs\. We note that the cross\-document label is defined by the*set of documents the question is grounded in*, not by the count of pages on which the final supporting evidence happens to fall: a question may depend on multiple PDFs through context, definitions, or cross\-references, yet its expert\-annotated evidence pages may concentrate in only a few documents\. The page\-level evidence statistics therefore underestimate the true cross\-document scope of these questions\. TheNone\-answer subset has 218 questions \(14\.4%\) and tests whether systems can abstain when the documents do not contain the required support\.

### 3\.3Benchmark Reliability and Validation

Because XL\-DocBench uses LLM\-generated candidates, reliability depends on human verification\. We treat automatic filtering as candidate selection, not final validation\. Of 3,550 synthetic candidates, 2,104 pass expert verification \(59\.3%\)\. We then remove lower\-difficulty items that are too easy after verification, leaving 1,519 examples in the final release subset\. Each retained example stores the question, final answer, answer format, reasoning type, expert\-annotated evidence pages, evidence quotes or sentence\-level snippets, and a typed verification rule\. Human verification is a main part of the benchmark, not a light audit\. In total, 194 human experts inspect retained candidates with full document access\. They re\-answer questions, mark the supporting pages and quotes, check whether the answer follows from the verification rule, flag ambiguity or missing support, and repair answer formats when the evidence is clear\. This step is necessary because the automatic path\-dependency check only works at the chapter or section level\. Experts must convert that coarse signal into reliable page\-level evidence\. This is a main reason the benchmark is difficult to reproduce by prompting alone\. Examples without clear evidence quotes and verification rules are removed\.

## 4Experiments

### 4\.1Evaluation Protocol

Input and pipeline settings\.For inference,*Img*denotes full\-document page\-image input, and*OCR*denotes per\-page OCR text capped at 80% of the model context window so the model still has room to generate a stable answer\. In both settings, the model receives the document representation and question, then returns a final answer without access to the expert\-annotated evidence pages\. Agent rows use PDF input; we use an LLM to extract the final answer from each agent response and then compute all metrics on the extracted answer\.

Models and metrics\.Completed runs cover GPT\-5\.2, GPT\-5\.4, Claude Opus 4\.6\[[2](https://arxiv.org/html/2608.00036#bib.bib25)\], Kimi\-K2\.5, DeepSeek\-V3\.2, Qwen3\.5 9B/35B\-A3B, and three RAG\-based pipelines: MDocAgent, SimpleDoc, and DeepRead\. Their nominal context windows are 1M tokens for GPT\-5\.4 and Claude Opus 4\.6, 256K for GPT\-5\.2, Kimi\-K2\.5, and all Qwen3\.5 variants, and 128K for DeepSeek\-V3\.2; with the 80% OCR cap, the usable OCR budgets are about 800K, 205K, and 102K tokens, respectively\. We report rule\-based accuracy under each typed verification rule as the primary metric, plus token F1 and ANLS as secondary surface\-form diagnostics\. Failed, missing, or unparsable predictions count as incorrect\.

### 4\.2Main Results by Reasoning Type

Table 3:Main results on the human\-verified XL\-DocBench subset\.Per\-reasoning\-type accuracy, grouped into the three tiers of the taxonomy, followed by aggregate metrics\.Img\(full document as page images\), orOCR\(per\-page OCR text capped at 80% of the model context window to reserve context for stable answer generation\); agent frameworks operate directly on PDFs\.T1 – Core:Cmp\.comparison,Ref\.reference chain\.T2 – Structural:Rnk\.ranking,Cov\.coverage,Rec\.reconciliation\.T3 – Advanced:SDiffset difference,Unansunanswerable,Tmptemporal,Cmplcompliance and rule\-special\-case checks,Ctfcounterfactual,Aggaggregation,Conconsistency\.Aggregate:Acc\.overall accuracy,F1token\-level F1,ANLSaverage normalised Levenshtein similarity\. Per\-columnbest\(green, bold\) andsecond\-best \(blue\) values are highlighted; ties are all shaded\.Table[3](https://arxiv.org/html/2608.00036#S4.T3)supports three observations\.First, OCR is the strongest one\-shot interface, but capacity alone is not the bottleneck\.OCR beats page\-image input on every completed pure\-model pair, with overall gains ranging from\+2\.5\+2\.5pts \(Qwen3\.5\-VL 9B: 19\.8→\\to22\.3\) to\+10\.4\+10\.4pts \(GPT\-5\.4: 26\.9→\\to37\.3\)\. The two 1M\-token models nonetheless differ by 2\.5 pts at equal usable OCR budget \(Claude Opus 4\.6 39\.8% vs\. GPT\-5\.4 37\.3%\), and 256K Kimi\-K2\.5 \(35\.8%\) remains close while 128K DeepSeek\-V3\.2 reaches only 29\.6% – models must still locate the right pages, suppress irrelevant text, and apply the typed rule\.Second, the hardest categories require tracking sets of evidence\.Best ranking, coverage, and set\-difference accuracies stay at 35\.8%, 30\.8%, and 36\.4%, while SimpleDoc\+GPT\-5\.4 reaches 55\.2% on counterfactual and 51\.1% on aggregation: once the right evidence is in context, models compute comparisons or counterfactuals reasonably well, but struggle to maintain a candidate set or check coverage\. Claude Opus 4\.6 OCR is the unusual one\-shot exception, leading on Aggregation \(46\.8\) and tying SimpleDoc\+GPT\-5\.4 on Consistency \(47\.6\) without retrieval\. The trade\-off shows on Unanswerable: Claude scores 66\.8, below GPT\-5\.2 OCR \(83\.6\), Qwen3\.5\-VL 35B\-A3B OCR \(84\.1\), and DeepSeek\-V3\.2 OCR \(78\.5\) – the strongest one\-shot reader is also the most willing to attempt an answer when support is missing\.Third, agent pipelines help only when retrieval produces usable context\.SimpleDoc\+GPT\-5\.4 is the strongest system at 44\.0% \(\+6\.7\+6\.7over GPT\-5\.4 OCR\), yet with the same backbone MDocAgent and DeepRead reach only 35\.0% and 32\.2%: the agent interface is not automatically better\. Retrieval can also hurt abstention – MDocAgent\+GPT\-5\.4 scores 22\.0% on unanswerable while several one\-shot OCR models exceed 78% – because partially relevant pages push the final model to answer when support is absent\.

![Refer to caption](https://arxiv.org/html/2608.00036v1/x3.png)Figure 4:Diagnostic performance breakdowns for all evaluated systems\.Columns separate full\-image input, OCR input, and agent pipelines; rows report accuracy by context length, number of expert\-annotated evidence pages, and evidence span\.
### 4\.3Diagnostic Breakdowns

Aggregate accuracy does not explain why a system fails\. Figure[4](https://arxiv.org/html/2608.00036#S4.F4)decomposes accuracy by context length, evidence\-page count, and evidence span – three pressures that the overall score conflates\. All direct readers degrade on longer contexts even though the 1M\-token models \(Claude Opus 4\.6, GPT\-5\.4\) and the 128K DeepSeek\-V3\.2 have very different windows, so the drop is not just truncation: models must still filter irrelevant pages and keep the right support active\. OCR makes pages searchable but does not select evidence; agents pass compact context to a smaller backbone, but only when their search step finds every required page\.

Table[5](https://arxiv.org/html/2608.00036#A0.T5)adds two checks\.Modality\.SimpleDoc\+GPT\-5\.4 leads on text, tables, charts, and mixed modalities, while Kimi\-K2\.5 OCR and Claude Opus 4\.6 OCR tie on image evidence – the best overall pipeline is not the best visual reader\.Document scope\.Cross\-document is not uniformly harder: GPT\-5\.4 OCR scores higher cross than single \(43\.03% vs\. 36\.63%\), SimpleDoc\+GPT\-5\.4 is balanced \(43\.03% vs\. 44\.09%\), and Claude Opus 4\.6 OCR is the only frontier one\-shot reader with a small cross\-document penalty \(−2\.53\-2\.53vs\. GPT\-5\.4 OCR’s\+6\.40\+6\.40\), suggesting its strength lies in deep single\-document reading\. Large negative gaps appear instead in unstable settings \(DeepRead\+DeepSeek\-V3\.2−14\.30\-14\.30, GPT\-5\.2 Img−9\.38\-9\.38\), so the harder step is selecting the right document and page, not the cross\-document label itself\.

![Refer to caption](https://arxiv.org/html/2608.00036v1/x4.png)Figure 5:Agent answer yield versus tool\-call rounds\. Accuracy as the retrieval rounds increases\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x5.png)Figure 6:Agent evidence\-hit dynamics\.Evidence hit rate as the retrieved/read page rounds increases\.Figures[5](https://arxiv.org/html/2608.00036#S4.F5)and[6](https://arxiv.org/html/2608.00036#S4.F6)separate agent failures into retrieval and evidence\-use errors\. High evidence hit with low answer yield indicates a rule\-following or answer\-normalization problem; low evidence hit indicates a page\-search or context\-construction problem\. The same final accuracy can therefore mask different failure modes\.

### 4\.4Discussion

The experiments show that window size is not the whole problem\. The two 1M\-token models \(Claude Opus 4\.6 and GPT\-5\.4\) share the same usable OCR budget, yet Claude Opus 4\.6 OCR – itself a one\-shot reader – still leads GPT\-5\.4 OCR by 2\.5 points and is in turn surpassed by the best agent pipeline\. This indicates that capacity\-driven gains and retrieval\-driven gains are partially substitutable rather than additive\. Some 256K and 128K models also improve when retrieval supplies compact evidence\. OCR helps one\-shot models by giving them searchable text, but it does not solve set tracking or abstention\. Agent pipelines help when retrieval returns compact evidence, as in SimpleDoc, but can hurt when search misses a required page or supplies distracting context\. These results show why long\-document QA needs page\-level evidence labels\. Without them, retrieval failure and reasoning failure are hard to separate\. The benchmark therefore reports reasoning\-type accuracy, modality and scope breakdowns, and retrieval checks rather than a single leaderboard score\.

## 5Conclusion

We introduced XL\-DocBench, a fully human\-verified benchmark for extra\-long document understanding\. It combines 1,519 questions over documents up to 2,303 pages with expert page\-level evidence, typed verification rules, and a twelve\-type reasoning taxonomy\. Current long\-context and agentic systems still struggle to find evidence, combine support, and satisfy explicit rules: the strongest pipeline reaches only 44\.0% overall, and the two 1M\-token one\-shot readers remain 4–7 points behind despite identical usable budgets\. Per\-type and per\-modality breakdowns further show that set\-tracking and abstention – not raw context length – are the binding constraints\.

## References

- \[1\]P\. Agrawal, S\. Antoniak, E\. B\. Hanna, B\. Bout, D\. Chaplot, J\. Chudnovsky, D\. Costa, B\. De Monicault, S\. Garg, T\. Gervet,et al\.\(2024\)Pixtral 12b\.arXiv preprint arXiv:2410\.07073\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[2\]\(2026\)Introducing claude opus 4\.6\.Note:[https://www\.anthropic\.com/news/claude\-opus\-4\-6](https://www.anthropic.com/news/claude-opus-4-6)Cited by:[§4\.1](https://arxiv.org/html/2608.00036#S4.SS1.p2.1)\.
- \[3\]S\. Appalaraju, B\. Jasani, B\. U\. Kota, Y\. Xie, and R\. Manmatha\(2021\)Docformer: end\-to\-end transformer for document understanding\.InICCV,pp\. 993–1003\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[4\]Y\. Bai, X\. Lv, J\. Zhang, H\. Lyu, J\. Tang, Z\. Huang, Z\. Du, X\. Liu, A\. Zeng, L\. Hou,et al\.\(2024\)Longbench: a bilingual, multitask benchmark for long context understanding\.InACL,pp\. 3119–3137\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[5\]Y\. K\. Chia, L\. Cheng, H\. P\. Chan, M\. Song, C\. Liu, M\. Aljunied, S\. Poria, and L\. Bing\(2025\)M\-longdoc: A benchmark for multimodal super\-long document understanding and A retrieval\-aware tuning framework\.InEMNLP,pp\. 9233–9250\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[6\]J\. Cho, D\. Mahata, O\. Irsoy, Y\. He, and M\. Bansal\(2024\)M3docrag: multi\-modal retrieval is what you need for multi\-page multi\-document understanding\.arXiv preprint arXiv:2411\.04952\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[7\]C\. Deng, J\. Yuan, P\. Bu, P\. Wang, Z\. Li, J\. Xu, X\. Li, Y\. Gao, J\. Song, B\. Zheng,et al\.\(2025\)Longdocurl: a comprehensive multimodal long document benchmark integrating understanding, reasoning, and locating\.InACL,pp\. 1135–1159\.Cited by:[Table 1](https://arxiv.org/html/2608.00036#S1.T1.33.33.33.33.4),[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[8\]Y\. Ding, S\. Luo, Y\. Dai, Y\. Jiang, Z\. Li, G\. Martin, and Y\. Peng\(2025\)A survey on mllm\-based visually rich document understanding: methods, challenges, and emerging trends\.arXiv preprint arXiv:2507\.09861\.Cited by:[§1](https://arxiv.org/html/2608.00036#S1.p1.1)\.
- \[9\]M\. Faysse, H\. Sibille, T\. Wu, B\. Omrani, G\. Viaud, C\. Hudelot, and P\. Colombo\(2024\)Colpali: efficient document retrieval with vision language models\.arXiv preprint arXiv:2407\.01449\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[10\]Google\(2026\)Gemini\-3\.Note:[https://aistudio\.google\.com/models/gemini\-3/](https://aistudio.google.com/models/gemini-3/)Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[11\]S\. Han, P\. Xia, R\. Zhang, T\. Sun, Y\. Li, H\. Zhu, and H\. Yao\(2025\)Mdocagent: a multi\-modal multi\-agent framework for document understanding\.arXiv preprint arXiv:2503\.13964\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[12\]A\. Hu, H\. Xu, L\. Zhang, J\. Ye, M\. Yan, J\. Zhang, Q\. Jin, F\. Huang, and J\. Zhou\(2025\)Mplug\-docowl2: high\-resolution compressing for ocr\-free multi\-page document understanding\.InACL,pp\. 5817–5834\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[13\]Y\. Huang, T\. Lv, L\. Cui, Y\. Lu, and F\. Wei\(2022\)Layoutlmv3: pre\-training for document ai with unified text and image masking\.InACM MM,pp\. 4083–4091\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[14\]C\. Jain, Y\. Wu, Y\. Zeng, J\. Liu, S\. Dai, Z\. Shao, Q\. Wu, and H\. Wang\(2025\)SimpleDoc: multi\-modal document understanding with dual\-cue page retrieval and iterative refinement\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 28398–28415\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[15\]G\. Kim, T\. Hong, M\. Yim, J\. Park, J\. Yim, W\. Hwang, S\. Yun, D\. Han, and S\. Park\(2021\)Donut: document understanding transformer without ocr\.arXiv preprint arXiv:2111\.156647\(15\),pp\. 2\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[16\]Z\. Li, H\. Tian, L\. Luo, Y\. Cao, and P\. Luo\(2026\)DeepRead: document structure\-aware reasoning to enhance agentic search\.arXiv preprint arXiv:2602\.05014\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[17\]J\. Liu, Y\. Song, B\. Y\. Lin, W\. Lam, G\. Neubig, Y\. Li, and X\. Yue\(2024\)Visualwebbench: how far have multimodal llms evolved in web page understanding and grounding?\.arXiv preprint arXiv:2404\.05955\.Cited by:[Table 1](https://arxiv.org/html/2608.00036#S1.T1.15.15.15.15.4)\.
- \[18\]Y\. Liu, B\. Yang, Q\. Liu, Z\. Li, Z\. Ma, S\. Zhang, and X\. Bai\(2026\)Textmonkey: an ocr\-free large multimodal model for understanding document\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[19\]Y\. Ma, J\. Li, Y\. Zang, X\. Wu, X\. Dong, P\. Zhang, Y\. Cao, H\. Duan, J\. Wang, Y\. Cao,et al\.\(2025\)Towards storage\-efficient visual document retrieval: an empirical study on reducing patch\-level embeddings\.InACL,pp\. 19568–19580\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[20\]Y\. Ma, Y\. Zang, L\. Chen, M\. Chen, Y\. Jiao, X\. Li, X\. Lu, Z\. Liu, Y\. Ma, X\. Dong,et al\.\(2024\)Mmlongbench\-doc: benchmarking long\-context document understanding with visualizations\.NeurIPS37,pp\. 95963–96010\.Cited by:[Table 1](https://arxiv.org/html/2608.00036#S1.T1.30.30.30.30.4),[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[21\]A\. Masry, X\. L\. Do, J\. Q\. Tan, S\. Joty, and E\. Hoque\(2022\)Chartqa: a benchmark for question answering about charts with visual and logical reasoning\.InACL,pp\. 2263–2279\.Cited by:[Table 1](https://arxiv.org/html/2608.00036#S1.T1.6.6.6.6.4),[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[22\]M\. Mathew, V\. Bagal, R\. Tito, D\. Karatzas, E\. Valveny, and C\. Jawahar\(2022\)Infographicvqa\.InWACV,pp\. 1697–1706\.Cited by:[Table 1](https://arxiv.org/html/2608.00036#S1.T1.9.9.9.9.4),[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[23\]M\. Mathew, D\. Karatzas, and C\. Jawahar\(2021\)Docvqa: a dataset for vqa on document images\.InCVPR,pp\. 2200–2209\.Cited by:[Table 1](https://arxiv.org/html/2608.00036#S1.T1.3.3.3.3.4),[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[24\]Qwen Team\(2026\)Qwen3\.5: towards native multimodal agents\.Note:[https://qwen\.ai/blog?id=qwen3\.5](https://qwen.ai/blog?id=qwen3.5)Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[25\]U\. Shaham, E\. Segal, M\. Ivgi, A\. Efrat, O\. Yoran, A\. Haviv, A\. Gupta, W\. Xiong, M\. Geva, J\. Berant,et al\.\(2022\)Scrolls: standardized comparison over long language sequences\.InEMNLP,pp\. 12007–12021\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[26\]R\. Tanaka, T\. Iki, T\. Hasegawa, K\. Nishida, K\. Saito, and J\. Suzuki\(2025\)Vdocrag: retrieval\-augmented generation over visually\-rich documents\.InCVPR,pp\. 24827–24837\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[27\]R\. Tanaka, K\. Nishida, K\. Nishida, T\. Hasegawa, I\. Saito, and K\. Saito\(2023\)Slidevqa: a dataset for document visual question answering on multiple images\.InAAAI,Vol\.37,pp\. 13636–13645\.Cited by:[Table 1](https://arxiv.org/html/2608.00036#S1.T1.27.27.27.27.5),[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[28\]R\. Tito, D\. Karatzas, and E\. Valveny\(2023\)Hierarchical multimodal transformers for multipage docvqa\.Pattern Recognition144,pp\. 109834\.Cited by:[Table 1](https://arxiv.org/html/2608.00036#S1.T1.18.18.18.18.4),[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[29\]J\. Van Landeghem, R\. Tito, Ł\. Borchmann, M\. Pietruszka, P\. Joziak, R\. Powalski, D\. Jurkiewicz, M\. Coustaty, B\. Anckaert, E\. Valveny,et al\.\(2023\)Document understanding dataset and evaluation \(dude\)\.InICCV,pp\. 19528–19540\.Cited by:[Table 1](https://arxiv.org/html/2608.00036#S1.T1.23.23.23.23.6),[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[30\]J\. Wang, L\. Jin, and K\. Ding\(2022\)Lilt: a simple yet effective language\-independent layout transformer for structured document understanding\.InACL,pp\. 7747–7757\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[31\]W\. Wang, Z\. Gao, L\. Gu, H\. Pu, L\. Cui, X\. Wei, Z\. Liu, L\. Jing, S\. Ye, J\. Shao,et al\.\(2025\)Internvl3\. 5: advancing open\-source multimodal models in versatility, reasoning, and efficiency\.arXiv preprint arXiv:2508\.18265\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[32\]R\. Xia, S\. Mao, X\. Yan, H\. Zhou, B\. Zhang, H\. Peng, J\. Pi, D\. Fu, W\. Wu, H\. Ye,et al\.\(2024\)Docgenome: an open large\-scale scientific document benchmark for training and testing multi\-modal large language models\.arXiv preprint arXiv:2406\.11633\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[33\]Y\. Xu, Y\. Xu, T\. Lv, L\. Cui, F\. Wei, G\. Wang, Y\. Lu, D\. Florencio, C\. Zhang, W\. Che,et al\.\(2021\)Layoutlmv2: multi\-modal pre\-training for visually\-rich document understanding\.InIJCNLP,pp\. 2579–2591\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[34\]Y\. Xu, M\. Li, L\. Cui, S\. Huang, F\. Wei, and M\. Zhou\(2020\)Layoutlm: pre\-training of text and layout for document image understanding\.InKDD,pp\. 1192–1200\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[35\]J\. Ye, A\. Hu, H\. Xu, Q\. Ye, M\. Yan, Y\. Dan, C\. Zhao, G\. Xu, C\. Li, J\. Tian,et al\.\(2023\)Mplug\-docowl: modularized multimodal large language model for document understanding\.arXiv preprint arXiv:2307\.02499\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[36\]S\. Yu, C\. Tang, B\. Xu, J\. Cui, J\. Ran, Y\. Yan, Z\. Liu, S\. Wang, X\. Han, Z\. Liu,et al\.\(2024\)Visrag: vision\-based retrieval\-augmented generation on multi\-modality documents\.arXiv preprint arXiv:2410\.10594\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p1.1)\.
- \[37\]S\. Zhao, Z\. Jin, S\. Li, and J\. Gao\(2025\)Finragbench\-v: a benchmark for multimodal rag with visual citation in the financial domain\.InEMNLP,pp\. 4215–4249\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[38\]F\. Zhu, W\. Lei, Y\. Huang, C\. Wang, S\. Zhang, J\. Lv, F\. Feng, and T\. Chua\(2021\)TAT\-qa: a question answering benchmark on a hybrid of tabular and textual content in finance\.InIJNLP,pp\. 3277–3287\.Cited by:[Table 1](https://arxiv.org/html/2608.00036#S1.T1.12.12.12.12.4),[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[39\]F\. Zhu, Z\. Liu, X\. Y\. Ng, H\. Wu, W\. Wang, F\. Feng, C\. Wang, H\. Luan, and T\. Chua\(2026\)MMDocBench: benchmarking large vision\-language models for fine\-grained visual document understanding and grounding\.InMMM,pp\. 74–88\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.
- \[40\]A\. Zou, W\. Yu, H\. Zhang, K\. Ma, D\. Cai, Z\. Zhang, H\. Zhao, and D\. Yu\(2025\)Docbench: a benchmark for evaluating llm\-based document reading systems\.InProceedings of the 4th International Workshop on Knowledge\-Augmented Methods for Natural Language Processing,pp\. 359–373\.Cited by:[§2](https://arxiv.org/html/2608.00036#S2.p2.1)\.

Table 4:Accuracy by document domain for all evaluated systems\. Bold pipeline names are group headers; rows underneath report pipeline–backbone combinations\.Table 5:Accuracy by evidence modality and document scope for all evaluated systems\. Gap is Cross–Single, so negative values indicate a cross\-document penalty\. Text\-combination modalities are assigned to the non\-text modality \(e\.g\., Table\+Text→\\rightarrowTable\); combinations of multiple non\-text modalities are grouped as Mix\.## Appendix AConstruction Pipeline Details

This appendix gives implementation details for the construction pipeline in §[3\.1](https://arxiv.org/html/2608.00036#S3.SS1)\. Each document is parsed into page images, OCR text, Markdown, tables, and figure metadata\. Headings, tables of contents, layout cues, and fallback page groups define a hierarchical tree; for document series, per\-document trees are attached to a shared root\. Candidate generation uses branch summaries, domain quotas, and reasoning\-type quotas\. We keep candidates that pass leave\-one\-branch\-out checks at the chapter or section level, iterative refinement checks, grounding checks, and artifact filters\. The released page\-level evidence does not come from this coarse automatic stage\. It is annotated and verified by 194 human experts with full document access\.

The filters target five failure modes: malformed answer schema, meaningless arithmetic such as page\-number subtraction, failed branch\-level path\-dependency, answerability without document evidence, and residual references to internal tree nodes\. The final released subset contains 1,519 examples from 3,550 generated candidates\. The no\-context filter is the largest single rejection source\. It gives the judge only the question and no document content; if the answer is still inferable from world knowledge, metadata, or wording leakage, the candidate is rejected\. After expert verification, we also remove lower\-difficulty items that pass support checks but do not sufficiently test long\-context evidence localization or multi\-step reasoning\. This produces the final 1,519\-example release subset\.

## Appendix BDiagnostic Breakdown Tables

Tables[4](https://arxiv.org/html/2608.00036#A0.T4)and[5](https://arxiv.org/html/2608.00036#A0.T5)provide breakdowns by document domain, evidence modality, and document scope for the evaluated systems\.

## Appendix CEvaluation Protocol Details

Each example includes an answer type and a typed verification rule\. Integer answers require exact normalized match\. Float answers specify units, rounding, and tolerance\. String answers include aliases and are evaluated with normalized alias/rule matching for accuracy, with token F1 and ANLS as secondary surface\-form metrics\. Unanswerable examples useNoneand include a rationale describing which required condition is absent from the documents\. These rules connect construction\-time grounding to evaluation\-time scoring and reduce the need for unconstrained free\-form judging\.

## Appendix DDataset Card

The released dataset card will document the benchmark motivation, composition, source documents, preprocessing, annotation process, validation process, intended use, out\-of\-scope use, licensing, maintenance plan, and known limitations\. The benchmark is intended for evaluating document\-understanding systems, not for making financial, medical, legal, or regulatory decisions\.

## Appendix EBroader Impacts and Limitations

The benchmark supports research on reliable document understanding, evidence\-based evaluation, and long\-context system design\. It can help reveal unsupported or poorly supported document answers before such systems are used in professional settings\. At the same time, high benchmark performance does not show that a model is safe for legal, medical, financial, regulatory, or other high\-stakes work\. The dataset uses public professional documents, but evaluated systems may still give misleading answers, fail to abstain, or cite incomplete evidence\. Users should treat XL\-DocBench as a research benchmark and diagnostic tool, not as a replacement for domain experts or formal review\. Limitations include possible style bias from LLM\-assisted candidate generation, finite human verification capacity, public\-document exposure in model pretraining, English and public\-source coverage limits, residual PDF parsing or OCR errors, and the smaller size of the cross\-document subset compared with the full benchmark\.

## Appendix FDeclaration of LLM Usage

We used LLMs in the construction pipeline to propose candidate questions, provisional answers, candidate evidence, verification rules, and automatic filtering or judging signals, as described in Appendix[H](https://arxiv.org/html/2608.00036#A8)\. These LLM\-generated outputs were provisional: LLMs were not used to replace human verification, determine the final benchmark labels, or make final decisions about the reported results\. All released answers, evidence annotations, reasoning labels, statistics, experimental conclusions, and paper edits were reviewed by the authors\.

## Appendix GFailure Case Studies

Figures[7](https://arxiv.org/html/2608.00036#A7.F7)and[8](https://arxiv.org/html/2608.00036#A7.F8)show representative failed cases from the evaluated systems\. These examples appear before the reasoning\-type case studies and show the main error modes discussed in §[4\.3](https://arxiv.org/html/2608.00036#S4.SS3)\. In both cases, the model output is not merely a surface\-form mismatch: the failure reflects a breakdown in evidence search, evidence use, or rule\-following answer generation\. The examples also explain why XL\-DocBench records page\-level evidence and typed verification rules\. Without these annotations, it is difficult to tell whether the system failed because it missed the supporting page, used the wrong evidence, or answered without satisfying the required condition\.

![Refer to caption](https://arxiv.org/html/2608.00036v1/x6.png)Figure 7:Failure case study 1\. The example illustrates a system failure on an evidence\-grounded long\-document question\. The model response does not satisfy the verified answer rule, showing how long\-context QA can fail even when the question appears answerable from the document\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x7.png)Figure 8:Failure case study 2\. The final response is inconsistent with the expert\-verified evidence\. Such cases motivate reporting evidence\-use checks in addition to final\-answer accuracy\.
## Appendix HSynthesis Prompts and Evaluation Algorithms

This section summarizes the prompts used by the data\-synthesis pipeline and the deterministic scoring procedure used in our experiments\. The boxes describe each stage’s role, constraints, and output format rather than the full implementation strings\.

### H\.1Data\-Synthesis Prompt Structure

Four\-stage synthesis chainExplorationproposes root\-level questions from the document tree\. The prompt asks the model to act as a benchmark designer rather than a summarizer: it should identify unresolved issues that require descending into lower sections, prefer verifiable closed\-form answers, and assign a reasoning type\. The pipeline rejects questions that can be answered from the root summary alone\.Refinementrewrites the parent question using child\-node summaries without changing the topic\. It narrows the answer space, selects the smallest sufficient set of next nodes, and keeps cross\-branch or cross\-document dependence when the item is designed to span multiple branches\.Evidence decompositionturns the refined question into atomic support checks\. Each sub\-question must isolate one concrete support point, preferably from a distinct page or section, and returns a compact answer plus a verification rule\.Final decisionwrites the benchmark\-ready QA item\. The prompt enforces answer\-format constraints, rejects trivial arithmetic and metadata\-derived answers, preserves the intended reasoning type, and outputs a provisional question, answer, candidate evidence pages, and typed verification rule for subsequent human verification\.

Quality constraints used across promptsPath dependencyrequires that a candidate cannot be answered from high\-level summaries alone\. The automatic check operates over chapter or section branches; final page\-level evidence is added by human experts\.Cross\-branch necessityrequires that removing any targeted branch weakens or breaks the question\. For multi\-document trees, the prompt forces evidence from at least two PDFs and uses document\-prefixed node IDs\.Answer disciplinebans boolean answers, counting\-only questions, page\-number arithmetic, ID arithmetic, and vague multi\-part questions\. Numeric answers must have meaningful units or domains, entity answers must be short, and single\-choice options must be mutually exclusive\.Artifact filteringrejects candidates answerable from world knowledge, document metadata, or wording leakage\. It also rejects questions whose apparent difficulty comes from formatting noise rather than document understanding\.

Editing and judge promptsPackaging editorshortens the final QA without changing its meaning or supported answer\. It removes unnecessary section numbering, keeps at least one phrase that helps retrieval, and normalizes answer surfaces\.Page\-stage repair editoris called when convergence checks fail\. It restores lost constraints, repairs drift from the refined question, and re\-inserts important fields such as criterion, threshold, comparison target, or mismatch\.Path\-dependency judgedecides whether the candidate still requires hierarchical navigation\. A high score means the answer needs evidence from multiple pages, sections, or tables rather than a single local span\.State\-assessment judgescores answerability, confidence, and benchmarkability for an in\-flight exploration state\. It also flags summary\-shortcut risk and ambiguity risk before the candidate reaches human verification\.

### H\.2Evaluation Code Sketch

Our reported metrics are computed by deterministic scripts\. Direct baselines are instructed to put the final answer in<answer\>tags; for agent baselines, we first extract a concise final answer from the agent response and then run the same scoring code\. The headline accuracy does not use an LLM judge\. The simplified Python\-style sketch below shows the main logic\.

Listing 1:Python\-style sketch of the deterministic evaluator\.defevaluate\_prediction\(response,gold,answer\_type\):

pred=extract\_answer\(response\)

pred\_n=normalize\(pred\)

gold\_n=normalize\(gold\)

ifanswer\_typein\{"none","unanswerable"\}:

acc=contains\_abstention\_phrase\(pred\_n\)

elifanswer\_typein\{"numeric","percentage"\}:

p=first\_number\(pred\_n\)

g=first\_number\(gold\_n\)

acc=pisnotNoneandgisnotNoneandrelative\_error\(p,g\)<=0\.05

elifanswer\_type=="single\_choice":

acc=first\_option\_label\(pred\)==first\_option\_label\(gold\)

else:

acc=\(gold\_ninpred\_n\)ornorm\_lev\_sim\(pred\_n,gold\_n\)\>=0\.8

return\{

"acc":int\(acc\),

"f1":token\_f1\(pred\_n,gold\_n\),

"anls":anls\(pred,gold\),

\}

defaggregate\_scores\(examples\):

scored=\[evaluate\_prediction\(e\.response,e\.gold,e\.answer\_type\)foreinexamples\]

overall=mean\_metrics\(scored\)

by\_reasoning=group\_and\_average\(scored,key="reasoning\_type"\)

by\_modality=group\_and\_average\(scored,key="evidence\_modality"\)

by\_scope=group\_and\_average\(scored,key="document\_scope"\)

by\_length=group\_and\_average\(scored,key="context\_or\_evidence\_bin"\)

returnoverall,by\_reasoning,by\_modality,by\_scope,by\_length

#### Scoring details\.

normalizelowercases text, removes common answer prefixes, drops articles, strips most punctuation while retaining decimal points, hyphens, and percent signs, and collapses whitespace\. Token F1 is set\-based overlap after normalization\. ANLS uses normalized Levenshtein similarity with a0\.50\.5threshold and does not apply the full normalization step, so punctuation and surface form still affect the distance\.

## Appendix IReasoning\-Type and Cross\-Document Case Studies

Figures[9](https://arxiv.org/html/2608.00036#A9.F9)–[22](https://arxiv.org/html/2608.00036#A9.F22)show cases from the final human\-verified benchmark\. Cross\-document cases appear first, followed by reasoning\-type cases\. Each case comes from the same construction pipeline used for the benchmark\. Documents are parsed into page text, layout, tables, figures, and page images\. Candidate questions are generated from document branches and assigned a reasoning type\. Automatic checks filter candidates for answer format, leave\-one\-branch sensitivity, and evidence consistency\. The remaining examples are sent to human verification\. The figures show final verified examples rather than raw generated candidates\. The question, typed answer, reasoning label, evidence pages, and marked evidence spans are the result of human review\. Human annotators inspected the source pages, corrected or removed unsupported evidence, and verified that the answer follows from the marked support\. They also resolved ambiguous cases and confirmed whether cross\-document cases require evidence from multiple PDFs\. These examples show how human evidence verification turns automatically proposed questions into reliable benchmark examples\.

![Refer to caption](https://arxiv.org/html/2608.00036v1/x8.png)Figure 9:Case study: Cross\-document temporal\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x9.png)Figure 10:Case study: Cross\-document reference chain\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x10.png)Figure 11:Case study: Comparison\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x11.png)Figure 12:Case study: Reference chain\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x12.png)Figure 13:Case study: Ranking\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x13.png)Figure 14:Case study: Coverage\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x14.png)Figure 15:Case study: Reconciliation\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x15.png)Figure 16:Case study: Set difference\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x16.png)Figure 17:Case study: Unanswerable\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x17.png)Figure 18:Case study: Temporal\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x18.png)Figure 19:Case study: Compliance\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x19.png)Figure 20:Case study: Counterfactual\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x20.png)Figure 21:Case study: Aggregation\.![Refer to caption](https://arxiv.org/html/2608.00036v1/x21.png)Figure 22:Case study: Consistency\.