SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

Hugging Face Daily Papers Papers
benchmark vision-language-models document-understanding long-context synthetic-data evaluation

Summary

SynthDocBench is a fully synthetic benchmark for long-context visual document understanding that systematically controls document length, layout, modality, and question type, revealing failure modes in current VLMs such as length degradation and positional sensitivity.

Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.
Original Article
View Cached Full Text

Cached at: 07/15/26, 04:22 PM

Paper page - SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

Source: https://huggingface.co/papers/2607.10400

Abstract

Visionlanguagemodels(VLMs)haveachievedstrongperformanceonvisualdocumentunderstandingbenchmarkssuchasDocVQA,ChartQA,andMMLongBench-Doc.However,real-worlddocumentscombinemultiplefactorssuchaslength,layoutcomplexity,modality,andquestiondifficulty,whichmakesitdifficulttoattributemodelfailurestospecificcauses.WeintroduceSynthDocBench,afullysyntheticbenchmarkforlong-contextvisualdocumentunderstandingthatsystematicallycontrolsfactorsincludingdocumentlength,layoutstructure,modalitycomposition,andquestiontype.Thebenchmarkisconstructedusingacombinatorialdesign,eachfactorisvariedindependentlyacrossgenerateddocuments,enablingcontrolledanalysisofmodelbehavior.DocumentsaregeneratedendtoendusinganLLMpipelineacrosssixlayoutarchetypes,witha40percentrandomoverridetopreventmodelsfromexploitingspuriouscorrelations.Additionally,SynthDocBenchspanslong-contextdocumentswithsubstantiallygreaterlengthandstructuraldiversitythanexistingbenchmarks.EvaluatingsevenfrontierVLMs,weuncoverthreefailuremodesthatexistingbenchmarkscannotsurface:sharpdegradationwithdocumentlength,asystematicpositionalsensitivityinwhichthemiddlethirdofadocumentishardestforfiveofsixmodelsandfiveofsixmodelsshowanegativeEarly-to-Latetrend(steepestdecline:8.3percentagepoints),andbreakdownofchartcomprehensioninlong-documentsettings.Theseresultssuggestthatcurrentmodelsmaybeoverfittingtobenchmarkartifactsratherthanachievingrobustlong-contextvisualdocumentunderstanding.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2607\.10400

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.10400 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.10400 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.10400 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

MemoryDocDataSet: A Benchmark for Joint Conversational Memory and Long Document Reasoning

arXiv cs.CL

MemoryDocDataSet is a new synthetic benchmark of 50 micro-worlds and 1,000 QA pairs designed to evaluate AI systems on the joint task of conversational memory and long-document reasoning simultaneously. The best baseline (RAG-Both) achieves only 0.358 overall F1, highlighting a significant gap in current systems' ability to unify conversational memory with long-document navigation.