ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Hugging Face Daily Papers Papers
benchmark document-extraction schema-guided enterprise evaluation grounding llm

Summary

ExtractBench is a new benchmark for schema-guided enterprise document extraction, evaluating value accuracy, record completeness, grounding, and cost across 4,869 pages of enterprise documents. The authors find that commercial VLMs struggle with long documents while coding agents are more accurate but costly, and LlamaExtract AgenticPlus leads on all metrics.

Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on https://huggingface.co/datasets/llamaindex/ExtractBench{HuggingFace} and https://github.com/run-llama/ExtractBench{GitHub}.
Original Article
View Cached Full Text

Cached at: 08/03/26, 05:30 AM

Paper page - ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Source: https://huggingface.co/papers/2607.29677

Abstract

Enterpriseworkflowsincreasinglyrelyonagentsforschema-guidedextraction:givenadocumentandauser-definedschema,theagentfaithfullyfollowstheschematoproducethecorrectoutputwithsourceevidenceasgroundingmetadata.WepresentExtractBench,abenchmarkforschema-guidedextractionand,toourknowledge,thefirsttoscorevalueaccuracy,recordcompletenessatscale,grounding,andmeasuredcosttogether.Theevaluationsystemcontains4,869pagesacross370enterprisedocuments,8businessdomains,and67documenttypes,withcleartagsdifferentiatingtheirchallengescenarios.Thescalableschemaandground-truthcurationpipelinecombinesindependent-systemagreementforrealdocuments,knownvaluesforsyntheticlists,andhumanverificationforforms.Wereportorder-insensitivevalueF1forvalueaccuracy,plustwogroundingmetricsforsourcetraceability:word-andpage-levelF1.CommercialVLMsperformwellonshortdocumentsbutoftentruncaterecordlistsonlongones,whilecodingagentsretainhigheraccuracyatmuchhighercost.LlamaExtractAgenticPlusranksfirstonallthreemetrics,withaccuracycomparabletocodingagentsatafractionofthecost.Datasetandevaluationcodeareavailableonhttps://huggingface.co/datasets/llamaindex/ExtractBench{HuggingFace}andhttps://github.com/run-llama/ExtractBench{GitHub}.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2607\.29677

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2607.29677 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2607.29677 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2607.29677 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles