@VikParuchuri: Structured extraction is hard to eval, and most benchmarks are biased, or have bad scoring/GT. That’s why we made OmniE…
Summary
OmniExtractBench is a new benchmark with 620 documents from multiple vendors, designed for fair evaluation of structured extraction, highlighting top performance by Datalab and Reducto.
View Cached Full Text
Cached at: 09/17/26, 12:26 PM
Structured extraction is hard to eval, and most benchmarks are biased, or have bad scoring/GT.
That’s why we made OmniExtractBench - 620 docs from multiple vendors (Datalab, Reducto, Extend, LlamaIndex), and fair scoring.
@datalabto and Reducto are at the top. https://t.co/j5UBto7cvm
Similar Articles
ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction
ExtractBench is a new benchmark for schema-guided enterprise document extraction, evaluating value accuracy, record completeness, grounding, and cost across 4,869 pages of enterprise documents. The authors find that commercial VLMs struggle with long documents while coding agents are more accurate but costly, and LlamaExtract AgenticPlus leads on all metrics.
@VikParuchuri: Datalab balanced mode extraction now scores 95.9% in our internal benchmark - more accurate than Reducto Deep Extract (…
Datalab's balanced mode extraction achieves 95.9% accuracy in internal benchmarks, surpassing Reducto Deep Extract (95.1%) at less than half the price, with full verification including citations and reasoning.
@VikParuchuri: We're open sourcing a 9B model that extracts structured data from documents at near-frontier performance. - 90.2% on ou…
Vik Paruchuri is open-sourcing a 9B model that extracts structured data from documents with near-frontier performance (90.2% on their benchmark, vs Gemini 3.5 Flash at 91.3%).
@llama_index: Most document extraction APIs can't tell you where a value came from. For ExtractBench, we scored grounding strictly: a…
ExtractBench is a new benchmark for document extraction that strictly scores grounding accuracy, showing VLMs and coding agents lack evidence while LlamaExtract Agentic Plus leads with high page-level and word-level performance.
Evaluating Structured Information Extraction with Open Models in a High Risk Public Sector Application
This paper benchmarks open-source OCR, LLM, and VLM systems for structured information extraction in a high-risk public sector application, finding that VLMs generally outperform OCR+LLM pipelines but most configurations struggle in zero-shot settings, emphasizing the critical role of input quality.