We benchmarked 18 LLMs on OCR (7k+ calls) — cheaper/old models oftentimes win. Full dataset + framework open-sourced. [R]
Summary
A comprehensive benchmark of 18 LLMs on OCR tasks (7k+ calls) reveals that cheaper and older models often match premium accuracy at a fraction of the cost, with full dataset and framework open-sourced.
Similar Articles
Open-Weight LLMs Have Caught Up on Accuracy (21 minute read)
A new benchmark, ClinReg, evaluates LLMs on real regulatory and clinical-trial tasks, finding that open-weight models now match closed-source models on accuracy at a fraction of the cost.
@llama_index: "OCR is just a feature now. Frontier models will eat it." We hear this constantly. The data says otherwise. Across thre…
LlamaIndex argues that document OCR is not being commoditized by frontier models, using benchmark data showing specialized parsers remain more accurate and cheaper.
Benchmarking LLMs
A study or report on benchmarking large language models, likely comparing performance across various tasks.
Benchmarking LLM Judges for Mobile Agent Evaluation
This paper introduces MobileJudgeBench, a benchmark with 931 human-annotated trajectories for systematically evaluating LLM-based judges on mobile agent tasks. It finds that simple baseline judges with sampled screenshots rival purpose-built methods, with the LLM backbone being the primary driver of quality.
@techNmak: A lightweight VLM that beats the giants at OCR. (1.7B parameters, SOTA on OmniDocBench) dots. ocr is a new multilingual…
dots.ocr is a new lightweight 1.7B parameter multilingual vision-language model that achieves state-of-the-art performance on OmniDocBench, outperforming much larger models (72B+) at document parsing and OCR tasks.