Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR

Hugging Face Daily Papers Papers
ocr evaluation-metrics llm-judge vision-language-models multilingual character-error-rate

Summary

论文提出 semCER，一种利用 LLM 评审按严重程度对字符编辑加权的 OCR 评估指标，相比标准 CER 更能贴合人类评分，并在英、越、中等 16 种语言上验证有效，同时发布了 6100 万条已评分编辑数据集与蒸馏打分模型。

Character Error Rate (CER), the standard metric for OCR and vision-language models (VLMs), charges a typo as much as a wrong name. We introduce semCER, which keeps CER's alignment and denominator and charges each grapheme edit a severity from an LLM judge, with no word segmentation. Normalized CER keeps the score of a whole transcription, where it tracks readers as well as semCER or better, and semCER breaks that score down into named edits. The charges track readers' ratings edit by edit in English, Vietnamese, and Chinese and put the edits readers rate most severe near the top of a sorted list. On pairs drawn where CER and semCER disagreed, readers side with semCER more often than with normalized CER in English and Chinese, and equally often in Vietnamese. Across 16 languages, the judge charges a wrong diacritic or tone mark almost nothing in Spanish but nearly a wrong letter's severity in Telugu. Where we can count real words, a general VLM's wrong letter forms one far above chance while a classical pipeline's stays near it. We release the 61 million judged edits, the aggregated reader ratings, and the distilled scorer at https://github.com/nguyenx2t/semcer.
Original Article
View Cached Full Text

Cached at: 10/16/26, 07:06 PM

Paper page - Equal CER, Unequal Damage: Severity-Weighted Character Error Rate for OCR

Source: https://huggingface.co/papers/2610.20461

Abstract

A new metric called semCER uses an LLM judge to weight grapheme edits by severity, improving alignment with human ratings over standard character error rate for OCR and vision-language models.

Character Error Rate (CER), the standard metric for OCR andvision-language models(VLMs), charges a typo as much as a wrong name. We introducesemCER, which keepsCER’s alignment and denominator and charges eachgrapheme edita severity from anLLM judge, with no word segmentation.Normalized CERkeeps the score of a whole transcription, where it tracks readers as well assemCERor better, andsemCERbreaks that score down into named edits. The charges track readers’ ratings edit by edit in English, Vietnamese, and Chinese and put the edits readers rate most severe near the top of a sorted list. On pairs drawn whereCERandsemCERdisagreed, readers side withsemCERmore often than withnormalized CERin English and Chinese, and equally often in Vietnamese. Across 16 languages, the judge charges a wrong diacritic or tone mark almost nothing in Spanish but nearly a wrong letter’s severity in Telugu. Where we can count real words, a general VLM’s wrong letter forms one far above chance while a classical pipeline’s stays near it. We release the 61 million judged edits, the aggregated reader ratings, and the distilled scorer at https://github.com/nguyenx2t/semcer.

View arXiv pageView PDFGitHub11Add to collection

Models citing this paper1

#### nguyenx2t/semcer-qwen3.5-0.8b 0.8B• Updatedabout 2 hours ago • 19

Datasets citing this paper1

#### nguyenx2t/semcer-judged-edits Viewer• Updatedabout 2 hours ago • 63.5M • 110

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.20461 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Evidence First, Arithmetic Second: A System Report and Failure Analysis for DocSem

arXiv cs.CL

A DocInsights 2026 shared-task system paper describing EviCalc, a PDF document QA pipeline that pairs evidence passage selection with LLM-generated arithmetic expressions evaluated in local code, reporting 8.61% joint accuracy on DocSem's test set along with descriptive failure analysis, including OCR and block-grouping errors that merged relevant passages.