Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

# Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged
Source: [https://arxiv.org/html/2610.10918](https://arxiv.org/html/2610.10918)
October 7, 2026

###### Abstract

Large Language Models \(LLMs\) are considered to be a more efficient and cost\-effective alternative to human judgment for Machine Translation \(MT\) evaluation\. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation\. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics \(MQM\) and Error Span Annotation \(ESA\) by comparing their agreement with human annotators\. We present results on a long\-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains\. Our results show that while LLM agreement with human annotators exceeds agreement between human annotators for some evaluation tasks, both vary substantially across annotation schemes, language pairs and domains and remain unreliable for most settings\. Furthermore, we identify challenges facing both human and LLM annotators: humans are particularly challenged by fine\-grained MQM annotations and low\-resource language pairs, while LLMs struggle with minor errors, wrong language variants and error span annotation\. Our results highlight the potential for improvement for both human and LLM annotation performance, possibly through human\-LLM collaborative annotation pipelines that address the reliability issues identified in this work\.

## 1Introduction

Translation evaluation is a challenging task, which requires evaluating multiple quality dimensions simultaneously\. It includes judging the faithful transfer of meaning from the source language to the target language, the naturalness and fluency of the translated text in addition to its adherence to the target language linguistic and cultural structures, and the usability of the translation in a given context\. Human evaluation is the gold standard to evaluate MT models\. However, human evaluation is often expensive and time\-consuming, slowing down MT model development\. Furthermore, human evaluation suffers from quality and reliability issues\. In addition to the highly subjective nature of translation evaluation, human evaluators differ in their skills, training, motivation and leniency\. Annotation tasks also differ in their design, guidelines, compensation and nature of evaluated text\. All these factors contribute to the unreliability of human evaluation\.

Several approaches aim to improve the reliability of human evaluation\. This includes aggregating multiple annotations from different annotators for the same source and translation texts\. Other work relies on more fine\-grained evaluation approaches that ground evaluation rating in detailed error annotations such as MQM\([Lommel et al\., 2014](https://arxiv.org/html/2610.10918#bib.bib16)\)and ESA\([Kocmi et al\., 2024](https://arxiv.org/html/2610.10918#bib.bib13)\)instead of providing a single quality rating\. While these fine\-grained MT evaluation approaches offer a more principled way for human evaluation which improves their quality, interpretability and consistency, they are more time\-consuming and expensive to obtain\.

Automatic MT evaluation metrics offer a solution to approximate human evaluation in a cost and time efficient way\. Early automatic metrics such as BLEU\([Papineni et al\., 2002](https://arxiv.org/html/2610.10918#bib.bib18)\)and chrF\([Popović, 2015](https://arxiv.org/html/2610.10918#bib.bib19)\)rely on lexical matching between the translation and a human\-produced reference translation\. While being efficient to compute, they suffer from a severe limitation of penalizing correct translations which deviate from the reference wording\.

With the advent of large pretrained language models, neural evaluation metrics emerged such as BLEURT\([Sellam et al\., 2020](https://arxiv.org/html/2610.10918#bib.bib23)\), COMET\([Rei et al\., 2020](https://arxiv.org/html/2610.10918#bib.bib21)\)and MetricX\([Juraska et al\., 2024](https://arxiv.org/html/2610.10918#bib.bib9)\), offering the ability to represent and compare text beyond the surface level, which was the main limitation of surface\-based automatic MT metrics\. Neural MT evaluation metrics are built by fine\-tuning language models on human evaluation data, with some offering error span annotation similar to MQM such as in xCOMET\([Guerreiro et al\., 2023](https://arxiv.org/html/2610.10918#bib.bib8)\)\. While these model\-based metrics often achieve better correlation with human judgment than surface\-based metrics, they suffer from multiple limitations, mainly limited generalization ability on unseen language pairs and domains and lack of interpretability\.

With LLMs showing more powerful abilities, researchers began exploring whether these models might provide a cost and time effective alternative for human evaluation, allowing for faster model iterations\. LLMs have been probed for performing automatic evaluation for a range of different tasks out\-of\-the\-box without the need for training them on human judgment data\. They also offer the ability to customize the evaluation process through prompting and providing explanation\. In the MT domain,[Kocmi and Federmann \(2023\)](https://arxiv.org/html/2610.10918#bib.bib12)introduced GEMBA, demonstrating that GPT\-3\.5 and GPT\-4 prompted with simple, language\-agnostic templates achieve state\-of\-the\-art correlation with MQM human judgments on WMT22 data on the system level— without any task\-specific fine\-tuning\.

In this work, we evaluate LLMs for MT evaluation following two popular MT evaluation schemes: MQM and ESA\. MQM involves fine\-grained error annotations where the quality score is automatically calculated out of error categories and severities whereas ESA provides simpler error annotations with only severity and the score is input directly by the annotators instead of being calculated\. Evaluating on these two different annotation schemes allows us to assess LLM performance across different levels of annotation granularity\. We present results on a long\-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs and domains\. We identify challenges facing both human and LLM annotators for fine\-grained MT evaluation\. Our analysis highlights the following main findings:

- •LLM agreement with human annotators is indistinguishable from agreement between human annotators for some evaluation tasks\.
- •Agreement between both LLM and human annotators varies substantially between annotation schemes, language pairs and domains\.
- •Agreement between human annotators is severely impacted when performing fine\-grained error annotations in MQM and for low\-resource language pairs\.
- •LLMs have particular challenges in identifying minor errors, flagging translations in the wrong language or variant and error span annotation, especially for rarer error categories\.
- •Low agreement levels highlight the need for improving both human and LLM evaluation reliability\.

## 2Related Work

### 2\.1Automatic MT Evaluation Metrics

The latest WMT25 metrics shared task\([Lavie et al\., 2025](https://arxiv.org/html/2610.10918#bib.bib15)\)included tasks for both score prediction and span\-level error detection\. The evaluation results showed that LLM\-based metrics are the best performing metrics at the system level however they are outperformed by surface and embedding\-based similarity metrics at the segment level\. The task also demonstrated that span\-level error detection remains the hardest challenge, with the best autoraters \(all LLM\-based\) achieving only 9–13% character\-level F1 compared to 47% for human annotators, which is consistent with our results\.

[Zhan et al\. \(2025\)](https://arxiv.org/html/2610.10918#bib.bib25)improved alignment of Large Reasoning Models with human judgment by post training them on synthetic data that emulates human evaluation process\. They identify minor errors as being more misaligned with human annotations than major errors — consistent with our findings\.

### 2\.2AI\-Assisted and Collaborative Human Annotation

[Zouhar et al\. \(2025\)](https://arxiv.org/html/2610.10918#bib.bib26)showed that AI\-assisted human ESA annotation increases error recall \(1\.6 spans per segment vs\. 0\.5 unassisted\), improves inter\-annotator consistency, and reduces annotation budget by 24% through pre\-filtering of error\-free segments\.[Riley et al\. \(2025\)](https://arxiv.org/html/2610.10918#bib.bib22)demonstrated that human revision of automatic MQM annotations \(GEMBA\-MQM, AutoMQM\) rather than annotating from scratch surfaces missed errors, improves annotation quality without additional cost, and reduces annotation time by up to 12%\.

[Song et al\. \(2025\)](https://arxiv.org/html/2610.10918#bib.bib24)confirmed that performing MQM annotations for translation pairs simultaneously yields higher inter\-annotator agreement on the side\-by\-side comparison results, better error span consistency and more reliability in identifying equal\-quality translations than standard independent MQM annotations\.

### 2\.3Reliability of LLMs as Annotators

[Bavaresco et al\. \(2025\)](https://arxiv.org/html/2610.10918#bib.bib1)evaluated 11 LLMs on a 20\-task NLP benchmark\. They found that no single model consistently excels, LLMs correlate more with non\-expert than expert annotations, and chain\-of\-thought and few\-shot prompting yield inconsistent gains\. They conclude that LLMs must be carefully validated before deployment as evaluators – a finding which our results also support\.

[Calderon et al\. \(2025\)](https://arxiv.org/html/2610.10918#bib.bib3)propose a leave\-one\-out statistical test for whether an LLM can replace a human annotator\. We follow a similar leave\-one\-out approach in our experiments with the difference that[Calderon et al\. \(2025\)](https://arxiv.org/html/2610.10918#bib.bib3)calculate alignment on the instance level while we calculate alignment on the test set level using Krippendorff’sα\\alpha\. They find that success of the test is highly task and annotation aspect\-dependent and that low human inter annotator agreement \(IAA\) makes the test unreliable — a caveat applicable to the results in our study\.

[Kasner et al\. \(2025\)](https://arxiv.org/html/2610.10918#bib.bib10)evaluate LLMs on ESA error span annotation on a subset of the WMT24 human evaluation data\. They found that human\-human IAA consistently exceeds LLM\-human IAA for span annotation, and that LLM\-generated character\-level span indices require post\-hoc re\-alignment — a challenge we independently observe and address\.

Previous work evaluated LLMs against human annotators for machine translation quality annotation focusing on either MQM or ESA, and it was limited to a small number of language pairs\. By contrast, our work compares the agreement of LLMs versus human annotators on both ESA and MQM, exploring both score and span annotation for both schemes\. Our MQM results cover the full MQM taxonomy on 70 language pairs and our ESA results cover multiple language pairs and domains, questioning the reliability of the human baseline across these dimensions\. We present results using two different annotator pools and annotation platforms and supported by statistical significance tests\.

## 3Experimental Setup

### 3\.1Data

We present our findings on three data sets:

- •WMT23 MQM\.The MQM annotations from the WMT23 Metrics Shared Task\([Freitag et al\., 2023](https://arxiv.org/html/2610.10918#bib.bib7)\)for two language pairs: English\-to\-German \(en\-de\) and Chinese\-to\-English \(zh\-en\), which include three independent annotation rounds per segment\.
- •Long\-Context Web MQM\.Our internal test set consisting of multi\-paragraph text extracted from crawled web pages\. Target translations were produced by human translators and a variety of MT systems, including neural and LLM\-based models\.
- •WMT25 ESA\.The Error Span Annotation \(ESA\)\([Kocmi et al\., 2024](https://arxiv.org/html/2610.10918#bib.bib13)\)data from the WMT25 Human Evaluation Data\([Kocmi et al\., 2025](https://arxiv.org/html/2610.10918#bib.bib11)\), covering 14 language pairs across 5 domains111Due to access restrictions imposed by the YouTube platform, WMT25 speech domain was not included in the analysis and represents a limitation of the current work\.\.

For our MQM annotations on the Long\-Context Web MQM test set, professional annotators followed guidelines adapted from[Freitag et al\. \(2021\)](https://arxiv.org/html/2610.10918#bib.bib6), selecting error spans in the source and/or target text and specifying the category and severity of each error using a specially designed interface\. We conducted two annotation rounds covering all segments, yielding two independent annotations per sample from different annotators\. Thus, our work presents results under two different annotation settings with different annotator pools and annotation platforms\. Appendix[A\.1](https://arxiv.org/html/2610.10918#A1.SS1)provides more details about the human annotation process for the Long\-Context Web MQM test set\. Tables[6](https://arxiv.org/html/2610.10918#A1.T6)and[7](https://arxiv.org/html/2610.10918#A1.T7)show data statistics for the Long\-Context Web MQM and WMT23 MQM test sets, respectively\. All data sets have segments which are predominantly long\-context\.

### 3\.2Large Language Models

We use the following LLMs for our evaluation: Gemini 2\.5 Flash\([Comanici et al\., 2025](https://arxiv.org/html/2610.10918#bib.bib4)\), GPT\-OSS\-120B\([OpenAI et al\., 2025](https://arxiv.org/html/2610.10918#bib.bib17)\), DeepSeek\-V3\.1\([DeepSeek\-AI, 2024](https://arxiv.org/html/2610.10918#bib.bib5)\)and Qwen3\.5\-397B\([Qwen Team, 2026](https://arxiv.org/html/2610.10918#bib.bib20)\)\. All inference is performed withtemperature=0\\texttt\{temperature\}=0andtop\_p=1\\texttt\{top\\\_p\}=1\. Open\-weight models are self\-hosted while Gemini 2\.5 Flash is run through Vertex AI in April–May 2026\.

The models receive an adapted version of the annotation guidelines provided to human annotators \(refer to Figure[6](https://arxiv.org/html/2610.10918#A1.F6)for MQM and Figure[7](https://arxiv.org/html/2610.10918#A1.F7)for ESA\)\. For WMT23, we use the original MQM error annotation instructions from[Freitag et al\. \(2021\)](https://arxiv.org/html/2610.10918#bib.bib6), which define the error taxonomy used in the WMT23 annotations\.

We report results using zero\-shot prompting, as adding few\-shot examples did not improve LLM\-human agreement\. We disable reasoning for all open\-weight models except for GPT\-OSS\-120B which was run withreasoning\_effort==low\. The output token limit is set for open\-weight models per task: 4096 tokens for WMT23 MQM and WMT25 ESA, and 8192 for Long\-Context Web MQM, raised to 16384 for DeepSeek\-V3\.1\. No explicit output\-token limit or reasoning effort is set for Gemini 2\.5 Flash, which therefore used the serving API’s default\. We use guided decoding to make the LLM response adhere to the specified output format\. Because every model is evaluated on the same segments, we exclude any segment for which at least one LLM failed to return a parseable annotation, removing 1\.1% of WMT23 MQM, 5\.1% of Long\-Context Web MQM, and 7\.3% of WMT25 ESA segments\.

A critical issue emerged when calculating span annotation agreement with LLM\-provided span indices: at least 80% of extracted spans based on these indices did not match the corresponding span text in the LLM output for all LLMs on MQM and ESA data sets\. To correct this, we re\-aligned spans by searching for the LLM\-provided span text within the source \(for source errors\) or target \(for target errors\) and recording the first match start and end indices\. This post\-processing step was necessary to enable accurate error localization for annotation comparison\.

### 3\.3MQM Scoring

MQM scores are calculated out of the error annotations provided by both humans and the LLM using the standard MQM scoring formula\([Freitag et al\., 2021](https://arxiv.org/html/2610.10918#bib.bib6)\)which weighs major errors by 5, minor errors by 1 and punctuation errors by 0\.1\.

### 3\.4Annotation Evaluation

We evaluate LLM annotations across the following aspects: scores, error counts, and span\-level annotations\. We do so by comparing LLM annotation agreement with human annotations, treating each human annotation round as an independent annotator\.

Score and Error Count Evaluation\.We use Krippendorff’sα\\alpha\([Krippendorff, 2004](https://arxiv.org/html/2610.10918#bib.bib14)\)with interval distance to measure inter\-annotator agreement \(IAA\) on segment\-level MQM and ESA scores as well as major and minor error counts per segment for MQM\. Krippendorff’sα\\alphais a widely\-used agreement coefficient that provides several advantages over correlation measures such as Pearson and Spearman: it accounts for chance agreement, handles multiple annotations and measures agreement on absolute scores rather than merely relative co\-variation — enabling assessment of how consistently annotators agree on absolute translation quality\. We compute LLM\-human IAA using a leave\-one\-out approach: each human annotation round is in turn excluded and replaced by the LLM annotation, and IAA is computed within the resulting annotator group\. The reported LLM\-human IAA is the average across all such groups\.

Span Annotation Evaluation\.We compute the following F1\-based metrics to evaluate error span identification and localization:

- •Character overlap:Measures character\-level overlap between predicted and gold spans\. Each predicted span is matched to the gold span with maximum character overlap \(category\-agnostic\), and F1 is computed based on the number of overlapping characters\.
- •Span overlap:Measures F1 for predicting spans that overlap with gold spans regardless of category and severity \(minimum one character overlap required\)\.
- •Category/severity\-specific span overlap:Same as span overlap but with the requirement of having the same error category/severity when matching with gold spans\.
- •No error:F1 for correctly identifying error\-free segments\.

F1 metrics are computed for all human\-human and human\-LLM annotation pairs and averaged within each group\.

### 3\.5Significance Testing

We tested the statistical significance of the hypothesis of LLM\-human agreement being equal to the human\-human agreement in our results\. We used two\-sided paired bootstrap resampling with replacement on samples clustered by source segments with 10,000 bootstrap iterations and treating a difference as significant atp<0\.05p<0\.05\. LLM\-human and human\-human agreement are calculated on the same resampled source segments for each iteration\. For Tables[2](https://arxiv.org/html/2610.10918#S4.T2)and[4](https://arxiv.org/html/2610.10918#A1.T4)comparing agreement across multiple language pairs within different domains or language pair directions, we used the Benjamini–Hochberg false discovery rate \(FDR\) procedure\([Benjamini and Hochberg, 1995](https://arxiv.org/html/2610.10918#bib.bib2)\)separately to each LLM within each table to correct the p\-values applied, controlling the FDR rate at 0\.05\.

## 4Results

### 4\.1MQM

Figure 1:Inter\-annotator agreement on the WMT23 MQM test set, showing Krippendorff’sα\\alphafor MQM scores, major error counts, and minor error counts\.#### 4\.1\.1Score and Error Count Evaluation

Figure[1](https://arxiv.org/html/2610.10918#S4.F1)shows agreement for each of MQM score, major error counts and minor error counts on the WMT23 test set\. For MQM score, human\-human agreement is significantly better than LLM\-human agreement for both en\-de and zh\-en\. The only exception is Gemini 2\.5 Flash\-human MQM score agreement for zh\-en which is on par with human\-human agreement\. Overall, agreement between different annotator combinations is well below the 0\.667 threshold for tentative agreement\.

Given that MQM scores are derived from major and minor error counts, we conduct agreement experiments for segment\-level major and minor error counts for different annotator pairs to better understand how each error type contributes to the overall score agreement\. For major errors, Gemini 2\.5 Flash\-human agreement approaches human\-human agreement for zh\-en but falls behind it for en\-de while other models’ agreement is further behind\. For minor errors, human\-human agreement substantially exceeds LLM\-human agreement for both language pairs, especially for zh\-en where the margin is larger\. This demonstrates that LLMs have more difficulty in identifying minor errors that involve subtle quality distinctions\. The difficulty of minor error annotation is particularly consequential for MT evaluation: as MT system quality improves, minor errors increasingly become the primary differentiating factor among closely ranked systems, making reliable minor error annotation essential for meaningful quality estimation\.

These findings are consistent with results on the Long\-Context MQM test set with wider language pair coverage \(Appendix[A\.2](https://arxiv.org/html/2610.10918#A1.SS2)\), which was independently annotated using different annotator pools and annotation platform\.

#### 4\.1\.2Span Annotation Evaluation

Error categoryH\-HGemGPTDSQwenSpan overlap\.51\.43∗\.44∗\.39∗\.39∗Character overlap\.43\.39∗\.31∗\.27∗\.35∗Severity minor\.33\.17∗\.16∗\.13∗\.12∗Mistranslation\.37\.35∗\.37\.32∗\.34∗Severity major\.45\.43∗\.43∗\.31∗\.41∗Awkward\.19\.13∗\.12∗\.01∗\.02∗Grammar\.18\.19\.10∗\.13∗\.16∗Punctuation\.28\.04∗\.06∗\.02∗\.03∗Inappropriate for context\.07\.05\.05\.05\.06Omission\.36\.30∗\.14∗\.06∗\.15∗Spelling\.14\.05∗\.04∗\.01∗\.06∗Addition\.26\.21∗\.11∗\.11∗\.16∗Source error\.11\.00∗\.00∗\.00∗\.00∗Register\.00\.02∗\.01∗\.01\.01∗Character encoding\.28\.02∗\.01∗\.33\.06∗Inconsistency\.08\.04∗\.03∗\.01∗\.05Untranslated text\.12\.00∗\.00∗\.00∗\.00∗No error\.10\.12\.07∗\.00∗\.15∗Other\.02\.00∗\.00∗\.00∗\.00∗Non translation\.17\.00∗\.00∗\.00∗\.00∗Date format\.10\.09\.00∗\.05\.10Inconsistent use\.01\.00\.00\.00\.00Text Breaking\.02\.00\.00\.00\.00Name format\.04\.04\.00∗\.00∗\.00∗Currency format\.04\.00\.00\.00\.00Table 1:Span\-annotation F1 by error category on the WMT23 MQM test set, averaged over language pairs and ordered by decreasing category frequency\. H\-H is agreement between human annotation rounds; the remaining columns are each LLM’s agreement with the human rounds\. Gem = Gemini 2\.5 Flash, GPT = GPT\-OSS\-120B, DS = DeepSeek\-V3\.1, Qwen = Qwen3\.5\-397B\.∗marks a difference from H\-H that is statistically significant atp<0\.05p<0\.05\. Categories whose human\-human baseline rests on fewer than 5 source segments are omitted, since no comparison against it is estimable: Address format, Time format\.Table[1](https://arxiv.org/html/2610.10918#S4.T1)shows span annotation evaluation on the WMT23 test set in terms of F1 score, averaged across all language pairs\. Agreement levels for high\-frequency error categories are notably higher than low\-frequency categories\. In addition, human\-human agreement exceeds LLM\-human agreement for most span annotation evaluation metrics with some exceptions which include Mistranslation where GPT\-OSS\-120B\-human is on par with human\-human agreement and Grammar where Gemini 2\.5 Flash\-human agreement is on par with human\-human agreement\. Human\-human agreement on span overlap reaches at most F1 score = 0\.51, indicating the inherent difficulty of error span annotation\. Notably, the agreement levels for the lower\-frequency error categories are extremely low for all annotator combinations, with many categories achieving zero F1 score by all LLMs such as Currency Format\. This suggests that while LLMs show stronger agreement with human annotators for frequent error categories such as mistranslation, they particularly struggle to identify rarer error categories, possibly because these errors are underrepresented in the LLM training data\. In terms of error severity, both LLM\-human and human\-human agreements are lower for minor errors than major errors, which confirms error count results in Section[4\.1](https://arxiv.org/html/2610.10918#S4.SS1)\. When it comes to identifying error\-free segments \(the No Error category\), human\-human agreement is significantly outperformed by Qwen3\.5\-397B\-human agreement and is no different from Gemini 2\.5 Flash\-human agreement while both DeepSeek\-V3\.1\-human and GPT\-OSS\-120B\-human agreements are significantly inferior to human\-human agreement\. However, all agreements remain at or below 0\.15 in absolute terms\. This likely stems from the long\-context nature of the test segments and the high disagreement on annotating minor errors: in near\-perfect translations spanning multiple paragraphs, annotators frequently disagree on whether subtle imperfections constitute minor errors, leading to inconsistent error\-free classifications\. Notably, DeepSeek\-V3\.1\-human agreement on identifying error\-free segments is very low across all test sets \(Tables[3](https://arxiv.org/html/2610.10918#S4.T3)and[5](https://arxiv.org/html/2610.10918#A1.T5)\)\. Span annotation results on the Long\-Context MQM test set \(Appendix[A\.2\.2](https://arxiv.org/html/2610.10918#A1.SS2.SSS2)\) mirror this pattern with human\-human agreement exceeding LLM\-human for most categories, and the gap is widening for the low\-frequency ones\.

### 4\.2ESA

#### 4\.2\.1Score Evaluation

![Refer to caption](https://arxiv.org/html/2610.10918v1/fig_esa_alpha_heatmap.png)Figure 2:Krippendorff’sα\\alphaagreement on ESA scores for each language pair and domain on the WMT25 ESA test set\. Within each domain, the columns are H\-H \(Human with Human\), Gem\-H \(Gemini\-2\.5\-Flash with Human\), GPT\-H \(GPT\-OSS\-120B with Human\), DS\-H \(DeepSeek\-V3\.1 with Human\), Qwen\-H \(Qwen3\.5\-397B with Human\)\. The dashed line on the colorbar marksα=0\.667\\alpha=0\.667\.∗\*marks cells where LLM\-human agreement differs significantly from the human\-human agreement of the same cell \(paired cluster bootstrap on source segments,p<0\.05p<0\.05\)\.Figure[2](https://arxiv.org/html/2610.10918#S4.F2)presents ESA score agreement for each language pair and domain on the WMT25 ESA test set\. Consistent with MQM score results, ESA scores exhibit agreement below the Krippendorff’sα\\alphathreshold for tentative conclusions \(0\.667\) for nearly all language pairs and domains\. Exceptions include en\-arz\_EG across all domains, though as discussed below, the high en\-arz\_EG agreement reflects systematic issues rather than annotation quality\. Both human\-human and LLM\-human agreement vary substantially across domains within the same language pair, indicating that domain characteristics significantly influence annotation difficulty\. For a few language pairs such as English\-to\-Japanese \(en\-ja\_JP\), Qwen3\.5\-397B\-human and Gemini 2\.5 Flash\-human agreement exceeds human\-human agreement across all domains\. Conversely, for language pairs such as English\-to\-Egyptian Arabic \(en\-arz\_EG\) and English\-to\-Bhojpuri \(en\-bho\_IN\), human\-human agreement substantially outperforms LLM\-human agreement across all domains\. A notable result is English\-to\-Maasai \(en\-mas\_KE\), which has a remarkably low agreement across all annotator pairs consistent with its status as a low\-resource language pair\.

A striking finding emerges for en\-arz\_EG: LLM\-human agreement shows negative Krippendorff’sα\\alphafor LLMs which indicates systematic disagreement whereas human\-human agreement is remarkably high \(\>0\.90 for all domains\), indicating systematic agreement\. Manual examination of sample annotations reveals the root cause: Most translations \(60%\) for en\-arz\_EG are in the wrong Arabic variant \(Modern Standard Arabic instead of Egyptian Arabic\)222Percentage of en\-arz\_EG translations in wrong variant are estimated by calculating percentage of translations which has a score of 0 by one of the human annotators and a single error span, which is how annotators are instructed to treat translations with wrong language according to the ESA annotation instructions in Figure[7](https://arxiv.org/html/2610.10918#A1.F7)\. While humans consistently identified these samples and scored them as zero, the LLMs consistently assigned these translations a high score because they are accurate translations despite explicit instructions in the ESA prompt \(Figure[7](https://arxiv.org/html/2610.10918#A1.F7)\) to score translation in the wrong language/variant by zero\. This failure mode highlights a critical limitation in the LLMs’ ability to distinguish between closely related language variants—a capability essential for locale\-specific quality evaluation\.

Domain\#LPsGeminiGPT\-OSSDeepSeekQwenDialogue21/00/10/11/0Edu20/10/11/10/1Literary124/61/91/73/6News141/60/111/111/7Social142/72/100/81/9All domains448/203/323/286/23

Table 2:Language pairs per domain whose LLM\-human Krippendorff’sα\\alphaagreement on ESA scores differs significantly from human\-human agreement, on the WMT25 ESA test set, shown separately for each LLM\. The first number counts language pairs where LLM\-human Krippendorff’sα\\alphasignificantly exceeds human\-human agreement, the second those where human\-human significantly exceeds LLM\-human\. LPs with no significant difference in either direction appear in neither number\. Gemini = Gemini 2\.5 Flash, GPT\-OSS = GPT\-OSS\-120B, DeepSeek = DeepSeek\-V3\.1, Qwen = Qwen3\.5\-397B\. Benjamini–Hochberg FDR applied separately to each LLM’s comparisons,p<0\.05p<0\.05\.Table[2](https://arxiv.org/html/2610.10918#S4.T2)shows the number of language pairs per domain where LLM\-human agreement significantly exceeds human\-human agreement versus the number of language pairs where human\-human agreement significantly exceeds LLM\-human agreement for ESA scores on the WMT25 ESA test set\. In contrast to MQM score agreement results, human\-human ESA score agreement significantly exceeds LLM\-human agreement for the majority of language pairs and domains for most LLMs\. This might be due to the fact that ESA has a simpler annotation scheme where scores are input directly by annotators rather than calculated from error spans, which likely reduces disagreement whereas MQM calculates the scores by weighting the span counts which might elevate the variance\. Comparing the LLM\-human agreement between different models, Gemini 2\.5 Flash\-human agreement is higher than other models’ agreement for most language pairs and domains\. Notably, Gemini 2\.5 Flash\-human agreement significantly outperforms human\-human agreement for 4 out of 12 language pairs in the Literary domain whereas it significantly underperforms human\-human agreement for 6 out of 14 for the news domain and 7 out of 14 for the social domain\.

#### 4\.2\.2Span Annotation Evaluation

Table[3](https://arxiv.org/html/2610.10918#S4.T3)shows span annotation evaluation on the WMT25 ESA test set in terms of F1 score, averaged across all language pairs and domains\. Unlike MQM, ESA has fewer error categories to measure agreement on\. Similar to the MQM span annotation results in Section[4\.1\.2](https://arxiv.org/html/2610.10918#S4.SS1.SSS2), human\-human agreement is significantly better than LLM\-human agreement for all span annotation evaluation metrics\. In addition, both LLM\-human and human\-human agreements on major errors are higher than those of minor errors\. Both humans and LLMs achieve low agreement on the missing span category\. Notably, LLM\-human agreement is on the same level for most categories with a few exceptions such as DeepSeek\-V3\.1\-human agreement \(0\.05\) being substantially lower than other models’ agreement on identifying error\-free segments\.

Span agreementH\-HGemGPTDSQwenCharacter overlap\.43\.17∗\.16∗\.17∗\.18∗Span overlap\.45\.30∗\.31∗\.31∗\.31∗Minor severity\.23\.15∗\.12∗\.16∗\.15∗Major severity\.32\.22∗\.21∗\.21∗\.22∗No error\.47\.37∗\.25∗\.05∗\.29∗Missing span\.20\.15∗\.09∗\.07∗\.11∗Table 3:Span\-annotation F1 by span\-agreement measure on the WMT25 ESA test set, averaged over language pairs and domains and ordered by decreasing frequency\. H\-H is agreement between human annotation rounds; the remaining columns are each LLM’s agreement with the human rounds\. Gem = Gemini 2\.5 Flash, GPT = GPT\-OSS\-120B, DS = DeepSeek\-V3\.1, Qwen = Qwen3\.5\-397B\.∗marks a difference from H\-H that is statistically significant atp<0\.05p<0\.05\.

### 4\.3Annotation Quality Validation

Figure 3:Inter\-annotator agreement for MQM scores on the Long\-Context Web MQM test set for en\_US\-hi\_IN and en\_US\-th\_TH, showing Gemini 2\.5 Flash’s agreement with each individual human annotation round and agreement between human annotation rounds\.We observe that for many language pairs, there is a substantial difference in the LLM\-human agreement across human annotation rounds\. For example Figure[3](https://arxiv.org/html/2610.10918#S4.F3)shows Gemini 2\.5 Flash’s agreement with each human annotation round on MQM scores for English\-to\-Hindi \(en\_US\-hi\_IN\) and English\-to\-Thai \(en\_US\-th\_TH\)\. For these language pairs the human\-human agreement is negative or close to zero and Gemini 2\.5 Flash’s agreement with one of the human annotation rounds substantially exceeds its agreement with the other\. This suggests that one of the two human annotation rounds has issues which cause inconsistencies for these language pairs and that LLMs can serve as diagnostic tools for flagging such problematic rounds for further investigation\.

## 5Conclusion

We evaluate LLMs as MQM and ESA annotators using inter\-annotator agreement on a long\-context test set of 70 language pairs and the publicly available WMT23 and WMT25 data, investigating both score and error span annotation agreement across a variety of language pairs, domains and annotation settings, using four LLMs: Gemini 2\.5 Flash, GPT\-OSS\-120B, DeepSeek\-V3\.1 and Qwen3\.5\-397B\. We demonstrate that LLM\-human agreement exceeds human\-human agreement for some language pairs and domains, which highlights the potential of using LLMs for machine translation quality estimation\. However, our findings suggest that LLMs are not yet a reliable substitute for human MT evaluation\. First, LLM\-human agreement levels vary considerably between different language pairs and domains and between different LLMs\. This highlights the need for conducting thorough LLM\-human agreement tests with tight quality controls before using LLMs for MT evaluation in specific domains and language pairs\. Second, and more importantly, agreement levels remain well below the Krippendorff’sα\\alphathreshold for tentative conclusion agreement \(0\.667\), particularly for low\-resource language pairs, highlighting a substantial potential for enhancing both human and LLM annotation performance to achieve more reliable annotations\. Following[Calderon et al\. \(2025\)](https://arxiv.org/html/2610.10918#bib.bib3), we caution against taking LLM\-human agreement superiority over human\-human agreement as an indicator that LLMs can substitute humans in these settings, given the weak human reliability\.

Our results identify several key challenges facing both humans and LLMs for machine translation quality evaluation\. LLM annotators have particularly low agreement with human annotators on minor errors and translations in the wrong language variant\. Furthermore, LLMs struggle with error span annotation for all error categories, particularly rarer errors, suggesting that targeted prompt engineering or fine\-tuning may offer a path to improvement\. Human annotators are particularly challenged on fine\-grained MQM error annotation and low\-resource language pairs, which needs to be addressed through clearer guidelines, better annotator training and improved annotation processes that reduce annotator fatigue\. We also show that LLMs may be used as diagnostic tools to flag inconsistencies in human annotations\. Although the identified weaknesses could be addressed independently for LLMs and humans, we believe a combined LLM\-human collaborative framework that leverages both complementary strengths would offer a more promising path to reliable and efficient MT evaluation\.

## 6Limitations

One limitation of this work is that the Krippendorff’sα\\alphathreshold for reliable agreement is not well\-established for the fine\-grained MT quality annotations such as MQM\. The current widely used 0\.667 threshold in the research community for establishing tentative agreement might not be realistic for such a task with inherent difficulty and subjectivity\. More research is needed to establish realistic annotator reliability thresholds under different settings such as text length, domain and number of annotations\. Furthermore, more research is needed to identify the factors for low\-agreement levels for both humans and LLMs, particularly for long\-context documents, such as annotator fatigue for humans and attention dispersion for LLMs\.

For licensing reasons we cannot publish our internally\-annotated Long\-Context Web MQM test sets, which presents a significant limitation of the replicability of the MQM results presented in this work\. While we tried to cover a variety of language pairs, domains and annotation schemes in our work, more studies should be conducted to gauge the LLM ability for MT quality evaluation on larger data sets spanning a wider set of language pairs and domains\.

We presented our results using four LLMs, a single prompt in a zero\-shot setting and a simple heuristic for span correction\. With the growing variety of available LLMs, more evaluation needs to be conducted using different LLMs with ablation for different prompting strategies and a better approach for span localization\.

Data contamination is a concern with results run through modern LLMs\. The cut\-off dates for the models used in this paper \(GPT\-OSS\-120B: June 2024, Gemini 2\.5 Flash: January 2025, DeepSeek\-V3\.1: July 2025, Qwen3\.5\-397B: estimated mid\-to\-late 2025\) are likely before the release of the WMT25 ESA test set\. However, the WMT23 MQM test set might be included in the training data of these models, which might inflate the LLM\-human agreement\. Our Long\-Context Web MQM has never been publicly released so it cannot be in the training data of any of these models\.

Significance testing used in this paper examined the null hypothesis of human\-human and LLM\-human agreements being equal\. A more refined statistical testing approach should in addition establish equivalence between human\-human and LLM\-human agreements\.

Using LLMs for evaluation poses the risk of amplifying biases in evaluated models\. This was visible in our results in which LLMs failed to penalize translations in Modern Standard Arabic where Egyptian Arabic was expected, biasing towards the variant with more representation in the training data\.

## References

- Bavaresco et al\. \(2025\)Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni\. 2025\.[LLMs instead of human judges? a large scale empirical study across 20 NLP evaluation tasks](https://doi.org/10.18653/v1/2025.acl-short.20)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 238–255, Vienna, Austria\. Association for Computational Linguistics\.
- Benjamini and Hochberg \(1995\)Yoav Benjamini and Yosef Hochberg\. 1995\.[Controlling the false discovery rate: A practical and powerful approach to multiple testing](http://www.jstor.org/stable/2346101)\.*Journal of the Royal Statistical Society\. Series B \(Methodological\)*, 57\(1\):289–300\.
- Calderon et al\. \(2025\)Nitay Calderon, Roi Reichart, and Rotem Dror\. 2025\.[The alternative annotator test for LLM\-as\-a\-judge: How to statistically justify replacing human annotators with LLMs](https://arxiv.org/abs/2501.10970)\.*Preprint*, arXiv:2501\.10970\.
- Comanici et al\. \(2025\)Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan\-Jiang Jiang, and 3416 others\. 2025\.[Gemini 2\.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities](https://arxiv.org/abs/2507.06261)\.*Preprint*, arXiv:2507\.06261\.
- DeepSeek\-AI \(2024\)DeepSeek\-AI\. 2024\.[Deepseek\-v3 technical report](https://arxiv.org/abs/2412.19437)\.*Preprint*, arXiv:2412\.19437\.
- Freitag et al\. \(2021\)Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey\. 2021\.[Experts, errors, and context: A large\-scale study of human evaluation for machine translation](https://doi.org/10.1162/tacl_a_00437)\.*Transactions of the Association for Computational Linguistics*, 9:1460–1474\.
- Freitag et al\. \(2023\)Markus Freitag, Nitika Mathur, Chi\-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster\. 2023\.[Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent](https://doi.org/10.18653/v1/2023.wmt-1.51)\.In*Proceedings of the Eighth Conference on Machine Translation*, pages 578–628, Singapore\. Association for Computational Linguistics\.
- Guerreiro et al\. \(2023\)Nuno M\. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F\. T\. Martins\. 2023\.[xCOMET: Transparent machine translation evaluation through fine\-grained error detection](https://arxiv.org/abs/2310.10482)\.*Preprint*, arXiv:2310\.10482\.
- Juraska et al\. \(2024\)Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag\. 2024\.[MetricX\-24: The Google submission to the WMT 2024 metrics shared task](https://doi.org/10.18653/v1/2024.wmt-1.35)\.In*Proceedings of the Ninth Conference on Machine Translation*, pages 492–504, Miami, Florida, USA\. Association for Computational Linguistics\.
- Kasner et al\. \(2025\)Zdeněk Kasner, Vilém Zouhar, Patrícia Schmidtová, Ivan Kartáč, Kristýna Onderková, Ondřej Plátek, Dimitra Gkatzia, Saad Mahamood, Ondřej Dušek, and Simone Balloccu\. 2025\.[Large language models as span annotators](https://arxiv.org/abs/2504.08697)\.*Preprint*, arXiv:2504\.08697\.
- Kocmi et al\. \(2025\)Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstantin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Howard Lakougna, Jessica Lundin, Christof Monz, Kenton Murray, and 10 others\. 2025\.[Findings of the WMT25 general machine translation shared task: Time to stop evaluating on easy test sets](https://doi.org/10.18653/v1/2025.wmt-1.22)\.In*Proceedings of the Tenth Conference on Machine Translation*, pages 355–413, Suzhou, China\. Association for Computational Linguistics\.
- Kocmi and Federmann \(2023\)Tom Kocmi and Christian Federmann\. 2023\.[Large language models are state\-of\-the\-art evaluators of translation quality](https://aclanthology.org/2023.eamt-1.19/)\.In*Proceedings of the 24th Annual Conference of the European Association for Machine Translation*, pages 193–203, Tampere, Finland\. European Association for Machine Translation\.
- Kocmi et al\. \(2024\)Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova\. 2024\.[Error span annotation: A balanced approach for human evaluation of machine translation](https://doi.org/10.18653/v1/2024.wmt-1.131)\.In*Proceedings of the Ninth Conference on Machine Translation*, pages 1440–1453, Miami, Florida, USA\. Association for Computational Linguistics\.
- Krippendorff \(2004\)Klaus Krippendorff\. 2004\.*Content Analysis: An Introduction to Its Methodology \(second edition\)*\.Sage Publications\.
- Lavie et al\. \(2025\)Alon Lavie, Greg Hanneman, Sweta Agrawal, Diptesh Kanojia, Chi\-Kiu Lo, Vilém Zouhar, Frederic Blain, Chrysoula Zerva, Eleftherios Avramidis, Sourabh Deoghare, Archchana Sindhujan, Jiayi Wang, David Ifeoluwa Adelani, Brian Thompson, Tom Kocmi, Markus Freitag, and Daniel Deutsch\. 2025\.[Findings of the WMT25 shared task on automated translation evaluation systems: Linguistic diversity is challenging and references still help](https://doi.org/10.18653/v1/2025.wmt-1.24)\.In*Proceedings of the Tenth Conference on Machine Translation*, pages 436–483, Suzhou, China\. Association for Computational Linguistics\.
- Lommel et al\. \(2014\)Arle Lommel, Aljoscha Burchardt, and Hans Uszkoreit\. 2014\.Multidimensional quality metrics \(MQM\): A framework for declaring and describing translation quality metrics\.*Tradumàtica: tecnologies de la traducció*, 0:455–463\.
- OpenAI et al\. \(2025\)OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K\. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, and 107 others\. 2025\.[gpt\-oss\-120b & gpt\-oss\-20b model card](https://arxiv.org/abs/2508.10925)\.*Preprint*, arXiv:2508\.10925\.
- Papineni et al\. \(2002\)Kishore Papineni, Salim Roukos, Todd Ward, and Wei\-Jing Zhu\. 2002\.[BLEU: a method for automatic evaluation of machine translation](https://doi.org/10.3115/1073083.1073135)\.In*Proceedings of the 40th Annual Meeting on Association for Computational Linguistics*, ACL ’02, page 311–318, USA\. Association for Computational Linguistics\.
- Popović \(2015\)Maja Popović\. 2015\.[chrF: character n\-gram F\-score for automatic MT evaluation](https://doi.org/10.18653/v1/W15-3049)\.In*Proceedings of the Tenth Workshop on Statistical Machine Translation*, pages 392–395, Lisbon, Portugal\. Association for Computational Linguistics\.
- Qwen Team \(2026\)Qwen Team\. 2026\.[Qwen3\.5: Towards native multimodal agents](https://qwen.ai/blog?id=qwen3.5)\.
- Rei et al\. \(2020\)Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie\. 2020\.[COMET: A neural framework for MT evaluation](https://doi.org/10.18653/v1/2020.emnlp-main.213)\.In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 2685–2702, Online\. Association for Computational Linguistics\.
- Riley et al\. \(2025\)Parker Riley, Daniel Deutsch, Mara Finkelstein, Colten DiIanni, Juraj Juraska, and Markus Freitag\. 2025\.[MQM re\-annotation: A technique for collaborative evaluation of machine translation](https://arxiv.org/abs/2510.24664)\.*Preprint*, arXiv:2510\.24664\.
- Sellam et al\. \(2020\)Thibault Sellam, Dipanjan Das, and Ankur Parikh\. 2020\.[BLEURT: Learning robust metrics for text generation](https://doi.org/10.18653/v1/2020.acl-main.704)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 7881–7892, Online\. Association for Computational Linguistics\.
- Song et al\. \(2025\)Yixiao Song, Parker Riley, Daniel Deutsch, and Markus Freitag\. 2025\.[Enhancing human evaluation in machine translation with comparative judgment](https://arxiv.org/abs/2502.17797)\.*Preprint*, arXiv:2502\.17797\.
- Zhan et al\. \(2025\)Runzhe Zhan, Zhihong Huang, Xinyi Yang, Lidia S\. Chao, Min Yang, and Derek F\. Wong\. 2025\.[Are large reasoning models good translation evaluators? Analysis and performance boost](https://arxiv.org/abs/2510.20780)\.*Preprint*, arXiv:2510\.20780\.
- Zouhar et al\. \(2025\)Vilém Zouhar, Tom Kocmi, and Mrinmaya Sachan\. 2025\.[AI\-assisted human evaluation of machine translation](https://arxiv.org/abs/2406.12419)\.In*Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics*\.

## Appendix AAppendix

### A\.1Long\-Context Web MQM Human Annotation

The Long\-Context Web MQM test set was constructed through an annotation vendor that handled annotator recruitment, annotation project management and quality control, and task\-specific training\. The vendor owns and maintains a secure annotation platform with no association with third\-party providers or crowdsourcing\. We provided the vendor with annotation guidelines\. We also provided the vendor with annotation UI mock\-ups to build the annotation UI\. We kept direct communication with the vendor to address annotation issues and guideline understanding\. The vendor hired and compensated annotators at market rates in their respective countries\. Annotators were required to be fluent in the source languages and native\-level speakers in the translated languages\. For rare language pairs, linguists were hired if professional graders were not available to ensure full language coverage\. Two independent annotation rounds were conducted for each language pair\. The segments in each round were distributed among different annotators\. Each round was worked on by multiple annotators with the requirement that the same segment was annotated by two different annotators in the two rounds\. Table[6](https://arxiv.org/html/2610.10918#A1.T6)provides statistics about the number of segments, the number of annotators, the number of translation systems and average segment length per language pair in our Long\-Context Web MQM test set\.

### A\.2Results on Long\-Context Web MQM Test Set

#### A\.2\.1Score and Error Count Evaluation

Figure 4:Inter\-annotator agreement on the Long\-Context Web MQM test set, showing Krippendorff’sα\\alphafor MQM scores, major error counts, and minor error counts, averaged over the language pairs of each translation direction group \(en\-X, X\-en, X\-Y\)\.Figure[4](https://arxiv.org/html/2610.10918#A1.F4)shows MQM score, major and minor counts agreement averaged by language pair direction\. For MQM score agreement for en\-X language pairs, both Gemini 2\.5 Flash and GPT\-OSS\-120B\-human average agreements are not significantly different from average human\-human agreement while DeepSeek\-V3\.1 and Qwen3\.5\-397B\-human average agreements are significantly behind average human\-human agreement\. For X\-en language pairs human\-human average agreement is significantly larger than LLM\-human average agreement for all LLMs except for GPT\-OSS\-120B\. For X\-Y language pairs, all LLM\-human average agreements are significantly larger than human\-human average agreement for all LLMs except for DeepSeek\-V3\.1\. It is worth noting that the human\-human average agreement for X\-Y language pairs is notably lower than human\-human agreement for en\-X and X\-en language pair directions, which explains why LLM\-human average agreement is significantly larger than human\-human agreement for X\-Y language pairs for most LLMs\. Furthermore, all agreement levels are well below the 0\.667 threshold for tentative agreement\.

For major error counts, LLM\-human average agreement outperforms or is on par with human\-human average agreement for the three language pair directions for most LLMs\. By contrast, LLM\-human average agreement on minor error counts is significantly inferior to human\-human average agreement for all language pair directions for most LLMs, which is similar to the results on the WMT23 test set in Section[4\.1](https://arxiv.org/html/2610.10918#S4.SS1)\.

Direction\#LPsGeminiGPT\-OSSDeepSeekQwenen\-X194/24/40/161/4X\-en191/32/30/180/8X\-Y3220/219/43/219/6All7025/725/113/5510/18

Table 4:Language pairs per translation direction whose LLM\-human Krippendorff’sα\\alphaon MQM scores differs significantly from human\-human agreement, on the Long\-Context Web MQM test set\. The first number counts language pairs where LLM\-human Krippendorff’sα\\alphasignificantly exceeds human\-human agreement, the second those where human\-human significantly exceeds LLM\-human\. LPs with no significant difference in either direction appear in neither number\. Gemini = Gemini 2\.5 Flash, GPT\-OSS = GPT\-OSS\-120B, DeepSeek = DeepSeek\-V3\.1, Qwen = Qwen3\.5\-397B\. Benjamini–Hochberg FDR applied separately to each LLM’s comparisons,p<0\.05p<0\.05\.Error categoryH\-HGemGPTDSQwenSpan overlap\.36\.33∗\.24∗\.23∗\.28∗Character overlap\.28\.25∗\.18∗\.16∗\.20∗Severity minor\.27\.19∗\.15∗\.15∗\.18∗Mistranslation\.29\.30∗\.19∗\.12∗\.21∗Severity major\.20\.24∗\.16∗\.10∗\.17∗Awkward\.13\.06∗\.07∗\.03∗\.07∗Grammar\.10\.09∗\.07∗\.06∗\.09∗Omission\.24\.20∗\.11∗\.02∗\.04∗Spelling\.17\.09∗\.07∗\.05∗\.05∗Addition\.18\.13∗\.12∗\.07∗\.09∗No error\.18\.15∗\.14∗\.00∗\.09∗Punctuation\.18\.04∗\.05∗\.02∗\.04∗Register\.05\.04∗\.02∗\.01∗\.03∗Inconsistency\.13\.08∗\.08∗\.02∗\.08∗Untranslated text\.21\.09∗\.08∗\.08∗\.10∗Source error\.13\.06∗\.00∗\.00∗\.00∗Wrong Language\.21\.13∗\.09∗\.00∗\.06∗Name\.05\.00∗\.00∗\.00∗\.02∗Currency\.23\.10∗\.04∗\.09∗\.12∗Other\.04\.00∗\.00∗\.00∗\.02∗Date\.25\.13∗\.08∗\.09∗\.11∗Non translation\.05\.09∗\.06\.08\.06Time\.22\.07∗\.05∗\.06∗\.07∗Table 5:Span\-annotation F1 by error category on the Long\-Context Web MQM test set, averaged over language pairs and ordered by decreasing category frequency\. H\-H is agreement between human annotation rounds; the remaining columns are each LLM’s agreement with the human rounds\. Gem = Gemini 2\.5 Flash, GPT = GPT\-OSS\-120B, DS = DeepSeek\-V3\.1, Qwen = Qwen3\.5\-397B\.∗marks a difference from H\-H that is statistically significant atp<0\.05p<0\.05\. Categories whose human\-human baseline rests on fewer than 5 source segments are omitted, since no comparison against it is estimable: Address, Mismatched locale, Telephone\.Table[4](https://arxiv.org/html/2610.10918#A1.T4)shows the number of language pairs per language pair direction where LLM\-human agreement significantly exceeds human\-human agreement versus the number of language pairs where human\-human agreement significantly exceeds LLM\-human agreement for MQM scores on the Long\-Context Web MQM test set\. For en\-X and X\-en directions, LLM\-human and human\-human agreements are not significantly different for most language pairs for all LLMs except for DeepSeek\-V3\.1 which has an overall low agreement with humans\. By contrast for X\-Y language pairs, Gemini 2\.5 Flash and GPT\-OSS\-120B\-human agreement outperform human\-human agreement for most language pairs, which also stems from the low human\-human agreement in this direction as demonstrated in Figure[4](https://arxiv.org/html/2610.10918#A1.F4)\.

Figure[5](https://arxiv.org/html/2610.10918#A1.F5)presents MQM score, major error count and minor error count agreement results for individual language pairs within each direction\. Both LLM\-human and human\-human agreement vary substantially between different language pairs\. However, the overall agreement falls well below the Krippendorff’sα\\alpha= 0\.667 threshold for tentative conclusions for nearly all language pairs, with rare exceptions where human\-human agreement reaches this threshold, including Russian\-to\-English \(ru\_RU\-en\_US\), Simplified Chinese\-to\-English \(zh\_CN\-en\_US\) and Traditional Chinese\-to\-English \(zh\_TW\-en\_US\)\. Agreement is particularly low for Non\-English pairs, which predominantly comprise low\-resource language pairs\. For humans this might indicate issues in the expertise and/or training of the human annotators for these language pairs\. For LLMs, this is expected as lower\-resource language pairs would be underrepresented in the LLM training data\.

#### A\.2\.2Span Annotation Evaluation

Table[5](https://arxiv.org/html/2610.10918#A1.T5)presents F1 scores for error category identification across annotator pairs, averaged over all language pairs\. Similar to the WMT23 test set results in Section[4\.1\.2](https://arxiv.org/html/2610.10918#S4.SS1.SSS2), agreement levels are low across all categories and annotator pairs with agreement reaching at most F1 score = 0\.36\.

Human\-human agreement significantly exceeds LLM\-human agreement for all span evaluation metrics except for mistranslation, major errors and non\-translation where Gemini 2\.5 Flash\-human agreement significantly exceeds human\-human agreement\. All LLMs achieve low F1 for several low\-frequency categories and all annotators have notably low agreement on identifying error\-free segments \(F1 score≤\\leq0\.18\) consistent with the WMT23 test set results in Section[4\.1\.2](https://arxiv.org/html/2610.10918#S4.SS1.SSS2)\.

![Refer to caption](https://arxiv.org/html/2610.10918v1/fig_lct_alpha_heatmap.png)Figure 5:Krippendorff’sα\\alphaagreement on MQM scores, major and minor error counts for each of the 70 language pairs of the Long\-Context Web MQM test set, grouped by translation direction \(en\-X, X\-en, X\-Y\)\. Within each group the columns are H\-H \(Human with Human\), Gem\-H \(Gemini\-2\.5\-Flash with Human\), GPT\-H \(GPT\-OSS\-120B with Human\), DS\-H \(DeepSeek\-V3\.1 with Human\), Qwen\-H \(Qwen3\.5\-397B with Human\)\. The dashed line on the colorbar marksα=0\.667\\alpha=0\.667\.∗\*marks cells where LLM\-human agreement differs significantly from the human\-human agreement of the same cell \(paired cluster bootstrap on source segments,p<0\.05p<0\.05\)\.TranslationQualityEvaluation

\#\#TaskDescription

Youareanexperttranslationqualityevaluator\.

Youaregivenatextinthesourcelanguageandacorrespondingtranslationinthetargetlanguage\.

YourtaskistoevaluatethetranslationusingMultidimensionalQualityMetrics\(MQM\)accordingtoinstructionsdefinedbelow\.

\#\#MQMErrorAnnotationInstructions

Pleasefollowtheseguidelinestoannotatealltranslationerrors\.

Pleasereadthesourceandtranslationtextsandannotatetranslationerrorsonebyone,takingthecontextofthewholetextintoconsideration\.Foreachtranslationerrorpleaseprovidethefollowinginformation:

Span:selecttheerrorspan\(s\)

Category:selecttheerrorcategoryfromtheerrorcategoriesdefinedbelow\.

Severity:selecttheerrorseverityfromtheerrorseveritiesdefinedbelow\.

\#\#\#ErrorSeverityLevels

SeverityDescription

MajorErrorsthatmayconfuseormisleadthereaderduetosignificantchangeinmeaning\.

MinorErrorsthatdon’tleadtolossofmeaningandwouldn’tconfuseormisleadthereaderbutwouldbenoticed,woulddecreasestylisticquality,fluencyorclarity,orwouldmakethecontentlessappealing\.

\#\#\#ErrorCategories

ErrorCategoryDescription

Accuracy/AdditionTranslationincludesinformationnotpresentinthesource

Accuracy/OmissionTranslationismissingcontentfromthesource

Accuracy/MistranslationTranslationdoesnotaccuratelyrepresentthesource

Accuracy/WrongLanguageTranslationincludessegmentsinwrongtargetlanguage\.Thisdoesn’tincludescenarioslikenamesorquotesthatareintendedtobeinanotherlanguage\.

Accuracy/MismatchedLocaleTranslationincludessegmentsinwronglocalevariant\.

Accuracy/Untranslated\-textSourcetextwasleftuntranslatedinthetargetwhenatranslationisexpected

Accuracy/Non\-translationImpossibletoreliablycharacterizedistincttranslationerrorsbecausethetranslationisgarbledorisunrelatedtothesourcetext

Fluency/PunctuationIncorrectpunctuationforlocaleorstyle

Fluency/SpellingIncorrectspellingorcapitalization

Fluency/GrammarProblemswithgrammar,otherthanorthography

Fluency/RegisterWronggrammaticalregister\(eg,inappropriatelyinformalpronouns\)\.

Fluency/InconsistencyTranslationofthesameword/phraseisinconsistentacrossthetext

Style/AwkwardTranslationhasinappropriatelanguagestyledespitebeingaccurateandgrammatical

Localeconvention/AddressAddressformatdoesnotfollowtargetlocaleconventions

Localeconvention/CurrencyCurrencyformatdoesnotfollowtargetlocaleconventions

Localeconvention/DateDateformatdoesnotfollowtargetlocaleconventions

Localeconvention/NameNameformatdoesnotfollowtargetlocaleconventions

Localeconvention/TelephoneTelephonenumberformatdoesnotfollowtargetlocaleconventions

Localeconvention/TimeTimeformatdoesnotfollowtargetlocaleconventions

Other/Source\-errorAnerrorinthesource\.\(e\.g\.thetextisnotintheexpectedlanguage,isgarbled,nonsensical,hasspellingerrorsortyposornoncoherent\)

Other/OtherAnyotherissueinthesourceortranslation

\#\#OutputFormat

ReturnaJSONobjectwiththefollowingfield:

\-errors\(list\):listoftranslationerrorsfound\.Eacherrorisanobjectwith:

\-category\(string\):errorcategoryfromthepredefinedlistabove\(e\.g\."Accuracy/Mistranslation"\)

\-severity\(boolean\):trueforMajorerrors,falseforMinorerrors

\-other\(string,optional\):free\-textdescriptionwhencategoryis"Other/Other"

\-spans\(list\):oneormoreerrorspans,eachwith:

\-start\(integer\):characterindexofthefirstcharacterofthespan\(0\-based,relativetotheparagraph\)

\-end\(integer\):characterindexofthelastcharacterofthespan\(inclusive,relativetotheparagraph\)

\-paraIndex\(integer\):indexoftheparagraphcontainingthespan\(0\-based\)

\-spanText\(string\):theactualtextoftheselectedspan

\-isSource\(boolean\):trueifthespanisonthesourcetext,falseifonthetarget

Iftherearenoerrors,returnanemptylist:\{"errors":\[\]\}

\#\#TranslationtoEvaluate

Source\(<source\_language\>\):<source\_text\>

Translation\(<target\_language\>\):<target\_text\>

\#\#YourEvaluation

Figure 6:MQM LLM evaluation prompt template\.TranslationQualityEvaluation

\#\#TaskDescription

Youareanexperttranslationqualityevaluator\.

Youaregivenatextinthesourcelanguageandacorrespondingtranslationinthetargetlanguage\.

YourtaskistoevaluatethetranslationusingErrorSpanAnnotationaccordingtoinstructionsdefinedbelow\.

\#\#ErrorAnnotationInstructions

Selectthepartoftranslationwhereyouhaveidentifiedatranslationerror\.Chooseerrorseverity:

\-Minorerrors:Style,grammar,wordchoicecouldbebetterormorenatural\.

\-Majorerrors:Themeaningischangedsignificantlyand/orthepartisreallyhardtounderstand\.

Tips:

\-Omittedcontent:Ifthetranslationismissingcontentfromthesource,setbothstart\_iandend\_ito"missing"\.

\-Highlightthewordorgeneralareaoftheerror\(itdoesn’tneedtobeexact\)\.Usemultiplehighlightsfordifferenterrors\.

\-Payparticularattentiontothelanguage/variantbeingevaluated\.Ifthetranslationisinthewronglanguageorlanguagevariant,markitfullyandassignit0\.

\-Payparticularattentiontotranslationconsistencybetweentextsacrossthewholedocument\.

\-Ifthetranslationcontainsadditionaltext\(e\.g\."Hereisthetranslation"\)oralternativesecondarytranslation,markitasamajorerror\.

RatingScale\(frombesttoworst\):

\-Perfect\[95\-100\]:Themeaningofthetranslationiscompletelyconsistentwiththesourceandreadslikeitwasformulatedbyanativespeaker\.

\-Retainscoremeaningwithminormistakes\[67\-94\]:Thetranslatedsentenceretainsthecoremeaningofthesourcesentence,butmayhavesomeminorgrammarmistakes,awkwardformulationsand/ormightomitsomeinsignificantparts\.

\-Preservessomemeaningwithmajormistakes\[6\-66\]:Thetranslatedsentencepreservessomeofthemeaningofthesourcesentencebutmissessignificantparts,orcontainsmajorerrors\.

\-Nonsenseorunrelated\[0\-5\]:Completelynonsensetranslationthatisnotrelatedtothesourcestring\.

Withineachratingcategory,bettertranslationsshouldgethigherscores\.

\#\#OutputFormat

ReturnaJSONobjectwiththefollowingfields:

\-score\(float\):overallqualityscoreintherange\[0\-100\]

\-errors\(list\):listoftranslationerrorsfound\.Eacherrorisanobjectwith:

\-start\_i\(integeror"missing"\):characterindexofthefirstcharacteroftheerrorspan\(0\-based\),or"missing"iftheerrorhasnolocalizedspan\(e\.g\.omission\)

\-end\_i\(integeror"missing"\):characterindexofthelastcharacteroftheerrorspan\(inclusive\),or"missing"iftheerrorhasnolocalizedspan

\-spanText\(string\):theactualtextoftheselectedspan,ornullifmissing

\-severity\(boolean\):trueforMajorerrors,falseforMinorerrors

Iftherearenoerrors,returnanemptylist:\{"score":<score\>,"errors":\[\]\}

\#\#TranslationtoEvaluate

Source\(<source\_language\>\):<source\_text\>

Translation\(<target\_language\>\):<target\_text\>

\#\#YourEvaluation

Figure 7:ESA LLM evaluation prompt template adapted from ESA guidelines\([Kocmi et al\., 2024](https://arxiv.org/html/2610.10918#bib.bib13)\)Table 6:Statistics for Long\-Context MQM Test Sets\. Ann shows the number of different annotators worked on each language pair in the two rounds, Segs shows the number of segments in each language pair, Sys shows the number of translation systems per language pair and Len shows average source segment length in words \(w\) for space\-using languages or characters \(c\) for non\-space languages\.LPSegsSysAnnLenLPSegsSysAnnLenar\_AE\-en\_US100111136wfr\_FR\-es\_ES20046157wde\_DE\-en\_US200410136wfr\_FR\-it\_IT200411157wde\_DE\-es\_ES200413136wfr\_FR\-pt\_BR200419157wde\_DE\-fr\_FR200423136wfr\_FR\-zh\_CN200415157wde\_DE\-it\_IT200413136whi\_IN\-en\_US100111142wde\_DE\-ja\_JP200416136wid\_ID\-en\_US100110133wde\_DE\-pt\_BR200423136wit\_IT\-de\_DE200418142wde\_DE\-zh\_CN200419136wit\_IT\-en\_US200412140wen\_US\-ar\_AE10018138wit\_IT\-es\_ES200412142wen\_US\-de\_DE200319136wit\_IT\-fr\_FR200416142wen\_US\-es\_ES200318136wit\_IT\-zh\_CN200418142wen\_US\-fr\_FR200317136wja\_JP\-en\_US20049195cen\_US\-hi\_IN100111138wja\_JP\-ko\_KR200413390cen\_US\-id\_ID10017138wja\_JP\-pt\_BR200415390cen\_US\-it\_IT200322136wja\_JP\-zh\_CN20049390cen\_US\-ja\_JP200310136wko\_KR\-en\_US200410141wen\_US\-ko\_KR20035136wko\_KR\-ja\_JP20041799wen\_US\-nl\_NL10016138wko\_KR\-zh\_CN20041899wen\_US\-pl\_PL10016138wnl\_NL\-en\_US10015137wen\_US\-pt\_BR200319136wpl\_PL\-en\_US10015141wen\_US\-ru\_RU10014138wpt\_BR\-en\_US200419141wen\_US\-th\_TH10013138wpt\_BR\-es\_ES200417146wen\_US\-tr\_TR10018138wpt\_BR\-fr\_FR200415146wen\_US\-uk\_UA10015138wpt\_BR\-zh\_CN200419146wen\_US\-vi\_VN10016138wru\_RU\-en\_US10016139wen\_US\-zh\_CN20038136wth\_TH\-en\_US5013884cen\_US\-zh\_TW100110138wtr\_TR\-en\_US10018138wes\_ES\-de\_DE200418150wuk\_UA\-en\_US10019137wes\_ES\-en\_US200421136wvi\_VN\-en\_US10015144wes\_ES\-fr\_FR200418150wzh\_CN\-en\_US20048215ces\_ES\-it\_IT200421150wzh\_CN\-es\_ES20047276ces\_ES\-pt\_BR200420150wzh\_CN\-fr\_FR200417276ces\_ES\-zh\_CN20048150wzh\_CN\-ja\_JP20047276cfr\_FR\-de\_DE200426157wzh\_CN\-ko\_KR200413276cfr\_FR\-en\_US200417138wzh\_TW\-en\_US10016202c

Table 7:Statistics for the WMT23 MQM test set\. Len shows average source segment length in words \(w\) for space\-using languages or characters \(c\) for non\-space languages\.LPSegsSysAnnLenen\_US\-de\_DE1901310144wzh\_CN\-en\_US179168264c

††Apple and the Apple logo are trademarks of Apple Inc\., registered in the U\.S\. and other countries and regions\.