Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators

# Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators
Source: [https://arxiv.org/html/2610.33741](https://arxiv.org/html/2610.33741)
Roman BokhyanEkaterina EnikeevaArtem MekhralievStepan VysotskyNikolay Karpachev♣\\clubsuitYandex,♢\\diamondsuitHSE University

###### Abstract

LLM\-based evaluators have rapidly become a standard tool for machine translation evaluation, yet the design choices that shape their behavior remain poorly understood\. We present a systematic study of how instruction design, few\-shot demonstrations, backbone model choice, and multi\-step workflows affect span\-level and ranking\-based MT evaluation\. We collect new expert RATE annotations on WMT24 and WMT25 English–Russian translations, with transfer validation on MQM\- and ESA\-annotated benchmarks across five additional language pairs\. Our experiments reveal four robust findings: instruction structure shapes evaluator behavior, high\-quality demonstrations matter more than quantity, frontier backbones exhibit distinct evaluator profiles \(recall\-, precision\-, or ranking\-oriented\), and fusing outputs across diverse backbones outperforms single\-evaluator multi\-call setups\. Although evaluator design exposes a large combinatorial space, several discovered design principles transfer consistently across annotation protocols and language pairs\. We synthesize these findings intoOrbit, a family of LLM\-based MT evaluators\. Our strongest configurations achieve expert\-level performance in error span detection and in Accuracy and Fluency ranking at roughly20×20\\timeslower cost than expert human annotation\.

Table 1:Overview of human baselines, automatic baselines, and our configurations\. Prompt\-based baselines andOrbit\-SCuse GPT\-5\.4\.Orbit\-MCis a multi\-call setup that fuses three evaluator backbones with an LLM fuser\.## 1Introduction

Machine translation \(MT\) quality has improved rapidly in recent years, with frontier LLM\-based systems approaching professional human translation quality on many language pairs and domains\([Kocmi et al\., 2024a](https://arxiv.org/html/2610.33741#bib.bib13)\)\. As translation quality improves, evaluation increasingly becomes the bottleneck: identifying subtle translation errors requires trained annotators, substantial annotation time, and careful quality control\. Automatic metrics remain useful at scale, but still lag behind expert human judgment on fine\-grained translation assessment, particularly for span\-level error localization and diagnostic analysis\([Lavie et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib17);[Freitag et al\., 2021](https://arxiv.org/html/2610.33741#bib.bib5)\)\.

Large language models have recently emerged as strong MT evaluators\([Kocmi and Federmann, 2023b](https://arxiv.org/html/2610.33741#bib.bib15)\)\. Prior work has explored both score\-based approaches, which predict a single quality score for a translation\([Juraska et al\., 2024](https://arxiv.org/html/2610.33741#bib.bib9)\), and span\-based approaches, which attempt to identify concrete translation errors following human evaluation protocols such as MQM\([Freitag et al\., 2021](https://arxiv.org/html/2610.33741#bib.bib5)\)and ESA\([Kocmi et al\., 2024b](https://arxiv.org/html/2610.33741#bib.bib16)\)\. However, evaluator design choices — prompt structure, demonstrations, workflow decomposition, and aggregation — are typically treated as implementation details, despite substantial evidence from in\-context learning research that demonstration design strongly affects downstream behavior\([Liu et al\., 2022](https://arxiv.org/html/2610.33741#bib.bib19);[Agrawal et al\., 2023](https://arxiv.org/html/2610.33741#bib.bib1)\)\. PrExMe explores prompt variation for scalar\-score MT evaluation\([Leiter and Eger, 2024](https://arxiv.org/html/2610.33741#bib.bib18)\)\. We instead study protocol\-grounded evaluator design for fine\-grained span\-level and ranking\-based assessment against expert annotations\.

This leaves a central question largely unexplored: which evaluator design choices robustly improve LLM\-based MT evaluation quality across datasets, protocols, and annotator settings?

In this work, we study LLM\-based MT evaluation as a structured evaluator design problem rather than as a fixed prompting configuration\. We conduct a systematic study across four major design dimensions: instruction design, few\-shot demonstration design, backbone model selection, and multi\-step evaluator workflows\. Our focus is span\-level MT evaluation under the RATE protocol\([Popov et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib25)\), which provides fine\-grained error spans, severity annotations, and separate Accuracy and Fluency judgments\.

To support this study, we collect new expert annotations on English–Russian WMT24 and WMT25 translations under the RATE protocol\. Each translation is independently annotated by multiple professional linguists, enabling direct evaluation of error span detection against human annotations rather than relying only on scalar quality\-score correlations\. Span\-level evaluation is particularly important for diagnostic MT assessment, since it localizes concrete translation failures and provides actionable feedback for both system developers and human editors\.

Our experiments reveal four robust findings: instruction structure shapes evaluator behavior, high\-quality demonstrations matter more than quantity, frontier backbones exhibit distinct evaluator profiles \(recall\-, precision\-, or ranking\-oriented\), and multi\-model fusion substantially outperforms single\-call evaluation\. Although evaluator design exposes a large combinatorial space, several discovered design principles transfer consistently across annotation protocols and language pairs\. Across these experiments, our strongest configurations consistently exceed assessor\-level agreement and match expert\-level error span detection performance \(Table[1](https://arxiv.org/html/2610.33741#S0.T1)\)\.

We encode these design principles inOrbit, a family of evaluator configurations instantiating the most robust choices identified in our study\. We refer to the resulting single\-call and multi\-call configurations asOrbit\-SCandOrbit\-MCrespectively\. Our strongest configurations achieve expert\-level error span detection performance\.Orbit\-MCalso reaches expert\-level Accuracy and Fluency ranking \([Table7](https://arxiv.org/html/2610.33741#S4.T7)\) and exceeds assessor agreement on MQM ranking \([Table1](https://arxiv.org/html/2610.33741#S0.T1)\)\. We additionally provide a preliminary study of how expert annotators interact with LLM\-generated reports in collaborative annotation workflows \(Appendix[H](https://arxiv.org/html/2610.33741#A8)\)\.

Our contributions are: \(i\) A systematic study of evaluator design for fine\-grained LLM\-based MT evaluation\. \(ii\) Evidence that several evaluator design principles transfer across annotation protocols \(RATE, MQM, ESA\), annotator pools, and five additional WMT language directions: en\-de, zh\-en, cs\-de, en\-it, and en\-zh\. \(iii\) New expert RATE annotations on WMT24 and WMT25 English\-\-Russian translations spanning multiple annotator expertise levels\.111[https://huggingface\.co/datasets/yandex/orbit\-mt\-eval](https://huggingface.co/datasets/yandex/orbit-mt-eval)\(iv\)Orbit, a family of single\- and multi\-call MT evaluators whose strongest configuration achieves expert\-level error span detection performance and surpasses assessor\-level translation ranking while costing roughly20×20\\timesless than expert human annotation\.

## 2Background and Related Work

### 2\.1Human MT Evaluation Protocols

MQM\([Lommel et al\., 2014](https://arxiv.org/html/2610.33741#bib.bib21);[Freitag et al\., 2021](https://arxiv.org/html/2610.33741#bib.bib5)\)and ESA\([Kocmi et al\., 2024b](https://arxiv.org/html/2610.33741#bib.bib16)\)serve as de facto gold standards in WMT evaluation campaigns\. Both ask annotators to identify error spans in translations\. MQM additionally assigns each span an error category and severity, which are aggregated into a final score, while ESA removes categories and instead uses spans as guidance for a final 0–100 quality score\.

RATE\([Popov et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib25)\)extends this setup with five\-point severity labels, a streamlined semantic/non\-semantic taxonomy, separate Accuracy and Fluency scores, and an overall score derived from span severities\.

### 2\.2LLM\-Based MT Evaluation

LLM\-based evaluation has emerged as a general paradigm for natural language assessment\([Zheng et al\., 2023](https://arxiv.org/html/2610.33741#bib.bib26);[Gu et al\., 2026](https://arxiv.org/html/2610.33741#bib.bib7);[Liu et al\., 2023](https://arxiv.org/html/2610.33741#bib.bib20);[Bavaresco et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib2)\)\. In MT evaluation, existing approaches can be broadly divided into score\-based, span\-based, and supervised methods\.

Score\-based approaches such as GEMBA\-DA\([Kocmi and Federmann, 2023b](https://arxiv.org/html/2610.33741#bib.bib15)\)and MetricX\([Juraska et al\., 2024](https://arxiv.org/html/2610.33741#bib.bib9)\)predict a single quality score for a translation segment\. Prompt\-based span evaluators more closely follow human annotation workflows by asking LLMs to identify translation errors directly\. Canonical examples include GEMBA\-MQM\([Kocmi and Federmann, 2023a](https://arxiv.org/html/2610.33741#bib.bib14)\)and AutoMQM\([Fernandes et al\., 2023](https://arxiv.org/html/2610.33741#bib.bib4)\), which prompt LLMs using MQM\-style instructions and a small set of demonstrations\. Subsequent work extends this paradigm with decomposition, post\-editing validation, and rubric\-based prompting\([Lu et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib22);[Kim, 2025](https://arxiv.org/html/2610.33741#bib.bib11);[Lu et al\., 2024](https://arxiv.org/html/2610.33741#bib.bib23)\)\. Multi\-agent approaches such as M\-MAD\([Feng et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib3)\)further decompose evaluation into interacting specialist agents\.

PrExMe\([Leiter and Eger, 2024](https://arxiv.org/html/2610.33741#bib.bib18)\)systematically explores more than 720 prompt templates for open\-source LLM\-based MT and summarization metrics, showing that seemingly minor prompt and output\-format choices can alter metric behavior\. Our study has a different focus: we measure direct agreement with multiple professional experts on error localization and severity, and jointly study demonstrations, backbone choice, and multi\-call workflows in addition to instruction variation\.

Supervised approaches instead train models on MQM\-style annotations to predict scores, spans, or both\. For example, xCOMET\([Guerreiro et al\., 2024](https://arxiv.org/html/2610.33741#bib.bib8)\)combines sentence\-level scoring with span detection, while recent work such as GemSpanEval\([Juraska et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib10)\)adapts frontier LLMs to span\-level evaluation tasks\.

Span\-level results are sensitive to annotation quality and annotator expertise, while systematic comparisons against multiple professional experts remain limited\([Freitag et al\., 2021](https://arxiv.org/html/2610.33741#bib.bib5);[Lavie et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib17)\)\.

Concurrent work by[Perrella et al\. \(2026\)](https://arxiv.org/html/2610.33741#bib.bib24)proposes alternative span\-matching formulations, which we discuss in Appendix[E](https://arxiv.org/html/2610.33741#A5)\.

## 3Experimental Setup

### 3\.1Design Study Protocol

Evaluator design exposes a large combinatorial space of potentially interacting decisions, including instruction structure, few\-shot demonstration selection, workflow decomposition, aggregation, and backbone model choice\. Exhaustive joint optimization across all dimensions is impractical\. We therefore study design dimensions sequentially, fixing variables after each stage to isolate the effect of subsequent decisions\. Our objective is not to find a universally optimal evaluator configuration, but to identify design choices that consistently improve span\-level MT evaluation quality and transfer across protocols and language pairs\.

### 3\.2Dataset and Annotation Setup

We collect new annotations on 600 source–translation pairs from WMT24[Kocmi et al\. \(2024a\)](https://arxiv.org/html/2610.33741#bib.bib13)and WMT25[Kocmi et al\. \(2025\)](https://arxiv.org/html/2610.33741#bib.bib12)English–Russian translations, extending the RATE\-annotated coverage to newer frontier LLM translation systems\. Detailed dataset statistics are in Appendix[A](https://arxiv.org/html/2610.33741#A1)\.

##### Sources and translations\.

Source segments are sampled from WMT24 and WMT25 across multiple domains\. We select 300 source segments and pair each with two translations, enabling segment\-level pairwise ranking accuracy \([Section3\.4](https://arxiv.org/html/2610.33741#S3.SS4)\) alongside span\-level evaluation\. Rather than sampling uniformly, we apply diversity sampling to ensure coverage across translation systems and quality levels, producing a challenging evaluation set\.

##### Annotation layers and human baselines\.

Each translation receives annotation at three levels of expertise\. Professional expert linguists annotate the dataset under the full RATE protocol in implicit side\-by\-side mode, yielding three independent expert annotations per segment \(Expert×\\times3\)\. Assessors — Russian\-native annotators with verified C1 English proficiency but no formal translation training — follow the same protocol \(Assessor\)\. We additionally include standard ESA annotations from the WMT evaluation campaign \(WMT annotator\), providing a link to the public benchmark setting\.

For evaluation, we keep two of the three expert annotations as gold references\. HumanErepresents pairwise expert agreement, computed by evaluating the third expert against each gold annotation and averaging — the same procedure used for all evaluated systems\. HumanA\-SbSand HumanA\-PWcorrespond to assessors evaluated in implicit side\-by\-side and point\-wise setups respectively, while HumanWMTfollows standard ESA annotation conditions\. Full annotator criteria and statistics are in Appendix[C](https://arxiv.org/html/2610.33741#A3)\.

##### Few\-shot pool\.

Separately from the evaluation set, we construct a dedicated few\-shot calibration pool consisting of 8 source segments across multiple domains, each translated by 12 systems\. Professional linguists annotate the pool collaboratively under the RATE protocol with consensus reconciliation, yielding more reliable demonstrations than single\-annotator outputs\. This pool is used exclusively for few\-shot selection \([Section4\.2](https://arxiv.org/html/2610.33741#S4.SS2)\) and does not overlap with the evaluation set\.

##### Multilingual validation\.

To test whether discovered evaluator design choices transfer beyond English–Russian and the RATE protocol, we additionally evaluate our best single\-call configuration on five WMT language directions with publicly available annotations\. From the WMT23 Metrics Shared Task\([Freitag et al\., 2023](https://arxiv.org/html/2610.33741#bib.bib6)\), we use MQM annotations on en\-de and zh\-en, sampling 450 examples per direction\. From the WMT25 evaluation shared task\([Lavie et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib17)\), we use ESA annotations on cs\-de, en\-it, and en\-zh, sampling 300 examples per direction\. Full results are in[Section5](https://arxiv.org/html/2610.33741#S5)\.

### 3\.3LLM Baselines

We compare against baselines representing several major evaluator paradigms: concise prompting, detailed protocol prompting, decomposition\-based workflows, and supervised span\-level evaluation\.

Prompt\-based baselines include GEMBA\-MQM\([Kocmi and Federmann, 2023a](https://arxiv.org/html/2610.33741#bib.bib14)\), GEMBA\-ESA, a reference\-free LLM judge used in the WMT25 General MT shared task\([Kocmi et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib12)\), and UMB\-v5, our abbreviation for theunified\-mqm\-boosted\-v5schema released with[Perrella et al\. \(2026\)](https://arxiv.org/html/2610.33741#bib.bib24)\. The latter is a detailed prompt that thoroughly describes the annotation task\. We additionally include MQM\-AE and ESA\-AE as our own adaptations of MQM and ESA human annotation guidelines into LLM prompts, keeping deviations from the originals minimal\. As a supervised baseline we include xCOMET\-XXL\([Guerreiro et al\., 2024](https://arxiv.org/html/2610.33741#bib.bib8)\), trained on MQM\-style annotations to predict both scores and spans\. All evaluation prompts are available in our[Hugging Face release](https://huggingface.co/datasets/yandex/orbit-mt-eval)\. Omitted methods are discussed in Appendix[D](https://arxiv.org/html/2610.33741#A4)\.

### 3\.4Evaluation Metrics

We evaluate MT evaluators along two dimensions: span\-level error detection and segment\-level pairwise ranking accuracy\. Throughout the paper, MQM scores are derived from RATE annotations following[Popov et al\. \(2025\)](https://arxiv.org/html/2610.33741#bib.bib25): RATE severity levels 2–3 map to MQM minor errors and levels 4–5 to major errors\. As in the original RATE\-to\-MQM conversion, RATE severity 1 \(“negligible or arguable” errors\) is excluded because standard MQM contains no equivalent severity level\.

##### Span\-level metrics\.

Predicted error spans are matched to gold annotations via bipartite matching with character overlap \(MP withτ=1\\tau\{=\}1\)\. We report micro\-averaged recallRR, precisionPP, and harmonic meanF1F\_\{1\}, following[Perrella et al\. \(2026\)](https://arxiv.org/html/2610.33741#bib.bib24)\. Segment\-level macro\-averaging gives equal weight to each segment regardless of its number of annotated errors and can therefore overstate performance for low\-recall evaluators\. To evaluate severe\-error behavior, we additionally report two complementary diagnostics:R4\+R\_\{4\+\}, recall over gold spans of severity≥4\\geq 4regardless of predicted severity, andP4\+P\_\{4\+\}, precision for predictions assigned severity≥4\\geq 4against gold major\-error spans\. For MQM\- and ESA\-annotated data,R4\+R\_\{4\+\}andP4\+P\_\{4\+\}are computed over major and critical errors\. We verify robustness under alternative span\-matching formulations in Appendix[E](https://arxiv.org/html/2610.33741#A5)\.

##### Ranking accuracy\.

For each pair of translations of the same source, we measure whether the evaluator correctly predicts which translation is better across Accuracy \(Acc\), Fluency \(FL\), and MQM dimensions\.

Table 2:Effect of instruction design \(zero\-shot, GPT\-5\.2 backbone\)\. GEMBA systems use their original few\-shot demonstrations\. All other systems are zero\-shot\.

## 4Systematic Study of LLM Evaluation Design

### 4\.1Instruction Design

How closely should evaluator instructions mirror human annotation protocols? Existing prompt\-based evaluators adopt substantially different assumptions: GEMBA\-MQM and AutoMQM rely on concise language\-agnostic prompts, while UMB\-v5 follows the opposite strategy with a detailed protocol\-oriented specification\. Neither line of work ablates the instruction content itself, leaving the relationship between protocol detail and evaluator behavior unmeasured\.

We compare three families of protocol\-adapted prompts \(MQM\-AE, ESA\-AE, RATE\-AE\), with RATE\-AE ablated into three variants of decreasing detail: RATE\-AEfull\+excontains full category definitions with inline error\-span examples \(as category\-boundary disambiguators, not few\-shot demonstrations\), RATE\-AEfullkeeps definitions and severity anchors without examples, and RATE\-AEbriefcompresses to a minimal task description, severity scale, and output format\. All experiments in this and the following section \([Section4\.2](https://arxiv.org/html/2610.33741#S4.SS2)\) use GPT\-5\.2 with temperature 0\.

[Table2](https://arxiv.org/html/2610.33741#S3.T2)shows large differences across instruction variants, with span F1 ranging from 27\.0 \(GEMBA\-MQM\) to 39\.3 \(RATE\-AEfull\+ex\)\. MQM\- and ESA\-style prompts achieve high precision but low recall, including on high\-severity errors — indicating genuinely missed problems rather than annotation granularity differences\. GEMBA systems exhibit similar behavior despite using few\-shot demonstrations\. The strongest RATE variant improves span F1 and major\-error recovery over the protocol baselines in this GPT\-5\.2 comparison\. Appendix[G](https://arxiv.org/html/2610.33741#A7)further shows that all tested RATE variants outperform the protocol baselines on F1 for Claude Opus 4\.6 and Gemini 3 Flash\. This effect is not specific to RATE\-annotated gold: RATE\-AEbriefalso outperforms MQM\-AE on F1 andR4\+R\_\{4\+\}when evaluated against MQM\-annotated WMT data \([Section5](https://arxiv.org/html/2610.33741#S5)\)\.

Within the RATE family, the relationship between instruction detail and evaluator quality is non\-monotonic: removing the inline examples is costly, with RATE\-AEfullfalling behind even the much shorter RATE\-AEbrief\. This indicates that detailed taxonomy definitions alone overconstrain evaluator behavior, while compact instructions preserving only task\-critical guidance generalize more effectively\. At the same time, the strong performance of RATE\-AEfull\+exindicates that detailed protocols remain useful when augmented with concrete examples, foreshadowing the demonstration\-based calibration we study in[Section4\.2](https://arxiv.org/html/2610.33741#S4.SS2)\. Ranking accuracy remains stable across variants, indicating that segment\-level judgments are less sensitive to instruction structure than to the backbone \([Section4\.3](https://arxiv.org/html/2610.33741#S4.SS3)\)\.

Overall, human annotation guidelines transfer effectively to LLM evaluators, but exhaustive taxonomy descriptions become less important once high\-quality demonstrations or compact task specifications are available\. We use RATE\-AEfull\+exas our primary zero\-shot setup and revisit this choice in[Section4\.2](https://arxiv.org/html/2610.33741#S4.SS2)\.

Table 3:Effect of few\-shot annotation quality\. Each value is the mean over 3 runs\. Brackets show the corresponding minimum and maximum\.
### 4\.2Few\-Shot Design

#### 4\.2\.1Does Annotation Quality Matter?

We compare three sources of few\-shot demonstrations: WMT25 shared task ESA annotations \(manually mapped to the RATE taxonomy and severity scale by our expert annotators\), single\-expert RATE annotations on WMT24 from[Popov et al\. \(2025\)](https://arxiv.org/html/2610.33741#bib.bib25), and our collaboratively curated annotations produced by multiple experts with consensus reconciliation\.

[Table3](https://arxiv.org/html/2610.33741#S4.T3)shows that demonstration quality matters substantially more than demonstration presence alone\. WMT annotations push F1 below the zero\-shot baseline across all three runs \(37\.0 vs 39\.3 mean\), suggesting that noisy demonstrations can actively mislead the evaluator about what constitutes a translation error\. RATE\-WMT24 broadly matches the zero\-shot baseline on mean F1\. Only our collaboratively curated annotations consistently improve both overall span recovery and major\-error detection\.

In contrast, ranking metrics improve over the zero\-shot baseline more uniformly across demonstration sources, indicating that span\-level evaluator behavior is substantially more affected by demonstration design than pairwise ranking judgments\. We therefore fix our curated annotation pool for all subsequent few\-shot experiments\.

Figure 1:Effect of few\-shot example count \(kk\) across span\-level and ranking metrics\. Each condition uses 5 random draws\. Boxes show IQR, points show individual runs\.
#### 4\.2\.2How Many Examples?

With the demonstration source fixed, we next vary the number of examples \(k∈\{1,2,4,8\}k\\in\\\{1,2,4,8\\\},[Figure1](https://arxiv.org/html/2610.33741#S4.F1)\)\.

Increasingkkproduces only modest changes in mean performance\. Instead, the primary effect of larger few\-shot sets is variance reduction:k=1k\{=\}1exhibits noticeably less stable behavior, whilek≥2k\\geq 2yields broadly consistent results across independently sampled configurations\. The most consistent mean trend is a slight improvement in R4\+, suggesting that additional demonstrations primarily improve major\-error recovery rather than general error detection\. We adoptk=2k\{=\}2for the remaining experiments, revisiting selection strategy in[Section4\.2\.3](https://arxiv.org/html/2610.33741#S4.SS2.SSS3)\.

#### 4\.2\.3Coverage\-Based Selection

The previous experiments showed that evaluator behavior depends strongly on demonstration design, motivating a more structured selection strategy\. We construct compact few\-shot sets that span the protocol’s error categories and severity range\. We evaluatek∈\{2,3,4\}k\\in\\\{2,3,4\\\}, corresponding to approximately 85%, 96%, and 100% category/severity coverage\.

Table 4:Effect of coverage\-based few\-shot selection\. Upper block: varyingkkwith RATE\-AEfull\+ex\. Lower block:k=2k\{=\}2with varying instruction\.[Table4](https://arxiv.org/html/2610.33741#S4.T4)shows that compact coverage\-based selection already substantially improves span\-level quality over the zero\-shot setup\. Performance is broadly comparable acrosskk, with a mild recall–precision trade\-off: smallerkkfavors recall, while largerkkimproves severity\-aware precision at the cost of recall\. This is consistent with the earlier observation that demonstration quantity has limited effect on mean performance, primarily reducing run\-to\-run variance\.

The coverage criterion itself is defined without optimizing metric scores\. We selectk=2k\{=\}2as the smallest configuration with comparable validation performance, balancing quality, prompt complexity, and inference cost\.

Revisiting instruction choice under this setup reverses the zero\-shot result from[Section4\.1](https://arxiv.org/html/2610.33741#S4.SS1): RATE\-AEbriefnow slightly surpasses RATE\-AEfull\+exon span F1, suggesting that high\-quality few\-shot demonstrations partially substitute for detailed taxonomy descriptions\. Instruction design nevertheless remains important: replacing the RATE\-style prompts with GEMBA\-MQM or GEMBA\-ESA substantially reduces span\-level quality even under the same few\-shot configuration\.

We refer to this single\-call configuration — RATE\-AEbriefpaired with coverage\-basedk=2k\{=\}2demonstrations — asOrbit\-SC\.[Section4\.3](https://arxiv.org/html/2610.33741#S4.SS3)evaluates this configuration across a range of frontier LLM backbones\.

Table 5:Model comparison usingOrbit\-SCacross frontier models\.

### 4\.3Model Comparison

How do frontier LLM backbones compare as MT evaluators? Using theOrbit\-SCconfiguration, we evaluate a range of frontier models as evaluator backbones\.[Table5](https://arxiv.org/html/2610.33741#S4.T5)reports the main results\. Extended metrics are provided in Appendix[F](https://arxiv.org/html/2610.33741#A6)\.

Different frontier models exhibit distinct evaluator profiles, with no single backbone dominating all evaluation dimensions\. Recent GPT models are recall\-oriented: GPT\-5\.4 achieves the strongest span\-level recovery and major\-error detection, while GPT\-4\.1 provides similar recall behavior at substantially lower cost\. Claude Opus 4\.6 exhibits the strongest precision\-oriented profile, leading on severity\-aware precision and direct\-assessment ranking while remaining competitive on span F1\. Gemini 3\.1 Pro combines ranking and precision strengths: it achieves the strongest MQM ranking accuracy together with the highest overall precision among span\-level evaluators\.

At the other end of the spectrum, Grok\-4 and DeepSeek\-V3\.2 substantially over\-predict errors at a large precision cost, with DeepSeek\-V3\.2 in particular flagging more than twice as many errors per segment as HumanE\. The remaining open\-weight models \(DeepSeek\-R1, Kimi K2 variants, Qwen\-235B\) sit consistently below the top tier across both span\-level and ranking\-based evaluation\.

Overall, strong evaluator quality is multi\-dimensional: different models favor different trade\-offs between recall, precision, severity calibration, and ranking behavior\. We adopt GPT\-5\.4 as the default backbone in the remaining experiments\. Appendix[G](https://arxiv.org/html/2610.33741#A7)repeats the matched design comparison on Claude Opus 4\.6 and Gemini 3 Flash to test whether the broader design conclusions transfer beyond GPT\-5\.4\.

### 4\.4Point\-Wise vs\. Side\-by\-Side Evaluation

Human MT evaluation is commonly conducted in an implicit side\-by\-side \(SbS\) setting, where one annotator sees multiple translations of the same source\([Freitag et al\., 2021](https://arxiv.org/html/2610.33741#bib.bib5)\)\. Our mainOrbit\-SCsetup is point\-wise \(PW\): each call evaluates one source–translation pair independently\. We adapt it to SbS by placing two translations of the same source in one call and returning a separate report for each\. The few\-shot sources remain fixed, with the second translation added to each demonstration\.

[Table6](https://arxiv.org/html/2610.33741#S4.T6)shows a clear trade\-off\. SbS improves Accuracy and Fluency ranking for all three backbones and improves or matches MQM ranking, but does not improve spanF1F\_\{1\}\. Paired context shifts predictions toward higher precision and lower recall\. PW is therefore convenient for evaluating an individual MT system, whereas SbS is better suited to ranking a fixed set of systems\. Reversing the presentation order produced no statistically significant differences\.

Table 6:Point\-wise \(PW\)Orbit\-SCand its side\-by\-side \(SbS\) adaptation\.
### 4\.5Multi\-Call Workflows

We next study multi\-call workflows as a design dimension: do additional LLM calls improve evaluator quality? We compare three approaches: task decomposition, iterative self\-review, and fusion across multiple evaluators \([Table7](https://arxiv.org/html/2610.33741#S4.T7)\)\.

#### 4\.5\.1Task decomposition

Inspired by M\-MAD\-style decomposition\([Feng et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib3)\), we split annotation into semantic and non\-semantic passes\. Concatenating both outputs improves recall, particularly for major errors, but substantially hurts precision due to overlapping and conflicting annotations\. LLM fusion recovers precision at the cost of recall\. In our setting, decomposition alone provides limited benefit over the single\-call baseline\.

#### 4\.5\.2Self\-review

We add a second\-pass reviewer that re\-examines the first\-pass output, either only adding missed spans or also editing existing predictions\. Self\-review consistently improves recall, especially on major errors, but also introduces borderline predictions that reduce precision\. Allowing the reviewer to edit existing spans partially recovers this precision loss and yields a small F1 gain over the single\-call setup\.

#### 4\.5\.3Model\-diversity fusion

Fusion across diverse evaluator backbones produces the strongest overall gains\. We refer to this multi\-call configuration — fusing Claude Opus 4\.6, GPT\-5\.4, and Gemini 3\.1 Pro outputs via Claude Opus 4\.6 as fuser — asOrbit\-MC\.Orbit\-MCmatches or exceeds expert\-to\-expert agreement on span F1 and major\-error recovery, while maintaining strong precision on high\-severity errors\.

Unlike fixed\-label ensembles, these structured reports do not provide a canonical unit for majority voting: models may identify the same error with different span boundaries and descriptions, or assign conflicting categories, severities, and segment scores\. A deterministic rule would therefore first require a separate span\-matching policy and arbitrary conflict\-resolution choices\. The semantic fuser is used precisely to perform this reconciliation\.

Fusion quality depends strongly on the fuser backbone itself\. Using the same three constituent models, GPT\-5\.4 as fuser falls several F1 points below Claude, while Gemini 3\.1 Pro as fuser produces the strongest MQM ranking accuracy\. Claude substantially outperforms GPT\-5\.4 as a fusion model despite weaker standalone recall \([Section4\.3](https://arxiv.org/html/2610.33741#S4.SS3)\), suggesting that strong evaluator generation and strong evaluator aggregation are distinct capabilities\.

Table 7:Multi\-call strategies\. C = Claude Opus 4\.6, G = GPT\-5\.4, Gem = Gemini 3\.1 Pro\. Fused = LLM\-merged report\.Evaluator setupFormatUSD/10k seg\.Human, one annotator passProfessional expert RATESbS$48,920Assessor RATESbS$6,500Assessor RATEPW$6,865ORBIT\-SC, varying backbone and formatGemini 3 FlashPW$20GPT\-4\.1PW$60GPT\-5\.4PW$85GPT\-5\.4SbS$75Claude Opus 4\.6PW$345Gemini 3\.1 ProPW$385Orbit\-MC\(3 models \+ fuser\)PW$1,720Table 8:API cost and estimated cost of one human annotator pass per 10,000 evaluated segments\.Table 9:Multilingual transfer on WMT23 MQM \(en\-de, zh\-en\) and WMT25 ESA \(cs\-de, en\-it, en\-zh\) data\. DS = direction\-specific demonstrations: two few\-shot examples in the target translation direction\.
#### 4\.5\.4Sources of diversity

Within model\-diversity fusion, expanding from two to three constituent models improves overall quality, while expanding the fusion set to five models by adding recall\-oriented backbones \(Grok\-4, Kimi\-K2\) further increases recall but reduces precision and F1, consistent with the over\-prediction tendencies observed in[Section4\.3](https://arxiv.org/html/2610.33741#S4.SS3)\.

Temperature and strategy diversity also improve over the single\-call baseline but produce consistently smaller gains than model diversity, indicating that complementary evaluator profiles are more valuable than repeated sampling from the same backbone\.

Overall, multi\-call workflows benefit most from genuine evaluator diversity rather than from iteration or decomposition over a single backbone\. The multi\-call configurationOrbit\-MCtherefore adopts three\-model fusion \(Claude Opus 4\.6, GPT\-5\.4, Gemini 3\.1 Pro\) via Claude as fuser, achieving the strongest span\-detection profile across all tested configurations\.

### 4\.6Cost and Deployment Trade\-offs

[Table8](https://arxiv.org/html/2610.33741#S4.T8)compares human and API costs at a common scale of 10,000 evaluated segments\. We use this volume as a representative data\-collection campaign for training or adapting an automatic evaluator rather than a one\-off evaluation set\. A segment is one source–translation pair\. PW is the standardOrbit\-SCsetup, with one pair per call\. Its SbS adaptation places two translations of the same source in one call and returns a separate report for each\. The human PW and SbS conditions use the same visibility distinction\. Thus, 300 SbS tasks and 600 PW tasks both yield 600 segment\-level reports\.

Human estimates use observed aggregate labor for one annotator pass rather than extrapolating median completion times\. Scaling these totals to 10,000 reports gives roughly 3,542 expert\-hours in SbS, 1,625 assessor\-hours in SbS, and 1,717 assessor\-hours in PW\. Monetary estimates are rounded to the nearest $5\. TheOrbit\-MCtotal includes three evaluators and the fuser\.

At this scale, one pass by a professional expert costs about $48,920\. An assessor pass costs $6,500 in SbS and $6,865 in PW\.Orbit\-MCcosts roughly $1,720, corresponding to a4×4\\timescost reduction from the least expensive assessor setup and a28×28\\timesreduction from professional expert RATE\. Point\-wiseOrbit\-SCcosts $20–$385 depending on the backbone, a further44–86×86\\timesreduction relative toOrbit\-MCwhile retaining the strong results in[Table5](https://arxiv.org/html/2610.33741#S4.T5)\. For GPT\-5\.4, SbS costs roughly $75 compared with $85 for separate PW calls\. The corresponding quality comparison is reported in[Section4\.4](https://arxiv.org/html/2610.33741#S4.SS4)\.

These dollar totals still understate the practical gap\. Human evaluation requires recruiting and screening annotators who are native speakers of the target language and have C1/C2 proficiency in the source language\. For expert pools, formal training or a degree in linguistics or translation is also preferred\. Protocol training, calibration, scheduling, and ongoing quality control add further overhead\([Freitag et al\., 2021](https://arxiv.org/html/2610.33741#bib.bib5);[Kocmi et al\., 2024b](https://arxiv.org/html/2610.33741#bib.bib16);[Popov et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib25)\)\. Much of this infrastructure must be established again for each language direction, which delays evaluation and makes procedures harder to standardize across pools\. API\-based evaluation can reuse a common workflow across directions, although model quality may still vary across languages and domains\.

## 5Multilingual Transfer

We test whether the evaluator design choices identified on English–Russian transfer across annotation protocols, annotation densities, and language pairs\. We evaluateOrbit\-SCandOrbit\-MCon WMT23 MQM \(en\-de, zh\-en\) and WMT25 ESA \(cs\-de, en\-it, en\-zh\) benchmarks using the same inference\-time configurations developed on RATE English–Russian data\. Human2denotes a second annotator scored against the gold annotation\.

Across both MQM and ESA benchmarks \([Table9](https://arxiv.org/html/2610.33741#S4.T9)\), the main findings of our design study transfer consistently beyond the English–Russian setting\.Orbit\-SCremains competitive across all evaluated language pairs, whileOrbit\-MCpreserves its strong recall\-oriented profile, improving major\-error recovery overOrbit\-SCon en\-de and remaining comparable on zh\-en\.

Compact RATE\-style prompting remains competitive on MQM\. Replacing the en\-ru few\-shot pool with direction\-specific demonstrations \(DS\) provides no substantial gain overOrbit\-SCon either MQM direction\.

Notably, the en\-ru pool matches direction\-specific demonstrations on en\-de spanF1F\_\{1\}while improvingR4\+R\_\{4\+\}and MQM, and improves all three metrics on zh\-en\. These results suggest that the advantage comes not from Russian itself but from how the pool was constructed\. The pool covers diverse error categories and severities, with annotations collaboratively reconciled by multiple experts\.

Transfer to ESA benchmarks exhibits substantially lower precision across all evaluators, likely reflecting the much sparser annotation density of ESA compared to MQM and RATE\. On spanF1F\_\{1\},Orbit\-MCprovides smaller gains than on MQM and is slightly belowOrbit\-SCon en\-it and en\-zh, reflecting that multi\-model fusion expands prediction sets in a regime where additional spans are heavily penalized\. Nevertheless, transfer gains remain clearest on major\-error recovery, whereOrbit\-MCconsistently exceeds Human2across all ESA directions\.

## 6Conclusion

We systematically study how instruction design, demonstrations, backbone choice, and multi\-call workflows shape LLM\-based MT evaluation across RATE, MQM, ESA, and multiple language pairs\. Curated, collaboratively reconciled demonstrations can match or outperform larger pools, model families show distinct evaluator profiles, and fusion yields the strongest overall results\.Orbitinstantiates these findings as single\- and multi\-call evaluators\. Its strongest configurations achieve expert\-level error span detection and Accuracy/Fluency ranking, whileOrbit\-MCexceeds assessor agreement on MQM ranking\.

## Limitations

##### Language coverage\.

Our main design study is conducted on a single high\-resource language pair \(English–Russian\)\. We validate the resultingOrbit\-SCandOrbit\-MCconfigurations on five additional WMT directions \(en\-de and zh\-en for MQM, and cs\-de, en\-it, and en\-zh for ESA\), but coverage of low\-resource pairs, non\-Latin scripts beyond Russian and Chinese, and language families outside Indo\-European and Sino\-Tibetan remains limited\. Findings on instruction language and few\-shot example selection in particular may require adaptation for typologically distant target languages\.

##### Closed\-source dependencies\.

Our strongest configurations \(Orbit\-SCwith GPT\-5\.4,Orbit\-MCfusing Claude Opus 4\.6, GPT\-5\.4, and Gemini 3\.1 Pro\) depend on closed\-source, API\-only models whose weights and training procedures are not public\. This affects reproducibility \(exact behaviour may shift across silent model updates\) and longevity \(specific model snapshots may be deprecated\)\. We mitigate the reproducibility concern through the broad cross\-model comparison in[Section4\.3](https://arxiv.org/html/2610.33741#S4.SS3), showing that qualitative conclusions hold across model families\.

##### Span\-matching methodology\.

Span\-levelF1F\_\{1\}depends on the choice of span\-matching formulation\. We use MP withτ=1\\tau\{=\}1as the primary formulation because our main question is whether the evaluator identifies the annotated error location\. MPP assigns partial credit in proportion to character overlap and is therefore more sensitive to span\-boundary agreement\. Appendix[E](https://arxiv.org/html/2610.33741#A5)reports MPP and LLM\-based reconciliation as complementary sensitivity analyses\.

##### Severity calibration\.

Appendix[F](https://arxiv.org/html/2610.33741#A6)shows that severity assignment is a distinct capability from error detection, with both human experts and LLM evaluators substantially under\-rating major errors when stricter severity thresholds are required\. We report detection\-orientedR4\+/P4\+R\_\{4\+\}/P\_\{4\+\}as primary metrics in the main paper because they isolate the more fundamental question of error localization, but downstream use cases that depend on accurate severity assignment \(e\.g\., quality estimation thresholds, automatic gating\) require additional severity calibration that is not the focus of this work\.

##### Human–LLM collaboration study\.

Our human–LLM collaboration experiments \(Appendix[H](https://arxiv.org/html/2610.33741#A8)\) are exploratory in scope: they measure annotation revision rather than ground\-truth correctness, use onlyOrbit\-MCas the reference configuration, and were conducted with a limited annotator group on a single language pair\. Whether the observed shifts in annotation behavior correspond to improved annotation quality, and how the effects generalize across configurations and annotator pools, remains to be studied\.

##### Subjectivity of MT evaluation\.

Translation quality assessment remains partly subjective even among trained experts\. The agreement between our expert annotators \([Table1](https://arxiv.org/html/2610.33741#S0.T1)\) provides a human\-agreement reference point for how closely any evaluator — human or LLM\-based — can be expected to match a single “ground truth”\. We treat expert agreement as the headline target rather than a hard ceiling, and recognize that ranking differences within this band may not reflect meaningful quality differences\.

## Ethics Statement

##### Annotator compensation and consent\.

RATE annotators received compensation at standard commercial translation rates for their region, with expert annotators earning approximately double the hourly wage of assessors, reflecting the higher qualifications required\. No personal data was collected during the annotation process, and all source segments were screened for potentially disturbing material prior to annotation\. All annotators were informed about how their annotations would be used — including for the development and evaluation of LLM\-based MT evaluators — prior to beginning the tasks\.

##### Human–LLM collaboration study\.

The collaborative annotation study in Appendix[H](https://arxiv.org/html/2610.33741#A8)followed the same compensation and consent procedures\. Annotators were informed that they would be reviewing LLM\-generated reports as part of their workflow, and that their pre\- and post\-hint annotations would be compared\. No identifying information about annotators is reported in our analyses\.

##### Annotator feedback\.

Following the annotation, we administered a feedback questionnaire\. The vast majority of annotators reported a positive experience and found the instructions clear\.

##### Use of closed\-source models\.

Our strongest configurations rely on closed\-source LLMs accessed via commercial APIs\. We discuss reproducibility and longevity implications of this choice in the Limitations section\.

## References

- Agrawal et al\. \(2023\)Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad\. 2023\.In\-context examples selection for machine translation\.In*Findings of the Association for Computational Linguistics: ACL 2023*\.
- Bavaresco et al\. \(2025\)Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, André F\. T\. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni\. 2025\.[LLMs instead of human judges? a large scale empirical study across 20 NLP evaluation tasks](https://doi.org/10.18653/v1/2025.acl-short.20)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 238–255, Vienna, Austria\. Association for Computational Linguistics\.
- Feng et al\. \(2025\)Zhaopeng Feng, Jiayuan Su, Jiamei Zheng, Jiahan Ren, Yan Zhang, Jian Wu, Hongwei Wang, and Zuozhu Liu\. 2025\.M\-MAD: Multidimensional multi\-agent debate for advanced machine translation evaluation\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*\. Association for Computational Linguistics\.
- Fernandes et al\. \(2023\)Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F\. T\. Martins, Graham Neubig, Ankush Garg, Jonathan H\. Clark, Markus Freitag, and Orhan Firat\. 2023\.The devil is in the errors: Leveraging large language models for fine\-grained machine translation evaluation\.In*Proceedings of the Eighth Conference on Machine Translation \(WMT\)*, Singapore\. Association for Computational Linguistics\.
- Freitag et al\. \(2021\)Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey\. 2021\.Experts, errors, and context: A large\-scale study of human evaluation for machine translation\.*Transactions of the Association for Computational Linguistics*, 9:1460–1474\.
- Freitag et al\. \(2023\)Markus Freitag, Nitika Mathur, Chi\-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frédéric Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster\. 2023\.Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent\.In*Proceedings of the Eighth Conference on Machine Translation \(WMT\)*, pages 578–628, Singapore\. Association for Computational Linguistics\.
- Gu et al\. \(2026\)Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Zhouchi Lin, Bowen Zhang, Lionel Ni, Wen Gao, Yuanzhuo Wang, and Jian Guo\. 2026\.[A survey on LLM\-as\-a\-judge](https://doi.org/10.1016/j.xinn.2025.101253)\.*The Innovation*, 7\(6\):101253\.
- Guerreiro et al\. \(2024\)Nuno M\. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F\. T\. Martins\. 2024\.xCOMET: Transparent machine translation evaluation through fine\-grained error detection\.*Transactions of the Association for Computational Linguistics \(TACL\)*\.
- Juraska et al\. \(2024\)Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag\. 2024\.MetricX\-24: The Google submission to the WMT 2024 metrics shared task\.In*Proceedings of the Ninth Conference on Machine Translation \(WMT\)*\.
- Juraska et al\. \(2025\)Juraj Juraska, Tobias Domhan, Mara Finkelstein, Tetsuji Nakagawa, Geza Kovacs, Daniel Deutsch, Pidong Wang, and Markus Freitag\. 2025\.MetricX\-25 and GemSpanEval: Google Translate submissions to the WMT25 evaluation shared task\.In*Proceedings of the Tenth Conference on Machine Translation \(WMT\)*, Suzhou, China\. Association for Computational Linguistics\.
- Kim \(2025\)Ahrii Kim\. 2025\.RUBRIC\-MQM: Span\-level LLM\-as\-judge in machine translation for high\-end models\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\)*, pages 147–165, Vienna, Austria\. Association for Computational Linguistics\.
- Kocmi et al\. \(2025\)Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstantin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Howard Lakougna, Jessica M\. Lundin, Christof Monz, Kenton Murray, and 10 others\. 2025\.[Findings of the WMT25 general machine translation shared task: Time to stop evaluating on easy test sets](https://aclanthology.org/2025.wmt-1.22/)\.In*Proceedings of the Tenth Conference on Machine Translation \(WMT\)*, Suzhou, China\. Association for Computational Linguistics\.
- Kocmi et al\. \(2024a\)Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popović, and 3 others\. 2024a\.Findings of the WMT24 general machine translation shared task: The LLM era is here but MT is not solved yet\.In*Proceedings of the Ninth Conference on Machine Translation \(WMT\)*, pages 1–46, Miami, Florida, USA\. Association for Computational Linguistics\.
- Kocmi and Federmann \(2023a\)Tom Kocmi and Christian Federmann\. 2023a\.GEMBA\-MQM: Detecting translation quality error spans with GPT\-4\.In*Proceedings of the Eighth Conference on Machine Translation \(WMT\)*, pages 768–775, Singapore\. Association for Computational Linguistics\.
- Kocmi and Federmann \(2023b\)Tom Kocmi and Christian Federmann\. 2023b\.Large language models are state\-of\-the\-art evaluators of translation quality\.In*Proceedings of the 24th Annual Conference of the European Association for Machine Translation \(EAMT\)*, pages 193–203, Tampere, Finland\.
- Kocmi et al\. \(2024b\)Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova\. 2024b\.Error span annotation: A balanced approach for human evaluation of machine translation\.In*Proceedings of the Ninth Conference on Machine Translation \(WMT\)*, Miami, Florida, USA\. Association for Computational Linguistics\.
- Lavie et al\. \(2025\)Alon Lavie, Greg Hanneman, Sweta Agrawal, Diptesh Kanojia, Chi\-Kiu Lo, Vilém Zouhar, Frederic Blain, Chrysoula Zerva, Eleftherios Avramidis, Sourabh Deoghare, Archchana Sindhujan, Jiayi Wang, David Ifeoluwa Adelani, Brian Thompson, Tom Kocmi, Markus Freitag, and Daniel Deutsch\. 2025\.[Findings of the WMT25 shared task on automated translation evaluation systems: Linguistic diversity is challenging and references still help](https://doi.org/10.18653/v1/2025.wmt-1.24)\.In*Proceedings of the Tenth Conference on Machine Translation*, pages 436–483, Suzhou, China\. Association for Computational Linguistics\.
- Leiter and Eger \(2024\)Christoph Leiter and Steffen Eger\. 2024\.[PrExMe\! large scale prompt exploration of open source LLMs for machine translation and summarization evaluation](https://doi.org/10.18653/v1/2024.emnlp-main.641)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 11481–11506, Miami, Florida, USA\. Association for Computational Linguistics\.
- Liu et al\. \(2022\)Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen\. 2022\.What makes good in\-context examples for GPT\-3?In*Proceedings of Deep Learning Inside Out \(DeeLIO 2022\): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures*\.
- Liu et al\. \(2023\)Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu\. 2023\.G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 2511–2522, Singapore\. Association for Computational Linguistics\.
- Lommel et al\. \(2014\)Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt\. 2014\.[Multidimensional quality metrics \(MQM\): A framework for declaring and describing translation quality metrics](https://doi.org/10.5565/rev/tradumatica.77)\.*Tradumàtica*, pages 455–463\.No\. 12\.
- Lu et al\. \(2025\)Qingyu Lu, Liang Ding, Kanjian Zhang, Jinxia Zhang, and Dacheng Tao\. 2025\.MQM\-APE: Toward high\-quality error annotation predictors with automatic post\-editing in LLM translation evaluators\.In*Proceedings of the 31st International Conference on Computational Linguistics \(COLING\)*\. Association for Computational Linguistics\.
- Lu et al\. \(2024\)Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao\. 2024\.Error analysis prompting enables human\-like translation evaluation in large language models\.In*Findings of the Association for Computational Linguistics: ACL 2024*, pages 8801–8816\. Association for Computational Linguistics\.
- Perrella et al\. \(2026\)Stefano Perrella, Eric Morales Agostinho, and Hugo Zaragoza\. 2026\.Span\-level machine translation meta\-evaluation\.*arXiv preprint arXiv:2603\.19921*\.
- Popov et al\. \(2025\)Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev, and Max Ryabinin\. 2025\.Refined assessment for translation evaluation: Rethinking machine translation evaluation in the era of human\-level systems\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 22079–22095, Suzhou, China\. Association for Computational Linguistics\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\. 2023\.Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.

## Appendix ADataset Statistics

The design study dataset comprises 600 source–translation pairs constructed from 300 English source segments sampled from WMT24–WMT25 English–Russian shared task data, each paired with two system translations\. Source selection prioritizes challenging segments where candidate translations disagree, ensuring evaluators must discriminate between subtle quality differences rather than identify obvious failures\.

The few\-shot pool uses separately selected sources and does not overlap with this 600\-pair evaluation set\.

##### Domain coverage\.

The dataset covers four domains \([Table10](https://arxiv.org/html/2610.33741#A1.T10)\), with literary, news, and speech segments slightly over\-represented relative to social media text\.

Table 10:Domain distribution of source–translation pairs\.
##### Translation systems\.

The 600 segments span 14 translation systems \([Table11](https://arxiv.org/html/2610.33741#A1.T11)\) including frontier closed\-source LLMs \(Claude\-3\.5/4, GPT\-4/4\.1, Gemini 2\.5\), production MT systems \(Dubformer, ONLINE\-G, Yandex Translate\), open\-weight LLMs \(Llama3\-70B, DeepSeek\-V3\), dedicated MT systems \(Unbabel\-Tower70B\), and human reference translations from the WMT campaign\.

Table 11:Translation system distribution\.
##### Annotation density\.

Each translation is annotated independently by three expert linguists under the RATE protocol \(Expert×\\times3\)\. Combining two annotations, translations contain on average10\.6410\.64error spans per segment \([Table12](https://arxiv.org/html/2610.33741#A1.T12)\), with severity 2 \(minor errors\) most frequent followed by severity 3 and severity 4 \(major\)\. Critical errors \(severity 5\) and trivial errors \(severity 1\) together account for roughly11%11\\%of annotated spans\. The per\-individual annotator rate is approximately halved \(5\.485\.48spans per segment\), consistent with partial overlap across independent expert annotations\.

Table 12:Mean error spans per segment by severity\. “Combined” aggregates two independent expert annotations\. Per\-annotator columns show individual outputs\.
##### Category distribution\.

[Table13](https://arxiv.org/html/2610.33741#A1.T13)reports the distribution across RATE error categories\. Fluency errors dominate \(6\.066\.06spans per segment, combined\), followed by Mistranslation \(2\.342\.34\)\. All other categories average below11span per segment\. This long\-tailed distribution motivates coverage\-based few\-shot selection \([Section4\.2](https://arxiv.org/html/2610.33741#S4.SS2)\): uniform random sampling would oversample Fluency and Mistranslation while underrepresenting rarer categories\.

Table 13:Mean error spans per segment by RATE category, aggregated across the two expert gold annotations\.Table 14:Span\-level metrics under three span\-matching formulations: MP withτ=1\\tau\{=\}1\(primary\), MPP\([Perrella et al\., 2026](https://arxiv.org/html/2610.33741#bib.bib24)\)\(partial overlap with partial credit\), and an LLM reconciliation proxy using GPT\-5\.4\. All metrics are micro\-averaged\.Table 15:Extended model comparison usingOrbit\-SC\(RATE\-AEbriefwith coverage\-basedk=2k\{=\}2demonstrations\)\. Non\-trivial, Major, and Critical report mean predicted spans per segment at severity≥2\\geq 2,≥4\\geq 4, and=5=5respectively\.R4\+R\_\{4\+\}measures recovery of gold major spans regardless of predicted severity, whileP4\+P\_\{4\+\}uses predictions assigned severity≥4\\geq 4\. The bullet variants use predicted severity≥3\\geq 3, providing one\-level tolerance around the major threshold\.
##### Source length and overall quality\.

The average source segment length is377377characters\. The mean RATE overall score \(a derived 0–10 quality score\) is9\.309\.30, indicating that the dataset contains publication\-grade translations rather than failure\-mode outputs\. Mean annotated span coverage per translation is6262characters, roughly16%16\\%of source length, with errors concentrated in localized regions rather than spread diffusely\.

## Appendix BArtifact Use and Licenses

WMT24 General MT data is released by the WMT organizers under terms permitting research use\. WMT23 MQM annotations\([Freitag et al\., 2023](https://arxiv.org/html/2610.33741#bib.bib6)\)are released under the Apache 2\.0 license via thegoogle/wmt\-mqm\-human\-evaluationrepository\. WMT25 General MT data, including ESA annotations\([Lavie et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib17)\), is publicly released by the WMT organizers via thewmt\-conference/wmt25\-general\-mtrepository for research purposes\. xCOMET\-XXL\([Guerreiro et al\., 2024](https://arxiv.org/html/2610.33741#bib.bib8)\)is released under CC\-BY\-NC\-SA\-4\.0\. We use it for non\-commercial research evaluation, consistent with its license terms\. Closed\-source LLM APIs \(GPT, Claude, Gemini\) were accessed in accordance with their respective terms of service\.

Our use of all existing artifacts is consistent with their intended research and non\-commercial evaluation purposes\. The expert\-annotated dataset, evaluation prompts, and accompanying paper artifacts are available at[https://huggingface\.co/datasets/yandex/orbit\-mt\-eval](https://huggingface.co/datasets/yandex/orbit-mt-eval)\.

## Appendix CAnnotator Pool

##### Expert annotators\.

Our Expert tier consists of professional translators who are native Russian speakers with verified C1/C2 English proficiency and substantial industry experience\. A university degree in linguistics or translation studies was preferred\. Candidates passed a multi\-stage selection process: a post\-editing test that required context analysis and source fact\-checking, followed by an interview\. After selection, experts received structured training on the RATE protocol — error categorization, severity calibration, and the Accuracy/Fluency scoring scales — with concrete examples for each category\.

##### Assessor\.

Our Assessor tier consists of in\-house crowd\-level evaluators: native Russian speakers with verified C1 English proficiency \(screened via a standardized proficiency test\)\. We do*not*require linguistics or translation degrees or professional translation experience\. Assessors received the same RATE\-protocol training materials as experts but were not subject to the post\-editing selection test\.

##### Pool size and workload\.

The pool comprises35expert translators and82assessors\. Each segment is annotated independently by three experts from the pool in implicit side\-by\-side mode \(Expert×\\times3\), and by assessors in two setups: implicit side\-by\-side \(HumanA\-SbS\) and point\-wise \(HumanA\-PW\)\. For evaluation we hold two expert annotations as gold and score the third against each of them and average, giving the HumanEagreement reported in Table[1](https://arxiv.org/html/2610.33741#S0.T1)\(Section[3\.3](https://arxiv.org/html/2610.33741#S3.SS3)\)\. Average workload per annotation pass was17segments per expert and22segments per assessor\. Median annotation times were14minutes per segment for experts and5minutes for assessors, compared to the∼\\sim68 seconds per segment that[Kocmi et al\. \(2025\)](https://arxiv.org/html/2610.33741#bib.bib12)report for English–Russian WMT25 ESA annotation\.

##### Training and dispute resolution\.

Before annotation began, all annotators participated in a calibration session in which a small set of pre\-annotated examples was reviewed and edge cases were discussed collaboratively\. For the few\-shot pool used in Section[4\.2](https://arxiv.org/html/2610.33741#S4.SS2), annotators worked in overlapping groups on subsets of segments and reconciled disagreements through discussion to produce consensus annotations more reliable than any single\-annotator output\.

## Appendix DMethods Not Included in Main Comparison

We do not include several recent MT evaluation approaches in our primary comparison due to differences in evaluation setup and workflow design\. M\-MAD\([Feng et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib3)\)explores debate\-style multi\-agent evaluation with substantially different orchestration strategies and computational budgets\. We draw on its decomposition idea in our multi\-call ablations \([Section4\.5](https://arxiv.org/html/2610.33741#S4.SS5)\) but do not reproduce the full multi\-agent system\. EAPrompt\([Lu et al\., 2024](https://arxiv.org/html/2610.33741#bib.bib23)\)produces error analyses through iterative prompting\. A controlled re\-evaluation by[Feng et al\. \(2025\)](https://arxiv.org/html/2610.33741#bib.bib3)reports lower span\-levelF1F\_\{1\}than GEMBA\-MQM, which is already included in our comparison\. MQM\-APE\([Lu et al\., 2025](https://arxiv.org/html/2610.33741#bib.bib22)\)relies on automatic post\-editing as a validation step, requiring multiple sequential LLM passes per segment and following a substantially different multi\-stage workflow\. More broadly, our goal is not to exhaustively benchmark every existing MT evaluation system, but to study evaluator design principles under a controlled experimental setup\.

## Appendix ESensitivity to Span Matching

[Perrella et al\. \(2026\)](https://arxiv.org/html/2610.33741#bib.bib24)recommend MPP with micro\-averaging to penalize imprecise span boundaries and avoid the preference of MP for longer spans\. Our target places more emphasis on error\-identity agreement, so we retain MP withτ=1\\tau\{=\}1as the primary formulation and compare it with MPP and an LLM\-based reconciliation proxy\. All metrics in this section are micro\-averaged\.

### E\.1Formal Definitions

We compare three matching formulations\.

##### MP withτ=1\\tau\{=\}1\(primary\)\.

Predicted error spans are matched to gold annotations via bipartite matching that maximizes the number of matched pairs\. A predicted–gold pair is eligible for matching iff their character overlap is≥1\\geq 1\. Each matched pair contributes a credit of1\.01\.0to both precision \(over predicted spans\) and recall \(over gold spans\)\.

##### MPP\([Perrella et al\., 2026](https://arxiv.org/html/2610.33741#bib.bib24)\)\.

Eligible predicted–gold pairs must overlap by at least one character\. Among eligible one\-to\-one matchings, MPP selects the matching that maximizes the resulting MPPF1F\_\{1\}\. Credit is then proportional to overlap\. For a matched pair withkkoverlapping characters, precision credit \(over the predicted span of lengthℓp\\ell\_\{p\}\) isk/ℓpk/\\ell\_\{p\}, and recall credit \(over the gold span of lengthℓg\\ell\_\{g\}\) isk/ℓgk/\\ell\_\{g\}\.

##### LLM\-based reconciliation proxy\.

For each translation, we present the predicted spans and gold annotations to GPT\-5\.4 and instruct it to match annotations that describe the same error, allowing character boundaries to differ\. Each matched pair contributes a credit of1\.01\.0\. We treat this as an exploratory reconciliation proxy rather than human adjudication\.

\(a\)Claude Opus 4\.6\.
\(b\)Gemini 3 Flash\.

Table 16:Matched design comparison across two backbones\.Orbit\-SCuses RATE\-AEbriefwith curated coverage \(k=2k\{=\}2\)\.

### E\.2Cross\-Formulation Comparison

Table[14](https://arxiv.org/html/2610.33741#A1.T14)reports all three formulations across our main systems\.

### E\.3Boundary Sensitivity under MPP

The matching formulation affects exact system rankings and the comparison between human and model agreement\. Under MPP, HumanEdrops by4\.94\.9recall points and5\.75\.7precision points\. Across the threeOrbit\-SCbackbones, recall changes by only1\.61\.6–2\.22\.2points, while precision falls by14\.314\.3–19\.519\.5points\.Orbit\-MCshows the same asymmetry\. This follows directly from MPP’s partial\-credit rule when a predicted span extends beyond the gold boundaries\.

Boundary fidelity is relevant for some downstream uses\. However, MPP can under\-credit error\-identity agreement when systems and experts use different span granularities\.

### E\.4LLM Reconciliation Proxy

The LLM reconciliation proxy provides a complementary view\. It places HumanEatF1=53\.1F\_\{1\}\{=\}53\.1, the threeOrbit\-SCconfigurations at51\.751\.7–54\.154\.1, andOrbit\-MCat56\.456\.4\. These results support the main error\-identity comparison, while MPP shows that the exact human–model ordering is sensitive to span boundaries\.

GPT\-5\.4 is both the reconciler and one of the evaluated backbones, so the proxy may share biases with the systems being judged\. We therefore use it as complementary evidence rather than as the primary meta\-evaluation method\.

### E\.5Methodological Takeaway

We retain MP withτ=1\\tau\{=\}1as primary because our main question is whether an evaluator identifies the annotated error location\. MPP provides a boundary\-sensitive view, while the LLM reconciliation proxy emphasizes error identity\. The three formulations support the same broad conclusion thatOrbitis a strong evaluator, but exact system ordering and the human–model comparison depend on the operational definition of a span match\.

## Appendix FModel Comparison: Extended Results

[Table15](https://arxiv.org/html/2610.33741#A1.T15)extends the model comparison from[Section4\.3](https://arxiv.org/html/2610.33741#S4.SS3)with per\-model span density \(mean predicted spans per segment, broken down by severity\) and severity\-calibrated variants of the major\-error metrics\.

##### Severity\-aware metrics\.

The mainR4\+R\_\{4\+\}metric measures recovery of gold spans with severity≥4\\geq 4regardless of predicted severity\. The mainP4\+P\_\{4\+\}metric measures precision for predictions assigned severity≥4\\geq 4against the same gold major\-error spans\. ForR4\+∙R\_\{4\+\}^\{\\bullet\}andP4\+∙P\_\{4\+\}^\{\\bullet\}, only predictions assigned severity≥3\\geq 3are eligible for matching\. These are severity\-aware variants with a one\-level tolerance around the major threshold\. The gap betweenR4\+R\_\{4\+\}andR4\+∙R\_\{4\+\}^\{\\bullet\}therefore measures how often real major errors are detected but assigned severity≤2\\leq 2\.

##### Severity calibration is a separate axis of evaluator behavior\.

As shown in[Table15](https://arxiv.org/html/2610.33741#A1.T15), the gap betweenR4\+R\_\{4\+\}andR4\+∙R\_\{4\+\}^\{\\bullet\}reveals substantial under\-rating of major errors across both human and LLM evaluators\. HumanEitself drops1111points \(65\.3→54\.065\.3\\to 54\.0\), confirming that severity assignment is genuinely hard even for trained experts\. LLM evaluators show gaps of comparable magnitude: GPT\-5\.4 drops from74\.774\.7to61\.361\.3\(−13\.4\-13\.4\), Claude Opus 4\.6 from72\.272\.2to56\.856\.8\(−15\.4\-15\.4\), and Gemini 3\.1 Pro from67\.867\.8to56\.856\.8\(−11\.0\-11\.0\)\. This indicates that detection and severity assignment are partially independent capabilities, and that the difficulty of severity calibration is not specific to LLM evaluators — trained experts exhibit the same pattern\. We adopt the detection\-orientedR4\+/P4\+R\_\{4\+\}/P\_\{4\+\}as primary metrics in the main paper because they isolate the comparatively more fundamental question of whether the evaluator finds the right spans, but report severity\-calibrated variants here as a complementary diagnostic\.

##### Span density patterns\.

The mean spans\-per\-segment columns clarify the recall–precision trade\-offs visible in the span\-level metrics \([Table5](https://arxiv.org/html/2610.33741#S4.T5)\)\. Backbones with strong precision \(Gemini 3\.1 Pro, Claude Opus 4\.6\) predict at or below the expert non\-trivial rate \(4\.394\.39and5\.095\.09vs\.5\.285\.28\), while recall\-oriented backbones \(GPT\-5\.4, Grok\-4\) predict above it but with major\-severity counts that remain plausible \(1\.551\.55and1\.361\.36vs\.1\.011\.01\)\. DeepSeek\-V3\.2 is the clear outlier:11\.9611\.96non\-trivial spans and6\.126\.12major errors per segment — roughly6×6\\timesthe expert major\-error rate — explaining its severe precision collapse \(P4\+=15\.3P\_\{4\+\}=15\.3\)\. Kimi\-K2 sits at the opposite extreme, under\-predicting both non\-trivial \(3\.493\.49\) and major \(0\.700\.70\) spans, consistent with its low overall recall \(R=22\.0R=22\.0\)\.

## Appendix GCross\-Backbone Design Replication

We evaluate the same 11 configurations with Claude Opus 4\.6 and Gemini 3 Flash using the final metric pipeline\. This matched set covers published baselines, RATE instruction variants, coverage\-based demonstrations, and the finalOrbit\-SCconfiguration\. The full comparison is reported in[Table16](https://arxiv.org/html/2610.33741#A5.T16)\.

On Gemini,Orbit\-SCis strongest on spanF1F\_\{1\}andR4\+R\_\{4\+\}\. On Claude, it has the strongestR4\+R\_\{4\+\}and is within0\.10\.1F1F\_\{1\}of the strongest configuration\. RATE\-based variants also outperform the protocol baselines onF1F\_\{1\}for both backbones, supporting transfer of the overall design while leaving individual instruction effects backbone\-dependent\.

## Appendix HHuman–LLM Collaborative Annotation

We conduct a preliminary study of how expert annotators interact with reports generated by theOrbit\-MCmulti\-call configuration\. On a held\-out set of 600 English–Russian segments \(disjoint from the design study data\), we compare two expert groups\. HumanE1first produced independent annotations and then re\-annotated the same segments after reviewingOrbit\-MCreports \(post\-hint, within\-subject design\)\. HumanE2received the reports before its initial annotation \(pre\-hint\)\.

In aggregate, access toOrbit\-MCreports increases annotation density and shifts annotators toward lower overall quality scores \([Table17](https://arxiv.org/html/2610.33741#A8.T17)\)\. Annotators do not adopt reports uncritically: post\-hint annotations retain fewer spans than the raw report contains, and slider scores shift toward but do not matchOrbit\-MC’s more severe assessments, indicating partial rather than wholesale integration of report signals\. These results are exploratory — the study measures annotation revision rather than ground\-truth correctness\.

Table 17:Effect ofOrbit\-MChints on expert annotation \(held\-out 600\-segment set\)\.