North Small Translate: Advanced Cost-Effective Translation (Cohere CAT+)

# North Small Translate: Advanced Cost-Effective Translation (Cohere CAT+)
Source: [https://arxiv.org/html/2609.13916](https://arxiv.org/html/2609.13916)
Alexandre BérardPhil BlunsomSamuel CahyawijayaShaun CassiniNicholas FrosstOna de GibertAidan GomezNithya GovindarajanShun KiyonoOlivia LascheLawrence RogersKelly MarchisioNikita MogheYash MoreCamila Moran\-HidalgoYiyang NanMichael SachsTrisha StarostinaDaan van StigtSpencer RarrickSebastian VincentIvan ZhangCohereEmail:[\*](mailto:)[kocmi@cohere\.com](mailto:[email protected])

###### Abstract

We presentNorth Small Translate, an open\-weight, LLM\-based machine translation \(MT\) model with instruction\-following capabilities built on the same foundation as Cohere’sCommand A\+, a mixture\-of\-experts architecture with 25 billion active parameters out of 218 billion total parameters\.North Small Translateis trained using difficulty sampling to obtain challenging documents and a five\-step training protocol combining supervised fine\-tuning, direct preference optimization, and online reinforcement learning\. We prioritized throughput through a non\-reasoning base model and supplemented with optional agentic capabilities to unlock translation quality gains\.North Small Translateis trained to perform MT\-related tasks, including post\-editing and quality estimation, as well as related tasks such as general instruction following\. The model achieves top MT performance across 50 languages in the class of models under 1T parameters, with no need to run expensive reasoning at inference time\.

Paramstotal \(active\)GEMBAWMT26xCOMETxlWMT24\+\+TerminologyLongcontextStructuredtranslationNorth Small Translate \(Agentic\)218B \(25B\)84\.480\.787\.947\.893\.3North Small Translate218B \(25B\)83\.680\.689\.748\.993\.7Mistral Large 3675B \(41B\)81\.678\.391\.220\.093\.8Qwen 3\.5 397B397B \(17B\)81\.678\.587\.019\.559\.4DeepL NextGen81\.481\.847\.856\.564\.5Gemma 4 31b31B \(31B\)79\.579\.292\.119\.495\.1Inkling Small276B \(12B\)78\.578\.791\.121\.591\.3Muse Glimmer 30B30B \(30B\)78\.277\.790\.132\.591\.7Command A\+218B \(25B\)76\.581\.068\.730\.573\.7GLM 5\.2744B \(40B\)76\.578\.878\.228\.692\.1Mistral Medium 3\.5128B \(128B\)75\.776\.790\.319\.394\.2GPT\-OSS 120B117B \(5B\)72\.376\.683\.820\.893\.2Command A Translate \(2025\)111B \(111B\)70\.076\.956\.90\.079\.5Google Translate68\.280\.838\.121\.348\.8Nemotron 3 Ultra 550B550B \(55B\)67\.184\.819\.7Qwen 3\.6 27B27B \(27B\)61\.989\.228\.9

Table 1:Aggregated results of our model against other top systems with under 1T parameters\.## 1Introduction

The landscape of machine translation \(MT\) has undergone a profound transformation in recent years, having almost fully transitioned to large language models \(LLMs\) that treat cross\-lingual generation as a complex instruction\-following task\. The paradigm shift has unlocked unprecedented capabilities, allowing models to leverage broad contextual understanding, perform robust translation, and adapt translations to various requests\.

We introduceNorth Small Translate\(submitted to the WMT General MT shared task asCohere CAT\+\), our best\-performing MT system built upon the same foundation as Cohere’sCommand A\+\([Cohere Team, 2026](https://arxiv.org/html/2609.13916#bib.bib6)\)\. It is a sparse mixture\-of\-experts \(MoE\) transformer with 25 billion active parameters out of a total of 218 billion parameters\.

North Small Translateis built for efficiency\. While most LLMs rely on test\-time reasoning to achieve state\-of\-the\-art performance, long reasoning traces introduce prohibitive latency and computational overhead for production\-level translation\. Model throughput is a primary target for practical MT deployment, so we explicitly trainedNorth Small Translateas a non\-reasoning model\. For latency\-insensitive applications, our optional*Agentic Translation*framework bridges the potential quality gap caused by the absence of intermediate reasoning steps\.

North Small Translateachieves top\-tier translation quality through five distinct training steps: \(i\) Coarse SFT to establish general non\-MT instruction\-following capabilities and extend the base model’s language coverage; \(ii\) Fine\-grained SFT using highly curated synthetic datasets to improve core MT quality and adjacent tasks like error detection and post\-editing; followed by three steps focusing primarily on human\-aligned translation quality: \(iii\) DPO; \(iv\) Online RL using an LLM\-as\-a\-judge reward; and \(v\) a minimal final DPO with reduced learning rate to resolve edge\-case regressions introduced in RL\.

We participate in the WMT 2026 General MT shared task\([Kocmi et al\., 2026](https://arxiv.org/html/2609.13916#bib.bib18)\), Terminology shared task\([Charkiewicz et al\., 2026](https://arxiv.org/html/2609.13916#bib.bib3)\), and Automated MT Evaluation shared tasks with our model under nameCohere CAT\+\. We focused our submission on a clean performance without relying on any specialized search heuristics or inference tricks such as Minimum Bayes Risk decoding\([Freitag et al\., 2022](https://arxiv.org/html/2609.13916#bib.bib10)\)\. Neither did we use the Agentic translation\.111Disclosure of conflict: the main author is also an organizer of the WMT General MT shared task\.One domain for General MT contains videos as sources, we usedCohere Transcribe222[https://cohere\.com/blog/transcribe](https://cohere.com/blog/transcribe)to obtain textual transcripts for English videos, while relying on the official provided transcripts for Chinese and Czech source data\.

## 2Training Details

### 2\.1Model Architecture

The list of pre\-training languages is:

inline,color=olive\!20\!whiteinline,color=olive\!20\!whitetodo:inline,color=olive\!20\!whiteArabic, Bengali, Czech, Danish, Dutch, English, Filipino, Finnish, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Simplified Chinese, Spanish, Swedish, Tamil, Telugu, Thai, Traditional Chinese, Turkish, Ukrainian, Urdu, Vietnamese\.The list of languages extended in post\-training:

inline,color=olive\!20\!whiteinline,color=olive\!20\!whitetodo:inline,color=olive\!20\!whiteAlbanian, Bulgarian, Catalan, Croatian, Estonian, Hungarian, Icelandic, Irish, Latvian, Lithuanian, Maltese, Serbian, Slovak, Slovenian\.
### 2\.2Training Approach

We build on top of findings from our previous models and also our previous Command\-A\-Translate\([Kocmi et al\., 2025a](https://arxiv.org/html/2609.13916#bib.bib16)\)which showcased the impact of Direct Preference Optimization \(DPO\)\([Rafailov et al\., 2023](https://arxiv.org/html/2609.13916#bib.bib30)\)training step on MT quality\.

We use five distinct training steps, each focused on a different aspect:

1. 1\.Initial coarse SFT: to cover general instruction\-following capabilities, and extend language coverage from pretrained 35 languages to 50 languages\. We fine\-tune four different models, and merge them\([Cohere et al\., 2025](https://arxiv.org/html/2609.13916#bib.bib5)\)\.
2. 2\.Fine\-grained SFT: focusing on improving the MT quality using primarily synthetic datasets and also improving MT related tasks such as error detection, terminology or post\-editing\.
3. 3\.DPO: primarily focused on improving quality\.
4. 4\.Online RL: online RL with GSPO\([Zheng et al\., 2025](https://arxiv.org/html/2609.13916#bib.bib36)\)and LLM\-as\-a\-Judge as the reward signal to polish performance and increase adequacy\.
5. 5\.Light targeted DPO with reduced learning rate: for fixing remaining problems, especially a few introduced by the RL step\.

[Table2](https://arxiv.org/html/2609.13916#S2.T2)highlights how the MT performance improved by each individual training step\. We see significant gains with each step except for the RL\. While this step improved some scores, especially adequacy, it degraded the performance over few low\-resource languages, which is due to weakness of the LLM judge on these languages\.

To fix the aforementioned regression, we introduced a final DPO step reduced learning rate and using only 5 warm\-up steps\. For this step, we focused only on a few problematic languages and capabilities\.

StepMT QualityCoarse SFT71\.3%Fine\-grained SFT76\.2%DPO80\.1%Online RL80\.0%Light DPO81\.8%Table 2:Training progression on the WMT25 evaluation with GEMBA\-ESA averaged over 14 languages\.
### 2\.3Data Preparation

While the main goal is to build a strong MT model, we additionally focused on building a strong instruction\-following model that also excels in MT\-related tasks, such as post\-editing, terminology following, quality estimation, or structured translation\.

For every training set, we checked that every translation pair was processed through a suite of automated verifiers designed to eliminate noisy or malformed text\. Specifically, we enforced strict structural consistency by verifying line and paragraph alignment, ensuring that the source and target segments contained the exact same number of lines and paragraphs\. Additionally, we applied language identification and filtered out instances of unwanted code\-switching\. Our filtering pipeline also removed segments containing broken Unicode characters and relied on various other internal classifiers to detect and discard anomalies\.

#### 2\.3\.1Quality of Machine Translation

A primary challenge in curating training corpora for top\-performing MT systems is the sparsity of genuinely difficult source material, as we highlighted in[Kocmi et al\. \(2025a\)](https://arxiv.org/html/2609.13916#bib.bib16)\. After our initial coarse SFT stage, the model translates without major errors for more than 90% of our training documents\. Consequently, randomly sampling source texts would yield a weak learning signal and fail to push the model’s quality boundaries\.

We use difficulty sampling to focus our synthetic generation exclusively on problematic instances from which the model can learn\. Building upon our previous work\([Kocmi et al\., 2025a](https://arxiv.org/html/2609.13916#bib.bib16)\), we extend difficulty sampling by using the initial base model with minimal post\-training itself as the baseline model and keep only documents where the model produces major errors, or its translation quality is lower than the human reference for parallel data\. For each source document, we randomly pick a high\-resource target language and translate it with an early stage model\. The generated translation is then automatically evaluated for errors with a judge\. For monolingual data, if the translation is largely error\-free, the source segment is classified as “easy” and removed from the candidate pool\. For parallel data, we check if the translation is better than the human reference to deem it “easy”\. By aggressively filtering out easily translated segments, we distill a highly concentrated corpus of difficult texts\. This ensures that our downstream generation of synthetic data is allocated to the documents where the model requires the most improvement\.

To achieve top MT performance across all supported languages, we construct our training corpora using two distinct data synthesis and refinement strategies, which we call*Best\-per\-Language Forward Translation*and*Post\-Edit Driven Preference Distillation*\. Both approaches are designed to maximize the quality and diversity of the training signal without relying on human\-annotated parallel data, which we use solely in the first stage of training\.

Best\-per\-Language Forward Translation relies on a standard forward translation approach combined with a Best\-of\-N selection\. We use a set of top\-performing language\-specific expert models, where for each language we select the top\-performing expert to generate translations for given source documents\. To ensure broad multilingual coverage and explicitly improve cross\-lingual transfer, we randomly assign the target language during generation\. While the majority of the source texts are in English, a significant part consists of non\-English sources, yielding a substantial volume of direct non\-English to non\-English translation pairs\. The resulting translation candidates are then evaluated by a judge and accepted only if their score is higher than 70 GEMBA\-ESA\. We describe details of the scoring in[Section2\.4](https://arxiv.org/html/2609.13916#S2.SS4)as we use this approach throughout the work\.

Our second strategy,*Post\-Edit Driven Preference Distillation*, is the most critical of our data preparation pipeline and is the primary driver for our DPO stage\. The core objective is to generate near on\-policy training data in an offline setting, creating highly effective preference pairs that directly target the model’s own weaknesses\. The orchestration pipeline operates in three distinct steps:

1. 1\.Generate a draft translation using the current candidate model\.
2. 2\.A separate judge model identifies and annotates error spans in the generated draft\. If no error is present, the sample is deemed “easy” and dropped from the training data\.
3. 3\.A third model acts as an automated post\-editor, taking the draft and fixing the errors\.
4. 4\.We iterate steps 2\-3 until no major error is discovered, or we pass 10 iterations\.

This multi\-model orchestration process yields a perfectly aligned preference pair: the original, flawed output from the candidate model serves as the worse completion, with the corrected, post\-edited version as the preferred completion\.Because the broken translation originates directly from the candidate model, the resulting dataset closely approximates on\-policy MT data\. This allows the offline DPO training to function with the precision of online reinforcement learning, penalizing the model’s actual failure modes\.This orchestration approach was instrumental in fixing persistent, model\-specific errors and driving the most significant quality improvements observed during our DPO stage\.

#### 2\.3\.2Instruction Following

To ensure our MT model has acceptable general instruction following abilities which would strengthen the translation\-related tasks and following specific MT instructions, we included non\-MT related instruct data in the first SFT stage\. We used some of the English\-only single\-turn datasets that were used to train Command A\+\. Additionally, we included multilingual datasets using general\-purpose multilingual prompts\. We have removed reasoning traces from all reused datasets\. We furthermore used all the aforementioned data in the second SFT stage, but significantly downsampled\.

#### 2\.3\.3Structured Translation

“Structured translation” is the process of translating natural language embedded in data formats such as JSON or HTML without breaking their underlying syntax\. It is a critical feature especially for translating webpages, documents or training data\.

While general instruction following datasets described above contain various formats, we focus in structured translation solely on JSON and HTML formats\. We prepare data two ways: naturally occurring structured data and combining paragraphs or sentences from existing high quality MT datasets into diverse structures\. For the former, we relied on our internal datasets containing JSONs and we then extracted the HTML data from various web based datasets\. For the latter part, this was done using top segments from our high quality sentences corpora\. After constructing this source data, we translated them using an internal model into 35 languages\. These translations were done in a field\-by\-field basis as well as complete structure translations\. When preparing the training data, we added prompt instruction controllability of levels of translation, from translating all structures in the source up to only focusing on translation of few keys in JSONs\. Lastly, the translations were filtered for structure retention and for translation quality\. On top of SFT, we also created a DPO dataset targeting only specific structured errors that our fine\-tuned models exhibit\. These included overly copying or parsing issues while ensuring the preferred completions remain as pristine as possible

#### 2\.3\.4Terminology Translation

Terminology translation is the task of producing a translation that adheres to a glossary of source\-target term pairs supplied in the prompt\. When a customer has an established vocabulary, such as product names, brand names or domain\-specific jargon, it is critical that a translation solution render these consistently and have the ability to use translation memories\. We focus on building a preference dataset for this capability\.

For each document, we prompt an LLM to identify the named entities in the source together with their renderings in the reference translation, to propose an alternative but equally valid rendering for each of them in the target language, and to rewrite the reference translation so that it uses these new renderings while leaving the rest of the text untouched\. The rewritten translation then becomes the preferred completion and the original reference the dis\-preferred one, with the newly proposed term pairs given to the model as the glossary\. Since the two completions differ only in how the terms are rendered, the preference signal isolates terminology adherence from the general translation quality\.

Notably, a model that blindly copies every glossary entry into its output would score well on such data\. To counterbalance this, we populate the glossary with negative entries for half the documents: source\-language words and phrases that are topically related to the document but donotoccur in it\. These are proposed by an LLM, filtered with a substring check against the source and both completions, and verified by a second LLM pass that also rejects inflected forms, compounds and transliterations\. A document never receives more negative entries than genuine ones\. The resulting dataset contains preference pairs from English into ten languages: Arabic, German, Spanish, French, Hindi, Italian, Japanese, Korean, Portuguese and Russian\. Following the strategy described in[Section2\.3\.8](https://arxiv.org/html/2609.13916#S2.SS3.SSS8), the glossary itself is presented in a randomized format, either as a JSON object, a JSON list of pairs, or one term pair per line with a varying separator, and is attached to a randomly selected instruction\. Whenever negative entries are present, the instruction is drawn from a separate set that warns the model that some entries may not be relevant to the source text\.

#### 2\.3\.5Tone and Idiomatic Translation

Human evaluation in early stages revealed that our model struggled with adhering to an expected tone, formality levels and usage of idioms\.

To improve the model’s adherence to formality instructions, we collected social\-domain data in eight languages with explicit formality distinctions: Dutch, Italian, Polish, Greek, Japanese, Spanish, Korean, and Portuguese\. Each sample was automatically assigned one of three labels using LLM:*formal*,*informal*, or*ambivalent*\. Both grammatical markers and stylistic cues were considered when determining the appropriate formality label\.

We then back\-translated each target\-language sample, into English using our internal MT model, producing supervised training triplets of source back\-translated English segment, target language with authentic formality level and associated formality level\. During training, each label was mapped to its corresponding natural\-language instruction in the model prompt highlighting the expected formality level\.

In order to tackle the idiom translation, we created multilingual synthetic datasets from existing English monolingual idiom datasets\. The collected trainset contained 50k idiomatic sentences paired with annotated idioms or literal paraphrases\.

For samples with literal paraphrases, we took them as the source texts for translation and translated them from English into 35 languages using our internal MT model\. We then scored each translation with a GEMBA\-ESA\([Kocmi and Federmann, 2023](https://arxiv.org/html/2609.13916#bib.bib20)\)with a strong LLM as the judge\. We retained only examples with a score of at least 70 points\. For idioms without the literal paraphrases, we adopted abest\-of\-Napproach where we translated each with a mix of several models and selecting the best candidate using the LLM judge\. Furthermore, we followed the same approach as above\. Finally, we took the top 10k candidates per language pair creating a dataset of 512k sentences\.

#### 2\.3\.6Error Detection and Quality Estimation

Error detection is the task of identifying erroneous spans in a given machine\-translated text, while quality estimation assessing the quality of the translated text\. Both are evaluated annually by the WMT Automatic Evaluation task\([Lavie et al\., 2025](https://arxiv.org/html/2609.13916#bib.bib22)\)\.

In our preliminary experiments, training on a fully human labels alone was only marginally better than the baseline, lagging far behind state\-of\-the\-art models, which may be caused by inconsistency in the human annotations\. We therefore focus on synthetic annotations of actual MT errors only, using a strong judge to assess error spans and scores\.

Inspired by[Treviso et al\. \(2024\)](https://arxiv.org/html/2609.13916#bib.bib34), we used an XML tag scheme to annotate the text\. We designed a unified XML tag scheme that can represent both: ESA\-style tags\([Kocmi et al\., 2024b](https://arxiv.org/html/2609.13916#bib.bib21)\)encoding only severity \(<error severity="minor or major"\>\), and MQM\-style tags\([Lommel et al\., 2014](https://arxiv.org/html/2609.13916#bib.bib23)\)that additionally carry an error category and subcategory \(<error severity="\.\.\." category="accuracy/mistranslation"\>\)\.

#### 2\.3\.7Post\-editing

Post\-editing is the process of improving existing translation to capture inaccuracies in fluency and adequacy\. Typically, one provides a tuple containing the source and raw translation, and expects an improved translation from an oracle\. The oracle can be a human or an automated translation system or model\. In our setup, we use another capable judge LLM to be an oracle\.

WMT APE came up with the formulation for Automatic Post\-editing, where people were given human post\-edits of some unknown MT system and were allowed to train ML models that can perform similar fixes \- when trained on supervised datasets comprising of the original translation, source text and the corrected output\([Bojar et al\., 2015](https://arxiv.org/html/2609.13916#bib.bib2);[Chatterjee et al\., 2018](https://arxiv.org/html/2609.13916#bib.bib4)\)\. As human\-based post\-edits are scarce and harder to scale, people have relied on synthetically generated edits made by LLMs\([Bang et al\., 2023](https://arxiv.org/html/2609.13916#bib.bib1);[Ki and Carpuat, 2024](https://arxiv.org/html/2609.13916#bib.bib15);[Singh et al\., 2025](https://arxiv.org/html/2609.13916#bib.bib32)\)\. Furthermore, one can fine\-tune the LLM to imbibe more faithful post\-editing capabilities\. We adopt this approach, and believe that the ability to post\-edit responses at inference time a useful trait for translation models\.

We use post\-editing both to refine training targets and to teach our model to repair drafts at inference\-time\. To generate our post\-training dataset, we use a base internal MT system that produces an initial translation; which is then passed on to a stronger LLM that post\-edits it\. We score the draft and the revision with reference\-free GEMBA\-ESA\([Kocmi and Federmann, 2023](https://arxiv.org/html/2609.13916#bib.bib20)\)and keep the pair only when the post\-edit improves the draft, restricting supervision to major quality gaps\. Each retained example is packed in two instruction formats: a\)*Blind*post\-editing, that provides the source and the raw translation as input, and b\)*Span\-assisted*post\-editing, which marks erroneous spans in the original translation with similar XML severity tags \(major or minor\), and provides this annotated translation together with the source as input\([Lommel et al\., 2014](https://arxiv.org/html/2609.13916#bib.bib23);[Kocmi et al\., 2024b](https://arxiv.org/html/2609.13916#bib.bib21)\)\.

#### 2\.3\.8Instruction Diversity

Standard approaches to instruction\-tuned MT often rely on static, hardcoded prompts \(e\.g\., "Translate this to \[Language\]"\)\. While functional, we found out fixed phrasing causes overfitting to the formulation\. To enhance robustness of instruction\-following capabilities, we replace static prompt instructions with a highly diversified instruction strategy during the training phase\.

We introduce variance across three distinct tiers\. First, we seed instructions with a curated set of manually prepared paraphrases to capture human\-written translation requests\. Second, we scale by automatically preparing thousands of instruction variants through paraphrasing, ensuring broad lexical and syntactic coverage\. Finally, we translate each variant into all supported languages\. When preparing the training dataset, we randomly select a prompt alternating the instruction language between English, the source language, and the target language\. Furthermore, we added various augmentations such as merging consecutive sentences into a single line, replacing new lines with random paragraph separators, enclosing the source and/or target text into XML tags or markdown\.

To strengthen instruction adherence, we included a small preference dataset in the last step of DPO\. We want the model to follow specified terminology and formality levels, but also to respect custom formatting or style constraints and potentially several constraints at once\. We build a dataset synthetically by using an LLM to add creative translation guidelines to existing translation prompts, then generating translations with best\-of\-N approach selecting best translation with a judge which specifically tests the guideline adherence\. The worst completion is then used as a negative example\.

### 2\.4Reliable Evaluation Metric

MetricAcc \(ties\)GEMBA\-ESA Fluency\+Adequacy51\.0GEMBA\-ESA50\.6xCOMETxl44\.9spBLEU43\.7Table 3:Metric meta\-evaluation comparing LLM\-as\-a\-judge \(with GPT 5\.4\) versus traditional metrics on internal human label translationsDetermining reliable evaluation metrics was crucial for our model development\. We conducted a meta\-evaluation using internal human annotations for 24k MT translated segments across 10 languages, applying the segment\-level pairwise accuracy methodology introduced in[Deutsch et al\. \(2023\)](https://arxiv.org/html/2609.13916#bib.bib9)\. As shown in[Table3](https://arxiv.org/html/2609.13916#S2.T3), our results confirm the[Lavie et al\. \(2025\)](https://arxiv.org/html/2609.13916#bib.bib22): LLM\-as\-a\-Judge paradigms significantly outperform traditional string\-based metrics like xComet or spBLEU\. Consequently, we focused on LLM\-as\-a\-judge across our work\. Furthermore, we designed an updated variant of the GEMBA\-ESA\([Kocmi and Federmann, 2023](https://arxiv.org/html/2609.13916#bib.bib20)\)prompt to evaluate fluency and adequacy separately before averaging the scores\. This decoupled approach yields slightly higher correlation with human judgments while providing the analytical flexibility to diagnose translation accuracy and fluency independently, as well as individual error annotations\. The full prompt that we used across the work with various judges is in[AppendixA](https://arxiv.org/html/2609.13916#A1)\.

### 2\.5Agentic Translation

To maximize inference throughput, we train a non\-reasoning model\. While this design choice significantly improves generation speed, it limits the potential quality gains that the model might achieve through intermediate reasoning steps\. To bridge this gap for quality\-critical applications, we implemented a complementary framework which we refer to as*Agentic Translation*\. This approach orchestrates a multi\-step workflow by utilizing the exact same underlying model across three distinct functional roles\. Initially, a translation agent processes the source text to generate a baseline draft\. Subsequently, an evaluation agent inspects this draft to identify and annotate any translation errors\. Finally, a post\-editing agent leverages the identified error spans to refine and correct the draft, producing the final output\. By decomposing the translation process into iterative, targeted steps, the agentic approach recovers the performance benefits typically associated with reasoning models\.

## 3Evaluation and Results

We analyze the performance of our model over MT capabilities and related tasks\. We compare it our previous models, top\-performing MT open\-weight systems, and leading providers as Google Translate and DeepL Pro \(Full list in[Section3\.1](https://arxiv.org/html/2609.13916#S3.SS1)\)\. We evaluate all systems in an identical setup, in a clean zero\-shot approach without any inference techniques\.

### 3\.1Benchmark Models

Next to our new model, we evaluate Cohere’s two previous models: Command A Translate\([Kocmi et al\., 2025a](https://arxiv.org/html/2609.13916#bib.bib16)\)and general purpose Command A\+\([Cohere Team, 2026](https://arxiv.org/html/2609.13916#bib.bib6)\)\. We compare them with a broad selection of top performing open\-weight models below 1T total parameters: Google’s Gemma 4 31B\([Gemma Team, 2026](https://arxiv.org/html/2609.13916#bib.bib11)\); Mistral Large 3\([Mistral AI Team, 2026](https://arxiv.org/html/2609.13916#bib.bib25)\)and Mistral Medium 3\.5\([Mistral AI, 2026](https://arxiv.org/html/2609.13916#bib.bib24)\); Alibaba’s Qwen3\.5 397B\([Qwen Team, 2026a](https://arxiv.org/html/2609.13916#bib.bib28)\)and Qwen3\.6 27B\([Qwen Team, 2026b](https://arxiv.org/html/2609.13916#bib.bib29)\); Z\.ai’s GLM\-5\.2\([GLM\-5\-Team et al\., 2026](https://arxiv.org/html/2609.13916#bib.bib12)\); OpenAI’s GPT\-OSS 120B\([OpenAI et al\., 2025](https://arxiv.org/html/2609.13916#bib.bib27)\); NVIDIA’s Nemotron 3 Ultra\([NVIDIA Research, 2026](https://arxiv.org/html/2609.13916#bib.bib26)\); Thinking Machines Lab’s Inkling\-Small\([Thinking Machines Lab, 2026](https://arxiv.org/html/2609.13916#bib.bib33)\); and Meta’s Muse Glimmer 30B\([Cuenca et al\., 2026](https://arxiv.org/html/2609.13916#bib.bib7)\)\. In addition, we compare against two leading commercial translation services: Google Translate and DeepL’s NextGen\.

We keep the reasoning on for every model that supports reasoning capabilities: specifically, Command A\+, Gemma 4 31B, Qwen3\.5\-397B, Qwen3\.6\-27B, GLM\-5\.2, GPT\-OSS 120B, and Muse Glimmer 30B\.

### 3\.2Machine Translation Capabilities

Our core focus is MT quality across all supported languages\. To honestly evaluate capabilities, we evaluate them on a testset released after we finished building the model: the WMT 2026 General MT test set\([Kocmi et al\., 2026](https://arxiv.org/html/2609.13916#bib.bib18)\)\. This dataset contains four distinct domains and translation modes\.

[Lavie et al\. \(2025\)](https://arxiv.org/html/2609.13916#bib.bib22)showed that traditional evaluation techniques such as xComet\([Guerreiro et al\., 2024](https://arxiv.org/html/2609.13916#bib.bib13)\)or MetricX\([Juraska et al\., 2024](https://arxiv.org/html/2609.13916#bib.bib14)\)no longer correlate well with human evaluation and instead strong frontier models excel at evaluation\. We observed the same on our internal evaluation described in[Section2\.4](https://arxiv.org/html/2609.13916#S2.SS4)\. Thus, we use GPT\-5\.6 Sol judge without references for evaluation, which we ask to separately judge adequacy and fluency\. As a final score we average both scores\.

WMT26 is a document\-level test set, where number of documents differ for each domain\. Building on the recommendation of the original test set, we weight each score by the number of segments in the document\. This way each domain is roughly equally represented with about 180 segments\.

The speech domain test set is build to test multimodal capabilities\. However, all reported models are evaluated on a text\-to\-text basis\. To obtain a source text, we use the provided automatic speech transcription\. Finally, we provide the LLM judge the human\-cleaned transcript to reduce noise from the ASR\.

Results are in[Table4](https://arxiv.org/html/2609.13916#S3.T4)\. North Small Translate scores the top performance across the set of all languages, especially highlighting our strength in European and Latin\-script based languages\. It reaches second\-best performance over languages of the Middle\-East and Asia\. Performance over each language is shown in the[Table6](https://arxiv.org/html/2609.13916#A0.T6)\.

Paramstotal \(active\)AlllanguagesEuropeME \+ AsiaLatinscriptOtherscriptsNorth Small Translate \(Agentic\)218B \(25B\)84\.482\.587\.283\.086\.5North Small Translate218B \(25B\)83\.682\.086\.182\.485\.4Mistral Large 3675B \(41B\)81\.680\.383\.580\.383\.5Qwen 3\.5 397B397B \(17B\)81\.681\.282\.081\.082\.4DeepL NextGen81\.482\.380\.082\.479\.9Gemma 4 31b31B \(31B\)79\.573\.588\.474\.986\.3Inkling Small276B \(12B\)78\.577\.879\.678\.279\.0Muse Glimmer 30B30B \(30B\)78\.276\.880\.377\.479\.5Command A\+218B \(25B\)76\.576\.077\.476\.077\.3GLM 5\.2744B \(40B\)76\.577\.575\.077\.874\.6Mistral Medium 3\.5128B \(128B\)75\.773\.678\.773\.678\.7GPT\-OSS 120B117B \(5B\)72\.370\.375\.271\.473\.5Command A Translate \(2025\)111B \(111B\)70\.068\.871\.769\.670\.5Google Translate68\.268\.068\.468\.967\.1Nemotron 3 Ultra 550B550B \(55B\)67\.162\.973\.364\.670\.8Qwen 3\.6 27B27B \(27B\)61\.959\.964\.959\.965\.0

Table 4:Machine translation capabilities aggregated over different groups of languages over the WMT26 testset\.We supplement results with a well\-known evaluation in the MT community: WMT24\+\+\([Deutsch et al\., 2025](https://arxiv.org/html/2609.13916#bib.bib8)\)\. WMT24\+\+ contains English to 55 human\-translated languages and dialects\. The original source text is from[Kocmi et al\. \(2024a\)](https://arxiv.org/html/2609.13916#bib.bib19)and covers multiple domains\. Each language pair contains 171 documents split into 998 mostly paragraph level segments\. We use the prompt instruction from[Deutsch et al\. \(2025\)](https://arxiv.org/html/2609.13916#bib.bib8)and we evaluate with xComet\-XL\([Guerreiro et al\., 2024](https://arxiv.org/html/2609.13916#bib.bib13)\)a metric widely used for system rankings\. The metric is a 3\.5B parameter model fine\-tuned on human judgment data\.

[Table1](https://arxiv.org/html/2609.13916#S0.T1)shows that North Small Translate reaches near\-top performance\. That said, caution is needed; as discussed in the[Section3\.6](https://arxiv.org/html/2609.13916#S3.SS6), WMT24\+\+ is significantly easier than WMT26, and mostly “solved” with top\-performing systems\.

### 3\.3Long\-Context Translation

We evaluate long\-context translation using a subset of the WMT25 General MT test set\([Kocmi et al\., 2025b](https://arxiv.org/html/2609.13916#bib.bib17)\)\. The English source long\-context comprises of two documents with approximately 9,900 words divided into two sections\. We concatenate them into a single source document, yielding one long\-form evaluation instance\. As part of the evaluation, we translate it into Egyptian Arabic, Czech, Estonian, Icelandic, Japanese, Korean, Russian, Simplified Chinese, and Ukrainian\. The complete document is provided to the model in a single request\.

Translations are evaluated against the WMT25 references using the reference\-based xCOMET\-XL metric\([Guerreiro et al\., 2024](https://arxiv.org/html/2609.13916#bib.bib13)\)\. Since the metric is not able to evaluate full document, we divide the translations at segment boundaries\. xCOMET\-XL independently scores each aligned source–hypothesis–reference triple, and returns the arithmetic mean over the 85 segments\.

[Table1](https://arxiv.org/html/2609.13916#S0.T1)shows that our model reachs second\-best MT quality, outperformed only by DeepL\.444DeepL is a closed\-source system accessed by API, so we cannot confirm implementation: for instance, whether it might pre\-parse the source text\.

### 3\.4Terminology Translation

We evaluate glossary\-conditioned translation using the WMT25 Terminology Translation Task\([Semenov et al\., 2025](https://arxiv.org/html/2609.13916#bib.bib31)\)\. We refer readers to that work for the dataset construction, domains, and official shared\-task protocol\. Our evaluation covers five translation directions: English to German, Spanish, Russian, and Traditional Chinese, and Traditional Chinese to English\.

For the scoring we use only the proper\-terminology condition setup\. Each model receives the source text together with a source–target glossary and is instructed to apply the supplied terminology while returning only the translation\.

We measure is corpus\-level terminology success rate: a dictionary pair is successful when its source term occurs in the input and its prescribed target form occurs in the translation\. As a final score in[Table1](https://arxiv.org/html/2609.13916#S0.T1), we micro average language pair scores from both WMT25 Terminology Tracks\.

### 3\.5Evaluation of Structured Translation

This task is a first of its kind, hence efforts were dedicated towards building test suites as well as designing relevant metrics for this task\. Our test suites were developed using internal datasets, from the same distribution of training data and also naturally occurring sentences that were combined to create diverse JSON structures\. The examples included two translation tasks: translating entire structures with all fields, and translating only user\-specified fields, designed to evaluate the model’s instruction\-following abilities\. We also included a markdown evaluation suite but our base models were already saturating their performance, thus we subsequently dropped this test suite\.

For evaluation of structured translation, we tracked two areas \- the quality of translation and primarily the consistency of the structured translation, which we report in this paper\. The evaluation has multiple metrics targeting various errors observed during model development\. We report an average score of all the metrics described below in[table1](https://arxiv.org/html/2609.13916#S0.T1)\.

- •Overall structure: Three metrics \-parsing accuracychecks if the produced translation adheres to the right delimiters from which the structured output can be extracted\.Decoding accuracydetermines if the extracted output can be decoded into a valid object \(e\.g\., a JSON or YAML object\)\. Andschema accuracychecks whether the decoded translation output matches the source schema \(e\.g\., the same JSON keys\)\. These metrics have cascading effect on the next structured metrics\.
- •Untranslation issues: The intermediate models exhibited different levels of copying in their outputs, from leaving intermediate fields untranslated to entire translations retained in the source language\. In addition, the instruction following test suite involved only translating a limited number of fields, requiring the remaining fields to be retained in their source language\. Thus, the evaluation measured \(i\) percentage of samples where all non translatable content was copied correctly and \(ii\) average per\-sample percentage of translatable fields that were incorrectly copied\. Lastly, a separate metric evaluated the proportion of examples that were left completely untranslated\.
- •Line count consistency: This metric operates on a per\-field level and checked if the lines in generated translation match exactly the number of lines in source translation\.

### 3\.6Quantifying Test Set Saturation

To verify the headroom for improvement for popular WMT testsets, we ran a human evaluation using the same system and the ESA protocol\([Kocmi et al\., 2024b](https://arxiv.org/html/2609.13916#bib.bib21)\)\.[Table5](https://arxiv.org/html/2609.13916#S3.T5)showx that for high\-resource languages, the WMT24 testset is much easier than WMT26—nearly solved\. This is especially problematic in combination with xComet where the score differences are more affected by the metric word preferences rather than the actual errors, which are getting rare\. Furthermore, we observed that for the WMT 2026 the software domain is the easiest domain, while other domains remain considerably more challenging\.

These results should be interpreted with some caution\. Although both evaluations were conducted using the Pearmut framework\([Zouhar and Kocmi, 2026](https://arxiv.org/html/2609.13916#bib.bib37)\), they differ in certain evaluation details, most notably affecting the slider anchors\. We argue, however, that these changes had minimal impact on the major error annotation, which remained comparable\. A second difference is that each testset was evaluated in a different month, alongside a distinct set of systems, and by a slightly different pool of annotators\. While the annotator pool could not be strictly controlled, we selected one system that was identical across both evaluations\. Thus, the table shows the percentage of errors for this specific model across both setups\. For context, this is a strong system that outperforms Command A\+, though it underperforms North Small Translate\.

LanguageWMT24WMT26Arabic \(MSA\)19\.7%34\.9%German22\.9%37\.1%Spanish \(Mexico\)14\.2%34\.9%French7\.6%13\.9%Italian11\.9%11\.5%Japanese14\.6%28\.9%Korean29\.9%19\.3%Portuguese \(Brazil\)6\.6%17\.5%Chinese \(Simplified\)14\.4%18\.1%Overall difficulty15\.8%24\.0%Table 5:Percentage of segments containing at least one major error\. Highlighting the saturation of the test sets\.

## 4Conclusion

We introduceNorth Small Translate, Cohere’s state\-of\-the\-art machine translation system\. A sparse mixture\-of\-experts non\-reasoning transformer\-based LLM with 25 billion active and 218 billion total parameters, it achieves top\-tier performance across 50 languages\.

We buildNorth Small Translatevia a five\-step training protocol spanning SFT, DPO, and online RL\. By using difficulty sampling and an orchestrated preference distillation pipeline, we focused our synthetic data generation and training signal strictly on the most challenging documents\. Consequently, the model excels in standard machine translation as well as complex related tasks, including structured translation, terminology adherence, and error detection\.

For latency\-insensitive applications requiring maximum quality, we incorporated an Agentic Translation framework, enabling the model to iteratively evaluate and post\-edit its own drafts\. We submitted our model to the WMT 2026 General MT shared task asCohere CAT\+, demonstrating the model’s strong out\-of\-the\-box capabilities\.

## Limitations

The evaluation of MT systems is fundamentally limited by the noise and limited discriminative power of automated benchmarks, and even of human evaluators\. Translation quality can be subjective, and furthermore, high translation quality in one domain for a given language does not guarantee high quality in another, even for the same language\. Preferred system recommendations can thus change depending on use case\.

## References

- Bang et al\. \(2023\)Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V\. Do, Yan Xu, and Pascale Fung\. 2023\.[A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity](https://doi.org/10.18653/v1/2023.ijcnlp-main.45)\.In*Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 675–718, Nusa Dua, Bali\. Association for Computational Linguistics\.
- Bojar et al\. \(2015\)Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi\. 2015\.[Findings of the 2015 workshop on statistical machine translation](https://doi.org/10.18653/v1/W15-3001)\.In*Proceedings of the Tenth Workshop on Statistical Machine Translation*, pages 1–46, Lisbon, Portugal\. Association for Computational Linguistics\.
- Charkiewicz et al\. \(2026\)Adrian Charkiewicz, Pinzhen Chen, Thierry Etchegoyhen, Harritxu Gete Ugarte, Kamil Guttmann, Xu Huang, David Ponce, Artur Nowakowski, Frédéric Odermatt, Arturo Oncevay, Kirill Semenov, Dawei Zhu, and Vilém Zouhar\. 2026\.Findings of the WMT26 terminology translation task\.In*To appear in Proceedings of the Eleventh Conference on Machine Translation*\. Association for Computational Linguistics\.
- Chatterjee et al\. \(2018\)Rajen Chatterjee, Matteo Negri, Raphael Rubino, and Marco Turchi\. 2018\.[Findings of the WMT 2018 shared task on automatic post\-editing](https://doi.org/10.18653/v1/W18-6452)\.In*Proceedings of the Third Conference on Machine Translation: Shared Task Papers*, pages 710–725, Belgium, Brussels\. Association for Computational Linguistics\.
- Cohere et al\. \(2025\)Team Cohere, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, et al\. 2025\.Command a: An enterprise\-ready large language model\.*arXiv preprint arXiv:2504\.00698*\.
- Cohere Team \(2026\)Cohere Team\. 2026\.[Introducing command a\+: Making sovereign agentic capabilities available to all](https://cohere.com/blog/command-a-plus)\.
- Cuenca et al\. \(2026\)Pedro Cuenca, Ben Burtenshaw, and Aritra Roy Gosthipaty\. 2026\.[Meta is back with muse glimmer: Local, agentic, multimodal, and open source](https://huggingface.co/blog/muse-glimmer)\.
- Deutsch et al\. \(2025\)Daniel Deutsch, Eleftheria Briakou, Isaac Rayburn Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, and Markus Freitag\. 2025\.[WMT24\+\+: Expanding the language coverage of WMT24 to 55 languages & dialects](https://doi.org/10.18653/v1/2025.findings-acl.634)\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 12257–12284, Vienna, Austria\. Association for Computational Linguistics\.
- Deutsch et al\. \(2023\)Daniel Deutsch, George Foster, and Markus Freitag\. 2023\.[Ties matter: Meta\-evaluating modern metrics with pairwise accuracy and tie calibration](https://doi.org/10.18653/v1/2023.emnlp-main.798)\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 12914–12929, Singapore\. Association for Computational Linguistics\.
- Freitag et al\. \(2022\)Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang\. 2022\.[High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics](https://doi.org/10.1162/tacl_a_00491)\.*Transactions of the Association for Computational Linguistics*, 10:811–825\.
- Gemma Team \(2026\)Gemma Team\. 2026\.[Gemma 4 technical report](https://arxiv.org/abs/2607.02770)\.*Preprint*, arXiv:2607\.02770\.
- GLM\-5\-Team et al\. \(2026\)GLM\-5\-Team, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, and 167 others\. 2026\.[Glm\-5: from vibe coding to agentic engineering](https://arxiv.org/abs/2602.15763)\.*Preprint*, arXiv:2602\.15763\.
- Guerreiro et al\. \(2024\)Nuno M\. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F\. T\. Martins\. 2024\.[xCOMET: Transparent machine translation evaluation through fine\-grained error detection](https://doi.org/10.1162/tacl_a_00683)\.*Transactions of the Association for Computational Linguistics*, 12:979–995\.
- Juraska et al\. \(2024\)Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag\. 2024\.[MetricX\-24: The Google submission to the WMT 2024 metrics shared task](https://doi.org/10.18653/v1/2024.wmt-1.35)\.In*Proceedings of the Ninth Conference on Machine Translation*, pages 492–504, Miami, Florida, USA\. Association for Computational Linguistics\.
- Ki and Carpuat \(2024\)Dayeon Ki and Marine Carpuat\. 2024\.[Guiding large language models to post\-edit machine translation with error annotations](https://doi.org/10.18653/v1/2024.findings-naacl.265)\.In*Findings of the Association for Computational Linguistics: NAACL 2024*, pages 4253–4273, Mexico City, Mexico\. Association for Computational Linguistics\.
- Kocmi et al\. \(2025a\)Tom Kocmi, Arkady Arkhangorodsky, Alexandre Berard, Phil Blunsom, Samuel Cahyawijaya, Théo Dehaze, Marzieh Fadaee, Nicholas Frosst, Matthias Galle, Aidan Gomez, Nithya Govindarajan, Wei\-Yin Ko, Julia Kreutzer, Kelly Marchisio, Ahmet Üstün, Sebastian Vincent, and Ivan Zhang\. 2025a\.[Command\-a\-translate: Raising the bar of machine translation with difficulty filtering](https://doi.org/10.18653/v1/2025.wmt-1.55)\.In*Proceedings of the Tenth Conference on Machine Translation*, pages 789–799, Suzhou, China\. Association for Computational Linguistics\.
- Kocmi et al\. \(2025b\)Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstantin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Howard Lakougna, Jessica Lundin, Christof Monz, Kenton Murray, and 10 others\. 2025b\.[Findings of the WMT25 general machine translation shared task: Time to stop evaluating on easy test sets](https://doi.org/10.18653/v1/2025.wmt-1.22)\.In*Proceedings of the Tenth Conference on Machine Translation*, pages 355–413, Suzhou, China\. Association for Computational Linguistics\.
- Kocmi et al\. \(2026\)Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Sergey Dukanov, Anton Dvorkovich, Mark Fishel, Markus Freitag, Samuel Frontull, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Stan Kharevich, Philipp Koehn, Zheng Li, Jean Maillard, Christof Monz, Alexander Murauski, and 12 others\. 2026\.Findings of the WMT26 general machine translation shared task: Contrastive dynamic human evaluation at scale\.In*Proceedings of the Eleventh Conference on Machine Translation*\. Association for Computational Linguistics\.
- Kocmi et al\. \(2024a\)Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popović, and 3 others\. 2024a\.[Findings of the WMT24 general machine translation shared task: The LLM era is here but MT is not solved yet](https://doi.org/10.18653/v1/2024.wmt-1.1)\.In*Proceedings of the Ninth Conference on Machine Translation*, pages 1–46, Miami, Florida, USA\. Association for Computational Linguistics\.
- Kocmi and Federmann \(2023\)Tom Kocmi and Christian Federmann\. 2023\.[Large language models are state\-of\-the\-art evaluators of translation quality](https://aclanthology.org/2023.eamt-1.19/)\.In*Proceedings of the 24th Annual Conference of the European Association for Machine Translation*, pages 193–203, Tampere, Finland\. European Association for Machine Translation\.
- Kocmi et al\. \(2024b\)Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova\. 2024b\.[Error span annotation: A balanced approach for human evaluation of machine translation](https://doi.org/10.18653/v1/2024.wmt-1.131)\.In*Proceedings of the Ninth Conference on Machine Translation*, pages 1440–1453, Miami, Florida, USA\. Association for Computational Linguistics\.
- Lavie et al\. \(2025\)Alon Lavie, Greg Hanneman, Sweta Agrawal, Diptesh Kanojia, Chi\-Kiu Lo, Vilém Zouhar, Frederic Blain, Chrysoula Zerva, Eleftherios Avramidis, Sourabh Deoghare, Archchana Sindhujan, Jiayi Wang, David Ifeoluwa Adelani, Brian Thompson, Tom Kocmi, Markus Freitag, and Daniel Deutsch\. 2025\.[Findings of the WMT25 shared task on automated translation evaluation systems: Linguistic diversity is challenging and references still help](https://doi.org/10.18653/v1/2025.wmt-1.24)\.In*Proceedings of the Tenth Conference on Machine Translation*, pages 436–483, Suzhou, China\. Association for Computational Linguistics\.
- Lommel et al\. \(2014\)Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt\. 2014\.Multidimensional quality metrics \(mqm\): A framework for declaring and describing translation quality metrics\.*Tradumàtica*, pages 0455–463\.
- Mistral AI \(2026\)Mistral AI\. 2026\.[Mistral medium 3\.5](https://docs.mistral.ai/models/mistral-medium-3-5-26-04)\.
- Mistral AI Team \(2026\)Mistral AI Team\. 2026\.[Ministral 3](https://arxiv.org/abs/2601.08584)\.*Preprint*, arXiv:2601\.08584\.
- NVIDIA Research \(2026\)NVIDIA Research\. 2026\.[Nemotron 3 ultra: Open, efficient mixture\-of\-experts hybrid mamba](https://arxiv.org/abs/2606.15007)\.*Preprint*, arXiv:2606\.15007\.
- OpenAI et al\. \(2025\)OpenAI, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K\. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, and 107 others\. 2025\.[gpt\-oss\-120b and gpt\-oss\-20b model card](https://arxiv.org/abs/2508.10925)\.*Preprint*, arXiv:2508\.10925\.
- Qwen Team \(2026a\)Qwen Team\. 2026a\.[Qwen3\.5\-omni technical report](https://arxiv.org/abs/2604.15804)\.*Preprint*, arXiv:2604\.15804\.
- Qwen Team \(2026b\)Qwen Team\. 2026b\.[Qwen3\.6\-27B: Flagship\-level coding in a 27b dense model](https://qwen.ai/blog?id=qwen3.6-27b)\.
- Rafailov et al\. \(2023\)Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D\. Manning, and Chelsea Finn\. 2023\.Direct preference optimization: your language model is secretly a reward model\.In*Proceedings of the 37th International Conference on Neural Information Processing Systems*, NIPS ’23, Red Hook, NY, USA\. Curran Associates Inc\.
- Semenov et al\. \(2025\)Kirill Semenov, Xu Huang, Vilém Zouhar, Nathaniel Berger, Dawei Zhu, Arturo Oncevay, and Pinzhen Chen\. 2025\.[Findings of the WMT25 terminology translation task: Terminology is useful especially for good MTs](https://doi.org/10.18653/v1/2025.wmt-1.30)\.In*Proceedings of the Tenth Conference on Machine Translation*, pages 554–576, Suzhou, China\. Association for Computational Linguistics\.
- Singh et al\. \(2025\)Anushka Singh, Aarya Pakhale, Mitesh M\. Khapra, and Raj Dabre\. 2025\.[Quality estimation and post\-editing using LLMs for Indic languages: How good is it?](https://aclanthology.org/2025.mtsummit-1.30/)In*Proceedings of Machine Translation Summit XX: Volume 1*, pages 388–398, Geneva, Switzerland\. European Association for Machine Translation\.
- Thinking Machines Lab \(2026\)Thinking Machines Lab\. 2026\.[Introducing inkling\-small](https://thinkingmachines.ai/news/inkling-small/)\.
- Treviso et al\. \(2024\)Marcos V Treviso, Nuno M Guerreiro, Sweta Agrawal, Ricardo Rei, José Pombal, Tania Vaz, Helena Wu, Beatriz Silva, Daan van Stigt, and Andre Martins\. 2024\.[xTower: A multilingual LLM for explaining and correcting translation errors](https://doi.org/10.18653/v1/2024.findings-emnlp.892)\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 15222–15239, Miami, Florida, USA\. Association for Computational Linguistics\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin\. 2017\.Attention is all you need\.*Advances in neural information processing systems*, 30\.
- Zheng et al\. \(2025\)Chujie Zheng, Shixuan Liu, Mingze Li, Xiong\-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin\. 2025\.[Group sequence policy optimization](https://arxiv.org/abs/2507.18071)\.*Preprint*, arXiv:2507\.18071\.
- Zouhar and Kocmi \(2026\)Vilém Zouhar and Tom Kocmi\. 2026\.[Pearmut: Human evaluation of translation made trivial](https://arxiv.org/abs/2601.02933)\.*Preprint*, arXiv:2601\.02933\.

Paramstotal \(active\)AlllanguagesAlbanianBengaliBulgarianCatalanCroatianCzechDanishDutchEgyptianArabicEstonianEuropeanPortugueseNorth Small Translate \(Agentic\)218B \(25B\)84\.478\.787\.580\.184\.079\.088\.889\.189\.584\.273\.590\.4North Small Translate218B \(25B\)83\.677\.982\.879\.583\.978\.389\.285\.389\.584\.072\.689\.6Mistral Large 3675B \(41B\)81\.676\.284\.983\.285\.381\.483\.986\.087\.978\.772\.086\.4Qwen 3\.5 397B397B \(17B\)81\.681\.283\.883\.382\.083\.481\.585\.587\.175\.475\.680\.7DeepL NextGen81\.476\.575\.884\.578\.776\.485\.388\.689\.557\.681\.087\.2Gemma 4 31b31B \(31B\)79\.576\.288\.878\.578\.379\.475\.588\.588\.987\.553\.187\.0Inkling Small276B \(12B\)78\.573\.880\.278\.482\.476\.778\.882\.583\.980\.173\.181\.1Muse Glimmer 30B30B \(30B\)78\.276\.082\.476\.980\.577\.079\.179\.682\.081\.673\.883\.1Command A\+218B \(25B\)76\.573\.479\.375\.478\.876\.083\.385\.082\.971\.865\.979\.5GLM 5\.2744B \(40B\)76\.579\.072\.678\.284\.186\.286\.385\.080\.973\.877\.369\.5Mistral Medium 3\.5128B \(128B\)75\.760\.480\.978\.483\.674\.180\.183\.085\.067\.761\.184\.3GPT\-OSS 120B117B \(5B\)72\.366\.473\.570\.873\.471\.871\.677\.478\.075\.062\.578\.8Command A Translate \(2025\)111B \(111B\)70\.055\.763\.759\.777\.266\.381\.175\.382\.981\.751\.183\.5Google Translate68\.267\.772\.968\.069\.869\.265\.470\.876\.853\.065\.651\.6Nemotron 3 Ultra 550B550B \(55B\)67\.152\.565\.861\.374\.660\.865\.170\.573\.679\.549\.285\.4Qwen 3\.6 27B27B \(27B\)61\.930\.168\.464\.752\.277\.770\.079\.079\.963\.652\.283\.2FilipinoFinnishFrenchGermanGreekHebrewHindiHungarianIcelandicIndonesianIrishItalianJapaneseNorth Small Translate \(Agentic\)82\.886\.190\.289\.789\.487\.188\.874\.867\.290\.169\.690\.387\.9North Small Translate82\.786\.690\.289\.789\.083\.388\.373\.867\.985\.868\.490\.287\.4Mistral Large 377\.580\.088\.787\.781\.180\.684\.681\.469\.186\.362\.688\.086\.7Qwen 3\.5 397B67\.983\.481\.987\.079\.782\.279\.682\.771\.486\.167\.186\.581\.2DeepL NextGen75\.385\.086\.686\.482\.785\.777\.882\.767\.287\.770\.688\.484\.6Gemma 4 31b86\.182\.690\.489\.674\.180\.889\.555\.441\.390\.018\.989\.090\.9Inkling Small74\.178\.582\.282\.079\.178\.979\.077\.569\.281\.267\.984\.480\.4Muse Glimmer 30B76\.376\.283\.884\.177\.877\.279\.978\.565\.782\.959\.182\.883\.7Command A\+73\.678\.285\.584\.581\.981\.679\.470\.353\.780\.154\.284\.879\.4GLM 5\.260\.772\.489\.185\.958\.572\.076\.173\.865\.178\.151\.784\.190\.0Mistral Medium 3\.570\.872\.888\.186\.876\.175\.881\.375\.148\.883\.944\.986\.384\.9GPT\-OSS 120B75\.069\.582\.579\.167\.967\.876\.770\.255\.780\.050\.779\.479\.9Command A Translate \(2025\)64\.762\.184\.283\.480\.279\.677\.268\.942\.182\.038\.484\.379\.2Google Translate66\.966\.478\.575\.164\.964\.971\.764\.464\.277\.565\.076\.771\.8Nemotron 3 Ultra 550B70\.057\.886\.983\.958\.484\.484\.959\.045\.276\.843\.483\.989\.9Qwen 3\.6 27B50\.158\.878\.375\.859\.419\.481\.169\.325\.483\.216\.884\.475\.6KoreanLatvianLithuanianMalayMalteseNorwegianBokmalPanjabiPersianPolishRomanianRussianSerbianSimplifiedChineseNorth Small Translate \(Agentic\)88\.971\.669\.687\.365\.090\.287\.887\.487\.288\.188\.874\.989\.1North Small Translate84\.471\.268\.587\.065\.690\.087\.687\.087\.287\.688\.173\.985\.4Mistral Large 386\.771\.277\.883\.146\.484\.583\.583\.585\.385\.685\.980\.387\.6Qwen 3\.5 397B83\.279\.278\.379\.770\.981\.284\.783\.983\.184\.886\.575\.986\.2DeepL NextGen87\.080\.180\.877\.769\.487\.877\.874\.382\.185\.884\.373\.585\.5Gemma 4 31b92\.148\.556\.889\.230\.688\.188\.189\.488\.983\.389\.969\.890\.4Inkling Small84\.169\.673\.979\.368\.281\.778\.581\.680\.780\.180\.163\.086\.1Muse Glimmer 30B82\.768\.474\.480\.758\.080\.880\.580\.478\.681\.180\.968\.785\.4Command A\+81\.165\.765\.778\.252\.685\.278\.779\.580\.082\.979\.574\.681\.3GLM 5\.286\.073\.878\.079\.358\.683\.937\.085\.773\.483\.388\.475\.292\.1Mistral Medium 3\.581\.555\.059\.377\.034\.782\.277\.081\.081\.080\.582\.874\.986\.8GPT\-OSS 120B78\.859\.462\.976\.748\.077\.974\.271\.374\.976\.674\.661\.281\.5Command A Translate \(2025\)79\.549\.959\.771\.936\.374\.348\.779\.880\.082\.778\.453\.080\.9Google Translate72\.665\.062\.767\.168\.371\.165\.266\.068\.467\.369\.666\.675\.4Nemotron 3 Ultra 550B87\.446\.749\.669\.631\.661\.369\.966\.467\.954\.977\.543\.587\.4Qwen 3\.6 27B63\.936\.78\.779\.810\.675\.026\.282\.177\.848\.185\.359\.691\.0SlovakSlovenianSpanishStandardArabicSwedishTamilTeluguThaiTraditionalChineseTurkishUkrainianUrduVietnameseNorth Small Translate \(Agentic\)84\.676\.190\.986\.689\.285\.785\.986\.788\.885\.386\.887\.189\.7North Small Translate81\.277\.290\.586\.788\.985\.785\.786\.788\.585\.786\.986\.989\.4Mistral Large 379\.374\.588\.480\.282\.782\.282\.584\.287\.882\.984\.780\.886\.6Qwen 3\.5 397B79\.181\.086\.785\.484\.683\.983\.882\.082\.884\.085\.978\.286\.5DeepL NextGen84\.081\.988\.482\.088\.477\.979\.881\.984\.685\.184\.175\.786\.7Gemma 4 31b71\.263\.190\.688\.188\.885\.887\.590\.587\.488\.187\.788\.590\.4Inkling Small77\.876\.685\.278\.984\.075\.677\.778\.181\.677\.282\.076\.382\.8Muse Glimmer 30B77\.574\.084\.774\.380\.979\.778\.776\.182\.578\.280\.979\.382\.8Command A\+77\.273\.987\.282\.083\.076\.575\.978\.353\.677\.878\.477\.481\.9GLM 5\.283\.673\.881\.786\.781\.346\.731\.884\.688\.885\.383\.884\.187\.7Mistral Medium 3\.571\.365\.687\.977\.882\.973\.973\.979\.084\.277\.881\.875\.383\.8GPT\-OSS 120B69\.066\.682\.372\.977\.770\.172\.772\.181\.973\.572\.575\.374\.6Command A Translate \(2025\)73\.359\.485\.682\.074\.766\.444\.261\.279\.176\.379\.356\.679\.9Google Translate65\.665\.476\.573\.069\.471\.478\.376\.541\.370\.465\.554\.078\.8Nemotron 3 Ultra 550B60\.756\.388\.168\.571\.954\.152\.165\.083\.867\.665\.770\.673\.3Qwen 3\.6 27B70\.942\.581\.661\.974\.534\.773\.253\.789\.939\.769\.775\.984\.6
Table 6:Machine translation quality of every supported language of the WMT26 testset\.## Appendix AGEMBA\-ESA Fluency\+Adequacy Template

### A\.1System prompt

You are an impartial judge whose task is to analyze machine translation outputs and evaluate their quality\.Based on the provided source text, target language, translation guidelines, and translation output, identify translation errors and assess their severity\. Then evaluate the quality of the translation according to the provided rubric criteria\. For each criterion, provide a brief rationale in English and a score within the allowed range\.Errors have "quote", "severity", and "category" attributes\. The "quote" value should quote the part of the translation that contains the error\. Use minor severity for imperfections or stylistic issues that do not impact the core message \(e\.g\., awkward phrasing\)\. Use major severity for errors that obscure the meaning, misrepresent the source, or change the message \(e\.g\., incorrect information or confusing wording\)\. The category must be one of: "accuracy", "fluency", "terminology", "style", "locale", "design", "mistranslation", "omission", "addition", "untranslated", "grammar", "spelling", "punctuation", "register", "unidiomatic", "inconsistency", or "other"\. If the error span is longer than a few words, use an ellipsis \(\.\.\.\) to shorten it\. If the error concerns the entire translation, such as a translation in the wrong language, set "quote" to "FULL\_TRANSLATION"\. For omissions \(missing source content\), set "quote" to the adjacent translation words with "OMISSION" between them\. If there are no errors, set "errors" to an empty array\.Return your error analysis and quality evaluation as a single, strictly valid JSON object matching the provided output format\.

### A\.2User prompt

\#\# Evaluation rubric\- Fluency score \[key: fluency\]: Measures how natural and grammatically correct the translation reads in the target language\. \(score: 0 to 100\)\- Adequacy score \[key: adequacy\]: Measures how completely the meaning of the source is conveyed by the translation\. \(score: 0 to 100\)\#\# Additional judging rules\#\#\# Important rules\- Hallucinations are major adequacy errors and should be penalized severely\.\- Outputs in the wrong language, dialect, or script are major fluency errors and should be penalized severely\. Borrowed words are allowed if valid in the target language\.\- Check translation consistency \(e\.g\., technical terms\) across the entire text\.\- Over\-translation and extraneous text \(e\.g\., introductory statements like "Here is your translation" or unprompted explanations\) are major adequacy errors\.\- Count guideline adherence issues toward the adequacy score; for these errors, set "quote" to the corresponding guideline\.\#\#\# Adequacy scale\- 81\-100 \(Very Good\): Complete meaning transfer; no information is lost, added, or distorted\.\- 61\-80 \(Good\): Near\-complete transfer; minor inaccuracies or omissions that do not affect the core message\.\- 41\-60 \(Acceptable\): Main ideas conveyed, but noticeable inaccuracies, omissions, or additions\.\- 21\-40 \(Borderline\): Partial transfer; frequent misinterpretation or omission confusing the core message\.\- 0\-20 \(Not acceptable\): Complete violation of meaning; large portions mistranslated, hallucinated, or missing\.\#\#\# Fluency scale\- 81\-100 \(Very Good\): Perfectly natural and grammatical; reads like a native text; requires no or minimal proofreading\.\- 61\-80 \(Good\): Mostly natural; minor awkwardness or slight grammatical imperfections; needs light proofreading\.\- 41\-60 \(Acceptable\): Uneven naturalness; noticeable awkward phrasing or structural issues; usable only after substantial revision\.\- 21\-40 \(Borderline\): Often unnatural; frequent grammatical errors that impede reading flow; requires major rewrite\.\- 0\-20 \(Not acceptable\): Incoherent, structurally broken, incomprehensible, or in the wrong language, dialect, or script; unusable without complete retranslation\.\#\# Source text\`\`\`\{\{SOURCE\_TEXT\}\}\`\`\`\#\# Translation language or dialect\{\{TARGET\_LANGUAGE\_OR\_DIALECT\}\}\#\# Translation guidelines\{\{TRANSLATION\_GUIDELINES\}\}\#\# Translation output\`\`\`\{\{TRANSLATION\_OUTPUT\}\}\`\`\`\#\# Expected output formatReturn a single, strictly valid JSON object with the following schema:\`\`\`json\{"errors": \[\{"quote": "<string\>","severity": "minor\|major","category": "<string\>"\}\],"fluency": \{"rationale": "<string\>","score": "<number\>"\},"adequacy": \{"rationale": "<string\>","score": "<number\>"\}\}\`\`\`Every string value must be valid JSON: escape each backslash as \\\\ and each double quote as \\"\. Do not emit raw control characters or trailing commas\.