ComplexityMT: Benchmarking the Interaction Between Text Complexity and Machine Translation

# ComplexityMT: Benchmarking the Interaction Between Text Complexity and Machine Translation
Source: [https://arxiv.org/html/2606.05421](https://arxiv.org/html/2606.05421)
Joseph Marvin Imperial1,3,Junhong Liang4,Belal Shoer4,Abdullah Barayan2,9, Rodrigo Wilkens5,Omar Mussa10,Dawn Knight2,Eugénio Ribeiro6,7,Ekaterina Kochmar4, Sowmya Vajjala8,Fernando Alva\-Manchego2,Harish Tayyar Madabushi1 1University of Bath,2Cardiff University,3National University Philippines,4MBZUAI, 5University of Exeter,6INESC\-ID Lisboa,7Instituto Universitário de Lisboa \(ISCTE\-IUL\), ISTAR 8National Research Council, Canada,9King Abdulaziz University,10Saudi Electronic University [jmri20@bath\.ac\.uk](https://arxiv.org/html/2606.05421v1/mailto:[email protected]),[alvamanchegof@cardiff\.ac\.uk](https://arxiv.org/html/2606.05421v1/mailto:[email protected])

###### Abstract

When a text is translated, does the translation retain the complexity of the original?We introduceComplexityMT, a new challenge for assessing how text complexity and machine translation interact with and influence each other, using the[Common European Framework of Reference for Languages](https://arxiv.org/html/2606.05421#id1.1.id1)\([CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)\) levels as the measure of text complexity\. Across six languages, including Arabic, Dutch, English, French, Hindi, and Russian, we evaluate three open\-weight models, one closed model, and a commercial machine translation system on two tasks: i\)correlationof[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)with translation difficulty, and ii\)shiftsin[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)levels of the source texts\. Our experiments show that higher[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)levels make texts more difficult to translate, and that machine translation shifts the CEFR level of the target text compared to the original source, for most languages\. These findings provide new insights for researchers and practitioners working on multilingual pedagogical content generation and MT difficulty estimation\.

CEFRCommon European Framework of Reference for LanguagesLLMLarge Language ModelMTMachine TranslationNLPNatural Language Processing
ComplexityMT: Benchmarking the Interaction Between Text Complexity and Machine Translation

Joseph Marvin Imperial1,3, Junhong Liang4, Belal Shoer4, Abdullah Barayan2,9,Rodrigo Wilkens5,Omar Mussa10,Dawn Knight2,Eugénio Ribeiro6,7,Ekaterina Kochmar4,Sowmya Vajjala8,Fernando Alva\-Manchego2,Harish Tayyar Madabushi11University of Bath,2Cardiff University,3National University Philippines,4MBZUAI,5University of Exeter,6INESC\-ID Lisboa,7Instituto Universitário de Lisboa \(ISCTE\-IUL\), ISTAR8National Research Council, Canada,9King Abdulaziz University,10Saudi Electronic University[jmri20@bath\.ac\.uk](https://arxiv.org/html/2606.05421v1/mailto:[email protected]),[alvamanchegof@cardiff\.ac\.uk](https://arxiv.org/html/2606.05421v1/mailto:[email protected])

![Refer to caption](https://arxiv.org/html/2606.05421v1/x1.png)Figure 1:We observe two limitations in existing machine translation models: i\) arobustnessproblem where they tend to produce lower quality translations \(via COMET or GEMBA\) for higher\-complexity source texts, and ii\) apreservationproblem where they tend to change the pedagogical complexity levels \(e\.g\., CEFR\) of the source text\.## 1Introduction

Creating reading materials tailored to a learner’s proficiency level has been a topic of interest in educational research for a long time\. Approaches such as traditional readability formulasKincaid et al\. \([1975](https://arxiv.org/html/2606.05421#bib.bib22)\); DuBay \([2004](https://arxiv.org/html/2606.05421#bib.bib18)\); Crossley et al\. \([2017](https://arxiv.org/html/2606.05421#bib.bib16)\)and proprietary measures such as Lexile scoresLennon and Burdick \([2004](https://arxiv.org/html/2606.05421#bib.bib24)\)were used in the past to evaluate existing materials and adapt them to specific reading levels\. More recently,[Natural Language Processing](https://arxiv.org/html/2606.05421#id4.4.id4)\([NLP](https://arxiv.org/html/2606.05421#id4.4.id4)\) research has enabled automation of two foundational tasks in this direction, namely automatic readability assessmentAluisio et al\. \([2010](https://arxiv.org/html/2606.05421#bib.bib2)\); Ciobanu et al\. \([2015](https://arxiv.org/html/2606.05421#bib.bib13)\); Xia et al\. \([2016](https://arxiv.org/html/2606.05421#bib.bib48)\); Deutsch et al\. \([2020](https://arxiv.org/html/2606.05421#bib.bib17)\); Vajjala \([2022](https://arxiv.org/html/2606.05421#bib.bib44)\)and text simplificationMaddela and Xu \([2018](https://arxiv.org/html/2606.05421#bib.bib28)\); Scarton and Specia \([2018](https://arxiv.org/html/2606.05421#bib.bib39)\); Nishihara et al\. \([2019](https://arxiv.org/html/2606.05421#bib.bib35)\); Alva\-Manchego et al\. \([2020](https://arxiv.org/html/2606.05421#bib.bib3)\); Maddela et al\. \([2021](https://arxiv.org/html/2606.05421#bib.bib27)\); Sheang and Saggion \([2021](https://arxiv.org/html/2606.05421#bib.bib42)\); Alva\-Manchego et al\. \([2025](https://arxiv.org/html/2606.05421#bib.bib5)\); Barayan et al\. \([2025](https://arxiv.org/html/2606.05421#bib.bib9)\), which could together support the creation of targeted reading level\-specific texts\. Research in this direction has understandably focused on high\-resource languages like English due to the abundance of data and models\.[Machine Translation](https://arxiv.org/html/2606.05421#id3.3.id3)\([MT](https://arxiv.org/html/2606.05421#id3.3.id3)\) offers a natural method to extend reading level\-appropriate texts into more languagesXu et al\. \([2016](https://arxiv.org/html/2606.05421#bib.bib49)\); Marchisio et al\. \([2019](https://arxiv.org/html/2606.05421#bib.bib29)\); Alva\-Manchego and Shardlow \([2022](https://arxiv.org/html/2606.05421#bib.bib4)\); Zouhar et al\. \([2026](https://arxiv.org/html/2606.05421#bib.bib50)\)\. But understanding whether existing MT models are actually fit for this purpose requires empirical evidence of how text complexity affects translation qualities and if it preserves the complexity of the source text being translated\. Understanding these interactions will guide how to leverage MT and text simplification to scale content generation across languages\.

The assumption that complex text could be difficult to translate is not new\. Text simplification was considered a pre\-processing step to reduce translation difficulty and improve translation quality, both in earlier MT models as well as more recently\(e\.g\., Chandrasekar et al\.,[1996](https://arxiv.org/html/2606.05421#bib.bib12); Mehta et al\.,[2020](https://arxiv.org/html/2606.05421#bib.bib30)\), and recent work has shown thatcomplex texts are difficult to translateShardlow and Alva\-Manchego \([2022](https://arxiv.org/html/2606.05421#bib.bib41)\)\. At the same time, a separate line of work describessimplification as a translation universal, i\.e\., translated text is shown to be more readable and less complex, as it relies on high\-frequency tokens in the target languageCorpas Pastor et al\. \([2008](https://arxiv.org/html/2606.05421#bib.bib14)\); Lu et al\. \([2021](https://arxiv.org/html/2606.05421#bib.bib26)\); Wastl et al\. \([2025](https://arxiv.org/html/2606.05421#bib.bib47)\)\. In all these cases, the notion of complexity has predominantly been binary\. Drawing on what appears to be contrasting viewpoints on the relationship between text complexity and machine translation, we revisit these questions in our paper, specifically in the context of pedagogical difficulty, which we operationalize using the[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)scaleCouncil of Europe \([2001](https://arxiv.org/html/2606.05421#bib.bib15)\)\. Thus, we pose the following research questions:

- •RQ1: How does the[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level of a textcorrelatewith translation difficulty?
- •RQ2: How does the[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level of a textchangeunder translation?

These questions, motivated by problems in multilingual pedagogically\-grounded content generation, also contribute to[MT](https://arxiv.org/html/2606.05421#id3.3.id3)difficulty estimation research\. To our knowledge, the interaction between MT and pedagogical difficulty\-based content generation has not been studied in past[NLP](https://arxiv.org/html/2606.05421#id4.4.id4)research, and this paper introduces a framework to study this through two tasks \(see Section[3](https://arxiv.org/html/2606.05421#S3)\)\.

## 2Related Work

We consider two strands of existing[NLP](https://arxiv.org/html/2606.05421#id4.4.id4)research as directly related to our current work:[MT](https://arxiv.org/html/2606.05421#id3.3.id3)difficulty estimation, and complexity\-controlled[MT](https://arxiv.org/html/2606.05421#id3.3.id3)\.

#### MT Difficulty Estimation

[MT](https://arxiv.org/html/2606.05421#id3.3.id3)is one of the core research problems in[NLP](https://arxiv.org/html/2606.05421#id4.4.id4), and the topics of[MT](https://arxiv.org/html/2606.05421#id3.3.id3)difficulty estimation, i\.e\., how difficult a text is for an[MT](https://arxiv.org/html/2606.05421#id3.3.id3)system to translate and how to mitigate this effect, have all received some attention in past research\. Various text\-level features \(e\.g\., text length and lexical diversity\) have been explored to model source text difficulty in machine translationHale and Campbell \([2002](https://arxiv.org/html/2606.05421#bib.bib20)\); Mishra et al\. \([2013](https://arxiv.org/html/2606.05421#bib.bib32)\); Li et al\. \([2014](https://arxiv.org/html/2606.05421#bib.bib25)\); Bugliarello et al\. \([2020](https://arxiv.org/html/2606.05421#bib.bib11)\); Araghi and Palangkaraya \([2024](https://arxiv.org/html/2606.05421#bib.bib7)\); Proietti et al\. \([2025](https://arxiv.org/html/2606.05421#bib.bib37)\); Zouhar et al\. \([2026](https://arxiv.org/html/2606.05421#bib.bib50)\)\.Proietti et al\. \([2025](https://arxiv.org/html/2606.05421#bib.bib37)\)have recently considered the[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)scaleCouncil of Europe \([2001](https://arxiv.org/html/2606.05421#bib.bib15)\)as a way to characterize translation difficulty\. Text simplification has been explored as a preprocessing step before[MT](https://arxiv.org/html/2606.05421#id3.3.id3), to make the text easier to \(machine\) translate and improve translation qualityChandrasekar et al\. \([1996](https://arxiv.org/html/2606.05421#bib.bib12)\); Mehta et al\. \([2020](https://arxiv.org/html/2606.05421#bib.bib30)\)\. However, to our knowledge, all prior research on[MT](https://arxiv.org/html/2606.05421#id3.3.id3)difficulty estimation has focused primarily on sentence\-level text and English source data and has been motivated primarily by improving[MT](https://arxiv.org/html/2606.05421#id3.3.id3)translation quality, rather than by a pedagogical justification like ours\.

In this paper, we look at[MT](https://arxiv.org/html/2606.05421#id3.3.id3)difficulty estimation by considering the[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)scale as the text difficulty measure, and[MT](https://arxiv.org/html/2606.05421#id3.3.id3)quality as the difficulty measure for machine translation\. Unlike other works on this topic, we go beyond sentence\-level analyses and English and also consider the document\-level across multiple languages to explore this question\.

#### Complexity Controlled MT

Generating translation variants by controlling for aspects such as politenessSennrich et al\. \([2016](https://arxiv.org/html/2606.05421#bib.bib40)\), formalityNadejde et al\. \([2022](https://arxiv.org/html/2606.05421#bib.bib33)\), and personalizationMirkin and Meunier \([2015](https://arxiv.org/html/2606.05421#bib.bib31)\)is well\-studied in[NLP](https://arxiv.org/html/2606.05421#id4.4.id4)research\. In this line of research, controlling for text complexity in[MT](https://arxiv.org/html/2606.05421#id3.3.id3)Agrawal and Carpuat \([2019](https://arxiv.org/html/2606.05421#bib.bib1)\); Marchisio et al\. \([2019](https://arxiv.org/html/2606.05421#bib.bib29)\); Tani et al\. \([2022](https://arxiv.org/html/2606.05421#bib.bib43)\); Zouhar et al\. \([2026](https://arxiv.org/html/2606.05421#bib.bib50)\)is somewhat related to our second research question, although the specific question of whether translation can preserve the original text complexity is not explored in previous work\. Further, this strand of research assumes that the target translation’s text complexity level is prespecified\.

In this paper, we explore whether translation preserves the source language’s text complexity in the target language\. Compared to previous works, we use a pedagogical complexity construct, CEFR, as our main reference for complexity for measuring shifts from MT models\.

## 3ComplexityMT: Benchmarking the Interaction between Text Complexity and Machine Translation

As discussed in Section[2](https://arxiv.org/html/2606.05421#S2), previous work has expressed contrasting views on the relationship between text complexity and[MT](https://arxiv.org/html/2606.05421#id3.3.id3)difficulty \(as measured by MT quality\)\. In that context, we introduceComplexityMT, a framework for assessing the impact of text complexity on[MT](https://arxiv.org/html/2606.05421#id3.3.id3)across two core aspects: robustness and preservation\.Robustnesscaptures the expectation that good[MT](https://arxiv.org/html/2606.05421#id3.3.id3)models should maintain translation quality across the text\-complexity spectrum, thereby addressing RQ1\.Preservationbuilds on the expectation that text complexity is maintained across translations, addressing RQ2\. The following sections describe our experimental pipelines to assess both aspects\.

### 3\.1ComplexityMT\-Robustness

This task assesses whether[MT](https://arxiv.org/html/2606.05421#id3.3.id3)quality correlates with the complexity of the source text\. Given a set of[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)\-labeled source texts and a set of target languages, the robustness of an[MT](https://arxiv.org/html/2606.05421#id3.3.id3)system is assessed as follows:

1. 1\.We translate each source textxxwith an assigned gold\-standard[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)levelℓx∈\{A​1,A​2,B​1,B​2,C​1,C​2\}\\ell\_\{x\}\\in\\\{A1,A2,B1,B2,C1,C2\\\}into each target languageLtgtL\_\{\\text\{tgt\}\}using the MT model under evaluation to produce translationyy;
2. 2\.We then estimate a reference\-free[MT](https://arxiv.org/html/2606.05421#id3.3.id3)quality scoreq​\(y\)∈\[0,1\]q\(y\)\\in\[0,1\];
3. 3\.Finally, we compute the Spearman correlationρ=corr​\(ℓx,q​\(y\)\)\\rho=\\mathrm\{corr\}\(\\ell\_\{x\},q\(y\)\)between the source[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level and the[MT](https://arxiv.org/html/2606.05421#id3.3.id3)quality score\.

We use the computed correlationρ\\rhobetween textual complexity and[MT](https://arxiv.org/html/2606.05421#id3.3.id3)quality as the main robustness metric, with values closer to zero indicating higher robustness\. A significant negative correlation \(ρ<0\\rho<0\) indicates that the translation quality decreases as the[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level increases, implying that higher\-level texts pose greater challenges for the[MT](https://arxiv.org/html/2606.05421#id3.3.id3)system\. Conversely, a positive correlation \(ρ\>0\\rho\>0\) suggests that higher\-level texts receive better quality scores, a less intuitive but theoretically possible outcome\.

### 3\.2ComplexityMT\-Preservation

This task assesses whether[MT](https://arxiv.org/html/2606.05421#id3.3.id3)systems preserve the[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)levels of the source texts upon translation\. Since obtaining manual text\-complexity annotations for the translations generated by each evaluated[MT](https://arxiv.org/html/2606.05421#id3.3.id3)system is not feasible, we rely on a pretrained multilingual[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level classifier \(see §[4\.4](https://arxiv.org/html/2606.05421#S4.SS4)for details\)\. To reduce the translation effects from the CEFR classifier’s calibration errors, we evaluate[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level preservation via a backtranslation procedure with a model\-anchored shift, rather than a direct gold\-vs\-classifier comparison, as described below:

1. 1\.Given a source textxxin languageLsrcL\_\{\\text\{src\}\}with the gold\-standard[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)levelℓgold\\ell\_\{\\text\{gold\}\}, we translatexxinto a pivot languageLpivL\_\{\\text\{piv\}\}with an[MT](https://arxiv.org/html/2606.05421#id3.3.id3)model, producing the forward translationyfwdy\_\{\\text\{fwd\}\};
2. 2\.We then translateyfwdy\_\{\\text\{fwd\}\}back intoLsrcL\_\{\\text\{src\}\}with the same[MT](https://arxiv.org/html/2606.05421#id3.3.id3)model, producing the back\-translationybacky\_\{\\text\{back\}\};
3. 3\.Next, we apply the[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level classifierffindependently to each translation output to obtain the forward and backtranslation[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level predictions:ℓ^fwd=f​\(yfwd\)\\hat\{\\ell\}\_\{\\text\{fwd\}\}=f\(y\_\{\\text\{fwd\}\}\)andℓ^back=f​\(yback\)\\hat\{\\ell\}\_\{\\text\{back\}\}=f\(y\_\{\\text\{back\}\}\);
4. 4\.Finally, we compute the model\-anchored[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level shift: Δ​ℓmodel=ℓ^back−ℓ^fwd\.\\Delta\\ell\_\{\\text\{model\}\}=\\hat\{\\ell\}\_\{\\text\{back\}\}\-\\hat\{\\ell\}\_\{\\text\{fwd\}\}\.\(1\)

We use the computed[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level shiftΔ​ℓmodel\\Delta\\ell\_\{\\text\{model\}\}as the main preservation metric, where a value ofΔ​ℓmodel=0\\Delta\\ell\_\{\\text\{model\}\}=0signals that the backtranslation process preserved the original[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level, while a value ofΔ​ℓmodel≠0\\Delta\\ell\_\{\\text\{model\}\}\\neq 0indicates a net[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level shift induced by the[MT](https://arxiv.org/html/2606.05421#id3.3.id3)model during translation\. The purpose of anchoring on the classifier’s forward prediction is thatit minimizes the classifier’s disagreement with the gold label of the source text, thereby leaving the effect exclusively to the classifier’s predictions\. We note that the classifier is applied to texts of two different languages on the two legs of backtranslation \(LpivL\_\{\\text\{piv\}\}foryfwdy\_\{\\text\{fwd\}\}andLsrcL\_\{\\text\{src\}\}forybacky\_\{\\text\{back\}\}\), thus per\-language prediction differences in the classifier are not canceled inΔ​ℓmodel\\Delta\\ell\_\{\\text\{model\}\}\. We address this by investigating robustness across three classifiers that are structurally different \(Section[5](https://arxiv.org/html/2606.05421#S5)\)\.

Table 1:The distribution of multilingual reference texts extracted from theUniversalCEFRtest split \(n=1,515n=1,515\) we used for this study across the CEFR levels and formats \(document\- and sentence\-level\)\.## 4Experimental Setup

In this section, we detail an implementation and application ofComplexityMT\. The framework can easily be extended to additional languages, used to evaluate other[MT](https://arxiv.org/html/2606.05421#id3.3.id3)systems, and improved through advances in automatic[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level prediction\.

### 4\.1Data

We use a subset of[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)\-labeled texts from theUniversalCEFR\(Imperial et al\.,[2025](https://arxiv.org/html/2606.05421#bib.bib21)\)test split at both the sentence and document levels\.111[https://huggingface\.co/UniversalCEFR](https://huggingface.co/UniversalCEFR)We filteredUniversalCEFRto extract the reference\-level texts that are associated with gold\-standard CEFR levels as reported in Table[3\.2](https://arxiv.org/html/2606.05421#S3.SS2)\. The sentence\-level data cover multiple source languages, including English, French, Arabic, Hindi, and Russian, while the document\-level data include English, French, and Dutch\. Each text is associated with an original[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)label from A1 to C2\. For analysis,[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)levels are assigned to an ordinal scale, where A1 = 1, A2 = 2, B1 = 3, B2 = 4, C1 = 5 and C2 = 6\. Table[3\.2](https://arxiv.org/html/2606.05421#S3.SS2)shows the distribution of[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)levels in the document\-level and sentence\-level subsets\. The document\-level subset contains 515 instances across three languages, while the sentence\-level subset contains 1,000 instances across five languages, with 200 instances per language\.

### 4\.2Translation Models

We evaluate five diverse[MT](https://arxiv.org/html/2606.05421#id3.3.id3)systems spanning different training objectives and scales: a general\-purpose LLMGPT\-5\.4OpenAI \([2025](https://arxiv.org/html/2606.05421#bib.bib36)\); three translation\-specialized LLMs,TowerInstruct\-7BAlves et al\. \([2024](https://arxiv.org/html/2606.05421#bib.bib6)\)andTranslateGemmaGoogle Translate Research Team et al\. \([2026](https://arxiv.org/html/2606.05421#bib.bib19)\)in its 4B and 12B variants; and a commercial translation system,Google Cloud Translation API\.222[https://cloud\.google\.com/translate](https://cloud.google.com/translate)All models were accessed between February and May 2026\.

### 4\.3Translation Quality Metrics

We select two reference\-free MT quality metrics forComplexityMT\-Robustness, specificallyCOMET333[https://github\.com/Unbabel/COMET](https://github.com/Unbabel/COMET)Rei et al\. \([2020](https://arxiv.org/html/2606.05421#bib.bib38)\)which measures quality via an encoder\-based multilingual BERT model, andGEMBA\-DA444[https://github\.com/MicrosoftTranslator/GEMBA](https://github.com/MicrosoftTranslator/GEMBA)Kocmi and Federmann \([2023](https://arxiv.org/html/2606.05421#bib.bib23)\), which measures quality via prompting GPT\-5\.4 for a direct assessment\. Both metrics have been shown to correlate strongly with human evaluation\. We use a uniform score scale of\[0,1\]\[0,1\]where higher values indicate better translations\.

### 4\.4CEFR Level Classifiers

To assess whether the[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level classifier impacts the results of the benchmark, we use a recent state\-of\-the\-artXLM\-Rmodel finetuned with the massively multilingual train split of theUniversalCEFRdata\. We verified that there were no overlaps between the train split used for training the XLM\-R CEFR classifier and the curated test split used for the evaluation forComplexityMT\-RobustnessandComplexityMT\-Preservation\. Thus, we guarantee that there is no data leakage within the experiments\. For comparison, we also perform cross\-analysis with two additional CEFR classifiers that were trained with the same data but that differ architecturally – one is based on the ModernBERT architectureWarner et al\. \([2025](https://arxiv.org/html/2606.05421#bib.bib46)\)and another is trained with 146 hand\-engineered linguistic features using Random Forest \(see Appendix[A\.3](https://arxiv.org/html/2606.05421#A1.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2606.05421v1/x2.png)Figure 2:Spearman correlation between source CEFR levels and COMET scores across MT models for sentence\-level texts\. \* indicates statistical significance \(p<0\.05p<0\.05\)\. Negative values indicate that translation quality, as measured by COMET, decreases as the CEFR level of the source text increases\.![Refer to caption](https://arxiv.org/html/2606.05421v1/x3.png)Figure 3:Spearman correlation between source CEFR levels and GEMBA\-DA scores across MT models for sentence\-level texts\. \* indicates statistical significance \(p<0\.05p<0\.05\)\. The significance pattern matches the results of COMET except for Russian where it diverges for GEMBA, although the absolute correlations are smaller\.![Refer to caption](https://arxiv.org/html/2606.05421v1/x4.png)

![Refer to caption](https://arxiv.org/html/2606.05421v1/x5.png)

Figure 4:Spearman correlation between source CEFR levels and COMET scores \(left\) and GEMBA scores \(right\) across MT models for document\-level texts\. \* indicates statistical significance \(p<0\.05p<0\.05\)\.![Refer to caption](https://arxiv.org/html/2606.05421v1/x6.png)Figure 5:Mean model\-anchored CEFR level shifts from back\-translations across MT models for sentence\-level texts\. We anchor theUniversalCEFRclassifier’s forward prediction to isolate the back\-translation’s effects from the classifier bias\.## 5Results

In this section, we discuss the results of our experiments exploring how text complexity and MT affect each other across languages\.

### 5\.1Translation Quality Declines with CEFR Level

We address RQ1 through theComplexityMT\-Robustnesstask described in Section[3\.1](https://arxiv.org/html/2606.05421#S3.SS1)\. Figures[2](https://arxiv.org/html/2606.05421#S4.F2)and[3](https://arxiv.org/html/2606.05421#S4.F3)present Spearman correlation heatmaps capturing the relationship between text complexity and MT quality across the five multilingual MT models at the sentence level, while Figure[4](https://arxiv.org/html/2606.05421#S4.F4)does the same for the document level\.

Across most source languages at the sentence level, we observe a distinct pattern wheretranslation quality correlates negatively with the source CEFR levelsfor Arabic and English, meaning that texts of higher CEFR levels are associated with lower translation quality according to COMET scores\. These significant correlations are observed within the range of−0\.16\-0\.16to−0\.40\-0\.40, and this pattern is consistent with the general expectation that higher\-proficiency texts, which tend to contain more complex syntactic structures, richer vocabulary, and denser information, pose greater challenges for MT models\. For GEMBA, we observe the same predominantly negative correlation pattern at smaller magnitudes\. Interestingly, Russian stands out as an outlier with positive COMET correlations with CEFR level not seen under GEMBA\. We posit that most MT models have been trained on higher\-level Russian texts, which are more likely to conform to conventional written registers, thereby yielding higher translation scores\.

At the document level, we observe thatnegative correlation between translation quality and CEFR levelsis stronger and more uniform\. For English and French, all MT models show significant negative correlations with both translation quality metrics whereρ≈−0\.47\\rho\\approx\-0\.47and−0\.34\-0\.34for COMET, andρ≈−0\.50\\rho\\approx\-0\.50and−0\.31\-0\.31for GEMBA\. For NL, this correlation is less pronounced withρ≈−0\.13\\rho\\approx\-0\.13and−0\.04\-0\.04\. We posit that the effect is more pronounced for documents than for sentences because longer inputs may carry more linguistic complexity, allowing translation difficulty to converge and yield less noise than for a single sentence\.

### 5\.2Translation Shifts CEFR at Document Level

We observe how the CEFR level of a text changes under MT by visualizing the mean model\-anchored level shifts from the backtranslation process forComplexityMT\-Preservation\(Section[3\.2](https://arxiv.org/html/2606.05421#S3.SS2)\) experiments across the five multilingual MT models\. Figures[5](https://arxiv.org/html/2606.05421#S4.F5)and[6](https://arxiv.org/html/2606.05421#S5.F6)show sentence\-level and document\-level results, respectively, and cross\-classifier robustness checks are reported in Table[5\.2](https://arxiv.org/html/2606.05421#S5.SS2)\.

At the sentence level, the mean CEFR shift is minimal atΔ​ℓ≤\+0\.07\\Delta\\ell\\leq\+0\.07across all MT models with 95% confidence interval excluding zero, indicating a small but statistically distinguishable upward shift\. However, this effect is not uniform across languages and is most evident with Russian, where it receives a higher CEFR classifier prediction by≈\+0\.6\\approx\+0\.6levels on average and up to\+0\.95\+0\.95max, whereas using it as the source text lowers it by≈−0\.6\\approx\-0\.6levels, down to roughly−1\.0\-1\.0\.

At the document level, we observe thatthe CEFR level shift is larger and consistently negative for documentswith meanΔ​ℓ\\Delta\\ellranging from−0\.16\-0\.16to−0\.31\-0\.31with confidence intervals excluding zero for all MT models\. Considering that these selected MT models differ in their training data, language coverage, and architectures, their similar level shifts in document\-level texts indicate that this is a systematic phenomenon across all models rather than an outlier related to a single model\. The three\-model comparison of CEFR level shifts in Table[5\.2](https://arxiv.org/html/2606.05421#S5.SS2)further confirms this observation, where all three CEFR classifiers produce downward shifts for all MT models, with a meanΔ​ℓ\\Delta\\ellof−0\.21\-0\.21,−0\.38\-0\.38, and−0\.28\-0\.28for XLM\-R, ModernBERT, and the Random Forest classifiers, respectively\.

![Refer to caption](https://arxiv.org/html/2606.05421v1/x7.png)Figure 6:Mean model\-anchored CEFR level shifts from back\-translations across MT models for document\-level texts\. Level shifts are larger at the document level but remain consistent in direction across MT models for each back\-translation\.Table 2:Mean model\-anchored CEFR level shift under three independently designed CEFR classifiers across MT models and granularities \(25% sample\), based onn=1500n=1500sentence\-level andn=270n=270document\-level instances\.### 5\.3Comparing Shifts with Finer\-Grained Complexity Features

To systematically investigate the correlation between source text complexity and translation quality, we design a small\-scale control experiment with sentence\-level texts evaluating four translation directions: EN→\\rightarrowFR, FR→\\rightarrowEN, EN→\\rightarrowRU, and RU→\\rightarrowEN\. We use 200 sentence pairs per model\-direction combination \(3,200 instances in total\), with translation quality measured via COMET\. We employspaCy555[https://spacy\.io](https://spacy.io/)to extract 37 source text linguistic complexity features, including length statistics, POS distribution, syntactic complexity, lexical richness, and argument structures, and compute Pearson correlation coefficients against COMET\.

Our linguistic feature analysis shows that translation directions involving Russian exhibit the strongest feature sensitivity\. EN→\\rightarrowRU and RU→\\rightarrowEN display markedly opposite correlation patterns across nearly all complexity features, indicating substantial directional asymmetry\. For instance, vocabulary\-related features such as content tokens and unique lemmas are negatively correlated with quality in the EN→\\rightarrowRU direction \(r≈−0\.20∗r\\approx\-0\.20^\{\*\}\), yet positively correlated in the RU→\\rightarrowEN direction \(r≈0\.19r\\approx 0\.19–0\.20∗0\.20^\{\*\}\)\. Word length features show the strongest associations in the RU→\\rightarrowEN direction \(up tor=0\.29∗r=0\.29^\{\*\}\), while POS ratio yields opposing significant correlations across FR→\\rightarrowEN \(r=−0\.25∗r=\-0\.25^\{\*\}\) and EN→\\rightarrowRU \(r=0\.19∗r=0\.19^\{\*\}\)\. These findings suggest that linguistic sensitivity is highly direction\-dependent and that source\-text complexity affects translation quality asymmetrically across language pairs\.

### 5\.4Translation Quality Does not Predict CEFR Shift

![Refer to caption](https://arxiv.org/html/2606.05421v1/x8.png)Figure 7:Scatterplot of MT quality via COMET and GEMBA versus model\-anchored[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level shifts for sentence and document\-level texts across[MT](https://arxiv.org/html/2606.05421#id3.3.id3)models\. The Spearman correlation is near zero in every panel \(\|ρ\|≤0\.12\|\\rho\|\\leq 0\.12\), indicating that translation quality doesnotpredict the magnitude or direction of its[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level shift\. This means that a high\-quality translation is just as likely to shift the CEFR level of a text as a lower\-quality translation\.We visualize per\-pair computed MT quality against the model\-anchored CEFR level shifts in Figure[7](https://arxiv.org/html/2606.05421#S5.F7)for COMET and GEMBA, and across sentence and document level texts\. We randomly sampled 100 and 30 data points for sentence\- \(55systems×\\times2020language pairs\) and document\-level \(5×65\\times 6language pairs\), respectively\.

In all four experiment runs, we obtain non\-significant near\-zero results for Spearman correlation \(ρ=\+0\.05\\rho=\+0\.05and−0\.01\-0\.01at the sentence level for COMET and GEMBA, and−0\.12\-0\.12and−0\.12\-0\.12at the document level\), indicating thattranslation quality and CEFR level shift are statistically independent\. From this, we conclude that the two subtasks underComplexityMT,ComplexityMT\-RobustnessandComplexityMT\-Preservation, therefore measure distinct yet complementary properties of MT behavior\. Since a high COMET or GEMBA does not lead to effective preservation of source CEFR levels, both a translation metric and a preservation metric may be needed for translation tasks, especially in MT applications sensitive to text complexity, such as educational content generation across multiple languages\. This result demonstrates the importance of theComplexityMTframework as a strong evaluation challenge for current and future MT models\.

## 6Discussion

#### Relationship between Machine Translation and Complexity

Prior works have formed two distinct viewpoints on the relation of translation and complexity: complex texts are complex to translateShardlow and Alva\-Manchego \([2022](https://arxiv.org/html/2606.05421#bib.bib41)\), and the translation process inherently simplifies textsCorpas Pastor et al\. \([2008](https://arxiv.org/html/2606.05421#bib.bib14)\); Wastl et al\. \([2025](https://arxiv.org/html/2606.05421#bib.bib47)\)\. Our empirical findings withComplexityMToffer support for both viewpoints\.ComplexityMT\-Robustnessresults show that translation quality indeed declines when the source text is of a higher CEFR level, whileComplexityMT\-Preservationresults provide evidence that the simplification universal phenomenon persists at the document level\. Beyond confirming both, we also showed that translation quality and complexity level shifts do not correlate with each other\. Therefore, our contribution to this discourse is not to resolve any disagreement—if such a disagreement exists—but to support both aspects by showing that they aredistinct dimensions of[MT](https://arxiv.org/html/2606.05421#id3.3.id3)behavior\.

#### Pedagogically Motivated Content Generation

Readability\-controlled simplification is studied as a method to adapt content to different levels of text complexity, while MT enables scaling the adapted content into multiple languages, which is one of the motivations for this research\. Our findings withComplexityMTshow that translation can shift CEFR levels differently across languages, and these resultscan inform content generation approaches by guiding when to simplify \(at source or at target\)\. For language pairs where translation tends to increase the complexity of the source text, our findings showthe need for technical pipeline improvements, such as adding a readability\-controlled simplification moduleat the target end, and vice versa\.

## 7Conclusion

We introducedComplexityMT, a new challenge for assessing how text complexity and machine translation interact, using the CEFR as the measure of text complexity\. Across six languages and five MT systems, our experiments showed that higher[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)levels make translation more difficult and that MT shifts the target text’s[CEFR](https://arxiv.org/html/2606.05421#id1.1.id1)level relative to the source in most languages\. These two effects are also statistically independent, where translation quality does not predict the magnitude of CEFR level shifts\. Together, these findings highlight the need to consider both translation quality and CEFR preservation when evaluating MT applications sensitive to text complexity, such multilingual educational content generation\.

## Limitations

We identify a few limitations to our work, in terms of how we operationalize text complexity and the data used, which we discuss below\.

#### Focus on CEFR

We use CEFRCouncil of Europe \([2001](https://arxiv.org/html/2606.05421#bib.bib15)\)as the reference pedagogical construct for text complexity in our study, as it is the most widely recognized language proficiency framework across the broader education and learning community\. We recognize that our findings may not directly translate to other, region\- or country\-specific pedagogical constructs such as the Common Core Standards \(CCS\) in the United States, or China’s Standards for English \(CSE\), to name a few\. However, the methodology we followed can be replicated with other constructs, if relevant data resources are available\.

#### Use of Automatic Classifiers for CEFR

We use three diverse, state\-of\-the\-art CEFR classifiers from\(Imperial et al\.,[2025](https://arxiv.org/html/2606.05421#bib.bib21)\), which were trained on the massively multilingual gold\-standardUniversalCEFRdataset\. While automated CEFR classifiers may have inherent prediction errors, their utility in our study remains appropriate and needed, considering that our main goal includes investigating how MT models can preserve CEFR levels of texts for educational content generation, for which these automatic CEFR classifiers will be a necessary resource in the process\.

#### Data and Linguistic Coverage

Our main results are anchored in the specific languages for which we obtained representative CEFR\-labeled reference texts fromUniversalCEFRImperial et al\. \([2025](https://arxiv.org/html/2606.05421#bib.bib21)\)\. This includes English, Dutch, and French at the document level, and English, French, Russian, Arabic, and Hindi at the sentence level\. We do not claim that our results will generalize to other languages, text formats \(e\.g\., phrase\-level\), or text types \(e\.g\., learner texts\) not tested in this work\.

#### Focus on Quantitative Analysis

Our results are primarily quantitative, given the straightforward goal of empirically investigating the effect of translation on text complexity and vice\-versa\. We acknowledge that a well\-constructed qualitative analysis would enrich and complement our work, but recognize that this can be conducted as a separate study and leave it for future work\.

## Acknowledgments

JMI is supported by the National University Philippines and the UKRI Centre for Doctoral Training in Accountable, Responsible, and Transparent AI \[EP/S023437/1\] of the University of Bath\.

## References

- Agrawal and Carpuat \(2019\)Sweta Agrawal and Marine Carpuat\. 2019\.[Controlling text complexity in neural machine translation](https://doi.org/10.18653/v1/D19-1166)\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\)*, pages 1549–1564, Hong Kong, China\. Association for Computational Linguistics\.
- Aluisio et al\. \(2010\)Sandra Aluisio, Lucia Specia, Caroline Gasperin, and Carolina Scarton\. 2010\.[Readability assessment for text simplification](https://aclanthology.org/W10-1001/)\.In*Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications*, pages 1–9, Los Angeles, California\. Association for Computational Linguistics\.
- Alva\-Manchego et al\. \(2020\)Fernando Alva\-Manchego, Carolina Scarton, and Lucia Specia\. 2020\.[Data\-driven sentence simplification: Survey and benchmark](https://doi.org/10.1162/coli_a_00370)\.*Computational Linguistics*, 46\(1\):135–187\.
- Alva\-Manchego and Shardlow \(2022\)Fernando Alva\-Manchego and Matthew Shardlow\. 2022\.[Towards readability\-controlled machine translation of COVID\-19 texts](https://aclanthology.org/2022.eamt-1.33/)\.In*Proceedings of the 23rd Annual Conference of the European Association for Machine Translation*, pages 287–288, Ghent, Belgium\. European Association for Machine Translation\.
- Alva\-Manchego et al\. \(2025\)Fernando Alva\-Manchego, Regina Stodden, Joseph Marvin Imperial, Abdullah Barayan, Kai North, and Harish Tayyar Madabushi\. 2025\.[Findings of the TSAR 2025 shared task on readability\-controlled text simplification](https://doi.org/10.18653/v1/2025.tsar-1.8)\.In*Proceedings of the Fourth Workshop on Text Simplification, Accessibility and Readability \(TSAR 2025\)*, pages 116–130, Suzhou, China\. Association for Computational Linguistics\.
- Alves et al\. \(2024\)Duarte Miguel Alves, José Pombal, Nuno M Guerreiro, Pedro Henrique Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G\. C\. de Souza, and Andre Martins\. 2024\.[Tower: An Open Multilingual Large Language Model for Translation\-Related Tasks](https://openreview.net/forum?id=EHPns3hVkj)\.In*First Conference on Language Modeling*\.
- Araghi and Palangkaraya \(2024\)Sahar Araghi and Alfons Palangkaraya\. 2024\.[The link between translation difficulty and the quality of machine translation: a literature review and empirical investigation](https://doi.org/10.1007/s10579-024-09735-x)\.*Language Resources and Evaluation*, 58\(4\):1093–1114\.
- Arase et al\. \(2022\)Yuki Arase, Satoru Uchida, and Tomoyuki Kajiwara\. 2022\.[CEFR\-based sentence difficulty annotation and assessment](https://doi.org/10.18653/v1/2022.emnlp-main.416)\.In*Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pages 6206–6219, Abu Dhabi, United Arab Emirates\. Association for Computational Linguistics\.
- Barayan et al\. \(2025\)Abdullah Barayan, Jose Camacho\-Collados, and Fernando Alva\-Manchego\. 2025\.[Analysing zero\-shot readability\-controlled sentence simplification](https://aclanthology.org/2025.coling-main.452/)\.In*Proceedings of the 31st International Conference on Computational Linguistics*, pages 6762–6781, Abu Dhabi, UAE\. Association for Computational Linguistics\.
- Breuker \(2022\)Mark Breuker\. 2022\.[CEFR Labelling and Assessment Services](https://library.oapen.org/bitstream/handle/20.500.12657/59316/1/978-3-031-17258-8.pdf#page=297)\.In*European Language Grid: A Language Technology Platform for Multilingual Europe*, pages 277–282\. Springer International Publishing Cham\.
- Bugliarello et al\. \(2020\)Emanuele Bugliarello, Sabrina J\. Mielke, Antonios Anastasopoulos, Ryan Cotterell, and Naoaki Okazaki\. 2020\.[It’s easier to translate out of English than into it: Measuring neural translation difficulty by cross\-mutual information](https://doi.org/10.18653/v1/2020.acl-main.149)\.In*Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 1640–1649, Online\. Association for Computational Linguistics\.
- Chandrasekar et al\. \(1996\)R\. Chandrasekar, Christine Doran, and B\. Srinivas\. 1996\.[Motivations and methods for text simplification](https://aclanthology.org/C96-2183/)\.In*COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics*\.
- Ciobanu et al\. \(2015\)Alina Maria Ciobanu, Liviu P\. Dinu, and Flaviu Pepelea\. 2015\.[Readability assessment of translated texts](https://aclanthology.org/R15-1014/)\.In*Proceedings of the International Conference Recent Advances in Natural Language Processing*, pages 97–103, Hissar, Bulgaria\. INCOMA Ltd\. Shoumen, BULGARIA\.
- Corpas Pastor et al\. \(2008\)Gloria Corpas Pastor, Ruslan Mitkov, Naveed Afzal, and Viktor Pekar\. 2008\.[Translation universals: do they exist? a corpus\-based NLP study of convergence and simplification](https://aclanthology.org/2008.amta-papers.5/)\.In*Proceedings of the 8th Conference of the Association for Machine Translation in the Americas: Research Papers*, pages 75–81, Waikiki, USA\. Association for Machine Translation in the Americas\.
- Council of Europe \(2001\)Council of Europe\. 2001\.[*Common European Framework of Reference for Languages: Learning, Teaching, Assessment*](https://rm.coe.int/1680459f97)\.Cambridge University Press\.
- Crossley et al\. \(2017\)Scott A\. Crossley, Stephen Skalicky, Mihai Dascalu, Danielle S\. McNamara, and Kristopher Kyle\. 2017\.[Predicting Text Comprehension, Processing, and Familiarity in Adult Readers: New Approaches to Readability Formulas](https://doi.org/10.1080/0163853x.2017.1296264)\.*Discourse Processes*, 54\(5\-6\):340–359\.
- Deutsch et al\. \(2020\)Tovly Deutsch, Masoud Jasbi, and Stuart Shieber\. 2020\.[Linguistic features for readability assessment](https://doi.org/10.18653/v1/2020.bea-1.1)\.In*Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications*, pages 1–17, Seattle, WA, USA → Online\. Association for Computational Linguistics\.
- DuBay \(2004\)William H\. DuBay\. 2004\.[*The Principles of Readability*](https://files.eric.ed.gov/fulltext/ED490073.pdf)\.Impact Information\.
- Google Translate Research Team et al\. \(2026\)Google Translate Research Team, Mara Finkelstein, Isaac Caswell, Tobias Domhan, Jan\-Thorsten Peter, Juraj Juraska, Parker Riley, Daniel Deutsch, Cole Dilanni, Colin Cherry, Eleftheria Briakou, Elizabeth Nielsen, Jiaming Luo, Sweta Agrawal, Wenda Xu, Erin Kats, Stephane Jaskiewicz, Markus Freitag, and David Vilar\. 2026\.[TranslateGemma Technical Report](https://doi.org/10.48550/arXiv.2601.09012)\.*Computing Research Repository*, arXiv:2601\.09012\.
- Hale and Campbell \(2002\)Sandra Hale and Stuart Campbell\. 2002\.[The interaction between text difficulty and translation accuracy](https://doi.org/10.1075/babel.48.1.02hal)\.*Babel*, 48\(1\):14–33\.
- Imperial et al\. \(2025\)Joseph Marvin Imperial, Abdullah Barayan, Regina Stodden, Rodrigo Wilkens, Ricardo Muñoz Sánchez, Lingyun Gao, Melissa Torgbi, Dawn Knight, Gail Forey, Reka R\. Jablonkai, Ekaterina Kochmar, Robert Joshua Reynolds, Eugénio Ribeiro, Horacio Saggion, Elena Volodina, Sowmya Vajjala, Thomas François, Fernando Alva\-Manchego, and Harish Tayyar Madabushi\. 2025\.[UniversalCEFR: Enabling open multilingual research on language proficiency assessment](https://doi.org/10.18653/v1/2025.emnlp-main.491)\.In*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages 9703–9755, Suzhou, China\. Association for Computational Linguistics\.
- Kincaid et al\. \(1975\)J\. Peter Kincaid, Robert P\. Fishburne Jr, Richard L\. Rogers, and Brad S\. Chissom\. 1975\.[Derivation of New Readability Formulas \(Automated Readability Index, Fog Count and Flesch Reading Ease Formula\) for Navy Enlisted Personnel](https://doi.org/10.21236/ada006655)\.Technical report, Institute for Simulation and Training, University of Central Florida\.
- Kocmi and Federmann \(2023\)Tom Kocmi and Christian Federmann\. 2023\.[Large language models are state\-of\-the\-art evaluators of translation quality](https://aclanthology.org/2023.eamt-1.19/)\.In*Proceedings of the 24th Annual Conference of the European Association for Machine Translation*, pages 193–203, Tampere, Finland\. European Association for Machine Translation\.
- Lennon and Burdick \(2004\)Colleen Lennon and Hal Burdick\. 2004\.[The lexile framework as an approach for reading measurement and success](https://cdn.lexile.com/m/resources/materials/Lennon__Burdick_2004.pdf)\.*Electronic Publication on www\.lexile\.com*\.
- Li et al\. \(2014\)Junyi Jessy Li, Marine Carpuat, and Ani Nenkova\. 2014\.[Assessing the discourse factors that influence the quality of machine translation](https://doi.org/10.3115/v1/P14-2047)\.In*Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 283–288, Baltimore, Maryland\. Association for Computational Linguistics\.
- Lu et al\. \(2021\)Xinyu Lu, Jipeng Qiang, Yun Li, Yunhao Yuan, and Yi Zhu\. 2021\.[An unsupervised method for building sentence simplification corpora in multiple languages](https://doi.org/10.18653/v1/2021.findings-emnlp.22)\.In*Findings of the Association for Computational Linguistics: EMNLP 2021*, pages 227–237, Punta Cana, Dominican Republic\. Association for Computational Linguistics\.
- Maddela et al\. \(2021\)Mounica Maddela, Fernando Alva\-Manchego, and Wei Xu\. 2021\.[Controllable text simplification with explicit paraphrasing](https://doi.org/10.18653/v1/2021.naacl-main.277)\.In*Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 3536–3553, Online\. Association for Computational Linguistics\.
- Maddela and Xu \(2018\)Mounica Maddela and Wei Xu\. 2018\.[A word\-complexity lexicon and a neural readability ranking model for lexical simplification](https://doi.org/10.18653/v1/D18-1410)\.In*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 3749–3760, Brussels, Belgium\. Association for Computational Linguistics\.
- Marchisio et al\. \(2019\)Kelly Marchisio, Jialiang Guo, Cheng\-I Lai, and Philipp Koehn\. 2019\.[Controlling the reading level of machine translation output](https://aclanthology.org/W19-6619/)\.In*Proceedings of Machine Translation Summit XVII: Research Track*, pages 193–203, Dublin, Ireland\. European Association for Machine Translation\.
- Mehta et al\. \(2020\)Sneha Mehta, Bahareh Azarnoush, Boris Chen, Avneesh Saluja, Vinith Misra, Ballav Bihani, and Ritwik Kumar\. 2020\.[Simplify\-then\-translate: Automatic preprocessing for black\-box translation](https://doi.org/10.1609/aaai.v34i05.6369)\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 34, pages 8488–8495\.
- Mirkin and Meunier \(2015\)Shachar Mirkin and Jean\-Luc Meunier\. 2015\.[Personalized machine translation: Predicting translational preferences](https://doi.org/10.18653/v1/D15-1238)\.In*Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing*, pages 2019–2025, Lisbon, Portugal\. Association for Computational Linguistics\.
- Mishra et al\. \(2013\)Abhijit Mishra, Pushpak Bhattacharyya, and Michael Carl\. 2013\.[Automatically predicting sentence translation difficulty](https://aclanthology.org/P13-2062/)\.In*Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 346–351, Sofia, Bulgaria\. Association for Computational Linguistics\.
- Nadejde et al\. \(2022\)Maria Nadejde, Anna Currey, Benjamin Hsu, Xing Niu, Marcello Federico, and Georgiana Dinu\. 2022\.[CoCoA\-MT: A dataset and benchmark for contrastive controlled MT with application to formality](https://doi.org/10.18653/v1/2022.findings-naacl.47)\.In*Findings of the Association for Computational Linguistics: NAACL 2022*, pages 616–632, Seattle, United States\. Association for Computational Linguistics\.
- Naous et al\. \(2024\)Tarek Naous, Michael J Ryan, Anton Lavrouk, Mohit Chandra, and Wei Xu\. 2024\.[ReadMe\+\+: Benchmarking multilingual language models for multi\-domain readability assessment](https://doi.org/10.18653/v1/2024.emnlp-main.682)\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 12230–12266, Miami, Florida, USA\. Association for Computational Linguistics\.
- Nishihara et al\. \(2019\)Daiki Nishihara, Tomoyuki Kajiwara, and Yuki Arase\. 2019\.[Controllable text simplification with lexical constraint loss](https://doi.org/10.18653/v1/P19-2036)\.In*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop*, pages 260–266, Florence, Italy\. Association for Computational Linguistics\.
- OpenAI \(2025\)OpenAI\. 2025\.[GPT\-5 System Card](https://doi.org/10.48550/arXiv.2601.03267)\.*Computing Research Repository*, arXiv:2601\.03267\.
- Proietti et al\. \(2025\)Lorenzo Proietti, Stefano Perrella, Vilém Zouhar, Roberto Navigli, and Tom Kocmi\. 2025\.[Estimating machine translation difficulty](https://doi.org/10.18653/v1/2025.findings-emnlp.1317)\.In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 24261–24285, Suzhou, China\. Association for Computational Linguistics\.
- Rei et al\. \(2020\)Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie\. 2020\.[COMET: A neural framework for MT evaluation](https://doi.org/10.18653/v1/2020.emnlp-main.213)\.In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, pages 2685–2702, Online\. Association for Computational Linguistics\.
- Scarton and Specia \(2018\)Carolina Scarton and Lucia Specia\. 2018\.[Learning simplifications for specific target audiences](https://doi.org/10.18653/v1/P18-2113)\.In*Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pages 712–718, Melbourne, Australia\. Association for Computational Linguistics\.
- Sennrich et al\. \(2016\)Rico Sennrich, Barry Haddow, and Alexandra Birch\. 2016\.[Controlling politeness in neural machine translation via side constraints](https://doi.org/10.18653/v1/N16-1005)\.In*Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 35–40, San Diego, California\. Association for Computational Linguistics\.
- Shardlow and Alva\-Manchego \(2022\)Matthew Shardlow and Fernando Alva\-Manchego\. 2022\.[Simple TICO\-19: A dataset for joint translation and simplification of COVID\-19 texts](https://aclanthology.org/2022.lrec-1.331/)\.In*Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 3093–3102, Marseille, France\. European Language Resources Association\.
- Sheang and Saggion \(2021\)Kim Cheng Sheang and Horacio Saggion\. 2021\.[Controllable sentence simplification with a unified text\-to\-text transfer transformer](https://doi.org/10.18653/v1/2021.inlg-1.38)\.In*Proceedings of the 14th International Conference on Natural Language Generation*, pages 341–352, Aberdeen, Scotland, UK\. Association for Computational Linguistics\.
- Tani et al\. \(2022\)Kazuki Tani, Ryoya Yuasa, Kazuki Takikawa, Akihiro Tamura, Tomoyuki Kajiwara, Takashi Ninomiya, and Tsuneo Kato\. 2022\.[A benchmark dataset for multi\-level complexity\-controllable machine translation](https://aclanthology.org/2022.lrec-1.726/)\.In*Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 6744–6752, Marseille, France\. European Language Resources Association\.
- Vajjala \(2022\)Sowmya Vajjala\. 2022\.[Trends, limitations and open challenges in automatic readability assessment research](https://aclanthology.org/2022.lrec-1.574/)\.In*Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pages 5366–5377, Marseille, France\. European Language Resources Association\.
- Vásquez\-Rodríguez et al\. \(2022\)Laura Vásquez\-Rodríguez, Pedro\-Manuel Cuenca\-Jiménez, Sergio Morales\-Esquivel, and Fernando Alva\-Manchego\. 2022\.[A benchmark for neural readability assessment of texts in Spanish](https://doi.org/10.18653/v1/2022.tsar-1.18)\.In*Proceedings of the Workshop on Text Simplification, Accessibility, and Readability \(TSAR\-2022\)*, pages 188–198, Abu Dhabi, United Arab Emirates \(Virtual\)\. Association for Computational Linguistics\.
- Warner et al\. \(2025\)Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli\. 2025\.[Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference](https://doi.org/10.18653/v1/2025.acl-long.127)\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 2526–2547, Vienna, Austria\. Association for Computational Linguistics\.
- Wastl et al\. \(2025\)Michelle Wastl, Jannis Vamvas, and Rico Sennrich\. 2025\.[Machine translation models are zero\-shot detectors of translation direction](https://doi.org/10.18653/v1/2025.findings-acl.59)\.In*Findings of the Association for Computational Linguistics: ACL 2025*, pages 1054–1074, Vienna, Austria\. Association for Computational Linguistics\.
- Xia et al\. \(2016\)Menglin Xia, Ekaterina Kochmar, and Ted Briscoe\. 2016\.[Text readability assessment for second language learners](https://doi.org/10.18653/v1/W16-0502)\.In*Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications*, pages 12–22, San Diego, CA\. Association for Computational Linguistics\.
- Xu et al\. \(2016\)Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison\-Burch\. 2016\.[Optimizing statistical machine translation for text simplification](https://doi.org/10.1162/tacl_a_00107)\.*Transactions of the Association for Computational Linguistics*, 4:401–415\.
- Zouhar et al\. \(2026\)Vilém Zouhar, Wenda Xu, Parker Riley, Juraj Juraska, Mara Finkelstein, Markus Freitag, and Daniel Deutsch\. 2026\.[Generating difficult\-to\-translate texts](https://doi.org/10.18653/v1/2026.mme-main.14)\.In*Proceedings of the First Workshop on Multilingual Multicultural Evaluation*, pages 204–219, Rabat, Morocco\. Association for Computational Linguistics\.

## Appendix AAppendix

### A\.1Libraries, Hyperparameters, and Configurations

We provide the full table of the Python libraries used and their corresponding versions in our experiments in Table[3](https://arxiv.org/html/2606.05421#A1.T3)\. Likewise, we also provide the configurations and hyperparameter values used for the three CEFR classifiers \(XLM\-R,ModernBERT, andRandom Forest\) in Table[4](https://arxiv.org/html/2606.05421#A1.T4)across all CEFR level predicitons in this study\.

Table 3:The Python libraries used in our experiments with with a main Python version of 3\.14\.4\.Table 4:Hyperparameter and model configurations for the three CEFR classifiers \(XLM\-R,ModernBERT, andRandom Forest\) explored in the study\.Table 5:Configurations for the reference\-free translation quality metrics COMET and GEMBA used in the study\.
### A\.2Utility Prompts

We provide the prompts we used for the LLM\-based MT models we used in this study, including GPT\-5\.4, TowerInstruct\-7B, TranslateGemma and for the LLM\-based translation quality metric GEMBA\.

GPT\-5\.4 Translation PromptTranslate the following \{src\_name\} sentence to \{tgt\_name\}\. Output only the translation, nothing else\. \{text\}

TowerInstruct\-7B Translation PromptTranslate the following text from \{src\_name\} into \{tgt\_name\}\. \{src\_name\}: \{text\} \{tgt\_name\}:

TranslateGemma Translation Prompt\{"role": "user", "content": \[\{ "type": "text", "source\_lang\_code": "\{src\}", "target\_lang\_code": "\{tgt\}", "text": "\{text\}" \}\]\}

GEMBA Quality Scoring PromptScore the following translation from \{src\_name\} to \{tgt\_name\} on a continuous scale from 0 to 100, where a score of zero means ‘‘no meaning preserved’’ and a score of one hundred means ‘‘perfect meaning and grammar’’\. \{src\_name\} source: ‘‘\{src\_text\}’’ \{tgt\_name\} translation: ‘‘\{tgt\_text\}’’ Score \(0\-\-100\):

### A\.3CEFR Classifier Reliability

To investigate the robustness of the automatic CEFR classifiers we used in this work, we conduct a pairwise reliability test by computing Cohen’sκ\\kappawith quadratic weights and exact\-match rate on a 25% random sample across the sentence and document\-level texts across languages\. We report the results in Table[6](https://arxiv.org/html/2606.05421#A1.T6)\. Results show that all three structurally diverse CEFR classifier models exhibit moderate to high agreement \(κquad\\kappa\_\{\\text\{quad\}\}\), with XLM\-R and ModernBERT showing the greatest agreement\. All three CEFR classifiers also achieve≈90\.0\+\\approx 90\.0\+in adjacent accuray \(±1\\pm 1\) and≈50\+\\approx 50\+in exact accuracy\.

Table 6:Pairwise CEFR\-label agreement between XLM\-R, ModernBERT, and the feature\-based Random Forest model on round\-trip text classification \(forward and back\-translations\) across five MT models \(25% sample\)\. Cohen’sκ\\kappawith quadratic weights and exact\-match rate\.
### A\.4Source\-Anchored Level Shift

In Section[3\.2](https://arxiv.org/html/2606.05421#S3.SS2)withComlexityMT\-Preservation, we discuss that we use a model\-anchored shift that uses the forward and back\-translationΔ​ℓmodel=ℓ^back−ℓ^fwd\\Delta\\ell\_\{\\text\{model\}\}=\\hat\{\\ell\}\_\{\\text\{back\}\}\-\\hat\{\\ell\}\_\{\\text\{fwd\}\}\. This represents the core translation process to quantify the CEFR level shift from the source and target languages\. To investigate a within\-language comparison from the back\-translation process, we report a formula which is a source\-anchored shift

Δ​ℓsource=ℓ^back−ℓ^orig,\\Delta\\ell\_\{\\text\{source\}\}=\\hat\{\\ell\}\_\{\\text\{back\}\}\-\\hat\{\\ell\}\_\{\\text\{orig\}\},\(2\)whereℓ^orig=f​\(x\)\\hat\{\\ell\}\_\{\\text\{orig\}\}=f\(x\)is the CEFR classifier’s prediction on the source text\. This allows measuring the CEFR level shift from the same language of the source and back\-translation texts\. We produce the same heatmap visualization for theΔ​ℓsource\\Delta\\ell\_\{\\text\{source\}\}and report the results in Figure[8](https://arxiv.org/html/2606.05421#A1.F8)for sentence level and Figure[9](https://arxiv.org/html/2606.05421#A1.F9)for document level\.

![Refer to caption](https://arxiv.org/html/2606.05421v1/x9.png)Figure 8:Mean model\-anchored CEFR level shifts from back\-translations across MT models for sentence\-level texts\. We anchor theUniversalCEFRclassifier’s source text as anchor \(vs\. the forward translation shown in Figure[5](https://arxiv.org/html/2606.05421#S4.F5)\) to investigate the effects of within\-language differences\.![Refer to caption](https://arxiv.org/html/2606.05421v1/x10.png)Figure 9:Mean model\-anchored CEFR level shifts from back\-translations across MT models for document\-level texts using the source text as the anchor\. Level shifts at the document level remain consistent with the sentence level of the same anchor\.At the sentence level, we observe that this within\-language shift is relatively small\. For English source texts, in particular, receive lower CEFR levels across all MT models with around−0\.21\-0\.21to−0\.41\-0\.41shifts, while French, Russian, and Hindi source texts obtain slightly higher CEFR levels compared to their original for most MT models\. We do note that this within\-language comparison is observable with Russian where the high sentence\-level CEFR level shifts as seen in Figure[5](https://arxiv.org/html/2606.05421#S4.F5)using forward translation as anchor is not observed in Figure[8](https://arxiv.org/html/2606.05421#A1.F8)using the source text with CEFR predicted level as anchor\.

At the document level, we observe a sharper pattern where Tower\-7B is the only outlier model that shows consistent within\-language decrease in CEFR level shifts across all backtranslations, with the highest being English to French at−0\.63\-0\.63\. Meanwhile, the other MT models more or less retain the source CEFR level with very minor deviations\. Again, we see the distinctiveness here in using the within\-language comparison, where it can cancel the classifier’s per\-language offset, and the backtranslation is able to return to the source text’s predicted CEFR level\.

### A\.5Use of LLMs

In producing this work, we used Grammarly for minor grammar and spelling corrections, ChatGPT and Claude Code for assistance with LaTeX table and figure formatting, troubleshooting code, and issues with Matplotlib visualizations\. All suggestions from these tools were scrutinized by the authors before integration into the paper\.

![Refer to caption](https://arxiv.org/html/2606.05421v1/x11.png)Figure 10:Pearson correlation coefficients between source\-text linguistic features and COMET scores across four translation directions\.