Multilingual Sentence Embeddings for Linguistic-Integrated Reliability Audit

# Multilingual Sentence Embeddings for Linguistic-Integrated Reliability Audit
Source: [https://arxiv.org/html/2607.17466](https://arxiv.org/html/2607.17466)
Ummugul Bezirhan, Ji Yoon Jung, Matthias von Davier Boston College Chestnut Hill, Massachusetts, USA \{bezirhan, jungjg, vondavim\}@bc\.edu

###### Abstract

Multilingual assessment systems commonly rely on translation for scoring and quality\-control processes\. We evaluate whether multilingual sentence embeddings can replace translated English input for Linguistic\-Integrated Reliability Auditing \(LiRA\) across 11 PIRLS constructed\-response items and three embedding models\. Native\-language embeddings reproduced translation\-based reliability estimates closely while recovering responses excluded after translation failure, with no meaningful change in reliability\.

Multilingual Sentence Embeddings for Linguistic\-Integrated Reliability Audit

Ummugul Bezirhan, Ji Yoon Jung, Matthias von DavierBoston CollegeChestnut Hill, Massachusetts, USA\{bezirhan, jungjg, vondavim\}@bc\.edu

## 1Introduction

International large\-scale assessments \(ILSAs\) such as PIRLS and TIMSS increasingly explore and implement automated approaches for scoring and quality control of constructed\-response \(CR\) items \(e\.g\.,Junget al\.,[2024](https://arxiv.org/html/2607.17466#bib.bib9); Tyacket al\.,[2024](https://arxiv.org/html/2607.17466#bib.bib17)\)\. Because these assessments operate across hundreds of languages simultaneously, multilingual processing remains a central methodological challenge\. Automated scoring systems must therefore support reliable semantic processing across linguistically diverse response sets while maintaining comparability across countries and languages\. A common strategy for multilingual automated scoring is the translate\-then\-score paradigm, in which student responses are translated into English prior to scoring\(Horbachet al\.,[2024](https://arxiv.org/html/2607.17466#bib.bib8); Junget al\.,[2024](https://arxiv.org/html/2607.17466#bib.bib9)\)\. This approach simplifies deployment by allowing a single English\-centric scoring system to be applied across multiple languages and countries\. More recent large language model \(LLM\)\-based scoring frameworks have further integrated translation directly into prompting workflows, enabling multilingual responses to be processed within a unified scoring framework\(Junget al\.,[2025b](https://arxiv.org/html/2607.17466#bib.bib11)\)\. Although these approaches have substantially improved scalability and multilingual coverage, they continue to rely on translation as an intermediate representation\. Even when scoring is performed directly on responses written in their native language\(Junget al\.,[2026](https://arxiv.org/html/2607.17466#bib.bib12)\), translation often remains a critical component retained for downstream processes such as quality\-control procedures, consistency checks, reliability analyses, and human review\. While LLM\-based context\-aware translation better preserves the original meaning and nuances than traditional neural machine translation\(Junget al\.,[2025b](https://arxiv.org/html/2607.17466#bib.bib11)\), any distortions introduced during translation may propagate beyond scoring into subsequent measurement and validation activities\.

Translation, however, is not necessarily a neutral preprocessing step\. Prior research in cross\-lingual representation learning has shown that semantic spaces are not fully isomorphic across languages, particularly for structurally distant or low\-resource languages\(Søgaardet al\.,[2018](https://arxiv.org/html/2607.17466#bib.bib15); Beinborn and Choenni,[2020](https://arxiv.org/html/2607.17466#bib.bib1)\)\. Even when translations appear semantically accurate at the surface level, translation may alter local neighborhood structure, pairwise similarity relationships, or embedding geometry\. Such distortions may be especially important for downstream tasks that rely directly on semantic similarity relationships among responses\(Bezirhanet al\.,[2026](https://arxiv.org/html/2607.17466#bib.bib2)\)\.

This issue is particularly relevant for LiRA \(Linguistic\-integrated Reliability Audit;Junget al\.,[2025a](https://arxiv.org/html/2607.17466#bib.bib10)\), a recently proposed reliability auditing framework for multilingual scoring systems\. LiRA estimates reliability using semantic similarity relationships among student responses\. For each response, the framework retrieves the most semantically similar neighboring responses in the embedding space and constructs a similarity\-weighted benchmark score using the neighboring responses’ labels\. Reliability is then estimated by assessing the agreement between the original scores and the neighborhood\-derived benchmark scores\. Because LiRA operates entirely in embedding space, translation\-induced changes in semantic neighborhoods may affect neighbor retrieval, benchmark construction, and ultimately, reliability estimates\.

Recent work has demonstrated that translation can introduce measurable semantic drift in multilingual embedding spaces\. Translation\-induced shifts have been observed in pairwise similarity, local neighborhood preservation, and global geometric alignment across multilingual sentence embedding models\(Artetxe and Schwenk,[2019](https://arxiv.org/html/2607.17466#bib.bib14); Conneau and Lample,[2019](https://arxiv.org/html/2607.17466#bib.bib4); Kornblithet al\.,[2019](https://arxiv.org/html/2607.17466#bib.bib13)\)\. However, it remains unclear whether these geometric changes meaningfully affect downstream reliability auditing tasks such as LiRA\. Importantly, advances in multilingual embedding models now allow semantic representations to be computed directly from native\-language responses without requiring translation into English\.

The present study investigates whether translation is necessary for LiRA\. Specifically, we compare an original translation\-based LiRA pipeline against a multilingual pipeline that embeds native\-language responses directly using multilingual sentence embeddings\. Using 11 PIRLS CR items across multiple languages, we evaluate whether multilingual LiRA can reproduce reliability estimates from translated English responses while reducing reliance on translation preprocessing and recovering responses excluded due to missing translations\.

## 2Conceptual Framework

Translation is not merely a technical preprocessing step; it can alter the linguistic and semantic characteristics of student responses\. Machine translation and LLM\-based translation systems are designed to produce fluent target\-language output, which may normalize features commonly observed in student writing, including fragmentary expressions, nonstandard spelling, and code\-switching\. In some cases, translation may also alter or omit information contained in the original response, particularly when responses are short, ambiguous, or context\-dependent\(Beinborn and Choenni,[2020](https://arxiv.org/html/2607.17466#bib.bib1); Somerset al\.,[2006](https://arxiv.org/html/2607.17466#bib.bib16)\)\. Such changes are especially relevant for short CR items, where a single idea or phrase may determine the assigned score\. Even minor shifts in meaning may therefore influence both automated scoring outcomes and downstream reliability analyses\.

These challenges are not distributed uniformly across languages\. Translation quality is generally lower for low\-resource and morphologically complex languages, where limited training data, rich inflectional systems, and greater orthographic variation increase the likelihood of translation errors\(Beinborn and Choenni,[2020](https://arxiv.org/html/2607.17466#bib.bib1)\)\. In operational settings, translation may also be bypassed or yield incomplete output, necessitating the exclusion of affected responses from subsequent analyses\. As a result, translation\-based pipelines may disproportionately affect the very languages and response types that are already most difficult to model reliably\.

Translation\-induced semantic drift does not necessarily arise from outright mistranslation\. Even when a translation is linguistically accurate, the translated expression may not carry exactly the same semantic associations as the original\. Research in cross\-lingual semantics has shown that concepts often differ in semantic scope across languages\(Conneauet al\.,[2018](https://arxiv.org/html/2607.17466#bib.bib3)\), and that projecting languages into a shared representation space can either sharpen fine\-grained distinctions through cross\-lingual sense disambiguation\(Faruqui and Dyer,[2014](https://arxiv.org/html/2607.17466#bib.bib6)\)or blur them through phenomena such as hubness, where a small number of representations appear disproportionately often among the nearest neighbors of others\(Dinuet al\.,[2014](https://arxiv.org/html/2607.17466#bib.bib5)\)\. Consequently, responses that appear equivalent to human readers may nevertheless occupy different locations within a semantic space and exhibit different neighborhood relationships\. For retrieval\-based methods that rely on semantic similarity, these local changes may be more consequential than shifts in global geometric structure\. For embedding\-based methods, these differences matter because they affect not only the representation of individual responses but also their relationships to surrounding responses\(Bezirhanet al\.,[2026](https://arxiv.org/html/2607.17466#bib.bib2)\)\. A translated response may remain broadly similar to its original version while exhibiting different nearest\-neighbor relationships, altered local density, or increased hubness\. Such changes are particularly relevant for retrieval\-based methods, which depend on local neighborhood structure rather than global geometric alignment\.

LiRA estimates reliability by leveraging semantic relationships among responses in an embedding space rather than relying on additional human scoring\(Junget al\.,[2025a](https://arxiv.org/html/2607.17466#bib.bib10)\)\. For each response, LiRA retrieves the most semantically similar neighboring responses, constructs a similarity\-weighted benchmark score from their labels, and estimates reliability through weighted exact agreement between the original score and this neighborhood\-derived benchmark\. Because LiRA operates entirely on semantic neighborhoods, its validity depends directly on the quality of the underlying representation space\. In translation\-based implementations, this space is constructed from English translations of multilingual responses\. Consequently, translation\-induced changes in local neighborhood structure may influence neighbor retrieval, benchmark construction, and ultimately reliability estimates\.

Recent multilingual sentence embedding models provide an alternative by representing responses directly in their native language within a shared multilingual semantic space\. If multilingual representations preserve the local neighborhood structure required for benchmark construction, translation may be unnecessary for semantic\-similarity\-based reliability auditing\. The present study evaluates this possibility by comparing translation\-based and multilingual LiRA pipelines under otherwise identical conditions\.

## 3Methods

### 3\.1Data and Design

We use PIRLS 2021 reading\-comprehension responses to 11 dichotomously scored CR items drawn from three reading passages\. The number of validly scored responses ranges from 12,729 to 15,794 per item, spanning the 29 languages of the participating PIRLS 2021 countries\.

The objective of the study was to isolate the effect of translation on LiRA reliability estimates\. To achieve this, the LiRA procedure and embedding model were held constant, with only the textual representation provided to the encoder varied\. Two processing conditions were evaluated\. In the translation\-based condition, the responses were translated into English using GPT\-4\.1 with a context\-rich translation framework prior to embedding\. This corresponds to the original LiRA implementation\(Junget al\.,[2025a](https://arxiv.org/html/2607.17466#bib.bib10)\)\. For the proposed translation\-free condition, the responses were embedded directly in their original language without a translation step\.

In both conditions, the same embedding model, LiRA parameters, and reliability estimation procedure were used\. Consequently, any differences in reliability estimates can be attributed to the use of translated versus native\-language representations rather than differences in model architecture or scoring methodology\.

### 3\.2The LiRA Procedure

We summarize LiRA formally to make explicit where translation can enter the reliability\-estimation process\. Consider a set ofNNscored responses to a single item\. Each responseiihas an observed human scorehi∈Sh\_\{i\}\\in S, whereSSis the item’s score set\. In the present study, all items are dichotomously scored, soS=\{0,1\}S=\\\{0,1\\\}\.

Letxix\_\{i\}denote the text representation of responseii\. In the translation\-based condition,xix\_\{i\}is the English translation of the response; in the translation\-free condition,xix\_\{i\}is the original native\-language response\. An encoderffmaps the response text to an embedding,

𝐞i=f​\(xi\)∈ℝd,\\mathbf\{e\}\_\{i\}=f\(x\_\{i\}\)\\in\\mathbb\{R\}^\{d\},\(1\)and similarity between two responses is computed using cosine similarity,

sim⁡\(i,j\)=𝐞i⋅𝐞j∥𝐞i∥​∥𝐞j∥\.\\operatorname\{sim\}\(i,j\)=\\frac\{\\mathbf\{e\}\_\{i\}\\cdot\\mathbf\{e\}\_\{j\}\}\{\\lVert\\mathbf\{e\}\_\{i\}\\rVert\\,\\lVert\\mathbf\{e\}\_\{j\}\\rVert\}\.\(2\)
For each responseii, LiRA retrieves the setNk​\(i\)N\_\{k\}\(i\)containing thekkmost similar responses toii, excluding the response itself; herek=3k=3, following the original LiRA implementation\. Within this neighborhood, the total similarity weight supporting each candidate scores∈Ss\\in Sis

wi​\(s\)=∑j∈Nk​\(i\)hj=ssim⁡\(i,j\),w\_\{i\}\(s\)=\\sum\_\{\\begin\{subarray\}\{c\}j\\in N\_\{k\}\(i\)\\\\ h\_\{j\}=s\\end\{subarray\}\}\\operatorname\{sim\}\(i,j\),\(3\)and the similarity\-weighted majority score is

si∗=arg⁡maxs∈S⁡wi​\(s\)\.s\_\{i\}^\{\*\}=\\arg\\max\_\{s\\in S\}w\_\{i\}\(s\)\.\(4\)
LiRA assigns this benchmark score only when the winning score receives a sufficient share of the total neighborhood weight\. Let

pi=wi​\(si∗\)∑j∈Nk​\(i\)sim⁡\(i,j\)\.p\_\{i\}=\\frac\{w\_\{i\}\(s\_\{i\}^\{\*\}\)\}\{\\sum\_\{j\\in N\_\{k\}\(i\)\}\\operatorname\{sim\}\(i,j\)\}\.\(5\)
Ifpip\_\{i\}exceeds the thresholdτ\\tau, the benchmark scoresi∗s\_\{i\}^\{\*\}is assigned; otherwise, responseiiis flagged as inconsistent\. Following the original implementation, we useτ=0\.60\\tau=0\.60\. Responses flagged as missing or untranslated are excluded from the translation\-based reliability calculation, and responses flagged as meaningless are assigned the lowest score category\. For each response assigned a numeric benchmark score, define the average cosine similarity of its retrieved neighbors,

c¯i=1k​∑j∈Nk​\(i\)sim⁡\(i,j\)\.\\bar\{c\}\_\{i\}=\\frac\{1\}\{k\}\\sum\_\{j\\in N\_\{k\}\(i\)\}\\operatorname\{sim\}\(i,j\)\.\(6\)
LetVVdenote the set of valid responses with both an observed human score and an assigned numeric LiRA benchmark\. Weighted Exact Agreement \(WEA\) is then

WEA=∑i∈Vc¯i​𝕀​\(hi=si∗\)∑i∈Vc¯i\.\\operatorname\{WEA\}=\\frac\{\\sum\_\{i\\in V\}\\bar\{c\}\_\{i\}\\,\\mathbb\{I\}\(h\_\{i\}=s\_\{i\}^\{\*\}\)\}\{\\sum\_\{i\\in V\}\\bar\{c\}\_\{i\}\}\.\(7\)
Responses with higher average neighborhood similarity therefore contribute more heavily to the reliability estimate\. This formalization makes clear where translation can affect LiRA\. All downstream quantities are functions of the response embedding\.

### 3\.3Multilingual Embedding Models

To assess the robustness of the proposed approach across embedding architectures, we evaluated three multilingual sentence encoders representing different modeling objectives: LaBSE\(Fenget al\.,[2022](https://arxiv.org/html/2607.17466#bib.bib7)\), which was designed for cross\-lingual alignment; Multilingual\-E5\(Wanget al\.,[2024](https://arxiv.org/html/2607.17466#bib.bib18)\), which was optimized for multilingual retrieval tasks; and Qwen3\-Embedding\(Zhanget al\.,[2025](https://arxiv.org/html/2607.17466#bib.bib19)\), a general\-purpose multilingual embedding model\. For each encoder, we constructed both translation\-based and translation\-free LiRA pipelines using identical retrieval, voting, and reliability\-estimation parameters\. This design isolates the effect of replacing translated responses with native\-language responses while holding the underlying encoder and all other components of the LiRA framework constant\.

### 3\.4Evaluation

Because GPT\-4\.1 translation occasionally fails or produces untranslated output, the translation\-based and translation\-free pipelines do not always operate on identical response sets\. To separate translation effects from sample\-composition effects, we conducted both matched\-sample and full\-sample analyses\. In addition, we recorded the number of rescued responses, defined as responses excluded because of translation failure but retained in the translation\-free pipeline\. To assess the stability of LiRA outputs across conditions, we also computed prediction agreement, defined as the proportion of responses that receive the same benchmark score under both pipelines, and inconsistency\-flag agreement, defined as the proportion of responses that receive the same inconsistency classification\.

## 4Results

Table[1](https://arxiv.org/html/2607.17466#S4.T1)summarizes LiRA reliability estimates obtained from the translation\-based and translation\-free pipelines across the three multilingual embedding models\. Across all encoders, reliability estimates were highly similar between conditions\. Mean WEA differences ranged from−0\.0030\-0\.0030for LaBSE to−0\.0086\-0\.0086for Multilingual\-E5, with all encoder\-level differences remaining less than0\.010\.01\. These results indicate that translation\-free LiRA reproduces translation\-based reliability estimates consistently across multilingual embedding architectures\. The findings therefore suggest that the observed equivalence is not specific to a particular encoder design but reflects a broader robustness of the LiRA framework to the use of native\-language representations\.

Table 1:Reliability and cross\-pipeline agreement by encoder, averaged across 11 items\.Item\-level differences were small throughout, as shown in Table[2](https://arxiv.org/html/2607.17466#S4.T2)\. For every encoder, all 11 items fell within about one and a half percentage point of the translated baseline, and the largest single\-item change across all encoders was−0\.0142\-0\.0142\(E5\)\. The slight ordering that does appear, LaBSE tracking the baseline most closely and E5 least closely, with Qwen between, is consistent with the encoders’ design objectives, but the spread is small enough that encoder choice is not consequential for the conclusion\. Because the encoders produced closely similar results, we use Qwen3\-Embedding, a strong general\-purpose multilingual encoder that also attained the highest absolute WEA in both conditions, for the item\-level analyses that follow\.

Table 2:Per\-itemΔ\\DeltaWEA by encoder\. Ref\. WEA is the Qwen translation\-based baseline shown for orientation\. Each encoder’sΔ\\DeltaWEA is calculated against its own translation\-based baseline\.Figure[1](https://arxiv.org/html/2607.17466#S4.F1)shows the per\-item translated and native WEA for the Qwen pipeline\. Across all 11 items, the two conditions are close, with native WEA within about one percentage point of translated WEA\. The largest item\-level gaps occur on the lower\-agreement items \(Item 8, 0\.917 vs\. 0\.906; Item 10, 0\.896 vs\. 0\.889\), while on one item \(Item 6\) the native pipeline slightly exceeds the translated one \(0\.925 vs\. 0\.927\)\. The pattern is one of close correspondence rather than uniform degradation\.

![Refer to caption](https://arxiv.org/html/2607.17466v1/figure1.png)Figure 1:LiRA WEA by item, English vs\. multilingual Qwen embeddings\.Although reliability estimates were highly similar across conditions, the translation\-based and translation\-free pipelines did not always produce identical benchmark assignments\. For Qwen3, benchmark agreement averaged 95\.3% across items, indicating that approximately 95% of responses received the same LiRA benchmark score under both pipelines\. Agreement in inconsistency classifications was effectively perfect across items, exceeding 99\.9% in all cases\.

The combination of high benchmark agreement and smallΔ\\DeltaWEA values suggests that LiRA reliability estimates are robust to moderate changes in local semantic neighborhoods\. While translation and native\-language embeddings may occasionally retrieve different neighboring responses, these differences rarely alter the overall reliability conclusions produced by the framework\.

Across the 11 items, 384 responses \(13–66 per item\) failed GPT\-4\.1 translation and were dropped from the translation\-based pipeline\. The multilingual pipeline recovered all of them under Qwen and E5, and 383 of 384 under LaBSE \(a single response LaBSE could not embed\)\. Including the rescued responses left reliability essentially unchanged; WEA on the full set differed from the matched set by at most 0\.0004 per item for every encoder, so the coverage gain comes without any measurable reliability cost\. Removing translation, therefore, makes the audit more inclusive: it scores responses that translation would discard, which are usually concentrated in low\-resource and morphologically complex languages\.

## 5Discussion

The purpose of this study was to evaluate whether translation is necessary for LiRA\-based reliability auditing in multilingual assessment systems\. Across 11 PIRLS CR items, three multilingual embedding models, and more than 160,000 scored responses, translation\-free LiRA produced reliability estimates that were highly similar to those obtained using translated English responses\. Mean differences in WEA remained below 1 percentage point for all embedding models, benchmark agreement exceeded 94% in every condition, and responses excluded due to translation failure were successfully recovered without altering reliability estimates\.

These findings indicate that multilingual sentence embeddings can reproduce translation\-based LiRA reliability estimates with only minimal differences, and this conclusion is robust across encoder architectures\. Although LaBSE, Multilingual\-E5, and Qwen3\-Embedding were developed with different objectives and training strategies, all three produced highly similar patterns of results\. The consistency of the findings suggests that the observed equivalence reflects a property of the LiRA framework itself rather than the behavior of any particular embedding model\.

The results also provide insight into the relationship between semantic drift and reliability auditing\. Prior work has shown that translation can alter semantic relationships, local neighborhood structure, and global embedding geometry \(e\.g\.,Conneauet al\.,[2018](https://arxiv.org/html/2607.17466#bib.bib3); Bezirhanet al\.,[2026](https://arxiv.org/html/2607.17466#bib.bib2)\)\. From a theoretical perspective, such changes could affect LiRA because benchmark scores are constructed directly from nearest\-neighbor relationships\. However, the present findings indicate that moderate changes in semantic neighborhoods do not necessarily translate into meaningful differences in reliability estimates\. Although approximately five percent of benchmark assignments differed between the translation\-based and translation\-free pipelines, the resulting changes in WEA were negligible\. This suggests that LiRA is relatively robust to modest perturbations in local semantic structure\.

The findings also have practical implications for multilingual assessment systems\. Translation has traditionally served as a mechanism for standardizing multilingual responses into a common language before scoring and downstream analysis\. While this approach simplifies processing, it introduces additional computational cost, dependence on translation quality, and the possibility of excluding responses that cannot be translated successfully\. The present results demonstrate that multilingual embeddings provide a viable alternative for reliability auditing\.

Several limitations should be acknowledged\. First, the study focused on dichotomously scored PIRLS reading\-comprehension items\. Whether similar results hold for longer responses, polytomous scoring schemes, or other assessment domains remains an open question for future research\. Second, although three multilingual embedding models were evaluated, the study did not examine the full range of contemporary multilingual encoders\. Future research should investigate whether the observed robustness extends to additional model families and emerging multilingual representation methods\. Finally, the present analysis focused on reliability estimation rather than automated scoring performance\. Translation may have different effects on predictive scoring models than on neighborhood\-based reliability auditing frameworks\.

As multilingual representation models continue to improve, translation\-free reliability auditing may provide a simpler, more scalable, and more inclusive approach for quality control in ILSAs\.

## References

- Massively multilingual sentence embeddings for zero\-shot cross\-lingual transfer and beyond\.Transactions of the association for computational linguistics7,pp\. 597–610\.Cited by:[§1](https://arxiv.org/html/2607.17466#S1.p4.1)\.
- L\. Beinborn and R\. Choenni \(2020\)Semantic drift in multilingual representations\.Computational Linguistics46\(3\),pp\. 571–603\.Cited by:[§1](https://arxiv.org/html/2607.17466#S1.p2.1),[§2](https://arxiv.org/html/2607.17466#S2.p1.1),[§2](https://arxiv.org/html/2607.17466#S2.p2.1)\.
- U\. Bezirhan, J\. Y\. Jung, and M\. von Davier \(2026\)Lost in translation: quantifying semantic drift in multilingual translations\.Note:Paper presentation at the Annual Meeting of the National Council on Measurement in Education, Los Angeles, CA, United StatesCited by:[§1](https://arxiv.org/html/2607.17466#S1.p2.1),[§2](https://arxiv.org/html/2607.17466#S2.p3.1),[§5](https://arxiv.org/html/2607.17466#S5.p3.1)\.
- A\. Conneau and G\. Lample \(2019\)Cross\-lingual language model pretraining\.Advances in neural information processing systems32\.Cited by:[§1](https://arxiv.org/html/2607.17466#S1.p4.1)\.
- A\. Conneau, R\. Rinott, G\. Lample, A\. Williams, S\. Bowman, H\. Schwenk, and V\. Stoyanov \(2018\)XNLI: evaluating cross\-lingual sentence representations\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2475–2485\.Cited by:[§2](https://arxiv.org/html/2607.17466#S2.p3.1),[§5](https://arxiv.org/html/2607.17466#S5.p3.1)\.
- G\. Dinu, A\. Lazaridou, and M\. Baroni \(2014\)Improving zero\-shot learning by mitigating the hubness problem\.arXiv preprint arXiv:1412\.6568\.Cited by:[§2](https://arxiv.org/html/2607.17466#S2.p3.1)\.
- M\. Faruqui and C\. Dyer \(2014\)Improving vector space word representations using multilingual correlation\.InProceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics,Vol\.2,Gothenburg, Sweden,pp\. 462–471\.External Links:[Document](https://dx.doi.org/10.3115/v1/E14-1049),[Link](https://aclanthology.org/E14-1049/)Cited by:[§2](https://arxiv.org/html/2607.17466#S2.p3.1)\.
- F\. Feng, Y\. Yang, D\. Cer, N\. Arivazhagan, and W\. Wang \(2022\)Language\-agnostic BERT sentence embedding\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Dublin, Ireland,pp\. 878–891\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.62),[Link](https://aclanthology.org/2022.acl-long.62/)Cited by:[§3\.3](https://arxiv.org/html/2607.17466#S3.SS3.p1.1)\.
- A\. Horbach, J\. Pehlke, R\. Laarmann\-Quante, and Y\. Ding \(2024\)Crosslingual content scoring in five languages using machine\-translation and multilingual transformer models\.International Journal of Artificial Intelligence in Education34\(4\),pp\. 1294–1320\.Cited by:[§1](https://arxiv.org/html/2607.17466#S1.p1.1)\.
- J\. Y\. Jung, U\. Bezirhan, and M\. von Davier \(2025a\)Reconceptualizing scoring reliability through linguistic similarity\.Educational and Psychological Measurement,pp\. 00131644251397428\.Cited by:[§1](https://arxiv.org/html/2607.17466#S1.p3.1),[§2](https://arxiv.org/html/2607.17466#S2.p4.1),[§3\.1](https://arxiv.org/html/2607.17466#S3.SS1.p2.1)\.
- J\. Y\. Jung, U\. Bezirhan, and M\. von Davier \(2026\)Optimizing automated scoring in ilsas with prompt compression\.Computers and Education: Artificial Intelligence,pp\. 100558\.Cited by:[§1](https://arxiv.org/html/2607.17466#S1.p1.1)\.
- J\. Y\. Jung, L\. Tyack, and M\. von Davier \(2024\)Combining machine translation and automated scoring in international large\-scale assessments\.Large\-scale Assessments in Education12\(1\),pp\. 10\.Cited by:[§1](https://arxiv.org/html/2607.17466#S1.p1.1)\.
- J\. Y\. Jung, L\. Tyack, and M\. von Davier \(2025b\)Towards the implementation of automated scoring in international large\-scale assessments: scalability and quality control\.Computers and Education: Artificial Intelligence8,pp\. 100375\.Cited by:[§1](https://arxiv.org/html/2607.17466#S1.p1.1)\.
- S\. Kornblith, M\. Norouzi, H\. Lee, and G\. Hinton \(2019\)Similarity of neural network representations revisited\.InInternational conference on machine learning,pp\. 3519–3529\.Cited by:[§1](https://arxiv.org/html/2607.17466#S1.p4.1)\.
- A\. Søgaard, S\. Ruder, and I\. Vulić \(2018\)On the limitations of unsupervised bilingual dictionary induction\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Melbourne, Australia,pp\. 778–788\.External Links:[Document](https://dx.doi.org/10.18653/v1/P18-1072),[Link](https://aclanthology.org/P18-1072/)Cited by:[§1](https://arxiv.org/html/2607.17466#S1.p2.1)\.
- H\. Somers, F\. Gaspari, and A\. Niño \(2006\)Detecting inappropriate use of free online machine translation by language students\. a special case of plagiarism detection\.InProceedings of the 11th Annual conference of the European Association for Machine Translation,Cited by:[§2](https://arxiv.org/html/2607.17466#S2.p1.1)\.
- L\. Tyack, L\. Khorramdel, and M\. von Davier \(2024\)Using convolutional neural networks to automatically score eight TIMSS 2019 graphical response items\.Computers and Education: Artificial Intelligence6,pp\. 100249\.Cited by:[§1](https://arxiv.org/html/2607.17466#S1.p1.1)\.
- L\. Wang, N\. Yang, X\. Huang, L\. Yang, R\. Majumder, and F\. Wei \(2024\)Multilingual E5 text embeddings: a technical report\.External Links:2402\.05672,[Link](https://arxiv.org/abs/2402.05672)Cited by:[§3\.3](https://arxiv.org/html/2607.17466#S3.SS3.p1.1)\.
- Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. Zhou \(2025\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.External Links:2506\.05176,[Link](https://arxiv.org/abs/2506.05176)Cited by:[§3\.3](https://arxiv.org/html/2607.17466#S3.SS3.p1.1)\.