The State and Fate of Multilingual, Contextual Evaluation in the NLP World

# The State and Fate of Multilingual, Contextual Evaluation in the NLP World
Source: [https://arxiv.org/html/2610.36990](https://arxiv.org/html/2610.36990)
###### Abstract

Multilingual evaluation benchmarks are the primary instrument for assessing whether large language models generalize beyond English, yet the adequacy of these benchmarks has received little systematic scrutiny\. We present a data\-driven audit of 51 recent multilingual benchmarks spanning 242 datasets and 219 languages, organized around three pillars:coverage,representativeness, andrigor\. Our analysis reveals that coverage is wide but thin with 36% of evaluated languages appearing in only a single benchmark, entire regions \(Oceania, the Americas, Central Asia\) are near\-zero, and a stark task equity gap leaves low\-resource languages evaluated on only 1–3 task categories versus 14 for high\-resource languages\. Representativeness is structurally compromised: translation from English remains the dominant construction strategy where 56% of all dataset–language instances are translated introducing artifacts and English\-centric framing, while culturally grounded content is concentrated in a handful of community\-driven benchmarks with narrow language scope\. The ecosystem thus forces a trade\-off between breadth and validity\. Rigor is undermined by benchmark contamination, including translated benchmark leakage and parallel corpus overlap that evade surface\-form detection\. We synthesize these findings into concrete recommendations for building evaluation frameworks that are natively constructed, culturally grounded, contamination\-aware, and designed to serve the communities whose languages they claim to evaluate\. The dashboard with all the metadata collected can be found at[https://aka\.ms/multilingual\-evals\.](https://aka.ms/multilingual-evals)

## 1Introduction

Evaluation benchmarks are the primary instrument through which the NLP community measures progress\. AI evaluation is critical beyond English, particularly for Global South languages, because most AI systems are designed from an English\-centric perspective, with Global North assumptions baked into their data, task design, and notions of success\([Akindotuni, 2025](https://arxiv.org/html/2610.36990#bib.bib33);[Lamentillo, 2025](https://arxiv.org/html/2610.36990#bib.bib38)\)\. Benchmark gains on English can obscure systematic failures in other languages and conditions where AI systems are deployed, potentially leading to marginalized communities being excluded, stereotyped, or exposed to high\-stakes failures\([Lachini, 2024](https://arxiv.org/html/2610.36990#bib.bib37);[Ahmad et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib43)\)\. Multilingual evaluation for Large Language Models \(LLMs\) has expanded rapidly in recent years, reflecting a growing recognition that English\-only evaluation is insufficient\([Weidinger et al\., 2022](https://arxiv.org/html/2610.36990#bib.bib36);[Mergen et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib35);[Guo et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib42);[Upadhayay, 2025](https://arxiv.org/html/2610.36990#bib.bib41);[Kargaran et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib40);[Zhang et al\., 2026](https://arxiv.org/html/2610.36990#bib.bib39)\)\. However, multilingual evaluation is not simply about measuring in another language what is measured in English, but about capturing the local contexts and cultural nuances that determine quality, usefulness and safety\. Recently at the India AI Impact Summit111[https://impact\.indiaai\.gov\.in/](https://impact.indiaai.gov.in/), the New Delhi Frontier AI Commitments make this explicit through the commitment titled ‘‘Strengthening Multilingual and Contextual Evaluations,’’ which calls for ensuring AI system effectiveness ‘‘across languages, cultures, and real\-world use cases’’222[https://www\.pib\.gov\.in/PressReleasePage\.aspx?PRID=2230201](https://www.pib.gov.in/PressReleasePage.aspx?PRID=2230201)\.

Progress on multilingual evaluation remains limited in important ways\([Wu et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib53);[Sinha et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib34);[Qin et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib52)\), with many efforts still being Western\-centric in their assumptions, priorities, and task formulations, often extending English benchmarks to other languages rather than grounding evaluation in the linguistic, cultural, and practical realities of diverse communities\([Smith, 2025](https://arxiv.org/html/2610.36990#bib.bib51);[Oh et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib50);[Mushtaq et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib49);[Zahraei and Asgari, 2025](https://arxiv.org/html/2610.36990#bib.bib44);[Plum et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib45)\)\. As a result, existing benchmarks are often not broad enough in the languages, varieties, and tasks they cover, nor sufficiently representative of how people actually use language technologies in different contexts\([McIntosh et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib46);[Ni et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib48);[Yang et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib47)\)\. In the spirit of[Joshi et al\. \(2020\)](https://arxiv.org/html/2610.36990#bib.bib107)”The State and Fate of Linguistic Diversity and Inclusion in the NLP World”, which took stock of languages represented in NLP and ones that are left behind, we offer a similar reckoning for multilingual evaluation by examining its growth and gaps\. We examine the current state of multilingual LLM evaluation through three key lenses: coverage, representativeness, and rigor, identify limitations in current practice and outline directions for building evaluation frameworks that are more meaningful, valid, and globally relevant\. We will release all structured metadata annotations, the full analysis codebase, and an interactive web dashboard that will allow users to explore benchmark coverage, representativeness, and per\-language statistics \(Appendix[C](https://arxiv.org/html/2610.36990#A3)\)\.

## 2Related Work

##### Multilingual Benchmarks and Language Coverage

Early cross\-lingual benchmarks such as XNLI\([Conneau et al\., 2018](https://arxiv.org/html/2610.36990#bib.bib108)\)established the paradigm of translating English datasets into multiple languages to perform multilingual evaluation\. This was scaled across dozens of languages and task types by large benchmarking efforts such as XTREME\-R, XGLUE, and MEGA\([Ruder et al\., 2021](https://arxiv.org/html/2610.36990#bib.bib101);[Liang et al\., 2020](https://arxiv.org/html/2610.36990#bib.bib104);[Ahuja et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib89)\)\. However, language coverage remains heavily skewed toward high\-resource Indo\-European and CJK languages\([Joshi et al\., 2020](https://arxiv.org/html/2610.36990#bib.bib107);[Blasi et al\., 2022](https://arxiv.org/html/2610.36990#bib.bib97);[Conneau et al\., 2020](https://arxiv.org/html/2610.36990#bib.bib8);[Ruder et al\., 2021](https://arxiv.org/html/2610.36990#bib.bib101)\), and low\-resource languages continue to lag behind\([Ahuja et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib89);[Asai et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib29);[Li et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib7)\)\. LLM\-powered data augmentation for multilingual commonsense reasoning also fails to produce meaningful text in certain low\-resource languages\([Doostmohammadi, 2026](https://arxiv.org/html/2610.36990#bib.bib5);[Gain et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib6)\)\. Furthermore, multilingual LLMs have been shown to use English as an internal pivot, with abstract representations closer to English than to other input languages, revealing a structural English\-centrism that needs to be addressed in evaluation\([Wendler et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib87);[Ye, 2025](https://arxiv.org/html/2610.36990#bib.bib4);[Blasi et al\., 2022](https://arxiv.org/html/2610.36990#bib.bib97);[Kumar et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib2);[Ahuja et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib89)\)\. These limitations have prompted calls for alternative paradigms such as performance prediction over per\-language evaluation sets\([Ahuja et al\., 2022](https://arxiv.org/html/2610.36990#bib.bib94)\)\.

##### Representativeness: Translation Artifacts and Cultural Grounding

A critical concern with multilingual benchmarks is their reliance on translation from English\([Huang et al\., 2025a](https://arxiv.org/html/2610.36990#bib.bib3)\), as exemplified by XNLI’s use of professionally translated MultiNLI data\([Conneau et al\., 2018](https://arxiv.org/html/2610.36990#bib.bib108)\)\. TyDiQA\([Clark et al\., 2020](https://arxiv.org/html/2610.36990#bib.bib102)\)addressed this by eliciting questions directly in each target language, avoiding translationese333Translationese refers to unnatural linguistic patterns arising from translation that fail to reflect how native speakers naturally express themselves\.artifacts and better capturing language\-specific information needs\. Several studies further highlight how cultural knowledge, pragmatic norms, and world knowledge diverge from the English\-centric assumptions embedded in translated datasets\([Hershcovich et al\., 2022](https://arxiv.org/html/2610.36990#bib.bib98);[Ruder et al\., 2021](https://arxiv.org/html/2610.36990#bib.bib101);[Conneau et al\., 2018](https://arxiv.org/html/2610.36990#bib.bib108);[Hou et al\., 2026](https://arxiv.org/html/2610.36990#bib.bib1);[Ebrahimi et al\., 2022](https://arxiv.org/html/2610.36990#bib.bib16);[Wendler et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib87)\)\. Language\-specific benchmarks such as MasakhaNER, IndicNLPSuite, AmericasNLI, and NusaX have shown that culturally grounded evaluation yields substantially different performance profiles than translated counterparts\([Adelani et al\., 2021](https://arxiv.org/html/2610.36990#bib.bib99);[Kakwani et al\., 2020](https://arxiv.org/html/2610.36990#bib.bib54);[Ebrahimi et al\., 2022](https://arxiv.org/html/2610.36990#bib.bib16);[Winata et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib92)\)\. However, even these benchmarks do not always reflect the priorities or lived realities of the communities they target\. Community\-driven efforts like Pariksha\([Watts et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib85)\)and Samiksha\([Bhat et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib32)\)help bridge this gap by incorporating stakeholder input, though such benchmarks remain scarce\. More recently, benchmarks like TUMLU\([Isbarov et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib80)\)for Turkic languages have begun prioritizing sociolinguistic authenticity over mere cross\-lingual alignment\.

##### Evaluation Rigor and Data Contamination

The integrity of benchmark\-based evaluation has come under increasing scrutiny as LLMs are trained on ever\-larger web corpora that may include benchmark test data\. Contamination has been argued to represent an existential threat to LLM evaluation, necessitating systematic analysis before conclusions are drawn from benchmark results\([Ahuja et al\., 2024b](https://arxiv.org/html/2610.36990#bib.bib58);[Sainz et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib88);[Jacovi et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib90);[Ahuja et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib89)\)\. Practical mitigation strategies have been proposed, including encrypting test data and demanding training exclusion controls from closed API model builders to proactively protect test data rather than retroactively detect leakage\([Jacovi et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib90);[Sainz et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib88);[Blasi et al\., 2022](https://arxiv.org/html/2610.36990#bib.bib97)\)\. The problem is further complicated by the opacity of training data in proprietary models, where contamination can only be inferred indirectly\([Sainz et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib88);[Jacovi et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib90);[Casalnuovo and Farrell, 2025](https://arxiv.org/html/2610.36990#bib.bib15)\)and agentic behavior showing evidence of knowledge of being tested444[https://www\.nist\.gov/caisi/cheating\-ai\-agent\-evaluations](https://www.nist.gov/caisi/cheating-ai-agent-evaluations)and deliberate cheating555[https://www\.anthropic\.com/engineering/eval\-awareness\-browsecomp](https://www.anthropic.com/engineering/eval-awareness-browsecomp)\. In the multilingual setting, contamination risks are compounded by the existence of parallel corpora and translated benchmark variants: a model trained on English test data may exhibit inflated performance on translated versions in other languages, yet this cross\-lingual leakage pathway remains poorly characterized\([Sainz et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib88);[Liang et al\., 2020](https://arxiv.org/html/2610.36990#bib.bib104);[Ruder et al\., 2021](https://arxiv.org/html/2610.36990#bib.bib101);[Conneau et al\., 2018](https://arxiv.org/html/2610.36990#bib.bib108);[Cruz and Aji, 2026](https://arxiv.org/html/2610.36990#bib.bib31)\)\. Contamination detection methods trained on one language often fail to generalize to linguistically dissimilar languages\([Macko et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib9)\)and usually rely on surface\-form overlap, which is inadequate for detecting semantic leakage through paraphrase or translation\([Cheng et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib57);[Macko et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib9);[Blevins and Zettlemoyer, 2022](https://arxiv.org/html/2610.36990#bib.bib95);[Fu et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib56);[Cruz and Aji, 2026](https://arxiv.org/html/2610.36990#bib.bib31)\)\.

## 3Survey of Existing Multilingual Benchmarks

We conduct a unified audit of existing multilingual benchmarks across coverage, representativeness, and rigor in four stages: benchmark collection, structured field extraction, manual verification, and contamination checks\.

We compile a corpus of 51 publicly released multilingual evaluation benchmarks, spanning diverse tasks \(NLI, QA, NER, sentiment analysis, commonsense reasoning, summarization, and others\), language resource levels, and language families\. We identify candidate benchmarks through a combination of literature search, leaderboard tracking, and community repositories\. We include both widely cited benchmarks \(e\.g\., XNLI, XQuAD, TyDiQA\) and recent community\-driven efforts targeting underrepresented languages and regions \(e\.g\., AfriSenti, MasakhaNER, IrokoBench\)\. The full list is provided in Table[8](https://arxiv.org/html/2610.36990#A3.T8)\. For fine\-grained analysis, each benchmark must be annotated with a consistent set of structured fields, including languages covered, language families, scripts, task types, construction methodology \(translated vs\. natively authored\), data sources, and licensing information\. To extract these fields at scale, we leverage Claude Opus 4\.6666[https://www\.anthropic\.com/news/claude\-opus\-4\-6](https://www.anthropic.com/news/claude-opus-4-6), prompting the model with the benchmark’s paper and metadata as context\. The extraction methodology is provided in the Appendix[A\.1](https://arxiv.org/html/2610.36990#A1.SS1)\. This semi\-automated approach enables consistent, structured annotation across all 51 benchmarks while substantially reducing manual effort\. We then perform a thorough manual verification of the annotations by cross\-referencing the extracted fields with the original papers, dataset cards, and associated repositories\.

### 3\.1Coverage

We analyze 51 benchmarks encompassing 242 distinct datasets and 219 unique languages along four dimensions: language, typological and geographic diversity, task coverage, and resource\-level balance\. Because many benchmarks aggregate existing datasets \(22 of the 51 are mixed or aggregation suites\), we conduct our analysis at thedataset–languagelevel to avoid inflating coverage counts when the same dataset appears in multiple benchmarks\.Unless otherwise specified, all subsequent analyses are conducted and reported at the dataset level\.

Language Coverage:Across the 51 benchmarks we survey, a total of219 unique languagesare evaluated across 242 datasets, suggesting reasonable coverage\. However, the distribution follows a steep power law, as shown in Figure[1](https://arxiv.org/html/2610.36990#S3.F1)that plots languages ranked by the number of datasets in which they appear\. A small cluster of high\-resource languages Hindi, Chinese, Spanish, Swahili, Arabic appear in 19 or more datasets, while48 of 219 languages \(37%\) appear in only a single dataset\. For these 48 languages, there exists no independent replication of evaluation results and no opportunity for cross\-dataset comparison\. This long\-tail structure has a direct methodological implication: claims of multilingual generalization rest on a dense evaluation of a handful of languages and single\-point estimates for the majority\.

Figure 1:Languages ranked by the number of datasets in which they appear\.The long tail reveals that 36% of evaluated languages appear in only a single dataset\.Typological and Geographical Coverage:Language count alone is an insufficient measure of diversity\. Two benchmark suites could each cover 20 languages yet differ vastly in typological range if one samples broadly across families and scripts while the other concentrates on closely related Indo\-European varieties\. Figure[2\(a\)](https://arxiv.org/html/2610.36990#S3.F2.sf1)shows the distribution of unique languages across the 29 language families represented in our survey\. Indo\-European languages account for83 of 219 languages \(38%\), while Atlantic\-Congo \(39 languages\) ranks second, but this representation is almost entirely attributable to Africa\-focused benchmarks like AfroBench, IrokoBench, and Uhuru\. Families such as Austroasiatic \(3 languages\), Tai\-Kadai \(3\), Japonic \(1\), and Quechuan \(2\) are poorly represented\. The entire Oceanic branch of Austronesian is absent\. Figure[2\(b\)](https://arxiv.org/html/2610.36990#S3.F2.sf2)reveals a similar distribution: Latin script accounts for the majority of evaluated languages, followed by Devanagari, Arabic, and CJK\. Scripts used by hundreds of millions of speakers such as Ge’ez \(Ethiopic\), Khmer, Myanmar appear in only 1–3 benchmarks\. Script diversity matters because models may exhibit systematically different tokenization efficiency\([Ahia et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib91)\), character\-level recall, and decoding accuracy across scripts, and these script\-specific failure modes remain invisible when evaluation concentrates on Latin and Devanagari\. We find similar trends in Geographical coverage as well with entire regions of the world such as Oceania and South America having very poor or no representation\.

\(a\)Language family representation\.\(b\)Script coverage \(top 15 scripts\)\.
Figure 2:Typological concentration in multilingual benchmarks\.\(a\)Indo\-European dominates at 38% of all languages, with many families represented by fewer than 5 languages\.\(b\)Latin script accounts for the majority; widely\-used scripts such as Ge’ez, Khmer, and Myanmar are barely represented\.Resource\-Level Distribution and Task Equity:We examine the resource\-level composition of evaluated languages using the taxonomy from[Joshi et al\. \(2020\)](https://arxiv.org/html/2610.36990#bib.bib107)\. Across the 242 distinct datasets, 65% of evaluated languages fall in the two lowest resource classes \(Scraping\-Bys and Left\-Behinds, levels 0–1\), while Winners \(level 5\) account for only 5% \(Figure[7](https://arxiv.org/html/2610.36990#A1.F7)in Appendix[A\.2](https://arxiv.org/html/2610.36990#A1.SS2)\)\. This distribution appears to favor low\-resource languages, but a finer\-grained analysis reverses the picture\. Figure[3](https://arxiv.org/html/2610.36990#S3.F3)disaggregates task coverage across 15 categories by resource level: the 10 Winner languages are each evaluated on a median of 14 task categories across multiple benchmarks, whereas the 160 low\-resource languages \(levels 0–2\) are typically tested on only 1–3 tasks in a single benchmark\. For Left\-Behinds, Translation alone accounts for 29% of all evaluation rows and Sentiment/Classification for 25%, while Coding, Safety/Toxicity, and Instruction Following each represent only 1%\. By contrast, Winners and Underdogs distribute evaluation more evenly, with Math/Reasoning \(11% and 8%\), Embedding/Retrieval \(9% and 6%\), and Safety \(7% and 9%\) all receiving meaningful attention\. In absolute terms, Safety/Toxicity evaluation is scarce—only roughly 25 dataset–language rows across all datasets—and these are concentrated almost entirely at higher resource levels, meaning that safety coverage for low\-resource languages is effectively nonexistent\. Figure[8](https://arxiv.org/html/2610.36990#A1.F8)\(Appendix\) cross\-tabulates task categories against world regions and confirms that QA and knowledge tasks dominate everywhere, while specialized capabilities show uneven coverage\. Thistask equity gapmeans that for most low\-resource languages, a model’s “evaluation” amounts to a narrow probe on translation and classification, making it far too limited to support claims of general competence\.

![Refer to caption](https://arxiv.org/html/2610.36990v1/09c_task_equity_fine.png)Figure 3:Task coverage by resource level\. Each cell shows the percentage of a resource level’s dataset–language rows devoted to that task category\.Left\-Behinds are overwhelmingly evaluated on Translation \(29%\) and Sentiment/Classification \(25%\); specialised tasks such as Coding, Safety, Instruction Following, each account for≤\\leq1% of their evaluation, whereas Winners are evaluated more evenly across all 15 categories\.The coverage analysis reveals an ecosystem that iswide but thin: many languages are nominally included, but the depth of evaluation measured by replication across datasets, typological diversity, task breadth, and resource\-level balance is concentrated in a small core\. The next section asks whether the quality of coverage that does exist is adequate\.

### 3\.2Representativeness

Coverage establisheswhatis evaluated; representativeness asks whether that evaluation isvalid\. A dataset may nominally cover a language, yet evaluate it using content translated from English, written by non\-native speakers, or detached from local cultural context and assumptions\. We audit the 242 datasets along two axes: translation and cultural grounding\. At the dataset–language level, 56% of all instances are translated from English, 36% are natively authored, and 8% are partially adapted\.

Translation vs\. Native Construction:The dominant construction strategy for multilingual datasets remains translation from English source datasets\. This approach scales efficiently as one annotation effort produces content in many languages but comes with the cost of translationese artifacts, English\-centric framing, and inherited contamination risk \(Section[3\.3](https://arxiv.org/html/2610.36990#S3.SS3)\)\([Artetxe et al\., 2020a](https://arxiv.org/html/2610.36990#bib.bib105);[Graham et al\., 2020](https://arxiv.org/html/2610.36990#bib.bib103)\)\. Figure[4\(a\)](https://arxiv.org/html/2610.36990#S3.F4.sf1)disaggregates translation status by world region\. In every region, a substantial fraction of languages are evaluated primarily through translated content\. Datasets covering Europe and East Asia are the most translation\-dependent, while Sub\-Saharan Africa shows the highest share of natively authored content, which is a direct consequence of language\-specific datasets created from scratch such as AfroBench\([Ojo et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib76)\), IrokoBench[Adelani et al\. \(2025\)](https://arxiv.org/html/2610.36990#bib.bib69), and MasakhaNER\([Adelani et al\., 2021](https://arxiv.org/html/2610.36990#bib.bib99)\)\.

Cultural Representativeness:Culturally grounded datasets reflect the norms, knowledge, and pragmatic conventions of the target language community\. A dataset can be natively authored yet culturally shallow \(e\.g\., factoid QA drawn from Wikipedia\), and a translated dataset can incorporate cultural adaptation \(transcreation\) to make it more culturally grounded\. However, in practice, the two are correlated: datasets constructed from scratch by native\-speaker communities are far more likely to be culturally grounded\. Figure[4\(b\)](https://arxiv.org/html/2610.36990#S3.F4.sf2)shows cultural grounding rates by region\. Datasets targeting Sub\-Saharan Africa and South Asia exhibit the highest cultural grounding rates, reflecting the deliberate design philosophy of community\-driven efforts like CulturalBench\([Chiu et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib81)\), Samiksha[Hamna et al\. \(2026\)](https://arxiv.org/html/2610.36990#bib.bib61), and TUMLU\([Isbarov et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib80)\)\. By contrast, datasets with broad cross\-regional scope tend to have lower cultural grounding as it is difficult to achieve at scale\. This finding reinforces the pattern: the datasets that are methodologically strongest along the representativeness axis are those with the narrowest language coverage\.

\(a\)Translation status by world region\.\(b\)Cultural grounding by world region\.
Figure 4:Representativeness of multilingual benchmarks by world region\.\(a\)Europe and East Asia are the most translation\-dependent; Sub\-Saharan Africa has the highest share of natively authored content\.\(b\)Community\-driven benchmarks in Sub\-Saharan Africa and South Asia achieve the highest cultural grounding rates; broad\-coverage suites score lower\.Annotator Demographics:Transparency around annotation workforces remains limited: only 8 of 51 benchmarks \(16%\) report annotator demographics such as geographic origin, ethnicity, or educational background, and only 17 \(33%\) disclose compensation details\. Researchers are the dominant creator type across datasets, with community volunteers and crowdworkers accounting for a small minority, raising questions about whose linguistic backgrounds and priorities shape “ground truth” for underrepresented languages\. The few benchmarks that do document their annotation pipeline in detail \(CulturalBench, Aya, and Uhuru\) tend to be the same ones that score highest on cultural grounding, suggesting that annotator transparency and evaluation validity go hand in hand\.

### 3\.3Rigor

#### 3\.3\.1Metrics

Metrics determine what counts as success, and poorly chosen metrics can give a misleading picture of model quality\. Most benchmarks rely either on standard reference\-based metrics to compare model outputs against ground truth, or on reference\-free metrics such as LLM judges\. However, many widely used metrics, such as BLEU, ROUGE, exact match, and Word Error Rate, were developed largely in English\-centric settings and can be poor proxies for quality in morphologically rich languages or languages with non\-standard spellings, where valid responses may differ substantially in surface form\. Similarly, although LLM judges are increasingly becoming standard in evaluation pipelines, prior work shows that they often align poorly with human judgments in multilingual and multicultural settings, particularly for low\-resource languages and in culturally grounded, context\-sensitive evaluations\([Hada et al\., 2024b](https://arxiv.org/html/2610.36990#bib.bib84);[Watts et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib85);[Hamna et al\., 2026](https://arxiv.org/html/2610.36990#bib.bib61);[Hada et al\., 2024a](https://arxiv.org/html/2610.36990#bib.bib83)\)\.

#### 3\.3\.2Contamination

DatasetTaskPrior WorkSTAMPDCQPaws\-XParaphrase Det\.9/1067\.9%32\.6%MgsmMath—45\.5%30\.9%XCopaCommonsense9/10——XQuADQA9/10——AfriMGSMMath—21\.0%18\.4%XnliNLI7/10——XStoryClozeStory Cloze7/10——FloresTranslation7/71\.2%—XLSumSummarization5/7——IndicParamKnowledge—4\.3%14\.7%MmmluKnowledge—2\.0%19\.6%Global MMLUKnowledge—1\.7%17\.5%IncludeKnowledge—1\.0%18\.8%MiluKnowledge—0\.0%21\.7%Table 1:Summary of contamination evidence across 14 multilingual datasets\.Prior Work: fraction of models flagged as contaminated by surface\-form and overlap methods\([Ahuja et al\., 2024a](https://arxiv.org/html/2610.36990#bib.bib82);[Ahuja et al\., 2024b](https://arxiv.org/html/2610.36990#bib.bib58);[Yao et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib86)\)\(details in Table[2](https://arxiv.org/html/2610.36990#A2.T2)\)\.STAMP: overall contamination rate \(% of model–language pairs flagged atp<0\.05p\{<\}0\.05\) from our analysis usingSTAMP\([Rastogi et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib30)\)\(details in Table[4](https://arxiv.org/html/2610.36990#A2.T4)\)\.DCQ: mean maximum recognition rate across models and languages using the Data Contamination Quiz\( \); 20% is the random\-chance floor \(details in Table[5](https://arxiv.org/html/2610.36990#A2.T5)\)\.Contamination is especially consequential in multilingual evaluation because much of the evaluation ecosystem isthin:36% of languagesappear in only one dataset \(Figure[1](https://arxiv.org/html/2610.36990#S3.F1)\), and56% of dataset–language rowsrely on translated content \(Figure[4\(a\)](https://arxiv.org/html/2610.36990#S3.F4.sf1)\)\. When a language’s evaluation rests on a single translated dataset, contamination through any pathwayinvalidates the only evidence of model competencefor that language\. Thetask equity gap\(Figure[3](https://arxiv.org/html/2610.36990#S3.F3)\) further amplifies this risk: for low\-resource languages evaluated on only 1–2 task categories, contamination on even a single task can dominate the reported result\.

Prior Contamination Analyses\.Several studies have investigated multilingual benchmark contamination using surface\-form overlap and membership inference methods\.[Ahuja et al\. \(2024a\)](https://arxiv.org/html/2610.36990#bib.bib82)found that GPT\-4, Llama\-2, and Gemini\-Pro are likely contaminated on PAWS\-X, XCopa, and XQuAD\.[Ahuja et al\. \(2024b\)](https://arxiv.org/html/2610.36990#bib.bib58)extended this analysis to newer models \(Llama\-3\.1, Mistral\-v0\.3, Gemma\-2\) and confirmed pervasive contamination across 7 benchmarks\.[Yao et al\. \(2024\)](https://arxiv.org/html/2610.36990#bib.bib86)further demonstrated that contaminationtransfers cross\-lingually: models contaminated on the English version of a dataset show inflated scores on non\-English translations \(Table[3](https://arxiv.org/html/2610.36990#A2.T3), Appendix\)\. Aggregating across these studies, 9 of the 10 models tested are contaminated on PAWS\-X, XCopa, and XQuAD, and 7 of 7 are contaminated on Flores \(Table[1](https://arxiv.org/html/2610.36990#S3.T1); per\-model details in Table[2](https://arxiv.org/html/2610.36990#A2.T2), Appendix\)\. However, these analyses examined a limited set of established datasets, leaving open the question of whether contamination extends to newer evaluation suites\.

Contamination Analysis with STAMP\.To provide an updated assessment, we applySTAMP\([Rastogi et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib30)\)to a broader and more contemporary suite of9 datasetsincluding recent ones such as Global MMLU, MILU, INCLUDE, and IndicParam alongside established ones like PAWS\-X, MGSM, and AfriMGSM across7 recent models\(Table[4](https://arxiv.org/html/2610.36990#A2.T4), Appendix\)\. STAMP makes minimal assumptions about data format, enabling reliable membership detection at trace levels while strictly preserving the semantic meaning and evaluation utility of the original content\. Following[Rastogi et al\. \(2025\)](https://arxiv.org/html/2610.36990#bib.bib30), we operationalize contamination detection by generating 9 paraphrased versions of each dataset item and measuring the performance gap between the original and its variants\. Paraphrases are produced usingQwen\-3\.5 397B, a large multilingual model chosen because its scale ensures high\-quality rephrasings across diverse writing systems\. Our results reveal that contamination remainspervasivefor older datasets:PAWS\-X\(67\.9% of model–language pairs flagged\),MGSM\(45\.5%\), andAfriMGSM\(21\.0%\) show significant contamination signals across all seven models \(Table[1](https://arxiv.org/html/2610.36990#S3.T1)\)\. By contrast, newer datasets—MILU \(0\.0%\), INCLUDE \(1\.0%\), and Global MMLU \(1\.7%\) exhibit markedly lower contamination rates, suggesting that dataset age and prominence are strong predictors of contamination risk\. One notable discrepancy emerges forFlores: prior surface\-form methods flag 7 of 7 models, yet STAMP detects contamination in only 1\.2% of model–language pairs\. This divergence likely reflects differences in what each method measures, i\.e\. surface\-form overlap versus memorization\-driven performance gaps and underscores that no single detection approach suffices for multilingual contamination\.

Complementary Analysis with DCQ\.We additionally apply theData Contamination Quiz \(DCQ\)[Golchin and Surdeanu \(2025\)](https://arxiv.org/html/2610.36990#bib.bib68)to 85 dataset–language partitions from 8 datasets across the same 7 models\. For each of 100 sampled items per partition, GPT\-4\.1 produces four word\-level synonym perturbations\. A bias\-detection quiz first presents only the four perturbations and a “none of these” option; the original item is then inserted into positions that the model under\-selected in this initial quiz\. Selecting the verbatim original provides a recognition signal while the initial responses estimate positional bias\. DCQ reports a conservative and a maximum recognition rate; we use the maximum rate in Table[1](https://arxiv.org/html/2610.36990#S3.T1)for consistency across models, while noting that taking the best rate across positions is optimistic and that 20% is the random\-chance floor\. At the dataset level, PAWS\-X \(32\.6%\) and MGSM \(30\.9%\) show the strongest average recognition, whereas the remaining dataset averages range from 14\.7% to 21\.7%\. Recognition is generally higher for English partitions: for example, English versus non\-English averages are 45\.7% versus 15\.3% for AfriMGSM, 44\.0% versus 29\.6% for MGSM, and 47\.6% versus 30\.1% for PAWS\-X\. These values measure recognition rather than the fraction of a dataset that leaked and should not be directly equated with STAMP’s significance\-based flag rate\. Full dataset–model results, language\-group comparisons, and validity caveats are reported in Appendix[B\.1](https://arxiv.org/html/2610.36990#A2.SS1)\.

Challenges of Multilingual Contamination Detection\.Both STAMP and DCQ rely on another LLM to construct the measuring instrument, so variation in that model’s competence across languages can be mistaken for variation in contamination\. For STAMP, semantically faithful watermarked rephrasing is harder across typologically distant and low\-resource languages; uneven paraphrase quality introduces noise into the paired test\. Its KGW watermarking scheme also partitions vocabulary through a tokenizer\-dependent hash function\([Kirchenbauer et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib60)\), so sparse or unstable tokenization can weaken the watermark and the test’s statistical power\. For DCQ, difficulty is determined by the quality of GPT\-4\.1’s decoys\. Perturbation distance ranges from 5\.8% for AfriMGSM in Amharic to 48\.1% for Global MMLU in English and varies with language resource level\. In the eight weakest partitions, decoys are too similar to the original for reliable separation, pinning recognition near the chance floor regardless of exposure; conversely, misspellings or semantic drift can make decoys easier to reject and inflate recognition\. This asymmetry is especially consequential because it can reproduce the expected finding: apparently lower contamination in low\-resource languages may reflect a weaker measuring instrument rather than less exposure\.

DCQ introduces a further identification problem because its prompt names the dataset, potentially conflating recognition of the dataset label with recognition of its instances\. Our natural replicate holds all 100 GSM8K items byte\-identical while changing only the label from AfriMGSM to MGSM; the resulting recognition difference ranges from 1 percentage point for Aya\-Expanse\-8B to 23 points for Gemma\-3\-12B \(Appendix[B\.1](https://arxiv.org/html/2610.36990#A2.SS1)\)\. This variation cannot be attributed to contamination and limits how finely models or languages can be compared\. Removing the dataset name would reduce the confound but also remove a cue on which DCQ’s recognition design depends\. Together, these limitations show that neither method is language\-neutral: multilingual contamination claims require per\-language instrument validation and convergence across complementary methods rather than direct comparison of raw scores\.

#### 3\.3\.3Reporting Practices in Model Releases

Based on our analysis, we ask: how do model developers report multilingual evaluation? We audit 23 recent model releases \(15 open\-weight, 8 closed\-models, 2024–2026\) across five binary transparency dimensions \(1\) whether training language composition is explicitly disclosed, \(2\) whether any multilingual evaluation dataset/benchmark is reported, \(3\) whether the specific evaluation languages are listed, \(4\) whether a contamination detection technique is named, and \(5\) whether contamination results are reported\. The composite transparency score is the sum \(0–5\) \(Figure[12](https://arxiv.org/html/2610.36990#A1.F12); full methodology and per\-model breakdown in Appendix[A\.7](https://arxiv.org/html/2610.36990#A1.SS7)\)\. The results are stark: 35% of releases reportzeromultilingual datasets, and 87% provide no contamination analysis\. Only 3 models \(all open\-weight\) report any contamination testing, and even these leave cross\-lingual contamination pathways unexamined\. When multilingual evaluationisreported, it concentrates on a handful of benchmarks, the model–dataset matrix is 93% sparse, while natively authored datasets from our survey \(AfroBench, IrokoBench, MILU\) are almost entirely unused\. Open\-weight models are more transparent \(mean score 2\.47/5\) than closed \(0\.75/5\), but both fall well below adequacy: the overall mean is 1\.87/5, and no closed model discloses training language composition or reports contamination\. These findings argue for a standardized multilingual evaluation card mandating per\-language dataset disclosure accounting for contamination\. Finally, multilingual safety evaluation is reported by only 4 of 23 models: Tiny Aya, Claude Opus 4\.6, Claude Sonnet 4\.6, and CommandA\. All other models either test safety only in English or do not report safety testing at all\.

## 4Discussion

Our audit of 51 recent multilingual benchmarks spanning 242 datasets reveals a consistent pattern across all three pillars\.Coverageis wide but thin: 36% of languages appear in only a single dataset and entire regions of the world are completely absent\. Multilingual safety datasets remain few in number and provide little coverage of low\-resource languages\.Representativenessis structurally compromised: the datasets with the broadest language coverage are the most translated and least culturally grounded, while the most valid datasets cover the fewest languages\. Annotator demographics are rarely reported, with researchers being the primary data creators of multilingual datasets, raising concerns about whose values and priorities are reflected in these datasets\.Rigoris undermined by multiple contamination pathways that are amplified by translation\-based construction, English\-centric metrics that do not always work well on diverse languages and a lack of meaningful reporting of multilingual evaluation in model releases\. These findings point to a systemic misalignment between how multilingual evaluation is currently conducted and what it would need to be to support trustworthy claims of cross\-lingual competence\. If these issues are not taken seriously, multilingual evaluation risks becoming performative: creating the appearance of inclusion while masking shallow support, overstating progress, and ultimately widening existing linguistic and cultural gaps in AI\.

Supporting a language is not simply a matter of translating prompts, reporting a score, or listing it in a benchmark; it requires evidence that the system works in ways that are meaningful for users in their local linguistic and cultural contexts\. Similarly, evaluating a language is not just measuring in another language what was originally defined in English, but assessing model behavior under the norms, knowledge systems, communicative practices, and risks that shape real use\. Our work aims to raise the bar for how multilingual evaluation is conducted and reported\. Reporting should make clear what was evaluated, how data was created or adapted, what contexts are covered, and what claims the results do and do not support\. Meaningful community participation must go beyond translation or validation to include communities in defining tasks, rubrics, and failure modes\. AI itself can also help scale multilingual evaluation through advances in multilingual synthetic data\([Chitale et al\., 2026](https://arxiv.org/html/2610.36990#bib.bib59)\), aligned multilingual LLM judges, and dynamic benchmarks to avoid contamination\. Multilingual evaluation must also move beyond model\-level assessment to include product, user, and real\-world impact evaluation777[https://eval\.playbook\.org\.ai/](https://eval.playbook.org.ai/), since model performance alone cannot capture how systems behave in deployed, culturally situated settings\.

There is an inherent tension between coverage and representativeness in multilingual evaluation: approaches that scale across many languages often depend on standardization, while representativeness requires deeper engagement with local variation, context, and heterogeneity\. If the field is committed to supporting everyone with AI, it must accept this tension as a fundamental reality: meaningful evaluation cannot always be reduced to uniform or fully replicable patterns across highly diverse settings, and some degree of variation must be expected as a consequence of taking linguistic, cultural, and contextual diversity seriously\.

## References

- Adelaniet al\.\(2021\)D\. I\. Adelani, J\. Abbott, G\. Neubig, D\. D’souza, J\. Kreutzer, C\. Lignos, C\. Palen\-Michel, H\. Buzaaba, S\. Rijhwani, S\. Ruder, S\. Mayhew, I\. A\. Azime, S\. H\. Muhammad, C\. C\. Emezue, J\. Nakatumba\-Nabende, P\. Ogayo, A\. Anuoluwapo, C\. Gitau, D\. Mbaye, J\. Alabi, S\. M\. Yimam, T\. R\. Gwadabe, I\. Ezeani, R\. A\. Niyongabo, J\. Mukiibi, V\. Otiende, I\. Orife, D\. David, S\. Ngom, T\. Adewumi, P\. Rayson, M\. Adeyemi, G\. Muriuki, E\. Anebi, C\. Chukwuneke, N\. Odu, E\. P\. Wairagala, S\. Oyerinde, C\. Siro, T\. S\. Bateesa, T\. Oloyede, Y\. Wambui, V\. Akinode, D\. Nabagereka, M\. Katusiime, A\. Awokoya, M\. MBOUP, D\. Gebreyohannes, H\. Tilaye, K\. Nwaike, D\. Wolde, A\. Faye, B\. Sibanda, O\. Ahia, B\. F\. P\. Dossou, K\. Ogueji, T\. I\. DIOP, A\. Diallo, A\. Akinfaderin, T\. Marengereke, and S\. OseiMasakhaNER: named entity recognition for African languages\.Transactions of the Association for Computational Linguistics9,pp\. 1116–1131\.External Links:[Link](https://aclanthology.org/2021.tacl-1.66/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00416)Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.36990#S3.SS2.p2.1)\.
- Adelaniet al\.\(2025\)D\. I\. Adelani, J\. Ojo, I\. A\. Azime, J\. Y\. Zhuang, J\. O\. Alabi, X\. He, M\. Ochieng, S\. Hooker, A\. Bukula, E\. A\. Lee, C\. I\. Chukwuneke, H\. Buzaaba, B\. K\. Sibanda, G\. K\. Kalipe, J\. Mukiibi, S\. Kabongo Kabenamualu, F\. Yuehgoh, M\. Setaka, L\. Ndolela, N\. Odu, R\. Mabuya, S\. Osei, S\. H\. Muhammad, S\. Samb, T\. K\. Guge, T\. V\. Sherman, and P\. StenetorpIrokoBench: a new benchmark for African languages in the age of large language models\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 2732–2757\.External Links:[Link](https://aclanthology.org/2025.naacl-long.139/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.139),ISBN 979\-8\-89176\-189\-6Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.18.6),[§3\.2](https://arxiv.org/html/2610.36990#S3.SS2.p2.1)\.
- Ahiaet al\.\(2023\)O\. Ahia, S\. Kumar, H\. Gonen, J\. Kasai, D\. Mortensen, N\. Smith, and Y\. TsvetkovDo all languages cost the same? tokenization in the era of commercial language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 9904–9923\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.614/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.614)Cited by:[§3\.1](https://arxiv.org/html/2610.36990#S3.SS1.p3.1)\.
- Ahmadet al\.\(2025\)A\. Ahmad, Y\. Vallès, and Y\. IdaghdourBias in ai systems: integrating formal and socio\-technical approaches\.Frontiers in Big Data8,pp\. 1686452\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p1.1)\.
- Ahujaet al\.\(2022\)K\. Ahuja, S\. Dandapat, S\. Sitaram, and M\. ChoudhuryBeyond static models and test sets: benchmarking the potential of pre\-trained models across tasks and languages\.InProceedings of NLP Power\! The First Workshop on Efficient Benchmarking in NLP,T\. Shavrina, V\. Mikhailov, V\. Malykh, E\. Artemova, O\. Serikov, and V\. Protasov \(Eds\.\),Dublin, Ireland,pp\. 64–74\.External Links:[Link](https://aclanthology.org/2022.nlppower-1.7/),[Document](https://dx.doi.org/10.18653/v1/2022.nlppower-1.7)Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1)\.
- Ahujaet al\.\(2023\)K\. Ahuja, H\. Diddee, R\. Hada, M\. Ochieng, K\. Ramesh, P\. Jain, A\. Nambi, T\. Ganu, S\. Segal, M\. Ahmed, K\. Bali, and S\. SitaramMEGA: multilingual evaluation of generative AI\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 4232–4267\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.258/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.258)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.7.6),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Ahujaet al\.\(2024a\)S\. Ahuja, D\. Aggarwal, V\. Gumma, I\. Watts, A\. Sathe, M\. Ochieng, R\. Hada, P\. Jain, M\. Ahmed, K\. Bali, and S\. SitaramMEGAVERSE: benchmarking large language models across languages, modalities, models and tasks\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 2598–2637\.External Links:[Link](https://aclanthology.org/2024.naacl-long.143/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.143)Cited by:[Table 2](https://arxiv.org/html/2610.36990#A2.T2),[§3\.3\.2](https://arxiv.org/html/2610.36990#S3.SS3.SSS2.p2.1),[Table 1](https://arxiv.org/html/2610.36990#S3.T1)\.
- Ahujaet al\.\(2024b\)S\. Ahuja, V\. Gumma, and S\. SitaramContamination report for multilingual benchmarks\.External Links:2410\.16186,[Link](https://arxiv.org/abs/2410.16186)Cited by:[Table 2](https://arxiv.org/html/2610.36990#A2.T2),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1),[§3\.3\.2](https://arxiv.org/html/2610.36990#S3.SS3.SSS2.p2.1),[Table 1](https://arxiv.org/html/2610.36990#S3.T1)\.
- Akindotuni \(2025\)D\. AkindotuniResource asymmetry in multilingual nlp: a comprehensive review and critique\.Journal of Computer and Communications13\(7\),pp\. 14–47\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p1.1)\.
- Artetxeet al\.\(2020a\)M\. Artetxe, G\. Labaka, and E\. AgirreTranslation artifacts in cross\-lingual transfer learning\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 7674–7684\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.618/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.618)Cited by:[§3\.2](https://arxiv.org/html/2610.36990#S3.SS2.p2.1)\.
- Artetxeet al\.\(2020b\)M\. Artetxe, S\. Ruder, and D\. YogatamaOn the cross\-lingual transferability of monolingual representations\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 4623–4637\.External Links:[Link](https://aclanthology.org/2020.acl-main.421/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.421)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.35.6)\.
- Asaiet al\.\(2021\)A\. Asai, J\. Kasai, J\. Clark, K\. Lee, E\. Choi, and H\. HajishirziXOR QA: cross\-lingual open\-retrieval question answering\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,K\. Toutanova, A\. Rumshisky, L\. Zettlemoyer, D\. Hakkani\-Tur, I\. Beltagy, S\. Bethard, R\. Cotterell, T\. Chakraborty, and Y\. Zhou \(Eds\.\),Online,pp\. 547–564\.External Links:[Link](https://aclanthology.org/2021.naacl-main.46/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.46)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.42.6)\.
- Asaiet al\.\(2024\)A\. Asai, S\. Kudugunta, X\. Yu, T\. Blevins, H\. Gonen, M\. Reid, Y\. Tsvetkov, S\. Ruder, and H\. HajishirziBUFFET: benchmarking large language models for few\-shot cross\-lingual transfer\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 1771–1800\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.9.6),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1)\.
- Bandarkaret al\.\(2024\)L\. Bandarkar, D\. Liang, B\. Muller, M\. Artetxe, S\. N\. Shukla, D\. Husa, N\. Goyal, A\. Krishnan, L\. Zettlemoyer, and M\. KhabsaThe belebele benchmark: a parallel reading comprehension dataset in 122 language variants\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 749–775\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.3.6)\.
- Baucellset al\.\(2025\)I\. Baucells, J\. Aula\-Blasco, I\. de\-Dios\-Flores, S\. Paniagua Suárez, N\. Perez, A\. Salles, S\. Sotelo Docio, J\. Falcão, J\. J\. Saiz, R\. Sepulveda Torres, J\. Barnes, P\. Gamallo, A\. Gonzalez\-Agirre, G\. Rigau, and M\. VillegasIberoBench: a benchmark for LLM evaluation in Iberian languages\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 10491–10519\.External Links:[Link](https://aclanthology.org/2025.coling-main.699/)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.45.6)\.
- Bayeset al\.\(2024\)E\. Bayes, I\. A\. Azime, J\. O\. Alabi, J\. Kgomo, T\. Eloundou, E\. Proehl, K\. Chen, I\. Khadir, N\. A\. Etori, S\. H\. Muhammad, C\. Mpanza, I\. P\. Thete, D\. Klakow, and D\. I\. AdelaniUhura: a benchmark for evaluating scientific question answering and truthfulness in low\-resource african languages\.External Links:2412\.00948,[Link](https://arxiv.org/abs/2412.00948)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.44.6)\.
- Bhatet al\.\(2025\)G\. Bhat, S\. Mukherjee, F\. Lalani, E\. Hadfield, D\. Siddarth, K\. Bali, S\. Sitaram,et al\.Building benchmarks from the ground up: community\-centered evaluation of llms in healthcare chatbot settings\.arXiv preprint arXiv:2509\.24506\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1)\.
- Blasiet al\.\(2022\)D\. Blasi, A\. Anastasopoulos, and G\. NeubigSystematic inequalities in language technology performance across the world’s languages\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 5486–5505\.External Links:[Link](https://aclanthology.org/2022.acl-long.376/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.376)Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Blevins and Zettlemoyer \(2022\)T\. Blevins and L\. ZettlemoyerLanguage contamination helps explains the cross\-lingual capabilities of English pretrained models\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 3563–3574\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.233/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.233)Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Casalnuovo and Farrell \(2025\)C\. Casalnuovo and M\. T\. FarrellReview of grey box/black box data contamination metrics on open and commercial models\.Technical reportSandia National Laboratories \(SNL\-CA\), Livermore, CA \(United States\)\.Note:Dataset contamination is a problem where benchmarks and tasks used to evaluate the capabilities of Large Language Models \(LLMs\) have been incorporated into the training dataset of the models\. This gives a false sense of performance that can overestimate how these models will function on truly unseen data\. This problem becomes worse with commercial LLMs with larger and non\-accessible training data, so techniques have been developed to try to measure the degree to which a model is contaminated with a benchmark’s data\. To understand the effectiveness of these techniques, particularly when evaluating contamination on coding tasks, we review trends and categorize techniques by the degree of access to the model that is required\. The research literature on this topic has reported mixed effectiveness of these techniques, so we select a set of black box \(text access only\) and grey box \(access to model loss/probabilities required\) techniques and apply them to both commercial and non\-commercial models\. We implement these metrics as part of a framework to test the contamination of Python code in LLMs to see to what extent we can replicate the effectiveness \(or ineffectiveness\) of these contamination detection techniques\. Though we find mixed results in the capabilities of these metrics to identify contamination, we do observe evidence that they can identify contamination \(broadly\) in fine\-tuned models when both a baseline and fine\-tuned model is present\. Additionally, similarity metrics were able to identify between contaminated and uncontaminated data even in situations where the data is distributionally similar \(e\.g\., drawn from the same set of code projects\)\.External Links:[Document](https://dx.doi.org/10.2172/2585515),[Link](https://www.osti.gov/biblio/2585515)Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Chenget al\.\(2025\)Y\. Cheng, Y\. Chang, and Y\. WuA survey on data contamination for large language models\.arXiv preprint arXiv:2502\.14425\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Chitaleet al\.\(2026\)P\. A\. Chitale, V\. Gumma, S\. Ahuja, P\. Kodali, M\. Uppadhyay, D\. Sudharsan, and S\. SitaramUPDESH: synthesizing grounded instruction tuning data for 13 indic languages\.External Links:2509\.21294,[Link](https://arxiv.org/abs/2509.21294)Cited by:[§4](https://arxiv.org/html/2610.36990#S4.p2.1)\.
- Chiuet al\.\(2025\)Y\. Y\. Chiu, L\. Jiang, B\. Y\. Lin, C\. Y\. Park, S\. S\. Li, S\. Ravi, M\. Bhatia, M\. Antoniak, Y\. Tsvetkov, V\. Shwartz, and Y\. ChoiCulturalBench: a robust, diverse and challenging benchmark for measuring LMs’ cultural knowledge through human\-AI red\-teaming\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 25663–25701\.External Links:[Link](https://aclanthology.org/2025.acl-long.1247/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1247),ISBN 979\-8\-89176\-251\-0Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.52.6),[§3\.2](https://arxiv.org/html/2610.36990#S3.SS2.p3.1)\.
- Clarket al\.\(2020\)J\. H\. Clark, E\. Choi, M\. Collins, D\. Garrette, T\. Kwiatkowski, V\. Nikolaev, and J\. PalomakiTyDi QA: a benchmark for information\-seeking question answering in typologically diverse languages\.Transactions of the Association for Computational Linguistics8,pp\. 454–470\.External Links:[Link](https://aclanthology.org/2020.tacl-1.30/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00317)Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1)\.
- Conneauet al\.\(2020\)A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. StoyanovUnsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th annual meeting of the association for computational linguistics,pp\. 8440–8451\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1)\.
- Conneauet al\.\(2018\)A\. Conneau, R\. Rinott, G\. Lample, A\. Williams, S\. Bowman, H\. Schwenk, and V\. StoyanovXNLI: evaluating cross\-lingual sentence representations\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2475–2485\.External Links:[Link](https://aclanthology.org/D18-1269/),[Document](https://dx.doi.org/10.18653/v1/D18-1269)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.23.6),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Cruz and Aji \(2026\)J\. C\. B\. Cruz and A\. F\. AjiLLM olympiad: why model evaluation needs a sealed exam\.arXiv preprint arXiv:2603\.23292\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Denget al\.\(2024\)Y\. Deng, W\. Zhang, S\. J\. Pan, and L\. BingMultilingual jailbreak challenges in large language models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=vESNKdEMGp)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.38.6)\.
- Dentanet al\.\(2025\)J\. Dentan, A\. Canesse, D\. Buscaldi, A\. Shabou, and S\. VanierMUCH: a multilingual claim hallucination benchmark\.arXiv preprint arXiv:2511\.17081\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.49.6)\.
- Devaneet al\.\(2025\)V\. Devane, M\. Nauman, B\. Patel, A\. M\. Wakchoure, Y\. Sant, S\. Pawar, V\. Thakur, A\. Godse, S\. Patra, N\. Maurya, S\. Racha, N\. K\. Singh, A\. Nagpal, P\. Sawarkar, K\. V\. Pundalik, R\. Saluja, and G\. RamakrishnanBhashaBench v1: a comprehensive benchmark for the quadrant of indic domains\.External Links:2510\.25409,[Link](https://arxiv.org/abs/2510.25409)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.51.6)\.
- Doddapaneniet al\.\(2023\)S\. Doddapaneni, R\. Aralikatte, G\. Ramesh, S\. Goyal, M\. M\. Khapra, A\. Kunchukuttan, and P\. KumarTowards leaving no indic language behind: building monolingual corpora, benchmark and models for indic languages\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 12402–12426\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.17.6)\.
- Doostmohammadi \(2026\)E\. DoostmohammadiToward understanding and enhancing the training and evaluation of language models: a study on vision, instruction tuning, and retrieval augmentation\.Vol\.2502,Linköping University Electronic Press\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1)\.
- Dussolleet al\.\(2025\)A\. Dussolle, A\. Cardeña Díaz, S\. Sato, and P\. DevineM\-IFEval: multilingual instruction\-following evaluation\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 6176–6191\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.344/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.344),ISBN 979\-8\-89176\-195\-7Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.48.6)\.
- Ebrahimiet al\.\(2022\)A\. Ebrahimi, M\. Mager, A\. Oncevay, V\. Chaudhary, L\. Chiruzzo, A\. Fan, J\. Ortega, R\. Ramos, A\. R\. Gonzales, I\. Meza\-Ruiz,et al\.AmericasNLI: evaluating zero\-shot natural language understanding of pretrained multilingual models in truly low\-resource languages\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 6279–6299\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1)\.
- Enevoldsenet al\.\(2025\)K\. Enevoldsen, I\. Chung, I\. Kerboua, M\. Kardos, A\. Mathur, D\. Stap, J\. Gala, W\. Siblini, D\. Krzeminski, G\. I\. Winata,et al\.MMTEB: massive multilingual text embedding benchmark\.InInternational Conference on Learning Representations,Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.2.6)\.
- Fuet al\.\(2025\)Y\. Fu, O\. Uzuner, M\. Yetisgen\-Yildiz, and F\. XiaDoes data contamination detection work \(well\) for llms? a survey and evaluation on detection assumptions\.InFindings of the Association for Computational Linguistics: NAACL 2025,pp\. 5235–5256\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Gainet al\.\(2025\)B\. Gain, D\. Bandyopadhyay, A\. Ekbal, and T\. N\. SinghBridging the linguistic divide: a survey on leveraging large language models for machine translation\.arXiv preprint arXiv:2504\.01919\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1)\.
- Golchin and Surdeanu \(2025\)S\. Golchin and M\. SurdeanuData contamination quiz: a tool to detect and estimate contamination in large language models\.Transactions of the Association for Computational Linguistics13,pp\. 809–830\.External Links:[Link](https://aclanthology.org/2025.tacl-1.37/),[Document](https://dx.doi.org/10.1162/tacl.a.20)Cited by:[§3\.3\.2](https://arxiv.org/html/2610.36990#S3.SS3.SSS2.p4.1)\.
- Goyalet al\.\(2022\)N\. Goyal, C\. Gao, V\. Chaudhary, P\. Chen, G\. Wenzek, D\. Ju, S\. Krishnan, M\. Ranzato, F\. Guzmán, and A\. FanThe flores\-101 evaluation benchmark for low\-resource and multilingual machine translation\.Transactions of the Association for Computational Linguistics10,pp\. 522–538\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.5.6)\.
- Grahamet al\.\(2020\)Y\. Graham, B\. Haddow, and P\. KoehnStatistical power and translationese in machine translation evaluation\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 72–81\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.6/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.6)Cited by:[§3\.2](https://arxiv.org/html/2610.36990#S3.SS2.p2.1)\.
- Guoet al\.\(2025\)Y\. Guo, S\. Conia, Z\. Zhou, M\. Li, S\. Potdar, and H\. XiaoDo large language models have an english accent? evaluating and improving the naturalness of multilingual llms\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3823–3838\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p1.1)\.
- Hadaet al\.\(2024a\)R\. Hada, V\. Gumma, M\. Ahmed, K\. Bali, and S\. SitaramMETAL: towards multilingual meta\-evaluation\.InFindings of the Association for Computational Linguistics: NAACL 2024,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 2280–2298\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.148/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.148)Cited by:[§3\.3\.1](https://arxiv.org/html/2610.36990#S3.SS3.SSS1.p1.1)\.
- Hadaet al\.\(2024b\)R\. Hada, V\. Gumma, A\. de Wynter, H\. Diddee, M\. Ahmed, M\. Choudhury, K\. Bali, and S\. SitaramAre large language model\-based evaluators the solution to scaling up multilingual evaluation?\.InFindings of the Association for Computational Linguistics: EACL 2024,Y\. Graham and M\. Purver \(Eds\.\),St\. Julian’s, Malta,pp\. 1051–1070\.External Links:[Link](https://aclanthology.org/2024.findings-eacl.71/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-eacl.71)Cited by:[§3\.3\.1](https://arxiv.org/html/2610.36990#S3.SS3.SSS1.p1.1)\.
- Hamnaet al\.\(2026\)H\. Hamna, G\. Bhat, S\. Mukherjee, F\. Lalani, E\. Hadfield, D\. Siddarth, K\. Bali, and S\. SitaramBuilding benchmarks from the ground up: community\-centered evaluation of llms in healthcare chatbot settings\.External Links:2509\.24506,[Link](https://arxiv.org/abs/2509.24506)Cited by:[§3\.2](https://arxiv.org/html/2610.36990#S3.SS2.p3.1),[§3\.3\.1](https://arxiv.org/html/2610.36990#S3.SS3.SSS1.p1.1)\.
- Hershcovichet al\.\(2022\)D\. Hershcovich, S\. Frank, H\. Lent, M\. de Lhoneux, M\. Abdou, S\. Brandl, E\. Bugliarello, L\. Cabello Piqueras, I\. Chalkidis, R\. Cui, C\. Fierro, K\. Margatina, P\. Rust, and A\. SøgaardChallenges and strategies in cross\-cultural NLP\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 6997–7013\.External Links:[Link](https://aclanthology.org/2022.acl-long.482/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.482)Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1)\.
- Houet al\.\(2026\)D\. X\. Hou, S\. Thapa, and L\. TayBridging cultures in the era of big data: a cross\-language equivalence framework in machine\-learning research with social media texts\.Advances in Methods and Practices in Psychological Science9\(1\),pp\. 25152459251398713\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1)\.
- Huanget al\.\(2025a\)C\. Huang, N\. Tashi, F\. Gao, Y\. Liu, J\. Li, H\. Tian, S\. Jiang, T\. Tsering, B\. Ma\-bao, R\. Duojie,et al\.Tibetan language and ai: a comprehensive survey of resources, methods and challenges\.arXiv preprint arXiv:2510\.19144\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1)\.
- Huanget al\.\(2025b\)X\. Huang, W\. Zhu, H\. Hu, C\. He, L\. Li, S\. Huang, and F\. YuanBenchMAX: a comprehensive multilingual evaluation suite for large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 16751–16774\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.909/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.909),ISBN 979\-8\-89176\-335\-7Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.20.6)\.
- Isbarovet al\.\(2025\)J\. Isbarov, A\. Akhundjanova, M\. Hajili, K\. Huseynova, D\. Gaynullin, A\. Rzayev, O\. Tursun, A\. Turdubaeva, I\. Saetov, R\. Kharisov, S\. Belginova, A\. Kenbayeva, A\. Alisheva, A\. Köksal, S\. Rustamov, and D\. AtamanTUMLU: a unified and native language understanding benchmark for Turkic languages\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 22816–22838\.External Links:[Link](https://aclanthology.org/2025.acl-long.1112/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1112),ISBN 979\-8\-89176\-251\-0Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.41.6),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.36990#S3.SS2.p3.1)\.
- Jacoviet al\.\(2023\)A\. Jacovi, A\. Caciularu, O\. Goldman, and Y\. GoldbergStop uploading test data in plain text: practical strategies for mitigating data contamination by evaluation benchmarks\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5075–5084\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.308/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.308)Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Jainet al\.\(2024\)D\. Jain, P\. Kumar, S\. Gehman, X\. Zhou, T\. Hartvigsen, and M\. SapPolygloToxicityPrompts: multilingual evaluation of neural toxic degeneration in large language models\.InFirst Conference on Language Modeling,Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.21.6)\.
- Jayakumaret al\.\(2026\)T\. Jayakumar, M\. S\. U\. R\. Khan, R\. Dabre, R\. Puduppully, and A\. KunchukuttanIndicIFEval: a benchmark for verifiable instruction\-following evaluation in 14 indic languages\.arXiv preprint arXiv:2602\.22125\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.24.6)\.
- Joshiet al\.\(2020\)P\. Joshi, S\. Santy, A\. Budhiraja, K\. Bali, and M\. ChoudhuryThe state and fate of linguistic diversity and inclusion in the NLP world\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 6282–6293\.External Links:[Link](https://aclanthology.org/2020.acl-main.560/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.560)Cited by:[§A\.1](https://arxiv.org/html/2610.36990#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.36990#S1.p2.1),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2610.36990#S3.SS1.p4.1)\.
- Kakwaniet al\.\(2020\)D\. Kakwani, A\. Kunchukuttan, S\. Golla, G\. NC, A\. Bhattacharyya, M\. M\. Khapra, and P\. KumarIndicNLPSuite: monolingual corpora, evaluation benchmarks and pre\-trained multilingual language models for indian languages\.InFindings of the association for computational linguistics: EMNLP 2020,pp\. 4948–4961\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.30.6),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1)\.
- Kanekoet al\.\(2026\)M\. Kaneko, A\. Niwa, and T\. BaldwinJailNewsBench: multi\-lingual and regional benchmark for fake news generation under jailbreak attacks\.InThe Fourteenth International Conference on Learning Representations,Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.16.6)\.
- Kargaranet al\.\(2025\)A\. H\. Kargaran, A\. Modarressi, N\. Nikeghbal, J\. Diesner, F\. Yvon, and H\. SchützeMexa: multilingual evaluation of english\-centric llms via cross\-lingual alignment\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 27001–27023\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p1.1)\.
- Kirchenbaueret al\.\(2023\)J\. Kirchenbauer, J\. Geiping, Y\. Wen, J\. Katz, I\. Miers, and T\. GoldsteinA watermark for large language models\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 17061–17084\.External Links:[Link](https://proceedings.mlr.press/v202/kirchenbauer23a.html)Cited by:[§3\.3\.2](https://arxiv.org/html/2610.36990#S3.SS3.SSS2.p5.1)\.
- KJet al\.\(2025\)S\. KJ, A\. Kumar, L\. Balaji, N\. Kotecha, V\. Jain, A\. Chadha, and S\. BhaduriIndicMMLU\-pro: benchmarking indic large language models on multi\-task language understanding\.External Links:2501\.15747,[Link](https://arxiv.org/abs/2501.15747)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.39.6)\.
- Kumaret al\.\(2022\)A\. Kumar, H\. Shrotriya, P\. Sahu, A\. Mishra, R\. Dabre, R\. Puduppully, A\. Kunchukuttan, M\. M\. Khapra, and P\. KumarIndicNLG benchmark: multilingual datasets for diverse NLG tasks in Indic languages\.InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 5363–5394\.External Links:[Link](https://aclanthology.org/2022.emnlp-main.360/),[Document](https://dx.doi.org/10.18653/v1/2022.emnlp-main.360)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.29.6)\.
- Kumaret al\.\(2024\)S\. Kumar, R\. Sanasam, and S\. NandiIndiSentiment140: sentiment analysis dataset for indian languages with emphasis on low\-resource languages using machine translation\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 7689–7698\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.15.6)\.
- Kumaret al\.\(2025\)S\. Kumar, V\. Balloli, M\. Ranjit, K\. Ahuja, S\. Sitaram, K\. Bali, T\. Ganu, and A\. NambiBridging the language gap: dynamic learning strategies for improving multilingual performance in llms\.InProceedings of the 31st International Conference on Computational Linguistics,pp\. 9209–9223\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1)\.
- Lachini \(2024\)K\. LachiniRethinking language assessment in the age of ai: a call for a paradigm shift\.Available at SSRN 5043405\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p1.1)\.
- Lamentillo \(2025\)A\. M\. LamentilloLinguistic pluralism as a core dimension of algorithmic fairness\.LSE International Development Review4\(2\)\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p1.1)\.
- Leonget al\.\(2023\)W\. Q\. Leong, J\. G\. Ngui, Y\. Susanto, H\. Rengarajan, K\. Sarveswaran, and W\. C\. TjhiBHASA: a holistic southeast asian linguistic and cultural evaluation suite for large language models\.External Links:2309\.06085Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.47.6)\.
- Liet al\.\(2025\)Y\. Li, Z\. Zhao, and C\. ScartonIt’s all about in\-context learning\! teaching extremely low\-resource languages to llms\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 29532–29547\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1)\.
- Lianget al\.\(2020\)Y\. Liang, N\. Duan, Y\. Gong, N\. Wu, F\. Guo, W\. Qi, M\. Gong, L\. Shou, D\. Jiang, G\. Cao, X\. Fan, R\. Zhang, R\. Agrawal, E\. Cui, S\. Wei, T\. Bharti, Y\. Qiao, J\. Chen, W\. Wu, S\. Liu, F\. Yang, D\. Campos, R\. Majumder, and M\. ZhouXGLUE: a new benchmark dataset for cross\-lingual pre\-training, understanding and generation\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 6008–6018\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.484/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.484)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.19.6),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2025\)Y\. Liu, Z\. Ma, X\. Jiang, J\. Hu, C\. ChangJing, and L\. LiMaXIFE: multilingual and cross\-lingual instruction following evaluation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 14252–14332\.External Links:[Link](https://aclanthology.org/2025.acl-long.698/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.698),ISBN 979\-8\-89176\-251\-0Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.14.6)\.
- Mackoet al\.\(2023\)D\. Macko, R\. Moro, A\. Uchendu, J\. Lucas, M\. Yamashita, M\. Pikuliak, I\. Srba, T\. Le, D\. Lee, J\. Simko,et al\.MULTITuDE: large\-scale multilingual machine\-generated text detection benchmark\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 9960–9987\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Maheshwariet al\.\(2025\)A\. Maheshwari, K\. Sharma, V\. Patel, and A\. MaheshwariIndicParam: benchmark to evaluate llms on low\-resource indic languages\.arXiv preprint arXiv:2512\.00333\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.31.6)\.
- McIntoshet al\.\(2025\)T\. R\. McIntosh, T\. Susnjak, N\. Arachchilage, T\. Liu, D\. Xu, P\. Watters, and M\. N\. HalgamugeInadequacies of large language model benchmarks in the era of generative artificial intelligence\.IEEE Transactions on Artificial Intelligence\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p2.1)\.
- Mergenet al\.\(2025\)A\. Mergen, N\. Çetin\-Kılıç, and M\. F\. ÖzbilginArtificial intelligence and bias towards marginalised groups: theoretical roots and challenges\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p1.1)\.
- Mhaskeet al\.\(2023\)A\. Mhaske, H\. Kedia, S\. Doddapaneni, M\. M\. Khapra, P\. Kumar, R\. Murthy, and A\. KunchukuttanNaamapadam: a large\-scale named entity annotated data for Indic languages\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 10441–10456\.External Links:[Link](https://aclanthology.org/2023.acl-long.582/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.582)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.33.6)\.
- Mushtaqet al\.\(2025\)A\. Mushtaq, I\. Taj, R\. Naeem, I\. Ghaznavi, and J\. QadirWorldView\-bench: a benchmark for evaluating global cultural perspectives in large language models\.arXiv preprint arXiv:2505\.09595\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p2.1)\.
- Niet al\.\(2025\)S\. Ni, G\. Chen, S\. Li, X\. Chen, S\. Li, B\. Wang, Q\. Wang, X\. Wang, Y\. Zhang, L\. Fan,et al\.A survey on large language model benchmarks\.arXiv preprint arXiv:2508\.15361\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p2.1)\.
- Ninget al\.\(2025\)Z\. Ning, T\. Gu, J\. Song, S\. Hong, L\. Li, H\. Liu, J\. Li, Y\. Wang, M\. Lingyu, Y\. Teng,et al\.Linguasafe: a comprehensive multilingual safety benchmark for large language models\.arXiv preprint arXiv:2508\.12733\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.26.6)\.
- Ohet al\.\(2025\)J\. Oh, I\. Cha, M\. Saxon, H\. Lim, S\. Bhatt, and A\. OhCulture is everywhere: a call for intentionally cultural evaluation\.arXiv preprint arXiv:2509\.01301\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p2.1)\.
- Ojewaleet al\.\(2026\)V\. Ojewale, I\. D\. Raji, and S\. VenkatasubramanianMulti\-lingual functional evaluation for large language models\.External Links:2506\.20793,[Link](https://arxiv.org/abs/2506.20793)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.43.6)\.
- Ojoet al\.\(2025\)J\. Ojo, O\. Ogundepo, A\. Oladipo, K\. Ogueji, J\. Lin, P\. Stenetorp, and D\. I\. AdelaniAfroBench: how good are large language models on African languages?\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 19048–19095\.External Links:[Link](https://aclanthology.org/2025.findings-acl.976/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.976),ISBN 979\-8\-89176\-256\-5Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.8.6),[§3\.2](https://arxiv.org/html/2610.36990#S3.SS2.p2.1)\.
- Parket al\.\(2025\)S\. Park, H\. Choi, M\. Kim, S\. An, X\. Wang, G\. Choi, and H\. KimFLUID QA: a multilingual benchmark for figurative language usage in dialogue across English, Chinese, and Korean\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 30280–30294\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1540/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1540),ISBN 979\-8\-89176\-332\-6Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.50.6)\.
- Plumet al\.\(2025\)A\. Plum, A\. Lutgen, C\. Purschke, and A\. RettingerIdentity\-aware large language models require cultural reasoning\.arXiv preprint arXiv:2510\.18510\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p2.1)\.
- Qinet al\.\(2025\)L\. Qin, Q\. Chen, Y\. Zhou, Z\. Chen, Y\. Li, L\. Liao, M\. Li, W\. Che, and P\. S\. YuA survey of multilingual large language models\.Patterns6\(1\)\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p2.1)\.
- Rastogiet al\.\(2025\)S\. Rastogi, P\. Maini, and D\. PruthiSTAMP your content: proving dataset membership via watermarked rephrasings\.InInternational Conference on Machine Learning,pp\. 51247–51272\.Cited by:[§3\.3\.2](https://arxiv.org/html/2610.36990#S3.SS3.SSS2.p3.1),[Table 1](https://arxiv.org/html/2610.36990#S3.T1)\.
- Ruderet al\.\(2023\)S\. Ruder, J\. H\. Clark, A\. Gutkin, M\. Kale, M\. Ma, M\. Nicosia, S\. Rijhwani, P\. Riley, J\. A\-Sarr, X\. Wang,et al\.Xtreme\-up: a user\-centric scarce\-data benchmark for under\-represented languages\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 1856–1884\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.6.6)\.
- Ruderet al\.\(2021\)S\. Ruder, N\. Constant, J\. Botha, A\. Siddhant, O\. Firat, J\. Fu, P\. Liu, J\. Hu, D\. Garrette, G\. Neubig, and M\. JohnsonXTREME\-R: towards more challenging and nuanced multilingual evaluation\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,M\. Moens, X\. Huang, L\. Specia, and S\. W\. Yih \(Eds\.\),Online and Punta Cana, Dominican Republic,pp\. 10215–10245\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.802/),[Document](https://dx.doi.org/10.18653/v1/2021.emnlp-main.802)Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.10.6),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Sainzet al\.\(2023\)O\. Sainz, J\. Campos, I\. García\-Ferrero, J\. Etxaniz, O\. L\. de Lacalle, and E\. AgirreNLP evaluation in trouble: on the need to measure LLM data contamination for each benchmark\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 10776–10787\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.722/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.722)Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px3.p1.1)\.
- Sharmaet al\.\(2026\)T\. Sharma, M\. Ravikiran, S\. K\. Behera, P\. Bhattacharya, A\. Bhattacharya, and R\. SalujaINDIC dialect: a multi task benchmark to evaluate and translate in indian language dialects\.arXiv preprint arXiv:2601\.10388\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.25.6)\.
- Siddhantet al\.\(2020\)A\. Siddhant, J\. Hu, M\. Johnson, O\. Firat, and S\. RuderXtreme: a massively multilingual multi\-task benchmark for evaluating cross\-lingual generalization\.InProceedings of the International Conference on Machine Learning,Vol\.2020,pp\. 4411–4421\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.11.6)\.
- Singhet al\.\(2024\)S\. Singh, F\. Vargus, D\. D’souza, B\. F\. Karlsson, A\. Mahendiran, W\. Ko, H\. Shandilya, J\. Patel, D\. Mataciunas, L\. O’Mahony,et al\.Aya dataset: an open\-access collection for multilingual instruction tuning\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 11521–11567\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.4.6)\.
- Sinhaet al\.\(2025\)S\. R\. Sinha, R\. Sheth, A\. Upperwal, and M\. SinghEka\-eval: an evaluation framework for low\-resource multilingual large language models\.arXiv preprint arXiv:2507\.01853\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p2.1)\.
- Smith \(2025\)O\. SmithCultural contexts in english language teaching: balancing global standards with local relevance\.IOSR Journal of Humanities and Social Science30\(10\),pp\. 16–28\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p2.1)\.
- Songet al\.\(2025\)S\. Song, S\. Jeong, E\. Kim, J\. Jin, D\. Kim, J\. Shin, and A\. OhMUG\-eval: a proxy evaluation framework for multilingual generation capabilities in any language\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 19488–19514\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.1061/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1061),ISBN 979\-8\-89176\-335\-7Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.12.6)\.
- Susantoet al\.\(2025\)Y\. Susanto, A\. V\. Hulagadri, J\. R\. Montalan, J\. G\. Ngui, X\. Yong, W\. Q\. Leong, H\. Rengarajan, P\. Limkonchotiwat, Y\. Mai, and W\. C\. TjhiSEA\-HELM: Southeast Asian holistic evaluation of language models\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 12308–12336\.External Links:[Link](https://aclanthology.org/2025.findings-acl.636/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.636),ISBN 979\-8\-89176\-256\-5Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.46.6)\.
- Upadhayay \(2025\)B\. UpadhayayEfficient and robust language adaptation in multilingual llms\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p1.1)\.
- Van Dorenet al\.\(2026\)M\. Van Doren, C\. Ford, J\. Barajas, and C\. Holland” Be my cheese?”: cultural nuance benchmarking for machine translation in multilingual llms\.arXiv preprint arXiv:2602\.04729\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.22.6)\.
- Vermaet al\.\(2025\)S\. Verma, M\. S\. U\. R\. Khan, V\. Kumar, R\. Murthy, and J\. SenMILU: a multi\-task Indic language understanding benchmark\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 10076–10132\.External Links:[Link](https://aclanthology.org/2025.naacl-long.507/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.507),ISBN 979\-8\-89176\-189\-6Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.32.6)\.
- Wattset al\.\(2024\)I\. Watts, V\. Gumma, A\. Yadavalli, V\. Seshadri, M\. Swaminathan, and S\. SitaramPARIKSHA: a large\-scale investigation of human\-LLM evaluator agreement on multilingual and multi\-cultural data\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 7900–7932\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.451/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.451)Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1),[§3\.3\.1](https://arxiv.org/html/2610.36990#S3.SS3.SSS1.p1.1)\.
- Weidingeret al\.\(2022\)L\. Weidinger, J\. Uesato, M\. Rauh, C\. Griffin, P\. Huang, J\. Mellor, A\. Glaese, M\. Cheng, B\. Balle, A\. Kasirzadeh,et al\.Taxonomy of risks posed by language models\.InProceedings of the 2022 ACM conference on fairness, accountability, and transparency,pp\. 214–229\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p1.1)\.
- Wendleret al\.\(2024\)C\. Wendler, V\. Veselovsky, G\. Monea, and R\. WestDo llamas work in English? on the latent language of multilingual transformers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15366–15394\.External Links:[Link](https://aclanthology.org/2024.acl-long.820/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.820)Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1)\.
- Winataet al\.\(2023\)G\. I\. Winata, A\. F\. Aji, S\. Cahyawijaya, R\. Mahendra, F\. Koto, A\. Romadhony, K\. Kurniawan, D\. Moeljadi, R\. E\. Prasojo, P\. Fung, T\. Baldwin, J\. H\. Lau, R\. Sennrich, and S\. RuderNusaX: multilingual parallel sentiment dataset for 10 Indonesian local languages\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,A\. Vlachos and I\. Augenstein \(Eds\.\),Dubrovnik, Croatia,pp\. 815–834\.External Links:[Link](https://aclanthology.org/2023.eacl-main.57/),[Document](https://dx.doi.org/10.18653/v1/2023.eacl-main.57)Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2025\)M\. Wu, W\. Wang, S\. Liu, H\. Yin, X\. Wang, Y\. Zhao, C\. Lyu, L\. Wang, W\. Luo, and K\. ZhangThe bitter lesson learned from 2,000\+ multilingual benchmarks\.arXiv preprint arXiv:2504\.15521\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p2.1)\.
- Xuanet al\.\(2025\)W\. Xuan, R\. Yang, H\. Qi, Q\. Zeng, Y\. Xiao, A\. Feng, D\. Liu, Y\. Xing, J\. Wang, F\. Gao,et al\.Mmlu\-prox: a multilingual benchmark for advanced large language model evaluation\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 1513–1532\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.13.6)\.
- Yanget al\.\(2025\)D\. Yang, D\. Hovy, D\. Jurgens, and B\. PlankSocially aware language technologies: perspectives and practices\.Computational Linguistics51,pp\. 689–703\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p2.1)\.
- Yaoet al\.\(2024\)F\. Yao, Y\. Zhuang, Z\. Sun, S\. Xu, A\. Kumar, and J\. ShangData contamination can cross language barriers\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 17864–17875\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.990/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.990)Cited by:[Table 2](https://arxiv.org/html/2610.36990#A2.T2),[Table 3](https://arxiv.org/html/2610.36990#A2.T3),[§3\.3\.2](https://arxiv.org/html/2610.36990#S3.SS3.SSS2.p2.1),[Table 1](https://arxiv.org/html/2610.36990#S3.T1)\.
- Ye \(2025\)H\. YeMultilinguality and inclusive language technologies for low\-resource languages\.Ph\.D\. Thesis,lmu\.Cited by:[§2](https://arxiv.org/html/2610.36990#S2.SS0.SSS0.Px1.p1.1)\.
- Zahraei and Asgari \(2025\)P\. S\. Zahraei and E\. AsgariI am aligned, but with whom? mena values benchmark for evaluating cultural alignment and multilingual bias in llms\.arXiv preprint arXiv:2510\.13154\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p2.1)\.
- Zhanget al\.\(2026\)L\. Zhang, Y\. Zhang, D\. Long, P\. Xie, M\. Zhang, J\. Li, and M\. ZhangProgressive adaptation of large language models for multilingual text ranking\.ACM Transactions on Information Systems\.Cited by:[§1](https://arxiv.org/html/2610.36990#S1.p1.1)\.
- Zhanget al\.\(2023\)W\. Zhang, M\. Aljunied, C\. Gao, Y\. K\. Chia, and L\. BingM3exam: a multilingual, multimodal, multilevel benchmark for examining large language models\.Advances in Neural Information Processing Systems36,pp\. 5484–5505\.Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.40.6)\.
- Zhaoet al\.\(2025\)R\. Zhao, B\. Chen, B\. Plank, and M\. A\. HedderichMAKIEval: a multilingual automatic WiKidata\-based framework for cultural awareness evaluation for LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 23104–23136\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.1256/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1256),ISBN 979\-8\-89176\-335\-7Cited by:[Table 8](https://arxiv.org/html/2610.36990#A3.T8.2.1.27.6)\.

## Appendix AAppendix

### A\.1Data Extraction Methodology

We survey 51 multilingual evaluation benchmarks and extract structured metadata at dataset\-level granularity through a multi\-step pipeline\. For each benchmark/dataset paper, Claude Opus 4\.6 extracts the constituent datasets, their member languages, and three key properties per dataset: whether the evaluation content was translated from another language \(typically English\), whether native speakers were involved in annotation, and whether the dataset is culturally grounded in the target locale rather than reflecting source\-culture assumptions\. The extraction operates at a three\-level hierarchy—benchmark→\\rightarrowdataset→\\rightarrowlanguage\. Language metadata is enriched deterministically from authoritative sources: family and macroarea, primary script from Glottolog888https://glottolog\.org/meta/downloads, and resource classification \(levels 0–5\) from[Joshi et al\. \(2020\)](https://arxiv.org/html/2610.36990#bib.bib107)’s taxonomy\. To avoid inflated counts from shared datasets appearing across multiple aggregation benchmarks, we deduplicate at the \(dataset, language\) level while retaining benchmark provenance\. Each property is assessed at the dataset level with an associated confidence rating \(high, medium, or low\) and a supporting evidence passage extracted from the paper\. These produced assessments are then surfaced in a web\-based human review interface \(Figure[5](https://arxiv.org/html/2610.36990#A1.F5)\), where reviewers inspect the extracted values and evidence, verify or correct language membership through interactive toggles, and provide final judgments via structured dropdowns\. Certain datasets are automatically flagged for mandatory review due to missing or uncertain property values \(e\.g\., unknown native annotator status\), indicated by coloredneeds reviewbadges\.

![Refer to caption](https://arxiv.org/html/2610.36990v1/plots_figures/webpage_images/annotate2.png)Figure 5:Human review interface showing extracted dataset properties with evidence passages and confidence ratings, alongside structured dropdowns for reviewer adjudication of translation status, native annotator involvement, and cultural grounding\.
### A\.2Geographic Distribution

Figure[6](https://arxiv.org/html/2610.36990#A1.F6)maps languages to macro\-regions\. Sub\-Saharan Africa leads in raw language count \(62 languages, 28%, driven by dedicated datasets\), followed by Europe \(45, 21%\) and South Asia \(43, 20%\)\. The Americas \(14\) and Central Asia \(9\) have gained modest representation through recent datasets, but Oceania remains near\-zero \(2 languages, 1%\)—a region home to over 1,400 living languages that remain almost entirely outside the evaluation perimeter\.

Figure 6:Number of unique languages covered by world region\.Takeaway:Coverage is concentrated in Sub\-Saharan Africa, Europe, and South Asia; Oceania remains near\-zero\.Figure 7:Resource\-level distribution of all evaluated languages\.Takeaway:65% of evaluated languages fall in the two lowest resource classes \(levels 0–1\), yet depth of evaluation remains concentrated in high\-resource languages\.
### A\.3Resource\-Level Distribution

Figure[7](https://arxiv.org/html/2610.36990#A1.F7)shows the resource\-level composition of all evaluated languages, complementing the task equity analysis in the section[3\.2](https://arxiv.org/html/2610.36990#S3.SS2)\(Figure[3](https://arxiv.org/html/2610.36990#S3.F3)\)\.

![Refer to caption](https://arxiv.org/html/2610.36990v1/06_task_by_region.png)Figure 8:Task category coverage by world region \(number of datatset evaluating each task\)\.Takeaway:QA/Knowledge dominates everywhere; specialized tasks \(safety, reasoning, coding\) are unevenly distributed, with Central Asia and MENA consistently thinnest\.
### A\.4Dataset Source Type Breakdown

Figure[9](https://arxiv.org/html/2610.36990#A1.F9)breaks down the 242 unique datasets by construction method\. While 128 datasets \(53%\) are originally authored, 85 \(35%\) are translated from English sources\. The remaining datasets are either adapted from existing resources \(15, 6%\) or use a mix of methods \(14, 6%\)\. This means that over a third of the evaluation data that multilingual models are tested on originates from English, carrying the associated risks of translationese artifacts and cultural misalignment\.

Figure 9:Distribution of dataset construction methods across 242 unique datasets\.35% of datasets are translated from English; only 53% are originally authored in the target language\(s\)\.
### A\.5Translation Status by Resource Level

Figure[10](https://arxiv.org/html/2610.36990#A1.F10)reveals how translation status interacts with resource level\. Across all resource levels, roughly 35–40% of dataset–language instances involve native data\. However, the composition shifts at the extremes: Left\-Behinds \(n=178n=178\) have the lowest native share \(∼20%\{\\sim\}20\\%\) and are overwhelmingly evaluated through translated datasets, while Winners \(n=360n=360\) maintain a higher native proportion \(∼40%\{\\sim\}40\\%\) but still rely heavily on translated data\. This pattern suggests that the most under\-served languages are also the most likely to be evaluated using data that does not originate from their speech communities\.

Figure 10:Translation status of dataset–language instances by Joshi resource level\.Takeaway:Left\-Behinds have the lowest native data share; the most under\-resourced languages are predominantly evaluated through translated datasets\.
### A\.6Native Annotators by Region

Figure[11](https://arxiv.org/html/2610.36990#A1.F11)examines whether dataset annotations were produced by native speakers of the target language, broken down by region\. Central Asia and South Asia have the highest native\-annotator share \(∼50%\{\\sim\}50\\%\), driven by dedicated regional datasets/benchmarks \(e\.g\., IndicGenBench, INCLUDE\)\. Sub\-Saharan Africa \(n=543n=543\) shows∼45%\{\\sim\}45\\%native annotators but a large “Unknown” fraction, reflecting that many aggregation benchmarks do not document annotator provenance\. The Americas \(n=23n=23\) have the lowest native share, consistent with minimal community\-driven dataset development for indigenous languages\.

Figure 11:Native vs\. non\-native annotator status by region\.Takeaway:Annotator provenance is under\-documented; where reported, South Asia and Central Asia lead in native\-speaker annotation, while the Americas lag\.
### A\.7Model Release Transparency Audit

Section[3\.3\.3](https://arxiv.org/html/2610.36990#S3.SS3.SSS3)reports summary statistics from our audit of 23 recent model releases\. Here we describe the methodology and provide the detailed per\-model breakdown\.

Figure 12:Multilingual evaluation transparency across 23 recent model releases \(15 open\-weight, 8 closed\)\.Left:Reporting rate across five transparency dimensions—only 65% report any multilingual evaluation, and just 13% report contamination results\.Right:Breakdown by model type—closed models disclose significantly less across all dimensions\.##### Methodology

We examine technical reports, model cards, and blog posts for 23 model releases spanning 2024–2026 \(15 open\-weight, 8 closed\-models\)\. Each release is scored on five binary transparency dimensions: \(1\) whether training language composition is explicitly disclosed, \(2\) whether any multilingual evaluation datasets is reported, \(3\) whether the specific evaluation languages are listed, \(4\) whether a contamination detection technique is named, and \(5\) whether contamination results are reported\. The composite transparency score is the sum \(0–5\)\.

##### Detailed Findings

##### Models with sparse multilingual evaluation evidence

include GPT, Grok, Mistral 3, BharatGen, Command\-R\+, OLMo, and Kimi K2\.5\. Kimi K2\.5, for example, is described by its authors as “very English centric\.” Similarly, while Sarvam is trained on 22 Indian languages, its reported evaluation appears to center on a newly introduced dataset/benchmark scored using an LLM judge, without corresponding results on other widely used multilingual datasets\. Consequently, the multilingual capabilities of these releases remain difficult to verify comprehensively, given the limited publicly reported evaluation evidence\.

##### Contamination testing

is reported by only 3 models: Llama\-3 \(ConTAM\), Gemma \(quality reweighting\), and GPT\-OSS \(TroubleshootingBench\)\. All are open\-weight\. Even these three use techniques designed for English benchmarks; cross\-lingual contamination pathways like translated dataset leakage, parallel corpus overlap, instruction\-tuning propagation go unexamined\.

##### Datasets concentration:

INCLUDE, FLORES\-200, and MGSM are the three most frequently cited multilingual datasets, each appearing in only 4 of 23 reports\. Natively authored, culturally grounded datasets like AfroBench, IrokoBench, MILU, CulturalBench, TUMLU are almost never cited\.

##### Open weight vs\. closed:

Open\-weight models average 2\.47/5 on the transparency score vs\. 0\.75/5 for closed models\. The gap is largest on training language disclosure \(60% vs\. 0%\), evaluation language listing \(73% vs\. 12%\), and contamination reporting \(20% vs\. 0%\)\. The Aya model family \(Aya\-23, Aya\-Expanse, Tiny Aya\) stands out as the transparency exemplar, reporting multilingual datasets across 23–70 languages\. However, even the most transparent open\-weight models rarely report contamination testing for multilingual datasets\.

##### Multilingual safety evaluation

is reported by only 4 of 23 models: Tiny Aya, Claude Opus 4\.6, Claude Sonnet 4\.6, and CommandA\. All other models either test safety only in English or do not report safety testing at all\.

## Appendix BContamination Detection Results

Llama\-3\.1\-8BLlama\-3\.1\-8B\-itMistral\-7B\-v0\.3Mistral\-7B\-v0\.3\-itGemma\-2\-9B\-itGemma\-2\-9BAya\-23\-8BGemma\-7B\-itLlama\-2\-7B\-itMistral\-7B\-v0\.1\-itFlores✗✗✗✗✗✗✗–––Paws\-X✗✗✗✗✗✗✓✗✗✗XCopa✗✗✗✗✗✗✓✗✗✗XLSum✓✓✗✗✗✗✗–––Xnli✗✗✗✗✗✗✗✓✓✓XQuad✗✗✗✗✗✗✗✓✗✗XStoryCloze✗✗✗✗✗✗✗✓✓✓

Table 2:Benchmark contamination presence across all evaluated models based on[Ahuja et al\. \(2024a\)](https://arxiv.org/html/2610.36990#bib.bib82),[Ahuja et al\. \(2024b\)](https://arxiv.org/html/2610.36990#bib.bib58), and[Yao et al\. \(2024\)](https://arxiv.org/html/2610.36990#bib.bib86)\.✗=contaminated,✓=not contaminated, – = not evaluated\.Phi2Phi3Phi3\.5\-miniPhi3\.5\-MoEGrinMoEAbel\-7BLLaMA2MistralQwen2GLM4LLaMA3Reflection2\.7B3\.8B3\.8B3\.8B×\\times163\.8B×\\times167B7B7B7B9B70B70BMMLU23\.8367\.2768\.6476\.6277\.5547\.0844\.8857\.2969\.0567\.3678\.5575\.83MMLU\-g25\.0285\.2987\.0091\.6592\.8368\.3772\.8782\.7189\.2384\.9192\.1788\.37difference1\.2018\.0218\.3615\.0315\.2821\.2927\.9925\.4220\.1817\.5513\.6212\.54ARC\-C42\.9280\.2059\.5654\.7863\.5750\.3436\.1864\.0884\.8186\.3561\.5256\.74ARC\-C\-g47\.2792\.1593\.9496\.5096\.2566\.0444\.7185\.7595\.2291\.8195\.9994\.45difference4\.3511\.9534\.3841\.7232\.6815\.708\.5321\.6710\.415\.4634\.4737\.71MathQA31\.3241\.1441\.1437\.4247\.6734\.3028\.7136\.8844\.3643\.0556\.5258\.29MathQA\-g38\.7049\.0647\.3843\.9656\.2735\.7136\.1845\.7749\.0356\.0461\.8463\.92difference7\.387\.926\.246\.548\.601\.417\.478\.894\.6712\.995\.325\.63

Table 3:Detecting inadvertent contamination in popular open\-source LLMs from[Yao et al\. \(2024\)](https://arxiv.org/html/2610.36990#bib.bib86)\. Each benchmark is tested in its original form and a paraphrased variant \(\-g\); thedifferencerow shows the generalizability gap\.Boldvalues indicate significantly lower generalizability, implying potential contamination\.ModelPAW\-SXGlobal MMLUIndicParamMILUMMMLUINCLUDEFLORESMGSMAFRiMGSMOverallAya\-Expanse\-32B100% \(4/4\)0% \(0/42\)0% \(0/6\)0% \(0/11\)0% \(0/14\)0% \(0/44\)4% \(1/28\)55% \(6/11\)16% \(3/19\)7\.8%Gemma\-3\-12B50% \(2/4\)12% \(5/40\)0% \(0/7\)0% \(0/11\)14% \(2/14\)5% \(2/44\)0% \(1/204\)55% \(6/11\)16% \(3/19\)5\.9%Tiny\-Aya\-Global100% \(4/4\)0% \(0/42\)0% \(0/12\)0% \(0/11\)0% \(0/14\)0% \(0/44\)2% \(5/204\)55% \(6/11\)26% \(5/19\)5\.5%Aya\-Expanse\-8B75% \(3/4\)0% \(0/42\)10% \(1/10\)0% \(0/11\)0% \(0/14\)0% \(0/44\)1% \(2/204\)64% \(7/11\)26% \(5/19\)5\.0%Qwen3\-4B100% \(4/4\)0% \(0/42\)8% \(1/12\)0% \(0/11\)0% \(0/14\)0% \(0/44\)0% \(1/204\)36% \(4/11\)21% \(4/19\)3\.9%Phi\-425% \(1/4\)0% \(0/42\)8% \(1/12\)0% \(0/11\)0% \(0/14\)2% \(1/44\)1% \(2/204\)18% \(2/11\)21% \(4/19\)3\.0%Llama\-3\.1\-8B25% \(1/4\)0% \(0/42\)0% \(0/10\)0% \(0/11\)0% \(0/14\)0% \(0/44\)1% \(3/204\)36% \(4/11\)20% \(2/10\)2\.9%Overall67\.9%1\.7%4\.3%0\.0%2\.0%1\.0%1\.2%45\.5%21\.0%4\.6%

Table 4:Contamination rate per model and dataset\. Each cell shows the fraction of languages flagged as contaminated \(p<0\.05p<0\.05\), with the count in parentheses\.Boldindicates the dataset is contaminated as a whole \(Fisher’s combinedp<0\.05p<0\.05\)\.### B\.1Data Contamination Quiz Results

We apply the Data Contamination Quiz \(DCQ\)\( \)to 85 dataset–language partitions, with 100 instances per partition\. Table[5](https://arxiv.org/html/2610.36990#A2.T5)reports the maximum recognition rate, averaged over languages within each dataset\. Table[6](https://arxiv.org/html/2610.36990#A2.T6)compares English partitions with the mean over non\-English partitions where both are available\. Because each quiz has five options, 20% is the random\-chance floor\. These are recognition rates rather than estimates of the fraction of training data contaminated\.

Eight partitions have perturbation distance below 10%, making their quizzes close to unsolvable and their low scores uninformative\. The bias\-detection abstention rate is also near zero for most models, indicating that models often guess rather than select “none of these\.” A byte\-identical replicate of the English GSM8K items under the AfriMGSM and MGSM dataset names produces a mean absolute difference of 9\.7 percentage points and a maximum difference of 23 points across models\. Individual DCQ values should therefore be treated as approximate\.

ModelAfriMGSMGlobal MMLUINCLUDEIndicParamMGSMMILUMMMLUPAWS\-XAya\-Expanse\-32B21\.328\.729\.716\.247\.427\.031\.854\.4Aya\-Expanse\-8B10\.812\.611\.58\.722\.313\.612\.438\.9Tiny\-Aya\-Global27\.521\.019\.115\.134\.616\.722\.811\.4Gemma\-3\-12B39\.421\.540\.636\.056\.636\.619\.741\.3Llama\-3\.1\-8B9\.28\.711\.19\.917\.99\.87\.333\.2Phi\-42\.322\.610\.613\.210\.624\.636\.820\.6Qwen3\-4B18\.17\.79\.33\.826\.813\.76\.528\.2Mean18\.417\.518\.814\.730\.921\.719\.632\.6

Table 5:DCQ maximum recognition rate \(%\) by model and benchmark, averaged over each benchmark’s language partitions\. The random\-chance floor is 20%\. The maximum rate selects the best raw hit rate across the tested answer positions and is therefore an optimistic estimate\.BenchmarkEnglishNon\-English MeanAfriMGSM45\.7%15\.3%Global MMLU26\.1%16\.6%MGSM44\.0%29\.6%MILU37\.9%18\.6%PAWS\-X47\.6%30\.1%Table 6:DCQ maximum recognition rate averaged across the seven models for English and non\-English language partitions\. INCLUDE, IndicParam, and MMMLU are omitted because the evaluated partitions do not include English\.TypeNameRelease DateModelAya\-Expanse\-32BDecember 2024Aya\-Expanse\-8BDecember 2024Tiny\-Aya\-GlobalFebruary 2026Gemma\-3\-12BMarch 2025Llama\-3\.1\-8BJuly 2024Phi\-4December 2024Qwen3\-4BApril 2025DatasetsPAW\-SXNovember 2019Global MMLUDecember 2024IndicParamNovember 2025MILUNovember 2024MMMLUSeptember 2020INCLUDENovember 2024FLORESJune 2021MGSMOctober 2022AFRiMGSMMay 2024Table 7:Release dates of models and datasets/benchmarks used in this study\.

## Appendix CReleased Resources and Interactive Dashboard

To facilitate community engagement with these findings, we will release three resources upon acceptance\. First, we will provide thestructured metadata annotationsfor all 51 benchmarks and 242 datasets, including per\-language fields for script, language family, resource level, task categories, translation status, native annotator provenance, and cultural grounding\. Second, we will release thefull analysis codebaseused to produce all figures, tables, and statistics in this paper, enabling researchers to reproduce our analysis and extend it as new datasets appear\. Third, we provide aninteractive web dashboardthat allows users to explore coverage and representativeness statistics, browse individual datasets via a searchable explorer, and look up per\-language dataset coverage\. The dashboard is designed to be a living resource that will be updated as new datasets are added to the survey\.

The dashboard provides four main views:

- •Coverage Analysis: Interactive visualisations of language, family, script, regional, resource\-level, and task\-type distributions, mirroring and extending the figures in this paper\. Users can click on individual bars to drill down into which datasets cover specific languages \(Figure[14](https://arxiv.org/html/2610.36990#A3.F14)\)\.
- •Representativeness: Translation status and cultural grounding breakdowns by region and resource level\.
- •Benchmark Explorer: A searchable, sortable table of all 51 benchmarks with expandable rows showing paper links, descriptions, language lists, task categories, and language family coverage \(Figure[15](https://arxiv.org/html/2610.36990#A3.F15)\)\.
- •Language Lookup: A per\-language search interface that returns all datasets/benchmarks covering a given language, with metadata on translation status and task types\.

![Refer to caption](https://arxiv.org/html/2610.36990v1/plots_figures/webpage_images/landing.png)Figure 13:Landing page of the interactive dashboard showing summary statistics \(51 benchmarks, 219 languages, 29 families, 29 scripts, 10 regions, 15 task types\) and navigation to the four main views: Coverage, Representativeness, Benchmark Explorer, and Language Lookup\.![Refer to caption](https://arxiv.org/html/2610.36990v1/plots_figures/webpage_images/coverage.png)Figure 14:Coverage Analysis view with interactive drill\-down\. Clicking on a bar in the language long\-tail chart opens a panel listing the specific benchmarks that cover languages appearing in exactly that number of benchmarks, along with matching languages for each benchmark\.![Refer to caption](https://arxiv.org/html/2610.36990v1/plots_figures/webpage_images/benchmak_explorer.png)Figure 15:Benchmark Explorer view\. Users can search, sort by column, and filter by region\. Expanding a row reveals the paper reference, description, year, benchmark type, task categories, and full language family breakdown\.\#BenchmarkTask CategoriesYearVenueCitation1MMTEBClassification / QA / Knowledge \(550 tasks\)2025ICLR\([Enevoldsen et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib14)\)2BelebeleQA / Knowledge2024ACL\([Bandarkar et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib13)\)3Aya\_Evaluation\_SuiteGeneration2024ACL\([Singh et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib12)\)4FLORES\-101Translation2022TACL\([Goyal et al\., 2022](https://arxiv.org/html/2610.36990#bib.bib11)\)5XTREME\_UPLinguistic Probing, Other, QA / Knowledge, …2023ACL\([Ruder et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib10)\)6MEGANER, NLI / Inference, QA / Knowledge, …2023EMNLP\([Ahuja et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib89)\)7AfroBenchLinguistic Probing, Math / Reasoning, NER, …2025ACL\([Ojo et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib76)\)8BUFFETNER, NLI / Inference, QA / Knowledge, …2024NAACL\([Asai et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib29)\)9XTREME\-REmbedding / Retrieval, Linguistic Probing, …2021EMNLP\([Ruder et al\., 2021](https://arxiv.org/html/2610.36990#bib.bib101)\)10XTREMEEmbedding / Retrieval, NLI / Inference, …2020ICML\([Siddhant et al\., 2020](https://arxiv.org/html/2610.36990#bib.bib28)\)11MUG\_EvalCoding, Instruction Following, Math / Reasoning2025EMNLP\([Song et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib73)\)12MMLU\-ProXQA / Knowledge2025EMNLP\([Xuan et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib27)\)13MaXIFEInstruction Following2025ACL\([Liu et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib79)\)14IndiSentiment140Sentiment / Classification2024NAACL\([Kumar et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib26)\)15JailNewsBenchSafety / Toxicity2026ICLR\([Kaneko et al\., 2026](https://arxiv.org/html/2610.36990#bib.bib25)\)16IndicXTREMEEmbedding / Retrieval, NER, NLI / Inference, …2023ACL\([Doddapaneni et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib24)\)17IrokoBenchMath / Reasoning, NLI / Inference, QA / Knowledge2025NAACL\([Adelani et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib69)\)18XGLUEEmbedding / Retrieval, Linguistic Probing, …2020EMNLP\([Liang et al\., 2020](https://arxiv.org/html/2610.36990#bib.bib104)\)19BenchMAXCoding, Instruction Following, …2025EMNLP\([Huang et al\., 2025b](https://arxiv.org/html/2610.36990#bib.bib72)\)20PTPSafety / Toxicity2024COLM\([Jain et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib23)\)21BeMyCheeseSentiment / Classification2026ArXiv\([Van Doren et al\., 2026](https://arxiv.org/html/2610.36990#bib.bib22)\)22XNLINLI / Inference2018EMNLP\([Conneau et al\., 2018](https://arxiv.org/html/2610.36990#bib.bib108)\)23IndicIFEvalInstruction Following2026ArXiv\([Jayakumar et al\., 2026](https://arxiv.org/html/2610.36990#bib.bib21)\)24INDIC\-DIALECTQA / Knowledge, Sentiment / Classification, …2026ArXiv\([Sharma et al\., 2026](https://arxiv.org/html/2610.36990#bib.bib20)\)25LinguaSafeSafety / Toxicity2025ArXiv\([Ning et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib19)\)26MAKIEVALSentiment / Classification2025EMNLP\([Zhao et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib74)\)27GSM8K\-IndicMath / Reasoning2025—sarvamai/gsm8k\-indic28IndicNLGGeneration, NLI / Inference, QA / Knowledge, …2022EMNLP\([Kumar et al\., 2022](https://arxiv.org/html/2610.36990#bib.bib96)\)29IndicNLPSuiteEmbedding / Retrieval, NER, NLI / Inference, …2020EMNLP\([Kakwani et al\., 2020](https://arxiv.org/html/2610.36990#bib.bib54)\)30IndicParamQA / Knowledge2025ArXiv\([Maheshwari et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib18)\)31MILUQA / Knowledge2025NAACL\([Verma et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib70)\)32NaamapadamNER2023ACL\([Mhaske et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib93)\)33TriviaQA\-IndicQA / Knowledge2025—sarvamai/trivia\-qa\-indic\-mcq34XQuADQA / Knowledge2020ACL\([Artetxe et al\., 2020b](https://arxiv.org/html/2610.36990#bib.bib106)\)35BoolQ\-IndicQA / Knowledge2025—sarvamai/boolq\-indic36MMLU\-IndicQA / Knowledge2025—sarvamai/mmlu\-indic37MultiJailSafety / Toxicity2024ICLR 2024\([Deng et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib67)\)38IndicMMLU\-ProQA / Knowledge2025ArXiv\([KJ et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib66)\)39M3ExamQA / Knowledge2023NeurIPS 2023\([Zhang et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib55)\)40TUMLUQA / Knowledge2025ACL\([Isbarov et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib80)\)41XORQAEmbedding / Retrieval2021NAACL\([Asai et al\., 2021](https://arxiv.org/html/2610.36990#bib.bib100)\)42CL\-IFEvalInstruction Following, Math / Reasoning2025ArXiv\([Ojewale et al\., 2026](https://arxiv.org/html/2610.36990#bib.bib65)\)43UhuraQA / Knowledge2024ArXiv\([Bayes et al\., 2024](https://arxiv.org/html/2610.36990#bib.bib64)\)44IberoBenchLinguistic Probing, Math / Reasoning, …2025COLING 2025\([Baucells et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib78)\)45SeaHELMGeneration, Instruction Following, …2025Findings of ACL 2025\([Susanto et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib75)\)46BHASALinguistic Probing, Math / Reasoning, NER, …2023ArXiv\([Leong et al\., 2023](https://arxiv.org/html/2610.36990#bib.bib63)\)47M\-IFEvalInstruction Following2025Findings of NAACL 2025\([Dussolle et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib71)\)48MUCHSentiment / Classification2026LREC\([Dentan et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib17)\)49FluidQAQA / Knowledge, Sentiment / Classification2025EMNLP 2025\([Park et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib77)\)50BhashaBench\_V1QA / Knowledge2025ArXiv\([Devane et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib62)\)51CulturalBenchQA / Knowledge2025ACL 2025\([Chiu et al\., 2025](https://arxiv.org/html/2610.36990#bib.bib81)\)

Table 8:Comprehensive overview of all 51 multilingual evaluation benchmarks surveyed\.