Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture

# Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture
Source: [https://arxiv.org/html/2608.28641](https://arxiv.org/html/2608.28641)
\\IfFontExistsTF

NotoSerifCJK\-subset\.otf\\IfFontExistsTFHaranoAjiMincho\-Regular\.otf\\IfFontExistsTFNoto Serif CJK KR\\IfFontExistsTFNoto Serif KR\\IfFontExistsTFNotoSerifDevanagari\-subset\.otf\\IfFontExistsTFNewCM10Devanagari\-Regular\.otf\\IfFontExistsTFNoto Serif Devanagari\\IfFontExistsTFNoto Sans Devanagari

Yunsu Kim Kaden Uhlig Ashwin Purohit Milind Agarwal Patrick Simianer Anil Arslan Kiarash Mokhtari Thomas Zenkel Johannes Mosig Gabriel Bretschner Shamik Bose Joern Wuebker John DeNero LILT, Inc\. contact@lilt\.com

###### Abstract

Most evaluations for coding agents are conducted exclusively in English, which does not reflect real\-world multilingual deployment\. We present Terminal\-Bench\-LILT, a suite of 300 authentic coding tasks in 10 languages: Arabic, Czech, German, Spanish, Hindi, Japanese, Korean, Serbian, Turkish, and Chinese\. Each task targets issues specific to non\-English software development that have no direct English equivalent, e\.g\., internationalization, encoding, text normalization, and cultural conventions\. All tasks are authored by native\-speaker programmers and validated through a multi\-stage quality control pipeline\. Evaluation of six frontier models reveals that even the strongest model reaches only 63\.1% pass rate, with many tasks unsolved by any model\. Performance varies substantially by language and does not track general coding benchmark rankings, highlighting that multilingual coding competence is a distinct and underexplored capability axis\. Sample tasks are available at[https://github\.com/lilt/terminal\-bench\-lilt](https://github.com/lilt/terminal-bench-lilt)\.

## 1Introduction

Software development is a global field in which English serves as the common working language: open\-source projects are coordinated in English\(Kocetkovet al\.,[2023](https://arxiv.org/html/2608.28641#bib.bib3); Lozhkovet al\.,[2024](https://arxiv.org/html/2608.28641#bib.bib4); Bhuiyanet al\.,[2026](https://arxiv.org/html/2608.28641#bib.bib1)\), and community resources such as Stack Overflow are written and answered largely in English\(Stack Overflow,[2024](https://arxiv.org/html/2608.28641#bib.bib21)\)\. Most programmers therefore read, write, and discuss code in English, and the models and tools built on this ecosystem are trained and tuned primarily for English and perform best in it\(GitHub,[2025a](https://arxiv.org/html/2608.28641#bib.bib22)\)\. The benchmarks used to evaluate these agents follow the same pattern: beyond being written almost entirely in English, they implicitly encode the data, locales, and conventions of an English\-speaking development context\(Chenet al\.,[2021](https://arxiv.org/html/2608.28641#bib.bib5); Jimenezet al\.,[2024](https://arxiv.org/html/2608.28641#bib.bib6); Jainet al\.,[2025](https://arxiv.org/html/2608.28641#bib.bib7); Merrillet al\.,[2026](https://arxiv.org/html/2608.28641#bib.bib8)\)\.

However, many developers are not native English speakers\(SlashData,[2025](https://arxiv.org/html/2608.28641#bib.bib23); GitHub,[2025b](https://arxiv.org/html/2608.28641#bib.bib24)\)and solve problems outside an English\-speaking context\. Particularly in local markets, developers work with native\-language data, configure locale\-specific settings, and build services that fit local conventions and culture\. These are real engineering challenges that rarely surface in English\-speaking contexts, and they directly shape how non\-English\-speaking developers experience coding agents\. The ideal coding agent proactively identifies and resolves implicit technical and cultural assumptions\. This is taken for granted in English\-centric tasks, while in non\-English contexts agents need explicit guidance to avoid pitfalls native speakers consider obvious\(Shenet al\.,[2024](https://arxiv.org/html/2608.28641#bib.bib27); Naouset al\.,[2024](https://arxiv.org/html/2608.28641#bib.bib28); Myunget al\.,[2024](https://arxiv.org/html/2608.28641#bib.bib29)\)\. This raises the expertise required of users and makes the agent tedious to use\.

No existing coding benchmark evaluates agents on this dimension, so we built Terminal\-Bench\-LILT, a suite of 300 terminal\-based coding tasks in 10 languages across three difficulty tiers\. Native programmers authored each task from a problem they had actually encountered, then put it through layered quality control combining automated checks and human review\.

I'montheTokyoNOC\(networkoperationscenter\)team,andIneedtofitourdaily\-statusloglinescleanlyintothefixed40\-columnmonitoringterminalweprintthemto\.

\.\.\.

\-StripANSIescapesequences\.Theytake0displaycolumnsandmustnotappearintheoutput\.

\-Converthalfwidthkatakanatoitsfullwidthequivalent\.Thehalfwidthvoicedmarkandsemi\-voicedmarkmergeintotheprecedingkana\.Theprolongedsoundmarkbecomesfullwidth\.

\.\.\.

\(a\)Instruction
![Refer to caption](https://arxiv.org/html/2608.28641v1/x1.png)\(b\)Example input/output\.

Figure 1:Theja\-terminal\-widthtask from Terminal\-Bench\-LILT\. The instruction in \(a\) is abridged for clarity\. In \(b\), each grid cell is one display column of the terminal\.Figure[1](https://arxiv.org/html/2608.28641#S1.F1)shows a task from Terminal\-Bench\-LILT\. The agent is asked to fit Japanese log lines into a fixed 40\-column terminal\. Each line mixes Japanese characters, ASCII, emoji, and terminal color escapes, which occupy different numbers of columns\. Further edge cases arise naturally from Japanese text itself, for example:

1. 1\.Unicode variation selectors, often used in names to force a specific visual variant, are not displayed and must be left out of the width calculation\. Counting code points, as Python’slen\(\)does, gives each one a column\.
2. 2\.Kanji have compatibility ideographs that look identical but have different bytes, e\.g\.,\\cjkfont神 = U\+FA19and\\cjkfont神 = U\+795E\. Applying Unicode NFKC without care merges them into the more common form, altering text the task only asked to reformat\.

Such issues are routine for a developer who handles Japanese terminal output and tedious to spell out\. The instruction therefore asks for it only implicitly, while the tests check exactly these points\. The task thus precisely measures a smooth user experience for a native developer working in Japanese\.

The rest of the paper is organized as follows\. Section[2](https://arxiv.org/html/2608.28641#S2)reviews prior coding benchmarks and related work\. Section[3](https://arxiv.org/html/2608.28641#S3)describes the design, construction, and quality control of Terminal\-Bench\-LILT\. Section[4](https://arxiv.org/html/2608.28641#S4)evaluates frontier models on the benchmark and analyzes the results\. Section[5](https://arxiv.org/html/2608.28641#S5)concludes, followed by an appendix listing sample tasks\.

## 2Related Work

#### Agentic Coding Benchmarks

Evaluation of code generation has moved from single\-function synthesis\(Chenet al\.,[2021](https://arxiv.org/html/2608.28641#bib.bib5)\)toward repository\-level and agentic settings: SWE\-bench\(Jimenezet al\.,[2024](https://arxiv.org/html/2608.28641#bib.bib6)\)poses real GitHub issues, LiveCodeBench\(Jainet al\.,[2025](https://arxiv.org/html/2608.28641#bib.bib7)\)provides contamination\-free competitive problems, and Terminal\-Bench\(Merrillet al\.,[2026](https://arxiv.org/html/2608.28641#bib.bib8)\)evaluates agents on command\-line tasks\. These benchmarks are all English\-only; we build on Terminal\-Bench and extend its task format to ten non\-English languages\.

#### Multilingual Coding Benchmarks

Many coding benchmarks that call themselves*multilingual*vary the*programming*language of the code while keeping the instruction in English\(Cassanoet al\.,[2023](https://arxiv.org/html/2608.28641#bib.bib30); Athiwaratkunet al\.,[2023](https://arxiv.org/html/2608.28641#bib.bib31); Zanet al\.,[2025](https://arxiv.org/html/2608.28641#bib.bib32)\)\. Others do vary the human language, but only in the instruction, translating English coding tasks into other natural languages\(Raihanet al\.,[2025](https://arxiv.org/html/2608.28641#bib.bib2); Penget al\.,[2024](https://arxiv.org/html/2608.28641#bib.bib33); Wanget al\.,[2023](https://arxiv.org/html/2608.28641#bib.bib34);[2024](https://arxiv.org/html/2608.28641#bib.bib35); Hofmanet al\.,[2025](https://arxiv.org/html/2608.28641#bib.bib15)\)\. They primarily measure model comprehension of translated prompts, not how useful a model is for developers in a non\-English context\.

In coding, this comprehension axis seems to be an inconsistent driver of difficulty\. For instance,Hofmanet al\.\([2025](https://arxiv.org/html/2608.28641#bib.bib15)\)translate SWE\-bench\(Jimenezet al\.,[2024](https://arxiv.org/html/2608.28641#bib.bib6)\)into ten languages, but the mean non\-English performance shows only a1\.3%1\.3\\%relative drop from English\. Meanwhile,Wanget al\.\([2024](https://arxiv.org/html/2608.28641#bib.bib35)\)report a relative decrease of at least13%13\\%in pass rate on HumanEval\-X\(Zhenget al\.,[2023](https://arxiv.org/html/2608.28641#bib.bib36)\)tasks translated into Chinese\. In these cases, it is hard to distinguish between errors introduced by translation and genuine limits of model capability; moreover, all tasks are drawn from an English context\.

Our benchmark is orthogonal to both lines of work: the programming language is incidental, and we vary the human language together with its regional and cultural context\. Our tasks are authored directly in a given language by native speakers, avoiding translation artifacts, and we keep only tasks that are genuinely language\- and culture\-specific, enabling a targeted evaluation of models’ usefulness for non\-English developers\.

#### Reasoning Language

One hypothesis is that a translated instruction leaves the underlying problem unchanged: a model can read a non\-English prompt yet still reason in English\(Schutet al\.,[2025](https://arxiv.org/html/2608.28641#bib.bib39)\)\.Baruaet al\.\([2026](https://arxiv.org/html/2608.28641#bib.bib38)\)find that reasoning in English outperforms reasoning in the target language, with the gap widening on multi\-step tasks\. For code generation,Nishigataet al\.\([2025](https://arxiv.org/html/2608.28641#bib.bib37)\)exploit this directly, tuning models to attach an English chain\-of\-thought to non\-English instructions\. Difficulty from the instruction language is thus limited, and such tuning can mitigate it further\. Our tasks instead rest on the locale\-specific knowledge the instructions leave implicit — a gap that reasoning alone does not close \(Section[4\.3](https://arxiv.org/html/2608.28641#S4.SS3.SSS0.Px2)\)\.

#### Cultural and Locale Knowledge

Benchmarks of everyday cultural knowledge expose large disparities between well\- and under\-represented cultures:Myunget al\.\([2024](https://arxiv.org/html/2608.28641#bib.bib29)\)report a best\-to\-worst culture gap of up to 57 percentage points for GPT\-4 on short\-answer questions, and related work documents cultural bias\(Naouset al\.,[2024](https://arxiv.org/html/2608.28641#bib.bib28)\)and the limits of cultural commonsense\(Shenet al\.,[2024](https://arxiv.org/html/2608.28641#bib.bib27)\)in LLMs\. These benchmarks, however, probe such knowledge in plain question answering, not in the work that models are actually used for\. Our benchmark tests the same knowledge in agentic coding, where an agent passes only if its code handles the local convention correctly\.

## 3Terminal\-Bench\-LILT

Terminal\-Bench\-LILT has 300 tasks spanning 10 languages, with 30 tasks in each language\. Each task runs under Harbor\(Harbor Framework Team,[2026](https://arxiv.org/html/2608.28641#bib.bib20)\)in an isolated environment with deterministic verification\. It follows the original Terminal\-Bench task structure\(Merrillet al\.,[2026](https://arxiv.org/html/2608.28641#bib.bib8)\), with additional fields and documentation for multilingual challenges:

- •Instructiondescribes a real\-world coding problem requiring the agent to write a solution script\. It is provided in both the native language and English\.
- •Environment\(Docker configuration\) together with any native\-language data assets required by the task\.
- •Verifierwith deterministic tests\.
- •Oracle solutionaccepted by the verifier\.
- •Metadatacontaining brief descriptions of the task, solution, and verifier, together with the ISO 639\-1 language code, challenge category and subcategory, difficulty, and infrastructure configuration\.
- •READMEexplaining the motivation behind the task design, its realism, and its linguistic and technical challenges in detail\.

Instructions were written*from scratch in the native target language*first\. Many tasks are grounded in locale\-specific data, such as files in legacy encodings and regional date formats, and are therefore most naturally described in the language in which the data was produced\. This approach also reflects how users instruct coding agents in their native languages, using colloquial phrasing and domain\-specific vocabulary\. Writing the instructions in English first and then translating them, as is common in prior work \(Section[2](https://arxiv.org/html/2608.28641#S2)\), could introduce translation artifacts and make the resulting tasks less representative of real\-world use\.

An English translation of the native instruction is also provided, making it accessible to non\-native speakers and enabling controlled experiments that isolate the effect of instruction language from task difficulty \(Section[4\.3](https://arxiv.org/html/2608.28641#S4.SS3)\)\.

### 3\.1Task Design

CategorySubcategoryScope / ExamplesInternationalizationLocale\-based TemplatingPlurals, numbers, suffixes, date formats, Korean*man*numberingCharacter RenderingRTL, Arabic shaping, Simplified vs\. Traditional Chinese, CyrillicInteraction with EnvironmentEncodingsNon\-Unicode/legacy file encodings, conversionSystem ConfigurationCJK input method debugging, RTL shell tools, locale\-correct sortingText ConversionNormalizationNFC/NFKC normalization, mojibake correctionML & Data ProcessingCase augmentation, tokenizer training, typo simulationVariant ConversionScript conversion, German spelling reformText HandlingDocument SearchStemming, stop word removal for agglutinative languagesUnicode Confusion ListsAnti\-spam character confusion in Cyrillic\-first applicationsUser AuthenticationNormalized username/password comparison, DB migrationCultural & OtherCalendar/Date ConversionLunar, Hijri, Umm al\-Qura, holiday calculationOtherCulture/Region\-specific issues that cannot be categorized into existing subcategoriesTable 1:Challenge taxonomy for Terminal\-Bench\-LILT tasks\.Table 2:Distribution of tasks by challenge category and language\.Each task includes one or more challenges specific to its target language, region, or culture\. We assign every task a primary category and subcategory from the challenge taxonomy in Table[1](https://arxiv.org/html/2608.28641#S3.T1); Table[2](https://arxiv.org/html/2608.28641#S3.T2)shows the distribution of tasks by category and language\. Descriptions of sample tasks are given in Appendix[A](https://arxiv.org/html/2608.28641#A1)\.

Regardless of category, every task follows three design principles:

1. 1\.A task should reflect a problem that a programmer in the target locale could plausibly encounter\. Its data, formats, and workflows should match those*actually used in practice*\.
2. 2\.A task’s difficulty should arise from*locale\-specific elements*, not from general engineering complexity\. A task that merely restates an ordinary programming problem in another language does not qualify\.
3. 3\.Locale\-specific challenges should be*embedded implicitly*in the engineering problem rather than stated directly\. This tests whether agents can supply the local knowledge that native developers take for granted, without requiring users to spell it out\.

The third principle is particularly important for cultural challenges where the task involves everyday knowledge in the target region beyond technical conventions such as encoding or sorting order\. When the required knowledge is too specialized, e\.g\., a traffic rule specific to one state, the task becomes asking whether a model happens to know it, even though modern agents can retrieve such facts through web search or reference files\. We avoid such esoteric knowledge recall and focus on native user experience by keeping the two rules below:

- •Any challenge should be*widely known*to ordinary adults or programmers in the target country\.
- •When an expert domain such as law or medicine is involved, it should serve only as realistic*context*, never as the challenge itself\. This is realized by citing or attaching the exact reference in the instruction\.

IhaveasmallcorpusofCzechsentencesat/app/data/sentences\.jsonl\.

Ineedascriptat/app/split\.pythatreads/app/data/sentences\.jsonlandwrites/app/output/lines\.jsonlwithonerecordperinputsentence,breakingeachsentenceintooneormoretypesetlinesthatfitaconfiguredtargetwidth\.

\.\.\.

ThelinesmustfollowCzechorthotypography\-thecanonicalrulesacompetentCzechtypographerwouldapply\.Thedescriptionbelowisillustrativeonly;therearefurtherpatternstoo,andallofthemapply\.

Single\-letterprepositionsandconjunctions\(\`k\`,\`s\`,\`v\`,\`z\`,\.\.\.\)mustneverendaline;wherevertheoriginaltexthasoneasastandalonetokenfollowedbyanotherword,IgluethemwithU\+00A0\(NBSP\)insteadofanASCIIspace\.Thesamenon\-breakingspacejoinsanumericquantitytoitsunitsymbol,thesectionsigntoafollowingarticlenumber\(\`§12\`\),titlesbeforeaname\(\`Dr\.\`,\`Mgr\.\`,\.\.\.\),\.\.\.

Thebundledsentencessampleonlyahandfulofthesepatterns;sincethescriptwillalsorunoveradifferentfile,IimplementthecanonicalCzechrulesinfullratherthanoverfittingtothebundledsentences\.

\.\.\.

\(a\)cs\-nbsp\-justify
Thefile/app/produkte\.csvcontainsalistofproductsfromourwholesalecatalogue\(columns:\`produkt\_id\`,\`bezeichnung\`\)\.Foraccounting,eachproductneedstheapplicableGermanVATrate\-\-\-eitherthereducedrate\(7%\)orthestandardrate\(19%\)\.

Aworkingstudentleftafirstdraftat/app/klassifikator\_draft\.py\.Inaspot\-check,ourtaxdepartmentflaggedseveralincorrectclassifications,sothedraftisnotacceptableas\-is\.

Pleasewriteacorrectedscript/app/klassifikator\.pythatreadstheinputfileandassignseachrowthecorrectVATrateunderGermantaxlawasof1January2026\.WritetheresultasaJSONarrayto/app/output/ergebnis\.json\.Eachentrymusthavethefields\`produkt\_id\`,\`bezeichnung\`,and\`mwst\_satz\`,where\`mwst\_satz\`istheinteger\`7\`or\`19\`\.Theordermustmatchtheinputfile\.

Exampleoutput:

\[

\{"produkt\_id":"P001","bezeichnung":"Roggenbrot1kg","mwst\_satz":7\},

\{"produkt\_id":"P002","bezeichnung":"Rotweintrocken0,75l","mwst\_satz":19\}

\]

Runwith\`python3/app/klassifikator\.py\`,noarguments\.Thedirectory/app/output/willbecreatedifneeded\.

\(b\)de\-umsatzsteuer\-klassifikation

Figure 2:Instructions for two sample tasks with different challenge scopes: \(a\) a bounded challenge and \(b\) an unbounded one fenced to a finite range\.
### 3\.2Test Design

Each task comes with a test suite combining basic integrity checks \(e\.g\., whether a solution script was created\), sample probes using the files provided with the task, and unseen probes using new inputs\. The unseen probes test generalization and discourage hard\-coded solutions, but remain within the intended challenge\.

Every tested behavior must be*inferable*from either the instruction, the input data, or common knowledge in the target locale\. The instruction should make clear that the provided files are examples rather than the complete input set; otherwise, unseen probes would test behavior that cannot be inferred from the instruction\. Requirements unrelated to the challenge, such as the output format, must be stated unambiguously\.

The scope of each challenge must also be testable\. When the challenge covers a*bounded*set of cases, the instruction may show only some of them and withhold the rest for testing, requiring a general solution\. For example, in Figure[2\(a\)](https://arxiv.org/html/2608.28641#S3.F2.sf1), the Czech non\-breaking\-space rules form a finite, closed set, so the instruction gives only a few example patterns and a small sample of sentences; the held\-out sentences exercise the patterns the samples never show, catching solutions that overfit to the visible ones\. When the cases are*unbounded*, the instruction and tests must instead define a finite range; otherwise the challenge is excluded\. For example, in Figure[2\(b\)](https://arxiv.org/html/2608.28641#S3.F2.sf2), the German VAT rates are unbounded legislated carve\-outs that no general solution captures, so the instruction fixes a finite range \(a specific legal snapshot and product list\) and the tests check only that range\.

### 3\.3Quality Control

Every task passes four review layers before entering the benchmark\. The layers are ordered by cost: automated checks run on every commit by the task author, and human reviewers see only the tasks that survive them\. Figure[3](https://arxiv.org/html/2608.28641#S3.F3)shows the full workflow for quality control\.

![Refer to caption](https://arxiv.org/html/2608.28641v1/x2.png)Figure 3:Quality control workflow for task submissions\.✓and✗indicate whether a task passes or fails each stage, respectively\. The responsible party is shown above each stage\.1. 1\.Deterministic checksvalidate the task’s structure and execution setup, including its file layout, Docker environment, metadata fields, and whether the oracle solution passes all tests\.
2. 2\.LLM judgesuse Gemini 3 Flash to assess the task’s semantic properties, such as parity between the native\-language and English instructions, alignment with the claimed task category, and whether agents can infer each tested behavior from the task materials or common local knowledge\.
3. 3\.Initial human reviewre\-audits the automated checks and assesses criteria that are difficult to automate: e\.g\., compliance with the AI\-use policy, realism of the coding scenario, and clarity of constraints\.
4. 4\.Final human reviewindependently repeats the initial review, checks for overlap with existing tasks, and curates the final set for balanced topic and difficulty distributions, recommending difficulty adjustments when needed\.

At any stage, if we found that a task could not be made valid and realistic with reasonable effort, we rejected it\. Overall, about 25% of reviewed tasks were rejected\.

#### Review Agents

Reviewing a task at this level of detail requires substantial time and sustained attention\. We therefore built two review agents to assist human reviewers: one assesses each task on its own, while the other compares it with existing tasks in the benchmark\. Each delegates subtasks to specialized agents and combines their findings into a single report with concrete citations\. We equipped the agents with deterministic tools whenever possible, e\.g\., to prepare execution logs and failure summaries, so the LLMs can focus on judgment\. Since deployment, we have periodically used human reviewer feedback on agent outputs to refine the rubrics, aligning automated review with human judgment and reducing review time\.

#### Tooling

To streamline the multi\-stage review process for hundreds of tasks, we developed an internal management application*Benchito*which tracks the stage of each submitted task \(Figure[4](https://arxiv.org/html/2608.28641#S3.F4)\)\. Program managers can filter and sort tasks by language, difficulty, author, or reviewer and compare task counts for a selected period with the target distribution\. It is integrated with the project’s communication channels and notifies the responsible author or reviewer when a task changes state or fails an automated check\. Access is restricted to the organization, with an audit trail of every action for accountability\.

![Refer to caption](https://arxiv.org/html/2608.28641v1/figures/benchito.png)Figure 4:Workflow management tool for Terminal\-Bench\-LILT\.

### 3\.4Team Management

Task quality depends on the expertise of both authors and reviewers\. We therefore used explicit selection criteria and structured training\.

#### Author Selection

For each language, we screened native\-speaker candidates for backgrounds in computer science or engineering and for multilingual experience\. Candidates who met these criteria completed an assessment of their ability to design tasks that challenge LLMs, which our reviewers evaluated manually\. Fewer than 10% of these candidates passed\.

#### Reviewer Selection

Initial reviewers were either research scientists with backgrounds in machine learning and NLP or task authors who had developed more than ten Terminal\-Bench\-LILT tasks\. Final reviewers were senior research scientists and engineers with extensive LLM benchmarking experience\.

#### Onboarding and Training

New task authors completed a first\-day checklist covering access provisioning and tool setup\. Each then attended a one\-hour training session on task architecture, Harbor, the taxonomy, example tasks, quality criteria, and the task submission workflow\. Before authoring tasks, they were required to demonstrate their understanding of the training materials, and reviewers provided detailed feedback on their first submission\. Reviewers remained available through a real\-time channel for ongoing feedback and held targeted calibration sessions when needed\.

### 3\.5Semi\-Automatic Task Creation

For experimental purposes and to supplement our dataset, we created a limited number of tasks through a partially automated workflow\. We built an agent equipped with specialized skills that operates in two stages:

1. 1\.Idea generation\.First, the agent researches linguistic and cultural software conventions from the web, extracts candidate challenges, and removes ideas similar to accepted or rejected tasks\. The remaining ideas are ranked by general quality and challenge specificity; a human reviewer approves the shortlist before development begins\. This stage runs once per language\.
2. 2\.Task generation\.For each approved idea, the agent produces a progress file with a development checklist\. Following this checklist iteratively, it implements the solution and unit tests, runs automated checks, and calibrates difficulty from agent trajectories\.

After calibration, each created task undergoes the same automated checks and review process as tasks written by humans \(Section[3\.3](https://arxiv.org/html/2608.28641#S3.SS3)\), with an additional initial stage in which a native speaker of the target language reviews only its linguistic quality\. This process produced approximately 10% of the Terminal\-Bench\-LILT tasks\.

## 4Experiments

On the full Terminal\-Bench\-LILT set, we evaluated six frontier LLMs across three providers: Claude Opus 4\.7\(Anthropic,[2026a](https://arxiv.org/html/2608.28641#bib.bib9)\)and 4\.8\(Anthropic,[2026b](https://arxiv.org/html/2608.28641#bib.bib10)\)from Anthropic, GPT\-5\.4\(OpenAI,[2026a](https://arxiv.org/html/2608.28641#bib.bib17)\)and 5\.5\(OpenAI,[2026b](https://arxiv.org/html/2608.28641#bib.bib18)\)from OpenAI, and Gemini 3\.1 Pro\(Google DeepMind,[2026a](https://arxiv.org/html/2608.28641#bib.bib13)\)and 3\.5 Flash\(Google DeepMind,[2026b](https://arxiv.org/html/2608.28641#bib.bib14)\)from Google\. Thinking/Reasoning level was set to the provider’s default for each model, following the same convention asMerrillet al\.\([2026](https://arxiv.org/html/2608.28641#bib.bib8)\)\. The agent was fixed to theterminus\-2harness and each task\-model pair was run five times\. We also manually inspected the failing trials to distinguish true language\-specific failures from infrastructure issues, e\.g\., random crashes, OOMs, API errors, or harness failures; any trial affected by the latter was re\-run so that all reported results reflect genuine task attempts\.

![Refer to caption](https://arxiv.org/html/2608.28641v1/x3.png)Figure 5:Task pass rate per model on Terminal\-Bench\-LILT, sorted by rate\. Error bars show 95% confidence intervals\.### 4\.1Main Results

Figure[5](https://arxiv.org/html/2608.28641#S4.F5)shows the aggregate pass rate across all trials for each model\. GPT\-5\.5, Gemini 3\.5 Flash, and Claude Opus 4\.8 form the leading group, while Claude Opus 4\.7 and Gemini 3\.1 Pro follow, with GPT\-5\.4 far behind\. The newer OpenAI model improves dramatically over its predecessor \(a 31\.9\-point gain\), while Google’s and Anthropic’s newer models improve more modestly, by 5\.3 and 3\.4 points\. Even the strongest model fails 36\.9% of trials, indicating that the benchmark is not saturated and leaves substantial room for improvement\.

#### Difficulty Calibration

We used the same trial results to estimate task difficulty followingMerrillet al\.\([2026](https://arxiv.org/html/2608.28641#bib.bib8)\)\. For each task, we pooled its 30 trial outcomes across the six models and computed the overall pass rate\. We classified tasks with pass rates of at least 66\.7% as easy, rates from 33\.3% to below 66\.7% as medium, and rates below 33\.3% as hard\. Table[3](https://arxiv.org/html/2608.28641#S4.T3)shows an approximate 4:3:3 ratio of easy, medium, and hard tasks\.

Table 3:Distribution of tasks by language and empirical difficulty\.

### 4\.2Performance Breakdown

Table 4:Per\-language pass rates\.σlang\\sigma\_\{\\mathrm\{lang\}\}andσmodel\\sigma\_\{\\mathrm\{model\}\}are the standard deviations \(in percentage points\) across languages and across models, respectively\.Table 5:Per\-category pass rates for the top three models\.Terminal\-Bench\-LILT enables analysis of model weaknesses by language and task category\. These comparisons are descriptive rather than controlled: empirical difficulty varies across languages, and category sizes are unequal\. Even with these limitations, the results reveal broad performance patterns worth investigating\.

#### Per\-Language Results

Table[4](https://arxiv.org/html/2608.28641#S4.T4)reports per\-language pass rates for the top three models\. Performance varies substantially across languages, as do the model rankings\.

Hindi and German are the two lowest\-scoring languages for every model, both around 50–53%\. The two leading models show different language strengths: GPT\-5\.5 leads on Arabic, Spanish, Japanese, and Serbian, whereas Gemini 3\.5 Flash leads on German and Korean\. The largest gaps between them occur in Japanese \(13\.4 points\), Spanish \(13\.3 points\), and Chinese \(11\.4 points\)\. Claude Opus 4\.8 generally falls behind these two models but leads on Czech and Hindi\.

Note that Gemini 3\.5 Flash shows notably more uniform performance across languages than the others \(σlang=4\.9\\sigma\_\{\\mathrm\{lang\}\}=4\.9\), with no language below 52%\. Among the ten languages, Hindi and Korean show the most consistent performance across models \(σmodel=0\.5\\sigma\_\{\\mathrm\{model\}\}=0\.5\), whereas Japanese, Spanish, and Chinese show the largest cross\-model variation\.

#### Per\-Category Results

We also report pass rates for the top models by task category \(Table[5](https://arxiv.org/html/2608.28641#S4.T5)\)\. Text handling is the hardest category for every model, whereas text conversion and environment interaction are the easiest\. The gap between a model’s best and worst category can be wide, reaching 18\.0 points for GPT\-5\.5 between text conversion and text handling\.

We hypothesize that this gap reflects how much language\-specific knowledge each category requires\. For example, text conversion relies on standardized transformations with direct library support \(e\.g\., Unicode normalization\), so a general solution often transfers across languages\. In contrast, text handling depends on locale\-specific rules that default tools handle incorrectly, such as Turkish case folding or stemming for agglutinative languages, forcing the model to supply the exact knowledge these tasks withhold\.

### 4\.3Ablation Studies

#### Effect of Instruction Language

Every Terminal\-Bench\-LILT task is accompanied by an English translation of its native instruction, which lets us run the same task with the translated instruction while holding the data, environment, and verifier fixed\. This isolates the effect of switching the instruction language, comparing the target language against the models’ strongest language, English\.

Figure[6](https://arxiv.org/html/2608.28641#S4.F6)shows this comparison in three languages with different scripts, using the top three models from the main results \(Section[4\.1](https://arxiv.org/html/2608.28641#S4.SS1)\)\. Across the nine model\-language configurations, English instructions never move the pass rate by more than 7 points, and move it by less than 5 in six of them, similar to the small, inconsistent comprehension effects reported for translated benchmarks \(Section[2](https://arxiv.org/html/2608.28641#S2)\)\. Even with English instructions, every configuration stays around 50–60%\. This confirms the premise behind our native\-first task design: instruction language is not the bottleneck; the hard part is working with native\-script data under locale\-specific rules\.

![Refer to caption](https://arxiv.org/html/2608.28641v1/x4.png)Figure 6:Native language vs\. English instruction ablation\. Every bar pair covers the same 30 tasks, scored in both conditions\. The number above each pair is the difference in pass rate, English minus native\.
#### Effect of Reasoning

![Refer to caption](https://arxiv.org/html/2608.28641v1/x5.png)Figure 7:Reasoning ablation\. Solid lines mark significant changes withp<\.05p<\.05in a two\-sided paired permutation test\.All LLMs tested in this work are capable of reasoning via chain\-of\-thought\(Weiet al\.,[2022](https://arxiv.org/html/2608.28641#bib.bib41)\), which may improve problem solving on coding tasks at the cost of more output tokens and longer runtimes\. To study this effect, we ran the evaluation with reasoning switched on and off and compared the results\.111When on, we used each provider’s default reasoning level:*medium*for GPT\-5\.5 and Gemini 3\.5 Flash,*high*for Claude Opus 4\.8\. Gemini 3\.5 Flash cannot disable reasoning entirely by design, so for the off condition we set it to its*minimal*level\.For this ablation, we effectively removed the tasks’ time limit, letting each agent run until it finished organically\. This study was run on 19 Terminal\-Bench\-LILT tasks that are hard under the empirical difficulty classification \(Table[3](https://arxiv.org/html/2608.28641#S4.T3)\)\. For comparison, we also ran the same reasoning ablation on 19 randomly sampled hard tasks from the original, English\-only Terminal\-Bench\(Merrillet al\.,[2026](https://arxiv.org/html/2608.28641#bib.bib8)\); note that their tasks pose pure engineering challenges, unlike ours, which also require linguistic and cultural knowledge\.

Figure[7](https://arxiv.org/html/2608.28641#S4.F7)shows the results\. While reasoning improves GPT\-5\.5, no significant effect can be observed with Gemini 3\.5 Flash or Claude Opus 4\.8\. With reasoning on, the three models converge to a similar level on the English tasks \(60\.0–65\.3%\), whereas on Terminal\-Bench\-LILT they range from 10\.5% to 35\.8%\. Reasoning thus lifts English performance to a shared level, but leaves large differences in multilingual capability across models\. Across models, reasoning inflates output token counts to 1\.4–2\.7×\\timestheir off/minimal\-reasoning levels, yet the large majority of the Terminal\-Bench\-LILT trials still fail\. Note that some tasks in German and Hindi stay at a 0% pass rate even with reasoning\.

## 5Conclusion

We introduce Terminal\-Bench\-LILT, an agentic coding benchmark of 300 tasks written by native\-speaker programmers in ten non\-English languages\. Each task requires language\-, region\-, and culture\-specific knowledge that a native developer takes for granted, built into everyday engineering work rather than tested in isolation\. A layered review pipeline retains only tasks that are realistic, verifiable, and genuinely locale\-specific\. The benchmark is far from saturated: frontier models still fail over a third of trials, many tasks go unsolved entirely, pass rates vary widely by language, and rankings do not follow general coding\-benchmark standing\. Our ablation studies show that this difficulty is not an artifact of non\-English prompting or of limited compute\. Terminal\-Bench\-LILT reveals gaps in multilingual capability across models, languages, and task categories that English benchmarks cannot expose\. Its native\-first design also offers a template for evaluation beyond English\.

## Ethics Statement

First, all tasks were written by native\-speaker programmers who contributed voluntarily with informed consent and compensation; we screened submissions for personal, proprietary, or sensitive data and collected no identifying information about contributors\. Second, our ten languages are a small, uneven sample and any single language spans many locales we do not fully cover, so low scores signal a measurement gap rather than a verdict on any language or its speakers\. Third, this is a diagnostic benchmark for a narrow capability: strong scores do not certify a model as fit for deployment, and weak scores should prompt further study, not exclusion of a language\. Fourth, we embed canary strings to limit training\-data contamination and plan to refresh the pool over time, and because these tasks execute code, all evaluation runs in sandboxed containers\.

## References

- Claude Opus 4\.7 System Card\.Technical reportAnthropic\.Cited by:[§4](https://arxiv.org/html/2608.28641#S4.p1.1)\.
- Anthropic \(2026b\)Claude Opus 4\.8 System Card\.Technical reportAnthropic\.Cited by:[§4](https://arxiv.org/html/2608.28641#S4.p1.1)\.
- B\. Athiwaratkun, S\. K\. Gouda, Z\. Wang, X\. Li, Y\. Tian, M\. Tan, W\. U\. Ahmad, S\. Wang, Q\. Sun, M\. Shang, S\. K\. Gonugondla, H\. Ding, V\. Kumar, N\. Fulton, A\. Farahani, S\. Jain, R\. Giaquinto, H\. Qian, M\. K\. Ramanathan, R\. Nallapati, B\. Ray, P\. Bhatia, S\. Sengupta, D\. Roth, and B\. Xiang \(2023\)Multi\-lingual evaluation of code generation models\.InThe Eleventh International Conference on Learning Representations \(ICLR\),Note:arXiv preprint arXiv:2210\.14868Cited by:[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Barua, S\. Eisape, K\. Yin, and A\. Suhr \(2026\)Long chain\-of\-thought reasoning across languages\.InThe Fourteenth International Conference on Learning Representations \(ICLR\),Note:arXiv preprint arXiv:2508\.14828Cited by:[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px3.p1.1)\.
- M\. H\. M\. Bhuiyan, M\. K\. Bala Kumar, and C\. Staicu \(2026\)“Write in English, nobody understands your language here”: a study of non\-English trends in open\-source repositories\.InProceedings of the IEEE/ACM 48th International Conference on Software Engineering \(ICSE ’26\),Note:arXiv preprint arXiv:2602\.19446External Links:[Document](https://dx.doi.org/10.1145/3744916.3787766)Cited by:[§1](https://arxiv.org/html/2608.28641#S1.p1.1)\.
- F\. Cassano, J\. Gouwar, D\. Nguyen, S\. Nguyen, L\. Phipps\-Costin, D\. Pinckney, M\. Yee, Y\. Zi, C\. J\. Anderson, M\. Q\. Feldman, A\. Guha, M\. Greenberg, and A\. Jangda \(2023\)MultiPL\-E: a scalable and polyglot approach to benchmarking neural code generation\.IEEE Transactions on Software Engineering49\(7\),pp\. 3675–3691\.Note:arXiv preprint arXiv:2208\.08227External Links:[Document](https://dx.doi.org/10.1109/TSE.2023.3267446)Cited by:[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. Zaremba \(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§1](https://arxiv.org/html/2608.28641#S1.p1.1),[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px1.p1.1)\.
- GitHub \(2025a\)Application card: GitHub Copilot Agents\.Note:GitHub DocsCited by:[§1](https://arxiv.org/html/2608.28641#S1.p1.1)\.
- GitHub \(2025b\)Octoverse: A new developer joins GitHub every second as AI leads TypeScript to \#1\.Note:The GitHub BlogCited by:[§1](https://arxiv.org/html/2608.28641#S1.p2.1)\.
- Google DeepMind \(2026a\)Gemini 3\.1 Pro: Model card\.Technical reportGoogle DeepMind\.Cited by:[§4](https://arxiv.org/html/2608.28641#S4.p1.1)\.
- Google DeepMind \(2026b\)Gemini 3\.5 Flash: Model card\.Technical reportGoogle DeepMind\.Cited by:[§4](https://arxiv.org/html/2608.28641#S4.p1.1)\.
- Harbor Framework Team \(2026\)Harbor: A framework for evaluating and optimizing agents and models in container environments\.Note:ZenodoSoftware, version v0\.16\.1\.\\doi10\.5281/zenodo\.20953922Cited by:[§3](https://arxiv.org/html/2608.28641#S3.p1.1)\.
- O\. Hofman, J\. Brokman, O\. Rachmil, S\. Bose, V\. Pahuja, T\. Shimizu, T\. Starostina, K\. Marchisio, S\. Goldfarb\-Tarrant, and R\. Vainshtein \(2025\)MAPS: A Multilingual Benchmark for Agent Performance and Security\.arXiv preprint arXiv:2505\.15935\.Cited by:[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px2.p2.2)\.
- N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2025\)LiveCodeBench: holistic and contamination free evaluation of large language models for code\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),Note:arXiv preprint arXiv:2403\.07974Cited by:[§1](https://arxiv.org/html/2608.28641#S1.p1.1),[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px1.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Note:arXiv preprint arXiv:2310\.06770Cited by:[§1](https://arxiv.org/html/2608.28641#S1.p1.1),[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px2.p2.2)\.
- D\. Kocetkov, R\. Li, L\. Ben Allal, J\. Li, C\. Mou, C\. Muñoz Ferrandis, Y\. Jernite, M\. Mitchell, S\. Hughes, T\. Wolf, D\. Bahdanau, L\. von Werra, and H\. de Vries \(2023\)The stack: 3 TB of permissively licensed source code\.Transactions on Machine Learning Research\.Note:arXiv preprint arXiv:2211\.15533Cited by:[§1](https://arxiv.org/html/2608.28641#S1.p1.1)\.
- A\. Lozhkov, R\. Li, L\. Ben Allal, F\. Cassano, J\. Lamy\-Poirier, N\. Tazi, A\. Tang, D\. Pykhtar, J\. Liu, Y\. Wei, T\. Liu, M\. Tian, D\. Kocetkov, A\. Zucker, Y\. Belkada, Z\. Wang, Q\. Liu, D\. Abulkhanov, I\. Paul, Z\. Li, W\. Li, M\. Risdal, J\. Li, J\. Zhu, T\. Y\. Zhuo, E\. Zheltonozhskii, N\. O\. O\. Dade, W\. Yu, L\. Krauß, N\. Jain, Y\. Su, X\. He, M\. Dey, E\. Abati, Y\. Chai, N\. Muennighoff, X\. Tang, M\. Oblokulov, C\. Akiki, M\. Marone, C\. Mou, M\. Mishra, A\. Gu, B\. Hui, T\. Dao, A\. Zebaze, O\. Dehaene, N\. Patry, C\. Xu, J\. McAuley, H\. Hu, T\. Scholak, S\. Paquet, J\. Robinson, C\. J\. Anderson, N\. Chapados, M\. Patwary, N\. Tajbakhsh, Y\. Jernite, C\. Muñoz Ferrandis, L\. Zhang, S\. Hughes, T\. Wolf, A\. Guha, L\. von Werra, and H\. de Vries \(2024\)StarCoder 2 and the stack v2: the next generation\.arXiv preprint arXiv:2402\.19173\.Cited by:[§1](https://arxiv.org/html/2608.28641#S1.p1.1)\.
- M\. A\. Merrill, A\. G\. Shaw, N\. Carlini,et al\.\(2026\)Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.InThe Fourteenth International Conference on Learning Representations \(ICLR\),Note:arXiv preprint arXiv:2601\.11868Cited by:[§1](https://arxiv.org/html/2608.28641#S1.p1.1),[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2608.28641#S3.p1.1),[§4\.1](https://arxiv.org/html/2608.28641#S4.SS1.SSS0.Px1.p1.1),[§4\.3](https://arxiv.org/html/2608.28641#S4.SS3.SSS0.Px2.p1.1),[§4](https://arxiv.org/html/2608.28641#S4.p1.1)\.
- J\. Myung, N\. Lee, Y\. Zhou, J\. Jin, R\. A\. Putri, D\. Antypas, H\. Borkakoty, E\. Kim, C\. Perez\-Almendros, A\. A\. Ayele, V\. Gutiérrez\-Basulto, Y\. Ibáñez\-García, H\. Lee, S\. H\. Muhammad, K\. Park, A\. S\. Rzayev, N\. White, S\. M\. Yimam, M\. T\. Pilehvar, N\. Ousidhoum, J\. Camacho\-Collados, and A\. Oh \(2024\)BLEnD: a benchmark for LLMs on everyday knowledge in diverse cultures and languages\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS 2024\) Datasets and Benchmarks Track,Note:arXiv preprint arXiv:2406\.09948Cited by:[§1](https://arxiv.org/html/2608.28641#S1.p2.1),[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px4.p1.1)\.
- T\. Naous, M\. J\. Ryan, A\. Ritter, and W\. Xu \(2024\)Having beer after prayer? measuring cultural bias in large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 16366–16393\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.862)Cited by:[§1](https://arxiv.org/html/2608.28641#S1.p2.1),[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px4.p1.1)\.
- Y\. Nishigata, W\. Ito, and K\. Kuramitsu \(2025\)Non\-English code generation with cross\-lingual chain of thought\.InProceedings of the 39th Pacific Asia Conference on Language, Information and Computation,C\. Huang, Y\. Harada, J\. Kim, N\. T\.M\. Huyen, L\. T\. Huong, P\. Hien, E\. Chersoni, L\. M\. Nguyen, R\. E\. O\. Roxas, and S\. Dita \(Eds\.\),Hanoi, Vietnam,pp\. 438–446\.Cited by:[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px3.p1.1)\.
- OpenAI \(2026a\)GPT\-5\.4 Thinking System Card\.Technical reportOpenAI\.Cited by:[§4](https://arxiv.org/html/2608.28641#S4.p1.1)\.
- OpenAI \(2026b\)GPT\-5\.5 System Card\.Technical reportOpenAI\.Cited by:[§4](https://arxiv.org/html/2608.28641#S4.p1.1)\.
- Q\. Peng, Y\. Chai, and X\. Li \(2024\)HumanEval\-XL: a multilingual code generation benchmark for cross\-lingual natural language generalization\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING\),Note:arXiv preprint arXiv:2402\.16694Cited by:[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px2.p1.1)\.
- N\. Raihan, A\. Anastasopoulos, and M\. Zampieri \(2025\)MHumanEval – a multilingual benchmark to evaluate large language models for code generation\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),Albuquerque, New Mexico,pp\. 11432–11461\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.570)Cited by:[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px2.p1.1)\.
- L\. Schut, Y\. Gal, and S\. Farquhar \(2025\)Do multilingual LLMs think in English?\.InICLR 2025 Workshop on Building Trust in LLMs and LLM Applications,Note:arXiv preprint arXiv:2502\.15603Cited by:[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Shen, L\. Logeswaran, M\. Lee, H\. Lee, S\. Poria, and R\. Mihalcea \(2024\)Understanding the capabilities and limitations of large language models for cultural commonsense\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 5668–5680\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.316)Cited by:[§1](https://arxiv.org/html/2608.28641#S1.p2.1),[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px4.p1.1)\.
- SlashData \(2025\)There are 47\.2 million developers in the world – Global developer population trends 2025\.Note:SlashData blogCited by:[§1](https://arxiv.org/html/2608.28641#S1.p2.1)\.
- Stack Overflow \(2024\)Can I ask a question in a language other than English?\.Note:Stack Overflow Help CenterCited by:[§1](https://arxiv.org/html/2608.28641#S1.p1.1)\.
- C\. Wang, Z\. Li, C\. Gao, W\. Wang, T\. Peng, H\. Huang, Y\. Deng, S\. Wang, and M\. R\. Lyu \(2024\)Exploring multi\-lingual bias of large code models in code generation\.arXiv preprint arXiv:2404\.19368\.Cited by:[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px2.p2.2)\.
- Z\. Wang, G\. Cuenca, S\. Zhou, F\. F\. Xu, and G\. Neubig \(2023\)MCoNaLa: a benchmark for code generation from multiple natural languages\.InFindings of the Association for Computational Linguistics: EACL 2023,Dubrovnik, Croatia,pp\. 265–273\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-eacl.20)Cited by:[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Vol\.35,pp\. 24824–24837\.Cited by:[§4\.3](https://arxiv.org/html/2608.28641#S4.SS3.SSS0.Px2.p1.1)\.
- D\. Zan, Z\. Huang, W\. Liu, H\. Chen, L\. Zhang, S\. Xin, L\. Chen, Q\. Liu, X\. Zhong, A\. Li, S\. Liu, Y\. Xiao, L\. Chen, Y\. Zhang, J\. Su, T\. Liu, R\. Long, K\. Shen, and L\. Xiang \(2025\)Multi\-SWE\-bench: a multilingual benchmark for issue resolving\.arXiv preprint arXiv:2504\.02605\.Cited by:[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px2.p1.1)\.
- Q\. Zheng, X\. Xia, X\. Zou, Y\. Dong, S\. Wang, Y\. Xue, Z\. Wang, L\. Shen, A\. Wang, Y\. Li, T\. Su, Z\. Yang, and J\. Tang \(2023\)CodeGeeX: a pre\-trained model for code generation with multilingual benchmarking on HumanEval\-X\.InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining \(KDD\),pp\. 5673–5684\.External Links:[Document](https://dx.doi.org/10.1145/3580305.3599790)Cited by:[§2](https://arxiv.org/html/2608.28641#S2.SS0.SSS0.Px2.p2.2)\.

## Appendix ASample Tasks

Table 6:Sample tasks in the benchmark\.Task IDCategoryDescriptionDifficultycs\-implicit\-alphabet\-frequencyCultural & OtherCount occurrences of each Czech alphabet letter in a text without being told what the Czech alphabet is — the agent must implicitly know all 42 letters including accented characters and the ‘ch’ digraph\.Mediumcs\-nbsp\-justifyCultural & OtherSplit Czech sentences into xelatex\-justified lines with no Overfull hbox, while honoring Czech orthotypographic non\-breaking\-space and soft\-hyphen rules\.Hardcs\-rozuctovani\-teplaCultural & OtherFix a script that audits a Czech SVJ heating\-cost allocation against vyhlaska 269/2015 Sb\.Mediumde\-bgb\-citation\-resolverCultural & OtherResolve a list of German civil\-code citations \(e\.g\. § 433 Abs\. 1 Satz 2\) against a BGB XML file\. Parse the hierarchical legal\-text format, count Sätze per Absatz while respecting period\-terminated legal abbreviations \(Abs\., Nr\., S\., gem\., i\.V\.m\.\), and return each citation’s exact text as JSON\.Mediumde\-umsatzsteuer\-klassifikationCultural & OtherClassify German products by statutory VAT rate \(7% reduced or 19% standard\) under §12 UStG and Anlage 2 \(Stand 1\.1\.2026\), fixing a broken keyword draft\. Many names are German compounds where the Grundwort sets the rate while the Bestimmungswort misleads, so decompose the compound first\.Hardes\-locale\-search\-analyzerText HandlingBuild a Spanish search analyzer that indexes a corpus, answers queries with locale\-correct collation, and emits snippets that preserve the original surface form\.Easyja\-terminal\-widthInternationalizationWrap each line of a NOC\-log file to a fixed 40\-column width: strip ANSI escapes, fold halfwidth katakana to fullwidth \(merging\\cjkfont̵ﾞﾟ\), measure width \(CJK/fullwidth/emoji = 2, ASCII = 1, variation\-selector clusters = one 2\-wide unit\), never splitting a wide char\. Bytes stay verbatim, so no NFC/NFKC\.Mediumsr\-cadastral\-parcel\-ledgerText ConversionNormalize Serbian cadastral parcel request text across registry aliases, script variants, area units, ownership shares, and Serbian collation into a JSON ledger\.Easysr\-locale\-sortCultural & OtherFix a buggy Serbian Latin sorter so the digraphs dž, lj and nj are treated as single letters in any word position, not only at word start\.Easytr\-username\-dedup\-casefoldText HandlingDetect duplicate user registrations in a Turkish CRM by applying the Turkish\-locale\-aware username canonicalization \(NFC \+ I\-pair pre\-fold \+ casefold\) that Python’s str\.lower\(\) and str\.casefold\(\) do not implement by default\.Hard