The IOL-AI Challenge: An Open Challenge towards Advancing Linguistic Reasoning

# The IOL-AI Challenge:
Source: [https://arxiv.org/html/2608.18011](https://arxiv.org/html/2608.18011)
An Open Challenge towards Advancing Linguistic Reasoning

affiliation=1name=Rita Berrada\\faaffiliation=2,3name= Dan\-Mircea Mirea\\faaffiliation=4name=Sara Rajaeeaffiliation=5name=Alexander Piperskiaffiliation=6name=Ana Meta Dolinaraffiliation=5name=Boris Iomdinaffiliation=7name=Andrey Nikulinaffiliation=8name=Mariya Shmatovaaffiliation=9name=Marzieh Fadaeeaffiliation=3name=Julia Kreutzer\\psaaffiliation=3

###### Abstract

Reasoning in LLMs is overwhelmingly studied in domains that provide a model with rules: mathematics and code\. Linguistic puzzles invert this: the solver must first discover the system before reasoning within it\. We present the IOL\-AI Challenge, an open\-science competition run on the unseen problems of the International Linguistics Olympiad \(IOL\) 2026 Individual Contest, evaluated both automatically and, for the first time, by members of the official IOL Jury under the same rubrics applied to human contestants\. The challenge drew 731 submissions from 46 teams under a strict compute budget \(one T4, 30 mins\)\. We additionally benchmark 15 unconstrained frontier and open models, with Claude Opus 4\.8 earning a jury score equivalent to a gold medal, while both resource\-constrained systems we submitted for jury grading scored in the range of the bottom 5% of contestants\. Capability was not determined by scale: 14B submissions outperform models twice their size, and gains come from decoding and output\-handling rather than model capacity\. We also found that automatic metrics rank systems exactly as the jury does, but compress the scale, upscoring weak systems by∼13\{\\sim\}13points and understating strong ones\. Our analysis shows that while frontier models might have prior knowledge about some of the problem languages, it does not significantly help them solve the linguistic reasoning tasks, leaving linguistic reasoning as a strong benchmarking proxy for generalizable reasoning skills\.

\\affiliations

University College London & Meta CentraleSupélec & McGill University Cohere Labs Princeton University University of Amsterdam Stockholm University Faculty of Liberal Arts and Sciences Universidade Federal de Goiás Independent

## 1Introduction

Reasoning has become one of the central paradigms for improving the capabilities of large language models\([12](https://arxiv.org/html/2608.18011#bib.bib20);[38](https://arxiv.org/html/2608.18011#bib.bib21);[46](https://arxiv.org/html/2608.18011#bib.bib9);[21](https://arxiv.org/html/2608.18011#bib.bib10)\)\. Rather than relying only on larger models, more data, or longer pretraining, recent systems increasingly improve their performance by allocating compute at inference time and generating intermediate reasoning\([55](https://arxiv.org/html/2608.18011#bib.bib22);[20](https://arxiv.org/html/2608.18011#bib.bib23)\), exploring alternative paths\([45](https://arxiv.org/html/2608.18011#bib.bib24);[7](https://arxiv.org/html/2608.18011#bib.bib25)\), verifying solutions, and revising their answers\([27](https://arxiv.org/html/2608.18011#bib.bib55);[59](https://arxiv.org/html/2608.18011#bib.bib26);[37](https://arxiv.org/html/2608.18011#bib.bib27)\)\.

The advances have been impressive yet the way we study reasoning has remained remarkably narrow\. The dominant tests of reasoning are mathematics\([53](https://arxiv.org/html/2608.18011#bib.bib28);[9](https://arxiv.org/html/2608.18011#bib.bib29);[60](https://arxiv.org/html/2608.18011#bib.bib31)\)and coding\([10](https://arxiv.org/html/2608.18011#bib.bib30);[26](https://arxiv.org/html/2608.18011#bib.bib32)\), which are domains with formal structures, unambiguous answers, and abundant automatically verifiable feedback\. However, real\-world problems often do not begin with a well\-specified system of rules\. Before solving the problem, a solver may first need to discover what the relevant structure is\. Most reasoning benchmarks abstract away this step: they provide the rules, the formalism, or the problem space, and evaluate how well a system can reason within it\.A more fundamental question is whether a system can infer a useful structure from evidence and then reason within the discovered structure\.This ability is critical for generalization: Solving new problems in unfamiliar domains requires identifying regularities, forming hypotheses, and testing them against new evidence\.

Natural language offers a particularly compelling domain in which to explore this question\. Human language is an intricate system of compositional structure that is highly variable, irregular, and context\-dependent\. Linguistic problems \(sometimes referred to as*linguistic puzzles*\)—a genre of puzzles where the solver is typically expected to disentangle linguistic phenomena of an unfamiliar language based on a limited corpus\([58](https://arxiv.org/html/2608.18011#bib.bib44);[13](https://arxiv.org/html/2608.18011#bib.bib39)\)—provide a natural testbed for this ability\. To understand a language that one has never encountered before, one must infer latent structure from sparse observations: identify recurring units, discover grammatical transformations, form hypotheses about meaning, and use these hypotheses to predict novel forms\([2](https://arxiv.org/html/2608.18011#bib.bib33)\)\.

The majority of linguistic problems come from linguistic olympiads, the largest of which is the International Linguistics Olympiad \(IOL\)111[https://ioling\.org/](https://ioling.org/)\. This annual competition unites secondary school students from many countries \(46 at the 2026 edition\), who compete in solving linguistic problems\. Each problem challenges contestants \(1\) to analyze linguistic data from one or more languages they have likely never seen before, \(2\) to deduce a set of rules that can fully explain the data, and \(3\) to use these rules to translate into and out of the unfamiliar language\. Contestants’ solutions are graded by an international body of linguists, polyglots and past IOL contestants called the IOL Jury, all of whom have substantial expertise in composing and evaluating linguistic problems\.

Prior work has uniformly reported that even frontier reasoning models consistently underperform at solving such puzzles\([19](https://arxiv.org/html/2608.18011#bib.bib1);[34](https://arxiv.org/html/2608.18011#bib.bib57);[35](https://arxiv.org/html/2608.18011#bib.bib58)\)\. We take past benchmarking efforts three steps further: \(1\) We engage the open\-science community to advance linguistic reasoning of open and resource\-constrained models in the form of a time\-bound challenge with a leaderboard:*The IOL\-AI Challenge*\. \(2\) We partner with organizers of the IOL to benchmark models on completely*new and unseen problems*from the IOL 2026 Individual Contest, which removes any concerns of benchmark leakage even for closed models\. \(3\) We obtain*expert\-level human evaluations*from the official IOL jury—which allows us to identify expert\-level judgments of the state of linguistic reasoning, where previously we had to rely on shallow automatic matching with official solutions\.

Our study thus constitutes the most recent, most thorough and in\-depth evaluation of linguistic reasoning capabilities of LLMs at scale\. In contrast to prior work, we find that proprietary models have caught up with the task, but open models lag wide behind, and under resource constraints mostly fail the task—indicating that generalizable reasoning is still a very open problem\.

## 2Related Work

Reasoning CapabilitiesThe evaluation of the reasoning capabilities of large reasoning models has recently become a principle of AI benchmarks to measure their progress\([46](https://arxiv.org/html/2608.18011#bib.bib9);[21](https://arxiv.org/html/2608.18011#bib.bib10)\)\. Reasoning datasets cover a wide range of tasks including, but not limited to, mathematics and theorem proving\([61](https://arxiv.org/html/2608.18011#bib.bib6);[56](https://arxiv.org/html/2608.18011#bib.bib7);[57](https://arxiv.org/html/2608.18011#bib.bib8)\), code generation\([30](https://arxiv.org/html/2608.18011#bib.bib11);[40](https://arxiv.org/html/2608.18011#bib.bib12)\), science and multidisciplinary\([50](https://arxiv.org/html/2608.18011#bib.bib13);[8](https://arxiv.org/html/2608.18011#bib.bib14)\), and visual puzzles\([11](https://arxiv.org/html/2608.18011#bib.bib15)\)\. Solving problems from the International Mathematical Olympiad \(IMO\)222[https://www\.imo\-official\.org/](https://www.imo-official.org/), which is widely known as the most difficult math contest, has been a big goal in mathematical reasoning\. In 2025,[28](https://arxiv.org/html/2608.18011#bib.bib51)showed how using a verification\-and\-refinement pipeline, which proved useful in mathematics given the easily verifiable correctness of the individual logic steps in the solution, drastically increases the performance of SotA modelson IMO problems and achieved a gold\-medal score\.

Linguistic Reasoning Benchmarks[51](https://arxiv.org/html/2608.18011#bib.bib61)proposed PuzzLing Machines including Linguistic Olympiad examples\. Later,[3](https://arxiv.org/html/2608.18011#bib.bib16)gathered a series of translation and linguistic reasoning problems from the UK Linguistic Olympiad, presenting the LingOly benchmark\. Similarly, Linguini, IOLBENCH, and LINGOLY\-TOO have been built using IOL problems, including a diverse set of low\-resource languages\([52](https://arxiv.org/html/2608.18011#bib.bib17);[23](https://arxiv.org/html/2608.18011#bib.bib18);[33](https://arxiv.org/html/2608.18011#bib.bib19)\)\. While prior benchmarks focus only on correct final answers, LOBSTER\([35](https://arxiv.org/html/2608.18011#bib.bib58)\)has provided step\-by\-step solutions for each problem using first an advanced LLM to generate them, and then human experts to post\-edit solutions\.

Advances in Linguistic Reasoning with AIAdvanced LLMs still struggle with linguistic reasoning, yet efforts to improve their performance remain surprisingly limited\. Trying to fill this gap,[62](https://arxiv.org/html/2608.18011#bib.bib62)has shown that LLMs’ performance improves in a multi\-turn step\-by\-step setting\. Moreover, providing analogical exemplars in the context enhances linguistic reasoning performance\([49](https://arxiv.org/html/2608.18011#bib.bib60)\)\.[19](https://arxiv.org/html/2608.18011#bib.bib1)study test\-time scaling of large\-scale LLMs and find that while benchmark performance can increase by a few points, it remains well below math and commonsense reasoning benchmarks\.

## 3Linguistic Reasoning at IOL

IOL started in 2003, growing steadily from just 36 contestants from 6 countries at its first edition in Borovets, Bulgaria to a record of 255 contestants from 46 countries at the current edition in Bucharest, Romania\. Each participating country can send one or two teams of 4 contestants, usually selected through the country’s national olympiad\. National olympiads333[https://ioling\.org/countries/](https://ioling.org/countries/)vary in size and similarity to the IOL, some having a different format in terms of number and length of linguistics puzzles\. There are other local or international linguistic olympiads, such as the Online Olympiad in Linguistics\.444[https://onling\.org/](https://onling.org/)The goals of IOL and other linguistic olympiads alike are:*to popularize linguistics at the pre\-university level, to**raise awareness of linguistic diversity and low\-resource languages globally, and to promote problem\-solving and linguistic skills beyond traditional school subjects*\. For a detailed history of linguistic olympiads and a discussion of their declared goals, the reader is referred to[39](https://arxiv.org/html/2608.18011#bib.bib34)\.

### 3\.1Linguistic Problems

The concept of linguistic problems goes back to a workbook by[22](https://arxiv.org/html/2608.18011#bib.bib43)and was first formalized by[58](https://arxiv.org/html/2608.18011#bib.bib44); more recent presentations of the genre include[13](https://arxiv.org/html/2608.18011#bib.bib39);[39](https://arxiv.org/html/2608.18011#bib.bib34);[54](https://arxiv.org/html/2608.18011#bib.bib38);[42](https://arxiv.org/html/2608.18011#bib.bib48);[41](https://arxiv.org/html/2608.18011#bib.bib47)\. A core idea behind this concept is the*principle of self\-sufficiency*, that is, the problem must be solvable without prior knowledge of any foreign languages or advanced linguistic concepts\. Low\-resource languages are typically chosen in order to minimize the risk of any solver being familiar with the language, thus offering them unfair advantage\. Solvers \(student contestants in the case of IOL\) are given enough information about a language, including but not limited to a few sentences and their translations, a numeral system, or a script sample\. To solve a given problem, students need to infer the relevant grammatical, phonological, and lexical patterns from the data, and later apply it to answer the assignments\. By default, contestants are expected to provide a theoretical explanation for their answers, using only concepts from secondary education curricula \(such as*consonant*,*suffix*, or*verb*\)\. Contestants are not required nor expected to have received prior training in linguistics—even though in recent years there has been a surge in training materials, solving problems assigned in past installments of linguistics olympiads remains the most common way of preparing for IOL\([4](https://arxiv.org/html/2608.18011#bib.bib35);[14](https://arxiv.org/html/2608.18011#bib.bib40);[15](https://arxiv.org/html/2608.18011#bib.bib41);[47](https://arxiv.org/html/2608.18011#bib.bib49);[48](https://arxiv.org/html/2608.18011#bib.bib50);[54](https://arxiv.org/html/2608.18011#bib.bib38);[41](https://arxiv.org/html/2608.18011#bib.bib47)\)\.

Linguistic problems are often classified by the subdiscipline of linguistics they illustrate—e\.g\., phonetics/phonology, morphology, syntax, lexical semantics, kinship terminology, number terms\([16](https://arxiv.org/html/2608.18011#bib.bib42)\), computational concepts\([36](https://arxiv.org/html/2608.18011#bib.bib46)\), corpus linguistics\([29](https://arxiv.org/html/2608.18011#bib.bib45)\), writing systems\. They can also be classified by problem structure\([39](https://arxiv.org/html/2608.18011#bib.bib34);[42](https://arxiv.org/html/2608.18011#bib.bib48);[41](https://arxiv.org/html/2608.18011#bib.bib47)\), the most common structures being “Rosetta stone problems”\([6](https://arxiv.org/html/2608.18011#bib.bib36)\), which involve translation or transcription assignments based on a bilingual corpus; and “chaos\-and\-order problems”, which involve matching assignments\.

Table 1:Examples from the IOL 2026 Individual Contest, structured into context and question to prompt LLMs\.
### 3\.2The IOL Problem Set

The IOL problem set consists of five problems meant to be solved individually by each contestant within 6 hours \(the Individual Contest\) and one problem designed to be solved collaboratively by national teams of 4 \(the Team Contest\)\. Any working language may be requested for the competition and assignments are equivalent in all working languages \([17](https://arxiv.org/html/2608.18011#bib.bib37)\)\. The problem set is crafted and multilingually rendered by the IOL Problem Committee in the year prior to each IOL\.

The individual problem set at IOL 2026 included a problem on phonetics and orthography in Central Alaskan Yup’ik \(P1\); a chaos\-and\-order problem on lexical semantics, in particular semantics of color terms, in Yélî Dnye \(P2\); a Rosetta stone problem on syntax in Iquito \(P3\); a chaos\-and\-order problem on kinship terminology in Sakurabiat \(P4\); and a Rosetta stone problem on verb morphology in Komnzo \(P5\)\.

### 3\.3Grading

Each problem is graded by a subset of the IOL jury\. In the weeks leading to IOL, after the problem set has been assigned, each grading group develops a grading scheme for their problem, deciding on which criteria to grade and how to weight them\. Grading schemes allocate points to assignments and the theory part \(the explanation of the rules\) separately, as criteria: each assignment is usually one criterion, whereas the theory portion contains multiple criteria, each corresponding to a main rule that the solvers need to identify\.

At IOL, each contestant’s submission for a problem is graded independently by at least two jurors within the grading group, with translation assistance from other jurors or external language specialists if needed\. In the event of a discrepancy between two or more jurors regarding any single criterion, the discrepancy must be resolved through discussion until consensus is achieved\.

## 4The IOL\-AI Challenge 2026

The challenge was set up as an open\-science competition preceding the IOL \(outreach efforts described in[AppendixA](https://arxiv.org/html/2608.18011#A1)\), with a compute\-restricted testing environment and a truly unseen test set that no participants had access to\. Submissions were accepted for a month until July 26, 2026 when the on\-site IOL competition started\.

### 4\.1Technical Setup

The TaskThe dataset includes the data from the 5 problems of the English version of the IOL 2026 Individual Contest, a total of 14 sub\-assignments or tasks that these problems collectively contain\. We distinguish 3 types of tasks: \(1\) Translation, where given a context, it is asked to translate a sentence from “Solverese” \(here: English\) to an unseen extremely low\-resource language, \(2\) Fill\-in Blanks, where given a set of examples, the model is asked to fill the missing forms, and \(3\) Match Letters, where the task is to match a set of phrases to their correspondences\.[Table1](https://arxiv.org/html/2608.18011#S3.T1)presents an example of test data with different types\. We split the tasks into a development \(8 tasks\) and a test set \(6 tasks\) balanced across task types, where the development set is used for computing automatic scores on a public leaderboard, while scores on the test set are not revealed during the competition, but only after its end, as part of a private leaderboard that contains scores for the complete set of tasks\. One task is excluded from automatic scoring as it requires only an explanation that cannot be graded automatically\. Submissions eligible to participate in the human evaluation had to provide a short textual explanation with every task solution as well\.

FormatWe follow Linguini in the task format\([52](https://arxiv.org/html/2608.18011#bib.bib17)\)where every data point has a context containing the given problem and a query for the asked question\. Most problems require returning a list of answers, such as filling in a list of blanks, or an ordered list of letters\. Reference solutions may also contain multiple valid answers for each problem, if there are any ambiguities\. Any visual clues are turned into text, and diagrams into JSON format, following the setup in\([34](https://arxiv.org/html/2608.18011#bib.bib57)\)\.

Automatic EvaluationFollowing previous work on linguistic reasoning\([52](https://arxiv.org/html/2608.18011#bib.bib17);[3](https://arxiv.org/html/2608.18011#bib.bib16)\), we employ ChrF\([44](https://arxiv.org/html/2608.18011#bib.bib59)\)and exact match \(EM\) to evaluate the final answers\. We report the geometric mean \(GM\) of these metrics because ChrF will likely overestimate true answer quality \(e\.g\., partial matches can be obtained from copying words from the task\), and EM will likely underestimate it \(e\.g\. answer formatting or minor morphological variations\)\. ChrF and EM are first computed on each individual example \(taking the highest score across reference solutions if there are multiple\), then averaged across examples\.555At the time of the AI competition, points were not assigned to tasks yet, so while the final human evaluation is point\-weighted \(more difficult problems giving more points\), the automatic scores are uniformly aggregated across problems\.

ImplementationThe challenge runs on Hugging Face \(HF\), built on the open\-sourcecompetitionsframework\. A submission is a public HF model repository holding a self\-contained inference script alongside the model weights themselves, that is run on an evaluation sandbox without network access and writes answers and optionally explanations into an output file\. This evaluation sandboxing is necessary to prevent leaking the IOL\-AI data\. The dataset was uploaded directly by members of the IOL jury and problem committee \(AP, DMM\) to a private HF dataset\. Neither participants nor other organizers ever saw the data\. Each submission runs as an isolated HF Job on a single NVIDIA T4 GPU \(16 GB VRAM, 8 vCPUs, 30 GB RAM\) under a 30\-minute wall\-clock limit\.

After each run, teams receive the score on the dev split \(scored server\-side based on the output file\); failed submissions additionally return an excerpt from the error log \(e\.g\., “time limit exceeded”\)\. Teams have up to 10 submissions per day and select two for the private leaderboard\.

### 4\.2Baselines

We provided four basic baselines, submitted through the same pipeline as participant entries\. They relied on four different open\-weights models of different scales, with slightly different amounts of prompt customization\.[AppendixC](https://arxiv.org/html/2608.18011#A3)describes them in detail\. The baselines were visible to participants on the leaderboard throughout the competition\.[Table2](https://arxiv.org/html/2608.18011#S4.T2)reports their automatic scores\.

Table 2:Automatic evaluation scores of organizer baselines\. EM = exact match, GM = geometric mean\.
### 4\.3Extended Model Benchmarking

To complement the resource\-constrained submissions, we also benchmark 15 existing reasoning models \(details in[AppendixD](https://arxiv.org/html/2608.18011#A4)\), including “frontier” models \(open and proprietary\), i\.e\., the largest, most resource\-consuming, most recent, and reasoning\-heavy models, and open “mid\-size” models to bridge the gap to the IOL resource constraints\. Given that the latter ones are individually not competitive with the former, we aggregate the answers of four mid\-sized models \(Gemma 4 31B, Qwen 3\.6 27b, DeepSeek R1 32B, GLM 4\.7 Flash\) in a best\-of\-nnensemble \(BoN; oracle selection: choosing the best scoring model for each task\) to measure the ceiling of what these models can currently achieve, inspired by earlier ensemble success\([19](https://arxiv.org/html/2608.18011#bib.bib1)\)\.

### 4\.4Human Evaluation at the IOL

We consider the top 10 challenge submissions with provided explanations along with the best closed\- and open\-source reasoning models for human evaluation, before selecting five systems to undergo human evaluation \(see[Section5\.2](https://arxiv.org/html/2608.18011#S5.SS2)\)\. Human evaluation is performed by a subset of the IOL jury \(at least 2 graders per problem\) using a very similar procedure to the grading of contestant solutions at IOL, including anonymization with respect to the model information and resolution of discrepancies between jury members through discussion\. Details are described in[AppendixG](https://arxiv.org/html/2608.18011#A7)\.

## 5Results

We first discuss the results from the automatic evaluation stage \([Section5\.1](https://arxiv.org/html/2608.18011#S5.SS1)\), covering a larger set of models, before zooming in on the five selected submissions for the human evaluation \([Section5\.2](https://arxiv.org/html/2608.18011#S5.SS2)\)\. Lastly we perform an analysis on the knowledge leveraged by top\-scoring models \([Section5\.3](https://arxiv.org/html/2608.18011#S5.SS3)\)\.

### 5\.1Automatic Evaluation

#### 5\.1\.1Submissions from the Open Challenge

We received 731 submissions \(80\.7% ran successfully\) from 46 external teams over the one\-month window\. We describe submission statistics in more detail in[AppendixH](https://arxiv.org/html/2608.18011#A8), and focus on the top 10 here, listed in[Table3](https://arxiv.org/html/2608.18011#S5.T3)\. Nine of the top ten entries run a quantized 14B Qwen model, the largest model to comply with the compute constraint\. The best submission more than doubles our strongest baseline with the same model \(GM 9\.40→\\rightarrow19\.79\), showing that decoding and output handling are effective levers under constrained model size\.

Table 3:Automatic evaluation scores for the top 10 from the leaderboard of the IOL\-AI challenge\.Submissions with⋆\\starwere selected for jury evaluation\. Dashed lines indicate first, second, and third place winners on the leaderboard\.Technique adoption and measured effectsWe inspected the archived repositories and tagged the techniques used in each submission\. Where a team submitted runs both with and without a technique, we measure its effect as the within\-team difference in mean private score \(nn= teams with runs of both kinds\), this controls for stronger teams simply trying more things\. Greedy decoding was near\-universal \(38 teams\), and 21 teams additionally used temperature sampling, mostly for retries\. Self\-consistency voting \(24 teams\) adds\+2\.08\+2\.08points \(n=21n=21\)\. Retry or reformat loops \(17 teams\) add\+1\.40\+1\.40\(n=14n=14\)\. Chain\-of\-thought prompting was widely tried \(33 teams\) but costs−1\.55\-1\.55\(n=25n=25\) and none of the top ten submissions used it\. JSON\-structured output \(16 teams\) is the most damaging at−4\.66\-4\.66\(n=10n=10\)\. Fine\-tuning or LoRA on past puzzles brought no gain, and neither did retrieval, consistent with problems that are self\-contained by construction\.

Leading ApproachesLeading submissions converge on a common recipe\. They lower the repetition penalty from its default greedy\-mode value of 1\.05 to 1\.0, a silent parameter that penalizes generating the same characters several times in a row, a frequent pattern in these languages\. None uses chain\-of\-thought prompting: it was tried repeatedly, and runs without it consistently scored higher\. All top entries anchor on a greedy pass, and when they sample, draw retries at temperature 0\.5 as time allows, replacing the greedy answer only on agreement\. Token budgets are set adaptively from the wall\-clock time remaining, and the submission file is written incrementally\. Part of this convergence reflects copying, since submission repositories were public throughout the competition: the 1st and 5th entries run the same script, differing only in whitespace, and their score difference reflects run\-to\-run variance rather than any difference in method\. We return to this in the Limitations\. The individual pipelines are described in[AppendixH](https://arxiv.org/html/2608.18011#A8)\.

#### 5\.1\.2Extended Models

Table 4:Automatic evaluation scores for the extended model set on the IOL 2026 problems\(after custom answer parsing\)\. LFMR = line format match rate\. Model outputs that were chosen for the jury are marked with⋆\\star\.[Table4](https://arxiv.org/html/2608.18011#S5.T4)contains the automatic evaluation scores for our selected 15 individual models and the ensemble\. We report line format match rate \(LFMR\) besides the quality scores to indicate where low scores are due to low format compliance\. Note that all submissions undergo a custom answer parsing process that e\.g\. strips markdown \([AppendixD](https://arxiv.org/html/2608.18011#A4)\)\. As expected from evaluations in prior work, proprietary models perform strongest, with a noticeable 19\-point gap between Claude\-Opus\-4\.8 and the following Gemini\-3\.6\-Flash and GPT5\.6\-Sol\. Open \(large\) models follow after a 7 point gap, with a large range in scores, going down to 10\.6\. Of the mid\-sized models, Gemma4\-31B\-It scores highest \(23\.2\), and the ensemble gains another 10 points on top of it, catching up with some of the larger open models\. Performance is therefore not strictly determined by scale: as[Figure1](https://arxiv.org/html/2608.18011#S5.F1)shows, resource\-constrained submissions at 14B double the organizer baselines and match or exceed the unconstrained open mid\-size tier, though all open systems remain far below the frontier\. As linguistic reasoning is not a typical post\-training task, and most models struggle with the output format, “tweaking” inference goes a long way\.

![Refer to caption](https://arxiv.org/html/2608.18011v1/images/GM_by_tier.png)Figure 1:Distribution of automatic scores \(GM\) by system tier, spanning from organizer baselines and resource\-constrained competition submissions to open mid\-size, open large, and proprietary frontier models\.
#### 5\.1\.3Comparison across Tasks

Among the top 10 submissions \([Figure2\(a\)](https://arxiv.org/html/2608.18011#S5.F2.sf1)\), difficulty separates cleanly by problem\. Iquito \(P3\) is the hardest: both of its tasks score 0% exact match for all ten teams\. Yup’ik \(P1\) and Yélî Dnye \(P2\) follow closely, with no team solving Yup’ik writing or either Yélî Dnye translation direction, and only the closed\-form items \(Yélî Dnye letter and picture matching, Yup’ik letter matching\) earning partial credit\. On these unsolved tasks, chrF often remains in the 23 to 53 range, indicating that answers contain some correct components\. Submissions perform best on Sakurabiat \(P4\), whose short, closed items \(birth order, number matching\) are answered by nearly every team\. Komnzo \(P5\) shows the largest asymmetry within a single problem: Komnzo→\\toEnglish is the best\-solved task across all submissions, while English→\\toKomnzo receives 0% throughout\. Teams are thus able to analyze Komnzo forms well enough to translate them into English, but not to produce correct forms in the language itself\.

Frontier models \([Figure2\(b\)](https://arxiv.org/html/2608.18011#S5.F2.sf2)\) score higher on every problem but rank them differently\. Yélî Dnye \(P2\) becomes the hardest problem \(about 20% geomean\), with both translation directions near 11%, whereas Iquito \(P3\), unsolved by all submissions, moves to the middle of the range\. Sakurabiat \(P4\) remains the easiest \(about 55%\), and the Komnzo \(P5\) asymmetry persists: English→\\toKomnzo stays low \(about 28\.5%\) even for models that succeed in the opposite direction\. On most problems, scores follow overall model strength, but translation tasks diverge from this pattern: only proprietary models obtain non\-zero scores on Yélî Dnye→\\toEnglish, while the same models fail Komnzo→\\toEnglish, where several open models earn substantial credit\.

![Refer to caption](https://arxiv.org/html/2608.18011v1/images/heatmap_submissions.png)\(a\)Top 10 submissions from the IOL\-AI challenge\.
![Refer to caption](https://arxiv.org/html/2608.18011v1/images/heatmap_frontier.png)\(b\)Frontier models\.

Figure 2:Breakdown of scores \(geomean %\) across the individual tasks of the 5 IOL 2026 problems, for the top 10 challenge submissions \(top\) and frontier models \(bottom\)\.

### 5\.2Human Evaluation by the IOL Jury

We selected the top 2 frontier models according to their automatic evaluation scores \(Claude Opus 4\.8 and Gemini\-3\.6\-Flash\) for human evaluation\. From the top submissions we selected the first one, and then manually went through subsequent ranks and verified the explanations, and found that many of them were either templated or meaningless \(copies of the same sentence\)\. The submission ranked 11th stood out as it provided structured explanations with schemata, so we selected it to provide a contrastive solution\. From the open mid\-size models, we submitted the ensemble, representing the ceiling of what a team of mid\-size models can achieve if assembled right\. In this way, we hope to capture the full spectrum from heavily tuned, severely resource\-constrained systems, over an ensemble of mid\-size open models, to proprietary frontier models\.

Table 5:Juryevaluation results by problem for the selected AI submissions compared to the automatic GM scores \(Aut\.\)\.#### 5\.2\.1Quantitative Comparison

The scores given by the IOL jurors showed the same rank\-order of the models as the automatic score \([Table5](https://arxiv.org/html/2608.18011#S5.T5)\)\. Notably, Opus 4\.8 received a score equivalent to that of an IOL gold medalist \(lowest gold medal this year: 70\.0\), a score that would have placed it in fourth place in this year’s ranking\. An example of its explanations for P5 are shown side\-by\-side with the reference solution in[Figure3](https://arxiv.org/html/2608.18011#S5.F3)\. Gemini 3\.6 received a score equivalent to that of a silver medal \(lowest silver medal this year: 58\.4\)\. The rest of the models were below the threshold for honorable mentions \(35\.3\), meaning they all performed worse than 50% of contestants this year\. In fact, the two models from the challenge would have both ranked in the bottom 5% of scores\. Agreement between the two protocols is near\-perfect in rank \(Spearmanρ=1\.00\\rho=1\.00, Pearsonr=0\.99r=0\.99,n=5n=5\) but not in magnitude\. The two proprietary models score*higher*with the jury than automatically \(\+4\.4\+4\.4and\+4\.1\+4\.1points\), whereas the ensemble and both challenge submissions score substantially*lower*\(−13\.2\-13\.2,−13\.9\-13\.9and−12\.1\-12\.1\); the jury’s range therefore spans 77\.0 points against 60\.5 automatically\.[Figure4](https://arxiv.org/html/2608.18011#S5.F4)breaks these differences down by problem: the automatic metric sits above the jury on nearly every problem for the weaker systems, while the direction flips for the two proprietary models on P1, P3 and P5\. Automatic scores overestimate the quality of the weaker submissions, due to absence of explanation scoring and overestimating influence of ChrF in the geometric mean; conversely, strong systems are under\-credited, because a correct analysis stated in prose earns jury points but nothing automatically\.

\(a\)IOL solution explanation\.\(b\)Claude 4\.8 Opus explanation\.
Figure 3:Schematic explanations provided in official IOL solutions for P5 in comparison to the top scoring model’s solution explanation that we elicit through prompting\.![Refer to caption](https://arxiv.org/html/2608.18011v1/images/aut_vs_jury.png)Figure 4:Difference between automatic and jury scores per problem for the five systems selected for human evaluation\. Cla = Claude Opus 4\.8, Gem = Gemini 3\.6 Flash, BoN = BoN Open \(oracle\), \#1 = arvindcr4, \#11 = cabanosss\.
#### 5\.2\.2Qualitative Comparison

Beyond the quantitative differences, the IOL jurors reported several recurring qualitative differences between the AI solutions and the human solutions they usually grade:

1. 1\.Local rather than consistent explanations\.Because our prompting procedure elicited a separate explanation for each task within a problem, the models sometimes gave different, and occasionally contradictory, analyses of the same phenomenon\. Strikingly, in P2–Yélî Dnye, the Gemini model proposed an incorrect theory in which nouns for objects were formed by reduplication from color terms for earlier assignments\. However, for assignment 4 \(which was unusual in that it explicitly asked for an explanation of how color terms are derived\), the model fully switched its explanation to the canonical one \(that some color terms are formed by reduplication while others are formed by comparison\)\. This suggests that the model was in principle able to recover the right set of rules, but that it did not get to this explanations in the other assignments\.
2. 2\.Verbosity: explanation traces and examples\.The models were generally more verbose than human contestants\. They often gave entire explanation traces, something that contestants are explicitly instructed not to do, and which they can learn not to do from seeing canonical solutions\. The models also often tried to explain a rule by giving an example sentence instead of the specific rule application \(e\.g\. in P3–Iquito, giving an example verb in the past tense and an example verb in the future tense, rather than stating the suffixes used to mark each tense\), which is likewise not rewarded with any marks\.
3. 3\.Hallucinations\.Lower\-performing models sometimes hallucinated data \(e\.g\., new terms such ascompletely blackin P2–Yélî Dnye, based onblackandcompletely whitewhich appear in the problem; or sounds that do not appear in the problem, such as \[\] or syllable\-initial consonant clusters in P1–Yup’ik\), outputted impossible answers to assignments \(e\.g\., wrong number of syllables due to miscounting in P1\), or provided non\-sensical explanations and examples \(e\.g\.,arvindcr4writing in P5\-Komnzofor example, “thafa” \(hold\) with “th” suffix indicates the subject, and “wath” indicates the object\)\.
4. 4\.Underspecified rules\.Lower\-performing models tended to hedge in their explanations using words such astypicallyoroften\(e\.g\.,arvindcr4in P3–Iquito:The verb typically comes after the subject, which is always true and hencetypicallyis not needed\)\. At IOL, such qualifications matter: a rule is expected to state precisely what the data support\.
5. 5\.Undergeneralization\.The frontier models tended to miss broad patterns that human contestants sometimes correctly delineated\. For instance in P5–Komnzo, no model parsed out morpheme\-a\-appearing in sentences in non\-recent past\. This suggests AI models could have difficulty generalizing from sparse data during linguistic reasoning\.
6. 6\.Overfitting or convoluted explanations\.Explanations were often more complex than those of human contestants \(e\.g\., in P5–Komnzo, Gemini incorrectly parsed the\-s\-stem fromnznästhandzwäsathand added a phonological modification \(The punctual stem for “call” \-näs\- reduces to \-äs\- when preceded by the labial glide /w/ in the 1s punctual prefix zw\- \(e\.g\., zw\-näs\-ath → zwäsath\)\.\)
7. 7\.Technical terminology without a complete analysis\.The frontier models \(Claude and Gemini\) sometimes used technical linguistic terminology \(e\.g\.,iambic feetin P1–Yup’ik,durative aspectin P5–Komnzo, or a specific kinship\-system annotation \(MBfor mother’s brother andFZfor father’s sister in P4–Sakurabiat\)\. However, these terms were not always used consistently or completely\. \(e\.g\., writingMB’s soninstead of the usualMBSin kinship terminology\)\.

Overall, we found that models do not fail to identify linguistic structure altogether—the strongest models clearly are able to achieve medal\-level scores\. However, there are differences in how that structure is used across a problem: human solutions tend to build a compact analysis that explains the data as a whole, while model solutions more often reason from one assignment to the next\. This difference is easy to miss when evaluation considers only the final answer, but becomes much more visible when the theory explanations themselves are graded\.

On top of the differences, the jury also highlighted similarities to human contestants’ solutions, such as the order in which phenomena are described in the explanations \(e\.g\., tending to start with word order in P5–Komnzo\), or the rank\-order of difficulty of the different phenomena\. Indeed, in line with the jurors’ impressions, the difficulty level of all theory and explanation criteria across the 5 grading schemes \(measured as the % models or contestants that met the criterion, i\.e\., solved that subunit of the problem correctly\) correlated strongly between humans and models \(Spearmanρ=0\.57\\rho=0\.57;[Figure5](https://arxiv.org/html/2608.18011#S5.F5)\)\. This link was observed despite the fact that some problems seemed overall easier for models \(P5–Komnzo\), whereas others were easier for human contestants \(P3–Iquito\)\. This finding suggests that linguistic reasoning comes in different levels of difficulty that is to some extend intrinsic, irrespective of whether the solver is human or machine\.

![Refer to caption](https://arxiv.org/html/2608.18011v1/images/criterion_difficulty_scatter.png)Figure 5:Comparison of the difficulty of criteria in the official IOL grading schemes between human contestants and models\. Difficulty is measured as the percentage of solutions that meet the criterion and receive the respective mark, split into human contestants and AI models\.

### 5\.3Analysis: Knowledge Probing

A key question is which knowledge successful models rely on \(not only on reasoning capabilities\)—one may speculate whether they post\-trained on linguistic tasks, or have included grammar books in their pretraining more than others\. We cannot perform training data analysis for most benchmarked models due to missing data releases, but we can estimate proxies of their leverage of prior knowledge by relating the following estimates to problem solving success:

1. 1\.Measure availability ofonline resourcesin or about the problem languages \(web crawl statistics, Wikipedia article length, number of Glottolog\([24](https://arxiv.org/html/2608.18011#bib.bib5)\)listed resources\)\. This estimates how much the models could have had prior exposure with the language or words in that language\.
2. 2\.Probe models about theirexplicit knowledge about the languagesby turning the contextual language facts of the problem context in a multiple\-choice task, and the lexical items from the problem language into a language identification task \(also multiple\-choice\)\. This gives us proxies about \(a\) prior language knowledge and \(b\) lexical familiarity or ability to deduce language identity from single words\.
3. 3\.Remove the language name and background informationfrom the problem context and measure how scores change across problem languages\. This tell us which models are able to leverage this information to bias their predictions, and how much they rely on it for solving the problems\.

Presence on the webAll problem languages would fall under the category of \(extremely\) “low\-resourced” languages in NLP\([31](https://arxiv.org/html/2608.18011#bib.bib56)\)beyond the scope of mainstream AI technology development\([5](https://arxiv.org/html/2608.18011#bib.bib53)\)\. However, Yup’ik \(esu\) and Yélî Dnye \(yle\) documents can be found in FineWeb\-2\([43](https://arxiv.org/html/2608.18011#bib.bib4)\)and GlotCC\-V1\([32](https://arxiv.org/html/2608.18011#bib.bib2)\)web crawls,666According to iso codes; without native speaker confirmation that these documents are actually in the languagesbut with less than 1MB each\. Yup’ik and Komnzo are each part of publicly available \(small\) speech or text corpora\([1](https://arxiv.org/html/2608.18011#bib.bib52);[18](https://arxiv.org/html/2608.18011#bib.bib3);[25](https://arxiv.org/html/2608.18011#bib.bib54)\)\. This means that there is a slight chance that models have encountered words in these languages before\. Short Wikipedia articles in the English Wikipedia exist for all five languages, with the ones about Yup’ik and Yélî Dnye being the most detailed, containing grammar and phonology descriptions \(Iquito and Komnzo: only phonology; Sakurabiat: shorter phonology, grammar\)—which means that models might recall some linguistic knowledge about these languages, if they have sufficiently memorized this data\. Iquito and Yup’ik have the largest number of resources listed in Glottolog \(around 50–60\)—resources that can be accessed if publicly available, but are not as likely to be included in pretraining data\. To summarize, Yup’ik is the most present in public web sources, and Iquito \(P3\) is the most documentationally rich but has a small general\-web footprint\. Hence, LLMs are likely to be most familiar with contents in Yup’ik \(P1\) and Yélî Dnye \(P2\) \(if at all\) and their grammars and phonology\. According to jury scores \([Table5](https://arxiv.org/html/2608.18011#S5.T5)\), Yup’ik and Sakurabiat were the best solved problems, but models struggled the most with Yélî Dnye, which indicates thatweb presence is not a reliable predictor of difficulty for this problem set\.

\(a\)Accuracy of frontier models on 4\-way MCQ knowledge probe tasks derived from IOL 2026 problems\. Language Context: questions targeting language context, Lexicon: identifying real forms from the problem language of a set of four options\.
\(b\)Correlating knowledge probe accuracy with overall IOL 2026 performance \(automatic score, GM\)\.

Table 6:Results of knowledge probing tasks for a subset of benchmarked models\.Direct knowledge probesWhen we directly a probe a subset of 9 of the large benchmarked models for either contextual or lexicon knowledge about each problem language \(details in[AppendixK](https://arxiv.org/html/2608.18011#A11)\), we find that they all score fairly highly \(random guess baseline would score at 25%\), as shown in[Table6\(a\)](https://arxiv.org/html/2608.18011#S5.T6.st1)\. All models score consistently higher on predicting contextual information about the languages \(72–94%\) than identifying lexical items \(37–75%\), indicating that theylikely know more about the language than about words in the language\. Comparing their mean accuracy in probing tasks with their ranking according to automatic \(GM\) scores on IOL 2026, we find a positive correlation between the two \(Pearson: 0\.38, Spearman: 0\.43\)\. Of the two sub\-tasks, correlation is higher for the lexicon probe, indicating this might be a more reliable predictor for task performance overall\. This finding is strictly correlational and does not allow causal conclusions—it might be an effect of scale, generally stronger reasoning or better memorization of e\.g\. Wikipedia, rather than indicating any linguistics\-specific training\.

Table 7:Automatic score change when language mentions and context are dropped from the task descriptions \(without custom answer parsing that would raise all scores\)\. Scores are aggregated across three repeated runs with different random seeds\.Table 8:Automatic score change on each problem for an average across three runs from models in[Table7](https://arxiv.org/html/2608.18011#S5.T7)\.Linguistic reasoning without language meta\-informationEach IOL task comes with a rich context and also background information about the language, e\.g\. language family, script, geographic region where it is spoken\. Models \(and humans\) might leverage this knowledge to transfer from related languages they might be more familiar with\. When we drop all mentions of the language and remove this context, we observe on average a loss in performance \([Table7](https://arxiv.org/html/2608.18011#S5.T7)\), but it is not significant according to 95% confidence intervals with three repeated runs\. Inspecting the effect per problem language \([Table8](https://arxiv.org/html/2608.18011#S5.T8)\), however, reveals that this removal of context is significantly hurting average model performance for Yélî Dnye \(P2\), with 6/7 models dropping an average of 5\.8 percentage points\. For this problem set, which is also the most difficult for all models, the presented language context is consistently the most helpful, because the language name itself contains a cue,777The language name “Yélî Dnye” constitutes a crucial cue for solving the problem 2b\), item 32 \(“nkéli dnye” translating to “English language”\) as “dnye” means “language”\.but overall the effect is small and noisy\.

Qualitative inspection of reasoning tracesFor models that disclose their reasoning traces \(not proprietary models\), we can also inspect whether they refer to typological or language family information about the problem language \(given or not\)\. A qualitative inspection reveals that language meta\-information is indeed frequently present in the reasoning traces across models\. This pattern is strongest for Sakurabiat problems \(P4\), where models refer to Tupian kinship structures and respective relational nouns, but as shown above there is no consistent effects on task scores when this association is removed\. For Yélî Dnye \(P2\), however, models refer to priors from Papuan/Oceanic/Austronesian languages, exploring answer paths beyond the explicitly stated information \(the problem states that it is a language isolate\)\. This analysis shows that reasoning models will try to leverage prior knowledge in their reasoning, but we cannot conclude that it will help them to solve the task\.

Overall, this analysis confirms that even when models have some knowledge about the problem languages or are familiar with words of these languages,it does not determine their ability to solve the respective linguistic problems\. The contextual meta\-information given with each problem might bias their priors, but the difficulty, as per design of the IOL, remains in deducing patterns from the presented linguistic context, and generalizing them to new forms\.

## 6Conclusion

We introduced the IOL\-AI challenge, the first evaluation of language models on genuinely unseen linguistic\-olympiad problems, graded by the expert jury that grades the human contestants\. Because the problems were unreleased, no model could have seen them or their solutions during training, ruling out benchmark leakage by construction\. The problem languages themselves have some public documentation, but our knowledge probing shows that what models know about them does not determine success\. Solving these problems requires inferring the linguistic system from the given data\. This makes linguistic reasoning a true test of model reasoning capabilities\.

The open challenge showed that inference\-time techniques have a large impact under tight resource constraints\. The winning submission more than doubled our strongest baseline with the same 14B model, with the margin coming from decoding and answer handling rather than model capacity\. Prior work reported that frontier models uniformly underperform on linguistic puzzles\. This is no longer the case: for the first time, models reached medal level\. Claude Opus 4\.8 scored above this year’s gold\-medal cutoff, equivalent to fourth place among 255 contestants, and Gemini 3\.6 Flash reached silver\. But this progress is confined to proprietary systems whose training and reasoning remain inaccessible, so it does little to advance open research on reasoning\. No resource\-constrained submission cleared the honorable\-mention threshold, both scoring in the bottom 5% of contestants, and even an oracle ensemble of mid\-size open models lands well short of the frontier\. Closing this gap with open, efficient systems is the central problem this challenge poses\.

Expert grading also changes what the scores mean: it is under the jury’s rubrics, not string matching, that frontier models reached medal level\. Comparing the two scoring methods, automatic metrics rank systems exactly as the jury does \(ρ=1\.00\\rho=1\.00,r=0\.99r=0\.99\), but compress the scale, inflating weak systems by∼13\{\\sim\}13points and under\-crediting strong ones, whose correct analyses in prose earn nothing automatically\. We release jury evaluations, model outputs, and scores888[https://github\.com/Cohere\-Labs\-Community/iol\-ai\-2026\-results](https://github.com/Cohere-Labs-Community/iol-ai-2026-results)to support progress on both linguistic reasoning systems and their measurement\.

## Limitations

Format RequirementsWe noticed that most models, including frontier models, struggle with the required answer format, outputting a list of individual solutions\. As a consequence, many submissions to the IOL\-AI challenge focused on working around format artifacts, and we had to craft post\-hoc format fixing rules for frontier model outputs to make sure their output format does not perturb automatic score rankings\. Future benchmarking for linguistic reasoning should re\-evaluate which prompting style is most favorable for LLMs across model scales\.

Test Set Size & DiversityThe evaluation on the IOL 2026 Individual Contest problems focuses on depth rather than breadth\. As a consequence, the total number of test samples is small for testing ML models and the statistical power low\. Since these tasks can be assumed to be out\-of\-domain for most models, the risk for high variance across multiple outputs under stochastic decoding is high, especially with long reasoning traces\. Small differences in automatic scoring should therefore not be over\-interpreted\. The score differences between the AI solutions participating in human evaluation completely align with the jury ranking, so we can be confident that differences of that magnitude \(min 5 points in GM\) tend to be meaningful and human noticeable\.

Oracle EnsembleThe BoN Open ensemble uses oracle selection against the reference answers\. It therefore reports an upper bound on what this pool of mid\-size open models could achieve under perfect routing, rather than the performance of a deployable system\. A further caveat is that a single model \(Gemma4\-31B\-It\) supplies the majority of the selected answers, so the ensemble’s advantage is only partly attributable to model diversity\.

Unequal Generation BudgetsWithin the mid\-size tier, DeepSeek\-R1\-32B was run with a 32,768\-token generation budget against 64,000 for the other three models \([Table11](https://arxiv.org/html/2608.18011#A9.T11)\), so its scores are not strictly comparable within that tier and likely understate it\.

Time LimitationThe open\-science competition ran for a short time window of a month, and a longer duration might have gathered more submissions, or have allowed participants to improve their submissions further\.

Independence of SubmissionsWe required all submissions to be public repositories on HuggingFace, so it was possible for participants to search for each other’s submission and potentially copy it\. While teaming up after individual submissions was explicitly forbidden, this remained a loophole\. As a consequence, we have many similar submissions, and top\-scoring submissions might not credit the work of other participants that laid the foundation\. Nevertheless, building on top of other peoples’ work and improving upon it is also in the spirit of advancing the state of the art\.

## Acknowledgments

We thank the participants of the challenge for their contributions to advancing the understanding of linguistic reasoning challenges for LLMs\. This work would not have been possible without the generous support of additional members of the IOL 2026 Jury – Mihai\-Alexandru Bratu, Aida Davletova, Jan Petr, Eimear McKnight, Stanislav Gurevich – who evaluated the submissions\. We especially thank Mihai, Aida and Jan for providing written impressions of the solutions\. In addition, the dataset for this challenge would have not existed without the work of the greater IOL Problem Committee and the authors of this year’s problems—DMM, Eimear McKnight, Lai Otsuka, and Vesko Milev\. We also thank Cohere as sponsors of the compute resources for the challenge\.

## References

- Ardilaet al\.\(2020\)R\. Ardila, M\. Branson, K\. Davis, M\. Kohler, J\. Meyer, M\. Henretty, R\. Morais, L\. Saunders, F\. Tyers, and G\. WeberCommon voice: a massively\-multilingual speech corpus\.InProceedings of the Twelfth Language Resources and Evaluation Conference,N\. Calzolari, F\. Béchet, P\. Blache, K\. Choukri, C\. Cieri, T\. Declerck, S\. Goggi, H\. Isahara, B\. Maegaard, J\. Mariani, H\. Mazo, A\. Moreno, J\. Odijk, and S\. Piperidis \(Eds\.\),Marseille, France,pp\. 4218–4222\(eng\)\.External Links:[Link](https://aclanthology.org/2020.lrec-1.520/),ISBN 979\-10\-95546\-34\-4Cited by:[§5\.3](https://arxiv.org/html/2608.18011#S5.SS3.p2.1)\.
- Aycocket al\.\(2025\)S\. Aycock, D\. Stap, D\. Wu, C\. Monz, and K\. Sima’anCan LLMs really learn to translate a low\-resource language from one grammar book?\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=aMBSY2ebPw)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p3.1)\.
- Beanet al\.\(2024\)A\. M\. Bean, S\. Hellsten, H\. Mayne, J\. Magomere, E\. A\. Chi, R\. A\. Chi, S\. A\. Hale, and H\. R\. KirkLINGOLY: a benchmark of olympiad\-level linguistic reasoning puzzles in low resource and extinct languages\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=cLga8GStdk)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p2.1),[§4\.1](https://arxiv.org/html/2608.18011#S4.SS1.p3.1)\.
- V\. I\. Belikov, E\. V\. Muravenko, and M\. E\. Alexeev \(Eds\.\) \(2006\)V\. I\. Belikov, E\. V\. Muravenko, and M\. E\. Alexeev \(Eds\.\)Zadači lingvističeskix olimpiad\. 1965–1975\.Izdatel’stvo MCNMO,Moscow\.Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p1.1)\.
- Blasiet al\.\(2022\)D\. Blasi, A\. Anastasopoulos, and G\. NeubigSystematic inequalities in language technology performance across the world’s languages\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 5486–5505\.External Links:[Link](https://aclanthology.org/2022.acl-long.376/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.376)Cited by:[§5\.3](https://arxiv.org/html/2608.18011#S5.SS3.p2.1)\.
- Bozhanov and Derzhanski \(2013\)B\. Bozhanov and I\. DerzhanskiRosetta stone linguistic problems\.InProceedings of the Fourth Workshop on Teaching Natural Language Processing\. 51th Annual Meeting of the Association for Computational Linguistics,I\. Derzhanski and D\. Radev \(Eds\.\),pp\. 1–8\.External Links:[Link](https://aclanthology.org/W13-3401.pdf)Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p2.1)\.
- Brownet al\.\(2024\)B\. Brown, J\. Juravsky, R\. Ehrlich, R\. Clark, Q\. V\. Le, C\. Ré, and A\. MirhoseiniLarge language monkeys: scaling inference compute with repeated sampling\.External Links:2407\.21787,[Link](https://arxiv.org/abs/2407.21787)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p1.1)\.
- Center for AI Safetyet al\.\(2026\)Center for AI Safety, Scale AI, and HLE Contributors ConsortiumA benchmark of expert\-level academic questions to assess AI capabilities\.Nature649,pp\. 1139–1146\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09962-4),2501\.14249,[Link](https://arxiv.org/abs/2501.14249)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p1.1)\.
- Chenet al\.\(2025a\)F\. Chen, A\. Raventos, N\. Cheng, S\. Ganguli, and S\. DruckmannRethinking fine\-tuning when scaling test\-time compute: limiting confidence improves mathematical reasoning\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=jvVQeSMeGM)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p2.1)\.
- Chenet al\.\(2025b\)S\. Chen, P\. Pusarla, and B\. RayDyCodeEval: dynamic benchmarking of reasoning capabilities in code large language models under data contamination\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=3BZyQqbytZ)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p2.1)\.
- Cholletet al\.\(2026\)F\. Chollet, M\. Knoop, G\. Kamradt, B\. Landers, and H\. PinkardARC\-agi\-2: a new challenge for frontier ai reasoning systems\.External Links:2505\.11831,[Link](https://arxiv.org/abs/2505.11831)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p1.1)\.
- DeepSeek\-AI \(2025\)DeepSeek\-AIDeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.External Links:2501\.12948,[Link](https://arxiv.org/abs/2501.12948)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p1.1)\.
- Derzhanski and Payne \(2010\)I\. Derzhanski and T\. PayneThe Linguistic Olympiads: academic competitions in linguistics for secondary school students\.InLinguistics at School: Language Awareness in Primary and Secondary Education,pp\. 213–226\.Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p3.1),[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p1.1)\.
- Derzhanski and Velinov \(2012\)I\. Derzhanski and A\. VelinovLingvističen kaleĭdoskop\.Prosveta\.Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p1.1)\.
- Derzhanski and Velinov \(2013\)I\. Derzhanski and A\. VelinovLingvistična mozaĭka: dopŭlneno izdanie\.Prosveta\.Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p1.1)\.
- Derzhanski and Veneva \(2018\)I\. Derzhanski and M\. VenevaLinguistic problems on number names\.InProceedings of the Third International Conference on Computational Linguistics in Bulgaria \(CLIB 2018\),pp\. 169–176\.External Links:[Link](https://aclanthology.org/2018.clib-1.21/)Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p2.1)\.
- Derzhanski \(2013\)I\. DerzhanskiMultilingual editing of linguistic problems\.InProceedings of the Fourth Workshop on Teaching Natural Language Processing\. 51th Annual Meeting of the Association for Computational Linguistics,I\. Derzhanski and D\. Radev \(Eds\.\),pp\. 27–34\.Cited by:[§3\.2](https://arxiv.org/html/2608.18011#S3.SS2.p1.1)\.
- Döhler \(2021\)C\. DöhlerKomnzo text corpus\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.4695271),[Link](https://doi.org/10.5281/zenodo.4695271)Cited by:[§5\.3](https://arxiv.org/html/2608.18011#S5.SS3.p2.1)\.
- Garnham and Shareghi \(2026\)J\. Garnham and E\. ShareghiCould language models win the international linguistics olympiad?\.In30th Conference on Computational Natural Language Learning,External Links:[Link](https://openreview.net/forum?id=dioc2KgOKg)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p5.1),[§2](https://arxiv.org/html/2608.18011#S2.p3.1),[§4\.3](https://arxiv.org/html/2608.18011#S4.SS3.p1.1)\.
- Geipinget al\.\(2025\)J\. Geiping, S\. M\. McLeish, N\. Jain, J\. Kirchenbauer, S\. Singh, B\. R\. Bartoldson, B\. Kailkhura, A\. Bhatele, and T\. GoldsteinScaling up test\-time compute with latent reasoning: a recurrent depth approach\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=S3GhJooWIC)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p1.1)\.
- Gemma Team \(2026\)Gemma TeamGemma 4 technical report\.External Links:2607\.02770,[Link](https://arxiv.org/abs/2607.02770)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p1.1),[§2](https://arxiv.org/html/2608.18011#S2.p1.1)\.
- Gleason \(1955\)H\. A\. GleasonWorkbook in descriptive linguistics\.Holt, Rinehart and Winston,New York\.Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p1.1)\.
- Goyal and Dan \(2025\)S\. Goyal and S\. DanIOLBENCH: benchmarking LLMs on linguistic reasoning\.External Links:2501\.04249,[Link](https://arxiv.org/abs/2501.04249)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p2.1)\.
- Hammarströmet al\.\(2026\)H\. Hammarström, R\. Forkel, M\. Haspelmath, and S\. BankGlottolog 5\.3\.InLeipzig: Max Planck Institute for Evolutionary Anthropology,External Links:[Link](https://doi.org/10.5281/zenodo.18840935)Cited by:[item 1](https://arxiv.org/html/2608.18011#S5.I2.i1.p1.1)\.
- Heet al\.\(2024\)T\. He, K\. Choi, L\. Tjuatja, N\. Robinson, J\. Shi, S\. Watanabe, G\. Neubig, D\. Mortensen, and L\. LevinWav2Gloss: generating interlinear glossed text from speech\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 568–582\.External Links:[Link](https://aclanthology.org/2024.acl-long.34/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.34)Cited by:[§5\.3](https://arxiv.org/html/2608.18011#S5.SS3.p2.1)\.
- Heet al\.\(2026\)Y\. He, L\. Yang, C\. C\. G\. Gonzalo, and H\. ChenEvaluating program semantics reasoning with type inference in system $f$\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=IA9RmaP0aw)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p2.1)\.
- Honget al\.\(2024\)R\. Hong, H\. Zhang, X\. Pang, D\. Yu, and C\. ZhangA closer look at the self\-verification abilities of large language models in logical reasoning\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 900–925\.External Links:[Link](https://aclanthology.org/2024.naacl-long.52/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-long.52)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p1.1)\.
- Huang and Yang \(2025\)Y\. Huang and L\. F\. YangWinning gold at IMO 2025 with a model\-agnostic verification\-and\-refinement pipeline\.External Links:2507\.15855,[Link](https://arxiv.org/abs/2507.15855)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p1.1)\.
- Iomdinet al\.\(2013\)B\. Iomdin, A\. Piperski, and A\. SominLinguistic problems based on text corpora\.InProceedings of the Fourth Workshop on Teaching Natural Language Processing\. 51th Annual Meeting of the Association for Computational Linguistics,I\. Derzhanski and D\. Radev \(Eds\.\),pp\. 9–17\.Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p2.1)\.
- Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. NarasimhanSWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p1.1)\.
- Joshiet al\.\(2020\)P\. Joshi, S\. Santy, A\. Budhiraja, K\. Bali, and M\. ChoudhuryThe state and fate of linguistic diversity and inclusion in the NLP world\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 6282–6293\.External Links:[Link](https://aclanthology.org/2020.acl-main.560/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.560)Cited by:[§5\.3](https://arxiv.org/html/2608.18011#S5.SS3.p2.1)\.
- Kargaranet al\.\(2025\)A\. H\. Kargaran, F\. Yvon, and H\. SchützeGlotCC: an open broad\-coverage commoncrawl corpus and pipeline for minority languages\.External Links:2410\.23825,[Link](https://arxiv.org/abs/2410.23825)Cited by:[§5\.3](https://arxiv.org/html/2608.18011#S5.SS3.p2.1)\.
- Khoujaet al\.\(2026\)J\. Khouja, L\. Yang, K\. Korgul, S\. Hellsten, V\. A\. Neacșu, H\. Mayne, R\. O\. Kearns, A\. M\. Bean, and A\. MahdiLINGOLY\-TOO: disentangling reasoning from knowledge with templatised orthographic obfuscation\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=CQIkN2uuBr)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p2.1)\.
- Kocmiet al\.\(2025\)T\. Kocmi, S\. Agrawal, E\. Artemova, E\. Avramidis, E\. Briakou, P\. Chen, M\. Fadaee, M\. Freitag, R\. Grundkiewicz, Y\. Hou, P\. Koehn, J\. Kreutzer, S\. Mansour, S\. Perrella, L\. Proietti, P\. Riley, E\. Sánchez, P\. Schmidtova, M\. Shmatova, and V\. ZouharFindings of the WMT25 multilingual instruction shared task: persistent hurdles in reasoning, generation, and evaluation\.InProceedings of the Tenth Conference on Machine Translation,B\. Haddow, T\. Kocmi, P\. Koehn, and C\. Monz \(Eds\.\),Suzhou, China,pp\. 414–435\.External Links:[Link](https://aclanthology.org/2025.wmt-1.23/),[Document](https://dx.doi.org/10.18653/v1/2025.wmt-1.23),ISBN 979\-8\-89176\-341\-8Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p5.1),[§4\.1](https://arxiv.org/html/2608.18011#S4.SS1.p2.1)\.
- Lianet al\.\(2025\)D\. Lian, R\. Huang, P\. Chen, C\. Lim, Y\. Lin, G\. Tseng, Z\. Lin, P\. Chen, and S\. HsiehLOBSTER: linguistics olympiad benchmark for structured evaluation on reasoning\.InProceedings of the 37th Conference on Computational Linguistics and Speech Processing \(ROCLING 2025\),K\. Chang, K\. Lu, C\. Yang, Z\. Tam, W\. Chang, and C\. Wang \(Eds\.\),National Taiwan University, Taipei City, Taiwan,pp\. 193–229\.External Links:[Link](https://aclanthology.org/2025.rocling-main.23/),ISBN 979\-8\-89176\-379\-1Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p5.1),[§2](https://arxiv.org/html/2608.18011#S2.p2.1)\.
- Littellet al\.\(2013\)P\. Littell, L\. Levin, J\. Eisner, and D\. R\. RadevIntroducing computational concepts in a Linguistics Olympiad\.InProceedings of the Fourth Workshop on Teaching Natural Language Processing\. 51th Annual Meeting of the Association for Computational Linguistics,I\. Derzhanski and D\. Radev \(Eds\.\),pp\. 18–26\.External Links:[Link](https://aclanthology.org/W13-3403/)Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p2.1)\.
- Liuet al\.\(2025\)X\. Liu, T\. Liang, Z\. He, J\. Xu, W\. Wang, P\. He, Z\. Tu, H\. Mi, and D\. YuTrust, but verify: a self\-verification approach to reinforcement learning with verifiable rewards\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=gA3fFAEXNT)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p1.1)\.
- Marjanovicet al\.\(2026\)S\. V\. Marjanovic, A\. Patel, V\. Adlakha, M\. Aghajohari, P\. BehnamGhader, M\. Bhatia, A\. Khandelwal, A\. Kraft, B\. Krojer, X\. H\. Lù, N\. Meade, D\. Shin, A\. Kazemnejad, G\. Kamath, M\. Mosbach, K\. Stanczak, and S\. ReddyDeepSeek\-r1 thoughtology: let’s think about LLM reasoning\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=BZwKsiRnJI)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p1.1)\.
- Martins \(2022\)E\. C\. MartinsOlimpíadas de linguística: mosaico de uma prática social baseada em problemas\.Ph\.D\. Thesis,Universidade de Brasília,Brasília\.Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p2.1),[§3](https://arxiv.org/html/2608.18011#S3.p1.1)\.
- Merrillet al\.\(2026\)M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan, J\. Shen, G\. Ye, H\. Lin, J\. Poulos, M\. Wang, M\. Nezhurina, D\. Lu, O\. M\. Mastromichalakis, Z\. Xu, Z\. Chen, Y\. Liu, R\. Zhang, L\. L\. Chen, A\. Kashyap, J\. Uslu, J\. Li, J\. Wu, M\. Yan, S\. Bian, V\. Sharma, K\. Sun, S\. Dillmann, A\. Anand, A\. Lanpouthakoun, B\. Koopah, C\. Hu, E\. K\. Guha, G\. H\. S\. Dreiman, J\. Zhu, K\. Krauth, L\. Zhong, N\. Muennighoff, R\. K\. Amanfu, S\. Tan, S\. Pimpalgaonkar, T\. Aggarwal, X\. Lin, X\. Lan, X\. Zhao, Y\. Liang, Y\. Wang, Z\. Wang, C\. Zhou, D\. Heineman, H\. Liu, H\. Trivedi, J\. Yang, J\. Lin, M\. Shetty, M\. Yang, N\. Omi, N\. Raoof, S\. Li, T\. Y\. Zhuo, W\. Lin, Y\. Dai, Y\. Wang, W\. Chai, S\. Zhou, D\. Wahdany, Z\. She, J\. Hu, Z\. Dong, Y\. Zhu, S\. Cui, A\. Saiyed, A\. Kolbeinsson, C\. M\. Rytting, R\. Marten, Y\. Wang, J\. Jitsev, A\. Dimakis, A\. Konwinski, and L\. SchmidtTerminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=a7Qa4CcHak)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p1.1)\.
- Neacșu \(2024\)V\. A\. NeacșuLinguistics Olympiad: training guide\.Language Science Press\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.10947862),[Link](https://zenodo.org/doi/10.5281/zenodo.10947862)Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p2.1)\.
- Nikulin \(2024\)A\. NikulinLas olimpiadas de lingüística, o cómo llevar la lingüística a la secundaria\.InAportes Disciplinares II SAEL,M\. M\. y Gonzalo Espinosa \(Ed\.\),pp\. 105–117\.Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p1.1),[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p2.1)\.
- Penedoet al\.\(2025\)G\. Penedo, H\. Kydlíček, V\. Sabolčec, B\. Messmer, N\. Foroutan, A\. H\. Kargaran, C\. Raffel, M\. Jaggi, L\. V\. Werra, and T\. WolfFineWeb2: one pipeline to scale them all – adapting pre\-training data processing to every language\.External Links:2506\.20920,[Link](https://arxiv.org/abs/2506.20920)Cited by:[§5\.3](https://arxiv.org/html/2608.18011#S5.SS3.p2.1)\.
- Popović \(2015\)M\. PopovićChrF: character n\-gram F\-score for automatic MT evaluation\.InProceedings of the Tenth Workshop on Statistical Machine Translation,O\. Bojar, R\. Chatterjee, C\. Federmann, B\. Haddow, C\. Hokamp, M\. Huck, V\. Logacheva, and P\. Pecina \(Eds\.\),Lisbon, Portugal,pp\. 392–395\.External Links:[Link](https://aclanthology.org/W15-3049/),[Document](https://dx.doi.org/10.18653/v1/W15-3049)Cited by:[§4\.1](https://arxiv.org/html/2608.18011#S4.SS1.p3.1)\.
- Qiuet al\.\(2026\)S\. Qiu, H\. Huang, S\. Zhong, P\. Zuo, and M\. LiHyPER: bridging exploration and exploitation for scalable LLM reasoning with hypothesis path expansion and reduction\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=G29kBVeIZt)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p1.1),[§2](https://arxiv.org/html/2608.18011#S2.p1.1)\.
- D\. R\. Radev \(Ed\.\) \(2013a\)D\. R\. Radev \(Ed\.\)Puzzles in logic, languages and computation: the green book\.Springer\.Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p1.1)\.
- D\. R\. Radev \(Ed\.\) \(2013b\)D\. R\. Radev \(Ed\.\)Puzzles in logic, languages and computation: the red book\.Springer\.Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p1.1)\.
- Ramji and Ramji \(2025\)R\. Ramji and K\. RamjiInductive linguistic reasoning with large language models\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 22783–22810\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1171/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1171),ISBN 979\-8\-89176\-256\-5Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p3.1)\.
- Reinet al\.\(2024\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: a graduate\-level google\-proof q&a benchmark\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Ti67584b98)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p1.1)\.
- Şahinet al\.\(2020\)G\. G\. Şahin, Y\. Kementchedjhieva, P\. Rust, and I\. GurevychPuzzLing Machines: A Challenge on Learning From Small Data\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 1241–1254\.External Links:[Link](https://aclanthology.org/2020.acl-main.115/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.115)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p2.1)\.
- Sánchezet al\.\(2025\)E\. Sánchez, B\. Alastruey, C\. Ropers, P\. Stenetorp, M\. Artetxe, and M\. R\. Costa\-jussàLinguini: a benchmark for language\-agnostic linguistic reasoning\.External Links:[Link](https://openreview.net/forum?id=QiyQJqpcYe)Cited by:[Appendix J](https://arxiv.org/html/2608.18011#A10.p1.1),[Appendix D](https://arxiv.org/html/2608.18011#A4.p1.1),[§2](https://arxiv.org/html/2608.18011#S2.p2.1),[§4\.1](https://arxiv.org/html/2608.18011#S4.SS1.p2.1),[§4\.1](https://arxiv.org/html/2608.18011#S4.SS1.p3.1)\.
- Shenet al\.\(2025\)M\. Shen, G\. Zeng, Z\. Qi, Z\. Hong, Z\. Chen, W\. Lu, G\. W\. Wornell, S\. Das, D\. D\. Cox, and C\. GanSatori: reinforcement learning with chain\-of\-action\-thought enhances LLM reasoning via autoregressive search\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=j4FXxMiDjL)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p2.1)\.
- Silva and Costa \(2022\)S\. É\. C\. d\. Silva and P\. T\. d\. CostaLinguística por problemas: explorando a linguagem de forma investigativa nas escolas\.LED,Belo Horizonte\.Cited by:[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p1.1)\.
- Snellet al\.\(2025\)C\. V\. Snell, J\. Lee, K\. Xu, and A\. KumarScaling LLM test\-time compute optimally can be more effective than scaling parameters for reasoning\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=4FWAwZtd2n)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p1.1)\.
- Wanget al\.\(2026\)Y\. Wang, P\. Zhang, J\. Tang, H\. Wei, B\. Yang, R\. Wang, C\. Sun, F\. Sun, J\. Zhang, J\. Wu, Q\. Cang, Y\. Zhang, F\. Huang, J\. Lin, F\. Huang, and J\. ZhouPolyMath: evaluating mathematical reasoning in multilingual contexts\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=B1vCImy6yI)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p1.1)\.
- Yanget al\.\(2023\)K\. Yang, A\. M\. Swope, A\. Gu, R\. Chalamala, P\. Song, S\. Yu, S\. Godil, R\. Prenger, and A\. AnandkumarLeanDojo: theorem proving with retrieval\-augmented language models\.InThirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=g7OX2sOJtn)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p1.1)\.
- Zaliznjak \(1963\)A\. A\. ZaliznjakLingvističeskie zadači\.InIssledovanija po strukturnoj tipologii,T\. N\. Mološnaja \(Ed\.\),Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p3.1),[§3\.1](https://arxiv.org/html/2608.18011#S3.SS1.p1.1)\.
- Zhanget al\.\(2025\)F\. Zhang, J\. Xu, C\. Wang, C\. Cui, Y\. Liu, and B\. AnIncentivizing LLMs to self\-verify their answers\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=MBDWO29Qq6)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p1.1)\.
- Zhanget al\.\(2026\)J\. Zhang, C\. Petrui, K\. Nikolić, and F\. TramèrRealMath: a continuous benchmark for evaluating language models on research\-level mathematics\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=RBssYVpQEr)Cited by:[§1](https://arxiv.org/html/2608.18011#S1.p2.1)\.
- Zhang and Math\-AI \(2025\)Y\. Zhang and T\. Math\-AIAmerican invitational mathematics examination \(aime\) 2025\.Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p1.1)\.
- Zhuet al\.\(2025\)H\. Zhu, Y\. Liang, W\. Xu, and H\. XuEvaluating large language models for in\-context learning of linguistic patterns in unseen low resource languages\.InProceedings of the First Workshop on Language Models for Low\-Resource Languages,H\. Hettiarachchi, T\. Ranasinghe, P\. Rayson, R\. Mitkov, M\. Gaber, D\. Premasiri, F\. A\. Tan, and L\. Uyangodage \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 414–426\.External Links:[Link](https://aclanthology.org/2025.loreslm-1.31/)Cited by:[§2](https://arxiv.org/html/2608.18011#S2.p3.1)\.

## Appendix AAccessibility and Engagement

To reduce barriers for participation, we hosted an online submission 101 session with a demo notebook that provides a sample submission based on a small open model\. The compute provided in a T4 GPU session on Colab \(session time limits are 12h for free usage\) is comparable to that available in the competition, which allowed competitors to optimize model choices and decoding strategies for the time and compute limits\. In addition, we provided participants with a list of resources about linguistic reasoning with AI, such as pointers to benchmarks and papers\.

We organized this challenge intentionally outside of any academic venue to broaden participation and extend it beyond an academic audience or tie it to the requirement of paper submissions \(classic shared task setup\)\. Instead, we advertised it heavily on social media \(X, LinkedIn, Bluesky\) and in open science communities\.We hosted a kick\-off fireside chat with a discussion around the skills needed for linguistic reasoning both from a human perspective and the machine learning perspective\. Additionally, we reached out to authors of related works and benchmarks to invite them to participate and/or share it with their network\.

## Appendix BTechnical Details

We substantially adapted the HF competition framework for the IOL\-AI competition: First, submission evaluation is re\-routed from per\-submission Spaces to HF Jobs after undocumented platform rate limits\. Second, every submission repository is automatically archived for reproducibility and post\-hoc human evaluation\. All modifications are publicly available in the competition Space repository\.

## Appendix CBaselines

Qwen2\.5\-1\.5B\-Instructis the example submission published on the competition platform and website to show participants what is expected: a minimal script with a minimal prompt, asking the model to answer each item on its own line\.Qwen2\.5\-7B\-Instructscales this example to a larger model, quantized to 4\-bit at load time withbitsandbytesunder the same prompt, showing that quantization fits the compute constraints\.Qwen2\.5\-14B\-Instruct\-AWQillustrates the second quantization option, shipping pre\-quantized AWQ weights, to fit the T4; here the prompt is not minimal but a reasoning prompt instructing the model on the output format expected for each task type\. Finally,Tiny Aya Global, a 3\.35B open multilingual model, uses an even more detailed prompt\. Together, the baselines show that larger models reach higher scores, but that careful prompt design lets a smaller model outperform a larger one\.

[SectionF\.2](https://arxiv.org/html/2608.18011#A6.SS2)lists the full evaluation prompts\.

## Appendix DExtended Model Benchmarking

We benchmark additional strong models that are outside the resource constraints of the competition\. We prioritize models scoring highly on the Artificial Analysis Intelligence Index999[https://artificialanalysis\.ai/evaluations/artificial\-analysis\-intelligence\-index](https://artificialanalysis.ai/evaluations/artificial-analysis-intelligence-index)and diversity across model providers\.101010We do not evaluate Anthropic’s Fable 5 as it requires data retention\.The evaluation hyperparameters can be found in[Table11](https://arxiv.org/html/2608.18011#A9.T11)\. We leverage Linguini\([52](https://arxiv.org/html/2608.18011#bib.bib17)\)as a development set to tune task and explanation prompt and postprocessing with an answer extraction script to further boost scores, and then use the IOL 2026 test set to select models for human evaluation\. We optimize their explanation output format for readability of the jury, to make it closer to typical IOL solutions\.[AppendixF](https://arxiv.org/html/2608.18011#A6)contain the prompts for benchmarking with and without explanations\.[Table9](https://arxiv.org/html/2608.18011#A4.T9)shows the results of the automatic evaluation for the extended model list, before and after custom answer parsing\.

Table 9:Evaluation scores for existing models on the IOL 2026 problem set, before and after answer parsing\. The BoN Open \(oracle\) ensemble builds directly on custom answer parsed answers\. EM = exact match, GM = geometric mean, LFMR = line format match rate\. Model outputs that were chosen for the jury are marked with⋆\\star\.Table 10:Per\-problem automatic scores for the BoN Open \(oracle\) ensemble\. Scores are on a 0–100 scale per problem, so they are directly comparable to jury scores expressed as a percentage of the 20 points available per problem\.
## Appendix EThe Challenge of Format Following

Automatic scoring is strict about format: model answers are compared to gold by position, so a missing or shifted line can zero out a whole row even when later items are right\.

Submissions\.Format following was a major focus in the open challenge\. 34 teams used regex answer extraction, and many added per\-task format prompts or retry loops\. Teams also dropped chain\-of\-thought on scoring runs\. CoT was tried by 33 teams but hurt on average, and none of the top 6 kept it: reasoning text made extraction harder and burned wall\-clock time\. The podium used answers\-only prompts and invested in parsers instead\. Because format was already enforced in the sandbox, post\-hoc extraction barely moves submission scores\.

Frontier models\.Format errors are still common in raw frontier explain outputs\. Light postprocessing removes most of them \([Figure6](https://arxiv.org/html/2608.18011#A5.F6)\): empty answers cannot be fixed, but superficial non\-conformities such as included item labels or JSON format as a one\-liner are easily repaired\.[Table9](https://arxiv.org/html/2608.18011#A4.T9)reports scores before and after custom answer parsing\. Parsing never lowers the score\. At best it raises GM by about 14 points \(e\.g\. GLM\-5\.2 from 30\.5 to 45\.0; Claude Opus from 61\.0 to 75\.1\)\. Already clean runs stay unchanged\.

![Refer to caption](https://arxiv.org/html/2608.18011v1/images/format_failure_prevalence_matrix.png)\(a\)IOL 2026 frontier: format failures before extract\.
![Refer to caption](https://arxiv.org/html/2608.18011v1/images/format_failure_prevalence_matrix_after.png)\(b\)IOL 2026 frontier: format failures after extract\.

Figure 6:Format\-failure prevalence for IOL 2026 frontier runs before and after light answer extraction\.
## Appendix FEvaluation Prompts

### F\.1Benchmarking Task Prompts

For Linguini evaluations, we use the task prompt given in[Figure7](https://arxiv.org/html/2608.18011#A6.F7)\.

Youaresolvingalinguisticpuzzle\.Alltheinformationyouneediscontainedinthecontextbelow\-nopriorknowledgeofthislanguageisrequired\.

Context:

\[CONTEXT\]

Question:

\[QUESTION\]

Answerwithonlytherequestedwordorphrase\.Donotincludeexplanationsinyourfinalanswer\.

Figure 7:Linguini evaluation prompt\.For out\-of\-the\-box benchmarking of frontier/open models where explanations are required for human evaluation, we use the task prompt given in[Figure8](https://arxiv.org/html/2608.18011#A6.F8)\. We verified on Linguini that the added explanation requirement does not have substantial impact on task performance of the top\-scoring models participating in the human evaluation\.

Youaresolvingalinguisticpuzzle\.Alltheinformationyouneediscontainedinthecontextbelow\-nopriorknowledgeofthislanguageisrequired\.

Context:

\[CONTEXT\]

Question:

\[QUESTION\]

Giveyourfinalanswer,thenanexplanationofhowyouarrivedatit\.

Inthefinalanswer,onlyincludetherequestedword\(s\)orphrase\(s\),withoutmarkdown\.

Intheexplanation,summarizeyourlinguisticanalysis,andusetablesorschemata\(inmarkdown\)toillustratederivedrules\.

Donotwritealongchain\-of\-thoughtorlengthynarration\.

Usethisformatexactly:

Answer:

<yourfinalansweronly\-therequestedword\(s\)orphrase\(s\)\>

Explanation:

<yourexplanation\>

Figure 8:Task prompt for Linguini benchmarking of out\-of\-the\-box models, including explanations\.For IOL we slightly modified it because we also allowed for answers in JSON format, see[Figure9](https://arxiv.org/html/2608.18011#A6.F9)\.

YouaresolvinganInternationalLinguisticsOlympiadproblem\.Alltheinformationyouneediscontainedinthecontextbelow\-nopriorknowledgeofthislanguageisrequired\.

Context:

\[CONTEXT\]

Question:

\[QUESTION\]

Answereverynumbered\(orlettered\)iteminthequestion,inorder\.

ReturneitheraJSONlistofstrings,oroneanswerperline\(nonumbering\)\.

Giveyourfinalanswer,thenanexplanationofhowyouarrivedatit\.

Inthefinalanswer,onlyincludetherequestedword\(s\)orphrase\(s\),withoutmarkdown\.

Intheexplanation,summarizeyourlinguisticanalysis,andusetablesorschemata\(inmarkdown\)toillustratederivedrules\.

Donotwritealongchain\-of\-thoughtorlengthynarration\.

Usethisformatexactly:

Answer:

<yourfinalansweronly\-therequestedword\(s\)orphrase\(s\)\>

Explanation:

<yourexplanation\>

Figure 9:Final IOL task prompt for benchmarking out\-of\-the\-box models, including explanations\.
### F\.2Baseline System Prompts

The four organizer baselines \([Table2](https://arxiv.org/html/2608.18011#S4.T2)\) use a chat format with a separate system prompt, unlike the single task prompts above\. We list the system prompt and the user template for each;\{context\}and\{query\}are filled in per test item\.

SYSTEM:YousolveInternationalLinguisticsOlympiadproblems\.Answereverynumbereditem\.Puteachansweronitsownline,inorder,withnonumberingandnoextratext\.

USER:

\{context\}

\{query\}

Figure 10:Qwen2\.5\-1\.5B\-Instruct \(starter\) and Qwen2\.5\-7B\-Instruct \(NF4\) baseline prompt\.SYSTEM:

YousolveInternationalLinguisticsOlympiadproblemsbyreasoningfromthedatayouaregiven\.Youmaymeetatasktypeyouhaveneverseen:readtheinstructionandtheexamples,andanswerinthesameformtheyuse\.Commontasktypesandwhattogive\-\-translation:thetranslatedformonly,inthelanguagethetaskasksfor;fill\_blanks:onlythemissingformforeachblank;match\_letters:onlytheoptionletter\(forexampleA,B,C\);text\_to\_num:thenumberindigits;num\_to\_text:thenumberwrittenoutinwords,inthelanguageasked;anyothertype:giveexactlywhattheinstructionasks,nothingelse\.Reasonstepbystepfirst\.ThenwritealinethatsaysexactlyFINALANSWERS:and,belowit,oneanswerperlineintheordertheitemsareasked\-\-thebareansweronly,nonumbering,noquotes,noextratext\.

USER:

\{context\}

\{query\}

Figure 11:Qwen2\.5\-14B\-Instruct\-AWQ baseline promptSYSTEM:

YousolveInternationalLinguisticsOlympiadproblemsbyreasoningfromthedatainCONTEXTyouaregiventosolvetheproblemsinQUERY\.

TherearecommonTASKTYPESthatwespecifybelow,butyoumaymeetaTASKTYPEyouhaveneverseen:readtheinstructionandtheexamples,andanswertheQUERYinthesameformtheyuse\.

CommonTASKTYPESandwhattoreturn:

‘translation‘:returnthetranslatedformonly,inthelanguagethetaskasksfor;

‘fill\_blanks‘:returnonlythemissingformforeachindicatedblank\(beware:thiscouldbemanydifferentthings:aword,apartofawordoraphonetictranscription\-\-\-paycloseattentiontowhatpartoftheCONTEXTismissinginQUERY\);

‘match\_letters‘:returnonlytheoptionletter\(forexampleA,B,C\);

‘text\_to\_num‘:returnthenumberindigits;

‘num\_to\_text‘:returnthenumberwrittenoutinwords,inthelanguageasked;

anyothertype:returnexactlywhattheinstructionasksfor,nothingelse\.

Asthefirstpartofyouranswer,reasonstepbystepabout\(1\)thelinguisticrulesthatcanbededucedfromthegivenexamplesinCONTEXT,and\(2\)howtoapplythemtothegivenproblemsinQUERY,and\(3\)inwhatformatanswersneedtobereturned\(words,numbers,phonetictranscriptions,\.\.\.\)\.

Thenwriteadraftofthefinalanswer\.Subsequently,compareitwiththeformatrequirementsagain,andverifyit’scompliantwiththededucedrules,anditiscomplete,i\.e\.hasananswerforeachelementinQUERY\.Ifnecessary,correctandrefine\.Finally,writealinethatsaysexactly‘FINALANSWERS:‘and,belowit,writetheanswerstotheitemsrequestedinQUERY\(notthoseinCONTEXT\),oneanswerperline\(separatedbynewline\)intheordertheitemsareaskedforintheQUERY\-\-thebareansweronly,nonumbering,noquotes,noextratext,accordingtothegivenTASKTYPE\.

USER:

CONTEXT:\{context\}

TASKTYPE:‘\{task\_type\}‘

QUERY:\{query\}

Figure 12:Tiny Aya Global baseline prompt

## Appendix GGrading of AI Solutions by the IOL Jury

In contrast to the in\-person grading at IOL, due to time constraints at the event, grading of the AI solutions is done remotely in the week following IOL\. 1–3 jury members from each grading group \(that graded each of the 5 problems at IOL\) volunteered to grade the AI solutions for each problem\. Each jury member received pdf printouts of the AI solutions and explanations, anonymized with respect to model information\.

Using the same grading schemes that were used at IOL \(which were crafted by each grading group in the months leading to the event\), each jury member then graded each AI paper individually, assigning points for each criterion specified in the grading scheme in a large spreadsheet\. The spreadsheet is designed so that individual jurors can work separately in their own sheets, and afterwards a comparison sheet highlights any*discrepancies*between the grades of 2 or more jurors\. At IOL, each contestant’s paper is graded by at least 2 jurors and discrepancies are resolved largely by in\-person discussion\. For the challenge, all problems were graded twice except problem 3, which was graded 3 times\. For problem 4, the IOL jury chair served as a second grader despite not being part of that grading group\. Discrepancies were resolved largely by virtual discussion \(through text\), and in a couple of cases by decision of the jury chair\.

Once all discrepancies were resolved, the scores for the 5 problems were summed up into one total score \(out of 100\) per submission\.

## Appendix HSubmissions

Submission StatisticsParticipants could form teams of up to three, but most competed individually: only four teams of two were formed\. Submissions were back\-loaded, with 43% arriving in the final 48 hours\. 590 \(80\.7%\) submissions ran successfully and received a score\. The 141 failures fall into four families: library and environment mismatches between the submission repository and the evaluation image \(54, 38\.3%\), errors in the participant inference script or at runtime \(32, 22\.7%\), exceeding the wall\-clock or memory limits \(27, 19\.1%\), and offline packaging mistakes such as code still attempting to reach the Hub at evaluation time \(19, 13\.5%\); the remaining 9 \(6\.4%\) fall outside these families\. Common fixes included shipping model weights inside the repository, bundling updated wheels for offline install, quantizing to fit the T4 \(AWQ, NF4, or GGUF\), and, against the timeout, using adaptive token budgets and writingsubmission\.csvincrementally so a killed run still scores\.

Leading PipelinesThe seven highest\-ranked entries all reuse our minimal system prompt verbatim \([SectionF\.2](https://arxiv.org/html/2608.18011#A6.SS2)\), and all lower the repetition penalty except Hul \(6\), which keeps the default\. The entries that sample draw up to 8 retries \(ranks 1, 4, 5\) or up to 24 \(ranks 2, 3\) at temperature 0\.5, for as long as time allows, and a retry only overrides the greedy answer when several samples agree on the same alternative; friedspaghetti \(4\) counts near\-identical strings as the same answer, so formatting variants do not split the vote\. Hul and jbuaba \(7\) never sample\. Token budgets are either derived from the time remaining \(ranks 1, 3, 4, 5, between 192 and 900 tokens per problem\) or fixed at 512 tokens \(ranks 2, 6, 7\)\. Incremental writing means a complete submission file is on disk before the model even loads, and is rewritten after every pass or row, so a crash or timeout still leaves a valid file\. Post\-processing is where the entries differ most: most pad or trim the answers to the expected number of items, Hul normalises them by task type, and jbuaba answers matching problems by reading the model’s probability for each option letter and assigning options so that none repeats\.

Submission Model UseNine of the top ten entries run a 14B Qwen model: eight use Qwen2\.5\-14B\-Instruct\-AWQ and one uses Qwen3\-14B; the remaining entry, at rank 9, runs a 4\.2B Qwen3\.5 model in 8\-bit GGUF\. Across the 42 teams with a scored final submission, model choice is similarly concentrated: 29 teams selected Qwen2\.5\-14B\-Instruct\-AWQ for their final submissions\.111111We identified each backbone from the weights shipped at the scored commit, readingconfig\.jsonor the GGUF header rather than repository or file names\.This concentration has two causes: Qwen2\.5\-14B\-Instruct\-AWQ is the largest model that fits on the T4 within the time limit, and it was the model presented in our submission 101 session\. Most teams therefore adopted it as a starting point and worked on the pipeline around it\. The best submission more than doubles our strongest baseline with the same model \(GM 9\.40→\\rightarrow19\.79\), so the margin at the top comes from decoding and output handling rather than model capacity\.

## Appendix IEvaluation Hyper\-parameters

[Table11](https://arxiv.org/html/2608.18011#A9.T11)reports the evaluation configurations that were used for benchmarking frontier and open models\. We set temperature and topppto whatever was the recommended configuration in model cards and the reasoning effort and token limit as high as possible\.

ModelProvidertemptop\_pReasoning EffortMax TokensClaude Opus 4\.8Bedrock1\.01\.0high128000GPT\-5\.6OpenAI1\.01high128000Kimi K2\.6TogetherAI1\.00\.95N/A256000GLM\-5\.2TogetherAI1\.00\.95max128000DeepSeek V4 ProTogetherAI1\.01high128000Qwen3\.7\-MaxTogetherAI0\.60\.95N/A65536InklingTogetherAI1\.00\.95high256000Nemotron 3 UltraTogetherAI0\.30\.95high65536Command A PlusCohere0\.60\.95N/A64000MiniMax M3TogetherAI1\.00\.95N/A131072Gemini 3\.6 FlashGoogle APIN/AN/Ahigh65536Gemma 4 31BSelf\-hosted1\.00\.95N/A64000Qwen3\.6 27BSelf\-hosted0\.60\.95 \(kk=20\)N/A64000DeepSeek R1 32BSelf\-hosted0\.60\.95N/A32768GLM 4\.7 FlashSelf\-hosted0\.60\.95 \(kk=40\)†N/A64000Table 11:Hyper\-parameters for evaluation\. Self\-hosted models were served with vLLM; “Max Tokens” is the generation budget, not the model’s context window\.†GLM 4\.7 Flash additionally requiresrepetition\_penalty=1\.05\{=\}1\.05; without it the model degenerates into repetition loops, which is reflected in its low format\-compliance and near\-zero scores\.
## Appendix JLinguini Scores

[Table12](https://arxiv.org/html/2608.18011#A10.T12)contains automatic evaluation results for the Linguini benchmark\([52](https://arxiv.org/html/2608.18011#bib.bib17)\)\.

Table 12:Out\-of\-the\-box benchmarking scores for existing models on Linguini\. EM = exact match, GM = geometric mean, LFMR = line format match rate\.
## Appendix KKnowledge Probing Details

In order to turn the language context per problem into probing tasks, we analyze each problem with an LLM, and prompt it to extract the relevant information \(either contextual information, or lexical items from the problem language\)\.[Table13](https://arxiv.org/html/2608.18011#A11.T13)shows examples\. All tasks and answer options are verified by a human annotator who edited especially alternative response options to avoid making the right answer option too obvious\. In total they span 33 questions about language context, and 40 about lexical items\.

Table 13:Examples of knowledge probing tasks derived from the problem statements from IOL 2026\. Bold\-faced answers are the correct answers\. Language Context = 4\-way MCQ over meta\-facts from the IOL blurbs; Lexicon = 4\-way MCQ over attested forms \(distractors: scrambled or other\-language\)\. Questions are shortened\.