KBS：用于基础模型评估的韩语基准测试套件

# KBS: A Korean Benchmark Suite for Foundation Model Evaluation
Source: [https://arxiv.org/html/2610.21227](https://arxiv.org/html/2610.21227)
Yeonkyoung So1Jongmin Kim1Sungmok Jung1Gyuseong Lee1 Sangho Kim1Jongyeon Park1Joonhak Lee1Seho Pyo1 Gyeongje Cho1Seorin Kim1JiSoo Kim1Suyoung Park1 Hyunji M\. Park1Yelim Ahn1Yeongho Seo2Jaejin Lee1,2

1Graduate School of Data Science, Seoul National University 2Dept\. of Computer Science and Engineering, Seoul National University \{kathy1028,jmin\.kim,tjdahrwjd,ksnannaya,ksh4931,iscopark67,hmjelee,hori14,avq159,seorin1116,jisoo\.kim,itforspark,mhj233,mileya,graysea2,jaejin\}@snu\.ac\.kr

###### Abstract

We present a Korean Benchmark Suite \(KBS\) comprising Ko\-ARC, Ko\-GSM8K, Ko\-EQ\-Bench, Ko\-WinoGrande, Ko\-LAMBADA, and Ko\-IFEval, covering six capabilities across 9,396 items\. The suite combines review, localization, direct Korean construction, and redesign for Korean, avoiding a translation\-only approach to benchmark adaptation\. We evaluate Korean\-developed models, multilingual open models, and closed API models under protocols that account for model access, prompt format, and scoring differences\. Results show that Korean benchmark scores depend on task construction, scoring rules, prompting, and model post\-training\. These findings suggest that Korean evaluation should report both final scores and the settings used to obtain them\.

## 1Introduction

Large language models \(LLMs\) have rapidly advanced in recent years and are now used across a wide range of domains requiring language understanding, generation, and reasoning\([Brown et al\., 2020](https://arxiv.org/html/2610.21227#bib.bib6);[B et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib5);[Zhou et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib7)\)\. As these models become more capable and more widely deployed, it becomes increasingly important to evaluate them using objective and standardized criteria\([Chang et al\., 2023](https://arxiv.org/html/2610.21227#bib.bib2);[Desai et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib1)\)\. Benchmarks play a central role in this process by providing shared tasks for comparing models and tracking progress across capabilities such as knowledge, reasoning, contextual understanding, and instruction following\([Clark et al\., 2018](https://arxiv.org/html/2610.21227#bib.bib13);[Cobbe et al\., 2021](https://arxiv.org/html/2610.21227#bib.bib14);[Hendrycks et al\., 2021](https://arxiv.org/html/2610.21227#bib.bib12);[Sakaguchi et al\., 2021](https://arxiv.org/html/2610.21227#bib.bib16);[Liang et al\., 2023](https://arxiv.org/html/2610.21227#bib.bib3);[Paech, 2024](https://arxiv.org/html/2610.21227#bib.bib15);[Zhou et al\., 2023](https://arxiv.org/html/2610.21227#bib.bib17);[Wang et al\., 2024b](https://arxiv.org/html/2610.21227#bib.bib4)\)\. However, benchmark scores are only useful when the benchmark measures the intended capability rather than artifacts of dataset construction or evaluation\([Durmus et al\., 2022](https://arxiv.org/html/2610.21227#bib.bib25);[Wu et al\., 2022](https://arxiv.org/html/2610.21227#bib.bib26);[Bean et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib24)\)\.

This issue is particularly important for languages outside a small set of major evaluation languages\. English benchmarks have become broad and diverse enough that researchers can evaluate models on many capabilities using widely recognized test sets\([Srivastava et al\., 2023](https://arxiv.org/html/2610.21227#bib.bib18);[Shen et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib8)\)\. For Korean, the situation has improved, and several valuable benchmarks have been introduced\([Park et al\., 2021](https://arxiv.org/html/2610.21227#bib.bib19);[Son et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib22);[Son et al\., 2025b](https://arxiv.org/html/2610.21227#bib.bib21)\)\. Still, it remains difficult to find Korean benchmark suites that both cover a broad range of capabilities and support close comparison with widely used English benchmarks\.

For this reason, Korean model evaluation often depends on adapting established English benchmarks\. The most straightforward way to do this is direct translation, but translation\-only evaluation can distort the benchmark itself\. Literal translation, mistranslation, stylistic inconsistency, and cultural mismatch may change task difficulty, alter the naturalness of the prompt, or even shift what the task is testing\([Chua et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib10);[Doddapaneni et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib9);[Xu et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib11)\)\. In such cases, differences in benchmark scores become harder to interpret, since they may reflect properties of the translation as much as the model’s actual capability in Korean\.

In this work, we present a Korean Benchmark Suite \(KBS\) for evaluating foundation models across multiple capabilities, including science question answering, mathematical reasoning, dialogue sentiment analysis, contextual reasoning, literary context understanding, and instruction following\. Our construction pipeline combines machine translation with author\-led review, linguistic correction, cultural localization, and cross\-validation\. For tasks that cannot be transferred appropriately by translation alone, we redesign the benchmark to preserve the intended capability in Korean rather than treating translation as a simple preprocessing step\.

Using the resulting benchmark suite, we evaluate both Korean LLMs and global LLMs with multilingual capability and examine how model behavior differs across tasks\. For multiple\-choice subsets, we also examine whether standard accuracy\-based evaluation may be affected by superficial properties of the answer choices, which can make benchmark results harder to interpret\([Cho et al\., 2026](https://arxiv.org/html/2610.21227#bib.bib23)\)\. More broadly, our study suggests that benchmarking in languages beyond English requires careful consideration of both how benchmarks are transferred across languages and how evaluation protocols shape the interpretation of benchmark scores\. The evaluation code and benchmark resources are publicly available\.111Code:[https://github\.com/mcrl/korean\_benchmarks](https://github.com/mcrl/korean_benchmarks)\. Benchmark resources:[https://huggingface\.co/collections/thunder\-research\-group/snu\-thunder\-llm\-korean\-benchmark\-suite](https://huggingface.co/collections/thunder-research-group/snu-thunder-llm-korean-benchmark-suite)\.

## 2Related Work

### 2\.1Benchmarking Foundation Models

Benchmarks have become a standard tool for evaluating foundation models\. By providing shared tasks and standardized metrics, they enable researchers to compare models across capabilities such as factual knowledge, reasoning, language understanding, code generation, and instruction following\. Widely used examples include MMLU\([Hendrycks et al\., 2021](https://arxiv.org/html/2610.21227#bib.bib12)\)for multi\-domain knowledge, ARC\([Clark et al\., 2018](https://arxiv.org/html/2610.21227#bib.bib13)\)for science question answering, GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.21227#bib.bib14)\)for mathematical reasoning, WinoGrande\([Sakaguchi et al\., 2021](https://arxiv.org/html/2610.21227#bib.bib16)\)and HellaSwag\([Zellers et al\., 2019](https://arxiv.org/html/2610.21227#bib.bib27)\)for commonsense and contextual reasoning, HumanEval\([Chen et al\., 2021](https://arxiv.org/html/2610.21227#bib.bib28)\)for code generation, and IFEval\([Zhou et al\., 2023](https://arxiv.org/html/2610.21227#bib.bib17)\)for instruction following\.

Beyond individual task\-specific datasets, holistic evaluation frameworks have attempted to aggregate diverse tasks into broader evaluation suites\. HELM\([Liang et al\., 2023](https://arxiv.org/html/2610.21227#bib.bib3)\)evaluates models across multiple scenarios and metrics, while BIG\-Bench\([Srivastava et al\., 2023](https://arxiv.org/html/2610.21227#bib.bib18)\)collects a large set of tasks designed to probe a wide range of model behaviors\. As model performance has improved, more challenging benchmarks have also been introduced to better differentiate frontier models\. Examples include MMLU\-Pro\([Wang et al\., 2024b](https://arxiv.org/html/2610.21227#bib.bib4)\), GPQA\([Rein et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib30)\), and expert\-level academic benchmarks\([Phan et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib31)\)\. Domain\-specific benchmarks, such as LegalBench\([Guha et al\., 2023](https://arxiv.org/html/2610.21227#bib.bib29)\)and MedQA\([Jin et al\., 2021](https://arxiv.org/html/2610.21227#bib.bib32)\), further show that general\-purpose benchmarks are often insufficient for evaluating specialized capabilities\.

### 2\.2Evaluation Sensitivity

Benchmark scores are not always straightforward indicators of model capability\. Recent work has shown that evaluation outcomes can depend on factors that are not directly related to the target skill being tested\. For example, small changes in prompt wording or formatting can lead to different measured performance, suggesting that benchmark results may depend on the particular evaluation template used\([Sclar et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib35);[Polo et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib40)\)\. In multiple\-choice question answering, model predictions can also be sensitive to the order, wording, length, format, or scoring of answer choices\([Zheng et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib34);[Pezeshkpour and Hruschka, 2024](https://arxiv.org/html/2610.21227#bib.bib33);[Wang et al\., 2024a](https://arxiv.org/html/2610.21227#bib.bib37);[Molfese et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib39)\)\. More fundamentally, models may sometimes rely on the answer choices themselves, rather than on the question, so high accuracy does not always imply that the model solved the task in the intended way\([Balepur et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib36);[Balepur et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib41);[Wang et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib38);[Cho et al\., 2026](https://arxiv.org/html/2610.21227#bib.bib23)\)\.

These findings suggest that benchmark design involves more than selecting a task and reporting accuracy\. The dataset construction process, prompt format, scoring rule, and evaluation protocol can all affect how benchmark scores should be interpreted\. We account for these concerns in our multiple\-choice evaluation by reporting NPSQ\-based accuracy, which helps distinguish question\-driven evidence from answer\-choice artifacts\.

Figure 1:Construction routes of the Korean Benchmark Suite\. Translation\-based tasks are adapted from English benchmarks through machine translation, correction, localization, and cross\-review\. Directly constructed tasks are built from Korean sources, while hybrid tasks combine translated items with Korean\-specific prompts and automatic checkers\.
### 2\.3Korean LLM Evaluation

Korean LLM evaluation has developed rapidly in recent years, but remains distributed across different goals, including general language understanding, cultural knowledge, professional knowledge, instruction following, and reasoning\. Early resources such as KLUE, KorNLI/KorSTS, and KoBEST have provided important foundations for Korean NLU evaluation\([Ham et al\., 2020](https://arxiv.org/html/2610.21227#bib.bib42);[Park et al\., 2021](https://arxiv.org/html/2610.21227#bib.bib19);[Jang et al\., 2022](https://arxiv.org/html/2610.21227#bib.bib20)\)\. KLUE has introduced a broad Korean language understanding benchmark, while KorNLI and KorSTS have adapted English NLI and STS resources into Korean through translation and human post\-editing\. KoBEST has further introduced Korean adaptations of several established tasks, motivated by the limitation that machine\-translated benchmarks may not fully reflect Korean linguistic characteristics\.

More recent benchmarks have expanded Korean evaluation toward native Korean sources and Korean\-specific knowledge\. HAE\-RAE Bench evaluates Korean cultural and contextual knowledge\([Son et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib22)\), while CLIcK draws from official Korean exams and textbooks to assess cultural and linguistic intelligence\([Kim et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib43)\)\. KMMLU provides large\-scale expert\-level multiple\-choice questions collected from Korean exams\([Son et al\., 2025b](https://arxiv.org/html/2610.21227#bib.bib21)\), with later variants such as KMMLU\-Redux and KMMLU\-Pro focusing on question quality and professional knowledge\([Hong et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib50)\)\. Other recent benchmarks target more specific dimensions, including national alignment and social values, Korean commonsense, and Korean mathematical reasoning\([Lee et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib44);[Seo et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib45);[Son et al\., 2025a](https://arxiv.org/html/2610.21227#bib.bib46);[Ko et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib47)\)\. In addition, the Open Ko\-LLM Leaderboard series has attempted to provide shared evaluation frameworks for Korean LLMs, but leaderboard\-based efforts remain limited when the underlying datasets, evaluation details, or long\-term maintenance are not fully available\([Park et al\., 2024](https://arxiv.org/html/2610.21227#bib.bib48);[Kim et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib49)\)\.

These efforts show that Korean LLM evaluation is no longer limited to a small number of datasets\. They also show a clear shift from simply machine\-translating English resources toward constructing tasks that better reflect Korean language use, culture, and real\-world evaluation needs\. Still, this expanding landscape raises a design question that is central to our work: how to determine when an English benchmark should be translated for comparability, localized for Korean naturalness, or redesigned because the original formulation no longer measures the same capability in Korean\.

Our work addresses this question by constructing a multi\-task Korean benchmark suite that combines reviewed translation, localization, and task redesign\. We keep links to established English benchmarks where comparison is useful, but modify or reconstruct tasks when direct transfer would distort the intended evaluation\. Together with the evaluation sensitivity considerations discussed above, this construction strategy supports more interpretable Korean benchmark results\.

## 3Constructing Korean Benchmarks

Figure 2:Examples illustrating why task redesign is needed when adapting English benchmarks to Korean\. Each panel contrasts direct transfer with the corresponding redesigned Korean task formulation\.### 3\.1Overview of the Korean Benchmark Suite

The Korean Benchmark Suite \(KBS\) evaluates foundation models across six capabilities: science question answering, mathematical reasoning, dialogue sentiment analysis, commonsense and contextual reasoning, long\-context literary understanding, and instruction following\. It is built around established English benchmarks—ARC\([Clark et al\., 2018](https://arxiv.org/html/2610.21227#bib.bib13)\), GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.21227#bib.bib14)\), EQ\-Bench\([Paech, 2024](https://arxiv.org/html/2610.21227#bib.bib15)\), WinoGrande\([Sakaguchi et al\., 2021](https://arxiv.org/html/2610.21227#bib.bib16)\), LAMBADA\([Paperno et al\., 2016](https://arxiv.org/html/2610.21227#bib.bib51)\), and IFEval\([Zhou et al\., 2023](https://arxiv.org/html/2610.21227#bib.bib17)\)—while adapting each task to Korean according to its linguistic and cultural characteristics\.

We construct the suite using three routes, as illustrated in Figure[1](https://arxiv.org/html/2610.21227#S2.F1)\. First, Ko\-ARC, Ko\-GSM8K, Ko\-EQ\-Bench, and Ko\-WinoGrande are adapted from their English counterparts through translation, correction, localization, and cross\-review\. Second, Ko\-LAMBADA is directly constructed from Korean literary texts because the original English task formulation does not transfer cleanly to Korean\. Third, Ko\-IFEval follows a hybrid strategy: transferable instructions are translated and localized, while Korean\-specific constraints and automatic checkers are newly developed\.

Table 1:Composition of the Korean Benchmark Suite\. Construction type indicates whether the benchmark is translated and refined from an English source, newly constructed from Korean sources, or combines translated items with newly constructed items for Korean\-specific constraints\. Ko\-IFEval consists of 541 translated items and 300 newly constructed items\.This task\-level construction strategy allows us to retain comparability with established English benchmarks where appropriate, while redesigning tasks when direct transfer would change the capability being evaluated\. The following subsections describe how English benchmarks are adapted to Korean, which tasks require redesign, and the resulting dataset statistics\.

### 3\.2Adapting English Benchmarks to Korean

For Ko\-ARC, Ko\-GSM8K, Ko\-EQ\-Bench, and Ko\-WinoGrande, we adapt the corresponding English benchmarks through a four\-stage pipeline: initial machine translation, author\-led correction, localization, and independent cross\-review\. The goal is to preserve the original task structure and answer key where possible, while removing artifacts that would make the Korean version unnatural or misleading\. Additional dataset\-specific correction and localization rules are provided in Appendix[A](https://arxiv.org/html/2610.21227#A1)\.

We first translate the English source items using the DeepL API222[https://www\.deepl\.com/products/api](https://www.deepl.com/products/api)\. We treat the machine\-translated output only as a draft, since direct translation often produces literal phrasing, inconsistent terminology, and unnatural Korean expressions\. Native Korean authors then revise each item, correcting both translation artifacts and issues in the original English items, such as typos, duplicate questions, incorrect answer labels, and incorrectly formatted solution annotations\. For domain\-specific tasks, authors also verify that scientific and mathematical terminology follows standard Korean usage and that the translated quantities, conditions, and explanations remain consistent with the answer\.

Localization is applied when a literal translation would make the item culturally unnatural or change the intended difficulty\. We adjust names, units, currency, everyday objects, and culturally specific scenarios while preserving the causal or logical structure of the original item\. For example, in math problems, we convert units and currencies to Korean conventions while keeping the arithmetic simple\. We also replace culturally unfamiliar situations with Korean equivalents when the answer remains unchanged\. For science questions, localization is applied selectively\. We standardize Korean terminology and wording, but we do not replace foreign scientific examples simply to make them Korean when the original context is part of the scientific content\.

Finally, each item is independently cross\-reviewed by an author who did not participate in the earlier correction and localization stage\. This pass checks whether the Korean item is fluent, whether the answer key remains valid, and whether localization has introduced unintended ambiguity\. If translation or localization makes the question ambiguous or changes the correct answer, we revise or remove the item\. This final check reduces translation artifacts while keeping the Korean dataset aligned with the original benchmark\.

### 3\.3Task Redesign for Korean

Table 2:Summary of evaluation formats\. Original multiple\-choice evaluation is used for open\-weight models with log\-likelihood access, while closed API models are evaluated using generation\-based variants\.The adaptation pipeline above is appropriate when an English task can be expressed in Korean without changing the capability being evaluated\. For Ko\-LAMBADA and Ko\-IFEval, however, direct transfer is insufficient because the original task formulation depends partly on English\-specific linguistic or formatting assumptions\. We therefore redesign these tasks for Korean rather than treating translation as the default\. Figure[2](https://arxiv.org/html/2610.21227#S3.F2)provides representative examples of these two redesign cases\.

#### Ko\-LAMBADA\.

Ko\-LAMBADA is constructed directly from copyright\-cleared Korean literary texts, primarily collected from Gongu Madang \(공유마당\)333[https://gongu\.copyright\.or\.kr/gongu/main/main\.do](https://gongu.copyright.or.kr/gongu/main/main.do), instead of translating the original English LAMBADA\. We choose direct construction for two reasons\. First, the original LAMBADA task asks models to predict the final word of an English passage\. In many cases, this final word is a noun or name that must be inferred from the preceding context\. In Korean, however, sentences typically end with predicates, so final\-word prediction would often test verb endings rather than long\-context understanding\. Second, machine\-translated literary passages often sound unnatural in Korean and tend to flatten literary expressions\. We therefore construct new items from Korean literary sources rather than translating English literary passages\.

To build candidate items, we automatically extract passage spans of an appropriate length from the collected Korean texts\. We select spans that contain a target noun or name that can be masked, together with another in\-context noun or name that can serve as a plausible distractor\. The target is then masked in the passage, and candidate items are manually reviewed to ensure that the answer is inferable from context, that the distractor is plausible but not equally valid, and that the masked sentence remains grammatical and natural\.

#### Ko\-IFEval\.

Ko\-IFEval follows a hybrid strategy\. Some original IFEval instructions transfer naturally to Korean after translation and localization, such as constraints involving required keywords, forbidden words, or output format\. Other constraints, on the other hand, depend on English\-specific writing conventions, such as alphabet\-based character constraints or word\-count rules that do not map cleanly onto Korean morphology and spacing\. For these cases, we add Korean\-specific instruction types and adapt the automatic checkers accordingly\. Examples include constraints based on Korean choseong \(초성; leading consonant\), Korean character\-count rules, and formatting conventions that can be checked automatically for Korean outputs\. As a result, Ko\-IFEval keeps the original goal of evaluating instruction following with automatically checkable constraints, while extending the constraint set to Korean\.

### 3\.4Dataset Statistics

Table[1](https://arxiv.org/html/2610.21227#S3.T1)summarizes the resulting benchmark suite\. The KBS contains 9,396 evaluation items across six capabilities\. The largest subset is Ko\-ARC, with 3,543 science question\-answering items across easy and challenge subsets, followed by Ko\-LAMBADA with 2,255 literary context\-understanding items\. Ko\-GSM8K, Ko\-WinoGrande, Ko\-IFEval, and Ko\-EQ\-Bench contribute 1,319, 1,267, 841, and 171 items, respectively\.

## 4Experiments

### 4\.1Evaluation Setup

#### Models\.

We evaluate both open\-weight models and closed API models\. Closed API models are evaluated through their official APIs: Claude Haiku 4\.5 \(claude\-haiku\-4\-5\-20251001\)\([ANTHROPIC, 2025](https://arxiv.org/html/2610.21227#bib.bib62)\)and Claude Opus 4\.7 \(claude\-opus\-4\-7\)\([ANTHROPIC, 2026](https://arxiv.org/html/2610.21227#bib.bib63)\)through the Anthropic API444[https://platform\.claude\.com/docs/en/about\-claude/models/overview](https://platform.claude.com/docs/en/about-claude/models/overview), GPT\-5\-mini \(gpt\-5\-mini\-2025\-08\-07\) and GPT\-5\.5 \(gpt\-5\.5\-2026\-04\-23\)\([Singh et al\., 2026](https://arxiv.org/html/2610.21227#bib.bib64)\)through the OpenAI API555[https://developers\.openai\.com/api/docs](https://developers.openai.com/api/docs)\. Open\-weight models are evaluated with the HuggingFace Transformers\([Wolf et al\., 2020](https://arxiv.org/html/2610.21227#bib.bib52)\)backend of Language Model Evaluation Harness \(lm\-evaluation\-harness\)\([Biderman et al\., 2026](https://arxiv.org/html/2610.21227#bib.bib53)\)\. All open\-model runs use bfloat16 precision on a single NVIDIA B200 GPU\.

We evaluate two groups of open\-weight models: Korean\-developed LLMs spanning five families \(KT Mi:dm 2\.0\([KT Tech\. Innovation Group, 2025](https://arxiv.org/html/2610.21227#bib.bib58)\), LG EXAONE 4\.0\([Bae et al\., 2026](https://arxiv.org/html/2610.21227#bib.bib57)\), Kakao Kanana\-2\([Kanana, 2025](https://arxiv.org/html/2610.21227#bib.bib65)\), Naver HyperCLOVAX\([Team\-HyperCLOVA, 2025](https://arxiv.org/html/2610.21227#bib.bib59)\), and SKT A\.X 4\.0\([SKT Telecom, 2025](https://arxiv.org/html/2610.21227#bib.bib56)\)\) and widely used multilingual LLMs \(Qwen 3\.5\([Qwen Team, 2026](https://arxiv.org/html/2610.21227#bib.bib61)\)and Gemma 4\([Google DeepMind, 2026](https://arxiv.org/html/2610.21227#bib.bib60)\)\)\. This model set allows direct comparison between Korean\-specialized and multilingual model families\. The full list of evaluated open\-weight models, parameter sizes, and HuggingFace model identifiers is provided in Appendix[C](https://arxiv.org/html/2610.21227#A3)\.

#### Evaluation formats\.

We use log\-likelihood\-based multiple\-choice evaluation for open\-weight models on Ko\-ARC, Ko\-WinoGrande, and Ko\-LAMBADA\. For the evaluated closed API models, token\-level log\-likelihoods are unavailable\. To compare the two model groups under a common response format, we additionally evaluate both groups on generative variants of these tasks\. Models are presented with explicitly labeled answer options and instructed to output a single label\. Ko\-ARC uses A\-D labels, while Ko\-WinoGrande and Ko\-LAMBADA use A\-B labels\. The final answer is extracted from the generated response with a regular expression over the option labels\.

#### Diagnostic translation variants\.

We additionally evaluate English and raw DeepL variants to examine performance differences associated with evaluation language and author\-led correction and localization\. These comparisons are restricted to variants with meaningful item\-level correspondence\. The DeepL variant uses the raw machine\-translated draft before correction and is available for Ko\-ARC, Ko\-GSM8K, and Ko\-EQ\-Bench\. The English variant uses the corresponding original English items\. For Ko\-WinoGrande, we report only the KBS\-English comparison because its Korean options were manually constructed and the sentence was revised accordingly, leaving no full raw\-DeepL counterpart\.

#### Prompt formatting\.

For open\-weight models, we use plain prompting for all models and additionally evaluate chat\-template prompting for instruction\-tuned or chat\-oriented models with a defined chat template\. Closed API models are evaluated only with chat\-style prompting, since their official APIs do not provide an equivalent plain prompting interface\.

#### Generation settings\.

All generation\-based evaluations use greedy decoding\. This includes Ko\-GSM8K, Ko\-IFEval, Ko\-EQ\-Bench, the generative multiple\-choice variants evaluated for both open\-weight and closed API models\. The model always selects the highest\-probability next token rather than sampling from the output distribution\. We disable sampling withdo\_sample=Falseand set the temperature to 0\.0\. Each reported result is based on a single run\.

#### Metrics\.

Table[2](https://arxiv.org/html/2610.21227#S3.T2)summarizes the task formats and metrics\. For multiple\-choice tasks, we report accuracy under raw log\-likelihood \(acc\), byte\-length normalized log\-likelihood \(acc\_norm\), and NPSQ\-based accuracy \(acc\_npsq\)\. NPSQ \(Normalized Probability Shift by the Question\) compares the model’s preference among answer choices conditioned on the question against its preference computed without the question, and selects the choice with the largest normalized increase in log\-probability when the question is added\. This separates question\-driven evidence from superficial choice preference that arises from lexical, length, or surface\-form biases over the answer choices alone\([Cho et al\., 2026](https://arxiv.org/html/2610.21227#bib.bib23)\)\.

For generation tasks, we report task\-specific automatic metrics, such as answer exact match for Ko\-GSM8K, strict and loose instruction\-following accuracy for Ko\-IFEval, and the custom Ko\-EQ\-Bench score\. The full evaluation protocol, including stop sequences, prompting templates, answer\-extraction rules, and metric definitions, is provided in Appendix[D](https://arxiv.org/html/2610.21227#A4)\.

### 4\.2Results

#### Overall performance\.

Among open\-weight models under default chat prompting, Gemma\-4\-31B\-it leads Ko\-ARC and Ko\-WinoGrande under NPSQ, and achieves the highest Ko\-GSM8K strict exact match \(82\.71%\) and Ko\-IFEval prompt\-level strict accuracy \(85\.02%\)\. On Ko\-LAMBADA, EXAONE\-4\.0\-32B achieves the highest NPSQ accuracy \(83\.68%\)\. The leading models change under plain prompting: Gemma\-4\-31B leads Ko\-ARC and Ko\-WinoGrande, Kanana\-2 variants lead Ko\-LAMBADA and Ko\-GSM8K, and A\.X 4\.0 Light leads Ko\-IFEval\. Complete results are provided in Appendix[F](https://arxiv.org/html/2610.21227#A6)\.

Table 3:ARC accuracy \(%\) using NPSQ under chat prompting\. All versions are restricted to the same 2,376 Easy and 1,167 Challenge items\. Scores are averaged equally over the two subsets and then over the models in each group\. DeepL denotes the raw machine\-translated version before author correction and localization\.
#### Translation and language effects\.

We use ARC for this comparison because it provides matched English, raw DeepL, and KBS variants under the same log\-likelihood MCQA setup\. Table[3](https://arxiv.org/html/2610.21227#S4.T3)compares the original English items, the raw DeepL translations before review, and the finalized KBS items using open\-weight chat\-template evaluations withacc\_npsq, restricted to items shared by all three versions\. Models score higher on KBS than on raw DeepL in all three model groups, with gains of 2\.35\-6\.91 percentage points\. These gains suggest that correcting translation errors and adapting questions to Korean contexts help models better understand the intended meaning of each question\. Models generally score higher on the English original than on the Korean variants on ARC\. However, the English\-Korean gap is task\-dependent: on WinoGrande, where only the KBS\-English comparison is meaningful, the gap is much smaller \(Table[19](https://arxiv.org/html/2610.21227#A6.T19)\)\.

#### Scoring differences in MCQA tasks\.

Table 4:Mean multiple\-choice accuracy of open\-weight models under three scoring rules: raw log\-likelihood \(acc\), byte\-length normalized log\-likelihood \(acc\_norm\), and NPSQ\-based accuracy \(acc\_npsq\)\.Table[4](https://arxiv.org/html/2610.21227#S4.T4)shows that the effect of the scoring rule is task\-dependent\. NPSQ\-based accuracy is lower than raw accuracy on Ko\-LAMBADA, especially under chat\-template prompting, where the mean score drops from 81\.22 to 69\.20\. Ko\-WinoGrande shows a smaller decrease under NPSQ\. In contrast, both Ko\-ARC subsets often obtain higher scores underacc\_npsqthan under raw accuracy, with the larger difference on Ko\-ARC\-Challenge\. Byte\-length normalization has its largest effect on Ko\-ARC\-Challenge\. These results suggest that answer\-choice priors affect Ko\-LAMBADA most strongly, while the impact of scoring rules varies with the structure of each multiple\-choice task\.

Figure 3:Performance of Qwen3\.5 Base \(pretrained\) and instruction\-tuned variants at five paired model sizes under plain prompting\. We report NPSQ accuracy for MCQA, strict exact match for Ko\-GSM8K, prompt\-level strict accuracy for Ko\-IFEval, and the EQ\-Bench score for Ko\-EQ\-Bench\. Accuracies are expressed as percentages\. 35B denotes 35B\-A3B\.
#### Effects of prompt formatting\.

Prompt format has a large effect on open\-weight models, but the direction of the effect differs by model family \(Appendix[F](https://arxiv.org/html/2610.21227#A6)reports the complete results, including plain prompting averages over the chat\-template subset\)\. For several chat models, applying the chat template substantially improves generation performance over plain prompting\. For example, Gemma 4 and EXAONE\-4\.0 model families show large gains under chat prompting across Ko\-GSM8K, Ko\-IFEval, and Ko\-EQ\-Bench\. These cases highlight that evaluations without the intended chat template can substantially underestimate the capabilities of instruction\-tuned models\.

Table 5:Qwen3\.5 results under default chat prompting and with thinking disabled \(Off\)\. Ko\-GSM8K reports strict exact match \(%\)\. Ko\-EQ\-Bench reports the EQ\-Bench score\. Both conditions use chat templates\.The reverse pattern appears in the Qwen3\.5 series\. Models larger than 2B perform well on Ko\-GSM8K under plain prompting but drop to almost zero under chat prompting, and flexible answer extraction does not recover the scores\. Output inspection suggests that the chat template leads these larger models to produce long English reasoning traces, often without reaching a final Korean answer \(Appendix[E](https://arxiv.org/html/2610.21227#A5)\)\. Prior work has also reported English\-centric representations and language inconsistency in multilingual reasoning\([Schut et al\., 2025](https://arxiv.org/html/2610.21227#bib.bib54);[Zhang et al\., 2026](https://arxiv.org/html/2610.21227#bib.bib55)\)\. We additionally evaluate six instruction\-tuned Qwen3\.5 models with thinking disabled throughenable\_thinking=False, while retaining chat templates\. As Table[5](https://arxiv.org/html/2610.21227#S4.T5)shows, the four larger models recover to 69\.60\-80\.67% strict exact match on Ko\-GSM8K and 53\.11\-72\.47 on Ko\-EQ\-Bench\. These results indicate that enabling thinking does not necessarily improve performance on these Korean tasks under the evaluated generation settings\. The recovery with thinking disabled highlights the need to account for reasoning mode when interpreting benchmark scores\.

A different pattern appears for some Korean\-developed models\. Mi:dm 2\.0 Base and Kanana\-2\-30b\-a3b\-instruct\-2601 obtain much higher flexible extract than strict match scores on Ko\-GSM8K under chat prompting, indicating that many outputs contain the correct numerical answer but do not follow the expected final answer format\. In these cases, the low strict score reflects answer format mismatch rather than complete reasoning failure\. Taken together, these results show that prompt format is not a minor implementation detail\. It interacts with the model’s post\-training and can change whether the model follows the intended task, switches to English reasoning on Korean inputs, or fails the required answer format\.

#### Effects of instruction tuning and model size\.

Figure[3](https://arxiv.org/html/2610.21227#S4.F3)compares Qwen3\.5 Base and instruction variants across model sizes under plain prompting\. Larger models generally perform better, especially on Ko\-ARC and Ko\-GSM8K\. However, instruction tuning is not a uniform gain in multilingual model families\. In Qwen3\.5, the instruction variant improves some scores, such as Ko\-GSM8K at larger sizes, but does not consistently outperform the Base model across MCQA tasks or Ko\-IFEval\. A similar pattern appears in Gemma under plain prompting, where instruction\-tuned models often underperform their Base counterparts\. On the other hand, the Kanana series shows progressive improvements on Ko\-GSM8K and Ko\-IFEval under plain prompting, with the Instruct variant also achieving the highest Ko\-EQ\-Bench score among the three variants\. These results suggest that instruction tuning effects depend on what the post\-training process is optimized for, including the target language, task format, and prompting style\.

#### Closed API models\.

Table[6](https://arxiv.org/html/2610.21227#S4.T6)compares the best open\-weight and closed API scores under a common generative MCQA format with default chat configurations\. The best closed scores exceed the best open\-weight scores by 1\.06\-2\.62 percentage points on Ko\-ARC and Ko\-LAMBADA, while the gap is larger on Ko\-WinoGrande \(8\.52 points\)\. With thinking disabled, Qwen3\.5\-27B achieves 98\.53% on Ko\-ARC\-Easy, further narrowing the gap to 0\.59 points\.

The relative performance also varies across generation tasks\. On Ko\-GSM8K, GPT\-5\.5 and Claude Opus 4\.7 achieve 96\.29% and 96\.06% strict exact match, respectively, compared with 82\.71% for Gemma\-4\-31B\-it, the best open\-weight model under default chat prompting\. On Ko\-IFEval, however, Gemma\-4\-31B\-it achieves 85\.02% prompt\-level strict accuracy, exceeding the best closed score of 81\.09% from GPT\-5\.5\. Complete results are provided in Appendix[F](https://arxiv.org/html/2610.21227#A6)\.

Table 6:Best generative MCQA accuracy \(%\) among open\-weight and closed API models under default chat configurations\. Gemma\-4\-31B\-it leads the open\-weight group on all four tasks\. GPT\-5\.5 leads the closed group on Ko\-ARC\-Challenge and Claude Opus 4\.7 on the other tasks\. Gap denotes the accuracy difference \(closed minus open\), in percentage points\. Qwen non\-thinking runs are reported separately in Appendix[F](https://arxiv.org/html/2610.21227#A6)\.

## 5Conclusion

We present a Korean Benchmark Suite for evaluating foundation models across six capabilities\. The suite is constructed through task\-level routes that combine reviewed translation and localization, direct Korean construction, and Korean\-specific redesign\. To our knowledge, few Korean evaluation suites jointly address broad capability coverage, task\-specific benchmark construction, differences in model access, and prompt\-formatting conditions\. For multiple\-choice tasks, we use log\-likelihood\-based scoring and NPSQ\-based accuracy for open\-weight models, and evaluate closed API models via label generation\. For generation tasks, both open\-weight and closed\-API models are evaluated based on model\-generated outputs\. We further compare plain prompting and chat\-template prompting for open\-weight models, where applicable, while closed\-API models are evaluated under chat\-style prompting\. The results show that Korean benchmark scores are shaped by task construction, scoring rules, prompt format, and model post\-training choices\. These factors can change both the measured score and the interpretation of model behavior\. We therefore suggest that Korean evaluations report not only final scores but also how each task is adapted, how each score is computed, and under which prompting and model\-access settings the score is obtained\.

## Limitations

Although the Korean Benchmark Suite covers six capabilities, it does not represent the full range of Korean language use, such as long\-form generation, multi\-turn dialogue, or safety\-related behavior\. Moreover, some tasks are adapted from English benchmarks, so residual translation or localization artifacts may remain despite author\-led review\. In addition, our evaluation relies on automatic metrics and single\-run greedy decoding, which may not capture all aspects of model behavior\. Future work should evaluate robustness across broader prompting and decoding conditions\.

KBS may be affected by contamination from publicly available source material\. The English\-Korean score gaps suggest sensitivity to evaluation language and adaptation, but do not rule out contamination\.

## Ethical Considerations

This work aims to improve the reliability and reproducibility of Korean foundation model evaluation by providing a benchmark suite with documented construction and evaluation protocols\. We expect the suite to support more transparent comparison of Korean and multilingual models, and to make evaluation resources for Korean more accessible to the research community\. We caution that benchmark scores should be interpreted only as reflecting performance under the specific tasks, prompts, scoring rules, and evaluation settings described in this paper\.

We have released the evaluation code under the MIT license\. Dataset subsets adapted from existing English benchmarks have been released under the licenses of their respective source datasets, including CC\-BY for WinoGrande, CC\-BY\-SA for ARC, MIT for GSM8K and EQ\-Bench, and Apache 2\.0 for IFEval\. Ko\-LAMBADA, which is newly constructed from copyright\-cleared Korean literary texts, has been released under CC\-BY\-NC\-SA 4\.0\.

## Acknowledgments

This work was partially supported by the National Research Foundation of Korea \(NRF\) under Grant No\. RS\-2023\-00222663 \(Center for Optimizing Hyperscale AI Models and Platforms\) and under Grant No\. A400\-20260031, and by the Institute for Information and Communications Technology Promotion \(IITP\) under Grant No\. 2018\-0\-00581 \(CUDA Programming Environment for FPGA Clusters\) and No\. RS\-2025\-02304554 \(Efficient and Scalable Framework for AI Heterogeneous Cluster Systems\), all funded by the Ministry of Science and ICT \(MSIT\) of Korea\. It was also partially supported by the Korea Health Industry Development Institute \(KHIDI\) under Grant No\. RS\-2025\-25454559 \(Frailty Risk Assessment and Intervention Leveraging Multimodal Intelligence for Networked Deployment in Community Care\), funded by the Ministry of Health and Welfare \(MOHW\) of Korea\. Additional support was provided by the BK21 Plus Program for Innovative Data Science Talent Education \(Department of Data Science, Seoul National University, No\. 5199990914569\) and the BK21 FOUR Program for Intelligent Computing \(Department of Computer Science and Engineering, Seoul National University, No\. 4199990214639\), both funded by the Ministry of Education \(MOE\) of Korea\. This work was also partially supported by the Advanced GPU Utilization Support Program, funded by the Ministry of Science and ICT \(MSIT\) of Korea and operated by the National IT Industry Promotion Agency \(NIPA\)\. Research facilities were provided by the Institute of Computer Technology \(ICT\) at Seoul National University\.

## References

- ANTHROPICSystem card: claude haiku 4\.5\.External Links:[Link](https://www-cdn.anthropic.com/7aad69bf12627d42234e01ee7c36305dc2f6a970.pdf)Cited by:[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px1.p1.1)\.
- ANTHROPIC \(2026\)ANTHROPICSystem card: claude opus 4\.7\.External Links:[Link](https://cdn.sanity.io/files/4zrzovbb/website/037f06850df7fbe871e206dad004c3db5fd50340.pdf)Cited by:[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px1.p1.1)\.
- Bet al\.\(2024\)S\. B, P\. R\. P, S\. M\. B, and K\. SThe evolution of large language model: models, applications and challenges\.In2024 International Conference on Current Trends in Advanced Computing \(ICCTAC\),pp\. 1–8\.External Links:[Document](https://dx.doi.org/10.1109/ICCTAC61556.2024.10581180)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1)\.
- Baeet al\.\(2026\)K\. Bae, E\. Choi, K\. Choi, S\. J\. Choi, Y\. Choi, K\. Han, S\. Hong, J\. Hwang, T\. Hwang, J\. Jang, H\. Jeon, K\. Jeon, G\. J\. Jo, H\. Jo, J\. Jung, E\. Kim, H\. Kim, J\. Kim, J\. Kim, S\. Kim, S\. Kim, S\. Kim, Y\. Kim, Y\. Kim, Y\. Kim, E\. H\. Lee, G\. Lee, H\. Lee, H\. Lee, J\. Lee, K\. Lee, S\. Park, Y\. M\. Paik, Y\. Park, Y\. Park, S\. Seo, S\. Yang, H\. Yeen, S\. Yi, and H\. YunEXAONE 4\.0: unified large language models integrating non\-reasoning and reasoning modes\.External Links:2507\.11407,[Link](https://arxiv.org/abs/2507.11407)Cited by:[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px1.p2.1)\.
- Balepuret al\.\(2024\)N\. Balepur, A\. Ravichander, and R\. RudingerArtifacts or abduction: how do LLMs answer multiple\-choice questions without the question?\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 10308–10330\.External Links:[Link](https://aclanthology.org/2024.acl-long.555/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.555)Cited by:[§2\.2](https://arxiv.org/html/2610.21227#S2.SS2.p1.1)\.
- Balepuret al\.\(2025\)N\. Balepur, R\. Rudinger, and J\. Boyd\-GraberWhich of these best describes multiple choice evaluation with LLMs? a\) forced B\) flawed C\) fixable D\) all of the above\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 3394–3418\.External Links:[Link](https://aclanthology.org/2025.acl-long.169/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.169),ISBN 979\-8\-89176\-251\-0Cited by:[§2\.2](https://arxiv.org/html/2610.21227#S2.SS2.p1.1)\.
- Beanet al\.\(2025\)A\. M\. Bean, R\. O\. Kearns, A\. Romanou, F\. S\. Hafner, H\. Mayne, J\. Batzner, N\. Foroutan, C\. Schmitz, K\. Korgul, H\. Batra, O\. Deb, E\. Beharry, C\. Emde, T\. Foster, A\. Gausen, M\. Grandury, S\. Han, V\. Hofmann, L\. Ibrahim, H\. Kim, H\. R\. Kirk, F\. Lin, G\. K\. Liu, L\. Luettgau, J\. Magomere, J\. Rystrøm, A\. Sotnikova, Y\. Yang, Y\. Zhao, A\. Bibi, A\. Bosselut, R\. Clark, A\. Cohan, J\. N\. Foerster, Y\. Gal, S\. A\. Hale, I\. D\. Raji, C\. Summerfield, P\. Torr, C\. Ududec, L\. Rocher, and A\. MahdiMeasuring what matters: construct validity in large language model benchmarks\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=mdA5lVvNcU)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1)\.
- Bidermanet al\.\(2026\)S\. Biderman, H\. Schoelkopf, L\. Sutawika, L\. Gao, J\. Tow, B\. Abbasi, A\. F\. Aji, P\. S\. Ammanamanchi, S\. Black, J\. Clive, A\. DiPofi, J\. Etxaniz, B\. Fattori, J\. Z\. Forde, C\. Foster, J\. Hsu, M\. Jaiswal, W\. Y\. Lee, H\. Li, C\. Lovering, N\. Muennighoff, E\. Pavlick, J\. Phang, A\. Skowron, S\. Tan, X\. Tang, K\. A\. Wang, G\. I\. Winata, F\. Yvon, and A\. ZouLessons from the trenches on reproducible evaluation of language models\.External Links:2405\.14782,[Link](https://arxiv.org/abs/2405.14782)Cited by:[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px1.p1.1)\.
- Brownet al\.\(2020\)T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell, S\. Agarwal, A\. Herbert\-Voss, G\. Krueger, T\. Henighan, R\. Child, A\. Ramesh, D\. M\. Ziegler, J\. Wu, C\. Winter, C\. Hesse, M\. Chen, E\. Sigler, M\. Litwin, S\. Gray, B\. Chess, J\. Clark, C\. Berner, S\. McCandlish, A\. Radford, I\. Sutskever, and D\. AmodeiLanguage models are few\-shot learners\.InProceedings of the 34th International Conference on Neural Information Processing Systems,NIPS ’20,Red Hook, NY, USA\.External Links:ISBN 9781713829546Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1)\.
- Changet al\.\(2023\)Y\. Chang, X\. Wang, J\. Wang, Y\. Wu, L\. Yang, K\. Zhu, H\. Chen, X\. Yi, C\. Wang, Y\. Wang, W\. Ye, Y\. Zhang, Y\. Chang, P\. S\. Yu, Q\. Yang, and X\. XieA survey on evaluation of large language models\.External Links:2307\.03109,[Link](https://arxiv.org/abs/2307.03109)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1)\.
- Chenet al\.\(2021\)M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. de Oliveira Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman, A\. Ray, R\. Puri, G\. Krueger, M\. Petrov, H\. Khlaaf, G\. Sastry, P\. Mishkin, B\. Chan, S\. Gray, N\. Ryder, M\. Pavlov, A\. Power, L\. Kaiser, M\. Bavarian, C\. Winter, P\. Tillet, F\. P\. Such, D\. Cummings, M\. Plappert, F\. Chantzis, E\. Barnes, A\. Herbert\-Voss, W\. H\. Guss, A\. Nichol, A\. Paino, N\. Tezak, J\. Tang, I\. Babuschkin, S\. Balaji, S\. Jain, W\. Saunders, C\. Hesse, A\. N\. Carr, J\. Leike, J\. Achiam, V\. Misra, E\. Morikawa, A\. Radford, M\. Knight, M\. Brundage, M\. Murati, K\. Mayer, P\. Welinder, B\. McGrew, D\. Amodei, S\. McCandlish, I\. Sutskever, and W\. ZarembaEvaluating large language models trained on code\.External Links:2107\.03374,[Link](https://arxiv.org/abs/2107.03374)Cited by:[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p1.1)\.
- Choet al\.\(2026\)G\. Cho, Y\. So, and J\. LeeChoices speak louder than questions\.InInternational Conference on Learning Representations,C\. Vondrick, B\. Hariharan, C\. Raffel, L\. Pinto, D\. Yang, and A\. Faust \(Eds\.\),Vol\.2026,pp\. 75943–75962\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/file/7ae2b0eedc560e3afbffd68445b8a220-Paper-Conference.pdf)Cited by:[Appendix D](https://arxiv.org/html/2610.21227#A4.SS0.SSS0.Px6.p1.1),[§1](https://arxiv.org/html/2610.21227#S1.p5.1),[§2\.2](https://arxiv.org/html/2610.21227#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px6.p1.1)\.
- Chuaet al\.\(2025\)L\. Chua, B\. Ghazi, Y\. Huang, P\. Kamath, R\. Kumar, P\. Manurangsi, A\. Sinha, C\. Xie, and C\. ZhangCrosslingual capabilities and knowledge barriers in multilingual large language models\.External Links:2406\.16135,[Link](https://arxiv.org/abs/2406.16135)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p3.1)\.
- Clarket al\.\(2018\)P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. TafjordThink you have solved question answering? try arc, the ai2 reasoning challenge\.External Links:1803\.05457,[Link](https://arxiv.org/abs/1803.05457)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.21227#S3.SS1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. SchulmanTraining verifiers to solve math word problems\.External Links:2110\.14168,[Link](https://arxiv.org/abs/2110.14168)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.21227#S3.SS1.p1.1)\.
- Desaiet al\.\(2024\)A\. P\. Desai, R\. Prajapati, T\. Ravi, M\. Luqman, and P\. YadavEmerging trends in llm benchmarking\.In2024 IEEE International Conference on Big Data \(BigData\),pp\. 8805–8807\.External Links:[Document](https://dx.doi.org/10.1109/BigData62323.2024.10825845)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1)\.
- Doddapaneniet al\.\(2025\)S\. Doddapaneni, M\. S\. U\. R\. Khan, D\. Venkatesh, R\. Dabre, A\. Kunchukuttan, and M\. M\. KhapraCross\-lingual auto evaluation for assessing multilingual LLMs\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 29297–29329\.External Links:[Link](https://aclanthology.org/2025.acl-long.1419/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1419),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p3.1)\.
- Durmuset al\.\(2022\)E\. Durmus, F\. Ladhak, and T\. HashimotoSpurious correlations in reference\-free evaluation of text generation\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 1443–1454\.External Links:[Link](https://aclanthology.org/2022.acl-long.102/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.102)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1)\.
- Google DeepMind \(2026\)Google DeepMindGemma 4 model card\.External Links:[Link](https://ai.google.dev/gemma/docs/core/model_card_4)Cited by:[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px1.p2.1)\.
- Guhaet al\.\(2023\)N\. Guha, J\. Nyarko, D\. E\. Ho, C\. Ré, A\. Chilton, A\. Narayana, A\. Chohlas\-Wood, A\. Peters, B\. Waldon, D\. N\. Rockmore, D\. Zambrano, D\. Talisman, E\. Hoque, F\. Surani, F\. Fagan, G\. Sarfaty, G\. M\. Dickinson, H\. Porat, J\. Hegland, J\. Wu, J\. Nudell, J\. Niklaus, J\. Nay, J\. H\. Choi, K\. Tobia, M\. Hagan, M\. Ma, M\. Livermore, N\. Rasumov\-Rahe, N\. Holzenberger, N\. Kolt, P\. Henderson, S\. Rehaag, S\. Goel, S\. Gao, S\. Williams, S\. Gandhi, T\. Zur, V\. Iyer, and Z\. LiLEGALBENCH: a collaboratively built benchmark for measuring legal reasoning in large language models\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p2.1)\.
- Hamet al\.\(2020\)J\. Ham, Y\. J\. Choe, K\. Park, I\. Choi, and H\. SohKorNLI and KorSTS: new benchmark datasets for Korean natural language understanding\.InFindings of the Association for Computational Linguistics: EMNLP 2020,T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 422–430\.External Links:[Link](https://aclanthology.org/2020.findings-emnlp.39/),[Document](https://dx.doi.org/10.18653/v1/2020.findings-emnlp.39)Cited by:[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. SteinhardtMeasuring massive multitask language understanding\.External Links:2009\.03300,[Link](https://arxiv.org/abs/2009.03300)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p1.1)\.
- Honget al\.\(2025\)S\. Hong, S\. Kim, G\. Son, S\. Kim, Y\. Hong, and J\. LeeFrom KMMLU\-redux to pro: a professional Korean benchmark suite for LLM evaluation\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 19067–19096\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.1038/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1038),ISBN 979\-8\-89176\-335\-7Cited by:[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p2.1)\.
- Janget al\.\(2022\)M\. Jang, D\. Kim, D\. S\. Kwon, and E\. DavisKoBEST: Korean balanced evaluation of significant tasks\.InProceedings of the 29th International Conference on Computational Linguistics,N\. Calzolari, C\. Huang, H\. Kim, J\. Pustejovsky, L\. Wanner, K\. Choi, P\. Ryu, H\. Chen, L\. Donatelli, H\. Ji, S\. Kurohashi, P\. Paggio, N\. Xue, S\. Kim, Y\. Hahm, Z\. He, T\. K\. Lee, E\. Santus, F\. Bond, and S\. Na \(Eds\.\),Gyeongju, Republic of Korea,pp\. 3697–3708\.External Links:[Link](https://aclanthology.org/2022.coling-1.325/)Cited by:[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p1.1)\.
- Jinet al\.\(2021\)D\. Jin, E\. Pan, N\. Oufattole, W\. Weng, H\. Fang, and P\. SzolovitsWhat disease does this patient have? a large\-scale open domain question answering dataset from medical exams\.Applied Sciences11\(14\)\.External Links:[Link](https://www.mdpi.com/2076-3417/11/14/6421),ISSN 2076\-3417,[Document](https://dx.doi.org/10.3390/app11146421)Cited by:[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p2.1)\.
- Kanana \(2025\)KananaRelease of the smarter and more efficient kanana\-2 open\-source models\.Note:Kakao Tech BlogOriginal Korean title: "더 똑똑하고 효율적인 Kanana\-2 오픈소스 공개"\. Accessed: 2026\-05\-02External Links:[Link](https://tech.kakao.com/posts/804)Cited by:[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px1.p2.1)\.
- Kimet al\.\(2024\)E\. Kim, J\. Suk, P\. Oh, H\. Yoo, J\. Thorne, and A\. OhCLIcK: a benchmark dataset of cultural and linguistic intelligence in Korean\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 3335–3346\.External Links:[Link](https://aclanthology.org/2024.lrec-main.296/)Cited by:[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p2.1)\.
- Kimet al\.\(2025\)H\. Kim, D\. Kim, J\. Kim, S\. Lee, Y\. Kim, and C\. ParkOpen Ko\-LLM leaderboard2: bridging foundational and practical evaluation for Korean LLMs\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 3: Industry Track\),W\. Chen, Y\. Yang, M\. Kachuee, and X\. Fu \(Eds\.\),Albuquerque, New Mexico,pp\. 266–273\.External Links:[Link](https://aclanthology.org/2025.naacl-industry.22/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-industry.22),ISBN 979\-8\-89176\-194\-0Cited by:[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p2.1)\.
- Koet al\.\(2025\)H\. Ko, G\. Son, and D\. ChoiUnderstand, solve and translate: bridging the multilingual mathematical reasoning gap\.InProceedings of the 5th Workshop on Multilingual Representation Learning \(MRL 2025\),D\. I\. Adelani, C\. Arnett, D\. Ataman, T\. A\. Chang, H\. Gonen, R\. Raja, F\. Schmidt, D\. Stap, and J\. Wang \(Eds\.\),Suzhuo, China,pp\. 78–95\.External Links:[Link](https://aclanthology.org/2025.mrl-main.6/),[Document](https://dx.doi.org/10.18653/v1/2025.mrl-main.6),ISBN 979\-8\-89176\-345\-6Cited by:[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p2.1)\.
- KT Tech\. Innovation Group \(2025\)KT Tech\. Innovation GroupMi:dm 2\.0: korea\-centric bilingual language models\.External Links:[Link](https://github.com/K-intelligence-Midm/Midm-2.0?tab=readme-ov-file)Cited by:[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px1.p2.1)\.
- Leeet al\.\(2024\)J\. Lee, M\. Kim, S\. Kim, J\. Kim, S\. Won, H\. Lee, and E\. ChoiKorNAT: LLM alignment benchmark for Korean social values and common knowledge\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 11177–11213\.External Links:[Link](https://aclanthology.org/2024.findings-acl.666/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.666)Cited by:[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p2.1)\.
- Lianget al\.\(2023\)P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar, B\. Newman, B\. Yuan, B\. Yan, C\. Zhang, C\. Cosgrove, C\. D\. Manning, C\. Ré, D\. Acosta\-Navas, D\. A\. Hudson, E\. Zelikman, E\. Durmus, F\. Ladhak, F\. Rong, H\. Ren, H\. Yao, J\. Wang, K\. Santhanam, L\. Orr, L\. Zheng, M\. Yuksekgonul, M\. Suzgun, N\. Kim, N\. Guha, N\. Chatterji, O\. Khattab, P\. Henderson, Q\. Huang, R\. Chi, S\. M\. Xie, S\. Santurkar, S\. Ganguli, T\. Hashimoto, T\. Icard, T\. Zhang, V\. Chaudhary, W\. Wang, X\. Li, Y\. Mai, Y\. Zhang, and Y\. KoreedaHolistic evaluation of language models\.External Links:2211\.09110,[Link](https://arxiv.org/abs/2211.09110)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p2.1)\.
- Molfeseet al\.\(2025\)F\. M\. Molfese, L\. Moroni, L\. Gioffré, A\. Scirè, S\. Conia, and R\. NavigliRight answer, wrong score: uncovering the inconsistencies of LLM evaluation in multiple\-choice question answering\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 18477–18494\.External Links:[Link](https://aclanthology.org/2025.findings-acl.950/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.950),ISBN 979\-8\-89176\-256\-5Cited by:[§2\.2](https://arxiv.org/html/2610.21227#S2.SS2.p1.1)\.
- Paech \(2024\)S\. J\. PaechEQ\-bench: an emotional intelligence benchmark for large language models\.External Links:2312\.06281,[Link](https://arxiv.org/abs/2312.06281)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1),[§3\.1](https://arxiv.org/html/2610.21227#S3.SS1.p1.1)\.
- Papernoet al\.\(2016\)D\. Paperno, G\. Kruszewski, A\. Lazaridou, N\. Q\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. FernándezThe LAMBADA dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),K\. Erk and N\. A\. Smith \(Eds\.\),Berlin, Germany,pp\. 1525–1534\.External Links:[Link](https://aclanthology.org/P16-1144/),[Document](https://dx.doi.org/10.18653/v1/P16-1144)Cited by:[§3\.1](https://arxiv.org/html/2610.21227#S3.SS1.p1.1)\.
- Parket al\.\(2024\)C\. Park, H\. Kim, D\. Kim, S\. Cho, S\. Kim, S\. Lee, Y\. Kim, and H\. LeeOpen Ko\-LLM leaderboard: evaluating large language models in Korean with Ko\-h5 benchmark\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 3220–3234\.External Links:[Link](https://aclanthology.org/2024.acl-long.177/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.177)Cited by:[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p2.1)\.
- Parket al\.\(2021\)S\. Park, J\. Moon, S\. Kim, W\. I\. Cho, J\. Y\. Han, J\. Park, C\. Song, J\. Kim, Y\. Song, T\. Oh, J\. Lee, J\. Oh, S\. Lyu, Y\. Jeong, I\. Lee, S\. Seo, D\. Lee, H\. Kim, M\. Lee, S\. Jang, S\. Do, S\. Kim, K\. Lim, J\. Lee, K\. Park, J\. Shin, S\. Kim, L\. Park, L\. Park, A\. Oh, J\. Ha \(NAVER AI Lab\), K\. Cho, and K\. ChoKLUE: korean language understanding evaluation\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,J\. Vanschoren and S\. Yeung \(Eds\.\),Vol\.1\.External Links:[Link](https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/98dce83da57b0395e163467c9dae521b-Paper-round2.pdf)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p2.1),[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p1.1)\.
- Pezeshkpour and Hruschka \(2024\)P\. Pezeshkpour and E\. HruschkaLarge language models sensitivity to the order of options in multiple\-choice questions\.InFindings of the Association for Computational Linguistics: NAACL 2024,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 2006–2017\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.130/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.130)Cited by:[§2\.2](https://arxiv.org/html/2610.21227#S2.SS2.p1.1)\.
- Phanet al\.\(2025\)L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi,et al\.Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.Cited by:[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p2.1)\.
- Poloet al\.\(2024\)F\. M\. Polo, R\. Xu, L\. Weber, M\. Silva, O\. Bhardwaj, L\. Choshen, A\. F\. M\. de Oliveira, Y\. Sun, and M\. YurochkinEfficient multi\-prompt evaluation of llms\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§2\.2](https://arxiv.org/html/2610.21227#S2.SS2.p1.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px1.p2.1)\.
- Reinet al\.\(2024\)D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. BowmanGPQA: a graduate\-level google\-proof q&a benchmark\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Ti67584b98)Cited by:[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p2.1)\.
- Sakaguchiet al\.\(2021\)K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. ChoiWinoGrande: an adversarial winograd schema challenge at scale\.Commun\. ACM64\(9\),pp\. 99–106\.External Links:ISSN 0001\-0782,[Link](https://doi.org/10.1145/3474381),[Document](https://dx.doi.org/10.1145/3474381)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.21227#S3.SS1.p1.1)\.
- Schutet al\.\(2025\)L\. Schut, Y\. Gal, and S\. FarquharDo multilingual llms think in english?\.External Links:2502\.15603,[Link](https://arxiv.org/abs/2502.15603)Cited by:[§4\.2](https://arxiv.org/html/2610.21227#S4.SS2.SSS0.Px4.p2.1)\.
- Sclaret al\.\(2024\)M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. SuhrQuantifying language models’ sensitivity to spurious features in prompt design or: how i learned to start worrying about prompt formatting\.External Links:2310\.11324,[Link](https://arxiv.org/abs/2310.11324)Cited by:[§2\.2](https://arxiv.org/html/2610.21227#S2.SS2.p1.1)\.
- Seoet al\.\(2024\)J\. Seo, J\. Lee, C\. Park, S\. Hong, S\. Lee, and H\. LimKoCommonGEN v2: a benchmark for navigating Korean commonsense reasoning challenges in large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 2390–2415\.External Links:[Link](https://aclanthology.org/2024.findings-acl.141/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.141)Cited by:[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p2.1)\.
- Shenet al\.\(2024\)Y\. Shen, K\. Song, X\. Tan, W\. Zhang, K\. Ren, S\. Yuan, W\. Lu, D\. Li, and Y\. ZhuangTaskBench: benchmarking large language models for task automation\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p2.1)\.
- Singhet al\.\(2026\)A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram, A\. Nathan, A\. Luo, A\. Helyar, A\. Madry, A\. Efremov, A\. Spyra, A\. Baker\-Whitcomb, A\. Beutel, A\. Karpenko, A\. Makelov, A\. Neitz, A\. Wei, A\. Barr, A\. Kirchmeyer, A\. Ivanov, A\. Christakis, A\. Gillespie, A\. Tam, A\. Bennett, A\. Wan, A\. Huang, A\. M\. Sandjideh, A\. Yang, A\. Kumar, A\. Saraiva, A\. Vallone, A\. Gheorghe, A\. G\. Garcia, A\. Braunstein, A\. Liu, A\. Schmidt, A\. Mereskin, A\. Mishchenko, A\. Applebaum, A\. Rogerson, A\. Rajan, A\. Wei, A\. Kotha, A\. Srivastava, A\. Agrawal, A\. Vijayvergiya, A\. Tyra, A\. Nair, A\. Nayak, B\. Eggers, B\. Ji, B\. Hoover, B\. Chen, B\. Chen, B\. Barak, B\. Minaiev, B\. Hao, B\. Baker, B\. Lightcap, B\. McKinzie, B\. Wang, B\. Quinn, B\. Fioca, B\. Hsu, B\. Yang, B\. Yu, B\. Zhang, B\. Brenner, C\. R\. Zetino, C\. Raymond, C\. Lugaresi, C\. Paz, C\. Hudson, C\. Whitney, C\. Li, C\. Chen, C\. Cole, C\. Voss, C\. Ding, C\. Shen, C\. Huang, C\. Colby, C\. Hallacy, C\. Koch, C\. Lu, C\. Kaplan, C\. Kim, C\. Minott\-Henriques, C\. Frey, C\. Yu, C\. Czarnecki, C\. Reid, C\. Wei, C\. Decareaux, C\. Scheau, C\. Zhang, C\. Forbes, D\. Tang, D\. Goldberg, D\. Roberts, D\. Palmie, D\. Kappler, D\. Levine, D\. Wright, D\. Leo, D\. Lin, D\. Robinson, D\. Grabb, D\. Chen, D\. Lim, D\. Salama, D\. Bhattacharjee, D\. Tsipras, D\. Li, D\. Yu, D\. Strouse, D\. Williams, D\. Hunn, E\. Bayes, E\. Arbus, E\. Akyurek, E\. Y\. Le, E\. Widmann, E\. Yani, E\. Proehl, E\. Sert, E\. Cheung, E\. Schwartz, E\. Han, E\. Jiang, E\. Mitchell, E\. Sigler, E\. Wallace, E\. Ritter, E\. Kavanaugh, E\. Mays, E\. Nikishin, F\. Li, F\. P\. Such, F\. de Avila Belbute Peres, F\. Raso, F\. Bekerman, F\. Tsimpourlas, F\. Chantzis, F\. Song, F\. Zhang, G\. Raila, G\. McGrath, G\. Briggs, G\. Yang, G\. Parascandolo, G\. Chabot, G\. Kim, G\. Zhao, G\. Valiant, G\. Leclerc, H\. Salman, H\. Wang, H\. Sheng, H\. Jiang, H\. Wang, H\. Jin, H\. Sikchi, H\. Schmidt, H\. Aspegren, H\. Chen, H\. Qiu, H\. Lightman, I\. Covert, I\. Kivlichan, I\. Silber, I\. Sohl, I\. Hammoud, I\. Clavera, I\. Lan, I\. Akkaya, I\. Kostrikov, I\. Kofman, I\. Etinger, I\. Singal, J\. Hehir, J\. Huh, J\. Pan, J\. Wilczynski, J\. Pachocki, J\. Lee, J\. Quinn, J\. Kiros, J\. Kalra, J\. Samaroo, J\. Wang, J\. Wolfe, J\. Chen, J\. Wang, J\. Harb, J\. Han, J\. Wang, J\. Zhao, J\. Chen, J\. Yang, J\. Tworek, J\. Chand, J\. Landon, J\. Liang, J\. Lin, J\. Liu, J\. Wang, J\. Tang, J\. Yin, J\. Jang, J\. Morris, J\. Flynn, J\. Ferstad, J\. Heidecke, J\. Fishbein, J\. Hallman, J\. Grant, J\. Chien, J\. Gordon, J\. Park, J\. Liss, J\. Kraaijeveld, J\. Guay, J\. Mo, J\. Lawson, J\. McGrath, J\. Vendrow, J\. Jiao, J\. Lee, J\. Steele, J\. Wang, J\. Mao, K\. Chen, K\. Hayashi, K\. Xiao, K\. Salahi, K\. Wu, K\. Sekhri, K\. Sharma, K\. Singhal, K\. Li, K\. Nguyen, K\. Gu\-Lemberg, K\. King, K\. Liu, K\. Stone, K\. Yu, K\. Ying, K\. Georgiev, K\. Lim, K\. Tirumala, K\. Miller, L\. Ahmad, L\. Lv, L\. Clare, L\. Fauconnet, L\. Itow, L\. Yang, L\. Romaniuk, L\. Anise, L\. Byron, L\. Pathak, L\. Maksin, L\. Lo, L\. Ho, L\. Jing, L\. Wu, L\. Xiong, L\. Mamitsuka, L\. Yang, L\. McCallum, L\. Held, L\. Bourgeois, L\. Engstrom, L\. Kuhn, L\. Feuvrier, L\. Zhang, L\. Switzer, L\. Kondraciuk, L\. Kaiser, M\. Joglekar, M\. Singh, M\. Shah, M\. Stratta, M\. Williams, M\. Chen, M\. Sun, M\. Cayton, M\. Li, M\. Zhang, M\. Aljubeh, M\. Nichols, M\. Haines, M\. Schwarzer, M\. Gupta, M\. Shah, M\. Y\. Guan, M\. Huang, M\. Dong, M\. Wang, M\. Glaese, M\. Carroll, M\. Lampe, M\. Malek, M\. Sharman, M\. Zhang, M\. Wang, M\. Pokrass, M\. Florian, M\. Pavlov, M\. Wang, M\. Chen, M\. Wang, M\. Feng, M\. Bavarian, M\. Lin, M\. Abdool, M\. Rohaninejad, N\. Soto, N\. Staudacher, N\. LaFontaine, N\. Marwell, N\. Liu, N\. Preston, N\. Turley, N\. Ansman, N\. Blades, N\. Pancha, N\. Mikhaylin, N\. Felix, N\. Handa, N\. Rai, N\. Keskar, N\. Brown, O\. Nachum, O\. Boiko, O\. Murk, O\. Watkins, O\. Gleeson, P\. Mishkin, P\. Lesiewicz, P\. Baltescu, P\. Belov, P\. Zhokhov, P\. Pronin, P\. Guo, P\. Thacker, Q\. Liu, Q\. Yuan, Q\. Liu, R\. Dias, R\. Puckett, R\. Arora, R\. T\. Mullapudi, R\. Gaon, R\. Miyara, R\. Song, R\. Aggarwal, R\. Marsan, R\. Yemiru, R\. Xiong, R\. Kshirsagar, R\. Nuttall, R\. Tsiupa, R\. Eldan, R\. Wang, R\. James, R\. Ziv, R\. Shu, R\. Nigmatullin, S\. Jain, S\. Talaie, S\. Altman, S\. Arnesen, S\. Toizer, S\. Toyer, S\. Miserendino, S\. Agarwal, S\. Yoo, S\. Heon, S\. Ethersmith, S\. Grove, S\. Taylor, S\. Bubeck, S\. Banesiu, S\. Amdo, S\. Zhao, S\. Wu, S\. Santurkar, S\. Zhao, S\. R\. Chaudhuri, S\. Krishnaswamy, Shuaiqi, Xia, S\. Cheng, S\. Anadkat, S\. P\. Fishman, S\. Tobin, S\. Fu, S\. Jain, S\. Mei, S\. Egoian, S\. Kim, S\. Golden, S\. Mah, S\. Lin, S\. Imm, S\. Sharpe, S\. Yadlowsky, S\. Choudhry, S\. Eum, S\. Sanjeev, T\. Khan, T\. Stramer, T\. Wang, T\. Xin, T\. Gogineni, T\. Christianson, T\. Sanders, T\. Patwardhan, T\. Degry, T\. Shadwell, T\. Fu, T\. Gao, T\. Garipov, T\. Sriskandarajah, T\. Sherbakov, T\. Korbak, T\. Kaftan, T\. Hiratsuka, T\. Wang, T\. Song, T\. Zhao, T\. Peterson, V\. Kharitonov, V\. Chernova, V\. Kosaraju, V\. Kuo, V\. Pong, V\. Verma, V\. Petrov, W\. Jiang, W\. Zhang, W\. Zhou, W\. Xie, W\. Zhan, W\. McCabe, W\. DePue, W\. Ellsworth, W\. Bain, W\. Thompson, X\. Chen, X\. Qi, X\. Xiang, X\. Shi, Y\. Dubois, Y\. Yu, Y\. Khakbaz, Y\. Wu, Y\. Qian, Y\. T\. Lee, Y\. Chen, Y\. Zhang, Y\. Xiong, Y\. Tian, Y\. Cha, Y\. Bai, Y\. Yang, Y\. Yuan, Y\. Li, Y\. Zhang, Y\. Yang, Y\. Jin, Y\. Jiang, Y\. Wang, Y\. Wang, Y\. Liu, Z\. Stubenvoll, Z\. Dou, Z\. Wu, and Z\. WangOpenAI gpt\-5 system card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px1.p1.1)\.
- SKT Telecom \(2025\)SKT TelecomA\.X 4\.0: foundation model specialized in korean, optimized for enterprise applications\.External Links:[Link](https://github.com/SKT-AI/A.X-4.0)Cited by:[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px1.p2.1)\.
- Sonet al\.\(2025a\)G\. Son, H\. Ko, and D\. ChoiMulti\-step reasoning in Korean and the emergent mirage\.InProceedings of the 3rd Workshop on Cross\-Cultural Considerations in NLP \(C3NLP 2025\),V\. Prabhakaran, S\. Dev, L\. Benotti, D\. Hershcovich, Y\. Cao, L\. Zhou, L\. Cabello, and I\. Adebara \(Eds\.\),Albuquerque, New Mexico,pp\. 10–21\.External Links:[Link](https://aclanthology.org/2025.c3nlp-1.2/),[Document](https://dx.doi.org/10.18653/v1/2025.c3nlp-1.2),ISBN 979\-8\-89176\-237\-4Cited by:[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p2.1)\.
- Sonet al\.\(2025b\)G\. Son, H\. Lee, S\. Kim, S\. Kim, N\. Muennighoff, T\. Choi, C\. Park, K\. M\. Yoo, and S\. BidermanKMMLU: measuring massive multitask language understanding in Korean\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 4076–4104\.External Links:[Link](https://aclanthology.org/2025.naacl-long.206/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.206),ISBN 979\-8\-89176\-189\-6Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p2.1),[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p2.1)\.
- Sonet al\.\(2024\)G\. Son, H\. Lee, S\. Kim, H\. Kim, J\. c\. Lee, J\. W\. Yeom, J\. Jung, J\. w\. Kim, and S\. KimHAE\-RAE bench: evaluation of Korean knowledge in language models\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),N\. Calzolari, M\. Kan, V\. Hoste, A\. Lenci, S\. Sakti, and N\. Xue \(Eds\.\),Torino, Italia,pp\. 7993–8007\.External Links:[Link](https://aclanthology.org/2024.lrec-main.704/)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p2.1),[§2\.3](https://arxiv.org/html/2610.21227#S2.SS3.p2.1)\.
- Srivastavaet al\.\(2023\)A\. Srivastava, A\. Rastogi, A\. Rao, A\. A\. M\. Shoeb, A\. Abid, A\. Fisch, A\. R\. Brown, A\. Santoro, A\. Gupta, A\. Garriga\-Alonso, A\. Kluska, A\. Lewkowycz, A\. Agarwal, A\. Power, A\. Ray, A\. Warstadt, A\. W\. Kocurek, A\. Safaya, A\. Tazarv, A\. Xiang, A\. Parrish, A\. Nie, A\. Hussain, A\. Askell, A\. Dsouza, A\. Slone, A\. Rahane, A\. S\. Iyer, A\. Andreassen, A\. Madotto, A\. Santilli, A\. Stuhlmüller, A\. Dai, A\. La, A\. Lampinen, A\. Zou, A\. Jiang, A\. Chen, A\. Vuong, A\. Gupta, A\. Gottardi, A\. Norelli, A\. Venkatesh, A\. Gholamidavoodi, A\. Tabassum, A\. Menezes, A\. Kirubarajan, A\. Mullokandov, A\. Sabharwal, A\. Herrick, A\. Efrat, A\. Erdem, A\. Karakaş, B\. R\. Roberts, B\. S\. Loe, B\. Zoph, B\. Bojanowski, B\. Özyurt, B\. Hedayatnia, B\. Neyshabur, B\. Inden, B\. Stein, B\. Ekmekci, B\. Y\. Lin, B\. Howald, B\. Orinion, C\. Diao, C\. Dour, C\. Stinson, C\. Argueta, C\. F\. Ramírez, C\. Singh, C\. Rathkopf, C\. Meng, C\. Baral, C\. Wu, C\. Callison\-Burch, C\. Waites, C\. Voigt, C\. D\. Manning, C\. Potts, C\. Ramirez, C\. E\. Rivera, C\. Siro, C\. Raffel, C\. Ashcraft, C\. Garbacea, D\. Sileo, D\. Garrette, D\. Hendrycks, D\. Kilman, D\. Roth, D\. Freeman, D\. Khashabi, D\. Levy, D\. M\. González, D\. Perszyk, D\. Hernandez, D\. Chen, D\. Ippolito, D\. Gilboa, D\. Dohan, D\. Drakard, D\. Jurgens, D\. Datta, D\. Ganguli, D\. Emelin, D\. Kleyko, D\. Yuret, D\. Chen, D\. Tam, D\. Hupkes, D\. Misra, D\. Buzan, D\. C\. Mollo, D\. Yang, D\. Lee, D\. Schrader, E\. Shutova, E\. D\. Cubuk, E\. Segal, E\. Hagerman, E\. Barnes, E\. Donoway, E\. Pavlick, E\. Rodola, E\. Lam, E\. Chu, E\. Tang, E\. Erdem, E\. Chang, E\. A\. Chi, E\. Dyer, E\. Jerzak, E\. Kim, E\. E\. Manyasi, E\. Zheltonozhskii, F\. Xia, F\. Siar, F\. Martínez\-Plumed, F\. Happé, F\. Chollet, F\. Rong, G\. Mishra, G\. I\. Winata, G\. de Melo, G\. Kruszewski, G\. Parascandolo, G\. Mariani, G\. Wang, G\. Jaimovitch\-López, G\. Betz, G\. Gur\-Ari, H\. Galijasevic, H\. Kim, H\. Rashkin, H\. Hajishirzi, H\. Mehta, H\. Bogar, H\. Shevlin, H\. Schütze, H\. Yakura, H\. Zhang, H\. M\. Wong, I\. Ng, I\. Noble, J\. Jumelet, J\. Geissinger, J\. Kernion, J\. Hilton, J\. Lee, J\. F\. Fisac, J\. B\. Simon, J\. Koppel, J\. Zheng, J\. Zou, J\. Kocoń, J\. Thompson, J\. Wingfield, J\. Kaplan, J\. Radom, J\. Sohl\-Dickstein, J\. Phang, J\. Wei, J\. Yosinski, J\. Novikova, J\. Bosscher, J\. Marsh, J\. Kim, J\. Taal, J\. Engel, J\. Alabi, J\. Xu, J\. Song, J\. Tang, J\. Waweru, J\. Burden, J\. Miller, J\. U\. Balis, J\. Batchelder, J\. Berant, J\. Frohberg, J\. Rozen, J\. Hernandez\-Orallo, J\. Boudeman, J\. Guerr, J\. Jones, J\. B\. Tenenbaum, J\. S\. Rule, J\. Chua, K\. Kanclerz, K\. Livescu, K\. Krauth, K\. Gopalakrishnan, K\. Ignatyeva, K\. Markert, K\. D\. Dhole, K\. Gimpel, K\. Omondi, K\. Mathewson, K\. Chiafullo, K\. Shkaruta, K\. Shridhar, K\. McDonell, K\. Richardson, L\. Reynolds, L\. Gao, L\. Zhang, L\. Dugan, L\. Qin, L\. Contreras\-Ochando, L\. Morency, L\. Moschella, L\. Lam, L\. Noble, L\. Schmidt, L\. He, L\. O\. Colón, L\. Metz, L\. K\. Şenel, M\. Bosma, M\. Sap, M\. ter Hoeve, M\. Farooqi, M\. Faruqui, M\. Mazeika, M\. Baturan, M\. Marelli, M\. Maru, M\. J\. R\. Quintana, M\. Tolkiehn, M\. Giulianelli, M\. Lewis, M\. Potthast, M\. L\. Leavitt, M\. Hagen, M\. Schubert, M\. O\. Baitemirova, M\. Arnaud, M\. McElrath, M\. A\. Yee, M\. Cohen, M\. Gu, M\. Ivanitskiy, M\. Starritt, M\. Strube, M\. Swędrowski, M\. Bevilacqua, M\. Yasunaga, M\. Kale, M\. Cain, M\. Xu, M\. Suzgun, M\. Walker, M\. Tiwari, M\. Bansal, M\. Aminnaseri, M\. Geva, M\. Gheini, M\. V\. T, N\. Peng, N\. A\. Chi, N\. Lee, N\. G\. Krakover, N\. Cameron, N\. Roberts, N\. Doiron, N\. Martinez, N\. Nangia, N\. Deckers, N\. Muennighoff, N\. S\. Keskar, N\. S\. Iyer, N\. Constant, N\. Fiedel, N\. Wen, O\. Zhang, O\. Agha, O\. Elbaghdadi, O\. Levy, O\. Evans, P\. A\. M\. Casares, P\. Doshi, P\. Fung, P\. P\. Liang, P\. Vicol, P\. Alipoormolabashi, P\. Liao, P\. Liang, P\. Chang, P\. Eckersley, P\. M\. Htut, P\. Hwang, P\. Miłkowski, P\. Patil, P\. Pezeshkpour, P\. Oli, Q\. Mei, Q\. Lyu, Q\. Chen, R\. Banjade, R\. E\. Rudolph, R\. Gabriel, R\. Habacker, R\. Risco, R\. Millière, R\. Garg, R\. Barnes, R\. A\. Saurous, R\. Arakawa, R\. Raymaekers, R\. Frank, R\. Sikand, R\. Novak, R\. Sitelew, R\. LeBras, R\. Liu, R\. Jacobs, R\. Zhang, R\. Salakhutdinov, R\. Chi, R\. Lee, R\. Stovall, R\. Teehan, R\. Yang, S\. Singh, S\. M\. Mohammad, S\. Anand, S\. Dillavou, S\. Shleifer, S\. Wiseman, S\. Gruetter, S\. R\. Bowman, S\. S\. Schoenholz, S\. Han, S\. Kwatra, S\. A\. Rous, S\. Ghazarian, S\. Ghosh, S\. Casey, S\. Bischoff, S\. Gehrmann, S\. Schuster, S\. Sadeghi, S\. Hamdan, S\. Zhou, S\. Srivastava, S\. Shi, S\. Singh, S\. Asaadi, S\. S\. Gu, S\. Pachchigar, S\. Toshniwal, S\. Upadhyay, S\. \(\. Debnath, S\. Shakeri, S\. Thormeyer, S\. Melzi, S\. Reddy, S\. P\. Makini, S\. Lee, S\. Torene, S\. Hatwar, S\. Dehaene, S\. Divic, S\. Ermon, S\. Biderman, S\. Lin, S\. Prasad, S\. T\. Piantadosi, S\. M\. Shieber, S\. Misherghi, S\. Kiritchenko, S\. Mishra, T\. Linzen, T\. Schuster, T\. Li, T\. Yu, T\. Ali, T\. Hashimoto, T\. Wu, T\. Desbordes, T\. Rothschild, T\. Phan, T\. Wang, T\. Nkinyili, T\. Schick, T\. Kornev, T\. Tunduny, T\. Gerstenberg, T\. Chang, T\. Neeraj, T\. Khot, T\. Shultz, U\. Shaham, V\. Misra, V\. Demberg, V\. Nyamai, V\. Raunak, V\. Ramasesh, V\. U\. Prabhu, V\. Padmakumar, V\. Srikumar, W\. Fedus, W\. Saunders, W\. Zhang, W\. Vossen, X\. Ren, X\. Tong, X\. Zhao, X\. Wu, X\. Shen, Y\. Yaghoobzadeh, Y\. Lakretz, Y\. Song, Y\. Bahri, Y\. Choi, Y\. Yang, Y\. Hao, Y\. Chen, Y\. Belinkov, Y\. Hou, Y\. Hou, Y\. Bai, Z\. Seid, Z\. Zhao, Z\. Wang, Z\. J\. Wang, Z\. Wang, and Z\. WuBeyond the imitation game: quantifying and extrapolating the capabilities of language models\.External Links:2206\.04615,[Link](https://arxiv.org/abs/2206.04615)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p2.1)\.
- Team\-HyperCLOVA \(2025\)Team\-HyperCLOVAHyperCLOVA X THINK Technical Report\.External Links:2506\.22403,[Link](https://arxiv.org/abs/2506.22403)Cited by:[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px1.p2.1)\.
- Wanget al\.\(2025\)H\. Wang, S\. Zhao, Z\. Qiang, N\. Xi, B\. Qin, and T\. LiuLLMs may perform MCQA by selecting the least incorrect option\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 5852–5862\.External Links:[Link](https://aclanthology.org/2025.coling-main.390/)Cited by:[§2\.2](https://arxiv.org/html/2610.21227#S2.SS2.p1.1)\.
- Wanget al\.\(2024a\)X\. Wang, C\. Hu, B\. Ma, P\. Rottger, and B\. PlankLook at the text: instruction\-tuned language models are more robust multiple choice selectors than you think\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=qHdSA85GyZ)Cited by:[§2\.2](https://arxiv.org/html/2610.21227#S2.SS2.p1.1)\.
- Wanget al\.\(2024b\)Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. ChenMMLU\-pro: a more robust and challenging multi\-task language understanding benchmark\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p2.1)\.
- Wolfet al\.\(2020\)T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. L\. Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. M\. RushTransformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Online,pp\. 38–45\.External Links:[Link](https://www.aclweb.org/anthology/2020.emnlp-demos.6)Cited by:[§4\.1](https://arxiv.org/html/2610.21227#S4.SS1.SSS0.Px1.p1.1)\.
- Wuet al\.\(2022\)Y\. Wu, M\. Gardner, P\. Stenetorp, and P\. DasigiGenerating data to mitigate spurious correlations in natural language inference datasets\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 2660–2676\.External Links:[Link](https://aclanthology.org/2022.acl-long.190/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.190)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1)\.
- Xuet al\.\(2025\)Z\. Xu, Y\. Wang, Y\. Huang, X\. Chen, J\. Zhao, M\. Jiang, and X\. ZhangCross\-lingual pitfalls: automatic probing cross\-lingual weakness of multilingual large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 8254–8284\.External Links:[Link](https://aclanthology.org/2025.acl-long.404/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.404),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p3.1)\.
- Zellerset al\.\(2019\)R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. ChoiHellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,A\. Korhonen, D\. Traum, and L\. Màrquez \(Eds\.\),Florence, Italy,pp\. 4791–4800\.External Links:[Link](https://aclanthology.org/P19-1472/),[Document](https://dx.doi.org/10.18653/v1/P19-1472)Cited by:[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p1.1)\.
- Zhanget al\.\(2026\)X\. Zhang, Y\. Liang, F\. Meng, S\. Zhang, K\. Huang, Y\. Chen, X\. Jinan, and J\. ZhouThink natively: unlocking multilingual reasoning with consistency\-enhanced reinforcement learning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 20766–20783\.External Links:[Link](https://aclanthology.org/2026.acl-long.951/),[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.951),ISBN 979\-8\-89176\-390\-6Cited by:[§4\.2](https://arxiv.org/html/2610.21227#S4.SS2.SSS0.Px4.p2.1)\.
- Zhenget al\.\(2024\)C\. Zheng, H\. Zhou, F\. Meng, J\. Zhou, and M\. HuangLarge language models are not robust multiple choice selectors\.External Links:2309\.03882,[Link](https://arxiv.org/abs/2309.03882)Cited by:[§2\.2](https://arxiv.org/html/2610.21227#S2.SS2.p1.1)\.
- Zhouet al\.\(2025\)C\. Zhou, Q\. Li, C\. Li, J\. Yu, Y\. Liu, G\. Wang, K\. Zhang, C\. Ji, Q\. Yan, L\. He, H\. Peng, J\. Li, J\. Wu, Z\. Liu, P\. Xie, C\. Xiong, J\. Pei, P\. S\. Yu, and L\. SunA comprehensive survey on pretrained foundation models: a history from bert to chatgpt\.International Journal of Machine Learning and Cybernetics16\(12\),pp\. 9851–9915\.Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1)\.
- Zhouet al\.\(2023\)J\. Zhou, T\. Lu, S\. Mishra, S\. Brahma, S\. Basu, Y\. Luan, D\. Zhou, and L\. HouInstruction\-following evaluation for large language models\.External Links:2311\.07911,[Link](https://arxiv.org/abs/2311.07911)Cited by:[§1](https://arxiv.org/html/2610.21227#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.21227#S2.SS1.p1.1),[§3\.1](https://arxiv.org/html/2610.21227#S3.SS1.p1.1)\.

## Appendix ADetailed Correction and Localization Rules

This section describes the dataset\-specific rules used during the author\-led correction and localization stage introduced in Section[3\.2](https://arxiv.org/html/2610.21227#S3.SS2)\. Throughout these stages, we aim to preserve the original reasoning requirements and difficulty, while verifying that the answer remains valid after localization\.

### A\.1Source\-Side Corrections in English Benchmarks

Some inconsistencies are present in the English source benchmarks rather than being introduced by translation\. We correct these cases during the correction and localization stage so that the Ko\-Xdataset reflects the intended task design of the correspondingXdataset rather than source\-side artifacts\. Representative cases include:

- •ARC\-Challenge near\-duplicates\.The itemsLEAP\_8\_10365andLEAP\_2000\_8\_2differ only in the surface form "one centimeter" vs\. "1 centimeter"\. We retain one item and remove the other\. Other near\-duplicate pairs are handled using the same criterion\.
- •ARC\-Easy unintended numeral\.The question stem "Which of the 27 following is the best use of a robot?" inMCAS\_2005\_8\_27contains an unintended "27", which we remove\.
- •GSM8K calculator annotations\.GSM8K marks calculator\-evaluable expressions with<<\.\.\.\>\>\. Redundant annotations such as<<8=8\>\>are removed, and annotations with incorrect arithmetic are corrected\. The original annotation convention is otherwise preserved for compatibility with existing GSM8K evaluation code\.

### A\.2Ko\-GSM8K: Names, Units, and Currency

#### Names\.

English given names are replaced with Korean given names sampled from the statistics of high\-frequency birth names recorded between 2008 and 2024 in the Korean Supreme Court e\-family Register666[https://stfamily\.scourt\.go\.kr/st/StFrrStatcsView\.do?pgmId=090000000025](https://stfamily.scourt.go.kr/st/StFrrStatcsView.do?pgmId=090000000025)\. Name substitutions are kept consistent within each problem so that the same person is referred to by the same Korean name throughout the problem\.

#### Units\.

Imperial units are converted to metric units\. When a numeric value is part of the arithmetic structure of the problem, we use an arithmetically simple metric counterpart rather than an exact conversion \(e\.g\., 10 feet→\\rightarrow3 meters rather than 3\.048 meters\)\. After each conversion, we recompute the answer to verify that the localized item preserves the intended difficulty\.

#### Currency\.

U\.S\. dollars are converted to Korean won using the fixed rate 1 USD = 1,000 KRW\. This rate is chosen to preserve the arithmetic structure of the original problems while keeping the quantities natural in Korean\. For example, if a problem asks for the cost of three items priced at $15 each, the original calculation is simply15×3=4515\\times 3=45\. Using a realistic exchange rate such as 1 USD = 1,512 KRW would turn this into15×1512×315\\times 1512\\times 3, adding unintended arithmetic complexity\. With the fixed rate, the problem becomes15000×3=4500015000\\times 3=45000, preserving the intended multiplication difficulty\. Although this abstracts away from exact exchange\-rate variation, the rounded conversion improves arithmetic clarity\.

#### Korean counting expressions\.

Korean counting expressions are selected according to the noun being counted\. For example, "5 pencils" is translated as "연필 5자루", where "자루" is the counting word used with pencil\-like objects\. We apply the same principle to other noun classes, including people, books, vehicles, and animals, etc\.

#### Everyday items, brands, and places\.

Everyday items, foods, brands, and place names that are less natural in Korean contexts are replaced with familiar Korean counterparts, provided that the substitution does not change the mathematical structure of the original problem\.

### A\.3Ko\-ARC: Cultural and Geographic Localization

ARC items often contain U\.S\.\-specific contextual details, such as states, national parks, coastlines, animal species, and industrial settings\. When these details are not essential to the scientific concept being tested, we replace them with Korean or East Asian counterparts that are more natural in Korean educational contexts\. Each substitution is checked against relevant Korean educational and public reference materials\. Table[7](https://arxiv.org/html/2610.21227#A1.T7)summarizes representative substitution rules\.

Table 7:Representative localization rules for Ko\-ARC\. Each substitution is checked to preserve both the underlying scientific mechanism and the correct answer\.
### A\.4Ko\-EQ\-Bench: Emotion Labels and Honorific Usage

#### Emotion\-label mapping\.

Some English emotion labels encode distinctions that do not map cleanly onto a single Korean label\. To ensure consistent annotation across items, we use a documented mapping table for emotion labels\. When a direct label mapping does not preserve the intended meaning in context, we apply an item\-specific revision\. For example, "Victimized" is mapped to "피해의식", which captures a perceived sense of "being wronged" rather than "the literal state of having been harmed"\. Idiomatic expressions are also translated by preserving their contextual meaning rather than their literal form\. For example, "haunted" is translated as "계속 떠오른다" when it refers to "a thought or memory that repeatedly comes to mind", rather than as the literal ghost\-related meaning "귀신이 나오는"\.

#### Honorific usage\.

Direct translation can produce repeated address forms in which a Korean name is followed by the polite suffix "씨" throughout a dialogue\. Since such repetition can sound unnatural or overly explicit in Korean, we reduce it where appropriate, while preserving the relationship between speakers and the emotional tone of the original item\.

### A\.5Ko\-WinoGrande: Names and Everyday Objects

English personal names are replaced with Korean given names \(e\.g\., Jessica→\\rightarrow지희\)\. The substitutions are kept consistent within each item and checked to preserve the original coreference structure\. Everyday objects that are uncommon or unnatural in Korean contexts are replaced with familiar Korean counterparts when the replacement does not change the commonsense relation required by the item \(e\.g\., toaster oven→\\rightarrow전자레인지\(microwave\) as Koreans typically don’t use toaster ovens\)\. Sentence structure is also adjusted to avoid excessive repetition of names, which is more marked in Korean than in English\.

### A\.6Ko\-IFEval: Hybrid Adaptation and Korean\-Specific Verifiers

#### Construction subsets\.

Ko\-IFEval is constructed through a hybrid strategy\. Items with indices 1,000\-3,757 form the translated subset, where original IFEval prompts are adapted through the four\-stage pipeline described in Section[3\.2](https://arxiv.org/html/2610.21227#S3.SS2)\. For these items, we translate and localize the prompt while preserving the original instruction\-following constraint whenever it can be evaluated reliably in Korean\. Localization includes replacing culturally unfamiliar genres with Korean counterparts\. For example, an instruction asking for an English "limerick" is rewritten as a request for "5행시", a Korean five\-line poem written from a given word or phrase\. We also convert verifier arguments, such as comparison relations and target strings, so that the automatic checker can evaluate Korean outputs directly\.

Items with indices 10,001\-10,300 form the Korean\-specific subset\. For these items, authors write Korean prompts from scratch using public\-domain Korean source material, including presidential speeches\. We define additional instruction types that reflect Korean writing conventions and implement verifier code for checking these constraints automatically\.

#### Instruction\-type expansion\.

Table[8](https://arxiv.org/html/2610.21227#A1.T8)summarizes how the instruction\-type inventory is expanded from 25 types in the original IFEval to 30 types in Ko\-IFEval\. The expansion is concentrated in instruction groups where direct transfer from English is insufficient, especially keyword constraints, length constraints, and detectable formatting\. Other instruction groups are retained without structural change\.

Table 8:Instruction\-type inventory in the original IFEval and Ko\-IFEval\. The "Changes" column summarizes Ko\-IFEval\-specific additions and adaptations\.
#### Verifier arguments\.

For Korean\-specific instruction types, each item includes Korean\-aware arguments in thekwargsfield\. For example, items with thedetectable\_format:number\_highlighted\_sectionsinstruction specifynum\_highlightstogether with a Korean comparison relation such as "이상\(greater than or equal to\)" or "미만\(less\)"\. Length\-constraint items also specify whethernum\_lettersshould be counted with or without whitespace\. The verifier code reads these fields directly, allowing the original IFEval scoring framework to be extended to Ko\-IFEval with a small compatibility layer\.

## Appendix BReview Process and Quality Control

### B\.1Participants and Task Assignments

Correction and review were conducted internally by members of the author team, rather than by external reviewers\. Fifteen authors participated in dataset construction and review\. Authors performing correction and cross\-review were native Korean speakers with at least CEFR777Common European Framework of Reference for Languages\.B2\-equivalent English proficiency\. The team included members with Korean\-English bilingual backgrounds, experience in English\-speaking educational environments, and translation\-related professional experience\. Relevant backgrounds included Korean language education, mathematics education, computer science, statistics, physics, geography, psychology, economics, accounting, and history\. Responsibilities were assigned with consideration of each subset’s requirements and the participants’ language and domain backgrounds\.

### B\.2Review Procedure

The correction and review procedures for Ko\-ARC, Ko\-GSM8K, Ko\-EQ\-Bench, and Ko\-WinoGrande are described in Section[3\.2](https://arxiv.org/html/2610.21227#S3.SS2)\. Ko\-LAMBADA followed a separate construction process based on Korean literary texts\. Passages and answer candidates were manually reviewed for contextual support and grammatical compatibility\. Ko\-IFEval also followed a separate adaptation and verification process\. Authors adapted 541 source prompts and wrote 300 additional Korean prompts, manually checking their consistency with instruction identifiers and verifier arguments and adapting the associated checks for Korean\.

### B\.3Recorded Revision Statistics

We quantify documented editing as the fraction of aligned items with a change to at least one task field between archived versions\. Changes include localization, word edits, and answer corrections\. Items are aligned by their original identifiers or source prompts\. We normalize line endings, Unicode representation, and leading or trailing whitespace, and compare serialized answer choices by their contents\. Table[9](https://arxiv.org/html/2610.21227#A2.T9)reports confirmed changes in records associated with initial editing, localization, and cross\-review\.

The high initial correction rates show that author revisions were needed across a large proportion of the machine\-translated items\. These revisions included restoring information omitted during translation, correcting inconsistent formatting, and revising wording that obscured the intended meaning\. In particular, missing task\-relevant information had to be recovered from the English source to preserve the conditions needed to answer the question\. These examples illustrate the substantive role of author\-led correction in preparing the translated drafts for Korean evaluation\.

Table 9:Recorded item revisions at each construction stage, reported as counts and percentages of aligned items\. Ko\-ARC\-Challenge statistics cover the 1,172\-item review pool before five items were excluded from the final dataset\. The separate Ko\-IFEval process is not included\.ProviderModelSizeHugging Face checkpoint*Korean\-developed LLMs*KTMi:dm 2\.0 Mini2\.3B[K\-intelligence/Midm\-2\.0\-Mini\-Instruct](https://huggingface.co/K-intelligence/Midm-2.0-Mini-Instruct)Mi:dm 2\.0 Base11\.5B[K\-intelligence/Midm\-2\.0\-Base\-Instruct](https://huggingface.co/K-intelligence/Midm-2.0-Base-Instruct)LGEXAONE\-4\.0\-1\.2B1\.2B[LGAI\-EXAONE/EXAONE\-4\.0\-1\.2B](https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-1.2B)EXAONE\-4\.0\-32B32B[LGAI\-EXAONE/EXAONE\-4\.0\-32B](https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-32B)KakaoKanana\-2\-30b\-a3b\-base\-260130B\-A3B[kakaocorp/kanana\-2\-30b\-a3b\-base\-2601](https://huggingface.co/kakaocorp/kanana-2-30b-a3b-base-2601)Kanana\-2\-30b\-a3b\-mid\-260130B\-A3B[kakaocorp/kanana\-2\-30b\-a3b\-mid\-2601](https://huggingface.co/kakaocorp/kanana-2-30b-a3b-mid-2601)Kanana\-2\-30b\-a3b\-instruct\-260130B\-A3B[kakaocorp/kanana\-2\-30b\-a3b\-instruct\-2601](https://huggingface.co/kakaocorp/kanana-2-30b-a3b-instruct-2601)NaverHyperCLOVAX\-SEED\-Text\-Instruct\-0\.5B0\.5B[HyperCLOVAX\-SEED\-Text\-Instruct\-0\.5B](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Text-Instruct-0.5B)HyperCLOVAX\-SEED\-Text\-Instruct\-1\.5B1\.5B[HyperCLOVAX\-SEED\-Text\-Instruct\-1\.5B](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Text-Instruct-1.5B)SKTA\.X 4\.0 Light7B[skt/A\.X\-4\.0\-Light](https://huggingface.co/skt/A.X-4.0-Light)*Multilingual LLMs*QwenQwen3\.5\-0\.8B\-Base0\.8B[Qwen/Qwen3\.5\-0\.8B\-Base](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base)Qwen3\.5\-0\.8B0\.8B[Qwen/Qwen3\.5\-0\.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B)Qwen3\.5\-2B\-Base2B[Qwen/Qwen3\.5\-2B\-Base](https://huggingface.co/Qwen/Qwen3.5-2B-Base)Qwen3\.5\-2B2B[Qwen/Qwen3\.5\-2B](https://huggingface.co/Qwen/Qwen3.5-2B)Qwen3\.5\-4B\-Base4B[Qwen/Qwen3\.5\-4B\-Base](https://huggingface.co/Qwen/Qwen3.5-4B-Base)Qwen3\.5\-4B4B[Qwen/Qwen3\.5\-4B](https://huggingface.co/Qwen/Qwen3.5-4B)Qwen3\.5\-9B\-Base9B[Qwen/Qwen3\.5\-9B\-Base](https://huggingface.co/Qwen/Qwen3.5-9B-Base)Qwen3\.5\-9B9B[Qwen/Qwen3\.5\-9B](https://huggingface.co/Qwen/Qwen3.5-9B)Qwen3\.5\-27B27B[Qwen/Qwen3\.5\-27B](https://huggingface.co/Qwen/Qwen3.5-27B)Qwen3\.5\-35B\-A3B\-Base35B\-A3B[Qwen/Qwen3\.5\-35B\-A3B\-Base](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-Base)Qwen3\.5\-35B\-A3B35B\-A3B[Qwen/Qwen3\.5\-35B\-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B)GoogleGemma\-4\-E2B5B[google/gemma\-4\-E2B](https://huggingface.co/google/gemma-4-E2B)Gemma\-4\-E2B\-it5B[google/gemma\-4\-E2B\-it](https://huggingface.co/google/gemma-4-E2B-it)Gemma\-4\-E4B8B[google/gemma\-4\-E4B](https://huggingface.co/google/gemma-4-E4B)Gemma\-4\-E4B\-it8B[google/gemma\-4\-E4B\-it](https://huggingface.co/google/gemma-4-E4B-it)Gemma\-4\-26B\-A4B26B\-A4B[google/gemma\-4\-26B\-A4B](https://huggingface.co/google/gemma-4-26B-A4B)Gemma\-4\-26B\-A4B\-it26B\-A4B[google/gemma\-4\-26B\-A4B\-it](https://huggingface.co/google/gemma-4-26B-A4B-it)Gemma\-4\-31B31B[google/gemma\-4\-31B](https://huggingface.co/google/gemma-4-31B)Gemma\-4\-31B\-it31B[google/gemma\-4\-31B\-it](https://huggingface.co/google/gemma-4-31B-it)Table 10:Open\-weight models evaluated\. Sizes are reported in billions of parameters\. For mixture\-of\-experts models, we also include the active\-parameter shorthand reported by the developer, such as30B\-A3B\. All checkpoints are accessed through the Hugging Face Hub\.

## Appendix COpen\-Weight Models Evaluated

Table[10](https://arxiv.org/html/2610.21227#A2.T10)lists all open\-weight models included in our evaluation, grouped by provider\. We report the model size and the exact Hugging Face checkpoint used for evaluation\. The Korean\-developed group spans five model families released by Korean companies, while the multilingual group includes two widely used model families with multilingual capabilities\.

## Appendix DDetailed Evaluation Protocol

#### Common settings\.

All evaluations are conducted withlm\-evaluation\-harness\. We use the test split for every task\. Each reported result is based on a single run\. For generation\-based tasks, we use greedy decoding withdo\_sample=Falseandtemperature=0\.0\.

#### Models\.

Closed API models are evaluated through their official APIs\. We use Claude Haiku 4\.5 \(claude\-haiku\-4\-5\-20251001\) and Claude Opus 4\.7 \(claude\-opus\-4\-7\) through the Anthropic API, and GPT\-5\-mini \(gpt\-5\-mini\-2025\-08\-07\) and GPT\-5\.5 \(gpt\-5\.5\-2026\-04\-23\) through the OpenAI API\. Open\-weight models are evaluated using the Hugging Face Transformers backend oflm\-evaluation\-harness; each model is identified by its exact Hugging Face checkpoint\. Open\-weight model inference uses bfloat16 precision on a single NVIDIA B200 GPU with 180 GB memory\.

Table 11:Detailed task configuration\. Original multiple\-choice tasks are evaluated only for open\-weight models with log\-likelihood access\. Generative multiple\-choice variants are used for both open\-weight and closed API models\. For Ko\-EQ\-Bench, the maximum generation length is 80 tokens for open\-weight models and 1024 tokens for closed API models\.
#### Stop sequences\.

For the generative variants of Ko\-ARC, Ko\-WinoGrande, and Ko\-LAMBADA, generation stops at one of\\n,</s\>, or<\|im\_end\|\>\. For Ko\-GSM8K, generation stops at문제:,</s\>, or<\|im\_end\|\>\. Ko\-IFEval uses no explicit stop sequence\. Ko\-EQ\-Bench stops at a blank line \(\\n\\n\), with maximum generation lengths of 80 tokens for open\-weight models and 1024 tokens for closed API models\.

#### Prompt formatting conditions\.

For open\-weight models, we distinguish between plain prompting and chat\-template prompting\. Plain prompting is used for all open\-weight checkpoints\. For instruction\-tuned or chat\-oriented checkpoints with a defined chat template, we additionally evaluate chat\-template prompting using the\-\-apply\_chat\_templateoption inlm\-evaluation\-harness\. This option formats each benchmark prompt according to the checkpoint\-specific chat template by adding model\-specific role markers, special tokens, or generation prefixes\. When few\-shot examples are used, the demonstrations and the test instance are represented as alternating user and model turns\.

For example, for a Gemma instruction\-tuned checkpoint, a task prompt containing Korean benchmark\-specific fields such as질문:and답변:is wrapped as:

> <bos\><\|turn\>user \[benchmark prompt\] <turn\|\> <\|turn\>model

Here, the Korean field labels are part of the benchmark prompt, whereas the turn markers and special tokens are added by the chat template\.

Closed API models are evaluated only through their official chat\-style APIs\. Since these APIs do not expose tokenizer\-level chat\-template control or a comparable plain prompting interface, we cannot evaluate closed models under the plain prompting condition\. Closed\-model results should therefore be interpreted as chat\-style prompting results, and compared most directly with the open\-weight chat\-template condition rather than the open\-weight plain condition\.

We report additional results for six instruction\-tuned Qwen3\.5 models withenable\_thinking=Falseon Ko\-GSM8K, Ko\-EQ\-Bench, and the four generative MCQA tasks, while retaining chat templates\. These results are reported separately from the default\-chat results\.

#### Prompting and answer extraction\.

For original multiple\-choice tasks, open\-weight models are evaluated using log\-likelihood\-based scoring\. For Ko\-ARC, the model receives the question prompt, and each answer choice is scored as a candidate continuation\. For Ko\-WinoGrande and Ko\-LAMBADA, each candidate is inserted into the blank and scored by log\-likelihood comparison\.

We evaluate generative multiple\-choice variants for both open\-weight and closed API models\. Ko\-ARC\-Easy\-gen and Ko\-ARC\-Challenge\-gen \(generation versions\) present four labeled options and require the model to output a single label:

> ``` 질문: {question} A. {choices.text[0]} B. {choices.text[1]} C. {choices.text[2]} D. {choices.text[3]} 반드시 A, B, C, D 중 하나의 문자로만 답하세요. 정답: ```

The final answer is extracted with the following regular expression:

> `\(?i\)\(?:정답\\s\*\[:：\]?\\s\*\)?\(\[A\-D\]\)\(?:\\s\*\(?:번\|입니다\|이에요\|가\)\|\[\\\.\)\\\]:：,\\s\]\|$\)`

Ko\-WinoGrande\-gen presents two labeled options and requires the model to output either A or B:

> ``` 문장의 빈칸에 들어갈 말로 더 자연스러운 선택지를 고르세요. 문장: {sentence} A. {option1} B. {option2} 반드시 A 또는 B 중 하나의 문자로만 답하세요. 정답: ```

Ko\-LAMBADA\-gen uses the same two\-choice format, with the masked passage provided as the sentence field:

> ``` 문장의 빈칸에 들어갈 말로 더 자연스러운 선택지를 고르세요. 문장: {text} A. {option_A} B. {option_B} 반드시 A 또는 B 중 하나의 문자로만 답하세요. 정답: ```

For Ko\-WinoGrande\-gen and Ko\-LAMBADA\-gen, the final answer is extracted with the same regular expression, except that the valid label range is A\-B:

> `\(?i\)\(?:정답\\s\*\[:：\]?\\s\*\)?\(\[A\-B\]\)\(?:\\s\*\(?:번\|입니다\|이에요\|가\)\|\[\\\.\)\\\]:：,\\s\]\|$\)`

If no valid label is matched, the response is counted as incorrect\.

For Ko\-GSM8K, the prompt format is문제: \{question\}\\n답:\. We use two independent answer extractors: a strict extractor that requires an explicit final\-answer marker and a flexible extractor that selects the final numeric match regardless of whether that marker is present\. During answer normalization, commas, the Korean currency marker "원", text before\#\#\#\#, and a trailing period are ignored\.

Ko\-IFEval uses thepromptfield directly and is evaluated with deterministic instruction\-following verifiers\. Ko\-EQ\-Bench also uses thepromptfield directly\. The model is expected to output four emotion\-score pairs, one per line, in the formemotion: score\. The reported EQ\-Bench scores are reproduced by a line\-based parser that allows spaces within Korean emotion labels and whitespace before the colon\. An output is parseable when the extracted label set consists of exactly the four reference emotions; otherwise, botheqbenchandpercent\_parseableare set to zero\.

#### Metric definitions\.

For multiple\-choice tasks,accdenotes the fraction of examples for which the correct option has the highest log\-likelihood\.acc\_normapplies length normalization to the per\-choice log\-likelihoods before comparison\.acc\_npsquses Normalized Probability Shift by the Question \(NPSQ;[Cho et al\., 2026](https://arxiv.org/html/2610.21227#bib.bib23)\)\. For a question\-related inputQQ, choice\-related inputCC, and answer candidatexx, NPSQ is defined as

NPSQ⁡\(Q,C,x\)=log⁡P⁡\(x∣Q,C\)−log⁡P⁡\(x∣C\)−log⁡P⁡\(x∣C\)\.\\mathrm\{NPSQ\}\(Q,C,x\)=\\frac\{\\log P\(x\\mid Q,C\)\-\\log P\(x\\mid C\)\}\{\-\\log P\(x\\mid C\)\}\.
The prediction is the option with the highest NPSQ score\. This scoring rule measures the relative increase in the model’s preference for each choice after the question is included, compared with a question\-free baseline that scores the choices without the question\. It therefore separates question\-driven evidence from choice\-only preferences such as choice\-length bias or surface\-form familiarity\.

For Ko\-GSM8K, we report two exact\-match variants:strict\-matchandflexible\-extract\. Thestrict\-matchmetric counts an example as correct only when the response contains an explicit final\-answer marker of the form\#\#\#\# numberand the extracted number matches the gold answer after normalization\. Theflexible\-extractmetric independently extracts the final numeric match in the response, regardless of whether an explicit final\-answer marker is present, and compares it with the gold answer after normalization\.

For Ko\-IFEval, we report four instruction\-following accuracies\. Prompt\-level strict accuracy counts a prompt as correct only when all verifiable instructions in that prompt are satisfied under the strict verifier\. Instruction\-level strict accuracy is the fraction of individual verifiable instructions that are satisfied\. Prompt\-level loose accuracy and instruction\-level loose accuracy use the same two aggregation levels, but apply the loose verifier, which allows common surface variations in the response before checking whether each instruction is followed\.

For Ko\-EQ\-Bench, we report two metrics:eqbenchandpercent\_parseable\. The model is instructed to output four emotion\-score pairs in the formemotion: score, with integer scores on a 0\-10 scale\.percent\_parseableis the percentage of examples for which the extracted label set consists of exactly the four reference emotions\. For parseable outputs,eqbenchmeasures how closely the predicted emotion intensity scores match the reference scores\. For each matched emotion, we compute the absolute difference between the predicted and reference scores and apply the same scaled penalty function used by the EQ\-Bench scorer\. The example\-level score is 100 for exact agreement and can be negative for sufficiently large errors\. Unparseable outputs receiveeqbench=0 andpercent\_parseable=0 before aggregation, so the reportedeqbenchscore reflects both emotion\-score accuracy and output format compliance\.

## Appendix EQwen3\.5 Behavior under Chat Prompting

We inspect representative Ko\-GSM8K outputs from Qwen3\.5\-9B and Qwen3\.5\-0\.8B under chat prompting\. The example uses a simple arithmetic problem about selling duck eggs\. Both models have enough information to solve the problem, and the correct answer is 18,000\. However, their generation behavior differs\. Qwen3\.5\-0\.8B directly solves the problem in Korean and reaches the required final answer format\. In contrast, Qwen3\.5\-9B begins with a long English reasoning trace and does not produce the final answer in the recorded output\. This example illustrates a failure to produce the final answer under default chat prompting; the thinking\-off results in Table[5](https://arxiv.org/html/2610.21227#S4.T5)show that performance depends on the evaluated generation settings\.

#### Qwen3\.5\-0\.8B\.

Qwen3\.5\-0\.8B produces a short Korean solution and reaches the final answer\.

> 보리의 하루는 16개의 알을 낳습니다\. 아침 식사로 3개, 머핀으로 4개, 나머지는 직거래 장터에서 16 \- 3 \- 4 = 9개의 알을 판매합니다\. 매일 직거래 장터에서 벌이는 돈은 9개 x 2000원 = <<9\*2000=18000\>\>18,000원입니다\. \#\#\#\# 18000

#### Qwen3\.5\-9B\.

Qwen3\.5\-9B begins solving the same problem in English and produces a long reasoning trace\. In the recorded output, the model does not reach the final answer\.

> Thinking Process: 1\. Analyze the Request: The user is presenting a math word problem in Korean and expects a solution in a specific format … 2\. Analyze the Problem: \* Duck lays 16 eggs per day\. \* Duck eats 3 eggs for breakfast\. \* Duck makes muffins with 4 eggs for friends\. \* Remaining eggs are sold at the direct market\. \* Selling price: 2000 won per egg\. 3\. Step\-by\-Step Calculation: \* Total eggs laid: 16 \* Eggs eaten: 3 \* Eggs for muffins: 4 \* Eggs remaining for sale:16−3−416\-3\-4 \* Calculation:16−3=1316\-3=13,13−4=13\-4=

## Appendix FFull Experimental Results

This section reports per\-model KBS results and model\-group summaries for language comparisons\. Tables[12](https://arxiv.org/html/2610.21227#A6.T12)and[13](https://arxiv.org/html/2610.21227#A6.T13)report open\-weight log\-likelihood MCQA results under plain and default chat prompting, respectively\. Tables[14](https://arxiv.org/html/2610.21227#A6.T14)and[15](https://arxiv.org/html/2610.21227#A6.T15)report open\-weight generation results under the two prompt formats\. Table[16](https://arxiv.org/html/2610.21227#A6.T16)reports closed API results for generative MCQA and generation tasks\. Table[17](https://arxiv.org/html/2610.21227#A6.T17)reports open\-weight generative MCQA results under default chat prompting\. Table[18](https://arxiv.org/html/2610.21227#A6.T18)reports Qwen3\.5 thinking\-off results for both generative MCQA and generation tasks\. Table[19](https://arxiv.org/html/2610.21227#A6.T19)summarizes language comparisons for ARC, GSM8K, WinoGrande, and EQ\-Bench\.

The thinking\-off comparison in Table[18](https://arxiv.org/html/2610.21227#A6.T18)focuses on Ko\-GSM8K, Ko\-EQ\-Bench, and generative MCQA, where the four larger Qwen3\.5 models obtain near\-zero scores under default chat prompting \(Tables[15](https://arxiv.org/html/2610.21227#A6.T15)and[17](https://arxiv.org/html/2610.21227#A6.T17)\)\. Ko\-IFEval does not exhibit the same near\-zero pattern and is not included in this comparison\.

Table 12:Open\-weight multiple\-choice results under plain prompting\. The chat\-template subset average uses the same model subset as the chat\-template prompting results in Table[13](https://arxiv.org/html/2610.21227#A6.T13)\. The highest score in each metric column is highlighted in red\.Table 13:Open\-weight multiple\-choice results under chat\-template prompting\. The highest score in each metric column is highlighted in red\.Table 14:Open\-weight generation results under plain prompting\. The chat\-template subset average uses the same model subset as the chat\-template prompting results in Table[15](https://arxiv.org/html/2610.21227#A6.T15)\. The highest score in each metric column is highlighted in red\.Table 15:Open\-weight generation results under chat\-template prompting\. The highest score in each metric column is highlighted in red\.Table 16:Closed API model results across all benchmark tasks under chat\-style prompting\.Table 17:Generative MCQA results for open\-weight models under chat\-style prompting\. The highest score in each metric column is highlighted in red\.Table 18:Generative MCQA and generation results for six instruction\-tuned Qwen3\.5 models withenable\_thinking=False, while retaining chat templates\.Table 19:Performance across English, raw DeepL, and KBS variants under chat\-style prompting, averaged over the same eight Korean\-developed, six Qwen3\.5, and four Gemma 4 models\. ARC and WinoGrande use NPSQ accuracy, GSM8K uses strict exact match, and EQ\-Bench uses its benchmark score\. DeepL results are unavailable for WinoGrande and excluded for EQ\-Bench because the Korean emotion labels in the translated prompts do not match the English reference labels used by the scorer\.