Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs

# Estimating Uncertainty from Reasoning: A Large-Scale Study of Multi- and Crosslingual MCQA Performance in LLMs
Source: [https://arxiv.org/html/2607.06327](https://arxiv.org/html/2607.06327)
Andrea Alfarano1,†Andrea Bacciu2,†Saab Mansour2Amin Mantrach2Marcello Federico2 1INSAIT, Sofia2Amazon andrea\.alfarano@insait\.ai \{andbac, saabm, mantrach, marcfede\}@amazon\.com

###### Abstract

Uncertainty estimation \(UE\) enables LLM\-powered systems to recognize when to abstain, yet existing research has predominantly focused on English\. We present the first large\-scale evaluation of UE methods across 22 languages, spanning high\-, mid\-, and low\-resource settings\. Using two human\-curated Q&A datasets, we compare open and closed box UE methods \(nine in total\) across different model sizes and architectures while eliciting long\-form reasoning, avoiding LLM\-as\-a\-judge and embedding\-based scoring, which can introduce evaluation noise\. We report three main actionable findings\. First, we find that prompting models to reason in English while keeping questions in low\-resource languages substantially improves UE performance, suggesting that comprehension of low\-resource languages is largely intact, and that the reliability bottleneck lies in generation rather than understanding\. Second, prompting models to reason in English closes the UE performance gap between low and high\-resource languages, demonstrating that generation language matters more than the question language\. Third, the choice of UE method should depend on model scale: at smaller scales, open\-box probability\-based methods outperform alternatives; at larger scales, closed\-box self\-verbalized uncertainty becomes superior\. Finally, we provide an analysis of threshold selection for selective prediction, offering guidance on calibrating abstention in multilingual settings\.

Estimating Uncertainty from Reasoning: A Large\-Scale Study of Multi\- and Crosslingual MCQA Performance in LLMs

## 1Introduction

Large Language Models \(LLMs\) have changed how people access and interact with information, supporting tasks from everyday planning to complex question\-answering\(Bommasani,[2021](https://arxiv.org/html/2607.06327#bib.bib1)\)\. Recent research has primarily focused on improving task performance, for example through chain\-of\-thought prompting\(Weiet al\.,[2022](https://arxiv.org/html/2607.06327#bib.bib11)\)and instruction tuning\(Ouyanget al\.,[2022](https://arxiv.org/html/2607.06327#bib.bib10)\), enabling models to solve increasingly complex problems\. However, high accuracy alone is insufficient: downstream systems must recognize when an LLM’s answer is not grounded in its knowledge, enabling them to abstain, defer to humans, or fall back to safer behavior\. This has motivated a parallel line of work on uncertainty estimation \(UE\), which seeks to determine when models know the answer, enabling LLM\-powered systems to recognize and communicate their lack of knowledge\(Kuhnet al\.,[2023](https://arxiv.org/html/2607.06327#bib.bib14)\)\. Existing research on LLM uncertainty has predominantly focused on English, leaving limited evidence on whether UE methods maintain their efficacy in other languages, especially in low\-resource settings \(Kuhnet al\.\([2023](https://arxiv.org/html/2607.06327#bib.bib14)\); Kossenet al\.\([2024](https://arxiv.org/html/2607.06327#bib.bib12)\); Santilliet al\.\([2025](https://arxiv.org/html/2607.06327#bib.bib51)\); Cecereet al\.\([2025](https://arxiv.org/html/2607.06327#bib.bib41)\), inter alia\)\. The only dedicated multilingual UE study we are aware of\(Xueet al\.,[2025](https://arxiv.org/html/2607.06327#bib.bib47)\)evaluates just three methods on five languages, relies on machine\-translated data without human post\-editing, and uses a multiple\-choice, short\-answer format rather than open\-ended generation\. Their dataset yields a median answer length of just one word, meaning this setup primarily evaluates how language affects question comprehension while offering limited evidence about uncertainty during longer, more linguistically rich generation\. Establishing multilingual trustworthiness through UE is also methodologically challenging because standard metrics such as AUROC rely on ground truth labels\. When ground truth is approximated via LLM\-as\-judge, BERTScore, or n\-gram overlap, these proxies introduce noise that can misrank uncertainty methods\(Santilliet al\.,[2025](https://arxiv.org/html/2607.06327#bib.bib51)\)\. In multilingual settings, this problem is compounded: neural judges may behave inconsistently across languages, introducing language\-dependent bias into UE comparisons\. As a result, existing English\-centric evaluations do not yet provide decisive evidence that UE methods transfer reliably to low\-resource languages and long\-form generation\. To address these challenges, we evaluate nine UE methods on two human\-curated QA datasets across 22 languages, covering high\-, mid\-, and low\-resource settings and representing80%80\\%of the world’s native speakers\. We elicit long\-form, language\-rich reasoning \(on average 150 words\) while preserving objective correctness from QA labels, thereby avoiding LLM\-as\-judge and embedding\-based proxies\.

Our results reveal three key patterns: \(1\) UE method performance varies largely across language families, consistent with effects of resource availability and linguistic distance; \(2\) generation language has a larger impact on UE than question language, a factor largely overlooked by prior work; and \(3\) while the closed\-box Self Verbalized method achieves the best overall performance, sampling\-based closed\-box methods degrade substantially on low\-resource languages, whereas open\-box probability\-based methods maintain more consistent performance across all language resource levels\.

We make the following contributions: \(i\) we propose a label\-grounded framework for evaluating UE on open\-ended multilingual generation without model\-based correctness metrics; \(ii\) a large\-scale study of UE across 9 models and 22 languages; and \(iii\) practical guidance for deploying uncertainty\-aware systems in multilingual settings\. Specifically, we address: \(RQ1\) which UE methods are robust across models and languages, \(RQ2\) how model scale affects UE efficacy, \(RQ3\) how the language of reasoning shapes UE quality, \(RQ4\) UE robustness under cross\-lingual answer settings, and \(RQ5\) threshold selection strategies for selective prediction\.

## 2Related Work and Background

Uncertainty estimation \(UE\) is critical for deploying large language models \(LLMs\) in high\-stakes environments, enabling systems to flag unreliable predictions\. Existing literature is commonly categorized by the degree of model access available:*open\-box methods*, which utilize internal model representations, and*closed\-box methods*, which rely solely on textual outputs\.

### 2\.1Open\-Box Methods

Open\-box approaches derive uncertainty directly from model internals, such as hidden states or token probabilities\.

Inference method approaches require no additional training and aggregate token\-level logits into sequence\-level scores\. Standard metrics include length\-normalized log\-probabilities, the geometric mean of token probabilities, and minimum token probability\(Fomichevaet al\.,[2020](https://arxiv.org/html/2607.06327#bib.bib64); Vazhentsevet al\.,[2023](https://arxiv.org/html/2607.06327#bib.bib30)\)\. Inference\-only methods remain the standard for open\-weights models due to their computational efficiency and lack of supervision requirements\.

### 2\.2Closed\-Box Methods

Closed\-box techniques are essential when access to logits or gradients is restricted \(e\.g\., commercial APIs\)\. These methods rely exclusively on the generated text\.

##### Self verbalized uncertainty

This approach prompts the LLM to explicitly state its confidence as a score or linguistic expression\(Kadavathet al\.,[2022](https://arxiv.org/html/2607.06327#bib.bib31); Tianet al\.,[2023c](https://arxiv.org/html/2607.06327#bib.bib34)\)\. While accessible, self\-verbalization is frequently miscalibrated and often struggles to discriminate correctness effectively compared to logit\-based baselines\(Xionget al\.,[2024](https://arxiv.org/html/2607.06327#bib.bib58); Groot and Valdenegro \- Toro,[2024](https://arxiv.org/html/2607.06327#bib.bib53)\)\.

##### Consistency and Sampling Strategies

Consistency\-based methods rely on the intuition that correct reasoning is stable across stochastic samples\.*Semantic Entropy*and its variants\(Kuhnet al\.,[2023](https://arxiv.org/html/2607.06327#bib.bib14); Farquharet al\.,[2024](https://arxiv.org/html/2607.06327#bib.bib22); Cecereet al\.,[2025](https://arxiv.org/html/2607.06327#bib.bib41)\)cluster multiple generations by meaning and compute entropy over these clusters\. Other works extend this method using graph theory, modeling generations as nodes in a similarity graph to derive uncertainty from structural properties\(Linet al\.,[2023](https://arxiv.org/html/2607.06327#bib.bib55); Vashurinet al\.,[2025](https://arxiv.org/html/2607.06327#bib.bib54)\)\.

### 2\.3Evaluation and Multilingual Challenges

UE performance is typically evaluated via*discrimination*AUROC but reliable evaluation remains a challenge\.

##### Ground Truth Matching

Santilliet al\.\([2025](https://arxiv.org/html/2607.06327#bib.bib51)\)demonstrate that automated ground truth matching like ROUGE or BERTScore correlate strongly with superficial features \(e\.g\., length\) rather than factual accuracy, inflating reported UE performance\. To address this, we avoid automated ground truth matching and rely on exact matching of multiple\-choice answers against human\-curated labels\. Prior work has also used MCQA exact matching to evaluate model reliability\. For example,Madhusudhanet al\.\([2025](https://arxiv.org/html/2607.06327#bib.bib8)\)study whether models select an explicit “I Don’t Know” option as a binary abstention behavior\. Our framework differs in three ways: we evaluate continuous\-valued uncertainty signals \(rather than binary abstention\), extract them from long\-form reasoning traces \(rather than the answer choice itself\), and operate across 22 languages \(rather than English only\)\.

##### Multilingual Uncertainty Estimation

While UE is well\-studied in English, multilingual evaluation remains underexplored\. To the best of our knowledge,Xueet al\.\([2025](https://arxiv.org/html/2607.06327#bib.bib47)\)present the only dedicated multilingual UE study, introducing*MlingConf*to benchmark three uncertainty methods across five languages\. However, their evaluation relies on machine\-translated data without human post\-editing and employs a short\-answer format with a median response length of just one word\. This setup primarily assesses how language affects question comprehension, offering limited insight into uncertainty during extended generation\. Our benchmark addresses this gap by eliciting long\-form reasoning in the target language while preserving objective correctness labels\.

![Refer to caption](https://arxiv.org/html/2607.06327v1/image_setting.png)Figure 1:Evaluation pipeline\. Unlike prior work that estimates uncertainty from predicted labels in English\-only settings, our framework \(1\) extracts uncertainty from the model’s*reasoning trace*and \(2\) evaluates across 22 languages\. The pipeline proceeds as: multilingual prompts \(A\) → LLM generation \(B\) → correctness evaluation against ground truth \(C\) → uncertainty estimation on reasoning \(D\) → method comparison via AUROC \(E\)\.

## 3Methodology

Our goal is to evaluate LLM UE methods across multiple languages, including low\-resource ones, while supporting long text generation and using model\-free metrics to maintain an unbiased setup\.

To achieve this, we leverage human\-curated Multiple\-Choice Question Answer \(MCQA\) datasets where the set of possible labels is a fixed set of options \(such as A, B, C, and D\)\. MCQA labels can be interpreted without relying on string\-matching, LLM\-as\-a\-judge, or embedding similarity techniques, thus providing ground truth without any approximation or language bias\. MCQA labels are too short to derive uncertainty estimates from the model; for this reason, we apply UE methods to the LLM’s reasoning text, similarly to the approach ofPodolak and Verma \([2025](https://arxiv.org/html/2607.06327#bib.bib69)\)\.

Specifically, we prompt the model to generate a reasoning explanation before producing its answer \(e\.g\., Chain of Thoughts\(Weiet al\.,[2022](https://arxiv.org/html/2607.06327#bib.bib11)\)\)\. We then apply uncertainty estimation methods to this reasoning text \(on average 150 words\), while using exact matching against MCQA labels to determine answer correctness\. This allows us to retain the reliability of MCQA labels as ground truth while extending UE evaluation to long text generation\. We deliberately choose MCQA over open\-ended QA datasets for two reasons: \(i\) open\-ended answers are extremely short factoids \(median 1–2 words\), so they do not provide longer text for UE than MCQA labels do, and \(ii\) their evaluation does not ensure a language\- and correctness\-unbiased setup\. A detailed justification is provided in Appendix[B](https://arxiv.org/html/2607.06327#A2)\. Finally, we evaluate the performance of each UE method using the Area Under the Receiver Operating Characteristic \(AUROC\) curve, which is the standard metric in the UE literature for measuring how well uncertainty scores correlate with answer correctness\. The complete process is illustrated in Figure[1](https://arxiv.org/html/2607.06327#S2.F1)\.

Datasets:We use two human\-curated MCQA datasets, namely Global\-MMLUSinghet al\.\([2025](https://arxiv.org/html/2607.06327#bib.bib63)\)\(with 4 possible choices\) and MMLU\-ProXXuanet al\.\([2025](https://arxiv.org/html/2607.06327#bib.bib59)\)\(with 10 possible choices\)\. We focus on the 22 languages that are common to both datasets \(the complete list of languages is available in Appendix[C](https://arxiv.org/html/2607.06327#A3)\)\. These two datasets are parallel, meaning that every question\-answer pair is available in every language, making our multilingual evaluation comparable across languages\. These two datasets cover a broad set of categories: Global\-MMLU contains 6 categories, while MMLU\-ProX contains 14 categories \(see Appendix[E](https://arxiv.org/html/2607.06327#A5)\)\. To ensure balanced representation, we sample 100 questions per category from each dataset, yielding 600 questions from Global\-MMLU and 1,400 from MMLU\-ProX, for a total of 2,000 unique questions\. Since both datasets are available in all 22 languages, this results in 44,000 language\-specific instances\. We further verify that the MCQA labels are uniformly distributed in both datasets, ruling out positional bias \(see App\.[J](https://arxiv.org/html/2607.06327#A10)\)\.

Models:Our evaluation spans nine models from both closed\-source and open\-source families\. For closed\-source models, we employ the Claude 4\.5 SonnetAnthropic \([2025](https://arxiv.org/html/2607.06327#bib.bib2)\)\- the state\-of\-the\-art model at the time of writing\. For open\-source models, we consider instruction\-tuned models including five Gemma3 models \(0\.27B, 1B, 4B, 12B, 27B\)\(Teamet al\.,[2025](https://arxiv.org/html/2607.06327#bib.bib7)\)and three Qwen3 models \(4B, 30B\-A3B, 235B\-A22B\)\(Yanget al\.,[2025](https://arxiv.org/html/2607.06327#bib.bib6)\), encompassing both dense and Mixture of Experts \(MoE\) architectures\. We report a scaling study on the Q&A accuracy in Figure[2](https://arxiv.org/html/2607.06327#S3.F2)\. In our experimentation we applied the quantization only to Qwen3 235B\-A22B to 4bit due to hardware constrains \(see our Hardware Infrastructure in Appendix[H](https://arxiv.org/html/2607.06327#A8)\)\. In Appendix[G](https://arxiv.org/html/2607.06327#A7), we investigate whether quantization impacts multilingual UE performance\. We selected Qwen3\-30B for this analysis as it is the largest model that fits in memory at full precision\. Results show no statistically significant effect when quantizing to 8\-bit and 4\-bit\.

![Refer to caption](https://arxiv.org/html/2607.06327v1/img/gemma_scaling_study_accuracy.png)Figure 2:Scaling study on MCQA accuracy on both datasets \(mean±\\pm95% CI\) over 22 languages\. Similar trends are observed in Qwen models\.Uncertainty Estimation Methods:Following the taxonomy in Section[2](https://arxiv.org/html/2607.06327#S2), we evaluate 9 methods, including both open\-box methods \(with access to model internal states\) and closed\-box methods\. Open\-box probability baselines include TokenEntropy, Maximum\-Sequence\-Probability \(Max Prob\), and Self Certainty\(Fomichevaet al\.,[2020](https://arxiv.org/html/2607.06327#bib.bib64)\)\. Closed\-box methods include Self\-verbalized uncertainty\(Tianet al\.,[2023a](https://arxiv.org/html/2607.06327#bib.bib67)\)and multi\-sample disagreement based on lexical overlap as Lexical Similarity \(ROUGE\-L\)\(Fomichevaet al\.,[2020](https://arxiv.org/html/2607.06327#bib.bib64)\), semantic clustering such as Semantic Entropy\(Kuhnet al\.,[2023](https://arxiv.org/html/2607.06327#bib.bib14)\), and graph\-based dispersion \(EigValLaplacian\_Jaccard, DegMat\_Jaccard, Eccentricity\_NLI\_Jaccard\)\(Linet al\.,[2024](https://arxiv.org/html/2607.06327#bib.bib50)\)\. More detailed method definitions are available in Appendix[A](https://arxiv.org/html/2607.06327#A1)\. All the prompts used are listed in Appendix[D](https://arxiv.org/html/2607.06327#A4)\. To run our experiments we leverage LM\-Polygraph\(Fadeevaet al\.,[2023](https://arxiv.org/html/2607.06327#bib.bib62)\), a framework that implements state\-of\-the\-art UE methods\. Generation parameters are those used in LM\-Polygraph, following the original paper implementations\.

## 4Results And Discussion

##### RQ1: Which UE methods are robust across models and languages?

![Refer to caption](https://arxiv.org/html/2607.06327v1/img/ue_methods_heatmap.png)Figure 3:Per\-language AUROC for nine UE methods across 22 languages on the dense Gemma3\-27B and MoE Qwen3\-235B\-A22B models\. Languages are sorted by mean AUROC \(median sorting produces identical rankings\)\. Closed\-box UE methods are marked with a black circle while open\-box are marked with a white circle\. In the left block we average across models, in the middle block across languages, while in the right block we average across languages and models\. Remarkably, the closed\-box methodSelf Verbalizedachieves top performance, rivaling open\-box methods such asToken EntropyandSelf Certainty—suggesting that effective UE is achievable without access to token\-level probabilities\.To address this question, Figure[3](https://arxiv.org/html/2607.06327#S4.F3)compares nine UE methods across 22 languages and two open\-source models \(Gemma3\-27B and Qwen3\-235B\)\. We excluded closed\-source models to ensure a fair comparison between open\-box and closed\-box approaches\.

The best\-performing method is “Self Verbalized”, which achieves an average AUROC of 0\.72 and demonstrates consistent performance across all languages and models\. The second\-ranked method is the open\-box “Token Entropy” with an AUROC of 0\.71, followed by “Self Certainty” at 0\.70\. The open\-box “Max Prob” performs reasonably well on average \(0\.68\) but degrades substantially on low\-resource languages, dropping to 0\.56 on Yoruba and 0\.57 on Nepali\.

Sampling\-based methods maintain reasonable performance on high\-resource languages but fail on low\-resource ones, with Eccentricity and Semantic Entropy dropping to near\-random performance on Yoruba \(0\.49 and 0\.50 AUROC\)\. These methods generate multiple reasoning traces and use their diversity as an uncertainty signal—the intuition being that confident models produce consistent outputs while uncertain models vary\. To understand why this fails, we analyze the diversity of reasoning traces across all 22 languages using pairwise Jaccard distance\. For high\-resource languages, the expected pattern holds: incorrect predictions produce more diverse reasoning than correct ones \(English: 0\.72 vs\. 0\.63,p<10−6p<10^\{\-6\}; German: 0\.73 vs\. 0\.65; Spanish: 0\.74 vs\. 0\.63; French: 0\.69 vs\. 0\.64\), with gaps of 0\.08–0\.11\. For low\-resource languages, this gap collapses to≤\\leq0\.03: Yoruba \(0\.93 vs\. 0\.93\), Swahili \(0\.93 vs\. 0\.92\), Bengali \(0\.83 vs\. 0\.80\)\. LLMs generate highly variable reasoning regardless of correctness, leaving no diversity signal to exploit\. Self\-verbalized approaches, by contrast, leverage meta\-cognitive capabilities\(Steyvers and Peters,[2025](https://arxiv.org/html/2607.06327#bib.bib56); Xuet al\.,[2025](https://arxiv.org/html/2607.06327#bib.bib57)\)that transfer more robustly across languages\.

##### RQ2: How does model scale affect multilingual UE method efficacy?

We investigate the effect of model parameters scale on UE performance, we evaluate the top\-2 open\-box \(Token Entropy, Self Certainty\) and top\-2 closed\-box \(Self Verbalized, Lexical Sim\.\) methods \(selected for figure clarity\) that scale best across all languages \(see RQ1\); for all other methods, see App\.[I](https://arxiv.org/html/2607.06327#A9)\. In Figure[4](https://arxiv.org/html/2607.06327#S4.F4), we computed the average across all the languages and averaging across datasets\. We select Qwen3 as it offers the broadest parameter range among the open\-source models in our study from 4B to 235B\. We observe a clear divergence in scaling behavior\. Token Entropy remains stable across all scales with overlapping confidence intervals indicating no statistically significant variation at the change of the parameters scale\. Self Verbalized, by contrast, improves substantially at increasing the parameter \(from 0\.65→\\rightarrow0\.66→\\rightarrow0\.77 AUROC\); the difference between 4B and 30B is not statistically significant as the CIs are overlapping but when scaling to 235B, the difference becomes statistically significant\. At 4B and 30B scale, Token Entropy significantly outperforms Self Verbalized, but this relationship reverses at 235B where Self Verbalized achieves the highest AUROC among all methods\. These findings have practical implications: at smaller scales, open\-box methods such as Token Entropy provide the most reliable uncertainty signal; at larger scales, Self Verbalized surpasses all alternatives by a significant margin, suggesting that meta\-cognitive abilities required for accurate self\-assessment emerge primarily at larger model scales, consistent with prior findings on self\-knowledge in LLMs\(Kadavathet al\.,[2022](https://arxiv.org/html/2607.06327#bib.bib31); Steyvers and Peters,[2025](https://arxiv.org/html/2607.06327#bib.bib56)\)\. However, that training pipelines are not uniform across model sizes within the same family: the Qwen3 Technical Report\(Yanget al\.,[2025](https://arxiv.org/html/2607.06327#bib.bib6)\)discloses that smaller Qwen3 variants are distilled from larger teacher models \(Qwen3\-235B\-A22B\), so part of the observed gap may also reflect that self\-verbalization capability are sensible to training procedures\. Furthermore, in Figure[5](https://arxiv.org/html/2607.06327#S4.F5)we show results across Gemma from 270M to 27B parameters, averaged across all nine UE methods\. A clear scaling trend emerges again: models below 1B parameters \(Gemma3\-0\.27B, Gemma3\-1B, Qwen3\-0\.6B\) perform at random baseline \(0\.50 AUROC\), indicating that UE methods fail to extract meaningful uncertainty signals from very small models\. Performance increases substantially between 1B and 4B parameters, with achieving 0\.60\-0\.65 AUROC\. Beyond 12B parameters, performance plateaus around 0\.67\-0\.70 AUROC across both model families\. We report Expected Calibration Error in App\.[K](https://arxiv.org/html/2607.06327#A11), where larger models show a slight tendency toward overconfidence\.

![Refer to caption](https://arxiv.org/html/2607.06327v1/img/UE_scaling.png)Figure 4:Effect of model scale on UE performance \(Qwen3 family\)\. Shaded regions denote 95% CIs\. Dashed line: random baseline\.![Refer to caption](https://arxiv.org/html/2607.06327v1/img/gemma_scaling_auroc.png)Figure 5:Uncertainty estimation performance \(AUROC ± CI 95%\) across varying model scales\. Results are averaged across 22 languages in both datasets\. Increasing model size generally yields higher AUROC scores\.
##### RQ3: How does the language of reasoning affect UE quality across languages?

To investigate whether the language used for reasoning influences multilingual UE, we compare two configurations:Multilingual, where the model reasons in the target language \(i\.e\., language of the question and answers\), andEnglish Reasoning, where reasoning is conducted in English regardless of the target language\. Figure[6](https://arxiv.org/html/2607.06327#S4.F6)reports AUROC with 95% confidence intervals across 22 languages, averaged across all datasets and the largest models \(Qwen 235B, Gemma3 27B, and Claude 4\.5 Sonnet\), grouped by WALS language genus111https://wals\.info/\.

On average, English reasoning improves UE quality \(0\.72 vs\. 0\.68 AUROC\)\. Crucially, this improvement is driven by low\-resource languages: English reasoning effectively brings their UE performance to the level of high\-resource languages\. For Yoruba, AUROC improves from 0\.58 to 0\.68; for Swahili, from 0\.58 to 0\.64; for Nepali, from 0\.66 to 0\.71—all statistically significant gains \(non\-overlapping CIs\)\. After applying English reasoning, these languages achieve AUROC scores comparable to Germanic and Romance languages \(0\.71–0\.73\), which show no significant change between conditions\. These results reveal an asymmetry between natural language understanding and generation\. In both conditions, the question remains in the target language—only the reasoning language changes\. If comprehension were the bottleneck, changing the reasoning language would not help: the model would not know what to reason about\. The fact that English reasoning closes the gap suggests that models comprehend low\-resource questions adequately but struggle to generate coherent reasoning in those languages\. By shifting generation to English, models leverage their stronger generative capabilities, producing stronger uncertainty signals\.

![Refer to caption](https://arxiv.org/html/2607.06327v1/img/plot_Global_Average_grouped.png)Figure 6:Effect of reasoning language on UE quality across 22 languages\. Solid bars represent reasoning in target language condition; hatched bars represent the English\-reasoning condition\. AUROC scores are averaged across all UE methods, datasets, and the largest models \(Qwen3\-235B, Gemma3\-27B, Claude Sonnet 4\.5\)\. Error bars denote 95% CI\. English reasoning significantly narrows the performance gap for South Asian and African languages\.
##### RQ4: How do UE methods perform under cross\-lingual answer settings?

Multilingual applications often involve cross\-lingual scenarios where different components of an input span multiple languages—a common situation in cross\-lingual retrieval\-augmented generation \(RAG\) systems where documents retrieved from multiple languages must be integrated\. We evaluate whether UE methods remain robust when this cross\-lingual complexity is introduced\. We construct instances where the question and reasoning are in the target language, but each answer option is drawn from a different language\. Leveraging the parallel nature of our multilingual datasets, we use aligned answer IDs across translations and sample each option from a different language variant\. For instance, an English question asking “What day comes after Monday?” would present answer options such as “Mardi” \(French, Tuesday\), “Mercoledì” \(Italian, Wednesday\), “Donnerstag” \(German, Thursday\), and “Viernes” \(Spanish, Friday\)\. Crucially, unlike RQ3, the reasoning language always follows the question language in both conditions\. We evaluate using the largest models in our study: Gemma3\-27B, Qwen3\-235B\-A22B, and Claude Sonnet 4\.5\.

![Refer to caption](https://arxiv.org/html/2607.06327v1/img/cross-lingual.png)Figure 7:Effect of cross\-lingual answer options on UE quality\. Solid bars: multilingual setting; hatched bars: cross\-lingual with each answer in a different language from the question language\. Reasoning follows the question language in both conditions\. AUROC scores are averaged across all UE methods, datasets, and the largest models \(Qwen3\-235B, Gemma3\-27B, Sonnet 4\.5\)\. Error bars denote 95% CIs\.Figure[7](https://arxiv.org/html/2607.06327#S4.F7)reports AUROC with 95% confidence intervals comparing the standard multilingual setting \(solid bars\) against the cross\-lingual setting \(hatched bars\)\. This setting represents a more challenging evaluation scenario than standard multilingual UE\. Despite this increased difficulty, the cross\-lingual setting does not degrade UE performance\. On average, AUROC remains stable \(0\.68 vs\. 0\.67\) with overlapping confidence intervals across all language families\. These results demonstrate that UE methods are robust to cross\-lingual answer configurations, suggesting they can be reliably deployed in multilingual systems where language mixing is common\.

##### RQ5: How can we find the best UE threshold across languages?

A key practical application of uncertainty estimation is selective prediction: using uncertainty scores to abstain from unreliable outputs\(Madhusudhanet al\.,[2025](https://arxiv.org/html/2607.06327#bib.bib8)\), thereby improving system trustworthiness\. The central challenge in multilingual settings is threshold calibration—selecting a filtering threshold on validation data for application at inference time\. Given an uncertainty scoreuuand thresholdtt, we accept the prediction ifu<tu<t, otherwise abstain \(e\.g\., defer to human review\)\. We selectttby maximizing F1\-score for error detection on validation data, comparing three calibration strategies: \(1\)tENt\_\{\\text\{EN\}\}, optimized on English data only and applied to all languages; \(2\)tGLOBALt\_\{\\text\{GLOBAL\}\}, optimized on pooled multilingual data; and \(3\)tLANGt\_\{\\text\{LANG\}\}, using language\-specific thresholds transferred to corresponding target languages\. We use one dataset as validation set for threshold tuning and the other as test set with Claude Sonnet 4\.5, comparing against Oracle upper bounds where thresholds are tuned directly on target test data\.

Table 1:Imapct of different threshold selection using Self\-Verbalized \(4\.5 Sonnet\)\. TP/TN: accepted predictions that are correct/wrong\. FP/FN: filtered predictions that are wrong/correct\. All values are percentages of total 44,000 samples averaged across the 2 datasets\.Table[1](https://arxiv.org/html/2607.06327#S4.T1)reports the confusion matrix for each strategy\. The selective prediction strategies reveal distinct trade\-offs between error reduction, coverage, and deployment complexity\. English\-only calibration \(tENt\_\{\\text\{EN\}\}\) offers the most practical entry point, requiring minimal data collection while achieving a 43% relative error reduction \(from 13% to 7\.7%\) and maintaining high coverage with only 6\.6% of correct predictions filtered out\. Pooled multilingual optimization \(tGLOBALt\_\{\\text\{GLOBAL\}\}\) strikes a middle ground, delivering 56% error reduction at the cost of moderate coverage loss \(13\.5% false negatives\), making it suitable for applications where validation data spans multiple languages\. Language\-specific calibration \(tLANGt\_\{\\text\{LANG\}\}\) achieves the strongest practical performance with 60% error reduction and 8\.1% error detection, nearly matching Oracle thresholds tuned on target data itself, though this comes at the expense of filtering out 14\.4% of correct predictions\. The Oracle strategy establishes the performance ceiling at 62% error reduction but requires target\-specific calibration data, limiting its practical applicability\. These results demonstrate that the choice of strategy depends critically on application requirements: safety\-critical systems benefit most fromtLANGt\_\{\\text\{LANG\}\}’s superior error detection, resource\-constrained deployments favortENt\_\{\\text\{EN\}\}’s simplicity, and balanced applications achieve optimal accuracy\-coverage trade\-offs withtGLOBALt\_\{\\text\{GLOBAL\}\}\.

## 5Conclusion & Future Work

We present the largest multilingual evaluation of LLM uncertainty estimation to date, covering 9 UE methods, 9 models \(5 Gemma3, 4 Qwen3, and Claude Sonnet 4\.5\) spanning 270M to 235B parameters, and 22 languages from high\- to low\-resource settings\. Our experiments yield three actionable findings\. First, generation language matters more than question language: prompting models to reason in English closes the UE performance gap for low\-resource languages, indicating that the reliability bottleneck lies in generation rather than comprehension\. Notably, Swahili, Yoruba, and Telugu—the lowest\-performing languages in our benchmark—achieve performance on par with English when reasoning is elicited in English\. Nevertheless, reasoning in the target language is important under certain scenarios \(eg explanations in target language for non English speakers\), and closing the gap is left for future work\.

Second, UE method selection should be scale\-aware\. All UE methods maintain stable performance across model scales, except for Self Verbalized, which improves at scale \(0\.65 AUROC at 4B to 0\.77 at 235B\), becoming the best\-performing approach with larger LLMs\. This trend is consistent with prior evidence that meta\-cognitive capabilities emerge primarily in larger models\(Tianet al\.,[2023b](https://arxiv.org/html/2607.06327#bib.bib16); Xionget al\.,[2024](https://arxiv.org/html/2607.06327#bib.bib58); Steyvers and Peters,[2025](https://arxiv.org/html/2607.06327#bib.bib56)\), although training\-pipeline differences across sizes \(e\.g\., distillation\) may also contribute\.

Third, threshold calibration transfers across datasets but benefits from language awareness\. English\-only calibration halves error rates, while multilingual calibration reduces errors by up to 77%\. We also observe that sampling\-based consistency methods degrade on low\-resource languages\. Since these methods rely on lexical overlap measures, sensitivity to surface\-level variation may explain this degradation, though further investigation is needed\. We release our evaluation framework to support research on trustworthy multilingual AI\. Future directions include extending to open\-ended generation with reliable correctness assessment and developing UE methods explicitly designed for cross\-linguistic robustness\.

## 6Limitations

We focus on two MCQA datasets, which, while providing reliable ground\-truth labels without model\-based proxies, may not fully represent the diversity of real\-world NLP applications\. The landscape of uncertainty estimation methods is vast, and we considered in our analysis 9 state\-of\-the\-art methods selected to cover the main paradigms \(probability\-based, verbalized, and consistency\-based\), we did not include any training\-based methods\.

## References

- Anthropic \(2025\)Introducing Claude Sonnet 4\.5\.Note:[https://www\.anthropic\.com/news/claude\-sonnet\-4\-5](https://www.anthropic.com/news/claude-sonnet-4-5)Accessed: 2026\-05\-25Cited by:[§3](https://arxiv.org/html/2607.06327#S3.p5.1)\.
- R\. Bommasani \(2021\)On the opportunities and risks of foundation models\.arXiv preprint arXiv:2108\.07258\.Cited by:[§1](https://arxiv.org/html/2607.06327#S1.p1.1)\.
- N\. Cecere, A\. Bacciu, I\. Fernández\-Tobías, and A\. Mantrach \(2025\)Monte Carlo temperature: a robust sampling strategy for LLM’s uncertainty quantification methods\.InProceedings of the 5th Workshop on Trustworthy NLP \(TrustNLP 2025\),T\. Cao, A\. Das, T\. Kumarage, Y\. Wan, S\. Krishna, N\. Mehrabi, J\. Dhamala, A\. Ramakrishna, A\. Galystan, A\. Kumar, R\. Gupta, and K\. Chang \(Eds\.\),Albuquerque, New Mexico,pp\. 305–320\.External Links:[Link](https://aclanthology.org/2025.trustnlp-main.21/),[Document](https://dx.doi.org/10.18653/v1/2025.trustnlp-main.21),ISBN 979\-8\-89176\-233\-6Cited by:[§1](https://arxiv.org/html/2607.06327#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.06327#S2.SS2.SSS0.Px2.p1.1)\.
- E\. Fadeeva, R\. Vashurin, A\. Tsvigun, A\. Vazhentsev, S\. Petrakov, K\. Fedyanin, D\. Vasilev, E\. Goncharova, A\. Panchenko, M\. Panov, T\. Baldwin, and A\. Shelmanov \(2023\)LM\-polygraph: uncertainty estimation for language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Y\. Feng and E\. Lefever \(Eds\.\),Singapore,pp\. 446–461\.External Links:[Link](https://aclanthology.org/2023.emnlp-demo.41/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-demo.41)Cited by:[§3](https://arxiv.org/html/2607.06327#S3.p6.1)\.
- S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal \(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630\(8017\),pp\. 625–630\.Cited by:[§2\.2](https://arxiv.org/html/2607.06327#S2.SS2.SSS0.Px2.p1.1)\.
- M\. Fomicheva, S\. Sun, L\. Yankovskaya, F\. Blain, F\. Guzmán, M\. Fishel, N\. Aletras, V\. Chaudhary, and L\. Specia \(2020\)Unsupervised quality estimation for neural machine translation\.Transactions of the Association for Computational Linguistics8,pp\. 539–555\.External Links:[Link](https://aclanthology.org/2020.tacl-1.35/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00330)Cited by:[§A\.1](https://arxiv.org/html/2607.06327#A1.SS1.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2607.06327#S2.SS1.p2.1),[§3](https://arxiv.org/html/2607.06327#S3.p6.1)\.
- T\. Groot and M\. Valdenegro \- Toro \(2024\)Overconfidence is key: verbalized uncertainty evaluation in large language and vision\-language models\.InProceedings of the 4th Workshop on Trustworthy Natural Language Processing \(TrustNLP 2024\),A\. Ovalle, K\. Chang, Y\. T\. Cao, N\. Mehrabi, J\. Zhao, A\. Galstyan, J\. Dhamala, A\. Kumar, and R\. Gupta \(Eds\.\),Mexico City, Mexico,pp\. 145–171\.External Links:[Link](https://aclanthology.org/2024.trustnlp-1.13/),[Document](https://dx.doi.org/10.18653/v1/2024.trustnlp-1.13)Cited by:[§2\.2](https://arxiv.org/html/2607.06327#S2.SS2.SSS0.Px1.p1.1)\.
- M\. Joshi, E\. Choi, D\. Weld, and L\. Zettlemoyer \(2017\)TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),R\. Barzilay and M\. Kan \(Eds\.\),Vancouver, Canada,pp\. 1601–1611\.External Links:[Link](https://aclanthology.org/P17-1147/),[Document](https://dx.doi.org/10.18653/v1/P17-1147)Cited by:[Appendix B](https://arxiv.org/html/2607.06327#A2.p1.2)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, D\. Drain, E\. Perez, N\. Schiefer, Z\. Hatfield\-Dodds, N\. DasSarma, E\. Tran\-Johnson,et al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§A\.2\.1](https://arxiv.org/html/2607.06327#A1.SS2.SSS1.p1.1),[§2\.2](https://arxiv.org/html/2607.06327#S2.SS2.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2607.06327#S4.SS0.SSS0.Px2.p1.2)\.
- J\. Kossen, J\. Han, M\. Razzak, L\. Schut, S\. Malik, and Y\. Gal \(2024\)Semantic entropy probes: robust and cheap hallucination detection in LLMs\.arXiv preprint arXiv:2406\.15927\.Cited by:[§1](https://arxiv.org/html/2607.06327#S1.p1.1)\.
- L\. Kuhn, Y\. Gal, and S\. Farquhar \(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.arXiv preprint arXiv:2302\.09664\.Cited by:[§A\.2\.2](https://arxiv.org/html/2607.06327#A1.SS2.SSS2.Px1.p1.4),[§1](https://arxiv.org/html/2607.06327#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.06327#S2.SS2.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2607.06327#S3.p6.1)\.
- T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee, K\. Toutanova, L\. Jones, M\. Kelcey, M\. Chang, A\. M\. Dai, J\. Uszkoreit, Q\. Le, and S\. Petrov \(2019\)Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 452–466\.External Links:[Link](https://aclanthology.org/Q19-1026/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00276)Cited by:[Appendix B](https://arxiv.org/html/2607.06327#A2.p1.2)\.
- C\. Lin \(2004\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,Barcelona, Spain,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§A\.2\.2](https://arxiv.org/html/2607.06327#A1.SS2.SSS2.Px2.p1.1)\.
- Z\. Lin, S\. Trivedi, and J\. Sun \(2023\)Generating with confidence: uncertainty quantification for black\-box large language models\.arXiv preprint arXiv:2305\.19187\.Cited by:[§A\.2\.2](https://arxiv.org/html/2607.06327#A1.SS2.SSS2.Px3.p1.2),[§2\.2](https://arxiv.org/html/2607.06327#S2.SS2.SSS0.Px2.p1.1)\.
- Z\. Lin, S\. Trivedi, and J\. Sun \(2024\)Generating with confidence: uncertainty quantification for black\-box large language models\.Transactions on Machine Learning Research\.Note:arXiv preprint arXiv:2305\.19187Cited by:[§3](https://arxiv.org/html/2607.06327#S3.p6.1)\.
- S\. Longpre, Y\. Lu, and J\. Daiber \(2021\)MKQA: a linguistically diverse benchmark for multilingual open domain question answering\.Transactions of the Association for Computational Linguistics9,pp\. 1389–1406\.External Links:[Link](https://aclanthology.org/2021.tacl-1.82/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00433)Cited by:[Appendix B](https://arxiv.org/html/2607.06327#A2.p1.2)\.
- N\. Madhusudhan, S\. T\. Madhusudhan, V\. Yadav, and M\. Hashemi \(2025\)Do LLMs know when to NOT answer? investigating abstention abilities of large language models\.InProceedings of the 31st International Conference on Computational Linguistics,O\. Rambow, L\. Wanner, M\. Apidianaki, H\. Al\-Khalifa, B\. D\. Eugenio, and S\. Schockaert \(Eds\.\),Abu Dhabi, UAE,pp\. 9329–9345\.External Links:[Link](https://aclanthology.org/2025.coling-main.627/)Cited by:[§2\.3](https://arxiv.org/html/2607.06327#S2.SS3.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2607.06327#S4.SS0.SSS0.Px5.p1.7)\.
- A\. Malinin and M\. Gales \(2021\)Uncertainty estimation in autoregressive structured prediction\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=jN5y-zb5Q7m)Cited by:[§A\.1](https://arxiv.org/html/2607.06327#A1.SS1.SSS0.Px1.p1.1)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2607.06327#S1.p1.1)\.
- J\. Podolak and R\. Verma \(2025\)Read your own mind: reasoning helps surface self\-confidence signals in llms\.arXiv preprint arXiv:2505\.23845\.Cited by:[§3](https://arxiv.org/html/2607.06327#S3.p2.1)\.
- A\. Santilli, A\. Golinski, M\. Kirchhof, F\. Danieli, A\. Blaas, M\. Xiong, L\. Zappella, and S\. Williamson \(2025\)Revisiting uncertainty quantification evaluation in language models: spurious interactions with response length bias results\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 743–759\.External Links:[Link](http://dx.doi.org/10.18653/v1/2025.acl-short.60),[Document](https://dx.doi.org/10.18653/v1/2025.acl-short.60)Cited by:[§1](https://arxiv.org/html/2607.06327#S1.p1.1),[§2\.3](https://arxiv.org/html/2607.06327#S2.SS3.SSS0.Px1.p1.1)\.
- S\. Singh, A\. Romanou, C\. Fourrier, D\. I\. Adelani, J\. G\. Ngui, D\. Vila\-Suero, P\. Limkonchotiwat, K\. Marchisio, W\. Q\. Leong, Y\. Susanto, R\. Ng, S\. Longpre, W\. Ko, S\. Ruder, M\. Smith, A\. Bosselut, A\. Oh, A\. F\. T\. Martins, L\. Choshen, D\. Ippolito, E\. Ferrante, M\. Fadaee, B\. Ermis, and S\. Hooker \(2025\)Global mmlu: understanding and addressing cultural and linguistic biases in multilingual evaluation\.External Links:2412\.03304,[Link](https://arxiv.org/abs/2412.03304)Cited by:[§3](https://arxiv.org/html/2607.06327#S3.p4.1)\.
- M\. Steyvers and M\. A\. Peters \(2025\)Metacognition and uncertainty communication in humans and large language models\.Current Directions in Psychological Science,pp\. 09637214251391158\.Cited by:[§4](https://arxiv.org/html/2607.06327#S4.SS0.SSS0.Px1.p3.2),[§4](https://arxiv.org/html/2607.06327#S4.SS0.SSS0.Px2.p1.2),[§5](https://arxiv.org/html/2607.06327#S5.p2.1)\.
- G\. Team, A\. Kamath, J\. Ferret, S\. Pathak, N\. Vieillard, R\. Merhej, S\. Perrin, T\. Matejovicova, A\. Ramé, M\. Rivière, L\. Rouillard, T\. Mesnard, G\. Cideron, J\. Grill, S\. Ramos, E\. Yvinec, M\. Casbon, E\. Pot, I\. Penchev, G\. Liu, F\. Visin, K\. Kenealy, L\. Beyer, X\. Zhai, A\. Tsitsulin, R\. Busa\-Fekete, A\. Feng, N\. Sachdeva, B\. Coleman, Y\. Gao, B\. Mustafa, I\. Barr, E\. Parisotto, D\. Tian, M\. Eyal, C\. Cherry, J\. Peter, D\. Sinopalnikov, S\. Bhupatiraju, R\. Agarwal, M\. Kazemi, D\. Malkin, R\. Kumar, D\. Vilar, I\. Brusilovsky, J\. Luo, A\. Steiner, A\. Friesen, A\. Sharma, A\. Sharma, A\. M\. Gilady, A\. Goedeckemeyer, A\. Saade, A\. Feng, A\. Kolesnikov, A\. Bendebury, A\. Abdagic, A\. Vadi, A\. György, A\. S\. Pinto, A\. Das, A\. Bapna, A\. Miech, A\. Yang, A\. Paterson, A\. Shenoy, A\. Chakrabarti, B\. Piot, B\. Wu, B\. Shahriari, B\. Petrini, C\. Chen, C\. L\. Lan, C\. A\. Choquette\-Choo, C\. Carey, C\. Brick, D\. Deutsch, D\. Eisenbud, D\. Cattle, D\. Cheng, D\. Paparas, D\. S\. Sreepathihalli, D\. Reid, D\. Tran, D\. Zelle, E\. Noland, E\. Huizenga, E\. Kharitonov, F\. Liu, G\. Amirkhanyan, G\. Cameron, H\. Hashemi, H\. Klimczak\-Plucińska, H\. Singh, H\. Mehta, H\. T\. Lehri, H\. Hazimeh, I\. Ballantyne, I\. Szpektor, I\. Nardini, J\. Pouget\-Abadie, J\. Chan, J\. Stanton, J\. Wieting, J\. Lai, J\. Orbay, J\. Fernandez, J\. Newlan, J\. Ji, J\. Singh, K\. Black, K\. Yu, K\. Hui, K\. Vodrahalli, K\. Greff, L\. Qiu, M\. Valentine, M\. Coelho, M\. Ritter, M\. Hoffman, M\. Watson, M\. Chaturvedi, M\. Moynihan, M\. Ma, N\. Babar, N\. Noy, N\. Byrd, N\. Roy, N\. Momchev, N\. Chauhan, N\. Sachdeva, O\. Bunyan, P\. Botarda, P\. Caron, P\. K\. Rubenstein, P\. Culliton, P\. Schmid, P\. G\. Sessa, P\. Xu, P\. Stanczyk, P\. Tafti, R\. Shivanna, R\. Wu, R\. Pan, R\. Rokni, R\. Willoughby, R\. Vallu, R\. Mullins, S\. Jerome, S\. Smoot, S\. Girgin, S\. Iqbal, S\. Reddy, S\. Sheth, S\. Põder, S\. Bhatnagar, S\. R\. Panyam, S\. Eiger, S\. Zhang, T\. Liu, T\. Yacovone, T\. Liechty, U\. Kalra, U\. Evci, V\. Misra, V\. Roseberry, V\. Feinberg, V\. Kolesnikov, W\. Han, W\. Kwon, X\. Chen, Y\. Chow, Y\. Zhu, Z\. Wei, Z\. Egyed, V\. Cotruta, M\. Giang, P\. Kirk, A\. Rao, K\. Black, N\. Babar, J\. Lo, E\. Moreira, L\. G\. Martins, O\. Sanseviero, L\. Gonzalez, Z\. Gleicher, T\. Warkentin, V\. Mirrokni, E\. Senter, E\. Collins, J\. Barral, Z\. Ghahramani, R\. Hadsell, Y\. Matias, D\. Sculley, S\. Petrov, N\. Fiedel, N\. Shazeer, O\. Vinyals, J\. Dean, D\. Hassabis, K\. Kavukcuoglu, C\. Farabet, E\. Buchatskaya, J\. Alayrac, R\. Anil, Dmitry, Lepikhin, S\. Borgeaud, O\. Bachem, A\. Joulin, A\. Andreev, C\. Hardin, R\. Dadashi, and L\. Hussenot \(2025\)Gemma 3 technical report\.External Links:2503\.19786,[Link](https://arxiv.org/abs/2503.19786)Cited by:[§3](https://arxiv.org/html/2607.06327#S3.p5.1)\.
- K\. Tian, E\. Mitchell, H\. Zhao, G\. Dziugaite, A\. Azaria, and S\. R\. Bowman \(2023a\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 12302–12328\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.575/)Cited by:[§3](https://arxiv.org/html/2607.06327#S3.p6.1)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning \(2023b\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.arXiv preprint arXiv:2305\.14975\.Cited by:[§A\.2\.1](https://arxiv.org/html/2607.06327#A1.SS2.SSS1.p1.1),[§5](https://arxiv.org/html/2607.06327#S5.p2.1)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. Manning \(2023c\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 5433–5442\.Cited by:[§2\.2](https://arxiv.org/html/2607.06327#S2.SS2.SSS0.Px1.p1.1)\.
- R\. Vashurin, E\. Fadeeva, A\. Vazhentsev, L\. Rvanova, D\. Vasilev, A\. Tsvigun, S\. Petrakov, R\. Xing, A\. Sadallah, K\. Grishchenkov, A\. Panchenko, T\. Baldwin, P\. Nakov, M\. Panov, and A\. Shelmanov \(2025\)Benchmarking uncertainty quantification methods for large language models with LM\-polygraph\.Transactions of the Association for Computational Linguistics13,pp\. 220–248\.External Links:[Link](https://aclanthology.org/2025.tacl-1.11/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00737)Cited by:[§2\.2](https://arxiv.org/html/2607.06327#S2.SS2.SSS0.Px2.p1.1)\.
- A\. Vazhentsev, A\. Tsvigun, R\. Vashurin, S\. Petrakov, D\. Vasilev, M\. Panov, A\. Panchenko, and A\. Shelmanov \(2023\)Efficient out\-of\-domain detection for sequence to sequence models\.InFindings of the Association for Computational Linguistics: ACL 2023,Toronto, Canada,pp\. 1430–1454\.Cited by:[§2\.1](https://arxiv.org/html/2607.06327#S2.SS1.p2.1)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InProceedings of the 36th International Conference on Neural Information Processing Systems,NIPS ’22,Red Hook, NY, USA\.External Links:ISBN 9781713871088Cited by:[§1](https://arxiv.org/html/2607.06327#S1.p1.1),[§3](https://arxiv.org/html/2607.06327#S3.p3.1)\.
- M\. Xiong, Z\. Hu, X\. Lu, Y\. LI, J\. Fu, J\. He, and B\. Hooi \(2024\)Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2\.2](https://arxiv.org/html/2607.06327#S2.SS2.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2607.06327#S5.p2.1)\.
- Z\. Xu, T\. Song, and Y\. Lee \(2025\)Confronting verbalized uncertainty: understanding how llm’s verbalized uncertainty influences users in ai\-assisted decision\-making\.International Journal of Human\-Computer Studies197,pp\. 103455\.Cited by:[§4](https://arxiv.org/html/2607.06327#S4.SS0.SSS0.Px1.p3.2)\.
- W\. Xuan, R\. Yang, H\. Qi, Q\. Zeng, Y\. Xiao, A\. Feng, D\. Liu, Y\. Xing, J\. Wang, F\. Gao, J\. Lu, Y\. Jiang, H\. Li, X\. Li, K\. Yu, R\. Dong, S\. Gu, Y\. Li, X\. Xie, F\. Juefei\-Xu, F\. Khomh, O\. Yoshie, Q\. Chen, D\. Teodoro, N\. Liu, R\. Goebel, L\. Ma, E\. Marrese\-Taylor, S\. Lu, Y\. Iwasawa, Y\. Matsuo, and I\. Li \(2025\)MMLU\-prox: a multilingual benchmark for advanced large language model evaluation\.External Links:2503\.10497,[Link](https://arxiv.org/abs/2503.10497)Cited by:[§3](https://arxiv.org/html/2607.06327#S3.p4.1)\.
- B\. Xue, H\. Wang, R\. Wang, S\. Wang, Z\. Wang, Y\. Du, B\. Liang, W\. Zhang, and K\. Wong \(2025\)MlingConf: a comprehensive study of multilingual confidence estimation on large language models\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 2535–2556\.External Links:[Link](https://aclanthology.org/2025.findings-acl.129/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.129),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2607.06327#S1.p1.1),[§2\.3](https://arxiv.org/html/2607.06327#S2.SS3.SSS0.Px2.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§3](https://arxiv.org/html/2607.06327#S3.p5.1),[§4](https://arxiv.org/html/2607.06327#S4.SS0.SSS0.Px2.p1.2)\.

## Appendix AUncertainty Estimation Methods

We classify Uncertainty Estimation \(UE\) methods into two primary categories based on the level of access required to the underlying Large Language Model \(LLM\)\.Open\-box methodsderive uncertainty from the model’s internal probability distributions \(logits\), whereasClosed\-box methodsoperate solely on the generated text, utilizing either self\-reflection or the consistency between multiple stochastic samples\.

Throughout, we denote the input prompt byxx, the model’s parameters byϕ\\phi, and the generated sequence by𝐲=\(y1,…,yT\)\\mathbf\{y\}=\(y\_\{1\},\\dots,y\_\{T\}\)\. The model’s predictive distribution at stepttis denoted aspϕ​\(yt∣𝐲<t,x\)p\_\{\\phi\}\(y\_\{t\}\\mid\\mathbf\{y\}\_\{<t\},x\)\.

### A\.1Open\-Box Methods

These methods require white\-box access to token\-level probabilities\. They are computationally efficient as they typically operate on a single greedy generation, but they are inapplicable to API\-based models where logits are unavailable\.

##### Maximum Sequence Probability

We estimate the confidence of a sequence𝐲\\mathbf\{y\}using its joint probability\. To normalize for varying sequence lengths, we compute the negative average log\-likelihood \(NLL\)\(Malinin and Gales,[2021](https://arxiv.org/html/2607.06327#bib.bib61)\):

uMSP​\(x\)=Norm​\(−1T​∑t=1Tlog⁡pϕ​\(yt∣𝐲<t,x\)\)\.u\_\{\\text\{MSP\}\}\(x\)=\\mathrm\{Norm\}\\left\(\-\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\log p\_\{\\phi\}\(y\_\{t\}\\mid\\mathbf\{y\}\_\{<t\},x\)\\right\)\.\(1\)Here,Norm​\(⋅\)\\mathrm\{Norm\}\(\\cdot\)denotes min–max normalization to the interval\[0,1\]\[0,1\]computed over the evaluation dataset\.

##### Mean Token Entropy

Rather than relying on the probability of the specific token chosen, this method measures the average distributional uncertainty \(Shannon entropy\) over the entire vocabulary𝒱\\mathcal\{V\}at each generation step\(Fomichevaet al\.,[2020](https://arxiv.org/html/2607.06327#bib.bib64)\):

uMTE​\(x\)=Norm​\(1T​∑t=1T∑v∈𝒱−pϕ​\(v\)​log⁡pϕ​\(v\)\),u\_\{\\text\{MTE\}\}\(x\)=\\mathrm\{Norm\}\\left\(\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\sum\_\{v\\in\\mathcal\{V\}\}\-p\_\{\\phi\}\(v\)\\log p\_\{\\phi\}\(v\)\\right\),\(2\)wherepϕ​\(v\)p\_\{\\phi\}\(v\)is shorthand forpϕ​\(v∣𝐲<t,x\)p\_\{\\phi\}\(v\\mid\\mathbf\{y\}\_\{<t\},x\)\.

### A\.2Closed\-Box Methods

These methods treat the LLM as a black box\. We further divide them intoVerbalizedmethods \(single generation, self\-reflection\) andConsistencymethods \(multiple generations, aggregation\)\.

#### A\.2\.1Verbalized Uncertainty

These approaches prompt the model to explicitly state its confidence, relying on the assumption that instruction\-tuned models possess internal calibration capabilities\(Kadavathet al\.,[2022](https://arxiv.org/html/2607.06327#bib.bib31); Tianet al\.,[2023b](https://arxiv.org/html/2607.06327#bib.bib16)\)\.

##### Self\-Certainty\.

We append a query to the context asking the model to rate its confidences∈\[0,1\]s\\in\[0,1\]\. The uncertainty is defined as the complement of the extracted score:

uSelfCert​\(x\)=1−sϕ​\(x\)\.u\_\{\\text\{SelfCert\}\}\(x\)=1\-s\_\{\\phi\}\(x\)\.\(3\)If the model fails to output a valid numerical score, we assign maximum uncertainty \(u=1u=1\)\.

##### Qualitative Verbalization

We prompt the model to describe its confidence using natural language \(e\.g\., “quite sure”, “very uncertain”\)\. We map these linguistic markers to discrete scalar valuesu∈\{0\.0,0\.25,0\.5,0\.75,1\.0\}u\\in\\\{0\.0,0\.25,0\.5,0\.75,1\.0\\\}based on a predefined lexical heuristic\.

#### A\.2\.2Consistency and Graph\-based Uncertainty

These methods approximate uncertainty by measuring the semantic dispersion ofkkstochastic samples𝒴=\{𝐲\(1\),…,𝐲\(k\)\}\\mathcal\{Y\}=\\\{\\mathbf\{y\}^\{\(1\)\},\\dots,\\mathbf\{y\}^\{\(k\)\}\\\}generated at temperatureT=1T=1\.

##### Semantic Entropy \(Jaccard\)

FollowingKuhnet al\.\([2023](https://arxiv.org/html/2607.06327#bib.bib14)\), we cluster the generations in𝒴\\mathcal\{Y\}based on Jaccard similarity\. LetC1,…,CmC\_\{1\},\\dots,C\_\{m\}be the resulting semantic clusters, andp^​\(Cj\)=\|Cj\|/k\\hat\{p\}\(C\_\{j\}\)=\|C\_\{j\}\|/kbe the empirical probability of clusterjj\. The uncertainty is the entropy of the cluster distribution:

uSemEnt​\(x\)=Norm​\(−∑j=1mp^​\(Cj\)​log⁡p^​\(Cj\)\)u\_\{\\text\{SemEnt\}\}\(x\)=\\mathrm\{Norm\}\\left\(\-\\sum\_\{j=1\}^\{m\}\\hat\{p\}\(C\_\{j\}\)\\log\\hat\{p\}\(C\_\{j\}\)\\right\)\(4\)

##### Lexical Similarity\.

We compute the average pairwise similarity using the ROUGE\-L\(Lin,[2004](https://arxiv.org/html/2607.06327#bib.bib60)\)metric as a proxy for consistency\. Higher disagreement implies higher uncertainty:

uLexSim​\(x\)=Norm​\(1−\(k2\)−1​∑i<jRougeL​\(𝐲\(i\),𝐲\(j\)\)\)u\_\{\\text\{LexSim\}\}\(x\)=\\mathrm\{Norm\}\\left\(1\-\\binom\{k\}\{2\}^\{\-1\}\\sum\_\{i<j\}\\mathrm\{RougeL\}\(\\mathbf\{y\}^\{\(i\)\},\\mathbf\{y\}^\{\(j\)\}\)\\right\)\(5\)

##### Graph\-Theoretic Measures\.

Inspired byLinet al\.\([2023](https://arxiv.org/html/2607.06327#bib.bib55)\), we construct a semantic similarity graphG=\(V,E\)G=\(V,E\)where nodes are generations and edges exist ifJac​\(𝐲\(i\),𝐲\(j\)\)≥τ\\mathrm\{Jac\}\(\\mathbf\{y\}^\{\(i\)\},\\mathbf\{y\}^\{\(j\)\}\)\\geq\\tau\. We extract three topological uncertainty features:

1. 1\.Laplacian Eigenvalue:We compute the spectral radius \(largest eigenvalue\) of the graph LaplacianLL, denotedλmax​\(L\)\\lambda\_\{\\max\}\(L\)\. A higherλmax\\lambda\_\{\\max\}indicates a partitioned graph structure \(conflicting answers\): uEig​\(x\)=Norm​\(λmax​\(L\)\)\.u\_\{\\text\{Eig\}\}\(x\)=\\mathrm\{Norm\}\(\\lambda\_\{\\max\}\(L\)\)\.\(6\)
2. 2\.Degree Concentration:We measure the dispersion of node degrees to detect if a dominant consensus cluster exists\. We define the score using the meanμd\\mu\_\{d\}and varianceσd2\\sigma\_\{d\}^\{2\}of the degrees: uDeg​\(x\)=Norm​\(1\(μd\+ϵ\)​\(σd\+ϵ\)\)\.u\_\{\\text\{Deg\}\}\(x\)=\\mathrm\{Norm\}\\left\(\\frac\{1\}\{\(\\mu\_\{d\}\+\\epsilon\)\(\\sigma\_\{d\}\+\\epsilon\)\}\\right\)\.\(7\)
3. 3\.Eccentricity:We measure the geometric spread of the generations using a weighted graph combining Jaccard and NLI scores\. Uncertainty is the mean eccentricity \(maximum shortest\-path distance from each node\): uEcc​\(x\)=Norm​\(1k​∑i=1kmaxj⁡dG​\(i,j\)\)\.u\_\{\\text\{Ecc\}\}\(x\)=\\mathrm\{Norm\}\\left\(\\frac\{1\}\{k\}\\sum\_\{i=1\}^\{k\}\\max\_\{j\}d\_\{G\}\(i,j\)\\right\)\.\(8\)

## Appendix BWhy not open\-ended QA?

An alternative to MCQA would be the use of open\-ended QA datasets such as TriviaQA\(Joshiet al\.,[2017](https://arxiv.org/html/2607.06327#bib.bib5)\), Natural Questions\(Kwiatkowskiet al\.,[2019](https://arxiv.org/html/2607.06327#bib.bib4)\), or their multilingual counterpart MKQA\(Longpreet al\.,[2021](https://arxiv.org/html/2607.06327#bib.bib3)\)\. We do not adopt this setup, because correctness on these datasets must still be assessed on the open\-ended string, which requires regex matching, an LLM\-as\-a\-judge, or embedding similarity, reintroducing exactly the biases in the correctness signal that we want to avoid\. Open\-ended answers are also extremely short factoids \(e\.g\. “Albert Einstein”\): the median is11word for TriviaQA and22words for Natural Questions, so they do not provide longer text for UE than MCQA labels do, and reasoning elicitation would be needed in either case\.

MCQA with elicited reasoning is therefore the only configuration that simultaneously satisfies all three requirements: \(1\) multilingual coverage including low\-resource languages, \(2\) sufficiently long generations for UE, \(3\) and unbiased model\-free correctness\.

## Appendix CLanguages

Our benchmark covers 22 languages given by the intersection of Global\-MMLU and MMLU\-ProX, so that every instance has a strictly parallel multiple\-choice question and shared answer key across all languages\. The languages and their ISO codes are: Arabic \(ar\), Bengali \(bn\), Czech \(cs\), German \(de\), English \(en\), Spanish \(es\), French \(fr\), Hindi \(hi\), Indonesian \(id\), Italian \(it\), Japanese \(ja\), Korean \(ko\), Nepali \(ne\), Portuguese \(pt\), Russian \(ru\), Serbian \(sr\), Swahili \(sw\), Telugu \(te\), Urdu \(ur\), Vietnamese \(vi\), Yoruba \(yo\), and Chinese \(zh\)\.

## Appendix DPrompts

This section details the prompts used in our experiments\. All uncertainty estimation methods share the same base prompt for answer generation\. Theself\-verbalizedmethod additionally employs a follow\-up prompt to elicit a confidence score from the model\.

### D\.1Answer Generation Prompt

Provideyourbestguessforthefollowingquestionabout\{subject\}selectingoneoftheoptions\.Beforegivingyouranswer,provideastep\-by\-stepexplanationofyourthoughtprocess\.Thenonanewlinegivetheguesswithnootherwordsorexplanation\.

Forexample:

Explanation:<onesentencestep\-by\-stepexplanationofyourthoughtprocess\>

Guess:<mostlikelyguess,asshortaspossible;notacompletesentence,justtheguess\!\>

Q:\{question\}

A\.\{choice\_A\}

B\.\{choice\_B\}

C\.\{choice\_C\}

D\.\{choice\_D\}

### D\.2Self\-Verbalized Confidence Prompt

Providetheprobabilitythatyourguessiscorrect\.GiveONLYtheprobability,nootherwordsorexplanation\.

Forexample:

Probability:<theprobabilitybetween0\.0and1\.0thatyourguessiscorrect,withoutanyextracommentarywhatsoever;justtheprobability\!\>

To transform the probability into the uncertainty score we computed the uncertainty as1−1\-probability score\.

## Appendix EQuestion Answering Categories

We report the list of categories in both datasets used\. The categories of Global\-MMLU are \[business, humanities, medical, other, stem, social sciences\]\. The categories of MMLU\-ProX are \[biology, business, chemistry, computer science, economics, engineering, health, history, law, math, other, philosophy, physics, psychology\]

## Appendix FCross\-lingual Q&A Accuracy comparison

This section provides task accuracy results for the cross\-lingual evaluation discussed in RQ4 \(Section[4](https://arxiv.org/html/2607.06327#S4)\)\. In the cross\-lingual setting, the question and reasoning are in the target language, but each answer option is drawn from a different language\.

We compared the task accuracy between the standard multilingual setting and the cross\-lingual setting\. As expected, the cross\-lingual setting is more challenging: accuracy drops by 4–6 percentage points across all models across all the scales\. This decrease reflects the additional difficulty of matching reasoning to answer options expressed in different languages\.

Crucially, while task accuracy decreases, uncertainty estimation performance \(AUROC\) remains stable \(see Figure[7](https://arxiv.org/html/2607.06327#S4.F7)\)\. This indicates that UE methods successfully capture model confidence regardless of the cross\-lingual configuration, and do not rely solely on lexical matching between reasoning and answer options\.

Beyond uncertainty estimation, we also investigate whether English reasoning improves task accuracy\. Table[2](https://arxiv.org/html/2607.06327#A6.T2)reports a comparison between language\-specific reasoning and English reasoning in terms of Q&A accuracy, showing the absolute difference \(Δabs\\Delta\_\{\\text\{abs\}\}\) and the relative improvement \(Δrel%\\Delta\_\{\\text\{rel\}\}\\%\) across the 22 languages considered in our study, averaged across the largest models of each family evaluated on Global\-MMLU and MMLU\-ProX\.

The results indicate that English reasoning consistently improves task accuracy, with an average absolute gain of\+0\.037\+0\.037\(relative improvement of\+5\.4%\+5\.4\\%\)\. The improvement is most pronounced for lower\-resource languages such as Yoruba \(YO,\+17\.6%\+17\.6\\%\), Nepali \(NE,\+10\.7%\+10\.7\\%\), and Swahili \(SW,\+10\.5%\+10\.5\\%\), while higher\-resource languages such as French \(FR\), Portuguese \(PT\), and German \(DE\) exhibit smaller gains\. This suggests that English reasoning not only helps models better recognize their uncertainty but also enhances their ability to answer correctly, particularly in languages where the model’s native reasoning capabilities are weaker\.

Table 2:Q&A accuracy comparison between language\-specific reasoning and English reasoning, averaged across the largest models of each family on Global\-MMLU and MMLU\-ProX\.
## Appendix GDoes the quantization impact multilingual UE performances?

In Figure[8](https://arxiv.org/html/2607.06327#A7.F8), we compare the performance of full\-precision Qwen3\-30B against its 8\-bit and 4\-bit quantizations\. We selected Qwen3\-30B as it is the largest model in our study that fits in memory at full precision, making it the most relevant candidate for quantization\. As can be noticed, there is no statistically significant impact from quantizing the model\.

![Refer to caption](https://arxiv.org/html/2607.06327v1/img/qwen30b_quantization_auroc.png)Figure 8:Uncertainty estimation performance for Qwen3\-30B with quantization to 8\-bit and 4\-bit\. Quantization has minimal impact on both accuracy and uncertainty estimation at this model scale\. \(± indicates 95% CI\)\.
## Appendix HHardware Infrastructure

All experiments were conducted on an Amazon EC2 g6e\.48xlarge instance with an AMD EPYC 7R13 processor \(192 vCPUs\), 1\.5 TiB of system memory, and 8 NVIDIA L40S GPUs \(48GB VRAM each, 384GB total\)\.

## Appendix IComplete Multilingual UE Method Scaling Effect

In Figure[9](https://arxiv.org/html/2607.06327#A9.F9), we report the model size scaling factor with all the UE methods\.

![Refer to caption](https://arxiv.org/html/2607.06327v1/img/ue_methods_scaling_all_methods.png)Figure 9:Effect of model scale on UE performance \(Qwen3 family\)\. Shaded regions denote 95% CIs\. Self Verbalized improves with statistical significance at 235B\. Dashed line: random baseline\. This figure complements the Figure[4](https://arxiv.org/html/2607.06327#S4.F4)presented in the main where for simplicity we show only 4 methods\.
## Appendix JLabel Distribution Analysis for Positional Bias

To rule out positional bias in our evaluation, we analyzed the distribution of correct answer labels across the two MCQA benchmarks used in our study\. Tables[3](https://arxiv.org/html/2607.06327#A10.T3)and[4](https://arxiv.org/html/2607.06327#A10.T4)show that, for both Global MMLU \(4 options, expected uniform=25%=25\\%\) and MMLU\-ProX \(10 options, expected uniform=10%=10\\%\), all labels fall within approximately±2\\pm 2percentage points of the uniform expectation\. We therefore conclude that label distributions are sufficiently balanced to exclude positional bias as a confounding factor in the results reported in the main paper\.

Table 3:Label distribution for Global MMLU\.Table 4:Label distribution for MMLU\-ProX\.
## Appendix KCalibration Analysis

Table[9](https://arxiv.org/html/2607.06327#A9.F9)extends the AUROC results from the main paper \(Table[10](https://arxiv.org/html/2607.06327#A11.F10)\) with Expected Calibration Error \(ECE\), which measures the alignment between predicted confidence and actual accuracy\. Lower ECE indicates better calibration\.

![Refer to caption](https://arxiv.org/html/2607.06327v1/img/gemma_scaling_ece.png)Figure 10:Effect of model scale on ECE\. \(Mean across langs and 95% CIs\)\.