The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context

# The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context
Source: [https://arxiv.org/html/2607.12963](https://arxiv.org/html/2607.12963)
Yanzhe Zhang Georgia Tech z\_yanzhe@gatech\.edu &Sanmi Koyejo Stanford University sanmi@stanford\.edu &Diyi Yang Stanford University diyiy@stanford\.edu

###### Abstract

As large language models \(LLMs\) grow more capable, they are increasingly deployed in context\-rich settings where task inputs are often accompanied by long, partially irrelevant context\. In a controlled setting, we find that state\-of\-the\-art models often appear robust to task\-irrelevant context at the aggregate level: prepending it to benchmark questions causes little change in overall accuracy\. This aggregate stability, however, masks significant per\-example instability\. Even semantically meaningless pseudo\-words, formed by randomly combining characters, can markedly shift model predictions on a small fraction of examples, degrading performance on some while improving it on others\. This two\-sided effect holds consistently across a wide range of models and datasets, yet the affected examples are largely model\-specific\. We further show that this instability is modulated by context type, context length, test\-time compute, and model development stage\. Together, our findings reveal context\-induced tail risks concealed by aggregate accuracy, motivating per\-example reliability evaluation of language models\.111Code and data are available at[https://github\.com/SALT\-NLP/illusion\-of\-robustness](https://github.com/SALT-NLP/illusion-of-robustness)\.

## 1Introduction

State\-of\-the\-art large language models \(LLMs\) are increasingly deployed as agents operating in context\-rich environments, where they rarely receive a clean task input in isolation\. A coding agent\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.12963#bib.bib8); Yanget al\.,[2024](https://arxiv.org/html/2607.12963#bib.bib35)\)may need to inspect long source files, execution logs, and intermediate tool outputs before making a small code edit\. A deep research agent\(OpenAI,[2025](https://arxiv.org/html/2607.12963#bib.bib36); Zhenget al\.,[2025](https://arxiv.org/html/2607.12963#bib.bib40)\)may condition on noisy web retrieval results before answering a factual question\. A computer\-use agent\(Xieet al\.,[2024](https://arxiv.org/html/2607.12963#bib.bib9); Wanget al\.,[2026a](https://arxiv.org/html/2607.12963#bib.bib37)\)may process a long sequence of screenshots before deciding its next step\. In all of these scenarios, the task\-relevant signal is embedded within a much larger amount of task\-irrelevant context\.

In this work, we revisit the common assumption that irrelevant context should not substantially alter a model’s answer\. We use a simplified setting to isolate the effect of task\-irrelevant context: we prepend it to questions from QA benchmarks and measure the change in performance\.

At the aggregate level, state\-of\-the\-art models appear robust: benchmark\-level accuracy changes by no more than±0\.9%\\pm 0\.9\\%forgpt\-5\.5and±2\.1%\\pm 2\.1\\%forgpt\-5\.4after adding task\-irrelevant context\. However, this apparent robustness hides substantial per\-example instability, as the underlying phenomenon is two\-sided: performance drops sharply on some questions, while others become “easier” and show large gains, even though no additional task\-relevant information is provided\. We refer to such prediction flips ascontext\-induced instability\.

This instability matters in real\-world deployments because a shift at a single step may trigger an irreversible action or propagate through a sequence of downstream decisions\. To characterize the phenomenon, we introduce two complementary metrics\.*Instability \(INS\)*measures the average absolute change in performance across all examples, capturing how sensitive a model is to irrelevant context regardless of whether the change helps or hurts\.*Worst\-tail Degradation \(WTD\)*, in contrast, measures the average performance drop within the worst\-affected tail, capturing the risk of severe context\-induced failures\.

We observe this phenomenon across different models on multiple datasets \(MMLU\-Pro\(Wanget al\.,[2024](https://arxiv.org/html/2607.12963#bib.bib11)\), GPQA\(Reinet al\.,[2024](https://arxiv.org/html/2607.12963#bib.bib41)\), Humanity’s Last Exam \(HLE\)\(Phanet al\.,[2025](https://arxiv.org/html/2607.12963#bib.bib13)\), and SimpleQA\(Weiet al\.,[2024](https://arxiv.org/html/2607.12963#bib.bib12)\)\), where INS reaches13\.6%13\.6\\%and WTD reaches53\.2%53\.2\\%\. Although the phenomenon itself is consistent, the specific examples it affects are model\-specific rather than simply data\-dependent: per\-example performance shifts are poorly correlated across models \(e\.g\., on MMLU\-Pro, the mean Pearsonrris0\.000\.00\), and the top\-affected examples have limited overlap \(e\.g\., on MMLU\-Pro, the mean Jaccard index is0\.090\.09for both the10%10\\%most improved and10%10\\%most degraded examples\)\. Consistent with this model specificity, question ambiguity is poorly correlated with performance change\. By contrast, a model’s uncertainty on each question \(without context\) is meaningfully associated with the magnitude of the corresponding performance change, but not its direction\. Furthermore, we show that context\-induced instability persists across a range of context types, tends to increase with context length, can be partially mitigated by scaling test\-time compute, and varies substantially across model development stages\.

Together, these results identify task\-irrelevant context as a concrete source of hidden reliability risk\(Rabanseret al\.,[2026](https://arxiv.org/html/2607.12963#bib.bib57)\)in context\-rich LLM deployments\. Example\-level prediction flips occur in both directions, with improvements and degradations largely canceling out at the aggregate level\. Moreover, the affected examples are highly model\-specific, making it difficult to anticipate which examples will fail based on the data alone\. We therefore argue that context robustness evaluation, including per\-example instability and worst\-tail degradation, should be incorporated into model development to support more trustworthy downstream applications\.

![Refer to caption](https://arxiv.org/html/2607.12963v1/x1.png)Figure 1:Context\-induced instability is a two\-sided effect\.We show two MMLU\-Pro examples where task\-irrelevant context respectively improves and degradesgpt\-5\.4’s responses by affecting commonsense and calculation\. Full examples are in Appendix[I](https://arxiv.org/html/2607.12963#A9)\.
## 2Setup

In our main setup, we use pseudo\-words to create task\-irrelevant contexts\. Similar to\(Mohriet al\.,[2026](https://arxiv.org/html/2607.12963#bib.bib38)\), we first generate a fixed vocabulary of random lowercase strings over the English alphabet \(a\-z\), with lengths between 3 and 8 characters, and filter out strings whose English frequency exceeds a threshold\. For each context, we sampled pseudo\-words uniformly from this vocabulary and concatenated them with spaces\. This procedure produces contexts that resemble word\-like text at the surface level while minimizing recognizable lexical or factual content\. Note that pseudo\-word contexts are not intended to be fully naturalistic\. Rather, they provide the cleanest way to isolate the effect\. More naturalistic alternatives, such as randomly sampled tokens, webpages, and question\-answer pairs, are tested in our ablation study \(Section[3\.3](https://arxiv.org/html/2607.12963#S3.SS3)\)\.

LetD=\{d1,…,dN\}D=\\\{d\_\{1\},\\ldots,d\_\{N\}\\\}be the target evaluation dataset, where each exampledid\_\{i\}has reference answeryiy\_\{i\}, and letC=\{c1,…,cM\}C=\\\{c\_\{1\},\\ldots,c\_\{M\}\\\}be a set of task\-irrelevant contexts\. For a modelfLMf\_\{\\mathrm\{LM\}\}, we definesi​j=Pr⁡\(Ai​j=1∣di,cj,fLM\)s\_\{ij\}=\\Pr\(A\_\{ij\}=1\\mid d\_\{i\},c\_\{j\},f\_\{\\mathrm\{LM\}\}\)as the latent correctness probability of the model on exampledid\_\{i\}when conditioned on contextcjc\_\{j\}, whereAi​j∈\{0,1\}A\_\{ij\}\\in\\\{0,1\\\}indicates whether the model’s response is judged correct\. We usesi​0s\_\{i0\}for the corresponding no\-context setting\.

For a given exampledid\_\{i\}, the average context\-induced performance change isΔi=1M​∑j=1Msi​j−si​0\\Delta\_\{i\}=\\frac\{1\}\{M\}\\sum\_\{j=1\}^\{M\}s\_\{ij\}\-s\_\{i0\}, which lies in\[−1,1\]\[\-1,1\]\.Δi<0\\Delta\_\{i\}<0indicates that irrelevant context hurts performance on exampledid\_\{i\}, whileΔi\>0\\Delta\_\{i\}\>0indicates that irrelevant context improves performance\. Across all examples, the average performance change isΔ=1N​∑i=1NΔi\\Delta=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\Delta\_\{i\}\.

From a reliability perspective, a desirable model would show little change under contexts that contain no task\-relevant evidence\. Thus, we define theInstability \(INS\)as the average absolute change:

ℐ=1N​∑i=1N\|Δi\|,\\mathcal\{I\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\|\\Delta\_\{i\}\|,whereℐ∈\[0,1\]\\mathcal\{I\}\\in\[0,1\]\. While INS captures the two\-sided sensitivity of model predictions, the more concerning case is when a model that would otherwise answer correctly becomes incorrect after conditioning on irrelevant context\. To isolate the harmful effect, we defineWorst\-tail Degradation \(WTD\)as the average degradation in the worstK%K\\%of examples\. Letm=⌈K100​N⌉m=\\left\\lceil\\frac\{K\}\{100\}N\\right\\rceil, and letΔ\(1\)≤Δ\(2\)≤⋯≤Δ\(N\)\\Delta\_\{\(1\)\}\\leq\\Delta\_\{\(2\)\}\\leq\\cdots\\leq\\Delta\_\{\(N\)\}denote the sorted values of\{Δi\}i=1N\\\{\\Delta\_\{i\}\\\}\_\{i=1\}^\{N\}in ascending order\. We define:

𝒲K=−1m​∑ℓ=1mΔ\(ℓ\),\\mathcal\{W\}\_\{K\}=\-\\frac\{1\}\{m\}\\sum\_\{\\ell=1\}^\{m\}\\Delta\_\{\(\\ell\)\},where𝒲K∈\[−1,1\]\\mathcal\{W\}\_\{K\}\\in\[\-1,1\]and a larger𝒲K\\mathcal\{W\}\_\{K\}indicates more severe degradation\.

Score EstimationSincesi​js\_\{ij\}is not directly observed, we estimate it in different ways depending on the model access and question format\. For API\-access models \(our main setting\), we estimatesi​js\_\{ij\}by averaging binary correctness over repeated sampled responses\. For local models with multiple\-choice questions \(Appendix[G](https://arxiv.org/html/2607.12963#A7)\), an easier way to estimate it is to use the probability assigned to the correct answer option under a fixed answer template\. In both cases, the resulting estimates^i​j\\widehat\{s\}\_\{ij\}is treated as a measurement of the same underlying latent correctness probability\.

Noise CorrectionWhile calculatingℐ^\\widehat\{\\mathcal\{I\}\}and𝒲^K\\widehat\{\\mathcal\{W\}\}\_\{K\}froms^i​j\\widehat\{s\}\_\{ij\}for API\-access models, finite sampling can induce a positive noise floor for both metrics even when irrelevant context has no true effect, since INS uses absolute values and WTD selects the worstK%K\\%tail\. To estimate this sampling\-induced noise, we use a bootstrap procedure based only on the no\-context samples\. For each exampledid\_\{i\}, each bootstrap trial constructs a synthetic null experiment in which all conditions are sampled from the same no\-context distribution\. Specifically, we independently resample one pseudo\-baseline group andMMpseudo\-context groups from the same set\. Using the same definitions, each bootstrap trial givesℐ^∗\\widehat\{\\mathcal\{I\}\}^\{\*\}and𝒲^K∗\\widehat\{\\mathcal\{W\}\}\_\{K\}^\{\*\}\. We average over bootstrap trials to get estimated noise floorsηℐ\\eta\_\{\\mathcal\{I\}\}andη𝒲K\\eta\_\{\\mathcal\{W\}\_\{K\}\}, and report noise\-adjusted metrics by subtracting them from the observed metrics:ℐadj=ℐ^−ηℐ\\mathcal\{I\}\_\{\\mathrm\{adj\}\}=\\widehat\{\\mathcal\{I\}\}\-\\eta\_\{\\mathcal\{I\}\}and𝒲K,adj=𝒲^K−η𝒲K\\mathcal\{W\}\_\{K,\\mathrm\{adj\}\}=\\widehat\{\\mathcal\{W\}\}\_\{K\}\-\\eta\_\{\\mathcal\{W\}\_\{K\}\}\. Note that this correction targets the mean null noise floor\. For uncertainty, we consider the bootstrap variance of the observed metricsℐ^\\widehat\{\\mathcal\{I\}\}and𝒲^K\\widehat\{\\mathcal\{W\}\}\_\{K\}\. Our adjustment should be interpreted as a noise\-floor correction rather than an unbiased estimator of the context effect\. In Appendix[A](https://arxiv.org/html/2607.12963#A1), we provide a complementary split\-sample analysis of WTD, where separate samples are used to rank examples and to estimateΔi\\Delta\_\{i\}within the selected tail\.

MMLU\-ProGPQAHLESimpleQAAVGΔ\\DeltaINS↓\\downarrowWTD↓\\downarrowAVGΔ\\DeltaINS↓\\downarrowWTD↓\\downarrowAVGΔ\\DeltaINS↓\\downarrowWTD↓\\downarrowAVGΔ\\DeltaINS↓\\downarrowWTD↓\\downarrowgpt\-5\.580\.280\.2\+0\.3\+0\.32\.42\.410\.010\.063\.863\.8\+0\.6\+0\.64\.94\.912\.212\.212\.912\.9\+0\.0\+0\.01\.41\.44\.34\.350\.050\.0−0\.9\-0\.92\.42\.411\.011\.0gpt\-5\.481\.281\.2−2\.1\-2\.14\.64\.626\.226\.263\.163\.1−0\.8\-0\.86\.26\.221\.721\.711\.511\.5−1\.5\-1\.53\.13\.118\.818\.831\.031\.0\+0\.4\+0\.42\.32\.37\.47\.4gpt\-4\.178\.078\.0\+0\.0\+0\.04\.34\.319\.719\.757\.357\.3−0\.9\-0\.95\.65\.622\.722\.73\.83\.8−0\.2\-0\.21\.01\.05\.45\.430\.430\.4−0\.7\-0\.73\.13\.111\.811\.8DeepSeek\-V4\-Pro79\.079\.0\+0\.2\+0\.24\.84\.815\.915\.970\.070\.0−0\.2\-0\.23\.03\.07\.77\.77\.87\.8−0\.2\-0\.21\.21\.24\.24\.241\.841\.8−5\.9\-5\.96\.96\.927\.427\.4Grok\-4\.20\-NR67\.967\.9\+0\.1\+0\.15\.45\.421\.421\.454\.554\.5\+0\.6\+0\.66\.66\.622\.722\.79\.19\.1−1\.1\-1\.12\.52\.515\.615\.619\.519\.5−0\.4\-0\.44\.74\.720\.720\.7Mistral\-Large\-374\.974\.9−3\.6\-3\.68\.38\.347\.347\.350\.750\.7−3\.0\-3\.011\.311\.353\.253\.25\.15\.1\+0\.0\+0\.02\.72\.712\.212\.228\.728\.7−1\.8\-1\.89\.69\.651\.651\.6gpt\-oss\-120b77\.477\.4−0\.6\-0\.63\.63\.615\.315\.363\.463\.4−2\.1\-2\.11\.21\.28\.58\.54\.74\.7−0\.2\-0\.20\.80\.83\.33\.311\.111\.1−0\.5\-0\.51\.81\.89\.29\.2gemini\-3\.1\-fl81\.781\.7\+2\.3\+2\.35\.95\.917\.617\.657\.257\.2\+8\.5\+8\.513\.613\.621\.521\.56\.56\.5\+0\.7\+0\.73\.13\.110\.610\.634\.534\.5−2\.0\-2\.07\.07\.034\.434\.4

Table 1:Main results \(%\\%\)\.AVG refers to the average baseline accuracy,Δ\\Deltarefers to the average performance change after adding contexts, and INS and WTD are noise\-floor adjusted\. Models on the top use no reasoning by default, while models on the bottom use the lowest reasoning effort\.![Refer to caption](https://arxiv.org/html/2607.12963v1/distribution_split.png)Figure 2:Per\-example performance change concentrates in the tails \(MMLU\-Pro\)\.For each model, we rank its examples from most improved to most degraded by the per\-example changeΔi\\Delta\_\{i\}, and keep the two10%10\\%tails: the most\-improved examples \(top panel\) and the most\-degraded examples \(bottom panel\)\. Each tail is split into2%2\\%\-wide percentile bins \(x\-axis\), and each bar reports the meanΔi\\Delta\_\{i\}in that bin for one model\. Solid bars show the observed change; the overlaid hatched bars show the corresponding estimated noise floor\.Table 2:Qualitative analysis of performance changes\.For questions whose accuracy changes by more than20%20\\%after adding task\-irrelevant context, we annotate the behavioral differences between the no\-context and with\-context settings using five pairwise comparisons per question\. The table reports the distribution \(%\\%\) of annotated error types across pairs\. For improved questions, the categories indicate the errors corrected by the added context; for degraded questions, they indicate the errors introduced\.
## 3Experiments

Experiment DetailsWe use the following four datasets: MMLU\-Pro\(Wanget al\.,[2024](https://arxiv.org/html/2607.12963#bib.bib11)\), GPQA\-Diamond\(Reinet al\.,[2024](https://arxiv.org/html/2607.12963#bib.bib41)\), Humanity’s Last Exam\(Phanet al\.,[2025](https://arxiv.org/html/2607.12963#bib.bib13)\), and SimpleQA Verified\(Weiet al\.,[2024](https://arxiv.org/html/2607.12963#bib.bib12); Haaset al\.,[2026](https://arxiv.org/html/2607.12963#bib.bib42)\)\. The first two datasets contain multiple\-choice questions, while the latter two contain open\-ended questions\. For each dataset, we randomly sample up to500500questions\. We createM=10M=10pseudo\-word contexts, each with10​k10ktokens by default \(21129\.921129\.9characters on average\)\. For consistency, we use token counts from thegpt\-oss\-120btokenizer to create contexts in this work\. Contexts are prepended to the question as an extra user message between the question and the system message\. We useK=10K=10for the WTD metric\. By default, we use API\-access models and report the adjusted version of INS and WTD, while providing extra results for local models in the ablation study and Appendix[G](https://arxiv.org/html/2607.12963#A7)\. For each question in the experiments, we use a temperature of1\.01\.0to samplep=20p=20times for the no\-context baseline andq=2q=2times for each context, so that both conditions have 20 trials in total\. We report results forgpt\-5\.5,gpt\-5\.4,gpt\-4\.1,DeepSeek\-V4\-Pro,Grok\-4\.20\-NR\(Grok 4\.20 non reasoning\),Mistral\-Large\-3,gpt\-oss\-120b, andgemini\-3\.1\-fl\(gemini\-3\.1\-flash\-lite\)\. By default, we use no reasoning setup to rule out the effect of reasoning tokens\. For models that cannot turn off reasoning, we use the lowest possible reasoning effort \(loworminimal\)\. We usegpt\-5\.4\-miniwithlowreasoning effort to grade all responses\. Extra experiment details are in Appendix[B](https://arxiv.org/html/2607.12963#A2)and prompt details are in Appendix[H](https://arxiv.org/html/2607.12963#A8)\.

### 3\.1Main results

We show the main result in Table[1](https://arxiv.org/html/2607.12963#S2.T1)\(unadjusted metrics, confidence intervals, and noise floors are shown in Appendix Table[6](https://arxiv.org/html/2607.12963#A2.T6)\)\. With the added pseudo\-word contexts, the average performance changeΔ\\Deltausually remains close to zero across models and benchmarks\. However, the INS metrics are usually much larger \(reaching13\.6%13\.6\\%\), suggesting that the aggregated stability is not due to per\-example stability but rather to two\-sided effects that cancel out\. If we take a closer look at the per\-example distribution \(Figure[2](https://arxiv.org/html/2607.12963#S2.F2)\), we find that the instability mostly comes from the top10%10\\%tail on both sides, accounting for88\.3%88\.3\\%of the total absolute change\. For the10%10\\%examples most negatively impacted, WTD indicates that the average performance degradation can be as large as53\.2%53\.2\\%\.

Among the evaluated models without reasoning,gpt\-5\.5is the most consistently stable across all four benchmarks\. Its INS is at most4\.9%4\.9\\%, and its WTD is at most12\.2%12\.2\\%\. Note that low INS or WTD should be interpreted together with the baseline accuracy\. For example,gpt\-4\.1has low INS and WTD on HLE, but its baseline AVG is only3\.8%3\.8\\%\. In this case, there is limited room for performance changes, so small INS can partly reflect a floor effect rather than genuine robustness\. More importantly, it is possible to achieve both strong baseline accuracy and low context sensitivity\. On SimpleQA, for instance,gpt\-5\.5reaches50\.0%50\.0\\%AVG with only2\.4%2\.4\\%INS and11\.0%11\.0\\%WTD, whereas several models with substantially lower AVG exhibit larger INS and WTD\.

Improvement vs\. DegradationModels can have similar INS but very different directional behavior\. For example, bothMistral\-Large\-3andgemini\-3\.1\-flexhibit large INS across several settings, but their instability manifests differently\.Mistral\-Large\-3tends to exhibit degradation, leading to particularly high WTD values:47\.3%47\.3\\%on MMLU\-Pro,53\.2%53\.2\\%on GPQA, and51\.6%51\.6\\%on SimpleQA\. In contrast,gemini\-3\.1\-floften shows relatively positive shifts, especially on MMLU\-Pro and GPQA, whereΔ\\Deltais\+2\.3%\+2\.3\\%and\+8\.5%\+8\.5\\%, respectively\. In Figure[2](https://arxiv.org/html/2607.12963#S2.F2),gemini\-3\.1\-flhas larger bars in the improved tail, whileMistral\-Large\-3is more prominent in the degraded tail\. Thus, high INS alone does not indicate whether the irrelevant context is mostly helpful or harmful; WTD is necessary to isolate the harmful tail\.

Qualitative AnalysisTo better understand context\-induced instability, we use LLM\-based annotations to analyze questions whose accuracy changes by more than20%20\\%after adding task\-irrelevant context forgpt\-5\.4\(details in Appendix[C](https://arxiv.org/html/2607.12963#A3)\)\. Table[2](https://arxiv.org/html/2607.12963#S2.T2)summarizes the types of errors that are corrected in improved examples and introduced in degraded examples\. Overall, the dominant error types reflect the nature of each benchmark\. On MMLU\-Pro, performance changes are primarily associated with calculation errors, including arithmetic mistakes, bookkeeping errors, and miscounts\. On HLE, changes are more often linked to the application of incorrect or overgeneralized rules, principles, or concepts\. On SimpleQA, nearly all changes correspond to factual hallucinations, consistent with its fact\-seeking format\. Importantly, improved and degraded examples do not follow identical error distributions\. In MMLU\-Pro and GPQA, degradation is disproportionately associated with newly introduced calculation errors\. In contrast, solving the wrong problem is more common among improved examples in MMLU\-Pro, GPQA, and HLE\. These results suggest that irrelevant context can alter the model’s predictions in qualitatively different ways\.

![Refer to caption](https://arxiv.org/html/2607.12963v1/correlation_heatmaps.png)Figure 3:Per\-example changes are uncorrelated across models\.We report Pearsonrrof per\-example performance change for all model pairs on MMLU\-Pro and SimpleQA\.
### 3\.2Model Specificity

When we examine per\-example changes\[Δ1,Δ2,⋯,ΔN\]\[\\Delta\_\{1\},\\Delta\_\{2\},\\cdots,\\Delta\_\{N\}\], there is almost no correlation across models, even among those with similar AVG/INS/WTD in Table[1](https://arxiv.org/html/2607.12963#S2.T1)\. On all four datasets, we plot the Pearson correlation between all models in Figure[3](https://arxiv.org/html/2607.12963#S3.F3)\(and Appendix Figure[7](https://arxiv.org/html/2607.12963#A2.F7)\)\. The mean Pearson correlation is0\.000\.00\(95%95\\%CI:\[−0\.02,0\.02\]\[\-0\.02,0\.02\]\) on MMLU\-Pro and similarly0\.000\.00\(95%95\\%CI:\[−0\.02,0\.01\]\[\-0\.02,0\.01\]\) on SimpleQA\. If we take a closer look at the correlations between model pairs, there is no noticeable positive correlation between any two models across these two datasets\. We also calculate the Jaccard overlap of the top10%10\\%improved/degraded examples\. On average, the Jaccard overlap is0\.090\.09\(95%95\\%CI:\[0\.08,0\.11\]\[0\.08,0\.11\]\) for improved examples and0\.090\.09\(95%95\\%CI:\[0\.08,0\.11\]\[0\.08,0\.11\]\) for degraded examples on MMLU\-Pro, which is close to the0\.050\.05random baseline\.

Table 3:Correlates of context\-induced performance change\.Spearmanρ\\rhowith 95% question\-bootstrap CIs over alln=1690n\{=\}1690questions\. We correlate question ambiguity and baseline uncertainty with the signed changeΔi\\Delta\_\{i\}\(direction\) and squared changeΔi2\\Delta\_\{i\}^\{2\}\(magnitude\)\. Raw estimates use all trials, while split\-sample estimates use disjoint trials for the feature and target\. We report both probability\-scale and smoothed log\-odds\-scale results\. Estimator details are in Appendix[F](https://arxiv.org/html/2607.12963#A6)\.Correlates of Context\-Induced ChangeFor each question, we measure ambiguity by asking three judge models to flag ambiguity or underspecification and summing their positive flags into a score in\{0,1,2,3\}\\\{0,1,2,3\\\}\(Appendix[E](https://arxiv.org/html/2607.12963#A5)\)\. We use the estimated Bernoulli variance of no\-context correctness,s^i​0​\(1−s^i​0\)\\widehat\{s\}\_\{i0\}\(1\-\\widehat\{s\}\_\{i0\}\), as a proxy for baseline predictive uncertainty\. Table[3](https://arxiv.org/html/2607.12963#S3.T3)correlates these features with the signed changeΔi\\Delta\_\{i\}, which captures whether context helps or hurts, and the squared changeΔi2\\Delta\_\{i\}^\{2\}, which captures its magnitude\. We report raw estimates together with split\-sample estimates that reduce shared\-trial noise on both the probability and log\-odds scales\. Ambiguity is consistently uncorrelated with either the direction or magnitude of change\. Baseline uncertainty is also nearly uncorrelated with direction\. Its association with magnitude is strong in the raw estimates and remains modest after split\-sampling \(ρ=0\.21\\rho\{=\}0\.21\)\. Importantly, the same association remains on the log\-odds scale, suggesting that it is not explained solely by uncertain questions having more room to change on the probability scale\. Intuitively, a model’s internal uncertainty helps identify questions that are more likely to change, but not whether the change will be beneficial or harmful\.

![Refer to caption](https://arxiv.org/html/2607.12963v1/instability_reliability_corr.png)Figure 4:Context\-induced instability holds across context types\.We show INS, WTD, and the correlation of per\-example performance change between different context types\. SQA refers to sequential question answering, WEB refers to random webpages, TOK refers to random tokens, and CHA refers to pseudo\-words\. All results are based ongpt\-5\.4and MMLU\-Pro\.![Refer to caption](https://arxiv.org/html/2607.12963v1/context_and_reasoning_scaling_one_row.png)Figure 5:Scaling effects of context length and reasoning effort\.We vary context length \(1010,100100,1​k1k, and10​k10k\) and reasoning effort \(none,low,medium,high\) on MMLU\-Pro\.
### 3\.3Ablation Study

Context TypeBeyond pseudo\-word contexts constructed by random characters \(CHA\), we further evaluate three more naturalistic types of task\-irrelevant context\. First,Random Tokens\(TOK\) are constructed by randomly sampling tokens from thegpt\-oss\-120btokenizer vocabulary\. Second,Random Webpages\(WEB\) are formed by randomly sampling and concatenating webpages fromHuggingFaceFW/fineweb\-edu\. Third,Sequential Question Answering\(SQA\) consists of multi\-turn conversation histories in which the model answers other randomly sampled questions from the same dataset\. For TOK and WEB, each context contains10​k10ktokens; for SQA, we use2020turns by default\. Unlike pseudo\-word contexts, these settings contain real words, webpages, or questions, and may therefore include superficially or semantically related information by chance\. We use them as a sanity check to test whether context\-induced instability also appears under more naturalistic forms of task\-irrelevant context, and to measure whether the affected questions are consistent\.

As shown in Figure[4](https://arxiv.org/html/2607.12963#S3.F4), all context types induce both INS and WTD, although the magnitudes vary across datasets and context types\. In both datasets, SQA yields the highest INS, likely because its question\-answer format most closely resembles the target task and therefore more strongly perturbs the model’s prediction\. The default pseudo\-word context yields the highest WTD on MMLU\-Pro but the lowest WTD on SimpleQA, suggesting that the same context type can have different relative effects depending on the benchmark\. Despite these differences in magnitude, the question\-level effects are positively correlated across context types, with mean Pearson correlations of0\.370\.37\(95%95\\%CI:\[0\.28,0\.46\]\[0\.28,0\.46\]\) on MMLU\-Pro and0\.240\.24\(95%95\\%CI:\[0\.16,0\.33\]\[0\.16,0\.33\]\) on SimpleQA\. This indicates that while the magnitude of instability depends on the form of irrelevant context, the questions affected by different context types partially overlap\. We report results with local models in Appendix[G](https://arxiv.org/html/2607.12963#A7)\.

Context LengthFigure[5](https://arxiv.org/html/2607.12963#S3.F5)\(Left\) studies how context length affects INS and WTD, using pseudo\-word contexts of length1010,100100,1​k1kand10​k10ktokens\. Surprisingly, even very short contexts of only1010pseudo\-word tokens induce nontrivial INS and WTD\. Increasing context length generally amplifies both effects, although the trend saturates at different lengths for different models\. For example, the WTD ofgpt\-5\.4stops increasing after1​k1kcontext tokens, whereas the WTD ofgpt\-5\.5saturates after100100context tokens\. We further find that the types of errors induced by short contexts are similar to those induced by long contexts\. For instance, with only1010pseudo\-word tokens, around60%60\\%of the newly introduced errors forgpt\-5\.4are still wrong\-calculation errors, comparable to the distribution observed with10​k10ktokens\. This suggests that context length primarily affects the extent to which questions are impacted, rather than fundamentally changing the qualitative failure modes\.

Test\-time ComputeFirst, we examine the effect of increasing reasoning effort for reasoning models in Figure[5](https://arxiv.org/html/2607.12963#S3.F5)\(Right\) with details in Appendix Table[8](https://arxiv.org/html/2607.12963#A2.T8)\. Forgpt\-5\.4, increasing reasoning effort fromnonetolowsharply reduces both INS and WTD, largely closing the gap withgpt\-5\.5\. However, additional reasoning does not monotonically reduce WTD\. For example, increasing reasoning effort frommediumtohighdoes not decrease WTD for eithergpt\-5\.4orgpt\-5\.5\. As a result, it does not fully eliminate the issue, and significantly degraded examples still exist\. A qualitative inspection ofgpt\-5\.4withhighreasoning effort shows that wrong calculation errors are largely eliminated, but solving the wrong problem and applying the wrong rule remain common failure modes\. Second, we apply Chain\-of\-Thought \(CoT\) prompting\(Weiet al\.,[2022](https://arxiv.org/html/2607.12963#bib.bib55)\)for a non\-reasoning modelgpt\-4\.1\(Full results in Appendix Table[9](https://arxiv.org/html/2607.12963#A8.T9)\)\. Similar to increasing reasoning effort, CoT prompting also reduces both INS and WTD across all datasets, although the degree of mitigation varies substantially: INS decreases from4\.3%4\.3\\%to1\.3%1\.3\\%and WTD from19\.7%19\.7\\%to6\.4%6\.4\\%on MMLU\-Pro\. On SimpleQA, INS decreases from3\.1%3\.1\\%to1\.7%1\.7\\%and WTD from11\.8%11\.8\\%to7\.6%7\.6\\%\. The remaining WTD is still relatively high on SimpleQA, suggesting that factual errors may be harder to recover even with a more explicit thinking process\. Overall, these results indicate that spending additional tokens at test time can mitigate context\-induced instability, but it does not provide a complete solution\.

![Refer to caption](https://arxiv.org/html/2607.12963v1/fullpipeline.png)

![Refer to caption](https://arxiv.org/html/2607.12963v1/pretraining.png)

![Refer to caption](https://arxiv.org/html/2607.12963v1/midtraining.png)

Figure 6:Context\-induced instability emerges across the training pipeline\.We track INS and WTD on MMLU forOlmo\-3\-7B\-Instructat checkpoints \(I\) after each training stage, \(II\) during pretraining, and \(III\) during midtraining\.Table 4:INS and WTD for local models on MMLU\.Left: comparison between base and post\-trained models\. Right: OLMo\-3 checkpoints with different midtraining data mixtures\.Model Development StageTo understand how context\-induced instability evolves during training, we evaluate local models on the original MMLU benchmark\(Hendryckset al\.,[2021](https://arxiv.org/html/2607.12963#bib.bib43)\), focusing on the OLMo\-3 family\(Olmoet al\.,[2026](https://arxiv.org/html/2607.12963#bib.bib44)\), which releases intermediate checkpoints throughout training\. Figure[6](https://arxiv.org/html/2607.12963#S3.F6)tracks no\-context performance \(AVG\), INS, and WTD across the development stages ofOlmo\-3\-7B\-Instruct: pretraining \(PRE\), midtraining \(MID\), context extension \(CXT\), supervised finetuning \(SFT\), direct preference optimization \(DPO\), and reinforcement learning from verifiable rewards \(RLVR\)\.\(I\)We find that INS and WTD gradually emerge during pretraining\. As task performance improves, the model also becomes increasingly sensitive to task\-irrelevant context\. During midtraining, both INS and WTD initially increase sharply, possibly reflecting a distribution shift in the training data, before gradually stabilizing at lower values, as task performance continues to improve\. Later stages exhibit different patterns\. SFT changes baseline performance only modestly, but substantially increases instability\. The subsequent DPO stage further improves task performance while also amplifying INS and WTD\. We observe a similar post\-training effect in Table[4](https://arxiv.org/html/2607.12963#S3.T4): bothQwen3\.5\-9Bandgemma\-4\-12B\-itexhibit higher INS and WTD than their corresponding base models\. These trends suggest that context\-induced instability is shaped throughout all stages\. In particular, improvements in benchmark performance do not necessarily imply improved robustness to irrelevant context; in some stages, especially post\-training, performance gains coincide with increased instability\.\(II\)We next examine whether instability is associated with the choice of data mixture for the same stage\. We analyze three OLMo\-3 midtraining checkpoints that start from the same 2T\-token pretrained checkpoint but use different data mixtures\. We find that midtraining on Generative/QA\-domain data is associated with higher INS and WTD than midtraining on code/math/thinking\-domain data, even though both mixtures substantially improve AVG performance\. Interestingly, the final OLMo\-3 mixture, which more evenly balances Generative/QA and code/math/thinking data, leads to the highest instability among the three\. While these comparisons are observational, they suggest that the composition of training data may also influence context\-induced instability\.

## 4Related work

Robustness to Prompt PerturbationsFor classic NLP tasks, CheckList\(Ribeiroet al\.,[2020](https://arxiv.org/html/2607.12963#bib.bib47)\)introduces invariance\-style tests for perturbations that should not affect model predictions\. For LLMs, a natural expectation is that predictions should be stable under prompt wording, formatting, and other semantically preserving variations, but prior work shows that this expectation often fails\.Sclaret al\.\([2024](https://arxiv.org/html/2607.12963#bib.bib48)\)show that meaning\-preserving prompt\-format changes can shift accuracy by up to76%76\\%\.Mizrahiet al\.\([2024](https://arxiv.org/html/2607.12963#bib.bib49)\)argue that single\-prompt evaluation is brittle and that models should be evaluated with diverse prompts\.Zhuoet al\.\([2024](https://arxiv.org/html/2607.12963#bib.bib50)\)provide instance\-level analyses of prompt sensitivity and find that robustness improves with more few\-shot examples and higher model confidence\.Salinas and Morstatter \([2024](https://arxiv.org/html/2607.12963#bib.bib51)\)further show that even minor formatting changes, such as adding a trailing space, can change an LLM’s answer\. In this work, the same invariance expectation extends beyond the prompt template to the input itself: a model’s per\-example performance should not change substantially when task\-irrelevant context is added\.

Robustness to Irrelevant ContextWhereas one line of work asks whether models can retrieve*relevant*evidence from the context, where performance depends strongly on where evidence appears\(Liuet al\.,[2024](https://arxiv.org/html/2607.12963#bib.bib6)\)and degrades as context length and task complexity increase\(Kamradt,[2023](https://arxiv.org/html/2607.12963#bib.bib25); Hsiehet al\.,[2024](https://arxiv.org/html/2607.12963#bib.bib26)\), a complementary line studies the effect of*irrelevant*context\.Shiet al\.\([2023](https://arxiv.org/html/2607.12963#bib.bib28)\)show that irrelevant information can substantially degrade arithmetic reasoning, whileYoranet al\.\([2024](https://arxiv.org/html/2607.12963#bib.bib29)\)show that retrieval\-augmented models can be harmed by irrelevant passages unless explicitly trained to handle them\. More recently,Wuet al\.\([2024](https://arxiv.org/html/2607.12963#bib.bib46)\)propose focused learning to reduce distraction in long\-context QA,Yanget al\.\([2025](https://arxiv.org/html/2607.12963#bib.bib45)\)introduce a controlled benchmark for distracting context in mathematical reasoning, andWanget al\.\([2026b](https://arxiv.org/html/2607.12963#bib.bib34)\)use automated tree search to generate adaptive distractions that strongly reduce performance\. Beyond strategically designed distractors,Levyet al\.\([2024](https://arxiv.org/html/2607.12963#bib.bib60)\)andDuet al\.\([2025](https://arxiv.org/html/2607.12963#bib.bib61)\)show that irrelevant filler text and even whitespace can degrade model performance, andRodionovet al\.\([2026](https://arxiv.org/html/2607.12963#bib.bib62)\)recently show that lengthy irrelevant context can shorten the reasoning trace and degrade performance\. Conversely,Shiet al\.\([2025](https://arxiv.org/html/2607.12963#bib.bib63)\)show that inserting a carefully tuned number of repeated meaningless tokens can improve performance\. However, each of these works examines only a limited set of datasets, models, and context types, and thus does not provide a comprehensive picture\. In this work, with broad coverage along these axes, we reveal a general phenomenon of context\-induced instability: even when aggregate accuracy remains stable, individual examples can undergo large, model\-specific performance shifts in either direction\.

Mechanisms of Context SensitivityA growing body of work seeks to explain the context\-related behavior of LLMs\.Wuet al\.\([2025](https://arxiv.org/html/2607.12963#bib.bib58)\)identify attention heads responsible for retrieving information from the context, andZhuet al\.\([2025](https://arxiv.org/html/2607.12963#bib.bib59)\)label relevant context spans and learn “focus” directions to mitigate distraction\. Closest to our setting,Niuet al\.\([2025](https://arxiv.org/html/2607.12963#bib.bib64)\)describe contextual entrainment: models assign higher probability to tokens that previously appeared in the context, regardless of their semantic relevance, and attribute the effect to a mechanistic token\-copying bias\.Kukrejaet al\.\([2026](https://arxiv.org/html/2607.12963#bib.bib65)\)further report that for non\-semantic context, this entrainment increases with model scale\. This work instead uses pseudo\-word context, which further minimizes lexical overlap between the context and the question or answer\. Even so, it still induces performance shifts on state\-of\-the\-art models and challenging benchmarks, suggesting that an overlap\-driven copying bias alone does not account for the effect\. We hope it offers a new perspective for understanding and improving context robustness\.

## 5Conclusion

In this work, we study context\-induced instability in LLMs by adding task\-irrelevant context to standalone benchmark questions\. Across models and datasets, we find that such context can substantially change per\-example behavior, degrading performance on some questions while improving it on others\. These affected examples are largely model\-specific, making them difficult to predict from the data alone\. We further find that context\-induced instability evolves across model development stages and can increase alongside benchmark performance, while additional test\-time reasoning mitigates but does not eliminate it\. Building on these findings, future work could explore several directions:

- •Improving measurement efficiency\.Reliably estimating tail metrics such as WTD currently requires repeated sampling over large datasets to ensure that the worst\-K%K\\%tail contains enough examples, resulting in substantial computational cost\.
- •Understanding the sources of instability\.Explaining why task\-irrelevant context destabilizes particular examples, from both training\-data and mechanistic interpretability perspectives, could make training and inference methods more robust\.
- •Diagnosing agent failures\.A deeper understanding of end\-to\-end agent failures could be achieved by more carefully isolating the effects of context and quantifying how instability at individual steps accumulates across multi\-step trajectories\.

## Acknowledgment

We would like to thank all SALT Lab members and friends for their feedback\. This work is supported by Open Philanthropy, Schmidt Sciences, and a grant under ONR N00014\-24\-1\-2532\.

## References

- Y\. Du, M\. Tian, S\. Ronanki, S\. Rongali, S\. B\. Bodapati, A\. Galstyan, A\. Wells, R\. Schwartz, E\. A\. Huerta, and H\. Peng \(2025\)Context length alone hurts LLM performance despite perfect retrieval\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 23281–23298\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.1264/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.1264),ISBN 979\-8\-89176\-335\-7Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p2.1)\.
- SimpleQA verified: a reliable factuality benchmark to measure parametric knowledge\.External Links:2509\.07968,[Link](https://arxiv.org/abs/2509.07968)Cited by:[§3](https://arxiv.org/html/2607.12963#S3.p1.8)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2021\)Measuring massive multitask language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d7KBjmI3GmQ)Cited by:[§3\.3](https://arxiv.org/html/2607.12963#S3.SS3.p5.1)\.
- C\. Hsieh, S\. Sun, S\. Kriman, S\. Acharya, D\. Rekesh, F\. Jia, and B\. Ginsburg \(2024\)RULER: what’s the real context size of your long\-context language models?\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=kIoBbc76Sy)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p2.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world github issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§1](https://arxiv.org/html/2607.12963#S1.p1.1)\.
- G\. Kamradt \(2023\)Needle in a haystack: pressure testing llms\.Note:[https://github\.com/gkamradt/LLMTest\_NeedleInAHaystack](https://github.com/gkamradt/LLMTest_NeedleInAHaystack)GitHub repositoryCited by:[§4](https://arxiv.org/html/2607.12963#S4.p2.1)\.
- D\. Kukreja, K\. Sah, G\. Gupta, A\. Anand, R\. R\. Shah, Z\. Wang, A\. B\. Ng, and E\. Cambria \(2026\)Better and worse with scale: how contextual entrainment diverges with model size\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 30194–30209\.External Links:[Link](https://aclanthology.org/2026.findings-acl.1509/),[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.1509),ISBN 979\-8\-89176\-395\-1Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p3.1)\.
- M\. Levy, A\. Jacoby, and Y\. Goldberg \(2024\)Same task, more tokens: the impact of input length on the reasoning performance of large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 15339–15353\.External Links:[Link](https://aclanthology.org/2024.acl-long.818/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.818)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p2.1)\.
- N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Link](https://aclanthology.org/2024.tacl-1.9/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p2.1)\.
- M\. Mizrahi, G\. Kaplan, D\. Malkin, R\. Dror, D\. Shahaf, and G\. Stanovsky \(2024\)State of what art? a call for multi\-prompt LLM evaluation\.Transactions of the Association for Computational Linguistics12,pp\. 933–949\.External Links:[Link](https://aclanthology.org/2024.tacl-1.52/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00681)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p1.1)\.
- C\. Mohri, J\. Duchi, and T\. Hashimoto \(2026\)A bitter lesson for data filtering\.External Links:2605\.19407,[Link](https://arxiv.org/abs/2605.19407)Cited by:[§2](https://arxiv.org/html/2607.12963#S2.p1.1)\.
- J\. Niu, X\. Yuan, T\. Wang, H\. Saghir, and A\. H\. Abdi \(2025\)Llama see, llama do: a mechanistic perspective on contextual entrainment and distraction in LLMs\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 16218–16239\.External Links:[Link](https://aclanthology.org/2025.acl-long.791/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.791),ISBN 979\-8\-89176\-251\-0Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p3.1)\.
- T\. Olmo, :, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison, J\. Morrison, J\. Poznanski, K\. Lo, L\. Soldaini, M\. Jordan, M\. Chen, M\. Noukhovitch, N\. Lambert, P\. Walsh, P\. Dasigi, R\. Berry, S\. Malik, S\. Shah, S\. Geng, S\. Arora, S\. Gupta, T\. Anderson, T\. Xiao, T\. Murray, T\. Romero, V\. Graf, A\. Asai, A\. Bhagia, A\. Wettig, A\. Liu, A\. Rangapur, C\. Anastasiades, C\. Huang, D\. Schwenk, H\. Trivedi, I\. Magnusson, J\. Lochner, J\. Liu, L\. J\. V\. Miranda, M\. Sap, M\. Morgan, M\. Schmitz, M\. Guerquin, M\. Wilson, R\. Huff, R\. L\. Bras, R\. Xin, R\. Shao, S\. Skjonsberg, S\. Z\. Shen, S\. S\. Li, T\. Wilde, V\. Pyatkin, W\. Merrill, Y\. Chang, Y\. Gu, Z\. Zeng, A\. Sabharwal, L\. Zettlemoyer, P\. W\. Koh, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi \(2026\)Olmo 3\.External Links:2512\.13961,[Link](https://arxiv.org/abs/2512.13961)Cited by:[§3\.3](https://arxiv.org/html/2607.12963#S3.SS3.p5.1)\.
- OpenAI \(2025\)Deep research system card\.Note:Accessed: June 2026External Links:[Link](https://cdn.openai.com/deep-research-system-card.pdf)Cited by:[§1](https://arxiv.org/html/2607.12963#S1.p1.1)\.
- L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi,et al\.\(2025\)Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.Cited by:[§1](https://arxiv.org/html/2607.12963#S1.p5.7),[§3](https://arxiv.org/html/2607.12963#S3.p1.8)\.
- S\. Rabanser, S\. Kapoor, P\. Kirgis, K\. Liu, S\. Utpala, and A\. Narayanan \(2026\)Towards a science of AI agent reliability\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=4LTwRqhmhk)Cited by:[§1](https://arxiv.org/html/2607.12963#S1.p6.1)\.
- D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman \(2024\)GPQA: a graduate\-level google\-proof q&a benchmark\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=Ti67584b98)Cited by:[§1](https://arxiv.org/html/2607.12963#S1.p5.7),[§3](https://arxiv.org/html/2607.12963#S3.p1.8)\.
- M\. T\. Ribeiro, T\. Wu, C\. Guestrin, and S\. Singh \(2020\)Beyond accuracy: behavioral testing of NLP models with CheckList\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 4902–4912\.External Links:[Link](https://aclanthology.org/2020.acl-main.442/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.442)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p1.1)\.
- G\. Rodionov, R\. Garipov, and G\. Yakushev \(2026\)Reasoning shift: how context silently shortens llm reasoning\.External Links:2604\.01161,[Link](https://arxiv.org/abs/2604.01161)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p2.1)\.
- A\. Salinas and F\. Morstatter \(2024\)The butterfly effect of altering prompts: how small changes and jailbreaks affect large language model performance\.InFindings of the Association for Computational Linguistics: ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 4629–4651\.External Links:[Link](https://aclanthology.org/2024.findings-acl.275/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.275)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p1.1)\.
- M\. Sclar, Y\. Choi, Y\. Tsvetkov, and A\. Suhr \(2024\)Quantifying language models’ sensitivity to spurious features in prompt design or: how i learned to start worrying about prompt formatting\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=RIu5lyNXjT)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p1.1)\.
- F\. Shi, X\. Chen, K\. Misra, N\. Scales, D\. Dohan, E\. H\. Chi, N\. Schärli, and D\. Zhou \(2023\)Large language models can be easily distracted by irrelevant context\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 31210–31227\.External Links:[Link](https://proceedings.mlr.press/v202/shi23a.html)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p2.1)\.
- Z\. Shi, Y\. Wan, Z\. Wang, Q\. Wang, F\. Yang, E\. Kreiss, and R\. Tang \(2025\)Meaningless tokens, meaningful gains: how activation shifts enhance llm reasoning\.External Links:2510\.01032,[Link](https://arxiv.org/abs/2510.01032)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p2.1)\.
- X\. Wang, B\. Wang, D\. Lu, J\. Yang, T\. Xie, J\. Wang, J\. Deng, X\. Guo, Y\. Xu, C\. H\. Wu, Z\. Shen, Z\. Li, R\. Li, X\. Li, J\. Chen, Z\. Boyuan, L\. PEIHANG, F\. Lei, R\. Cao, Y\. Fu, D\. Shin, M\. Shin, H\. Jiarui, Y\. Wang, J\. Chen, Y\. Ye, D\. Zhang, Y\. Wang, H\. Wang, D\. Yang, V\. Zhong, Y\. Charles, Z\. Yang, and T\. Yu \(2026a\)OpenCUA: open foundations for computer\-use agents\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=6iRZvJiC9Q)Cited by:[§1](https://arxiv.org/html/2607.12963#S1.p1.1)\.
- Y\. Wang, Z\. Xu, Y\. Huang, C\. Gao, S\. Wu, J\. Ye, P\. Chen, X\. Chen, and X\. Zhang \(2026b\)Adaptive distraction: probing LLM contextual robustness with automated tree search\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=sWIbJWiEuA)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p2.1)\.
- Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen \(2024\)MMLU\-pro: a more robust and challenging multi\-task language understanding benchmark\.InThe Thirty\-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=y10DM6R2r3)Cited by:[§1](https://arxiv.org/html/2607.12963#S1.p5.7),[§3](https://arxiv.org/html/2607.12963#S3.p1.8)\.
- J\. Wei, N\. Karina, H\. W\. Chung, Y\. J\. Jiao, S\. Papay, A\. Glaese, J\. Schulman, and W\. Fedus \(2024\)Measuring short\-form factuality in large language models\.External Links:2411\.04368,[Link](https://arxiv.org/abs/2411.04368)Cited by:[§1](https://arxiv.org/html/2607.12963#S1.p5.7),[§3](https://arxiv.org/html/2607.12963#S3.p1.8)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§3\.3](https://arxiv.org/html/2607.12963#S3.SS3.p4.8)\.
- W\. Wu, Y\. Wang, G\. Xiao, H\. Peng, and Y\. Fu \(2025\)Retrieval head mechanistically explains long\-context factuality\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=EytBpUGB1Z)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p3.1)\.
- Z\. Wu, B\. Liu, R\. Yan, L\. Chen, and T\. Delteil \(2024\)Reducing distraction in long\-context language models by focused learning\.External Links:2411\.05928,[Link](https://arxiv.org/abs/2411.05928)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p2.1)\.
- T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei,et al\.\(2024\)Osworld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.Advances in Neural Information Processing Systems37,pp\. 52040–52094\.Cited by:[§1](https://arxiv.org/html/2607.12963#S1.p1.1)\.
- J\. Yang, C\. E\. Jimenez, A\. Wettig, K\. Lieret, S\. Yao, K\. Narasimhan, and O\. Press \(2024\)SWE\-agent: agent\-computer interfaces enable automated software engineering\.InProceedings of the 38th International Conference on Neural Information Processing Systems,NIPS ’24,Red Hook, NY, USA\.External Links:ISBN 9798331314385Cited by:[§1](https://arxiv.org/html/2607.12963#S1.p1.1)\.
- M\. Yang, E\. Huang, L\. Zhang, M\. Surdeanu, W\. Y\. Wang, and L\. Pan \(2025\)How is LLM reasoning distracted by irrelevant context? an analysis using a controlled benchmark\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 13329–13347\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.674/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.674),ISBN 979\-8\-89176\-332\-6Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p2.1)\.
- O\. Yoran, T\. Wolfson, O\. Ram, and J\. Berant \(2024\)Making retrieval\-augmented language models robust to irrelevant context\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ZS4m74kZpH)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p2.1)\.
- Y\. Zheng, D\. Fu, X\. Hu, X\. Cai, L\. Ye, P\. Lu, and P\. Liu \(2025\)DeepResearcher: scaling deep research via reinforcement learning in real\-world environments\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 414–431\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.22/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.22),ISBN 979\-8\-89176\-332\-6Cited by:[§1](https://arxiv.org/html/2607.12963#S1.p1.1)\.
- Y\. Zhu, R\. Li, D\. Wang, D\. Haehn, and X\. Liang \(2025\)Focus directions make your language models pay more attention to relevant contexts\.External Links:2503\.23306Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p3.1)\.
- J\. Zhuo, S\. Zhang, X\. Fang, H\. Duan, D\. Lin, and K\. Chen \(2024\)ProSA: assessing and understanding the prompt sensitivity of LLMs\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 1950–1976\.External Links:[Link](https://aclanthology.org/2024.findings-emnlp.108/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.108)Cited by:[§4](https://arxiv.org/html/2607.12963#S4.p1.1)\.

## Appendix ASplit\-sample WTD

For WTD, the naive estimator ranks questions byΔ^i\\widehat\{\\Delta\}\_\{i\}, keeps them=⌈K100​N⌉m=\\left\\lceil\\frac\{K\}\{100\}N\\right\\rceilmost degraded of theNNquestions, and reports the negative mean\. Evaluating the same noisy estimates used for selection exaggerates degradation\. The main paper uses a bootstrap procedure as its default noise\-floor correction; here we report a complementary split\-sample estimation in Table[5](https://arxiv.org/html/2607.12963#A2.T5)\.

To remove the bias, we select and measure on*disjoint*trials\. We split each side into two:

- •The baseline trials are split into two disjoint halvesB1,B2B\_\{1\},B\_\{2\};
- •The context trials are split into two disjoint halvesC1,C2C\_\{1\},C\_\{2\}\.

On each half, we form an independent estimate of the change,

d1=mean​\(C1\)−mean​\(B1\),d2=mean​\(C2\)−mean​\(B2\),\\displaystyle d\_\{1\}\\;=\\;\\mathrm\{mean\}\(C\_\{1\}\)\-\\mathrm\{mean\}\(B\_\{1\}\),\\qquad d\_\{2\}\\;=\\;\\mathrm\{mean\}\(C\_\{2\}\)\-\\mathrm\{mean\}\(B\_\{2\}\),\(1\)We use one half to*select*the tail and the other to*measure*it,

S=\{the​m​questions with the smallest​d1\},𝒲^K=−1m​∑i∈Sd2,i\.\\displaystyle S\\;=\\;\\\{\\,\\text\{the \}m\\text\{ questions with the smallest \}d\_\{1\}\\,\\\},\\qquad\\widehat\{\\mathcal\{W\}\}\_\{K\}\\;=\\;\-\\frac\{1\}\{m\}\\sum\_\{i\\in S\}d\_\{2,i\}\.\(2\)Because the selected setSSdepends only ond1d\_\{1\}, andd2d\_\{2\}is independent ofd1d\_\{1\}given the question,𝔼​\[d2,i∣i∈S\]\\mathbb\{E\}\[d\_\{2,i\}\\mid i\\in S\]equals the true change of each selected question\. Thus, measurement is unbiased for the questions selected by the noisy ranking\. The selected set can still differ from the oracle tail ranked by the latentΔi\\Delta\_\{i\}\. A single split is noisy, so we repeat over random splits and average𝒲^K\\widehat\{\\mathcal\{W\}\}\_\{K\}\. By default, the two halves are also used*symmetrically*\(select ond1d\_\{1\}/ measured2d\_\{2\}, and select ond2d\_\{2\}/ measured1d\_\{1\}, then average\): every trial feeds both selection and measurement\.

## Appendix BExtra Experiment Details

While building the pseudo\-word vocabulary, the Zipf frequency threshold used for the English word filtering is1\.51\.5, and the total vocabulary size is1000010000\.

For each dataset, we first apply keyword\-based filtering to remove questions that might trigger abuse policies \(e\.g\., biosafety and cybersecurity\)\. Then we randomly sample 500 examples for MMLU\-Pro, HLE \(text\-only questions\), and SimpleQA Verified\. Since GPQA\-Diamond originally has only 198 questions and 8 are filtered, we use the remaining 190 questions\.

We use the following parameters consistently\. To estimate the noise floor, we construct10001000synthetic null experiments as described in Section[2](https://arxiv.org/html/2607.12963#S2)\. To estimate each95%95\\%confidence interval, we bootstrap questions with replacement10001000times, retaining all conditions and trials associated with each sampled question and recomputing the complete reported statistic\. In split\-sample and split\-half experiments, we average over200200random trial splits\.

Table 5:Split\-sample WTD with 95% CIs\.Table 6:For the reported INS and WTD in Table[1](https://arxiv.org/html/2607.12963#S2.T1), we show the observedℐ^\\widehat\{\\mathcal\{I\}\}with its95%95\\%CI and its noise floorηℐ\\eta\_\{\\mathcal\{I\}\}and the observed𝒲^K\\widehat\{\\mathcal\{W\}\}\_\{K\}with its95%95\\%CI and its noise floorη𝒲K\\eta\_\{\\mathcal\{W\}\_\{K\}\}\.![Refer to caption](https://arxiv.org/html/2607.12963v1/correlation_heatmaps_app.png)Figure 7:The correlation of per\-example performance change between all models on GPQA and HLE\. We report Pearsonrrfor all model pairs\.Table 7:Randomized split\-half reliability of per\-question performance changes within each model: mean Pearsonrrover splits with 95% CIs\.Table 8:Performance and average number of response/reasoning tokens for reasoning models\.
## Appendix CDetails of Qualitative Analysis

`System Prompt User Prompt For qualitative analysis, for each question with a large performance change, we fetch all the correct answers from the high performance condition and all the wrong answers from the low performance condition, randomly sample five \(correct response, wrong response\) pairs, and feed each pair to the prompt above to get the label based on gpt\-5\.5 with high reasoning\.`

`Appendix D Randomized split\-half reliability In Figure 3, since Δ^i\\widehat\{\\Delta\}\_\{i\} is estimated from 2020 baseline versus 2020 context trials, it carries sampling noise which attenuates correlations toward zero\. To estimate the corresponding noise ceiling, we compute randomized split\-half reliability, which measures the reproducibility of a model’s instability profile across random partitions of its trials\. Using the notation in Appendix A, Table 7 reports the Pearson correlation between the vectors d1d\_\{1\} and d2d\_\{2\} across questions\. Split\-half rr is substantial for every model and benchmark: it averages 0\.730\.73 on MMLU\-Pro \(range 0\.600\.60–0\.870\.87\), 0\.620\.62 on GPQA, 0\.600\.60 on SimpleQA, and 0\.500\.50 on HLE\. Because each half uses only half the trials, split\-half rr is noisier than the full estimate Δ^i\\widehat\{\\Delta\}\_\{i\}\. The Spearman\-Brown formula corrects for this, ρ=2​r1/21\+r1/2\\rho\\;=\\;\\frac\{2\\,r\_\{1/2\}\}\{1\+r\_\{1/2\}\}, giving ρ≥0\.75\\rho\\geq 0\.75 across all eight models on MMLU\-Pro and comparable values elsewhere\. For two independently\-noisy measurements with ρi,ρj\\rho\_\{i\},\\rho\_\{j\}, the observed correlation relates to the true correlation ρi​j\\rho\_\{ij\} by robs=ρi​j​ρi​ρjr\_\{\\mathrm\{obs\}\}=\\rho\_\{ij\}\\sqrt\{\\rho\_\{i\}\\rho\_\{j\}\}, so attenuation multiplies true correlations by the factor ρi​ρj\\sqrt\{\\rho\_\{i\}\\rho\_\{j\}\}\. On MMLU\-Pro, this factor is at least 0\.750\.75, so a cross\-model correlation of 0\.300\.30 would still be observed as ≥0\.22\\geq 0\.22\. Therefore, the near\-zero cross\-model Pearson correlations cannot be explained by sampling noise alone; they indicate a genuine lack of shared structure in the examples affected by context across models\. Appendix E Details of Ambiguity annotation System Prompt User Prompt We use the prompt above with gemini\-3\.5\-flash \(minimal reasoning\), gpt\-5\.5 \(no reasoning\), and gpt\-4\.1 to produce three flags for each question\. We avoid reasoning as much as possible to ensure models do not overthink the problems and focus on the surface\-form ambiguity\. To reflect the general property of each question, we use the total number of flags each question received as its general ambiguity level \(Level 0: 57\.9%57\.9\\%, Level 1: 25\.2%25\.2\\%, Level 2: 11\.6%11\.6\\%, Level 3: 5\.3%5\.3\\%\), and correlate it with the performance change on gpt\-5\.4 to avoid bias\. Note that regular human annotators cannot provide reliable judgments about the ambiguity of these questions, since they usually require domain\-specific knowledge to understand the questions and their backgrounds\. Appendix F Correlation Estimator Details Split\-sample estimation For a question ii we estimate the no\-context accuracy s^i​0\\widehat\{s\}\_\{i0\}, the with\-context accuracy s^i​c\\widehat\{s\}\_\{ic\}, and the performance change Δ^i=s^i​c−s^i​0\\widehat\{\\Delta\}\_\{i\}=\\widehat\{s\}\_\{ic\}\-\\widehat\{s\}\_\{i0\} from a finite number of repeated trials\. The baseline uncertainty feature ui=s^i​0​\(1−s^i​0\)u\_\{i\}=\\widehat\{s\}\_\{i0\}\(1\-\\widehat\{s\}\_\{i0\}\) and the change Δ^i\\widehat\{\\Delta\}\_\{i\} are computed from the same baseline trials, so their sampling noise is shared\. We remove the coupling by estimating the feature and the target from disjoint trials\. • The baseline trials are split into three disjoint folds F,B1,B2F,B\_\{1\},B\_\{2\} \(\|F\|≈nb/3\|F\|\\approx n\_\{b\}/3, with the remainder divided evenly between B1B\_\{1\} and B2B\_\{2\}\); • The context trials are split into two disjoint folds C1,C2C\_\{1\},C\_\{2\}\. The uncertainty feature is estimated only from FF: s^F​\(1−s^F\)\\widehat\{s\}^\{F\}\(1\-\\widehat\{s\}^\{F\}\)\. The externally annotated ambiguity level carries no trial noise and is held constant across splits\. The two targets are signed: Δ^i=mean​\(context\)−mean​\(B1∪B2\),\\displaystyle\\widehat\{\\Delta\}\_\{i\}\\;=\\;\\mathrm\{mean\}\(\\mathrm\{context\}\)\-\\mathrm\{mean\}\(B\_\{1\}\\cup B\_\{2\}\), \(3\) squared: d1​d2,d1=mean​\(C1\)−mean​\(B1\),d2=mean​\(C2\)−mean​\(B2\)\.\\displaystyle d\_\{1\}d\_\{2\},\\qquad d\_\{1\}=\\mathrm\{mean\}\(C\_\{1\}\)\-\\mathrm\{mean\}\(B\_\{1\}\),\\;\\;d\_\{2\}=\\mathrm\{mean\}\(C\_\{2\}\)\-\\mathrm\{mean\}\(B\_\{2\}\)\. \(4\) Because FF is disjoint from B1∪B2B\_\{1\}\\cup B\_\{2\}, the feature shares no trials with the signed target\. Because d1d\_\{1\} and d2d\_\{2\} are built from disjoint trials \(B1,C1B\_\{1\},C\_\{1\} vs\. B2,C2B\_\{2\},C\_\{2\}\), their noises are independent and 𝔼​\[d1​d2\]=Δi2\\mathbb\{E\}\[d\_\{1\}d\_\{2\}\]=\\Delta\_\{i\}^\{2\}\. Probability and log\-odds scales\. Probability\-scale changes can depend mechanically on baseline accuracy: under a common log\-odds shift η\\eta, a small probability change is approximately Δi≈η​si​0​\(1−si​0\)\\Delta\_\{i\}\\approx\\eta s\_\{i0\}\(1\-s\_\{i0\}\)\. We therefore repeat the analysis on a smoothed log\-odds scale\. For kk correct responses among nn trials, we use the empirical logit g​\(k,n\)=log⁡k\+1/2n−k\+1/2,g\(k,n\)=\\log\\frac\{k\+1/2\}\{n\-k\+1/2\}, and replace probability differences by differences in gg\. The same disjoint folds are used for the split\-sample log\-odds analysis\. The uncertainty\-magnitude correlation remains \+0\.21\+0\.21 under both scales, indicating that probability\-scale compression alone does not explain the result\. Homogeneous\-shift null\. As an additional check, we simulate a null in which every question receives the same log\-odds shift, using the Mantel\-Haenszel common log\-odds ratio η^=−0\.19\\widehat\{\\eta\}=\-0\.19\. Across 200 simulations of the same split\-sample estimator, the uncertainty\-squared\-change correlation lies in \[−0\.02,\+0\.04\]\[\-0\.02,\+0\.04\] on both scales, below the observed \+0\.21\+0\.21\. Reliability ceiling\. Disjoint trials remove shared\-noise bias, but they do not eliminate attenuation from estimating both the feature and target with a finite number of trials\. Let relF\\mathrm\{rel\}\_\{F\} and relT\\mathrm\{rel\}\_\{T\} denote the reliabilities of the feature and target, estimated by correlating replicate estimates constructed from disjoint trials\. Under classical independent measurement error, ρceiling=relF​relT\\rho\_\{\\mathrm\{ceiling\}\}=\\sqrt\{\\mathrm\{rel\}\_\{F\}\\,\\mathrm\{rel\}\_\{T\}\} gives the attenuation ceiling for Pearson correlation; we use it as an approximate reliability ceiling for the reported Spearman correlation\. For the uncertainty\-squared\-change correlation, this approximate ceiling is 0\.430\.43 on both the probability and log\-odds scales\. The observed split\-sample correlations of \+0\.21\+0\.21 are therefore about half of the corresponding reliability ceilings\. For signed change, the approximate ceiling is 0\.550\.55 on both scales, making the near\-zero signed correlations unlikely to be explained solely by low reliability\. Figure 8: \(Top\) The correlation of per\-example performance change between Olmo\-3 checkpoints and across local model families on MMLU\. Olmo\-3\-7B and Olmo\-3\-7B\-Ins\(truct\) on the right correspond to the CXT and RLVR checkpoints on the left\. \(Bottom\) INS and WTD induced by different types of context on Olmo\-3\-7B\-Instruct and the correlation of per\-example performance change between contexts\. Appendix G Details of Local Model Experiments For local model experiments, we use MMLU \(randomly sampled 500 questions\) and the same prompt format as our main setting, with the following differences: \(1\) We use 1​k1k tokens of context instead of 10​k10k\. Note that we use the corresponding tokenizers of the tested models\. For WEB/TOK/CHA, we truncate the same context in the main setting at 1​k1k tokens\. For SQA, we keep as many complete \(question, answer\) pairs as possible generated by gpt\-5\.4 within 1​k1k tokens\. \(2\) We use the model\-specific non\-reasoning chat template for each model, while we prepend “\*\*Correct Answer:” to the assistant’s response and fetch the probability of the ground truth option tokens \(e\.g\., “ A”, “ B”\) as s^i​j\\widehat\{s\}\_\{ij\}\. \(3\) There is no noise correction while calculating INS and WTD\. For pretrained checkpoints without a chat template, we use the default chat template below: Default Chat Template We replicate the experiments in Figure 3 and Figure 4 using local models, as shown in Figure 8\. We find that: \(1\) The correlation of per\-example performance change is still near random across model families \(e\.g\., Olmo, Qwen, and Gemma\)\. \(2\) Even within the same model family, the correlation between the base version and the post\-trained version could also be low \(e\.g\., 0\.010\.01 for Gemma\)\. By examining correlations across different stages of Olmo\-3\-7B\-Instruct, the correlation changes most during midtraining and SFT, suggesting that a relatively lightweight training procedure like SFT can largely shape the instability\. \(3\) The correlation between different context types \(mean Pearson rr 0\.700\.70 with 95%95\\% CI: \[0\.67,0\.74\]\[0\.67,0\.74\]\) is higher than that in Figure 4, as there is no sampling noise\. Appendix H Prompt Setup H\.1 Prompts for question answering By default, we use a simple prompt that asks the model to provide answers with brief explanations\. System Prompt User Prompt \(Multi\-choice question\) User Prompt \(General QA\) MMLU\-Pro GPQA HLE SimpleQA AVG Δ\\Delta INS↓\\downarrow WTD↓\\downarrow AVG Δ\\Delta INS↓\\downarrow WTD↓\\downarrow AVG Δ\\Delta INS↓\\downarrow WTD↓\\downarrow AVG Δ\\Delta INS↓\\downarrow WTD↓\\downarrow w/o CoT 78\.078\.0 \+0\.0\+0\.0 4\.34\.3 19\.719\.7 57\.357\.3 −0\.9\-0\.9 5\.65\.6 22\.722\.7 3\.83\.8 −0\.2\-0\.2 1\.01\.0 5\.45\.4 30\.430\.4 −0\.7\-0\.7 3\.13\.1 11\.811\.8 w/ CoT 81\.281\.2 \+0\.0\+0\.0 1\.31\.3 6\.46\.4 64\.164\.1 \+1\.4\+1\.4 2\.32\.3 3\.73\.7 4\.64\.6 −0\.3\-0\.3 0\.80\.8 4\.84\.8 32\.232\.2 −0\.3\-0\.3 1\.71\.7 7\.67\.6 Table 9: The effect of Chain\-of\-Thought \(CoT\) prompting on gpt\-4\.1\. w/o CoT w/ CoT Baseline Context Baseline Context MMLU\-Pro 304304 326326 503503 536536 GPQA 589589 620620 884884 925925 HLE 11781178 11871187 14731473 14971497 SimpleQA 111111 132132 207207 248248 Table 10: gpt\-4\.1 response length across benchmarks\. We provide the full CoT experiment results in Table 9 and Table 10\. User Prompt for CoT \(Multi\-choice question\) User Prompt for CoT \(General QA\) H\.2 Prompts for grading Our grading prompt clearly instructs the model to extract the final derived answer\. System Prompt \(Multi\-choice question\) System Prompt \(General QA\) User Prompt Appendix I Examples I\.1 Context Examples We show some \(truncated\) examples of different context types below: CHA TOK WEB SQA \(Model\-specific chat template is applied\) I\.2 Question & Response Examples We present examples of context\-induced instability for each category in Table 2\. All responses are from gpt\-5\.4 with reasoning effort none\. I\.2\.1 Wrong Problem Question \(Ground Truth: G\) Response \(no\-context, solving the wrong problem since Z is not a vowel\.\) Response \(with context, correct\) I\.2\.2 Wrong Fact Question \(Ground Truth: Liverpool FC\) Response \(no\-context, correct\) Response \(with context, retrieved the wrong fact\) I\.2\.3 Wrong Rule Question \(Ground Truth: J\) Response \(no\-context, applying a wrong rule for short\-term loss\) Response \(with context, correct\) I\.2\.4 Wrong Calculation Question \(Ground Truth: H\) Response \(no\-context, correct\) Response \(with context, wrong calculation of 11 \+ 9\)`