Stable Miscalibration in Large Language Models: A Practical View of High-Confidence Errors

# 1 Introduction
Source: [https://arxiv.org/html/2608.13591](https://arxiv.org/html/2608.13591)
marginparsep has been altered\. topmargin has been altered\. marginparpush has been altered\. The page layout violates the ICML style\.Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you\. We’re not able to reliably undo arbitrary changes to the style\. Please remove the offending package\(s\), or layout\-changing commands and try again\.

Stable Miscalibration in Large Language Models: A Practical View of High\-Confidence Errors

Akira Okutomi1

††footnotetext:1ToppyMicroServices OÜ, Tallinn, Estonia\. Correspondence to: Akira Okutomi, Tokyo, Japan\.
2nd Workshop on Epistemic Intelligence in Machine Learning \(EIML@ICML 2026\), Seoul, South Korea\. Copyright 2026 by the author\(s\)\.###### Abstract

High\-confidence errors in large language models are often treated as evidence of fragile internal inference\. We study a different possibility: stable miscalibration, where a confident wrong answer remains locally stable under small perturbations\. We combine two diagnostics: a label\-aware output\-level audit score that ranks domains by confidence variation and overconfident mistakes under a forced\-answer baseline, and an internal sensitivity probe that measures hidden\-state movement\. On a multi\-domain binary factual audit set, this audit score tracks where abstention\-aware self\-critique reduces decision loss, although direct labeled baselines rank the same gain more strongly\. Internally, self\-critical prompting consistently reduces hidden\-state sensitivity across layers in three open\-weight models\. This supports prompt\-induced local stabilization rather than a purely output\-level abstention pattern, but it does not imply calibration: audit\-defined overconfident errors are not clearly more locally sensitive than confidently correct answers, so some high\-confidence errors may be stable and miscalibrated rather than simply fragile\.

High\-confidence hallucinations in large language models \(LLMs\) are often described as a sign of internal fragility: if a small prompt or context change can flip the answer, then a high\-confidence error looks like a fragile failure modeJiet al\.\([2023](https://arxiv.org/html/2608.13591#bib.bib11)\); Manakulet al\.\([2023](https://arxiv.org/html/2608.13591#bib.bib9)\); Jooet al\.\([2025](https://arxiv.org/html/2608.13591#bib.bib10)\)\. That picture is plausible, but it is not the only one\. A wrong answer can also be*stable*: the model may stay near the same internal state under small perturbations and still produce a confident mistake\. We refer to this possibility as*stable miscalibration*\.

This distinction matters for uncertainty estimation\. If high\-confidence errors mainly come from local fragility, then perturbation sensitivity should separate overconfident mistakes from correct high\-confidence answers\. If they instead reflect stable miscalibration, then high\-confidence errors may matter for decisions without being unusually fragile\. For hard\-to\-interpret LLMs, a full internal explanation is often out of reach, so we focus on what can be learned from confidence responses and small internal probes\.

We therefore ask a narrower but sharper question: when high\-confidence errors matter in practice, what can we infer from the confidence responses and perturbation behavior that we can actually measure? High\-confidence errors can be viewed as failures of epistemic awareness: the model acts as if its evidence is sufficient even when the decision remains unreliable\.

Our work sits at the intersection of abstention\-aware decision policiesWenet al\.\([2025](https://arxiv.org/html/2608.13591#bib.bib1)\); Tayebatiet al\.\([2025](https://arxiv.org/html/2608.13591#bib.bib2)\); An and Xu \([2025](https://arxiv.org/html/2608.13591#bib.bib3)\), calibration and overconfidence in LLMsGuoet al\.\([2017](https://arxiv.org/html/2608.13591#bib.bib4)\); Kadavathet al\.\([2022](https://arxiv.org/html/2608.13591#bib.bib5)\); Xiaoet al\.\([2025](https://arxiv.org/html/2608.13591#bib.bib6)\); Wanget al\.\([2026](https://arxiv.org/html/2608.13591#bib.bib7)\), and perturbation or consistency probes for factual reliabilityJiet al\.\([2024](https://arxiv.org/html/2608.13591#bib.bib8)\); Manakulet al\.\([2023](https://arxiv.org/html/2608.13591#bib.bib9)\); Jooet al\.\([2025](https://arxiv.org/html/2608.13591#bib.bib10)\)\. We make three claims\.

1. 1\.Stable miscalibration is a useful uncertainty hypothesis\.A linear–Gaussian closed\-loop abstraction suggests that local stability and correctness need not coincide\.
2. 2\.A label\-aware audit score ranks where intervention helps\.A domain\-level score built from policy\-level confidence variation and overconfident mistakes under a forced\-answer baseline correlates with where abstention\-aware self\-critique reduces policy\-aware loss\. Because the score uses observed correctness, it should be read as a labeled audit diagnostic\.
3. 3\.Prompt\-induced local stabilization is visible internally, but it is not a fragility separator\.Across three open\-weight models, the self\-critical prompt lowers layer\-wise hidden\-state sensitivity\. However, audit\-defined overconfidently wrong items do not show a clear sensitivity gap relative to audit\-defined confidently correct items\.

The main point is not a full mechanistic reduction, but a practical lesson: uncertainty\-related behavior that matters for decisions can sometimes be studied through confidence responses even when the internal mechanism remains only partly understood\.

## 2Related Work

The closest related work falls into three groups\. First, abstention\-aware prediction treats refusal as a decision option rather than a failure mode, and recent LLM work emphasizes that the usefulness of abstention depends on both calibration and task valueWenet al\.\([2025](https://arxiv.org/html/2608.13591#bib.bib1)\); Tayebatiet al\.\([2025](https://arxiv.org/html/2608.13591#bib.bib2)\); An and Xu \([2025](https://arxiv.org/html/2608.13591#bib.bib3)\)\. Our policy\-aware loss follows this view: the question is not only whether a policy is accurate, but whether it answers in the right places\.

Second, calibration work shows that language models can remain overconfident, and recent studies question whether verbalized confidence reliably turns into good decisionsGuoet al\.\([2017](https://arxiv.org/html/2608.13591#bib.bib4)\); Kadavathet al\.\([2022](https://arxiv.org/html/2608.13591#bib.bib5)\); Xiaoet al\.\([2025](https://arxiv.org/html/2608.13591#bib.bib6)\); Wanget al\.\([2026](https://arxiv.org/html/2608.13591#bib.bib7)\)\. We build on this literature, but our goal is narrower than a new calibration benchmark\. We ask whether a practical score built from observable confidence behavior can rank where intervention is likely to help\.

Third, perturbation and consistency probes use either internal representations or output changes to study reliability under small input variationJiet al\.\([2024](https://arxiv.org/html/2608.13591#bib.bib8)\); Manakulet al\.\([2023](https://arxiv.org/html/2608.13591#bib.bib9)\); Jooet al\.\([2025](https://arxiv.org/html/2608.13591#bib.bib10)\)\. We use both views at once: an output\-level score for domain ranking and a layer\-wise internal probe for testing whether overconfident mistakes are actually more locally fragile\.

## 3A Practical View of Stable Miscalibration

We use a linear–Gaussian closed\-loop model only as a simple abstractionKalman \([1960](https://arxiv.org/html/2608.13591#bib.bib12)\), not as a literal model of LLM inference\. A minimal local version is

zt\+1=A​zt\+b\+ηt,ηt∼𝒩​\(0,Σ\)\.z\_\{t\+1\}=Az\_\{t\}\+b\+\\eta\_\{t\},\\qquad\\eta\_\{t\}\\sim\\mathcal\{N\}\(0,\\Sigma\)\.\(1\)Ifρ​\(A\)<1\\rho\(A\)<1, small perturbations are damped and the state converges locally\. The fixed pointz⋆=\(I−A\)−1​bz^\{\\star\}=\(I\-A\)^\{\-1\}b, however, can still lie in a decision region whose label disagrees with the truth\. Thus local stability is compatible with systematic miscalibration\. This is the picture we call*stable miscalibration*\. Here we focus on what this picture implies for quantities we can measure directly\.

The practical takeaway is simple\. If stable miscalibration is real, then two observable signals become important\. First, confidence may move substantially when the same item is examined under nearby decision policies\. Second, the reference policy may accumulate a noticeable mass of confident mistakes even when its behavior is locally stable\. Our domain score is designed to capture exactly those two signals\.

For LLMs, we summarize domain\-level risk with an observable score built from confidence movement and an overconfident\-wrong \(OCW\) indicator:

Hproxy​\(d\)\\displaystyle H\_\{\\mathrm\{proxy\}\}\(d\)=1\|Dd\|∑i∈Dd\[Stdc∈\{C​0,C​1,C​2\}\(pi,c\)\\displaystyle=\\frac\{1\}\{\|D\_\{d\}\|\}\\sum\_\{i\\in D\_\{d\}\}\\Big\[\\mathrm\{Std\}\_\{c\\in\\\{C0,C1,C2\\\}\}\(p\_\{i,c\}\)\(2\)\+λOCW\(i\)\]\.\\displaystyle\\qquad\\qquad\+\\;\\lambda\\,\\mathrm\{OCW\}\(i\)\\Big\]\.Here C0 is the forced\-answer baseline, C1 is the cautious\-abstention policy, and C2 is the self\-critical abstention policy \(see Section[4](https://arxiv.org/html/2608.13591#S4)\)\. The valuepi,c∈\[0,1\]p\_\{i,c\}\\in\[0,1\]is the reportedP​\(correct\)P\(\\mathrm\{correct\}\)for itemiiunder policycc, with abstentions mapped to the neutral value0\.50\.5\. The overconfident\-wrong \(OCW\) indicator is11for an item when C0 is overconfidently wrong: it answers the item, exceeds the high\-confidence threshold, and is wrong; otherwise it is0\. The first term is large when confidence changes across policies; the second is large when the forced\-answer baseline makes a confident mistake\. We use a single high\-confidence cutoff of 0\.8 and setλ=1\\lambda=1throughout\.

Because the overconfident\-wrong term uses correctness,HproxyH\_\{\\mathrm\{proxy\}\}is a labeled audit diagnostic rather than a deployment\-time estimator for unlabeled inputs\. Its purpose is not to replace direct outcome diagnostics such as C0 error rate or C0 Brier risk, but to separate two intervention\-relevant signals: policy movement and overconfident\-error mass\. Operationally, a larger value means that a domain shows policy movement, overconfident failure, or both\. The linear–Gaussian picture is only a guide: it illustrates why local stability and correctness can separate, making these observable signals natural audit quantities\.

Two domains can therefore have similar average accuracy and still receive different proxy scores\. One may have relatively stable confidence and few confident mistakes, while another may look safe on average but become risky once nearby policies or abstention are introduced\. The score is designed for this second question: not “which domain is hardest in the abstract,” but “which domain is most exposed to costly overconfidence and most likely to benefit from intervention\.”

## 4Experimental Setup

We useN=532N=532short binary factual items grouped into 11 domains\. This is a frozen audit set, not a named public benchmark; the reproduction bundle includes the item strings, domain labels, gold labels, and policy outputs\. Domain sizes range from 35 to 64 items\. The domains cover medical epidemiology, social stats, geo travel, cultural industry, dev region, entertainment event, literature media, macro index, history diplomacy, sports, and technical standard\. Every item is evaluated under the same three policies, so domain\-wise comparisons are paired throughout\. The appendix gives representative item examples, and the artifact repository contains the frozen inputs used for all reported numbersOkutomi \([2026](https://arxiv.org/html/2608.13591#bib.bib16)\)\.

The item file was frozen before policy evaluation\. Binary gold labels were assigned independently of model outputs, and items with missing or ambiguous written truth conditions were excluded from the frozen audit set\. Domain labels are used only for aggregation and are not shown to the model\.

#### Policies\.

C0 is a forced\-answer baseline that must output “Yes” or “No”\. C1 allows abstention when the item is too under\-specified to answer responsibly\. C2 adds an explicit self\-critical check before the final decision and may also abstain\. All three policies use the same item wording; only the decision policy changes\.

#### Output\-level policy model\.

All C0–C2 policy logs were collected with “gpt\-4\.1\-mini”\. The logging script did not set an explicit temperature parameter; exact replay may therefore depend on the API default and hosted model snapshot\. Each policy is prompted to return a decision and a verbalP​\(correct\)P\(\\mathrm\{correct\}\)score; the gold label is never included in the prompt\. We use this verbal confidence as a reported confidence signal, not as a calibrated probability guaranteed by the model\. We release the frozen policy\-output CSV used for the output\-level analyses\.

#### Confidence and policy\-aware loss\.

When a policy answers, it also reportsP​\(correct\)∈\[0,1\]P\(\\mathrm\{correct\}\)\\in\[0,1\]\. If it abstains, we assign the neutral value0\.50\.5, which yields a fixed squared\-loss penalty of0\.250\.25\. We evaluate policy gains with a Brier\-style squared lossBrier \([1950](https://arxiv.org/html/2608.13591#bib.bib13)\); Gneiting and Raftery \([2007](https://arxiv.org/html/2608.13591#bib.bib14)\)on the probability assigned to the positive “Yes” label\. Domain\-wise gain is always measured relative to C0\.

This design matters because the evaluation is intentionally intervention\-aware\. A policy can help either by improving the answer itself or by refusing where the baseline tends to be confidently wrong\. The loss therefore rewards useful abstention without treating abstention as cost\-free\.

For an item with binary labelyi∈\{0,1\}y\_\{i\}\\in\\\{0,1\\\}, we write

S​Epolicy​\(i,c\)=\(p~i,c−yi\)2,SE\_\{\\mathrm\{policy\}\}\(i,c\)=\(\\tilde\{p\}\_\{i,c\}\-y\_\{i\}\)^\{2\},\(3\)wherep~i,c\\tilde\{p\}\_\{i,c\}is the probability assigned to the positive “Yes” label under policycc\. If the policy answers “Yes”, thenp~i,c=P​\(correct\)\\tilde\{p\}\_\{i,c\}=P\(\\mathrm\{correct\}\); if it answers “No”, thenp~i,c=1−P​\(correct\)\\tilde\{p\}\_\{i,c\}=1\-P\(\\mathrm\{correct\}\); and if it abstains, thenp~i,c=0\.5\\tilde\{p\}\_\{i,c\}=0\.5\. Domain\-wise gains are paired averages of the change in this loss relative to C0\.

#### Internal probe\.

To probe local internal sensitivity, we use three open\-weight instruction\-tuned models: Llama\-3\.1\-8B\-Instruct, DeepSeek\-R1\-Distill\-Llama\-8B, and Qwen2\.5\-7B\-Instruct\. We add Gaussian embedding perturbations with scaleσ=0\.01\\sigma=0\.01and average over 40 trials per item\. The confidently correct \(CC\) and overconfidently wrong \(OCW\) groups are inherited from the frozen “gpt\-4\.1\-mini” C0 audit logs, not recomputed from each open\-weight model’s own answers\. An item is CC when that C0 log is correct with reported confidence at least 0\.8, and OCW when that C0 log is wrong with reported confidence at least 0\.8\. The threshold is a fixed high\-confidence cutoff used for this audit, not an optimized constant\. Thus, the probe asks whether audit\-model high\-confidence failures show greater hidden\-state sensitivity when the same item text is run through open\-weight models\. We then compare the same items under a self\-critical prompt that asks the model to check for missing evidence or counter\-considerations before answering\. This comparison also tests whether self\-critique induces prompt\-level local stabilization in hidden states, not only whether OCW items are more fragile\. Section[5](https://arxiv.org/html/2608.13591#S5)also reports denserσ\\sigmasweeps and a curated semantic\-rewrite check on two models, using 12 confidently correct and 12 overconfidently wrong items per model\.

At each chosen layer, the probe measures how much the final\-token hidden state moves when the input embedding is perturbed\. We use this signal only as a local diagnostic\. A larger value means that the representation moves more under the perturbation; it does not by itself identify the source of the movement\.

Concretely, letϵ\\epsilonbe the injected embedding\-space perturbation, with norm controlled by the noise scaleσ\\sigma\. Ifhℓcleanh\_\{\\ell\}^\{\\mathrm\{clean\}\}andhℓnoisyh\_\{\\ell\}^\{\\mathrm\{noisy\}\}are the clean and perturbed final\-token hidden states at layerℓ\\ell, then the per\-trial local sensitivity is

Sℓ​\(ϵ\)=‖hℓnoisy−hℓclean‖2‖ϵ‖2\.S\_\{\\ell\}\(\\epsilon\)=\\frac\{\\\|h\_\{\\ell\}^\{\\mathrm\{noisy\}\}\-h\_\{\\ell\}^\{\\mathrm\{clean\}\}\\\|\_\{2\}\}\{\\\|\\epsilon\\\|\_\{2\}\}\.\(4\)We average this quantity over trials, then compare means across models, prompts, and the audit\-defined CC/OCW subsets\.

Table 1:Compact setup summary\.

## 5Results

### 5\.1Output\-level audit proxy: where self\-critique helps

Figure[1](https://arxiv.org/html/2608.13591#S5.F1)plots normalizedHproxy​\(d\)H\_\{\\mathrm\{proxy\}\}\(d\)against the domain\-wise gain of C2 over C0\. We define this gain asGC​2​\(d\)=S​E¯C​0​\(d\)−S​E¯C​2​\(d\)G\_\{C2\}\(d\)=\\overline\{SE\}\_\{C0\}\(d\)\-\\overline\{SE\}\_\{C2\}\(d\), so positive values indicate improvement\. In Figure[2](https://arxiv.org/html/2608.13591#S5.F2), we instead plotΔ​S​Ec​\(d\)=S​E¯c​\(d\)−S​E¯C​0​\(d\)\\Delta SE\_\{c\}\(d\)=\\overline\{SE\}\_\{c\}\(d\)\-\\overline\{SE\}\_\{C0\}\(d\), so negative values indicate improvement\. The association in Figure[1](https://arxiv.org/html/2608.13591#S5.F1)is positive: domains with larger proxy values tend to be the domains where self\-critique reduces policy\-aware loss most\. The rank correlation is Spearmanρ=0\.71\\rho=0\.71with a bootstrap 95% interval of\[0\.12,0\.97\]\[0\.12,0\.97\]\.

The domain pattern is also interpretable\. Medical epidemiology and social stats show the largest gains under C2, while geo travel and cultural industry also improve\. By contrast, entertainment event and literature media do not benefit\. We therefore interpretHproxyH\_\{\\mathrm\{proxy\}\}as a*domain\-ranking*signal for intervention value, not as a causal estimate\.

![Refer to caption](https://arxiv.org/html/2608.13591v1/figures/fig_proxy_predicts_gain_v1.png)Figure 1:HproxyH\_\{\\mathrm\{proxy\}\}tracks where self\-critique helps on the labeled audit set\.Each point is a domain\. The dashed line is a Theil–Sen fit shown only for visualization\.Table 2:Domain\-ranking baselines\.Spearman correlations compare each score with C2 gain over C0 across 11 domains\. Predictive entropy is computed from C0’s reported confidence\. Label\-aware rows use gold correctness and should be read as retrospective audit baselines, not deployment\-time scores\.Table[2](https://arxiv.org/html/2608.13591#S5.T2)makes the baseline comparison explicit\. Predictive entropy and confidence variation alone do not rank the domains where C2 helps\. Label\-aware C0 error and Brier risk rank them strongly, as expected, because they measure observed failures directly, and in this small domain\-level comparison they are stronger pure rankers thanHproxyH\_\{\\mathrm\{proxy\}\}\. We therefore do not claim thatHproxyH\_\{\\mathrm\{proxy\}\}is the best predictor of C2 gain; it is a structured diagnostic showing whether high\-confidence baseline errors coincide with policy movement, rather than a performance\-optimized predictor\. A full semantic\-entropy baseline would require multiple sampled answers and semantic clusteringFarquharet al\.\([2024](https://arxiv.org/html/2608.13591#bib.bib15)\); the single\-answer policy logs do not contain that distribution, so we leave that comparison to a separate multi\-sample follow\-up\.

Table[3](https://arxiv.org/html/2608.13591#S5.T3)shows the abstention trade\-off directly\. C1 is the most conservative: it reduces the overall overconfident\-wrong rate from0\.2110\.211to0\.0280\.028, but coverage falls to0\.4270\.427\. C2 keeps more coverage \(0\.5710\.571\) while still lowering the overconfident\-wrong rate to0\.0640\.064\. Taken together, Figure[1](https://arxiv.org/html/2608.13591#S5.F1)and Table[3](https://arxiv.org/html/2608.13591#S5.T3)support a limited practical message: the score helps describe*why*a domain may benefit from abstention\-aware intervention, even when simpler labeled diagnostics rank the domains more strongly\.

The domain breakdown is also important because the gains are not uniform\. Under C2, the strongest paired improvements appear in medical epidemiology \(−0\.101\-0\.101\), social stats \(−0\.099\-0\.099\), and geo travel \(−0\.064\-0\.064\), while entertainment event \(\+0\.039\+0\.039\), literature media \(\+0\.049\+0\.049\), and technical standard \(\+0\.064\+0\.064\) become worse\.

![Refer to caption](https://arxiv.org/html/2608.13591v1/figures/fig_delta_brier_domain_v7.png)Figure 2:Domain\-wise change in policy\-aware loss relative to C0\.Negative values indicate lower policy\-aware loss than the forced\-answer baseline\. The domain pattern is mixed rather than uniform: some domains benefit substantially from abstention\-aware self\-critique, while others regress\.Figure[2](https://arxiv.org/html/2608.13591#S5.F2)makes this heterogeneity visible\. C1 and C2 are not globally better than C0 in every domain; their value lies in shifting the error profile in places where confident mistakes are common and abstention can be used productively\.

Table 3:Selective metrics under abstention\-aware policies\.Coverage is answer rate; answer yield treats abstentions as incorrect\.
### 5\.2Internal probe: prompt\-induced stabilization without a clear fragility gap

The internal probe separates a prompt\-level effect from a group\-level fragility test\. At the prompt level, self\-critical prompting makes hidden states less locally responsive, lowering layer\-wise sensitivitySℓS\_\{\\ell\}across depth in all three representative models\. This supports an internal stabilization interpretation rather than a purely output\-level abstention pattern\. The boundary is equally important: lower sensitivity is a stability signal, not calibration\. The reduction does not produce a clear separation between audit\-defined confidently correct \(CC\) and overconfidently wrong \(OCW\) items\.

![Refer to caption](https://arxiv.org/html/2608.13591v1/figures/fig_internal_sensitivity_1x3.png)Figure 3:Prompt\-induced local stabilization\.The self\-critical prompt lowers mean layer\-wise sensitivitySℓS\_\{\\ell\}across depth in all three representative models; error bars show standard errors over items\. Lower sensitivity indicates a smaller hidden\-state response to the injected local perturbation, not evidence of calibrated correctness\.Table[4](https://arxiv.org/html/2608.13591#S5.T4)gives the corresponding final\-layer numbers\. In short, the self\-critical prompt lowers the overall sensitivity in all three models, while the audit\-defined OCW–CC gaps remain small at the tested scale\. The bootstrap intervals include zero, and the standardized gaps are about0\.130\.13pooled standard deviations or less\. We therefore read the residual differences as weak drift, not evidence that the audit\-defined overconfident\-error items are uniquely fragile\.

![Refer to caption](https://arxiv.org/html/2608.13591v1/figures/fig_sigma_sweep_deepseek.png)![Refer to caption](https://arxiv.org/html/2608.13591v1/figures/fig_sigma_sweep_qwen.png)

![Refer to caption](https://arxiv.org/html/2608.13591v1/figures/fig_semantic_check_deepseek.png)![Refer to caption](https://arxiv.org/html/2608.13591v1/figures/fig_semantic_check_qwen.png)

Figure 4:Robustness checks\.DeepSeek\-R1 \(top\) and Qwen2\.5\-7B \(bottom\) under denser Gaussian sweeps \(left\) and semantic rewrites \(right\)\. The audit\-defined OCW–CC gap remains small and does not steadily grow across either perturbation family\.We also checked whether this conclusion depends on the default perturbation choice\. Figure[4](https://arxiv.org/html/2608.13591#S5.F4)shows denser Gaussian sweeps for DeepSeek\-R1 and Qwen2\.5\-7B and curated semantic rewrites for the same two models; the rewrites preserve the intended truth condition while changing surface wording and context\. These checks tell the same story: the OCW–CC gap stays small and does not steadily grow across either perturbation family\. They are still limited, but they make the “no clear gap” result less likely to be an artifact of the single default Gaussian setting\.

The two result blocks answer different uncertainty questions\. The output\-level proxy asks where intervention changes decision quality; it does not by itself establish stable miscalibration\. The internal probe asks a narrower follow\-up question: whether items marked as overconfidently wrong by the audit logs look uniquely fragile in open\-weight models\. Our data support the intervention\-ranking claim more clearly than any mechanism claim\.

Table 4:Internal sensitivity summary\.“All” columns report final\-layer mean sensitivity over all items under the standard and self\-critical \(SC\) prompts\. “CC” and “OCW” are inherited from the frozen C0 audit logs and report final\-layer sensitivity under the standard prompt only\. Gap is OCW–CC with a bootstrap 95% CI over items; standardized gap is in pooled\-SD units\.

## 6Discussion and Limitations

The claim is narrow\. The study uses a frozen 532\-item binary factual audit set rather than a broad public benchmark, so the rankings may not transfer to larger datasets, multi\-class tasks, or open\-ended generation\.HproxyH\_\{\\mathrm\{proxy\}\}is also a labeled ranking signal, not an absolute instability measure or a label\-free deployment estimator; in this audit, the direct labeled baselines in Table[2](https://arxiv.org/html/2608.13591#S5.T2)rank C2 gain more strongly\. Because the audit set is author\-curated rather than a public benchmark with independent multi\-annotator validation, label errors or item artifacts may affect the rankings\. Finally, the bridge from stability intuition to empirical score is local\. The output\-level audit and internal probe also use different model families: frozen policy logs establish the intervention pattern, while open\-weight models test whether an analogous CC–OCW fragility gap appears where hidden states are accessible\. Thus the probe does not isolate each open\-weight model’s own high\-confidence errors\.

Thus, “no clear CC–OCW gap” does not mean that hidden\-state instability never matters\. It means that, under the present probe, we do not see evidence for that local\-fragility account\. Consequently, prompt\-induced local quieting should be treated as a stability signal, not as evidence of calibrated correctness\. Future work should test the audit score on larger public benchmarks, extend it to multi\-class and open\-ended tasks, and develop label\-free approximations based on answer consistency, semantic entropy, calibrated confidence, or retrieval disagreement\. It should also compare more directly with calibration and uncertainty\-estimation methods, and test whether the CC–OCW gap changes under different confidence cutoffs, perturbation families, model\-specific CC/OCW definitions, and practical effect\-size rules; the present claim is labeled audit triage\.

Stable miscalibration is also a trust risk: a system can appear robust because its answers are locally stable, while still giving confident wrong answers in high\-impact domains\. The audit should therefore be used as a periodic labeled evaluation workflow rather than as a live safety filter: auditors can collect domain\-specific labeled items, rank domains by overconfident\-error mass and policy movement, and allocate abstention, retrieval, or human\-review interventions where risk is concentrated\.

## Impact Statement

This work studies high\-confidence errors and abstention\-aware auditing for large language models, with the intended impact of improving reliability under uncertainty\. The diagnostics should be used for labeled auditing, risk reporting, and human review, not as a guarantee of correctness or as a way to hide uncertainty\.

#### Reproducibility\.

The reproducibility artifact is archived at[https://osf\.io/ndz69/](https://osf.io/ndz69/?view_only=0920b52c74af4ded84f0086d5d773457)\. After extracting “OSF\.zip”, start with “README\.md”; the longer guide is “paper/eiml2026/repro/README\.md”\. The main command is “python scripts/reproduce\_eiml\.py”\. The default reproduction path uses frozen CSV inputs and does not call external APIs\.

#### Disclosure\.

LLMs were used for language editing and synthetic item drafting\. The author reviewed all claims, labels, code, and results\.

The views expressed are solely those of the author and do not represent any employer, client, or affiliated organization; no employer, client, proprietary, or personal data was used\.

## Appendix

#### Figure 4 detail\.

For the four\-domain DeepSeek subset,σ\\sigmaranges from0\.00250\.0025to0\.050\.05\. Across eight scales, the signed final\-layer audit\-defined OCW–CC gap ranges from−0\.185\-0\.185to\+0\.009\+0\.009, so this subset shows no stable positive OCW sensitivity gap\.

#### Audit materials\.

The OSF repository \(Okutomi, 2026\) contains the audit data, prompts, examples, and policy outputs\. C0/C1/C2 are forced\-answer, cautious\-abstention, and self\-critical; the probe compares standard and self\-critical prompts\.

## References

- Teaching LLMs to abstain via fine\-grained semantic confidence reward\.arXiv preprint arXiv:2510\.24020\.External Links:2510\.24020,[Link](https://arxiv.org/abs/2510.24020)Cited by:[§1](https://arxiv.org/html/2608.13591#S1.p4.1),[§2](https://arxiv.org/html/2608.13591#S2.p1.1)\.
- G\. W\. Brier \(1950\)Verification of forecasts expressed in terms of probability\.Monthly Weather Review78\(1\),pp\. 1–3\.External Links:[Document](https://dx.doi.org/10.1175/1520-0493%281950%29078%3C0001%3AVOFEIT%3E2.0.CO%3B2)Cited by:[§4](https://arxiv.org/html/2608.13591#S4.SS0.SSS0.Px3.p1.3)\.
- S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. Gal \(2024\)Detecting hallucinations in large language models using semantic entropy\.Nature630,pp\. 625–630\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07421-0),[Link](https://www.nature.com/articles/s41586-024-07421-0)Cited by:[§5\.1](https://arxiv.org/html/2608.13591#S5.SS1.p3.2)\.
- T\. Gneiting and A\. E\. Raftery \(2007\)Strictly proper scoring rules, prediction, and estimation\.Journal of the American Statistical Association102\(477\),pp\. 359–378\.External Links:[Document](https://dx.doi.org/10.1198/016214506000001437)Cited by:[§4](https://arxiv.org/html/2608.13591#S4.SS0.SSS0.Px3.p1.3)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.70,pp\. 1321–1330\.External Links:[Link](https://proceedings.mlr.press/v70/guo17a.html)Cited by:[§1](https://arxiv.org/html/2608.13591#S1.p4.1),[§2](https://arxiv.org/html/2608.13591#S2.p2.1)\.
- Z\. Ji, D\. Chen, E\. Ishii, S\. Cahyawijaya, Y\. Bang, B\. Wilie, and P\. Fung \(2024\)LLM internal states reveal hallucination risk faced with a query\.InProceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP,Miami, Florida, US,pp\. 88–104\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.blackboxnlp-1.6),[Link](https://aclanthology.org/2024.blackboxnlp-1.6/)Cited by:[§1](https://arxiv.org/html/2608.13591#S1.p4.1),[§2](https://arxiv.org/html/2608.13591#S2.p3.1)\.
- Z\. Ji, N\. Lee, R\. Frieske, T\. Yu, D\. Su, Y\. Xu, E\. Ishii, Y\. J\. Bang, A\. Madotto, and P\. Fung \(2023\)Survey of hallucination in natural language generation\.ACM Computing Surveys55\(12\),pp\. 1–38\.External Links:[Document](https://dx.doi.org/10.1145/3571730),[Link](https://doi.org/10.1145/3571730)Cited by:[§1](https://arxiv.org/html/2608.13591#S1.p1.1)\.
- S\. Joo, K\. Min, J\. Koo, and K\. Jung \(2025\)Black\-box hallucination detection via consistency under the uncertain expression\.arXiv preprint arXiv:2509\.21999\.External Links:2509\.21999Cited by:[§1](https://arxiv.org/html/2608.13591#S1.p1.1),[§1](https://arxiv.org/html/2608.13591#S1.p4.1),[§2](https://arxiv.org/html/2608.13591#S2.p3.1)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan, A\. Jones, N\. Schiefer, N\. Joseph, N\. DasSarma, S\. McCandlish, C\. Olsson,et al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.External Links:2207\.05221,[Link](https://arxiv.org/abs/2207.05221)Cited by:[§1](https://arxiv.org/html/2608.13591#S1.p4.1),[§2](https://arxiv.org/html/2608.13591#S2.p2.1)\.
- R\. E\. Kalman \(1960\)A new approach to linear filtering and prediction problems\.Journal of Basic Engineering82\(1\),pp\. 35–45\.External Links:[Document](https://dx.doi.org/10.1115/1.3662552)Cited by:[§3](https://arxiv.org/html/2608.13591#S3.p1.3)\.
- P\. Manakul, A\. Liusie, and M\. Gales \(2023\)SelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp\. 9004–9017\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.557),[Link](https://aclanthology.org/2023.emnlp-main.557/)Cited by:[§1](https://arxiv.org/html/2608.13591#S1.p1.1),[§1](https://arxiv.org/html/2608.13591#S1.p4.1),[§2](https://arxiv.org/html/2608.13591#S2.p3.1)\.
- A\. Okutomi \(2026\)OSF artifact repository for stable miscalibration in large language models\.Note:Available at[https://osf\.io/ndz69/](https://osf.io/ndz69/?view_only=0920b52c74af4ded84f0086d5d773457)Reproducibility artifactCited by:[§4](https://arxiv.org/html/2608.13591#S4.p1.1)\.
- S\. Tayebati, D\. Kumar, N\. Darabi, D\. Jayasuriya, R\. Krishnan, and A\. R\. Trivedi \(2025\)Learning conformal abstention policies for adaptive risk management in large language and vision\-language models\.arXiv preprint arXiv:2502\.06884\.External Links:2502\.06884,[Link](https://arxiv.org/abs/2502.06884)Cited by:[§1](https://arxiv.org/html/2608.13591#S1.p4.1),[§2](https://arxiv.org/html/2608.13591#S2.p1.1)\.
- J\. Wang, Y\. Zhou, S\. Devic, and D\. Fu \(2026\)Are LLM decisions faithful to verbal confidence?\.arXiv preprint arXiv:2601\.07767\.External Links:2601\.07767,[Link](https://arxiv.org/abs/2601.07767)Cited by:[§1](https://arxiv.org/html/2608.13591#S1.p4.1),[§2](https://arxiv.org/html/2608.13591#S2.p2.1)\.
- B\. Wen, J\. Yao, S\. Feng, C\. Xu, Y\. Tsvetkov, B\. Howe, and L\. L\. Wang \(2025\)Know your limits: a survey of abstention in large language models\.Transactions of the Association for Computational Linguistics13,pp\. 529–556\.External Links:[Link](https://aclanthology.org/2025.tacl-1.26/)Cited by:[§1](https://arxiv.org/html/2608.13591#S1.p4.1),[§2](https://arxiv.org/html/2608.13591#S2.p1.1)\.
- J\. Xiao, B\. Hou, Z\. Wang, R\. Jin, Q\. Long, W\. J\. Su, and L\. Shen \(2025\)Restoring calibration for aligned large language models: a calibration\-aware fine\-tuning approach\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 68364–68390\.External Links:[Link](https://proceedings.mlr.press/v267/xiao25b.html)Cited by:[§1](https://arxiv.org/html/2608.13591#S1.p4.1),[§2](https://arxiv.org/html/2608.13591#S2.p2.1)\.