Auditing Conformal Prediction for Language Models Under Inference-Time Model Shift

# Auditing Conformal Prediction for Language Models Under Inference-Time Model Shift
Source: [https://arxiv.org/html/2610.28937](https://arxiv.org/html/2610.28937)
Achref Doula1,2,Max Mühlhäuser1,2 1Tongji University, Shanghai, China 2Technical University of Darmstadt, Darmstadt, Germany††thanks:Corresponding author:achref\.doula@gmail\.com*Author preprint**Submitted 01\.05\.2026, accepted 24\.09\.2026,*

###### Abstract

Conformal prediction \(CP\) for language models is typically evaluated under a*matched*protocol, in which calibration and deployment use the same post\-training inference configuration, such as quantization mode and inference temperature\. In deployment, these choices often differ from those used at calibration to meet memory or throughput constraints or serving policies\. We argue that this exposes a limitation in current CP evaluation protocols for language models\. We formalize the underlying phenomenon as*inference\-time model shift*: a change in the conformal score distribution induced by inference configuration rather than by the input–label distribution, and theoretically characterize how it breaks split\-conformal exchangeability even when the task distribution and trained model are unchanged\. We then evaluate its consequences across88language models,44finite\-label benchmarks, and1212inference configurations\. We find that matched evaluation can substantially overstate deployment\-time coverage, and that the effect is heterogeneous across models and datasets\. We argue that inference configuration should be reported as a first\-class experimental axis in CP evaluations for language\-model systems\.

## 1Introduction

Conformal prediction \(CP\) is a distribution\-free framework for uncertainty quantification: given a trained predictor and a held\-out calibration set, it returns a prediction set guaranteed to contain the true label with a user\-specified probability1−α1\-\\alphain finite samples[Vovk et al\. \(2005\)](https://arxiv.org/html/2610.28937#bib.bib5);[Angelopoulos and Bates \(2023\)](https://arxiv.org/html/2610.28937#bib.bib2)\. The guarantee is obtained by calibrating a threshold on nonconformity scores, which are real\-valued scores of disagreement between the predictor’s output and a candidate label, and rests on an important assumption: that calibration and test scores are exchangeable\. A substantial literature studies how this assumption fails when the data\-generating distribution shifts, including covariate shift, label shift, online drift, and other forms of non\-exchangeability[Tibshirani et al\. \(2019\)](https://arxiv.org/html/2610.28937#bib.bib11);[Gibbs and Candès \(2021\)](https://arxiv.org/html/2610.28937#bib.bib12);[Oliveira et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib15)\. In each case, the source of failure is a change in the joint distributionPX​YP\_\{XY\}between calibration and deployment\.

For language models, calibration and test scores depend not only onPX​YP\_\{XY\}but also on the*inference configuration*used to generate the predictive distribution from which scores are computed: post\-training choices that shape the predictive distribution without altering the trained model itself, including numerical representation \(e\.g\., quantization mode\) and inference parameters \(e\.g\., temperature\)\. In deployed systems, the inference configuration is not always held fixed for reasons unrelated to the data or the model: quantization is changed to satisfy memory or throughput constraints, and inference temperature varies with system defaults, serving policies, or user\-facing controls\. Existing CP evaluations for language models, however, typically adopt a*matched*protocol, in which calibration and test points are generated under the same inference configuration; reported coverage then estimates the reliability of one fixed score\-generating procedure, a valid target only when the deployment configuration is fixed and known in advance[Kumar et al\. \(2023\)](https://arxiv.org/html/2610.28937#bib.bib42);[Quach et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib18);[Mohri and Hashimoto \(2024\)](https://arxiv.org/html/2610.28937#bib.bib43);[Cherian et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib19);[Su et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib20);[Ulmer et al\. \(2024b\)](https://arxiv.org/html/2610.28937#bib.bib44);[Gui et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib22);[Zhong et al\. \(2025\)](https://arxiv.org/html/2610.28937#bib.bib30)\.

When the calibration and deployment configurations differ, even with the trained model and the task distribution unchanged, calibration and test scores are generated by different mechanisms, and the exchangeability assumption underlying split\-conformal validity need not hold\. We refer to this phenomenon as*inference\-time model shift*: a change in the conformal score distribution induced by inference configuration rather than by the input–label distribution\. It is distinct from classical dataset shift becausePX​YP\_\{XY\}is fixed; the shift lives in the mapping from the input to the resulting predictive distribution, and hence in the induced score distribution\.

Figure 1:MMLU\-Prowith deployment fixed tonf4​@​T=0\.5\\texttt\{nf4\}@T\{=\}0\.5and nominal target0\.90\.9\. Matched calibration tracks target across eight model families; mismatched calibration, differing only in inference configuration between calibration and deployment, produces substantial coverage degradation\.The empirical consequences of*inference\-time model shift*are nontrivial\. Figure[1](https://arxiv.org/html/2610.28937#S1.F1)reports coverage onMMLU\-Prowhen deployment is fixed to a single configuration \(nf4quantization, temperatureT=0\.5T\{=\}0\.5\) and calibration uses, in turn, each of the twelve configurations in a3×43\{\\times\}4grid of quantization modes and temperatures\. Matched calibration tracks the nominal0\.900\.90target across eight small language models, with mean coverage0\.9020\.902\. Mismatched calibration produces mean coverage0\.6390\.639and worst\-case coverage0\.5080\.508\. The task distribution and the trained model are unchanged across all twelve calibrations; the degradation is attributable solely to the change in score\-generating procedure between calibration and deployment\. Matched evaluation, by construction, cannot surface this degradation, which exposes a limitation of matched\-only CP evaluation protocols for language models\.

We characterize this setting theoretically and audit it empirically\. We formalize inference\-time model shift as a property of the conformal score distribution, and we show that matched validity does not transport across inference configurations once score\-level exchangeability is broken\. The empirical audit covers88language models on44finite\-label benchmarks \(multiple\-choice QA and classification\) across a1212\-configuration grid\. We restrict the empirical scope to discrete\-label tasks because they admit a controlled comparison: candidate labels, predictive distributions, and conformal scores are directly comparable across inference configurations, isolating inference\-configuration mismatch from confounds introduced by open\-ended generation\.

Two findings emerge\. First, matched evaluation can substantially overstate deployment\-time coverage, with the failures concentrated in low\-temperature regimes and amplified, but not solely caused, by aggressive quantization\. Second, and more consequential for evaluation practice, the effect is*heterogeneous*: the same type of calibration–deployment mismatch produces severe undercoverage in one setting and mild degradation, or even overcoverage, in another\. This implies that inference configuration is an experimental axis on the same footing as the dataset and the model\.

Contributions\.We make the following contributions\.

- •We identify a limitation in current CP evaluation protocols for language models and formalize the underlying phenomenon as*inference\-time model shift*, a distinct source of non\-exchangeability characterized by a change in the score\-generating procedure under a fixed task distribution, under which matched validity is not transportable across inference configurations\.
- •We provide an empirical audit across88language models,44finite\-label benchmarks, and1212inference configurations, finding that mismatch effects are large but neither monotone in configuration nor consistent across models and datasets, and propose an evaluation protocol that reports matched validity alongside mismatch sensitivity and deployment\-assumption\-specific baselines\.

## 2Related Work

#### Conformal prediction under shift and non\-exchangeability\.

CP provides distribution\-free uncertainty quantification under exchangeability, with modern developments covering regression, classification, adaptive coverage, and broader risk\-controlled prediction sets[Doula et al\. \(2025\)](https://arxiv.org/html/2610.28937#bib.bib6);[Lei et al\. \(2018\)](https://arxiv.org/html/2610.28937#bib.bib1);[Sadinle et al\. \(2019\)](https://arxiv.org/html/2610.28937#bib.bib7);[Romano et al\. \(2020\)](https://arxiv.org/html/2610.28937#bib.bib8);[Angelopoulos et al\. \(2021\)](https://arxiv.org/html/2610.28937#bib.bib9);[Angelopoulos et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib10)\. A substantial literature studies what happens when the matched i\.i\.d\. setting fails, including covariate shift, label shift, online distribution drift, and other forms of non\-exchangeability[Doula \(2026\)](https://arxiv.org/html/2610.28937#bib.bib4);[Tibshirani et al\. \(2019\)](https://arxiv.org/html/2610.28937#bib.bib11);[Gibbs and Candès \(2021\)](https://arxiv.org/html/2610.28937#bib.bib12);[Gibbs and Candès \(2024\)](https://arxiv.org/html/2610.28937#bib.bib13);[Podkopaev and Ramdas \(2021\)](https://arxiv.org/html/2610.28937#bib.bib14);[Prinster et al\. \(2022\)](https://arxiv.org/html/2610.28937#bib.bib16);[Oliveira et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib15);[Kasa et al\. \(2025\)](https://arxiv.org/html/2610.28937#bib.bib17)\. These works are closest in spirit to ours because they analyze when standard conformal guarantees degrade outside matched calibration and test conditions\. However, their source of failure is typically a change in the data\-generating distribution or dependence structure\. In contrast, we keep the task distribution fixed and study a different source of non\-exchangeability: deployment\-time changes in inference configuration\. The same checkpoint and the same inputs can therefore induce different conformal score distributions under different temperatures or quantization modes\.

#### Conformal prediction and uncertainty for language models\.

Recent work has brought CP to language and foundation models, including multiple\-choice question answering[Kumar et al\. \(2023\)](https://arxiv.org/html/2610.28937#bib.bib42), conformal language modeling for generated candidate sets[Quach et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib18), conformal factuality and claim filtering[Mohri and Hashimoto \(2024\)](https://arxiv.org/html/2610.28937#bib.bib43);[Cherian et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib19), black\-box or API\-only settings[Su et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib20), correctness\-coverage guarantees for open\-ended generation[Wang et al\. \(2024b\)](https://arxiv.org/html/2610.28937#bib.bib21), non\-exchangeable conformal language generation[Ulmer et al\. \(2024b\)](https://arxiv.org/html/2610.28937#bib.bib44), selective uncertainty testing[Wang et al\. \(2025\)](https://arxiv.org/html/2610.28937#bib.bib45), and trust\- or alignment\-oriented selection for foundation\-model outputs[Gui et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib22)\. These works establish CP as a promising tool for language\-model reliability, but their evaluation targets differ from ours: they typically calibrate and evaluate under a fixed inference setup, or study other sources of uncertainty and non\-exchangeability\. Our focus is the evaluation protocol itself: whether a conformal predictor calibrated under one post\-training inference configuration remains reliable when deployment uses another\. Appendix[B](https://arxiv.org/html/2610.28937#A2)provides a more detailed summary of this distinction\.

#### Calibration, temperature, and quantized language models\.

A parallel literature studies calibration and uncertainty estimation for neural networks and language models\. Temperature scaling remains a standard post\-hoc calibration baseline[Guo et al\. \(2017\)](https://arxiv.org/html/2610.28937#bib.bib23), and calibration is also central for pretrained transformers and question\-answering language models[Desai and Durrett \(2020\)](https://arxiv.org/html/2610.28937#bib.bib24);[Jiang et al\. \(2021\)](https://arxiv.org/html/2610.28937#bib.bib25)\. Recent work further studies uncertainty estimation for free\-form or long\-form LLM generations[Ulmer et al\. \(2024a\)](https://arxiv.org/html/2610.28937#bib.bib26);[Huang et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib27)\. On the systems side, quantization has become a standard deployment mechanism for LLM serving, with work on low\-bit inference and quantization\-aware training[Lin et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib28);[Liu et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib29); recent work also shows that quantization can materially affect calibration itself[Zhong et al\. \(2025\)](https://arxiv.org/html/2610.28937#bib.bib30)\. This literature motivates temperature and quantization as practically relevant inference axes, but it does not formulate calibration\-to\-deployment inference mismatch as a distinct conformal\-validity problem\. Our contribution is to connect these deployment choices to exchangeability of conformal scores and to treat inference configuration as part of the CP evaluation design\.

## 3A Theoretical Characterization of Inference\-Time Model Shift

Standard matched CP evaluation estimates reliability for a fixed inference pipeline: calibration and deployment use the same score\-generating procedure\. In deployed language\-model systems, however, the inference configuration may change after calibration\. We formalize this mismatch and show why matched validity need not describe deployment\-time reliability even when the task distribution is unchanged\. We consider classification and multiple\-choice tasks with finite candidate label set𝒴⁡\(x\)\\mathcal\{Y\}\(x\)\. Let\(X,Y\)∼PX​Y\(X,Y\)\\sim P\_\{XY\}denote the task distribution, assumed unchanged between calibration and deployment\. At inference time, predictions are generated under a configurationC∈𝒞C\\in\\mathcal\{C\}, where𝒞=𝒬×𝒯\\mathcal\{C\}=\\mathcal\{Q\}\\times\\mathcal\{T\}is the product of a finite set of quantization modes𝒬\\mathcal\{Q\}and temperatures𝒯\\mathcal\{T\}\. For eachc∈𝒞c\\in\\mathcal\{C\}, the model induces a predictive distributionπ\(c\)​\(x\)∈Δ\|𝒴⁡\(x\)\|\\pi^\{\(c\)\}\(x\)\\in\\Delta^\{\|\\mathcal\{Y\}\(x\)\|\}\. Given a nonconformity scoreA:Δ\|𝒴⁡\(x\)\|×𝒴⁡\(x\)→ℝA:\\Delta^\{\|\\mathcal\{Y\}\(x\)\|\}\\times\\mathcal\{Y\}\(x\)\\to\\mathbb\{R\}, the score induced under configurationccisV\(c\):=A⁡\(π\(c\)​\(X\),Y\)V^\{\(c\)\}:=A\(\\pi^\{\(c\)\}\(X\),Y\), and we writeFcF\_\{c\}for the distribution ofV\(c\)V^\{\(c\)\}\.

###### Definition 3\.1\(Inference\-time model shift\)\.

Inference\-time model shift occurs if there existc,c′∈𝒞c,c^\{\\prime\}\\in\\mathcal\{C\}such thatFc≠Fc′F\_\{c\}\\neq F\_\{c^\{\\prime\}\}\. Equivalently, the conformal score distribution changes with inference configuration even though the task distributionPX​YP\_\{XY\}is fixed\.

This setting is distinct from classical dataset shift\. The input distribution is fixed\. The shift arises from the mechanism that maps an inputxxto a predictive distributionπ\(c\)​\(x\)\\pi^\{\(c\)\}\(x\), and hence to a conformal score\. At calibration time, we assume access to tuples\(Xi,Yi,Ci,πi\)\(X\_\{i\},Y\_\{i\},C\_\{i\},\\pi\_\{i\}\)withπi=π\(Ci\)​\(Xi\)\\pi\_\{i\}=\\pi^\{\(C\_\{i\}\)\}\(X\_\{i\}\), so that the score\-generating configuration is observed\. At test time, the conformal layer receives only a predictive vectorπ=π\(C\)​\(X\)\\pi=\\pi^\{\(C\)\}\(X\), while the underlying configurationCCmay be latent\.

To make the evaluation target explicit, letS^ccal\\widehat\{S\}\_\{c\_\{\\mathrm\{cal\}\}\}denote a conformal predictor calibrated under configurationccalc\_\{\\mathrm\{cal\}\}, and define

Cov⁡\(ccal→cdep\):=ℙ⁡\(Y∈S^ccal​\(X,π\(cdep\)​\(X\)\)\)\.\\mathrm\{Cov\}\(c\_\{\\mathrm\{cal\}\}\\\!\\to\\\!c\_\{\\mathrm\{dep\}\}\):=\\mathbb\{P\}\\\!\\left\(Y\\in\\widehat\{S\}\_\{c\_\{\\mathrm\{cal\}\}\}\(X;\\pi^\{\(c\_\{\\mathrm\{dep\}\}\)\}\(X\)\)\\right\)\.Matched evaluation estimatesCov⁡\(c→c\)\\mathrm\{Cov\}\(c\\\!\\to\\\!c\)for a fixed configurationcc\. Many deployments, however, care about reliability whencdepc\_\{\\mathrm\{dep\}\}may differ fromccalc\_\{\\mathrm\{cal\}\}\.

The consequence for split conformal prediction is immediate\. If calibration is performed underccalc\_\{\\mathrm\{cal\}\}and deployment undercdepc\_\{\\mathrm\{dep\}\}, then calibration scores have distributionFccalF\_\{c\_\{\\mathrm\{cal\}\}\}while test scores have distributionFcdepF\_\{c\_\{\\mathrm\{dep\}\}\}\. WheneverFccal≠FcdepF\_\{c\_\{\\mathrm\{cal\}\}\}\\neq F\_\{c\_\{\\mathrm\{dep\}\}\}, calibration and test scores are generally not exchangeable, so the usual split\-conformal proof no longer applies\.

###### Proposition 3\.2\(Matched validity is not transportable across inference configurations\. Proof in Appendix[A\.2](https://arxiv.org/html/2610.28937#A1.SS2)\.\)\.

Consider split conformal prediction with scoreAA\. If calibration scores are generated underccalc\_\{\\mathrm\{cal\}\}and test scores undercdepc\_\{\\mathrm\{dep\}\}, andFccal≠FcdepF\_\{c\_\{\\mathrm\{cal\}\}\}\\neq F\_\{c\_\{\\mathrm\{dep\}\}\}, then calibration and test scores are, in general, not exchangeable, and the usual split\-conformal coverage guarantee need not hold\.

The source of failure is therefore not a change in the task distribution, but a change in the score\-generating mechanism itself\. Matched evaluation remains appropriate for the special case of a fixed known deployment configuration; the problem is that it need not answer the broader deployment question when inference configuration can vary after calibration\. We present a more complete theoretical analysis in Appendix[A](https://arxiv.org/html/2610.28937#A1)\.

## 4Reliability Targets Under Deployment Uncertainty

Once inference configuration is allowed to vary after calibration, “is CP valid?” is no longer a single question\. The relevant reliability target depends on the deployment assumption\. If deployment configuration is fixed and known in advance, matched calibration is the correct target\. If deployment follows a stable mixture over configurations, mixture\-marginal validity is the relevant target; we include pooled calibration as a baseline for that setting and defer its formal treatment to the appendix\. The main text focuses on the stronger regime in which deployment configuration is unobserved at test time and reliability should remain meaningful for individual latent deployment states\.

We study two conservative baselines for that regime\.MaxThrprovides worst\-case protection across configurations by hedging against the largest configuration\-specific threshold\.Latent\-Robustuses the observed predictive vector to restrict that hedge to a conformal set of plausible latent configurations\. These baselines are not intended as universally superior replacements for matched CP; they correspond to stricter deployment targets and, in our default construction,Latent\-Robustcan be even more conservative because the total error budget is split across two stages\.

### 4\.1Max\-threshold calibration

A simple robust baseline is to calibrate separately for each configuration and then use the largest resulting threshold at deployment\. Let𝒟cal=\{\(Xi,Yi,Ci,πi\)\}i=1n\\mathcal\{D\}\_\{\\mathrm\{cal\}\}=\\\{\(X\_\{i\},Y\_\{i\},C\_\{i\},\\pi\_\{i\}\)\\\}\_\{i=1\}^\{n\}, and for eachc∈𝒞c\\in\\mathcal\{C\}let

𝒟cal\(c\)=\{\(Xi,Yi,πi\)∈𝒟cal:Ci=c\}\.\\mathcal\{D\}\_\{\\mathrm\{cal\}\}^\{\(c\)\}=\\\{\(X\_\{i\},Y\_\{i\},\\pi\_\{i\}\)\\in\\mathcal\{D\}\_\{\\mathrm\{cal\}\}:C\_\{i\}=c\\\}\.From𝒟cal\(c\)\\mathcal\{D\}\_\{\\mathrm\{cal\}\}^\{\(c\)\}, compute a configuration\-specific split\-conformal thresholdq^c\\hat\{q\}\_\{c\}at level1−α1\-\\alphausing the scoresVi\(c\):=A⁡\(πi,Yi\)V\_\{i\}^\{\(c\)\}:=A\(\\pi\_\{i\},Y\_\{i\}\)\. TheMaxThrpredictor then uses the global thresholdq^max:=maxc∈𝒞⁡q^c\\hat\{q\}\_\{\\max\}:=\\max\_\{c\\in\\mathcal\{C\}\}\\hat\{q\}\_\{c\}and returns

S^max​\(x\)=\{y∈𝒴⁡\(x\):A⁡\(π,y\)≤q^max\}\.\\hat\{S\}\_\{\\max\}\(x\)=\\\{y\\in\\mathcal\{Y\}\(x\):A\(\\pi,y\)\\leq\\hat\{q\}\_\{\\max\}\\\}\.
This construction is deliberately conservative: it ignores any structure in the observed predictive vector and protects against deployment uncertainty by hedging against the largest configuration\-specific threshold\.

###### Proposition 4\.1\(Max\-threshold validity, Proof in Appendix[A\.5](https://arxiv.org/html/2610.28937#A1.SS5)\.\)\.

Assume that, for eachc∈𝒞c\\in\\mathcal\{C\}, the calibration data in𝒟cal\(c\)\\mathcal\{D\}\_\{\\mathrm\{cal\}\}^\{\(c\)\}and a test point generated underccare exchangeable with respect toAA, so that the configuration\-specific predictor based onq^c\\hat\{q\}\_\{c\}is valid at level1−α1\-\\alpha\. Then

ℙ⁡\(Y∈S^max​\(X\)∣C=c\)≥1−αfor every​c∈𝒞\.\\mathbb\{P\}\(Y\\in\\hat\{S\}\_\{\\max\}\(X\)\\mid C=c\)\\geq 1\-\\alpha\\qquad\\text\{for every \}c\\in\\mathcal\{C\}\.Consequently,S^max\\hat\{S\}\_\{\\max\}is also valid marginally for any deployment distribution onCC\.

### 4\.2Latent\-robust calibration

MaxThrprotects against latent deployment variation with a single global hedge\. Our second baseline retains configuration\-specific calibration while using the observed predictive vector to infer which deployment configurations are plausible for the current test point\.

We split the calibration data into two disjoint parts,𝒟cal=𝒟1∪𝒟2\\mathcal\{D\}\_\{\\mathrm\{cal\}\}=\\mathcal\{D\}\_\{1\}\\cup\\mathcal\{D\}\_\{2\}\. In Stage 1, we perform configuration\-conditional calibration: for eachc∈𝒞c\\in\\mathcal\{C\}, let

𝒟1\(c\)=\{\(Xi,Yi,πi\)∈𝒟1:Ci=c\}\\mathcal\{D\}\_\{1\}^\{\(c\)\}=\\\{\(X\_\{i\},Y\_\{i\},\\pi\_\{i\}\)\\in\\mathcal\{D\}\_\{1\}:C\_\{i\}=c\\\}and compute a thresholdq^c\\hat\{q\}\_\{c\}at level1−αstage11\-\\alpha\_\{\\mathrm\{stage1\}\}from the scoresVi\(c\):=A⁡\(πi,Yi\)V\_\{i\}^\{\(c\)\}:=A\(\\pi\_\{i\},Y\_\{i\}\)\. This yields configuration\-specific prediction sets

S^c​\(x\)=\{y∈𝒴⁡\(x\):A⁡\(π,y\)≤q^c\}\.\\hat\{S\}\_\{c\}\(x\)=\\\{y\\in\\mathcal\{Y\}\(x\):A\(\\pi,y\)\\leq\\hat\{q\}\_\{c\}\\\}\.
In Stage 2, the deployment configuration is treated as latent\. We therefore construct from the observed predictive vector a label\-free signature that is informative about configuration identity\. Our primary signature is

Z⁡\(x\)=\(\|S^c​\(x\)\|\)c∈𝒞,Z\(x\)=\\big\(\|\\hat\{S\}\_\{c\}\(x\)\|\\big\)\_\{c\\in\\mathcal\{C\}\},the vector of hypothetical set sizes induced by the Stage\-1 thresholds\. Using𝒟2\\mathcal\{D\}\_\{2\}, we build a conformal set predictorΓ^​\(Z​\(x\)\)⊆𝒞\\hat\{\\Gamma\}\(Z\(x\)\)\\subseteq\\mathcal\{C\}such that the true deployment configuration is contained inΓ^​\(Z​\(x\)\)\\hat\{\\Gamma\}\(Z\(x\)\)with probability at least1−βstage21\-\\beta\_\{\\mathrm\{stage2\}\}\.

The final threshold is then chosen adaptively as

q^Γ​\(x\)=maxc∈Γ^​\(Z​\(x\)\)⁡q^c,\\hat\{q\}\_\{\\Gamma\}\(x\)=\\max\_\{c\\in\\hat\{\\Gamma\}\(Z\(x\)\)\}\\hat\{q\}\_\{c\},with fallbackmaxc∈𝒞⁡q^c\\max\_\{c\\in\\mathcal\{C\}\}\\hat\{q\}\_\{c\}ifΓ^​\(Z​\(x\)\)=∅\\hat\{\\Gamma\}\(Z\(x\)\)=\\varnothing, and the final prediction set is

S^Γ​\(x\)=\{y∈𝒴⁡\(x\):A⁡\(π,y\)≤q^Γ​\(x\)\}\.\\hat\{S\}\_\{\\Gamma\}\(x\)=\\\{y\\in\\mathcal\{Y\}\(x\):A\(\\pi,y\)\\leq\\hat\{q\}\_\{\\Gamma\}\(x\)\\\}\.
###### Theorem 4\.2\(Latent\-robust validity\. Proof in Appendix[A\.6](https://arxiv.org/html/2610.28937#A1.SS6)\.\)\.

Assume \(i\) for eachc∈𝒞c\\in\\mathcal\{C\}, the Stage\-1 calibration data in𝒟1\(c\)\\mathcal\{D\}\_\{1\}^\{\(c\)\}and a test point generated underccare exchangeable with respect toAA, so thatS^c\\hat\{S\}\_\{c\}is valid at level1−αstage11\-\\alpha\_\{\\mathrm\{stage1\}\}; and \(ii\) the Stage\-2 calibration tuples in𝒟2\\mathcal\{D\}\_\{2\}and the test tuple are exchangeable with respect to\(C,Z\)\(C,Z\), so thatℙ⁡\(C∈Γ^​\(Z⁡\(X\)\)\)≥1−βstage2\\mathbb\{P\}\(C\\in\\hat\{\\Gamma\}\(Z\(X\)\)\)\\geq 1\-\\beta\_\{\\mathrm\{stage2\}\}\. Then

ℙ⁡\(Y∈S^Γ​\(X\)\)≥1−αstage1−βstage2\.\\mathbb\{P\}\(Y\\in\\hat\{S\}\_\{\\Gamma\}\(X\)\)\\geq 1\-\\alpha\_\{\\mathrm\{stage1\}\}\-\\beta\_\{\\mathrm\{stage2\}\}\.

The key distinction fromMaxThris that the hedge is no longer global\. However, this adaptivity should not be confused with reduced conservativeness\. Under our default two\-stage construction, part of the total error budget is allocated to latent\-state identification and part to label coverage, soLatent\-Robustcan produce larger prediction sets thanMaxThr\. We therefore treat it as a stricter reference baseline for latent deployment uncertainty rather than as a uniformly better method\.

## 5Evaluation

We study inference\-time model shift as an*evaluation problem*through three questions\. First, can matched evaluation materially misstate deployment\-time coverage under calibration–deployment mismatch? Second, how much of this effect is attributable to temperature and to quantization, respectively? Third, how do calibration procedures aligned with different deployment assumptions compare in coverage under these regimes?

### 5\.1Experimental Setup

Tasks and models\.We evaluate on four discrete\-label benchmarks:CommonsenseQA[Talmor et al\. \(2019\)](https://arxiv.org/html/2610.28937#bib.bib31),DBPedia\-14[Zhang et al\. \(2015\)](https://arxiv.org/html/2610.28937#bib.bib32),MMLU\-Pro[Wang et al\. \(2024a\)](https://arxiv.org/html/2610.28937#bib.bib33), andMobileActions[Google \(2025\)](https://arxiv.org/html/2610.28937#bib.bib34)\. These datasets span multiple\-choice and classification settings and differ substantially in difficulty, label structure, and output uncertainty\. We study eight small language models:Qwen2\.5\-1\.5B,Qwen2\.5\-3B,Qwen2\.5\-7B[Qwen Team \(2024\)](https://arxiv.org/html/2610.28937#bib.bib35),DeepSeek\-R1\-Distill\-Qwen\-7B[Guo et al\. \(2025\)](https://arxiv.org/html/2610.28937#bib.bib41),Gemma\-2\-2B[Gemma Team et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib36),Llama\-3\.2\-3B[Grattafiori et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib37),Phi\-3\-Mini[Abdin et al\. \(2024\)](https://arxiv.org/html/2610.28937#bib.bib38), andMistral\-7B[Jiang et al\. \(2023\)](https://arxiv.org/html/2610.28937#bib.bib40)\.

Inference configurations\.An inference configuration is a pairc=\(q,t\)∈𝒞=𝒬×𝒯c=\(q,t\)\\in\\mathcal\{C\}=\\mathcal\{Q\}\\times\\mathcal\{T\}, where𝒬\\mathcal\{Q\}is the set of quantization modes and𝒯\\mathcal\{T\}is the set of token\-logit temperature values used before candidate\-label scoring\. In the main benchmark, we evaluate the grid𝒬=\{fp16,int8,nf4\},𝒯=\{0\.5,1\.0,1\.5,2\.0\}\\mathcal\{Q\}=\\\{\\texttt\{fp16\},\\texttt\{int8\},\\texttt\{nf4\}\\\},\\;\\mathcal\{T\}=\\\{0\.5,1\.0,1\.5,2\.0\\\}, yielding\|𝒞\|=12\|\\mathcal\{C\}\|=12inference configurations\. Appendix[C\.4](https://arxiv.org/html/2610.28937#A3.SS4)gives the implementation details for the quantization and temperature procedures\.

CP Setup\.In the main experiments, we use theLACnonconformity score withα=0\.1\\alpha=0\.1\. Results for additional nonconformity scores, RAPS[Angelopoulos et al\. \(2021\)](https://arxiv.org/html/2610.28937#bib.bib9)and APS[Romano et al\. \(2020\)](https://arxiv.org/html/2610.28937#bib.bib8), are reported in Appendix[F\.1](https://arxiv.org/html/2610.28937#A6.SS1)\.

Data splits\.For each dataset–model–configuration combination, we use disjoint calibration and evaluation sets; exact split sizes are reported in the appendix\. All results are averaged over 3 independent runs with different random splits\. ForLatent\-Robust, the calibration set is further split into Stage 1 and Stage 2 portions of 50% and 50%\. In the main setting, we useαstage1=α/2\\alpha\_\{\\mathrm\{stage1\}\}=\\alpha/2andβstage2=α/2\\beta\_\{\\mathrm\{stage2\}\}=\\alpha/2, so thatαstage1\+βstage2=α\\alpha\_\{\\mathrm\{stage1\}\}\+\\beta\_\{\\mathrm\{stage2\}\}=\\alpha\. Appendix[C\.3](https://arxiv.org/html/2610.28937#A3.SS3)gives the implementation details of the Stage\-2 latent\-configuration predictor\.

Methods compared\.We compare six procedures:

- •Matched: standard split conformal prediction calibrated under the true deployment configuration;
- •Mismatch Mean: standard split conformal prediction, averaged over all off\-diagonal calibration configurations for a fixed deployment configuration;
- •Mismatch Worst: the minimum coverage over all off\-diagonal calibration configurations for a fixed deployment configuration;
- •Pooled: calibration on the pooled configuration mixture;
- •MaxThr: the maximum of configuration\-specific thresholds;
- •Latent\-Robust: the two\-stage conservative construction from Section[4](https://arxiv.org/html/2610.28937#S4)\.

#### Metrics\.

Our primary metric in the main paper is empirical coverage, which measures reliability relative to the nominal target1−α1\-\\alpha\. We defer average prediction\-set size and other efficiency analyses to Appendices[C](https://arxiv.org/html/2610.28937#A3)and[D](https://arxiv.org/html/2610.28937#A4)\. For a fixed deployment configurationcdepc\_\{\\mathrm\{dep\}\}, letCov⁡\(ccal→cdep\)\\mathrm\{Cov\}\(c\_\{\\mathrm\{cal\}\}\\\!\\rightarrow\\\!c\_\{\\mathrm\{dep\}\}\)denote empirical coverage when calibration usesccalc\_\{\\mathrm\{cal\}\}and deployment usescdepc\_\{\\mathrm\{dep\}\}\. We summarize mismatch by

MismatchMean⁡\(cdep\)=1\|𝒞\|−1​∑ccal≠cdepCov⁡\(ccal→cdep\)\\mathrm\{MismatchMean\}\(c\_\{\\mathrm\{dep\}\}\)=\\frac\{1\}\{\|\\mathcal\{C\}\|\-1\}\\sum\_\{c\_\{\\mathrm\{cal\}\}\\neq c\_\{\\mathrm\{dep\}\}\}\\mathrm\{Cov\}\(c\_\{\\mathrm\{cal\}\}\\\!\\rightarrow\\\!c\_\{\\mathrm\{dep\}\}\)and

MismatchWorst⁡\(cdep\)=minccal≠cdep⁡Cov⁡\(ccal→cdep\)\.\\mathrm\{MismatchWorst\}\(c\_\{\\mathrm\{dep\}\}\)=\\min\_\{c\_\{\\mathrm\{cal\}\}\\neq c\_\{\\mathrm\{dep\}\}\}\\mathrm\{Cov\}\(c\_\{\\mathrm\{cal\}\}\\\!\\rightarrow\\\!c\_\{\\mathrm\{dep\}\}\)\.Full benchmark configuration, including dataset split sizes, model versions, score\-function hyperparameters, and latent\-split settings, is given in Appendix[C](https://arxiv.org/html/2610.28937#A3)\. Full results including set size and standard deviations are reported in Appendix[D](https://arxiv.org/html/2610.28937#A4)\. Results with different nonconformity scores \(APS, LAC, RAPS\) are reported in Appendix[F\.1](https://arxiv.org/html/2610.28937#A6.SS1), and results with differentα\\alphavalues are reported in Appendix[F\.2](https://arxiv.org/html/2610.28937#A6.SS2)\.

### 5\.2Experiment 1: Reliability Under Calibration–Deployment Mismatch

We fix deployment to quantizationnf4and temperatureT=0\.5T=0\.5\. Figure[2](https://arxiv.org/html/2610.28937#S5.F2)reports coverage separately for each dataset–model pair, averaged over three runs\. Across all four datasets,Matchedtracks the nominal0\.900\.90target tightly, whileMismatch Worstfalls below target for almost every model\. The robust baselines,MaxThrandLatent\-Robust, restore target coverage in nearly all cells\. As expected, this coverage recovery comes with increased prediction\-set size; detailed efficiency results are reported in Appendix[D](https://arxiv.org/html/2610.28937#A4)\.

MMLU\-Proexhibits the strongest degradation\. All eight models fall below target underMismatch Mean,Mismatch Worst, andPooled; descriptive averages over models drop from0\.9020\.902underMatchedto0\.6390\.639underMismatch Meanand0\.5080\.508underMismatch Worst, withPooledat0\.6860\.686\.CommonsenseQAfollows the same qualitative pattern more mildly: all eight models fall below target under the three mismatch summaries, but the corresponding average drops are smaller \(matched0\.9120\.912; mismatch mean0\.8270\.827; mismatch worst0\.7890\.789; pooled0\.8180\.818\)\.

DBPedia\-14differs in kind rather than degree\. The descriptive averages are comparatively close to target \(0\.901/0\.863/0\.827/0\.8670\.901/0\.863/0\.827/0\.867\), butMismatch Meanspans a wide model\-level range \(0\.7950\.795to0\.9460\.946\): the same summary statistic conceals substantial degradation for some model families and near\-matched behavior for others\.MobileActionsis the most stable dataset, with average mismatch effects within a few points of target, but it is not invariant:Mismatch Worststill falls below target for 6 of 8 models, indicating that even comparatively stable settings retain adverse calibration–deployment pairs\. Full per\-model tables are in Appendix[D\.1](https://arxiv.org/html/2610.28937#A4.SS1)\.

Two observations cut across the four datasets\. First, the gap betweenMismatch MeanandMismatch Worstis consistently large, so any summary that averages over off\-diagonal calibration choices understates the worst\-case reliability cost\. Second, the heterogeneity is structural:MMLU\-ProandCommonsenseQAfail uniformly across models whileDBPedia\-14fails selectively, and a single reporting protocol that fixes one calibration choice cannot capture both regimes\.

Figure 2:Coverage with deployment fixed tonf4​@​T=0\.5\\texttt\{nf4\}@T=0\.5\.Coverage is shown for 6 CP approaches across eight models and four datasets\.
### 5\.3Experiment 2: Quantization Effects at Fixed Deployment Temperature

We next isolate the role of quantization by fixing deployment temperature toT=0\.5T=0\.5and averaging over deployment quantization modes\. Figure[3](https://arxiv.org/html/2610.28937#S5.F3)reports coverage separately for each dataset–model pair\. For each model and run, coverage is first averaged over deployment quantization modes at fixedT=0\.5T=0\.5, and the plotted bar is then averaged over runs\.

The mismatch effect persists\. The qualitative pattern from Experiment 1 is preserved across all four datasets:Matchedtracks target,Mismatch Worstfalls below target for all 8 models on every dataset, andMaxThrandLatent\-Robustrestore coverage\.MMLU\-Proagain shows the largest drop, with descriptive averages over models of0\.9060\.906\(Matched\),0\.6480\.648\(Mismatch Mean\),0\.5200\.520\(Mismatch Worst\), and0\.6950\.695\(Pooled\)—essentially unchanged from the single\-configuration values reported in Experiment 1\.CommonsenseQAlikewise remains uniformly mismatch\-sensitive, with averages0\.912/0\.833/0\.796/0\.8240\.912/0\.833/0\.796/0\.824\.

Heterogeneity also persists\.DBPedia\-14continues to span a wide model\-level range underMismatch Mean\(0\.7770\.777to0\.9520\.952\), so averaging over deployment quantization does not collapse the cross\-model variation\.MobileActionsremains the most stable dataset on average, butMismatch Worststill falls below target for all 8 models, indicating that adverse calibration–deployment pairs survive the within\-run averaging\. We present full per\-model tables in Appendix[D\.2](https://arxiv.org/html/2610.28937#A4.SS2)\.

The results indicate that the degradation observed is not specific to a particular quantization mode\. Low\-temperature deployment alone already defines a regime in which standard CP is substantially more sensitive to calibration–deployment mismatch, regardless of the deployment quantization mode\.

Figure 3:Coverage with deployment temperature fixed toT=0\.5T=0\.5, averaged over deployment quantization modes\.Coverage is shown for all six procedures across eight models and four datasets\.
### 5\.4Experiment 3: Temperature Effects at Fixed Deployment Quantization

Finally, we isolate the role of temperature by fixing deployment quantization tonf4and averaging over deployment temperatures\. This complements Experiment 2 by asking whether the mismatch effect persists when quantization is held fixed and only the inference regime varies\. Figure[4](https://arxiv.org/html/2610.28937#S5.F4)reports coverage separately for each dataset–model pair\. For each model and run, coverage is first averaged over deployment temperatures at fixednf4, and the plotted bar is then averaged over runs\.

The mismatch effect persists, but more selectively than in Experiments 1 and 2\. The clearest sign is thatMismatch Worststill falls below target for all 8 of 8 models on every dataset, whereasMismatch Meannow falls below target on only some datasets: 8 of 8 onMMLU\-Pro, 1 of 8 onCommonsenseQA, and an intermediate count onDBPedia\-14andMobileActions\. Averaging over temperatures attenuates the average mismatch effect for many model–dataset pairs but does not eliminate adverse calibration–deployment pairs\.

MMLU\-Proagain shows the strongest degradation: descriptive averages over models drop from0\.9010\.901\(Matched\) to0\.8420\.842\(Mismatch Mean\) and0\.7190\.719\(Mismatch Worst\)\.DBPedia\-14again shows the strongest cross\-model heterogeneity, withMismatch Meanspanning0\.5670\.567to0\.9220\.922andMismatch Worstspanning0\.2200\.220to0\.8900\.890, a range so wide that some model families are nearly invariant under the averaging while others collapse\. This rules out a clear monotone ordering of inference configurations along which mismatch severity could be predicted\.

With Experiment 2, the results show that neither temperature nor quantization alone accounts for the observed degradation\. Each axis is sufficient to induce mismatch effects when the other is held fixed\. We present further results including full per\-model tables in Appendix[D\.3](https://arxiv.org/html/2610.28937#A4.SS3)\.

Figure 4:Coverage with deployment quantization fixed tonf4, averaged over deployment temperatures\.Coverage is shown for all six procedures across eight models and four datasets\.

## 6Discussion

The central claim of this paper is evaluative: for deployed language\-model systems, inference configuration is part of the validity problem itself, not an implementation detail to be held fixed or averaged away\. Standard CP evaluations ask whether the guarantee holds when calibration and deployment share an inference pipeline\. Our results show this is insufficient\. Even with the task distribution and trained model fixed, changing the inference configuration between calibration and deployment can move the conformal score distribution enough to invalidate split\-conformal validity, and matched\-only evaluation cannot, by construction, surface this failure\.

The empirical picture shows the challenging nature of inference\-time model shift\. In our experiments, low\-temperature deployment is consistently the most mismatch\-sensitive regime, and aggressive quantization can amplify the effect, but the failures admit no monotone ordering over inference configurations\. Their magnitude varies with the dataset and model family, and the direction of the effect can change across settings\.

Implications for evaluation practice\.Our findings motivate a concrete revision of reporting practice\. We argue that evaluations of CP for language models should treat inference configuration as a first\-class experimental axis, on the same footing as the dataset and the model\. Concretely, this entails three reporting steps: \(i\) state the deployment assumption being targeted: fixed configuration, stable mixture, or unobserved at test time, since each implies a different reliability target; \(ii\) report sensitivity to calibration–deployment mismatch along the inference axes that may vary at deployment, rather than matched coverage alone; and \(iii\) where the deployment configuration is not known in advance, report a baseline aligned with the targeted deployment regime\. In the setting studied here, these reporting steps require no new theoretical machinery, but they do require treating inference configuration as part of the experimental design\. We view this as a necessary step toward conformal guarantees that remain meaningful when language\-model systems are actually deployed\.

Limitations\.Our analysis is primarily a characterization rather than a mechanistic account: we identify when mismatch matters but do not provide a predictive theory of when a given configuration change will induce undercoverage, overcoverage, or only mild effects\. The audit is carried out on a finite deployment grid, which captures many realistic serving choices but does not address richer deployment variation, including continuous temperatures or temporal drift in serving policies\.

## 7Conclusion

We identify*inference\-time model shift*as a distinct failure mode for conformal prediction in language models: even under a fixed task distribution, changing deployment\-time inference configuration can alter the conformal score distribution and break the assumptions behind matched calibration\. Across eight small language models, four datasets, and a grid of quantization and temperature settings, this effect leads matched\-only evaluation to substantially overstate deployment\-time reliability\. These results motivate a simple change in practice: conformal evaluations for language models should treat inference configuration as part of the experimental design, not as a fixed implementation detail\.

## References

- \[1\]\(2024\)Phi\-3 technical report: a highly capable language model locally on your phone\.arXiv preprint arXiv:2404\.14219\.External Links:[Link](https://arxiv.org/abs/2404.14219)Cited by:[§5\.1](https://arxiv.org/html/2610.28937#S5.SS1.p1.1)\.
- \[2\]A\. N\. Angelopoulos and S\. Bates\(2023\)Conformal prediction: a gentle introduction\.Foundations and Trends in Machine Learning16\(4\),pp\. 494–591\.Cited by:[§1](https://arxiv.org/html/2610.28937#S1.p1.1)\.
- \[3\]A\. N\. Angelopoulos, S\. Bates, A\. Fisch, L\. Lei, and T\. Schuster\(2024\)Conformal risk control\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=33XGfHLtZg)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1)\.
- \[4\]A\. N\. Angelopoulos, S\. Bates, J\. Malik, and M\. I\. Jordan\(2021\)Uncertainty sets for image classifiers using conformal prediction\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=eNdiU_DbM9)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.28937#S5.SS1.p3.1)\.
- \[5\]J\. J\. Cherian, I\. Gibbs, and E\. J\. Candès\(2024\)Large language model validity via enhanced conformal prediction methods\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[Table 1](https://arxiv.org/html/2610.28937#A2.T1.5.5.1.1.1),[§1](https://arxiv.org/html/2610.28937#S1.p2.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px2.p1.1)\.
- \[6\]S\. Desai and G\. Durrett\(2020\)Calibration of pre\-trained transformers\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),Online,pp\. 295–302\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.21),[Link](https://aclanthology.org/2020.emnlp-main.21/)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px3.p1.1)\.
- \[7\]T\. Dettmers, M\. Lewis, Y\. Belkada, and L\. Zettlemoyer\(2022\)GPT3\.int8\(\): 8\-bit matrix multiplication for transformers at scale\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 30318–30332\.Cited by:[§C\.4](https://arxiv.org/html/2610.28937#A3.SS4.p1.1)\.
- \[8\]T\. Dettmers, A\. Pagnoni, A\. Holtzman, and L\. Zettlemoyer\(2023\)Qlora: efficient finetuning of quantized llms\.Advances in neural information processing systems36,pp\. 10088–10115\.Cited by:[§C\.4](https://arxiv.org/html/2610.28937#A3.SS4.p1.1)\.
- \[9\]A\. Doula, T\. Güdelhöfer, M\. Mühlhäuser, and A\. S\. Guinea\(2025\)Conformal prediction for semantically\-aware autonomous perception in urban environments\.InProceedings of The 8th Conference on Robot Learning,Proceedings of Machine Learning Research, Vol\.270,pp\. 2629–2654\.Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1)\.
- \[10\]A\. Doula\(2026\)Conformal calibration transfer\.InProceedings of the 43rd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.306,pp\. 26291–26318\.External Links:[Link](https://proceedings.mlr.press/v306/doula26a.html)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1)\.
- \[11\]Gemma Team, M\. Riviere, S\. Pathak, P\. G\. Sessa, C\. Hardin, S\. Bhupatiraju, L\. Hussenot, T\. Mesnard,et al\.\(2024\)Gemma 2: improving open language models at a practical size\.arXiv preprint arXiv:2408\.00118\.External Links:[Link](https://arxiv.org/abs/2408.00118)Cited by:[§5\.1](https://arxiv.org/html/2610.28937#S5.SS1.p1.1)\.
- \[12\]I\. Gibbs and E\. J\. Candès\(2021\)Adaptive conformal inference under distribution shift\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 1660–1672\.Cited by:[§1](https://arxiv.org/html/2610.28937#S1.p1.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1)\.
- \[13\]I\. Gibbs and E\. J\. Candès\(2024\)Conformal inference for online prediction with arbitrary distribution shifts\.Journal of Machine Learning Research25\(162\),pp\. 1–36\.Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1)\.
- \[14\]Google\(2025\)Mobile actions: a dataset for on\-device function calling\.Note:[https://huggingface\.co/datasets/google/mobile\-actions](https://huggingface.co/datasets/google/mobile-actions)Hugging Face dataset card, accessed 2026\-04\-21Cited by:[§5\.1](https://arxiv.org/html/2610.28937#S5.SS1.p1.1)\.
- \[15\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.External Links:[Link](https://arxiv.org/abs/2407.21783)Cited by:[§5\.1](https://arxiv.org/html/2610.28937#S5.SS1.p1.1)\.
- \[16\]Y\. Gui, Y\. Jin, and Z\. Ren\(2024\)Conformal alignment: knowing when to trust foundation models with guarantees\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 73884–73919\.Cited by:[Table 1](https://arxiv.org/html/2610.28937#A2.T1.5.10.1.1.1),[§1](https://arxiv.org/html/2610.28937#S1.p2.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px2.p1.1)\.
- \[17\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.70,pp\. 1321–1330\.External Links:[Link](https://proceedings.mlr.press/v70/guo17a.html)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px3.p1.1)\.
- \[18\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang,et al\.\(2025\)DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645,pp\. 633–638\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§5\.1](https://arxiv.org/html/2610.28937#S5.SS1.p1.1)\.
- \[19\]Y\. Huang, Y\. Liu, R\. Thirukovalluru, A\. Cohan, and B\. Dhingra\(2024\)Calibrating long\-form generations from large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Miami, Florida, USA,pp\. 13441–13460\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.785),[Link](https://aclanthology.org/2024.findings-emnlp.785/)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px3.p1.1)\.
- \[20\]A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand,et al\.\(2023\)Mistral 7b\.arXiv preprint arXiv:2310\.06825\.External Links:[Link](https://arxiv.org/abs/2310.06825)Cited by:[§5\.1](https://arxiv.org/html/2610.28937#S5.SS1.p1.1)\.
- \[21\]Z\. Jiang, J\. Araki, H\. Ding, and G\. Neubig\(2021\)How can we know when language models know? on the calibration of language models for question answering\.Transactions of the Association for Computational Linguistics9,pp\. 962–977\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00407),[Link](https://aclanthology.org/2021.tacl-1.57/)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px3.p1.1)\.
- \[22\]K\. Kasa, Z\. Zhang, H\. Yang, and G\. W\. Taylor\(2025\)Adapting prediction sets to distribution shifts without labels\.InProceedings of the Forty\-first Conference on Uncertainty in Artificial Intelligence,Proceedings of Machine Learning Research, Vol\.286,pp\. 1990–2010\.External Links:[Link](https://proceedings.mlr.press/v286/kasa25a.html)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1)\.
- \[23\]B\. Kumar, C\. Lu, G\. Gupta, A\. Palepu, D\. Bellamy, R\. Raskar, and A\. Beam\(2023\)Conformal prediction with large language models for multi\-choice question answering\.arXiv preprint arXiv:2305\.18404\.Cited by:[Table 1](https://arxiv.org/html/2610.28937#A2.T1.5.2.1.1.1),[§1](https://arxiv.org/html/2610.28937#S1.p2.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px2.p1.1)\.
- \[24\]J\. Lei, M\. G’Sell, A\. Rinaldo, R\. J\. Tibshirani, and L\. Wasserman\(2018\)Distribution\-free predictive inference for regression\.Journal of the American Statistical Association113\(523\),pp\. 1094–1111\.External Links:[Document](https://dx.doi.org/10.1080/01621459.2017.1307116)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1)\.
- \[25\]J\. Lin, J\. Tang, H\. Tang, S\. Yang, W\. Chen, W\. Wang, G\. Xiao, X\. Dang, C\. Gan, and S\. Han\(2024\)AWQ: activation\-aware weight quantization for on\-device llm compression and acceleration\.Proceedings of Machine Learning and Systems6\.External Links:[Link](https://proceedings.mlsys.org/paper_files/paper/2024/hash/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px3.p1.1)\.
- \[26\]Z\. Liu, B\. Oguz, C\. Zhao, E\. Chang, P\. Stock, Y\. Mehdad, Y\. Shi, R\. Krishnamoorthi, and V\. Chandra\(2024\)LLM\-QAT: data\-free quantization aware training for large language models\.InFindings of the Association for Computational Linguistics: ACL 2024,Bangkok, Thailand,pp\. 467–484\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.26),[Link](https://aclanthology.org/2024.findings-acl.26/)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px3.p1.1)\.
- \[27\]C\. Mohri and T\. Hashimoto\(2024\)Language models with conformal factuality guarantees\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 36029–36047\.Cited by:[Table 1](https://arxiv.org/html/2610.28937#A2.T1.5.4.1.1.1),[§1](https://arxiv.org/html/2610.28937#S1.p2.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px2.p1.1)\.
- \[28\]R\. I\. Oliveira, P\. Orenstein, T\. Ramos, and J\. V\. Romano\(2024\)Split conformal prediction and non\-exchangeable data\.Journal of Machine Learning Research25\(225\),pp\. 1–38\.Cited by:[§1](https://arxiv.org/html/2610.28937#S1.p1.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1)\.
- \[29\]A\. Podkopaev and A\. Ramdas\(2021\)Distribution\-free uncertainty quantification for classification under label shift\.InProceedings of the Thirty\-Seventh Conference on Uncertainty in Artificial Intelligence,Proceedings of Machine Learning Research, Vol\.161,pp\. 844–853\.External Links:[Link](https://proceedings.mlr.press/v161/podkopaev21a.html)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1)\.
- \[30\]D\. Prinster, A\. Liu, and S\. Saria\(2022\)JAWS: auditing predictive uncertainty under covariate shift\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 35907–35920\.Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1)\.
- \[31\]V\. Quach, A\. Fisch, T\. Schuster, A\. Yala, J\. H\. Sohn, T\. S\. Jaakkola, and R\. Barzilay\(2024\)Conformal language modeling\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=pzUhfQ74c5)Cited by:[Table 1](https://arxiv.org/html/2610.28937#A2.T1.5.3.1.1.1),[§1](https://arxiv.org/html/2610.28937#S1.p2.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px2.p1.1)\.
- \[32\]Qwen Team\(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.External Links:[Link](https://arxiv.org/abs/2412.15115)Cited by:[§5\.1](https://arxiv.org/html/2610.28937#S5.SS1.p1.1)\.
- \[33\]Y\. Romano, M\. Sesia, and E\. J\. Candès\(2020\)Classification with valid and adaptive coverage\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2610.28937#S5.SS1.p3.1)\.
- \[34\]M\. Sadinle, J\. Lei, and L\. Wasserman\(2019\)Least ambiguous set\-valued classifiers with bounded error levels\.Journal of the American Statistical Association114\(525\),pp\. 223–234\.Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1)\.
- \[35\]J\. Su, J\. Luo, H\. Wang, and L\. Cheng\(2024\)API is enough: conformal prediction for large language models without logit\-access\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Miami, Florida, USA,pp\. 978–993\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.54),[Link](https://aclanthology.org/2024.findings-emnlp.54/)Cited by:[Table 1](https://arxiv.org/html/2610.28937#A2.T1.5.6.1.1.1),[§1](https://arxiv.org/html/2610.28937#S1.p2.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px2.p1.1)\.
- \[36\]A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant\(2019\)CommonsenseQA: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),Minneapolis, Minnesota,pp\. 4149–4158\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1421),[Link](https://aclanthology.org/N19-1421/)Cited by:[§5\.1](https://arxiv.org/html/2610.28937#S5.SS1.p1.1)\.
- \[37\]R\. J\. Tibshirani, R\. F\. Barber, E\. J\. Candès, and A\. Ramdas\(2019\)Conformal prediction under covariate shift\.InAdvances in Neural Information Processing Systems,Vol\.32,pp\. 2526–2536\.Cited by:[§1](https://arxiv.org/html/2610.28937#S1.p1.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px1.p1.1)\.
- \[38\]D\. Ulmer, M\. Gubri, H\. Lee, S\. Yun, and S\. Oh\(2024\)Calibrating large language models using their generations only\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Bangkok, Thailand,pp\. 15440–15459\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.824),[Link](https://aclanthology.org/2024.acl-long.824/)Cited by:[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px3.p1.1)\.
- \[39\]D\. Ulmer, C\. Zerva, and A\. F\. T\. Martins\(2024\)Non\-exchangeable conformal language generation with nearest neighbors\.InFindings of the Association for Computational Linguistics: EACL 2024,pp\. 1909–1929\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-eacl.129)Cited by:[Table 1](https://arxiv.org/html/2610.28937#A2.T1.5.8.1.1.1),[§1](https://arxiv.org/html/2610.28937#S1.p2.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px2.p1.1)\.
- \[40\]V\. Vovk, A\. Gammerman, and G\. Shafer\(2005\)Algorithmic learning in a random world\.Springer\.Cited by:[§1](https://arxiv.org/html/2610.28937#S1.p1.1)\.
- \[41\]Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen\(2024\)MMLU\-Pro: a more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems,Vol\.37\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/ad236edc564f3e3156e1b2feafb99a24-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[§5\.1](https://arxiv.org/html/2610.28937#S5.SS1.p1.1)\.
- \[42\]Z\. Wang, J\. Duan, L\. Cheng, Y\. Zhang, Q\. Wang, X\. Shi, K\. Xu, H\. T\. Shen, and X\. Zhu\(2024\)ConU: conformal uncertainty in large language models with correctness coverage guarantees\.InFindings of the Association for Computational Linguistics: EMNLP 2024,Miami, Florida, USA,pp\. 6886–6898\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-emnlp.404),[Link](https://aclanthology.org/2024.findings-emnlp.404/)Cited by:[Table 1](https://arxiv.org/html/2610.28937#A2.T1.5.7.1.1.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px2.p1.1)\.
- \[43\]Z\. Wang, Q\. Wang, Y\. Zhang, T\. Chen, X\. Zhu, X\. Shi, and K\. Xu\(2025\)SConU: selective conformal uncertainty in large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 19052–19075\.Cited by:[Table 1](https://arxiv.org/html/2610.28937#A2.T1.5.9.1.1.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px2.p1.1)\.
- \[44\]X\. Zhang, J\. Zhao, and Y\. LeCun\(2015\)Character\-level convolutional networks for text classification\.Advances in Neural Information Processing Systems28\.External Links:[Link](https://arxiv.org/abs/1509.01626)Cited by:[§5\.1](https://arxiv.org/html/2610.28937#S5.SS1.p1.1)\.
- \[45\]M\. Zhong, G\. Wang, Y\. Chuang, and N\. Zou\(2025\)Quantized can still be calibrated: a unified framework to calibration in quantized large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 30503–30517\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.1473),[Link](https://aclanthology.org/2025.acl-long.1473/)Cited by:[Table 1](https://arxiv.org/html/2610.28937#A2.T1.5.11.1.1.1),[§1](https://arxiv.org/html/2610.28937#S1.p2.1),[§2](https://arxiv.org/html/2610.28937#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix

## Appendix AAdditional Theory and Full Statements

This appendix completes the technical development from the main text\. We first fix notation and recall the split\-conformal threshold used throughout\. We then formalize the reliability targets induced by different deployment assumptions and provide full proofs for the statements in Sections[3](https://arxiv.org/html/2610.28937#S3)and[4](https://arxiv.org/html/2610.28937#S4)\. The purpose of these results is not to characterize all effects of inference\-configuration mismatch, but to make precise which guarantees remain valid under which deployment regimes\.

### A\.1Setup and split\-conformal preliminaries

We use the notation from Sections[3](https://arxiv.org/html/2610.28937#S3)and[4](https://arxiv.org/html/2610.28937#S4)\. For a calibration sample

𝒟=\{\(Xi,Yi,πi\)\}i=1m\\mathcal\{D\}=\\\{\(X\_\{i\},Y\_\{i\},\\pi\_\{i\}\)\\\}\_\{i=1\}^\{m\}and nonconformity scoreA:Δ\|𝒴⁡\(x\)\|×𝒴⁡\(x\)→ℝA:\\Delta^\{\|\\mathcal\{Y\}\(x\)\|\}\\times\\mathcal\{Y\}\(x\)\\to\\mathbb\{R\}, define the calibration scores

Vi:=A\(πi,Yi\),i=1,…,m\.V\_\{i\}:=A\(\\pi\_\{i\},Y\_\{i\}\),\\qquad i=1,\\dots,m\.LetV\(1\)≤⋯≤V\(m\)V\_\{\(1\)\}\\leq\\cdots\\leq V\_\{\(m\)\}denote the order statistics\. For target miscoverageα∈\(0,1\)\\alpha\\in\(0,1\), define

kα​\(m\):=⌈\(m\+1\)​\(1−α\)⌉k\_\{\\alpha\}\(m\):=\\left\\lceil\(m\+1\)\(1\-\\alpha\)\\right\\rceiland the split\-conformal threshold

q^α​\(𝒟\):=\{V\(kα​\(m\)\),if​kα​\(m\)≤m,\+∞,if​kα​\(m\)=m\+1\.\\widehat\{q\}\_\{\\alpha\}\(\\mathcal\{D\}\):=\\begin\{cases\}V\_\{\(k\_\{\\alpha\}\(m\)\)\},&\\text\{if \}k\_\{\\alpha\}\(m\)\\leq m,\\\\\[3\.99994pt\] \+\\infty,&\\text\{if \}k\_\{\\alpha\}\(m\)=m\+1\.\\end\{cases\}Given a predictive vectorπ\\pifor a test inputxx, the associated prediction set is

S^α​\(x,π,𝒟\):=\{y∈𝒴⁡\(x\):A⁡\(π,y\)≤q^α​\(𝒟\)\}\.\\widehat\{S\}\_\{\\alpha\}\(x;\\pi,\\mathcal\{D\}\):=\\\{y\\in\\mathcal\{Y\}\(x\):A\(\\pi,y\)\\leq\\widehat\{q\}\_\{\\alpha\}\(\\mathcal\{D\}\)\\\}\.
We use the following standard split\-conformal fact repeatedly\.

###### Lemma A\.1\(Standard split\-conformal validity\)\.

Let\{\(Xi,Yi,πi\)\}i=1m\\\{\(X\_\{i\},Y\_\{i\},\\pi\_\{i\}\)\\\}\_\{i=1\}^\{m\}be a calibration sample and let\(Xm\+1,Ym\+1,πm\+1\)\(X\_\{m\+1\},Y\_\{m\+1\},\\pi\_\{m\+1\}\)be a test tuple\. If them\+1m\+1tuples are exchangeable andVi=A⁡\(πi,Yi\)V\_\{i\}=A\(\\pi\_\{i\},Y\_\{i\}\), then

ℙ⁡\(Ym\+1∈S^α​\(Xm\+1,πm\+1,𝒟\)\)=ℙ⁡\(Vm\+1≤q^α​\(𝒟\)\)≥1−α\.\\mathbb\{P\}\\\!\\left\(Y\_\{m\+1\}\\in\\widehat\{S\}\_\{\\alpha\}\(X\_\{m\+1\};\\pi\_\{m\+1\},\\mathcal\{D\}\)\\right\)=\\mathbb\{P\}\\\!\\left\(V\_\{m\+1\}\\leq\\widehat\{q\}\_\{\\alpha\}\(\\mathcal\{D\}\)\\right\)\\geq 1\-\\alpha\.

We also use the following monotonicity property\.

###### Lemma A\.2\(Monotonicity in the threshold\)\.

Ifq′≥qq^\{\\prime\}\\geq q, then for every pair\(x,π\)\(x,\\pi\),

\{y∈𝒴⁡\(x\):A⁡\(π,y\)≤q\}⊆\{y∈𝒴⁡\(x\):A⁡\(π,y\)≤q′\}\.\\\{y\\in\\mathcal\{Y\}\(x\):A\(\\pi,y\)\\leq q\\\}\\subseteq\\\{y\\in\\mathcal\{Y\}\(x\):A\(\\pi,y\)\\leq q^\{\\prime\}\\\}\.

###### Proof\.

Immediate from the definition of the prediction set\. ∎

### A\.2Proof of Proposition[3\.2](https://arxiv.org/html/2610.28937#S3.Thmtheorem2)

We restate the proposition for convenience\.

###### Proposition\(Matched validity is not transportable across inference configurations\)\.

Consider split conformal prediction with scoreAA\. If calibration scores are generated underccalc\_\{\\mathrm\{cal\}\}and test scores undercdepc\_\{\\mathrm\{dep\}\}, andFccal≠FcdepF\_\{c\_\{\\mathrm\{cal\}\}\}\\neq F\_\{c\_\{\\mathrm\{dep\}\}\}, then calibration and test scores are, in general, not exchangeable, and the usual split\-conformal coverage guarantee need not hold\.

###### Proof\.

LetV1,…,VnV\_\{1\},\\dots,V\_\{n\}denote the calibration scores generated underccalc\_\{\\mathrm\{cal\}\}, and letVn\+1V\_\{n\+1\}denote the test score generated undercdepc\_\{\\mathrm\{dep\}\}\. By construction,

Vi∼Fccalfor​i=1,…,n,Vn\+1∼Fcdep\.V\_\{i\}\\sim F\_\{c\_\{\\mathrm\{cal\}\}\}\\quad\\text\{for \}i=1,\\dots,n,\\qquad V\_\{n\+1\}\\sim F\_\{c\_\{\\mathrm\{dep\}\}\}\.If the vector\(V1,…,Vn,Vn\+1\)\(V\_\{1\},\\dots,V\_\{n\},V\_\{n\+1\}\)were exchangeable, then all coordinates would have the same marginal distribution\. SinceFccal≠FcdepF\_\{c\_\{\\mathrm\{cal\}\}\}\\neq F\_\{c\_\{\\mathrm\{dep\}\}\}, this is impossible in general\. Hence calibration and test scores are not exchangeable in general\.

The standard split\-conformal proof applies by ranking the test score among the calibration scores, and therefore requires exchangeability of then\+1n\+1scores\. Once exchangeability is absent, that proof no longer applies\.

To see that the coverage guarantee can in fact fail, consider the simple score\-level counterexample in which

V1=⋯=Vn=0a\.s\.,Vn\+1=1a\.s\.V\_\{1\}=\\cdots=V\_\{n\}=0\\quad\\text\{a\.s\.\},\\qquad V\_\{n\+1\}=1\\quad\\text\{a\.s\.\}ThenFccal≠FcdepF\_\{c\_\{\\mathrm\{cal\}\}\}\\neq F\_\{c\_\{\\mathrm\{dep\}\}\}\. For anyα<1\\alpha<1, the split\-conformal threshold computed from the calibration scores isq^α=0\\widehat\{q\}\_\{\\alpha\}=0, while the test score equals11almost surely\. Therefore

ℙ⁡\(Vn\+1≤q^α\)=0,\\mathbb\{P\}\(V\_\{n\+1\}\\leq\\widehat\{q\}\_\{\\alpha\}\)=0,and equivalently the resulting conformal predictor has coverage00\. Thus the usual split\-conformal coverage guarantee need not hold under calibration–deployment mismatch in the score\-generating configuration\. ∎

### A\.3Reliability targets under deployment uncertainty

Once inference configuration may vary after calibration, there is no single universal notion of deployment\-time validity\. The relevant target depends on the deployment assumption\.

###### Definition A\.4\(Fixed\-configuration validity\)\.

Letc∈𝒞c\\in\\mathcal\{C\}be a fixed deployment configuration\. A predictorS^\\widehat\{S\}is valid at level1−α1\-\\alphafor configurationccif

ℙ⁡\(Y∈S^​\(X\)∣C=c\)≥1−α\.\\mathbb\{P\}\\\!\\left\(Y\\in\\widehat\{S\}\(X\)\\mid C=c\\right\)\\geq 1\-\\alpha\.

###### Definition A\.5\(Mixture\-marginal validity\)\.

LetGGbe a deployment distribution on𝒞\\mathcal\{C\}\. A predictorS^\\widehat\{S\}is mixture\-marginally valid at level1−α1\-\\alphaunderGGif

ℙ⁡\(Y∈S^​\(X\)\)≥1−α\\mathbb\{P\}\\\!\\left\(Y\\in\\widehat\{S\}\(X\)\\right\)\\geq 1\-\\alphawhenC∼GC\\sim G\.

###### Definition A\.6\(Configuration\-conditional validity\)\.

A predictorS^\\widehat\{S\}is configuration\-conditionally valid at level1−α1\-\\alphaif

ℙ⁡\(Y∈S^​\(X\)∣C=c\)≥1−αfor every​c∈𝒞\.\\mathbb\{P\}\\\!\\left\(Y\\in\\widehat\{S\}\(X\)\\mid C=c\\right\)\\geq 1\-\\alpha\\qquad\\text\{for every \}c\\in\\mathcal\{C\}\.

These targets are distinct\. Fixed\-configuration validity refers to one known deployment state\. Mixture\-marginal validity averages over a deployment mixture\. Configuration\-conditional validity is stronger: it requires validity for every configuration separately\.

The procedures studied in the main text align with these targets as follows:

- •Matchedtargets fixed\-configuration validity\.
- •Pooledtargets mixture\-marginal validity under a stable deployment mixture\.
- •MaxThrtargets configuration\-conditional validity under the assumptions of Proposition[4\.1](https://arxiv.org/html/2610.28937#S4.Thmtheorem1)\.
- •Latent\-Robust, as analyzed in Theorem[4\.2](https://arxiv.org/html/2610.28937#S4.Thmtheorem2), combines configuration\-specific Stage\-1 calibration with a Stage\-2 conformal localization step and yields a marginal coverage guarantee at level1−αstage1−βstage21\-\\alpha\_\{\\mathrm\{stage1\}\}\-\\beta\_\{\\mathrm\{stage2\}\}\.

The last point is important: althoughLatent\-Robustis motivated by latent deployment uncertainty, the theorem in the main text is an unconditional coverage statement\. It does*not*by itself imply the same configuration\-conditional guarantee as Proposition[4\.1](https://arxiv.org/html/2610.28937#S4.Thmtheorem1); obtaining such a result would require a stronger, configuration\-conditional form of Stage\-2 control\.

The following elementary implication will also be useful\.

###### Lemma A\.7\(Configuration\-conditional validity implies mixture\-marginal validity\)\.

IfS^\\widehat\{S\}satisfies

ℙ⁡\(Y∈S^​\(X\)∣C=c\)≥1−αfor every​c∈𝒞,\\mathbb\{P\}\\\!\\left\(Y\\in\\widehat\{S\}\(X\)\\mid C=c\\right\)\\geq 1\-\\alpha\\qquad\\text\{for every \}c\\in\\mathcal\{C\},thenS^\\widehat\{S\}is mixture\-marginally valid at level1−α1\-\\alphafor any deployment distribution on𝒞\\mathcal\{C\}\.

###### Proof\.

By the law of total probability,

ℙ⁡\(Y∈S^​\(X\)\)=∑c∈𝒞ℙ⁡\(C=c\)​ℙ​\(Y∈S^​\(X\)∣C=c\)≥∑c∈𝒞ℙ⁡\(C=c\)​\(1−α\)=1−α\.\\mathbb\{P\}\(Y\\in\\widehat\{S\}\(X\)\)=\\sum\_\{c\\in\\mathcal\{C\}\}\\mathbb\{P\}\(C=c\)\\,\\mathbb\{P\}\(Y\\in\\widehat\{S\}\(X\)\\mid C=c\)\\geq\\sum\_\{c\\in\\mathcal\{C\}\}\\mathbb\{P\}\(C=c\)\\,\(1\-\\alpha\)=1\-\\alpha\.∎

### A\.4Pooled calibration and mixture\-marginal validity

We now state formally the guarantee targeted by pooled calibration\.

###### Proposition A\.8\(Pooled calibration under a stable deployment mixture\)\.

LetGGbe a deployment distribution on𝒞\\mathcal\{C\}\. Suppose the calibration tuples

\{\(Xi,Yi,Ci,πi\)\}i=1m\\\{\(X\_\{i\},Y\_\{i\},C\_\{i\},\\pi\_\{i\}\)\\\}\_\{i=1\}^\{m\}and the test tuple\(Xm\+1,Ym\+1,Cm\+1,πm\+1\)\(X\_\{m\+1\},Y\_\{m\+1\},C\_\{m\+1\},\\pi\_\{m\+1\}\)are exchangeable under the joint law induced byPX​YP\_\{XY\}, the mixtureGG, and the configuration\-dependent predictive mechanismc↦π\(c\)​\(⋅\)c\\mapsto\\pi^\{\(c\)\}\(\\cdot\)\. LetS^pool\\widehat\{S\}\_\{\\mathrm\{pool\}\}be the split\-conformal predictor obtained by pooling all calibration tuples and ignoring configuration labels when computing the threshold\. Then

ℙ⁡\(Ym\+1∈S^pool​\(Xm\+1\)\)≥1−α\.\\mathbb\{P\}\\\!\\left\(Y\_\{m\+1\}\\in\\widehat\{S\}\_\{\\mathrm\{pool\}\}\(X\_\{m\+1\}\)\\right\)\\geq 1\-\\alpha\.That is, pooled calibration is mixture\-marginally valid under the stable deployment mixtureGG\.

###### Proof\.

LetVi:=A⁡\(πi,Yi\)V\_\{i\}:=A\(\\pi\_\{i\},Y\_\{i\}\)fori=1,…,m\+1i=1,\\dots,m\+1\. Since the tuples are exchangeable andViV\_\{i\}is a measurable function of theii\-th tuple, the pooled scoresV1,…,Vm\+1V\_\{1\},\\dots,V\_\{m\+1\}are exchangeable\. Applying Lemma[A\.1](https://arxiv.org/html/2610.28937#A1.Thmtheorem1)to the pooled calibration sample yields

ℙ⁡\(Ym\+1∈S^pool​\(Xm\+1\)\)≥1−α\.\\mathbb\{P\}\\\!\\left\(Y\_\{m\+1\}\\in\\widehat\{S\}\_\{\\mathrm\{pool\}\}\(X\_\{m\+1\}\)\\right\)\\geq 1\-\\alpha\.This is exactly the claimed mixture\-marginal validity statement\. ∎

### A\.5Proof of Proposition[4\.1](https://arxiv.org/html/2610.28937#S4.Thmtheorem1)

We restate the proposition for convenience\.

###### Proposition\(Max\-threshold validity\)\.

Assume that, for eachc∈𝒞c\\in\\mathcal\{C\}, the calibration data in𝒟cal\(c\)\\mathcal\{D\}\_\{\\mathrm\{cal\}\}^\{\(c\)\}and a test point generated underccare exchangeable with respect toAA, so that the configuration\-specific predictor based onq^c\\widehat\{q\}\_\{c\}is valid at level1−α1\-\\alpha\. Then

ℙ⁡\(Y∈S^max​\(X\)∣C=c\)≥1−αfor every​c∈𝒞\.\\mathbb\{P\}\(Y\\in\\widehat\{S\}\_\{\\max\}\(X\)\\mid C=c\)\\geq 1\-\\alpha\\qquad\\text\{for every \}c\\in\\mathcal\{C\}\.Consequently,S^max\\widehat\{S\}\_\{\\max\}is also valid marginally for any deployment distribution onCC\.

###### Proof\.

Fixc∈𝒞c\\in\\mathcal\{C\}\. By assumption, the configuration\-specific split\-conformal predictor

S^c​\(x\):=\{y∈𝒴⁡\(x\):A⁡\(π,y\)≤q^c\}\\widehat\{S\}\_\{c\}\(x\):=\\\{y\\in\\mathcal\{Y\}\(x\):A\(\\pi,y\)\\leq\\widehat\{q\}\_\{c\}\\\}satisfies

ℙ⁡\(Y∈S^c​\(X\)∣C=c\)≥1−α\.\\mathbb\{P\}\(Y\\in\\widehat\{S\}\_\{c\}\(X\)\\mid C=c\)\\geq 1\-\\alpha\.By construction,

q^max=maxc′∈𝒞⁡q^c′≥q^c\.\\widehat\{q\}\_\{\\max\}=\\max\_\{c^\{\\prime\}\\in\\mathcal\{C\}\}\\widehat\{q\}\_\{c^\{\\prime\}\}\\geq\\widehat\{q\}\_\{c\}\.Hence, by Lemma[A\.2](https://arxiv.org/html/2610.28937#A1.Thmtheorem2),

S^c​\(x\)⊆S^max​\(x\)for every​x\.\\widehat\{S\}\_\{c\}\(x\)\\subseteq\\widehat\{S\}\_\{\\max\}\(x\)\\qquad\\text\{for every \}x\.Therefore

ℙ⁡\(Y∈S^max​\(X\)∣C=c\)≥ℙ⁡\(Y∈S^c​\(X\)∣C=c\)≥1−α\.\\mathbb\{P\}\(Y\\in\\widehat\{S\}\_\{\\max\}\(X\)\\mid C=c\)\\geq\\mathbb\{P\}\(Y\\in\\widehat\{S\}\_\{c\}\(X\)\\mid C=c\)\\geq 1\-\\alpha\.Sincec∈𝒞c\\in\\mathcal\{C\}was arbitrary, the configuration\-conditional statement holds for every configuration\.

The marginal validity statement now follows from Lemma[A\.7](https://arxiv.org/html/2610.28937#A1.Thmtheorem7)\. ∎

### A\.6Proof of Theorem[4\.2](https://arxiv.org/html/2610.28937#S4.Thmtheorem2)

We restate the theorem for convenience\.

###### Theorem\(Latent\-robust validity\)\.

Assume:

1. 1\.for eachc∈𝒞c\\in\\mathcal\{C\}, the Stage\-1 calibration data in𝒟1\(c\)\\mathcal\{D\}\_\{1\}^\{\(c\)\}and a test point generated underccare exchangeable with respect toAA, so thatS^c\\widehat\{S\}\_\{c\}is valid at level1−αstage11\-\\alpha\_\{\\mathrm\{stage1\}\}; and
2. 2\.the Stage\-2 calibration tuples in𝒟2\\mathcal\{D\}\_\{2\}and the test tuple are exchangeable with respect to\(C,Z\)\(C,Z\), so that ℙ⁡\(C∈Γ^​\(Z⁡\(X\)\)\)≥1−βstage2\.\\mathbb\{P\}\\\!\\left\(C\\in\\widehat\{\\Gamma\}\(Z\(X\)\)\\right\)\\geq 1\-\\beta\_\{\\mathrm\{stage2\}\}\.

Then

ℙ⁡\(Y∈S^Γ​\(X\)\)≥1−αstage1−βstage2\.\\mathbb\{P\}\\\!\\left\(Y\\in\\widehat\{S\}\_\{\\Gamma\}\(X\)\\right\)\\geq 1\-\\alpha\_\{\\mathrm\{stage1\}\}\-\\beta\_\{\\mathrm\{stage2\}\}\.

###### Proof\.

Let

E:=\{C∈Γ^\(Z\(X\)\)\}E:=\\\{C\\in\\widehat\{\\Gamma\}\(Z\(X\)\)\\\}denote the event that the Stage\-2 conformal set of plausible configurations contains the true deployment configuration\.

Define the configuration\-specific Stage\-1 predictor

S^C​\(X\):=\{y∈𝒴⁡\(X\):A⁡\(π,y\)≤q^C\},\\widehat\{S\}\_\{C\}\(X\):=\\\{y\\in\\mathcal\{Y\}\(X\):A\(\\pi,y\)\\leq\\widehat\{q\}\_\{C\}\\\},whereq^C\\widehat\{q\}\_\{C\}denotes the Stage\-1 threshold corresponding to the realized configurationCC\.

On the eventEE, the realized configurationCCbelongs toΓ^​\(Z​\(X\)\)\\widehat\{\\Gamma\}\(Z\(X\)\)\. Therefore, by definition ofq^Γ​\(X\)\\widehat\{q\}\_\{\\Gamma\}\(X\),

q^Γ​\(X\)=maxc∈Γ^​\(Z​\(X\)\)⁡q^c≥q^C\.\\widehat\{q\}\_\{\\Gamma\}\(X\)=\\max\_\{c\\in\\widehat\{\\Gamma\}\(Z\(X\)\)\}\\widehat\{q\}\_\{c\}\\geq\\widehat\{q\}\_\{C\}\.Hence, by Lemma[A\.2](https://arxiv.org/html/2610.28937#A1.Thmtheorem2),

S^C​\(X\)⊆S^Γ​\(X\)on​E\.\\widehat\{S\}\_\{C\}\(X\)\\subseteq\\widehat\{S\}\_\{\\Gamma\}\(X\)\\qquad\\text\{on \}E\.Equivalently,

\{Y∉S^Γ\(X\)\}∩E⊆\{Y∉S^C\(X\)\}\.\\\{Y\\notin\\widehat\{S\}\_\{\\Gamma\}\(X\)\\\}\\cap E\\subseteq\\\{Y\\notin\\widehat\{S\}\_\{C\}\(X\)\\\}\.It follows that

ℙ​\(Y∉S^Γ​\(X\)\)\\displaystyle\\mathbb\{P\}\(Y\\notin\\widehat\{S\}\_\{\\Gamma\}\(X\)\)=ℙ\(\{Y∉S^Γ\(X\)\}∩E\)\+ℙ\(\{Y∉S^Γ\(X\)\}∩Ec\)\\displaystyle=\\mathbb\{P\}\\bigl\(\\\{Y\\notin\\widehat\{S\}\_\{\\Gamma\}\(X\)\\\}\\cap E\\bigr\)\+\\mathbb\{P\}\\bigl\(\\\{Y\\notin\\widehat\{S\}\_\{\\Gamma\}\(X\)\\\}\\cap E^\{c\}\\bigr\)≤ℙ⁡\(Y∉S^C​\(X\)\)\+ℙ⁡\(Ec\)\.\\displaystyle\\leq\\mathbb\{P\}\(Y\\notin\\widehat\{S\}\_\{C\}\(X\)\)\+\\mathbb\{P\}\(E^\{c\}\)\.
We now bound the two terms separately\.

For the first term, use the law of total probability and the Stage\-1 validity assumption:

ℙ​\(Y∉S^C​\(X\)\)\\displaystyle\\mathbb\{P\}\(Y\\notin\\widehat\{S\}\_\{C\}\(X\)\)=∑c∈𝒞ℙ⁡\(C=c\)​ℙ​\(Y∉S^c​\(X\)∣C=c\)\\displaystyle=\\sum\_\{c\\in\\mathcal\{C\}\}\\mathbb\{P\}\(C=c\)\\,\\mathbb\{P\}\(Y\\notin\\widehat\{S\}\_\{c\}\(X\)\\mid C=c\)≤∑c∈𝒞ℙ⁡\(C=c\)​αstage1=αstage1\.\\displaystyle\\leq\\sum\_\{c\\in\\mathcal\{C\}\}\\mathbb\{P\}\(C=c\)\\,\\alpha\_\{\\mathrm\{stage1\}\}=\\alpha\_\{\\mathrm\{stage1\}\}\.
For the second term, assumption \(ii\) gives

ℙ⁡\(Ec\)=ℙ⁡\(C∉Γ^​\(Z⁡\(X\)\)\)≤βstage2\.\\mathbb\{P\}\(E^\{c\}\)=\\mathbb\{P\}\\\!\\left\(C\\notin\\widehat\{\\Gamma\}\(Z\(X\)\)\\right\)\\leq\\beta\_\{\\mathrm\{stage2\}\}\.
Combining the two bounds yields

ℙ⁡\(Y∉S^Γ​\(X\)\)≤αstage1\+βstage2\.\\mathbb\{P\}\(Y\\notin\\widehat\{S\}\_\{\\Gamma\}\(X\)\)\\leq\\alpha\_\{\\mathrm\{stage1\}\}\+\\beta\_\{\\mathrm\{stage2\}\}\.Equivalently,

ℙ⁡\(Y∈S^Γ​\(X\)\)≥1−αstage1−βstage2,\\mathbb\{P\}\(Y\\in\\widehat\{S\}\_\{\\Gamma\}\(X\)\)\\geq 1\-\\alpha\_\{\\mathrm\{stage1\}\}\-\\beta\_\{\\mathrm\{stage2\}\},which proves the theorem\. ∎

### A\.7Remarks and non\-claims

We close with several clarifications that are useful when interpreting the main results\.

## Appendix BMore on related work

Table[1](https://arxiv.org/html/2610.28937#A2.T1)summarizes how representative prior work relates to the calibration–deployment inference\-configuration mismatch studied in this paper\. The review focuses on whether inference configuration is treated as an experimental axis for conformal validity, rather than on whether a paper reports every low\-level generation parameter\. Matched calibration is the right target when the deployment configuration is fixed and known\. Our setting instead asks whether coverage transports when calibration and deployment differ only in inference configuration\.

Table 1:Representative prior work and its relation to calibration–deployment inference\-configuration mismatch\.
## Appendix CSupplementary Experimental Details

This section records the benchmark configuration used to generate the supplementary results\.

#### Reporting convention\.

All primary supplementary tables report results separately for each dataset–model pair\. Every tabulated performance entry is reported as mean±\\pmstandard deviation across the three random runs\. We do not average across models in the primary appendix evidence, since averaging across model families would obscure an important part of the phenomenon under study\. When deployment axes are averaged, such as averaging over deployment quantization at fixed temperature or averaging over deployment temperature at fixed quantization, that aggregation is part of the experimental design and is performed within run before summarization across runs\.

### C\.1Datasets and split sizes

Table 2:Dataset sizes used throughout the benchmark\.Table[2](https://arxiv.org/html/2610.28937#A3.T2)reports the calibration and test sizes used for the four benchmark datasets\. These split sizes are fixed throughout the reported experiments\.

### C\.2Models, score functions, and deployment grid

The benchmark covers eight small language models: Qwen2\.5\-1\.5B\-Instruct, Qwen2\.5\-3B\-Instruct, Qwen2\.5\-7B\-Instruct, DeepSeek\-R1\-Distill\-Qwen\-7B, Gemma\-2\-2B\-it, Llama\-3\.2\-3B\-Instruct, Phi\-3\-mini\-4k\-instruct, and Mistral\-7B\-Instruct\-v0\.3\.

We evaluate three nonconformity scores: APS, LAC, and RAPS\. The RAPS hyperparameters are fixed to

λ=0\.01,kreg=1\.\\lambda=0\.01,\\qquad k\_\{\\mathrm\{reg\}\}=1\.
The inference\-configuration grid is the Cartesian product of

𝒬=\{fp16,int8,nf4\}and𝒯=\{0\.5,1\.0,1\.5,2\.0\},\\mathcal\{Q\}=\\\{\\mathrm\{fp16\},\\mathrm\{int8\},\\mathrm\{nf4\}\\\}\\qquad\\text\{and\}\\qquad\\mathcal\{T\}=\\\{0\.5,1\.0,1\.5,2\.0\\\},yielding 12 total deployment configurations\.

### C\.3Latent split implementation and setting

The main\-paper latent setting uses a 50/50 Stage\-1/Stage\-2 split of the calibration data:

For the default setting atα=0\.1\\alpha=0\.1, the results use

αstage1=0\.05,βstage2=0\.05\.\\alpha\_\{\\mathrm\{stage1\}\}=0\.05,\\qquad\\beta\_\{\\mathrm\{stage2\}\}=0\.05\.Unless otherwise stated, the main appendix tables use the default setting above\.

#### Stage\-2 latent\-configuration predictor\.

In our implementation, the Stage\-2 conformal predictorΓ^\\hat\{\\Gamma\}is built using a multinomial logistic\-regression router\. After the Stage\-1 thresholds\{q^c\}c∈𝒞\\\{\\hat\{q\}\_\{c\}\\\}\_\{c\\in\\mathcal\{C\}\}are fixed, we compute the set\-size signature

Z⁡\(x\)=\(\|S^c​\(x\)\|\)c∈𝒞,Z\(x\)=\\bigl\(\|\\hat\{S\}\_\{c\}\(x\)\|\\bigr\)\_\{c\\in\\mathcal\{C\}\},where

S^c​\(x\)=\{y∈𝒴⁡\(x\):A⁡\(π,y\)≤q^c\}\\hat\{S\}\_\{c\}\(x\)=\\\{y\\in\\mathcal\{Y\}\(x\):A\(\\pi,y\)\\leq\\hat\{q\}\_\{c\}\\\}is the hypothetical prediction set obtained by applying the Stage\-1 threshold for configurationcc\. The router takesZ⁡\(x\)Z\(x\)as input and predicts the inference configurationC∈𝒞C\\in\\mathcal\{C\}\. To preserve the split\-conformal interpretation of the Stage\-2 configuration set, we split the Stage\-2 data into a router\-training splitD2,trainD\_\{2,\\mathrm\{train\}\}and a router\-calibration splitD2,calD\_\{2,\\mathrm\{cal\}\}\. The router is fit onD2,trainD\_\{2,\\mathrm\{train\}\}and then held fixed when computing configuration nonconformity scores onD2,calD\_\{2,\\mathrm\{cal\}\}\. Letp^​\(c∣Z​\(x\)\)\\hat\{p\}\(c\\mid Z\(x\)\)denote the router’s predicted probability for configurationcc\. We use the configuration nonconformity score

B⁡\(Z⁡\(x\),C\)=1−p^​\(C∣Z⁡\(x\)\),B\(Z\(x\),C\)=1\-\\hat\{p\}\(C\\mid Z\(x\)\),so that configurations assigned higher router probability have smaller nonconformity\. The Stage\-2 thresholdr^β\\hat\{r\}\_\{\\beta\}is the standard split\-conformal quantile of

\{B⁡\(Zi,Ci\):\(Xi,Ci\)∈D2,cal\}\\\{B\(Z\_\{i\},C\_\{i\}\):\(X\_\{i\},C\_\{i\}\)\\in D\_\{2,\\mathrm\{cal\}\}\\\}at miscoverage levelβstage2\\beta\_\{\\mathrm\{stage2\}\}\. At test time, the plausible latent\-configuration set is

Γ^​\(Z⁡\(x\)\)=\{c∈𝒞:1−p^​\(c∣Z⁡\(x\)\)≤r^β\}\.\\hat\{\\Gamma\}\(Z\(x\)\)=\\\{c\\in\\mathcal\{C\}:1\-\\hat\{p\}\(c\\mid Z\(x\)\)\\leq\\hat\{r\}\_\{\\beta\}\\\}\.The LATENT\-ROBUST threshold is then

q^Γ​\(x\)=maxc∈Γ^​\(Z​\(x\)\)⁡q^c,\\hat\{q\}\_\{\\Gamma\}\(x\)=\\max\_\{c\\in\\hat\{\\Gamma\}\(Z\(x\)\)\}\\hat\{q\}\_\{c\},with fallback tomaxc∈𝒞⁡q^c\\max\_\{c\\in\\mathcal\{C\}\}\\hat\{q\}\_\{c\}ifΓ^​\(Z​\(x\)\)=∅\\hat\{\\Gamma\}\(Z\(x\)\)=\\emptyset\.

### C\.4Quantization and temperature implementation\.

For each model, we instantiate the same pretrained checkpoint under each quantization mode𝒬∈\{fp16,int8,nf4\}\\mathcal\{Q\}\\in\\\{\\mathrm\{fp16\},\\mathrm\{int8\},\\mathrm\{nf4\}\\\}\. The fp16 setting uses half\-precision inference, while the int8 and nf4 settings use post\-training low\-bit inference following standard LLM quantization practice\[[7](https://arxiv.org/html/2610.28937#bib.bib39),[8](https://arxiv.org/html/2610.28937#bib.bib3)\]\. In all cases, quantization changes only the numerical representation used at inference time: the model checkpoint, training procedure, prompt template, candidate labels, calibration split, and evaluation split are held fixed\. No model weights are fine\-tuned or otherwise updated between inference configurations\.

Temperature is applied before candidate\-label scoring, at the token\-logit level\. Letℓq,k​\(v∣x,y<k\)\\ell\_\{q,k\}\(v\\mid x,y\_\{<k\}\)denote the pre\-softmax logit assigned, under quantization modeqq, to vocabulary tokenvvat positionkkwhen scoring candidate labely=\(y1,…,ym\)y=\(y\_\{1\},\\ldots,y\_\{m\}\)conditioned on inputxxand the candidate prefixy<ky\_\{<k\}\. For temperaturet\>0t\>0, we form the temperature\-scaled token distribution

pq,t​\(v∣x,y<k\)=exp⁡\(ℓq,k​\(v∣x,y<k\)/t\)∑v′∈𝒱exp⁡\(ℓq,k​\(v′∣x,y<k\)/t\),p\_\{q,t\}\(v\\mid x,y\_\{<k\}\)=\\frac\{\\exp\\\!\\left\(\\ell\_\{q,k\}\(v\\mid x,y\_\{<k\}\)/t\\right\)\}\{\\sum\_\{v^\{\\prime\}\\in\\mathcal\{V\}\}\\exp\\\!\\left\(\\ell\_\{q,k\}\(v^\{\\prime\}\\mid x,y\_\{<k\}\)/t\\right\)\},where𝒱\\mathcal\{V\}is the model vocabulary\. We then compute the candidate\-label score from the temperature\-scaled token probabilities as the length\-normalized log\-likelihood

sq,t​\(y∣x\)=1m​∑k=1mlog⁡pq,t​\(yk∣x,y<k\)\.s\_\{q,t\}\(y\\mid x\)=\\frac\{1\}\{m\}\\sum\_\{k=1\}^\{m\}\\log p\_\{q,t\}\(y\_\{k\}\\mid x,y\_\{<k\}\)\.The finite\-label predictive distribution used by conformal prediction is obtained by exponentiating and normalizing these candidate\-label log\-scores over the candidate set,

πq,t​\(y∣x\)=exp⁡\(sq,t​\(y∣x\)\)∑y′∈𝒴⁡\(x\)exp⁡\(sq,t​\(y′∣x\)\),y∈𝒴⁡\(x\)\.\\pi\_\{q,t\}\(y\\mid x\)=\\frac\{\\exp\\\!\\left\(s\_\{q,t\}\(y\\mid x\)\\right\)\}\{\\sum\_\{y^\{\\prime\}\\in\\mathcal\{Y\}\(x\)\}\\exp\\\!\\left\(s\_\{q,t\}\(y^\{\\prime\}\\mid x\)\\right\)\},\\qquad y\\in\\mathcal\{Y\}\(x\)\.Thus, changingttchanges the token\-level predictive distribution before candidate\-label likelihoods are computed, while changingqqchanges the numerical representation used to compute the logits\. The conformal score is then computed fromπq,t\(⋅∣x\)\\pi\_\{q,t\}\(\\cdot\\mid x\)and the true label\. Therefore, changing\(q,t\)\(q,t\)changes the inference\-time score\-generating procedure while leaving the task distribution, trained checkpoint, prompt template, candidate labels, and data splits unchanged\.

### C\.5Compute resources

All experiments were run on a local workstation equipped with an AMD Ryzen Threadripper PRO CPU and a single NVIDIA RTX A6000 GPU with 48GB of GPU memory\. Model inference, including fp16, int8, and nf4 configurations, was executed on the GPU\. CPU resources were used for data loading, preprocessing, conformal calibration, the logistic\-regression router used by LATENT\-ROBUST, and result aggregation\. The experiments were run sequentially over the dataset–model–configuration grid\.

## Appendix DAdditional Main\-Benchmark Results

Unless otherwise stated, all tables in this section use LAC withα=0\.1\\alpha=0\.1and the main\-paper latent setting

αstage1=βstage2=α/2,d1=0\.5\.\\alpha\_\{\\mathrm\{stage1\}\}=\\beta\_\{\\mathrm\{stage2\}\}=\\alpha/2,\\qquad d\_\{1\}=0\.5\.Each performance entry reports mean±\\pmstandard deviation across runs for coverage and set size\.

### D\.1Deployment fixed tonf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5

Table 3:Coverage and average prediction\-set size for CommonsenseQA with deployment fixed tonf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5, using LAC withα=0\.1\\alpha=0\.1\. Each entry reports mean±\\pmstandard deviation across runs\.Table 4:Coverage and average prediction\-set size for DBPedia\-14 with deployment fixed tonf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5, using LAC withα=0\.1\\alpha=0\.1\. Each entry reports mean±\\pmstandard deviation across runs\.Table 5:Coverage and average prediction\-set size for MMLU\-Pro with deployment fixed tonf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5, using LAC withα=0\.1\\alpha=0\.1\. Each entry reports mean±\\pmstandard deviation across runs\.Table 6:Coverage and average prediction\-set size for MobileActions with deployment fixed tonf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5, using LAC withα=0\.1\\alpha=0\.1\. Each entry reports mean±\\pmstandard deviation across runs\.Tables[3](https://arxiv.org/html/2610.28937#A4.T3)–[6](https://arxiv.org/html/2610.28937#A4.T6)correspond to the main deployment setting emphasized in the paper\. Several points are visible at the per\-model level\. First, the matched baseline generally remains close to target, consistent with the standard split\-conformal setting\. Second, the degradation under calibration–deployment mismatch is often substantial, but its magnitude varies sharply across model families\. Third, the strongest failures remain concentrated in MMLU\-Pro, where Table[5](https://arxiv.org/html/2610.28937#A4.T5)shows that multiple models exhibit large drops fromMatchedtoMismatch Meanand especially toMismatch Worst\. By contrast, Table[6](https://arxiv.org/html/2610.28937#A4.T6)shows that MobileActions is more stable for many models, though worst\-case mismatch can still fall below target\.

The merged presentation in Tables[3](https://arxiv.org/html/2610.28937#A4.T3)–[6](https://arxiv.org/html/2610.28937#A4.T6)also makes the reliability–efficiency tradeoff explicit\. In settings whereMaxThrandLatent\-Robustrecover coverage, they generally do so by increasing average prediction\-set size\. This is especially visible on the harder tasks, where robust coverage recovery is paired with a nontrivial efficiency cost\. The attached standard deviations are typically small relative to the method gaps, indicating that the qualitative conclusions are not driven by a single random split\.

### D\.2Deployment temperature fixed toT=0\.5T=0\.5

Table 7:Coverage and average prediction\-set size for CommonsenseQA with deployment temperature fixed toT=0\.5T=0\.5and deployment quantization averaged within run, using LAC withα=0\.1\\alpha=0\.1\. Each entry reports mean±\\pmstandard deviation across runs\.Table 8:Coverage and average prediction\-set size for DBPedia\-14 with deployment temperature fixed toT=0\.5T=0\.5and deployment quantization averaged within run, using LAC withα=0\.1\\alpha=0\.1\. Each entry reports mean±\\pmstandard deviation across runs\.Table 9:Coverage and average prediction\-set size for MMLU\-Pro with deployment temperature fixed toT=0\.5T=0\.5and deployment quantization averaged within run, using LAC withα=0\.1\\alpha=0\.1\. Each entry reports mean±\\pmstandard deviation across runs\.Table 10:Coverage and average prediction\-set size for MobileActions with deployment temperature fixed toT=0\.5T=0\.5and deployment quantization averaged within run, using LAC withα=0\.1\\alpha=0\.1\. Each entry reports mean±\\pmstandard deviation across runs\.Tables[7](https://arxiv.org/html/2610.28937#A4.T7)–[10](https://arxiv.org/html/2610.28937#A4.T10)report the experiment in which deployment temperature is fixed toT=0\.5T=0\.5and the deployment quantization mode is averaged within run by design\. The resulting tables preserve the same qualitative picture seen in the main setting\. In particular, Tables[7](https://arxiv.org/html/2610.28937#A4.T7)and[9](https://arxiv.org/html/2610.28937#A4.T9)show that several model–dataset pairs on CommonsenseQA and MMLU\-Pro remain strongly mismatch\-sensitive, indicating that the phenomenon is not attributable only to one particular deployment quantization mode\. This supports the interpretation that low\-temperature deployment itself defines a regime of elevated sensitivity to calibration–deployment mismatch\.

At the same time, the per\-model view in Tables[7](https://arxiv.org/html/2610.28937#A4.T7)–[10](https://arxiv.org/html/2610.28937#A4.T10)continues to show that the effect is not uniform across model families\. Some models remain comparatively stable under the same averaging regime in which others degrade sharply\. This reinforces one of the paper’s central points: there is no single monotone ordering of configurations that cleanly explains all failures\.

### D\.3Deployment quantization fixed tonf4\\mathrm\{nf4\}

Table 11:Coverage and average prediction\-set size for CommonsenseQA with deployment quantization fixed tonf4\\mathrm\{nf4\}and deployment temperature averaged within run, using LAC withα=0\.1\\alpha=0\.1\. Each entry reports mean±\\pmstandard deviation across runs\.Table 12:Coverage and average prediction\-set size for DBPedia\-14 with deployment quantization fixed tonf4\\mathrm\{nf4\}and deployment temperature averaged within run, using LAC withα=0\.1\\alpha=0\.1\. Each entry reports mean±\\pmstandard deviation across runs\.Table 13:Coverage and average prediction\-set size for MMLU\-Pro with deployment quantization fixed tonf4\\mathrm\{nf4\}and deployment temperature averaged within run, using LAC withα=0\.1\\alpha=0\.1\. Each entry reports mean±\\pmstandard deviation across runs\.Table 14:Coverage and average prediction\-set size for MobileActions with deployment quantization fixed tonf4\\mathrm\{nf4\}and deployment temperature averaged within run, using LAC withα=0\.1\\alpha=0\.1\. Each entry reports mean±\\pmstandard deviation across runs\.Tables[11](https://arxiv.org/html/2610.28937#A4.T11)–[14](https://arxiv.org/html/2610.28937#A4.T14)report the experiment in which deployment quantization is fixed tonf4\\mathrm\{nf4\}and deployment temperature is averaged within run\. These tables show that substantial mismatch effects remain visible for multiple models even after fixing quantization, which indicates that temperature is itself a primary driver of inference\-time model shift\. This complements the previous subsection: neither temperature nor quantization alone fully explains the phenomenon, and the strongest discrepancies arise when both dimensions of configuration are allowed to vary\.

The model\-disaggregated results in Tables[11](https://arxiv.org/html/2610.28937#A4.T11)–[14](https://arxiv.org/html/2610.28937#A4.T14)again matter here\. In several cases the same broad type of mismatch produces noticeably different effects across models, including different degrees of undercoverage and conservativeness\. The appendix therefore supports the main\-paper claim that matched\-only reporting can be materially misleading even under a fixed benchmark task distribution\.

## Appendix ECalibration–Deployment Mismatch Structure

We next visualize the full pairwise vanilla calibration→\\rightarrowdeployment structure under LAC withα=0\.1\\alpha=0\.1\. Each figure is model\-specific and contains four panels, one for each dataset\. The left subfigure reports empirical coverage, while the right subfigure reports the corresponding coverage gap relative to the nominal target of0\.90\.9\. Rows index calibration configuration and columns index deployment configuration\.

These figures are intentionally reported at the model level rather than after averaging across models\. This is important because the direction and magnitude of mismatch effects vary across model families\.

![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_cov_qwen_1_5b.png)\(a\)Coverage\.
![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_gap_qwen_1_5b.png)\(b\)Coverage gap from target\.

Figure 5:Pairwise vanilla calibration→\\rightarrowdeployment structure for Qwen2\.5\-1\.5B under LAC withα=0\.1\\alpha=0\.1\. Each panel corresponds to one dataset\. White cell annotations report the exact values used to construct the heatmap\.![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_cov_qwen_3b.png)\(a\)Coverage\.
![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_gap_qwen_3b.png)\(b\)Coverage gap from target\.

Figure 6:Pairwise vanilla calibration→\\rightarrowdeployment structure for Qwen2\.5\-3B under LAC withα=0\.1\\alpha=0\.1\.![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_cov_qwen_7b.png)\(a\)Coverage\.
![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_gap_qwen_7b.png)\(b\)Coverage gap from target\.

Figure 7:Pairwise vanilla calibration→\\rightarrowdeployment structure for Qwen2\.5\-7B under LAC withα=0\.1\\alpha=0\.1\.![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_cov_deepseek_r1_qwen7b.png)\(a\)Coverage\.
![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_gap_deepseek_r1_qwen7b.png)\(b\)Coverage gap from target\.

Figure 8:Pairwise vanilla calibration→\\rightarrowdeployment structure for DeepSeek\-R1\-Distill\-Qwen\-7B under LAC withα=0\.1\\alpha=0\.1\.![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_cov_gemma_2b.png)\(a\)Coverage\.
![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_gap_gemma_2b.png)\(b\)Coverage gap from target\.

Figure 9:Pairwise vanilla calibration→\\rightarrowdeployment structure for Gemma\-2\-2B under LAC withα=0\.1\\alpha=0\.1\.![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_cov_llama_3_2_3b.png)\(a\)Coverage\.
![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_gap_llama_3_2_3b.png)\(b\)Coverage gap from target\.

Figure 10:Pairwise vanilla calibration→\\rightarrowdeployment structure for Llama\-3\.2\-3B under LAC withα=0\.1\\alpha=0\.1\.![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_cov_phi_3_mini.png)\(a\)Coverage\.
![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_gap_phi_3_mini.png)\(b\)Coverage gap from target\.

Figure 11:Pairwise vanilla calibration→\\rightarrowdeployment structure for Phi\-3\-mini under LAC withα=0\.1\\alpha=0\.1\.![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_cov_mistral_7b.png)\(a\)Coverage\.
![Refer to caption](https://arxiv.org/html/2610.28937v1/pairwise_gap_mistral_7b.png)\(b\)Coverage gap from target\.

Figure 12:Pairwise vanilla calibration→\\rightarrowdeployment structure for Mistral\-7B under LAC withα=0\.1\\alpha=0\.1\.Figures[5](https://arxiv.org/html/2610.28937#A5.F5)–[12](https://arxiv.org/html/2610.28937#A5.F12)reveal structure that is difficult to see from summary statistics alone\. First, the diagonal matched cells are generally close to target, as expected under the standard split\-conformal setting\. Second, off\-diagonal behavior is highly configuration\-dependent, with localized regions of severe undercoverage and other regions that are conservative\. Third, these patterns differ materially across models\. The same broad type of mismatch can therefore look relatively benign in one model family and clearly unreliable in another\. This is exactly why model\-level reporting is necessary for the present evaluation question\.

Because each heatmap cell is numerically annotated, Figures[5](https://arxiv.org/html/2610.28937#A5.F5)–[12](https://arxiv.org/html/2610.28937#A5.F12)also support direct inspection of specific calibration→\\rightarrowdeployment pairs without sacrificing the visual overview of the mismatch structure\. The coverage\-gap panels are especially useful for isolating undercoverage regions, while the raw coverage panels make the absolute magnitude of the deviations easy to interpret\.

## Appendix FAdditional Robustness Analyses

### F\.1Robustness to the score function

Panel A: APS

Panel B: LAC

Panel C: RAPS

Table 15:Score\-function robustness on CommonsenseQA at deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5withα=0\.1\\alpha=0\.1\. Each panel reports mean±\\pmstandard deviation across runs\.Panel A: APS

Panel B: LAC

Panel C: RAPS

Table 16:Score\-function robustness on DBPedia\-14 at deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5withα=0\.1\\alpha=0\.1\. Each panel reports mean±\\pmstandard deviation across runs\.Panel A: APS

Panel B: LAC

Panel C: RAPS

Table 17:Score\-function robustness on MMLU\-Pro at deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5withα=0\.1\\alpha=0\.1\. Each panel reports mean±\\pmstandard deviation across runs\.Panel A: APS

Panel B: LAC

Panel C: RAPS

Table 18:Score\-function robustness on MobileActions at deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5withα=0\.1\\alpha=0\.1\. Each panel reports mean±\\pmstandard deviation across runs\.Tables[15](https://arxiv.org/html/2610.28937#A6.T15)–[18](https://arxiv.org/html/2610.28937#A6.T18)compare APS, LAC, and RAPS at deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5withα=0\.1\\alpha=0\.1\. The qualitative mismatch pattern persists across score functions, although the absolute level of conservativeness differs across APS, LAC, and RAPS\. In particular, APS and RAPS are often more conservative than LAC, but neither removes the main evaluative issue: matched\-only reporting can still overstate deployment\-time reliability under calibration–deployment mismatch\. This robustness check therefore strengthens the claim that the phenomenon is not an artifact of one particular nonconformity score\.

The integrated mean±\\pmstandard deviation reporting in Tables[15](https://arxiv.org/html/2610.28937#A6.T15)–[18](https://arxiv.org/html/2610.28937#A6.T18)also shows that the score\-function differences are typically larger than the run\-to\-run variability\. This is consistent with the view that the observed changes reflect systematic properties of the score construction rather than instability of the benchmark\.

### F\.2Robustness to the target miscoverage level

Panel A:α=0\.05\\alpha=0\.05

Panel B:α=0\.1\\alpha=0\.1

Panel C:α=0\.2\\alpha=0\.2

Panel D:α=0\.3\\alpha=0\.3

Table 19:Robustness to the target miscoverage level on CommonsenseQA at deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5using LAC\. Each panel reports coverage gap from target and average prediction\-set size as mean±\\pmstandard deviation across runs\.Panel A:α=0\.05\\alpha=0\.05

Panel B:α=0\.1\\alpha=0\.1

Panel C:α=0\.2\\alpha=0\.2

Panel D:α=0\.3\\alpha=0\.3

Table 20:Robustness to the target miscoverage level on DBPedia\-14 at deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5using LAC\. Each panel reports coverage gap from target and average prediction\-set size as mean±\\pmstandard deviation across runs\.Panel A:α=0\.05\\alpha=0\.05

Panel B:α=0\.1\\alpha=0\.1

Panel C:α=0\.2\\alpha=0\.2

Panel D:α=0\.3\\alpha=0\.3

Table 21:Robustness to the target miscoverage level on MMLU\-Pro at deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5using LAC\. Each panel reports coverage gap from target and average prediction\-set size as mean±\\pmstandard deviation across runs\.Panel A:α=0\.05\\alpha=0\.05

Panel B:α=0\.1\\alpha=0\.1

Panel C:α=0\.2\\alpha=0\.2

Panel D:α=0\.3\\alpha=0\.3

Table 22:Robustness to the target miscoverage level on MobileActions at deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5using LAC\. Each panel reports coverage gap from target and average prediction\-set size as mean±\\pmstandard deviation across runs\.Tables[19](https://arxiv.org/html/2610.28937#A6.T19)–[22](https://arxiv.org/html/2610.28937#A6.T22)report the coverage gap from target, defined as empirical coverage minus\(1−α\)\(1\-\\alpha\), together with average prediction\-set size, forα∈\{0\.05,0\.1,0\.2,0\.3\}\\alpha\\in\\\{0\.05,0\.1,0\.2,0\.3\\\}in the deployment settingnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5using LAC\. The main qualitative ranking of methods remains stable across target levels\. Across datasets and models,Matchedtends to remain near target, mismatch baselines can substantially under\-cover, and the more conservative robust methods recover coverage more reliably\. This indicates that the central phenomenon is not tied to a single nominal target level\.

The size columns in Tables[19](https://arxiv.org/html/2610.28937#A6.T19)–[22](https://arxiv.org/html/2610.28937#A6.T22)are also informative: as the target level changes, the efficiency cost of robust recovery changes in tandem\. This again reinforces the point that deployment robustness should be evaluated jointly in terms of reliability and prediction\-set size, not by coverage alone\.

## Appendix GLatent\-RobustDiagnostics

Table 23:Diagnostics forLatent\-Robuston CommonsenseQA in the main\-paper setting: LAC,α=0\.1\\alpha=0\.1, deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5,αstage1=βstage2=0\.05\\alpha\_\{\\mathrm\{stage1\}\}=\\beta\_\{\\mathrm\{stage2\}\}=0\.05, andd1=0\.5d\_\{1\}=0\.5\. Each entry reports mean±\\pmstandard deviation across runs\.Table 24:Diagnostics forLatent\-Robuston DBPedia\-14 in the main\-paper setting: LAC,α=0\.1\\alpha=0\.1, deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5,αstage1=βstage2=0\.05\\alpha\_\{\\mathrm\{stage1\}\}=\\beta\_\{\\mathrm\{stage2\}\}=0\.05, andd1=0\.5d\_\{1\}=0\.5\. Each entry reports mean±\\pmstandard deviation across runs\.Table 25:Diagnostics forLatent\-Robuston MMLU\-Pro in the main\-paper setting: LAC,α=0\.1\\alpha=0\.1, deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5,αstage1=βstage2=0\.05\\alpha\_\{\\mathrm\{stage1\}\}=\\beta\_\{\\mathrm\{stage2\}\}=0\.05, andd1=0\.5d\_\{1\}=0\.5\. Each entry reports mean±\\pmstandard deviation across runs\.Table 26:Diagnostics forLatent\-Robuston MobileActions in the main\-paper setting: LAC,α=0\.1\\alpha=0\.1, deploymentnf4​@​T=0\.5\\mathrm\{nf4\}@T=0\.5,αstage1=βstage2=0\.05\\alpha\_\{\\mathrm\{stage1\}\}=\\beta\_\{\\mathrm\{stage2\}\}=0\.05, andd1=0\.5d\_\{1\}=0\.5\. Each entry reports mean±\\pmstandard deviation across runs\.Tables[23](https://arxiv.org/html/2610.28937#A7.T23)–[26](https://arxiv.org/html/2610.28937#A7.T26)clarify howLatent\-Robustachieves coverage recovery\. Besides final coverage and average prediction\-set size, they report the empirical coverage of the latent configuration\-set predictorΓ^​\(Z​\(X\)\)\\widehat\{\\Gamma\}\(Z\(X\)\)and the average size of the predicted configuration set\. Interpreting these diagnostics jointly is important\. When the inferred configuration set is large, the method hedges more broadly across plausible latent deployment states and correspondingly pays a larger efficiency cost\. Conversely, smaller inferred configuration sets allow a less conservative threshold\.

The diagnostics in Tables[23](https://arxiv.org/html/2610.28937#A7.T23)–[26](https://arxiv.org/html/2610.28937#A7.T26)therefore show that the method’s conservativeness is not arbitrary\. It is mediated by how difficult it is to infer the latent deployment configuration from the observed predictive vector\. Generally, the adaptive hedge is driven primarily by latent\-configuration predictions rather than repeated reversion to the global worst\-case threshold\. The attached standard deviations also make clear that these diagnostic quantities are themselves reasonably stable across runs\.

## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]\.
4. Justification: The abstract and introduction state the main contribution: identifying inference\-time model shift as a source of non\-exchangeability for conformal prediction in language models and evaluating its empirical consequences across models, datasets, and inference configurations\. The scope is also clearly limited to finite\-label benchmarks and post\-training inference\-configuration changes\.
5. 2\.Limitations
6. Question: Does the paper discuss the limitations of the work performed by the authors?
7. Answer:\[Yes\]\.
8. Justification: Section 6 discusses limitations, including that the work is primarily a characterization rather than a mechanistic theory, that the empirical study uses a finite deployment grid, and that it does not cover richer deployment variation such as continuous temperatures or temporal drift in serving policies\.
9. 3\.Theory assumptions and proofs
10. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
11. Answer:\[Yes\]\.
12. Justification: The main text states the assumptions and targets for inference\-time model shift, MAXTHR, POOLED, and LATENT\-ROBUST\. Appendix A provides the corresponding definitions, lemmas, full statements, and proofs\.
13. 4\.Experimental result reproducibility
14. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
15. Answer:\[Yes\]\.
16. Justification: Section 5 and Appendix C specify the datasets, models, inference\-configuration grid, calibration/evaluation splits, score functions, robust baselines, quantization and temperature implementation, and the Stage\-2 LATENT\-ROBUST router construction\. The supplementary material includes the source code, configuration files, and processed experimental artifacts sufficient to reproduce the main reported tables and figures\.
17. 5\.Open access to data and code
18. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
19. Answer:\[Yes\]\.
20. Justification: The supplementary material includes the source code and data package with scripts, configuration files, processed experimental artifacts, and instructions for reproducing the main coverage and prediction\-set\-size results\. The package documents how to access the public datasets and model checkpoints used in the experiments\.
21. 6\.Experimental setting/details
22. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
23. Answer:\[Yes\]\.
24. Justification: Section 5 specifies the datasets, models, inference configurations, conformal score setup, data splits, compared methods, and metrics\. Appendix C further specifies split sizes, model versions, score functions, deployment grid, latent split settings, the logistic\-regression router, quantization/temperature implementation, and compute resources\.
25. 7\.Experiment statistical significance
26. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
27. Answer:\[Yes\]\.
28. Justification: The experiments are averaged over three independent random splits, and the appendix tables report mean±\\pmstandard deviation across runs for coverage and average prediction\-set size\. The paper also states how aggregation over deployment axes is performed before summarizing across runs\.
29. 8\.Experiments compute resources
30. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
31. Answer:\[Yes\]\.
32. Justification: Appendix C reports the compute environment used for the experiments, including the CPU class, GPU type, GPU memory, and how CPU and GPU resources were used across inference, calibration, routing, and aggregation\.
33. 9\.Code of ethics
35. Answer:\[Yes\]\.
36. Justification: The work evaluates uncertainty\-quantification protocols for publicly available language models and benchmark datasets, does not introduce human\-subject experiments, and does not release a high\-risk model or scraped dataset\. The study is intended to improve transparency and reliability reporting for deployed language\-model systems\.
37. 10\.Broader impacts
38. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
39. Answer:\[Yes\]\.
40. Justification: The paper discusses implications for evaluation practice and reliability reporting for language\-model systems\. The main positive impact is improved transparency around conformal reliability claims; the main potential negative impact is that conformal guarantees could be overinterpreted if deployment assumptions and inference\-configuration mismatch are not reported\.
41. 11\.Safeguards
42. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
43. Answer:\[N/A\]\.
44. Justification: The paper does not release a new pretrained model, image generator, scraped dataset, or other high\-risk asset\. The supplementary material contains the source code and processed experimental artifacts derived from existing public datasets and model outputs\.
45. 12\.Licenses for existing assets
46. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
47. Answer:\[Yes\]\.
48. Justification: The paper cites the original datasets and pretrained model checkpoints used in the experiments, and the supplementary material documents the corresponding sources, licenses, and terms of use where available\. The released artifacts are derived from these public assets and are provided consistently with their usage terms\.
49. 13\.New assets
50. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
51. Answer:\[Yes\]\.
52. Justification: The supplementary material includes the code and processed experimental artifacts derived from public datasets and model outputs\. The accompanying documentation describes the contents of the package, how the artifacts were generated, how to reproduce the reported results, and the intended use and limitations of the released materials\.
53. 14\.Crowdsourcing and research with human subjects
54. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
55. Answer:\[N/A\]\.
56. Justification: The paper does not involve crowdsourcing experiments or research with human subjects\. All experiments are performed on existing benchmark datasets and pretrained language models\.
57. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
58. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
59. Answer:\[N/A\]\.
60. Justification: The paper does not involve crowdsourcing or research with human subjects, so IRB approval or equivalent human\-subjects review is not applicable\.
61. 16\.Declaration of LLM usage
62. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
63. Answer:\[Yes\]\.
64. Justification: Language models are central experimental objects in this work\. The paper describes the pretrained language models used, the inference configurations applied to them, and how the resulting predictive distributions are used to compute conformal scores; any LLM use for writing, editing, or formatting does not affect the scientific methodology or originality of the research\.