Subtype Robustness Is Not Just Accuracy: Calibration Under Unseen Subtype Shift

# Subtype Robustness Is Not Just Accuracy: Calibration Under Unseen Subtype Shift
Source: [https://arxiv.org/html/2608.00928](https://arxiv.org/html/2608.00928)
###### Abstract

Subtype robustness asks whether a model keeps the correct coarse prediction when test examples come from fine\-grained subtypes absent from training but still inside a known coarse category\. Prior work studies this almost entirely through accuracy\. We ask whether the model also stays calibrated\. We present the first systematic study of the question across ImageNet, BREEDS, iNaturalist and CIFAR\-100 with five architectures\. Calibration breaks down on unseen subtypes, where accuracy drops while confidence barely follows, leaving the model systematically overconfident exactly where it has become less accurate\. At matched accuracy loss, generic image corruption causes a much larger drop in confidence, so the effect is not a general consequence of losing accuracy\. The model reacts to visible degradation but not to in\-taxonomy novelty\. Recalibration tuned on seen subtypes narrows the gap but does not close it, and out\-of\-distribution scores flag the affected inputs only weakly\. Subtype robustness should therefore be evaluated through calibration, not accuracy alone\.111The code and data are available at https://github\.com/yibo\-hu\-lab/subtype\-robustness

## Introduction

Deep learning systems are often deployed in settings where the correct semantic decision remains unchanged, even though its fine\-grained realizations continue to vary\. In hierarchical label spaces, this means that test examples may come from fine\-grained subtypes absent from training, while their correct coarse labels remain within the known taxonomy\. This is the setting ofsubtype robustness: the model need not recover the exact unseen subtype, but it should still preserve the correct coarse semantic prediction\. Figure[1](https://arxiv.org/html/2608.00928#Sx1.F1)illustrates it\.

![Refer to caption](https://arxiv.org/html/2608.00928v1/x1.png)Figure 1:Toy example of subtype robustness\. The model is trained onairplaneandautomobile\(vehicle\), andbirdandcat\(animal\), then evaluated on unseen subtypesshipanddeerfrom the same supertypes\. The unseenshipis correctly predicted asvehicle, and the unseendeeris misclassified asvehicleinstead ofanimal, yet the model is equally confident in both\.This setting is distinct from classical out\-of\-distribution \(OOD\) detection: the input still carries a valid coarse label, so the failure is a wrong coarse prediction\. In many applications the coarse decision is the primary target, and failing to recover the exact subtype costs less than predicting the wrong coarse category\(Bertinetto et al\.[2020](https://arxiv.org/html/2608.00928#bib.bib3); Garg, Sani, and Anand[2022](https://arxiv.org/html/2608.00928#bib.bib9)\)\. Real\-world systems routinely meet new variants of known categories while the coarse decision stays the same\. A species classifier deployed for biodiversity monitoring, for example, continually meets newly catalogued species under a known genus or family: the fine\-grained label is novel, the coarse taxonomic decision is not\.

Prior work on subtype robustness has focused primarily on accuracy\.Hendrycks and Dietterich \([2019](https://arxiv.org/html/2608.00928#bib.bib13)\)reported seen–unseen subtype gaps, andSanturkar et al\. \([2021](https://arxiv.org/html/2608.00928#bib.bib36)\)introduced BREEDS for controlled subtype\-split evaluation\. Accuracy alone, however, gives an incomplete view\. A model’s coarse accuracy can fall on unseen subtypes while its confidence barely follows, leaving it systematically overconfident on the predictions it now gets wrong\. This overconfidence is a calibration failure: a calibrated model’s confidence matches its accuracy\(Guo et al\.[2017](https://arxiv.org/html/2608.00928#bib.bib11)\), so confidence that stays high while accuracy falls is by definition miscalibrated\. To our knowledge, calibration under subtype\-level novelty is unexplored\.

Common image corruptions behave differently\. As the image degrades, models become less accurate and less confident, as observed broadly under distribution shift\(Ovadia et al\.[2019](https://arxiv.org/html/2608.00928#bib.bib35)\): the image looks different, and the model reacts\. Subtype shift breaks that link, because the image looks normal and its coarse label is still correct\.

We formalize subtype robustness as a same\-coarse\-label, unseen\-fine\-subtype shift, read off a single model through matched accuracy, confidence, and calibration gaps \(Δacc\\Delta\_\{\\mathrm\{acc\}\},Δconf\\Delta\_\{\\mathrm\{conf\}\},Δcal\\Delta\_\{\\mathrm\{cal\}\}\), and make three contributions\.

- •Confidence does not follow accuracy\.Across ImageNet, BREEDS \(Living\-17, Nonliving\-26\), iNaturalist, and CIFAR\-100, and across architectures, coarse accuracy drops on unseen subtypes while confidence barely follows, leaving the model overconfident where it has become less accurate\. The size of the accuracy drop varies by dataset: steep where the taxonomy is sparse, mild where it is densely sampled\. The signal also fails per sample: failure\-detection AUROC and risk–coverage both degrade\.
- •The failure does not appear under matched generic corruption\.At matched accuracy loss, generic image corruption lowers confidence far more than subtype novelty does\.
- •The post\-hoc tools we test do not recover the lost calibration\.Recalibration tuned on seen subtypes narrows but does not close the seen–unseen calibration gap, and inference\-time confidence and OOD scores flag unseen subtypes only weakly\.

## Related Work

#### Subtype robustness and subpopulation shift\.

The subtype robustness setting traces toHendrycks and Dietterich \([2019](https://arxiv.org/html/2608.00928#bib.bib13)\), whose Appendix G measures accuracy on held\-out subtypes of known coarse categories, and it has remained largely under\-explored since\. The closest benchmark continuation is BREEDS, which constructs hierarchy\-aware source–target splits with disjoint subtype support under curated ImageNet superclasses\(Santurkar et al\.[2021](https://arxiv.org/html/2608.00928#bib.bib36)\)\. In the failure\-detection literature,Jaeger et al\. \([2023](https://arxiv.org/html/2608.00928#bib.bib20)\)include a CIFAR\-100 “sub\-class shift” condition in their evaluation of misclassification\-detection metrics\. Broader subpopulation\- and domain\-shift benchmarks provide an umbrella, though the underlying shift definitions differ\(Koh et al\.[2021](https://arxiv.org/html/2608.00928#bib.bib24); Yang et al\.[2023](https://arxiv.org/html/2608.00928#bib.bib43)\)\.

#### Hierarchical classification and semantic error structure\.

A related line studies hierarchy\-aware prediction, where errors within a taxonomy are not equally severe and a coarse\-label error is treated differently from a fine\-grained one\(Deng et al\.[2010](https://arxiv.org/html/2608.00928#bib.bib6); Barz and Denzler[2019](https://arxiv.org/html/2608.00928#bib.bib2); Bertinetto et al\.[2020](https://arxiv.org/html/2608.00928#bib.bib3); Karthik et al\.[2021](https://arxiv.org/html/2608.00928#bib.bib23); Garg, Sani, and Anand[2022](https://arxiv.org/html/2608.00928#bib.bib9); Jain, Karthik, and Gandhi[2023](https://arxiv.org/html/2608.00928#bib.bib21)\)\.

#### Calibration and recalibration under shift\.

Calibration methods assess whether model confidence is aligned with empirical correctness\(Guo et al\.[2017](https://arxiv.org/html/2608.00928#bib.bib11); Widmann, Lindsten, and Zachariah[2019](https://arxiv.org/html/2608.00928#bib.bib42); Minderer et al\.[2021](https://arxiv.org/html/2608.00928#bib.bib33)\)\. Modern networks can become substantially miscalibrated under distribution shift, including OOD and novel\-class settings\(Hendrycks, Mazeika, and Dietterich[2019](https://arxiv.org/html/2608.00928#bib.bib15); Kong et al\.[2020](https://arxiv.org/html/2608.00928#bib.bib26); Hu et al\.[2021](https://arxiv.org/html/2608.00928#bib.bib18); Hu and Khan[2021](https://arxiv.org/html/2608.00928#bib.bib17)\)\. Large pretrained models are reported to be better calibrated and more robust to such shift\(Minderer et al\.[2021](https://arxiv.org/html/2608.00928#bib.bib33)\)\. A common fix for calibration under shift is to re\-weight the calibration data toward the test distribution\(Wang et al\.[2025](https://arxiv.org/html/2608.00928#bib.bib41); Hu et al\.[2026](https://arxiv.org/html/2608.00928#bib.bib19); Cheung, Veeraraghavan, and Balakrishnan[2026](https://arxiv.org/html/2608.00928#bib.bib5); Kong et al\.[2026](https://arxiv.org/html/2608.00928#bib.bib25)\)\. Finally,Lyu, Zhou, and Zhong \([2025](https://arxiv.org/html/2608.00928#bib.bib32)\)offer a candidate mechanism: under\-represented groups receive inflated training logits\. Our setting is the extreme case, because unseen subtypes are absent from training rather than merely rare\.

#### Out\-of\-distribution detection\.

Detection methods flag inputs whose label lies outside the taxonomy, using output\-based scores such as the maximum softmax probability\(Hendrycks and Gimpel[2017](https://arxiv.org/html/2608.00928#bib.bib14)\)and the energy score\(Liu et al\.[2020](https://arxiv.org/html/2608.00928#bib.bib30)\), or feature\-based scores such as the Mahalanobis distance\(Lee et al\.[2018](https://arxiv.org/html/2608.00928#bib.bib29)\)\. We therefore ask whether these scores respond to a novel subtype whose coarse label is still valid\. Benchmarks in this area have been argued to conflate semantic with covariate shift, with detectors responding mainly to the former\(Zhu et al\.[2024](https://arxiv.org/html/2608.00928#bib.bib44)\)\. Subtype shift fits neither cleanly: the label stays inside the taxonomy, as in covariate shift, yet what changes is semantic rather than appearance\.

## Problem Setup

### Hierarchical classification

Multiclass classification maps instancesx∈𝒳x\\in\\mathcal\{X\}to one ofK≥2K\\geq 2discrete labels𝒦:=\{1,…,K\}\\mathcal\{K\}:=\\\{1,\\dots,K\\\}\. We assume these labels are partitioned intoJJmutually exclusive, nonempty subsets𝒦1,…,𝒦J\\mathcal\{K\}\_\{1\},\\dots,\\mathcal\{K\}\_\{J\}indexed by𝒥:=\{1,…,J\}\\mathcal\{J\}:=\\\{1,\\dots,J\\\}, with2≤J≤K2\\leq J\\leq K\. We call the labelsk∈𝒦k\\in\\mathcal\{K\}subtypesand the partsj∈𝒥j\\in\\mathcal\{J\}supertypes, and writeg​\(k\)g\(k\)for the supertype of subtypekk, so thatg​\(⋅\):𝒦→𝒥g\(\\cdot\):\\mathcal\{K\}\\to\\mathcal\{J\}is surjective\.

###### Definition 1\(Seen/unseen subtypes\)\.

We assume a nonempty subset𝒮⊆𝒦\\mathcal\{S\}\\subseteq\\mathcal\{K\}ofseen subtypes, whose complement𝒰:=𝒦∖𝒮\\mathcal\{U\}:=\\mathcal\{K\}\\setminus\\mathcal\{S\}is the set ofunseen subtypes, so that1≤\|𝒮\|<K1\\leq\|\\mathcal\{S\}\|<K\. Every supertype is represented among the seen subtypes:g​\(𝒮\)=g​\(𝒰\)=𝒥g\(\\mathcal\{S\}\)=g\(\\mathcal\{U\}\)=\\mathcal\{J\}\.

Each part𝒦j=g−1​\(j\)\\mathcal\{K\}\_\{j\}=g^\{\-1\}\(j\)therefore splits into𝒮j:=𝒦j∩𝒮\\mathcal\{S\}\_\{j\}:=\\mathcal\{K\}\_\{j\}\\cap\\mathcal\{S\}, the seen subtypes underjj, and𝒰j:=𝒦j∩𝒰\\mathcal\{U\}\_\{j\}:=\\mathcal\{K\}\_\{j\}\\cap\\mathcal\{U\}, the unseen ones\. We writenj:=\|𝒮j\|n\_\{j\}:=\|\\mathcal\{S\}\_\{j\}\|for how many seen subtypes a supertype has\. The Analysis section relates this quantity to the size of the calibration collapse\.

### Training scheme

All models are trained using only examples whose subtype labels belong to𝒮\\mathcal\{S\}, and predict the supertype\. The key constraint is that no examples from𝒰\\mathcal\{U\}are used for representation learning, classifier training, calibration, or model selection, except in the post\-hoc calibration experiment where we state otherwise\. Model selection and any post\-hoc calibrator are fit on*seen\-val*, a held\-out set of*images*drawn from the same seen subtypes used for training, never on test\. Every split is carved at the image level within𝒮\\mathcal\{S\}, so no subtype label is withheld from training\. The four\-way train / seen\-val / test\-seen / test\-unseen split is detailed in Appendix[A](https://arxiv.org/html/2608.00928#A1)\.

#### Training procedure\.

All main\-table models are trained*direct\-coarse*: a single network optimized end\-to\-end on the supertype label from a random initialization, uniformly across all datasets, as in BREEDS\(Santurkar et al\.[2021](https://arxiv.org/html/2608.00928#bib.bib36)\)\.Hendrycks and Dietterich \([2019](https://arxiv.org/html/2608.00928#bib.bib13)\)instead reuse a fine\-pretrained frozen backbone, a two\-stage protocol on a pretrained network\. We report the protocol \(one\- vs\. two\-stage\) and initialization \(from\-scratch vs\. ImageNet\-pretrained\) axes as controls in Appendix[5](https://arxiv.org/html/2608.00928#A1.T5), where the protocol is neutral and pretraining mitigates the overconfidence without removing it\.

### Evaluation scheme

#### Accuracy\.

We evaluate on both seen and unseen subtype splits, always scoring the predicted supertype\. For an example\(x,y\)\(x,y\), the coarse\-label error is𝟏​\{j^​\(x\)≠g​\(y\)\}\\mathbf\{1\}\\\{\\widehat\{j\}\(x\)\\neq g\(y\)\\\}\. Accuracy is computed separately on examples withy∈𝒮y\\in\\mathcal\{S\}andy∈𝒰y\\in\\mathcal\{U\}\.

#### Calibration\.

Letpj​\(x\)p\_\{j\}\(x\)denote the model probability assigned to supertypejj\. We define the confidence and correctness of a prediction as

c​\(x\)\\displaystyle c\(x\)=maxj∈𝒥⁡pj​\(x\),\\displaystyle=\\max\_\{j\\in\\mathcal\{J\}\}p\_\{j\}\(x\),r​\(x,y\)\\displaystyle r\(x,y\)=𝟏​\{arg⁡maxj∈𝒥⁡pj​\(x\)=g​\(y\)\}\.\\displaystyle=\\mathbf\{1\}\\\{\\arg\\max\\nolimits\_\{j\\in\\mathcal\{J\}\}p\_\{j\}\(x\)=g\(y\)\\\}\.We measure calibration with the expected calibration error \(ECE\) over1515equal\-width bins\(Naeini, Cooper, and Hauskrecht[2015](https://arxiv.org/html/2608.00928#bib.bib34)\)\(Appendix[7](https://arxiv.org/html/2608.00928#A1.T7)covers the estimator and its binning bias\)\. Given that ECE is a binned estimator, we corroborate every gap with an equal\-mass recomputation and with two binning\-free proper scores\(Gneiting and Raftery[2007](https://arxiv.org/html/2608.00928#bib.bib10)\), the negative log\-likelihood \(NLL\) and the Brier score\(Brier[1950](https://arxiv.org/html/2608.00928#bib.bib4)\), all in Appendix[A](https://arxiv.org/html/2608.00928#A1.SS0.SSS0.Px4)\. Since ECE averages\|acc^−conf^\|\|\\widehat\{\\mathrm\{acc\}\}\-\\widehat\{\\mathrm\{conf\}\}\|, all three quantities share one scale, and we report accuracy, confidence, ECE, and the seen−\-unseen gaps defined below inpercentage pointsthroughout\.

#### Per\-sample discrimination\.

Calibration is an aggregate property and can be orthogonal to whether confidence discriminates correct from incorrect predictions per sample\.On each split we therefore also report the two standard failure\-detection measures\(Jaeger et al\.[2023](https://arxiv.org/html/2608.00928#bib.bib20)\): the areas under the ROC curve \(AUROC, whetherc​\(x\)c\(x\)separates correct from incorrect predictions\) and the risk–coverage curve \(AURC, lower is better\)\. Both are rank\-based, so unlike the calibration gap, a loss here cannot be repaired by rescaling confidence\.

#### Subtype accuracy, confidence, and calibration gaps\.

We summarize each model by three matched seen\-versus\-unseen gaps,Δacc=acc𝒮−acc𝒰\\Delta\_\{\\mathrm\{acc\}\}=\\mathrm\{acc\}\_\{\\mathcal\{S\}\}\-\\mathrm\{acc\}\_\{\\mathcal\{U\}\},Δconf=c¯𝒮−c¯𝒰\\Delta\_\{\\mathrm\{conf\}\}=\\bar\{c\}\_\{\\mathcal\{S\}\}\-\\bar\{c\}\_\{\\mathcal\{U\}\}, andΔcal=ECE𝒰−ECE𝒮\\Delta\_\{\\mathrm\{cal\}\}=\\mathrm\{ECE\}\_\{\\mathcal\{U\}\}\-\\mathrm\{ECE\}\_\{\\mathcal\{S\}\}, wherec¯∙\\bar\{c\}\_\{\\bullet\}is the mean confidencec​\(x\)c\(x\)on the split\. A model is*calibration\-robust*to subtype shift whenΔcal\\Delta\_\{\\mathrm\{cal\}\}stays small\. When insteadΔconf≪Δacc\\Delta\_\{\\mathrm\{conf\}\}\\ll\\Delta\_\{\\mathrm\{acc\}\}, the two gaps*dissociate*, and we call the resulting failure*silent overconfidence*\.

## Experiments

### Research Questions

RQ1 \(Calibration under subtype shift\)\.When a model loses accuracy on unseen subtypes, does it stay calibrated there, and does high confidence still indicate a correct prediction?

RQ2 \(Comparison with generic corruption\)\.If generic image corruption is tuned to cost the same accuracy as subtype shift, does it cost the same confidence?

RQ3 \(Standard remedies\)\.Can the calibration gap be closed after training, either by \(a\) recalibrating on seen subtypes or \(b\) catching novel subtypes at inference with standard OOD scores?

### Datasets and Evaluation Settings

ImageNet\-25\(IN\-25\)\(Deng et al\.[2009](https://arxiv.org/html/2608.00928#bib.bib7)\)is the largest of our benchmarks: 25 broad categories as supertypes, with 383 seen subtypes \(ImageNet\-1K synsets\) and 3,872 held\-out unseen subtypes \(ImageNet\-22K\-only synsets\) under those same categories\. Its number of seen subtypes per supertype is deliberately uneven \(dogmany,fungusfew\), inherited from ImageNet\-1K, and we do not rebalance it\.

BREEDS\(Santurkar et al\.[2021](https://arxiv.org/html/2608.00928#bib.bib36)\)is derived from the same ImageNet\-1K classes but purpose\-built for hierarchy\-aware seen/unseen subtype splits\. We take two tasks that span the living/man\-made divide,Living\-17\(Liv\-17; 17 supertypes; natural, animate\) andNonliving\-26\(NL\-26; 26 supertypes; man\-made\), each supertype split into two seen and two unseen subtypes\.

iNaturalist\(iNat\-25\)\(Van Horn et al\.[2021](https://arxiv.org/html/2608.00928#bib.bib38)\)carries a real Linnaean taxonomy \(order⊇\\supseteqfamily⊇\\supseteqgenus⊇\\supseteqspecies\), so an unseen subtype is literal: a held\-out species under a known order\. We use 25 orders as supertypes, splitting each order’s species into 3,427 seen and 3,427 held out\. It is also the only benchmark that supplies a model\-free notion of subtype distance \(taxonomic rank\)\.

CIFAR\-100\(Krizhevsky[2009](https://arxiv.org/html/2608.00928#bib.bib27)\)is a low\-resolution sanity check rather than a primary benchmark: 20 supertypes, 60 of the 100 fine classes seen and 40 held out, at32×3232\\times 32with two small models \(AlexNet, ResNet\-18\)\.

Table 1:Accuracy, confidence and ECE on seen and unseen subtypes, with the seen−\-unseen gap for each, all inpercentage points\. On every row ECE is close to confidence minus accuracy, henceΔcal≈Δacc−Δconf\\Delta\_\{\\mathrm\{cal\}\}\\approx\\Delta\_\{\\mathrm\{acc\}\}\-\\Delta\_\{\\mathrm\{conf\}\}\. Appendix[A](https://arxiv.org/html/2608.00928#A1.SS0.SSS0.Px4)checks the gap against equal\-mass binning and against two binning\-free proper scores\.Across all datasets we evaluate on a per\-supertype\-balanced test set\. Appendix[A](https://arxiv.org/html/2608.00928#A1)gives full construction, splits, and per\-class counts, and Appendix[8](https://arxiv.org/html/2608.00928#A1.T8)details the iNaturalist taxonomy build\.

### Models and Baselines

We compare five architectures:

- •Historical baseline:AlexNet\(Krizhevsky, Sutskever, and Hinton[2012](https://arxiv.org/html/2608.00928#bib.bib28)\)\.
- •Residual models:ResNet\-18 and ResNet\-50\(He et al\.[2016](https://arxiv.org/html/2608.00928#bib.bib12)\)\.
- •Modern backbones:ViT\-B/16\(Dosovitskiy et al\.[2021](https://arxiv.org/html/2608.00928#bib.bib8)\)and ConvNeXt\-T\(Liu et al\.[2022](https://arxiv.org/html/2608.00928#bib.bib31)\)\.

On the full\-resolution \(224×224224\\times 224\) datasets \(IN\-25, BREEDS, iNat\-25\) we sweep all five architectures; CIFAR\-100 uses only the small models \(AlexNet, ResNet\-18\) that train competently at32×3232\\times 32\.

For calibration, we evaluate two standard post\-hoc calibrators, tuned on a held\-out image split from the seen subtypes and then applied to unseen subtypes: temperature scaling and vector scaling\(Guo et al\.[2017](https://arxiv.org/html/2608.00928#bib.bib11)\)\.

For detection, we use three standard scores\. Two read the output distribution: maximum softmax probability \(MSP\)\(Hendrycks and Gimpel[2017](https://arxiv.org/html/2608.00928#bib.bib14)\)and the energy score\(Liu et al\.[2020](https://arxiv.org/html/2608.00928#bib.bib30)\)\. The third reads the representation: the Mahalanobis distance to the nearest seen\-class feature centroid\(Lee et al\.[2018](https://arxiv.org/html/2608.00928#bib.bib29)\)\. An output score can only flag what the model already doubts; a feature score can flag inputs that look novel even when the output is confident\.

The full training and evaluation recipe is given in Table[5](https://arxiv.org/html/2608.00928#A1.T5)\. The two controlled variations reported separately, the fine\-pretrained\-frozen protocol on CIFAR\-100 and BREEDS and an ImageNet\-1K\-pretrained backbone on IN\-25, are covered in Appendix[5](https://arxiv.org/html/2608.00928#A1.T5)\.

### Accuracy drops but confidence barely follows \(RQ1\)

We first confirm the accuracy phenomenon reported by prior subtype and subpopulation\-shift work, then turn to calibration\. Table[1](https://arxiv.org/html/2608.00928#Sx4.T1)reports what each model does on unseen subtypes and how far that falls from the seen split\. Coarse accuracy drops by2929–4040points on CIFAR\-100, IN\-25 and BREEDS and by a milder77–1313on iNat\-25, but confidence drops by only22–1616\. On unseen subtypes accuracy lands at3030–70%70\\%while confidence stays at6464–90%90\\%\. ECE therefore rises from22–2222points on seen subtypes to99–4646on unseen, so miscalibration increases by55–2828points on every model\. The gap is not a binning artifact: it is unchanged to within0\.70\.7points under equal\-mass bins, and the binning\-free NLL and Brier both rise from seen to unseen on every row \(Appendix[A](https://arxiv.org/html/2608.00928#A1.SS0.SSS0.Px4)\)\.

![Refer to caption](https://arxiv.org/html/2608.00928v1/x2.png)Figure 2:Seen versus unseen subtypes \(ResNet\-50, direct\-coarse from scratch\)\. Accuracy drops far more than confidence does, and the calibration error rises from22–1010points on seen subtypes to1919–2828on unseen\.Figure[2](https://arxiv.org/html/2608.00928#Sx4.F2)makes the dissociation visible for ResNet\-50, one metric per panel: accuracy falls steeply from seen to unseen subtypes, confidence barely moves, and the calibration error rises on every dataset\. Per\-architecture reliability diagrams are in Appendix[A](https://arxiv.org/html/2608.00928#A1.SS0.SSS0.Px6)\.

Confidence also degrades as a per\-sample signal\. On unseen subtypes the failure\-detection AUROC falls on every dataset, by0\.150\.15–0\.180\.18on CIFAR\-100, IN\-25 and BREEDS and by0\.050\.05on iNat\-25, consistent with its much smaller accuracy gap, and the risk–coverage AURC rises2\.3×2\.3\\timesto18×18\\times\(Appendix[6](https://arxiv.org/html/2608.00928#A1.F6)\)\. Used to decide when to abstain, such a model lets through far more errors while answering the same fraction of inputs\. This part of the failure is harder to repair thanΔcal\\Delta\_\{\\mathrm\{cal\}\}, because AUROC and AURC depend on the order of the confidence scores rather than their values, and recalibration changes only the values\.

### Confidence holds up under subtype shift, unlike under matched generic corruption \(RQ2\)

Miscalibration under distribution shift is already known\(Ovadia et al\.[2019](https://arxiv.org/html/2608.00928#bib.bib35); Hendrycks, Mazeika, and Dietterich[2019](https://arxiv.org/html/2608.00928#bib.bib15)\); the question is whether subtype novelty behaves like the shifts that literature studies\. We compare against a matched generic\-shift arm built from the ImageNet\-C corruptions\(Hendrycks and Dietterich[2019](https://arxiv.org/html/2608.00928#bib.bib13)\)\. We apply all1515of them, at all55severities, to the seen\-subtype images of every dataset, leaving the coarse labels unchanged\. Corrupting our own seen images rather than using the released ImageNet\-C copies matters here: the clean and corrupted arms are then literally the same images, so the only thing that changes is image quality\.

The comparison rests on two choices\. First, accuracy loss must be matched: a corruption that destroys more accuracy will naturally cost more confidence, so confidence drops are only comparable at equal accuracy loss\. We therefore interpolate each corruption along its own severity curve to the accuracy loss of that dataset’s subtype shift\. Second, no corruption is hand\-picked: every corruption that can reach that accuracy loss contributes one point, and we report the mean over all of them\. Matching a single best\-fitting corruption instead would be fragile, since which one fits best changes with the severity grid\.

A corruption enters only when its five severities bracket the subtype accuracy loss, which leaves1414of the1515corruptions on IN\-25,1111on each BREEDS task, and33on iNat\-25, whose subtype gap is smaller than most corruptions cost even at severity 1\. Figure[3](https://arxiv.org/html/2608.00928#Sx4.F3)collects the surviving arms\. At matched accuracy loss, generic corruption costs17\.317\.3points of confidence on average against9\.99\.9for subtype shift \(18\.618\.6vs\.10\.810\.8on IN\-25,19\.019\.0vs\.8\.98\.9on Liv\-17,23\.123\.1vs\.16\.216\.2on NL\-26,8\.38\.3vs\.3\.83\.8on iNat\-25\)\. The model loses the same accuracy either way, yet on every dataset it gives up less confidence to the subtype shift: it reacts to visible degradation and stays blind to in\-taxonomy novelty\. The corruption control is run on ResNet\-50; the dissociation betweenΔacc\\Delta\_\{\\mathrm\{acc\}\}andΔconf\\Delta\_\{\\mathrm\{conf\}\}it builds on is present on all five architectures \(Table[1](https://arxiv.org/html/2608.00928#Sx4.T1)\)\.

![Refer to caption](https://arxiv.org/html/2608.00928v1/x3.png)Figure 3:Confidence given up at matched accuracy loss \(ResNet\-50, direct\-coarse from scratch\); the corruption value is the mean over every ImageNet\-C corruption that can reach that dataset’s subtype accuracy loss\. Both arms use the clean accuracy of the same seen images as reference, so the subtype gaps differ marginally from Table[1](https://arxiv.org/html/2608.00928#Sx4.T1)\.
### Post\-hoc recalibration narrows but does not close the seen–unseen calibration gap \(RQ3a\)

We tune the two post\-hoc calibrators on a held\-out image split from the seen subtypes and apply them to unseen subtypes, a transfer that looks safe because the task and the coarse labels are unchanged and only the set of fine subtypes differs\.

Table[2](https://arxiv.org/html/2608.00928#Sx4.T2)gives the numbers and Figure[4](https://arxiv.org/html/2608.00928#Sx4.F4)the shape, both for ResNet\-50\. Temperature scaling lowers the unseen ECE on every dataset, by1818–57%57\\%\(IN\-2521\.7→17\.821\.7\\\!\\to\\\!17\.8, iNat\-2519\.3→8\.319\.3\\\!\\to\\\!8\.3\)\. Read on its own that looks like a repair, but what the calibrator fixes is the split it was tuned on\. On IN\-25 the seen ECE falls to almost nothing \(2\.52\.5uncalibrated,1\.41\.4under temperature,1\.01\.0under vector scaling\), while every bin of the unseen split stays below the diagonal in Figure[4](https://arxiv.org/html/2608.00928#Sx4.F4)\. That leavesΔcal\\Delta\_\{\\mathrm\{cal\}\}at77–1717\. iNat\-25 is where recalibration helps most, and even there the calibrated unseen ECE \(8\.38\.3\) is six times the calibrated seen ECE \(1\.41\.4\)\. The richer vector scaling improves the seen split most and transfers worst to unseen subtypes \(Liv\-17 unseen ECE25\.325\.3vs\.19\.619\.6under plain temperature\), because it may overfit the seen split it was tuned on\. On every dataset the calibrated unseen ECE stays far above the calibrated seen ECE\. Recalibration has been reported to transfer across hierarchy levels for vision\-language models\(Tu et al\.[2024](https://arxiv.org/html/2608.00928#bib.bib37)\)\. That transfer does not hold under discriminative subtype shift\. The same holds for all five architectures on all four datasets: temperature scaling never drivesΔcal\\Delta\_\{\\mathrm\{cal\}\}to zero \(Appendix[9](https://arxiv.org/html/2608.00928#A1.T9)\)\.

A temperature fit on the unseen split itself does close most of the gap, but it cannot be chosen without knowing which inputs are unseen, which is what motivates the detection question below \(Appendix[10](https://arxiv.org/html/2608.00928#A1.T10)\)\.

Table 2:Post\-hoc calibration \(ResNet\-50, direct\-coarse from scratch\), in percentage points\. Each calibrator is tuned on a held\-out image split from the seen subtypes and then applied to unseen subtypes;*None*is the uncalibrated model\.![Refer to caption](https://arxiv.org/html/2608.00928v1/x4.png)Figure 4:Reliability diagrams on the unseen split \(ResNet\-50 direct\-coarse from scratch,1515equal\-width bins; seen\-fit temperatureT𝒮∗=1\.2T^\{\*\}\_\{\\mathcal\{S\}\}=1\.2on IN\-25 and1\.651\.65on iNat\-25\)\. Bars below the diagonal are overconfident; each panel carries its unseen and seen ECE\.
### Inference\-time scores flag novel subtypes only weakly \(RQ3b\)

We ask whether a standard detection score can separate unseen\-subtype from seen\-subtype inputs, so that a system could at least know when to distrust its own confidence\. We score every test input with the maximum softmax probability \(MSP\)\(Hendrycks and Gimpel[2017](https://arxiv.org/html/2608.00928#bib.bib14)\), the logit\-based energy score\(Liu et al\.[2020](https://arxiv.org/html/2608.00928#bib.bib30)\), and the feature\-based Mahalanobis distance to the nearest seen\-class centroid\(Lee et al\.[2018](https://arxiv.org/html/2608.00928#bib.bib29); Heng and Soh[2026](https://arxiv.org/html/2608.00928#bib.bib16); Barker, Bethell, and Gerasimou[2026](https://arxiv.org/html/2608.00928#bib.bib1)\)\. Each score is evaluated on its own by the*novelty\-detection*AUROC: the probability that a randomly drawn unseen\-subtype input is scored as more novel than a randomly drawn seen\-subtype one, so0\.50\.5means the two populations are indistinguishable\. This is a different question from the failure\-detection AUROC of RQ1, which asks whether confidence separates correct from incorrect predictions inside a single split\.

Table 3:Novelty\-detection AUROC of each score \(ResNet\-50, direct\-coarse from scratch\): how well it ranks unseen\-subtype above seen\-subtype inputs,0\.50\.5= indistinguishable\. This is a triage signal, not the failure\-detection AUROC of Appendix[6](https://arxiv.org/html/2608.00928#A1.F6)and not standard out\-of\-taxonomy OOD rejection\. Restricting each score to inputs classified correctly on both splits lowers it by at most0\.040\.04\(Appendix[6](https://arxiv.org/html/2608.00928#A1.F6)\)\.All three scores separate the two populations poorly \(Table[3](https://arxiv.org/html/2608.00928#Sx4.T3)\)\. The two output\-based scores, MSP and energy, reach only∼0\.70\{\\sim\}0\.70on the three larger benchmarks and0\.560\.56on iNat\-25\. That is above chance but far from usable: at a threshold that flags5%5\\%of seen\-subtype inputs, MSP catches only1212–16%16\\%of unseen\-subtype inputs \(6%6\\%on iNat\-25\), because the two score distributions overlap heavily\. The feature\-based Mahalanobis distance sits just below chance on every dataset \(0\.450\.45–0\.470\.47\), ranking unseen subtypes as slightly*less*novel than seen ones\. A network trained only on the coarse label collapses same\-supertype subtypes together, so an unseen subtype ends up looking like the seen ones in feature space\.

Confidence correlates with correctness, so this separation could be trivial error detection rather than novelty detection\. Restricting each score to inputs the model classifies correctly on both splits lowers MSP and energy by at most0\.040\.04\(Appendix[6](https://arxiv.org/html/2608.00928#A1.F6)\), so the signal does respond to novelty, but far too weakly to act on\. No architecture does better \(Appendix[9](https://arxiv.org/html/2608.00928#A1.T9)\)\. On a from\-scratch model, then, confidence does not track subtype novelty, post\-hoc recalibration does not remove it, and inference\-time detection surfaces it only weakly\.

### Calibration collapse varies by superclass and tracks seen\-subtype diversity

The per\-class collapse is highly heterogeneous: some coarse classes keep near\-perfect calibration on unseen subtypes, others collapse completely \(per\-supertype accuracy and ECE in Appendix[A](https://arxiv.org/html/2608.00928#A1.SS0.SSS0.Px11)\)\. This matches the mechanism ofLyu, Zhou, and Zhong \([2025](https://arxiv.org/html/2608.00928#bib.bib32)\): under\-represented groups receive inflated training logits, and so become overconfident without losing accuracy\. Subtype shift is the extreme case, since unseen subtypes are absent from training rather than merely rare\.

We ask whether the collapse is predictable from seen\-side signals\. The one that tracks it is the number of seen subtypes per coarse class,nj=\|𝒮j\|n\_\{j\}=\|\\mathcal\{S\}\_\{j\}\|\. On IN\-25 \(the only dataset where it varies\), supertypes with few seen subtypes are the most overconfident on unseen ones while richly\-sampled ones \(e\.g\.dog, 117 subtypes\) stay calibrated \(Figure[5](https://arxiv.org/html/2608.00928#Sx4.F5); Spearmanρ=−0\.82\\rho=\-0\.82\), consistent with a sublinear \(≈1/nj\\approx\\\!1/\\sqrt\{n\_\{j\}\}\) coverage rate\(Jo, Kim, and Chae[2026](https://arxiv.org/html/2608.00928#bib.bib22)\)\. The trend is not a difficulty artifact: regressing per\-supertype overconfidence on1/nj1/\\sqrt\{n\_\{j\}\}and seen accuracy shows1/nj1/\\sqrt\{n\_\{j\}\}to be the dominant predictor \(t=4\.2t=4\.2\), while seen accuracy carries no signal \(t=0\.0t=0\.0\)\.

A controlledkk\-sweep supports a causal reading: holding the per\-class image budget fixed and varying only the seen\-subtype count shrinksΔcal\\Delta\_\{\\mathrm\{cal\}\}monotonically from3131\(k=1k\{=\}1\) to2323\(k=4k\{=\}4; Appendix[11](https://arxiv.org/html/2608.00928#A1.T11)\)\.

The count is useful in aggregate but not per class\. Ranking supertypes by it and keeping only the better\-sampled half roughly halves the overconfidence in the retained supertypes \(10\.210\.2, against21\.621\.6across all supertypes\), so it works as a coarse triage signal\. It does not say which individual supertype will collapse, and the two obvious alternatives, feature dispersion and confidence margin, carry no signal at all\.

Initialization matters more than protocol\. Switching to the two\-stage fine\-pretrained\-frozen protocol keepsΔcal\\Delta\_\{\\mathrm\{cal\}\}within66points, while an ImageNet\-1K\-pretrained backbone lowers it from19\.219\.2to12\.812\.8on IN\-25 without closing it \(Appendix[5](https://arxiv.org/html/2608.00928#A1.T5)\)\.

![Refer to caption](https://arxiv.org/html/2608.00928v1/x5.png)Figure 5:Per\-supertype overconfidence \(mean confidence−\-accuracy on unseen subtypes\) against the number of seen subtypesnjn\_\{j\}\(IN\-25, ResNet\-50 direct\-coarse from scratch\)\. Each point is one of the2525supertypes; the curve is a1/nj1/\\sqrt\{n\_\{j\}\}guide \(Spearmanρ=−0\.82\\rho=\-0\.82\)\.
### Discussion and Future Work

The practical upshot is that subtype robustness must be audited with calibration, since accuracy alone hides the failure\. A model that loses accuracy on novel subtypes goes on reporting almost the same confidence there, and neither post\-hoc recalibration nor inference\-time detection recovers the lost calibration\. The two failure modes are complementary: an aggregate miscalibration that a single temperature could repair only with oracle knowledge of which inputs are unseen, and a per\-sample discrimination loss that no monotone rescaling can touch\. Post\-hoc recalibration addresses the first and inference\-time detection the second, so neither reaches this regime\.

Subtype shift also behaves differently from generic covariate shift\. At matched accuracy loss, visible corruption reduces confidence much more sharply, whereas confidence barely moves on unseen subtypes even though the inputs look ordinary and still have a valid coarse label \(RQ2\)\. Together with the loss of per\-sample discrimination on unseen subtypes \(RQ1\), this shows that subtype robustness has been evaluated too narrowly\. Preserving coarse\-label accuracy is not enough if confidence no longer reflects reliability, so subtype robustness should be treated as a joint requirement on prediction and confidence\.

We study image classification, and we isolate the phenomenon rather than propose a training method, so several directions stay open\. The most direct is dense prediction, where subtype shift acts at the pixel or region level in semantic segmentation and object detection\. Beyond vision, whether the same silent overconfidence arises in language or tabular models is untested, and so is whether foundation and vision\-language models behave differently\. Our one pretraining control points only to partial mitigation \(Appendix[5](https://arxiv.org/html/2608.00928#A1.T5)\), where an ImageNet\-pretrained backbone narrows the gap without closing it\.

Three method\-side handles are worth pursuing\. Training\-time objectives such as evidential learning, outlier exposure, or subtype\-diversity augmentation would attack the miscalibration at its source\. A detector designed for in\-taxonomy novelty rather than out\-of\-taxonomy rejection would give the triage signal that the off\-the\-shelf scores of RQ3b fail to provide\. A larger diversity sweep than ourk=1k\{=\}1to44would turn the seen\-subtype effect from a supported direction into a quantitative law\. Until such tools exist, the immediate evaluation requirement is clear: confidence reliability should be reported alongside coarse accuracy whenever subtype robustness is claimed\.

#### Generative AI use statement\.

Generative AI tools were used solely for language editing and suggestions on the organization of the exposition\. All research questions, experimental design, analyses, results, and conclusions were developed by the authors\. The authors reviewed and verified all AI\-assisted edits, including all references, and take full responsibility for the content of the manuscript\.

## Acknowledgments

This work used Jetstream2 at Indiana University through ACCESS allocation CIS260254 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support \(ACCESS\) program, which is supported by U\.S\. National Science Foundation grants \#2138259, \#2138286, \#2138307, \#2137603, and \#2138296\. Results were also obtained using the Chameleon testbed, supported by the National Science Foundation\. We thank the Jetstream2, ACCESS, and Chameleon support teams for the computational infrastructure used in this work\. We also thank Yutong Wang for helpful discussions during the early stages of this project\.

## References

- Barker, Bethell, and Gerasimou \(2026\)Barker, C\.; Bethell, D\.; and Gerasimou, S\. 2026\.Robust Adversarial Quantification via Conflict\-Aware Evidential Deep Learning\.In*International Conference on Learning Representations*\.
- Barz and Denzler \(2019\)Barz, B\.; and Denzler, J\. 2019\.Hierarchy\-based image embeddings for semantic image retrieval\.In*WACV*\.
- Bertinetto et al\. \(2020\)Bertinetto, L\.; Mueller, R\.; Tertikas, K\.; Samangooei, S\.; and Lord, N\. A\. 2020\.Making better mistakes: Leveraging class hierarchies with deep networks\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 12506–12515\.
- Brier \(1950\)Brier, G\. W\. 1950\.Verification of forecasts expressed in terms of probability\.*Monthly Weather Review*, 78\(1\): 1–3\.
- Cheung, Veeraraghavan, and Balakrishnan \(2026\)Cheung, M\. Y\.; Veeraraghavan, A\.; and Balakrishnan, G\. 2026\.COMPASS: Robust Feature Conformal Prediction for Medical Segmentation Metrics\.In*International Conference on Learning Representations*\.
- Deng et al\. \(2010\)Deng, J\.; Berg, A\. C\.; Li, K\.; and Fei\-Fei, L\. 2010\.What does classifying more than 10,000 image categories tell us?In*European conference on computer vision*, 71–84\. Springer\.
- Deng et al\. \(2009\)Deng, J\.; Dong, W\.; Socher, R\.; Li, L\.\-J\.; Li, K\.; and Fei\-Fei, L\. 2009\.ImageNet: A Large\-Scale Hierarchical Image Database\.In*CVPR*\.
- Dosovitskiy et al\. \(2021\)Dosovitskiy, A\.; Beyer, L\.; Kolesnikov, A\.; Weissenborn, D\.; Zhai, X\.; Unterthiner, T\.; Dehghani, M\.; Minderer, M\.; Heigold, G\.; Gelly, S\.; Uszkoreit, J\.; and Houlsby, N\. 2021\.An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale\.In*International Conference on Learning Representations*\.
- Garg, Sani, and Anand \(2022\)Garg, A\.; Sani, D\.; and Anand, S\. 2022\.Learning hierarchy aware features for reducing mistake severity\.In*European Conference on Computer Vision*, 252–267\. Springer\.
- Gneiting and Raftery \(2007\)Gneiting, T\.; and Raftery, A\. E\. 2007\.Strictly proper scoring rules, prediction, and estimation\.*Journal of the American Statistical Association*, 102\(477\): 359–378\.
- Guo et al\. \(2017\)Guo, C\.; Pleiss, G\.; Sun, Y\.; and Weinberger, K\. Q\. 2017\.On calibration of modern neural networks\.In*International Conference on Machine Learning*, 1321–1330\. PMLR\.
- He et al\. \(2016\)He, K\.; Zhang, X\.; Ren, S\.; and Sun, J\. 2016\.Deep Residual Learning for Image Recognition\.In*Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Hendrycks and Dietterich \(2019\)Hendrycks, D\.; and Dietterich, T\. 2019\.Benchmarking Neural Network Robustness to Common Corruptions and Perturbations\.In*International Conference on Learning Representations*\.
- Hendrycks and Gimpel \(2017\)Hendrycks, D\.; and Gimpel, K\. 2017\.A Baseline for Detecting Misclassified and Out\-of\-Distribution Examples in Neural Networks\.In*International Conference on Learning Representations*\.
- Hendrycks, Mazeika, and Dietterich \(2019\)Hendrycks, D\.; Mazeika, M\.; and Dietterich, T\. 2019\.Deep Anomaly Detection with Outlier Exposure\.In*International Conference on Learning Representations*\.
- Heng and Soh \(2026\)Heng, A\.; and Soh, H\. 2026\.Know When to Abstain: Optimal Selective Classification with Likelihood Ratios\.In*International Conference on Learning Representations*\.
- Hu and Khan \(2021\)Hu, Y\.; and Khan, L\. 2021\.Uncertainty\-aware reliable text classification\.In*Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining*, 628–636\.
- Hu et al\. \(2021\)Hu, Y\.; Ou, Y\.; Zhao, X\.; Cho, J\.\-H\.; and Chen, F\. 2021\.Multidimensional uncertainty\-aware evidential neural networks\.In*Proceedings of the AAAI conference on artificial intelligence*, volume 35, 7815–7822\.
- Hu et al\. \(2026\)Hu, Z\.; Zhang, Z\.; Wang, Y\.; Rutkowski, L\.; and Tao, D\. 2026\.CoFact: Conformal Factuality Guarantees for Language Models under Covariate Shift\.In*International Conference on Learning Representations*\.
- Jaeger et al\. \(2023\)Jaeger, P\. F\.; Lüth, C\. T\.; Klein, L\.; and Bungert, T\. J\. 2023\.A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification\.In*International Conference on Learning Representations*\.
- Jain, Karthik, and Gandhi \(2023\)Jain, K\.; Karthik, S\.; and Gandhi, V\. 2023\.Test\-time amendment with a coarse classifier for fine\-grained classification\.*Advances in Neural Information Processing Systems*, 36: 63372–63384\.
- Jo, Kim, and Chae \(2026\)Jo, S\. H\.; Kim, S\.; and Chae, M\. 2026\.Mitigating Spurious Correlation via Distributionally Robust Learning with Hierarchical Ambiguity Sets\.In*International Conference on Learning Representations*\.
- Karthik et al\. \(2021\)Karthik, S\.; Prabhu, A\.; Dokania, P\. K\.; and Gandhi, V\. 2021\.No cost likelihood manipulation at test time for making better mistakes in deep networks\.In*International Conference on Learning Representations*\.
- Koh et al\. \(2021\)Koh, P\. W\.; Sagawa, S\.; Marklund, H\.; Xie, S\. M\.; Zhang, M\.; Balsubramani, A\.; Hu, W\.; Yasunaga, M\.; and Liang, P\. 2021\.WILDS: A Benchmark of in\-the\-Wild Distribution Shifts\.In*International Conference on Machine Learning*, 5637–5664\. PMLR\.
- Kong et al\. \(2026\)Kong, J\.; Tang, W\.; Kong, D\.; Kong, L\.; Yang, G\.; and Jiang, B\. 2026\.Adaptive Conformal Prediction via Mixture\-of\-Experts Gating Similarity\.In*International Conference on Learning Representations*\.
- Kong et al\. \(2020\)Kong, L\.; Jiang, H\.; Zhuang, Y\.; Lyu, J\.; Zhao, T\.; and Zhang, C\. 2020\.Calibrated Language Model Fine\-Tuning for In\-and Out\-of\-Distribution Data\.In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\)*, 1326–1340\.
- Krizhevsky \(2009\)Krizhevsky, A\. 2009\.Learning Multiple Layers of Features from Tiny Images\.Technical report, University of Toronto\.
- Krizhevsky, Sutskever, and Hinton \(2012\)Krizhevsky, A\.; Sutskever, I\.; and Hinton, G\. E\. 2012\.ImageNet Classification with Deep Convolutional Neural Networks\.In Pereira, F\.; Burges, C\.; Bottou, L\.; and Weinberger, K\., eds\.,*Advances in Neural Information Processing Systems*, volume 25\. Curran Associates, Inc\.
- Lee et al\. \(2018\)Lee, K\.; Lee, K\.; Lee, H\.; and Shin, J\. 2018\.A Simple Unified Framework for Detecting Out\-of\-Distribution Samples and Adversarial Attacks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Liu et al\. \(2020\)Liu, W\.; Wang, X\.; Owens, J\. D\.; and Li, Y\. 2020\.Energy\-based Out\-of\-distribution Detection\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Liu et al\. \(2022\)Liu, Z\.; Mao, H\.; Wu, C\.\-Y\.; Feichtenhofer, C\.; Darrell, T\.; and Xie, S\. 2022\.A ConvNet for the 2020s\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, 11976–11986\.
- Lyu, Zhou, and Zhong \(2025\)Lyu, J\.; Zhou, K\.; and Zhong, Y\. 2025\.A Statistical Theory of Overfitting for Imbalanced Classification\.arXiv:2502\.11323\.
- Minderer et al\. \(2021\)Minderer, M\.; Djolonga, J\.; Romijnders, R\.; Hubis, F\.; Zhai, X\.; Houlsby, N\.; Tran, D\.; and Lucic, M\. 2021\.Revisiting the calibration of modern neural networks\.*Advances in neural information processing systems*, 34: 15682–15694\.
- Naeini, Cooper, and Hauskrecht \(2015\)Naeini, M\. P\.; Cooper, G\.; and Hauskrecht, M\. 2015\.Obtaining well calibrated probabilities using bayesian binning\.In*Proceedings of the AAAI conference on artificial intelligence*, volume 29\.
- Ovadia et al\. \(2019\)Ovadia, Y\.; Fertig, E\.; Ren, J\.; Nado, Z\.; Sculley, D\.; Nowozin, S\.; Dillon, J\. V\.; Lakshminarayanan, B\.; and Snoek, J\. 2019\.Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty under Dataset Shift\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Santurkar et al\. \(2021\)Santurkar, S\.; Tsipras, D\.; Ilyas, A\.; and Madry, A\. 2021\.BREEDS: Benchmarks for Subpopulation Shift\.In*International Conference on Learning Representations*\.
- Tu et al\. \(2024\)Tu, W\.; Deng, W\.; Campbell, D\.; Gould, S\.; and Gedeon, T\. 2024\.An Empirical Study Into What Matters for Calibrating Vision\-Language Models\.In*International Conference on Machine Learning*\.
- Van Horn et al\. \(2021\)Van Horn, G\.; Cole, E\.; Beery, S\.; Wilber, K\.; Belongie, S\.; and Mac Aodha, O\. 2021\.Benchmarking Representation Learning for Natural World Image Collections\.*arXiv preprint arXiv:2103\.16483*\.
- Wallin, Kahl, and Hammarstrand \(2025\)Wallin, E\.; Kahl, F\.; and Hammarstrand, L\. 2025\.ProHOC: Probabilistic Hierarchical Out\-of\-Distribution Classification via Multi\-Depth Networks\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Wallin, Kahl, and Hammarstrand \(2026\)Wallin, E\.; Kahl, F\.; and Hammarstrand, L\. 2026\.Semi\-Supervised Hierarchical Open\-Set Classification\.In*Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision \(WACV\)*\.
- Wang et al\. \(2025\)Wang, N\.\-S\.; Yaldiz, D\. N\.; Bakman, Y\. F\.; and Karimireddy, S\. P\. 2025\.Conformal Prediction Adaptive to Unknown Subpopulation Shifts\.*arXiv preprint arXiv:2506\.05583*\.
- Widmann, Lindsten, and Zachariah \(2019\)Widmann, D\.; Lindsten, F\.; and Zachariah, D\. 2019\.Calibration tests in multi\-class classification: A unifying framework\.*Advances in neural information processing systems*, 32\.
- Yang et al\. \(2023\)Yang, Y\.; Zhang, H\.; Katabi, D\.; and Ghassemi, M\. 2023\.Change is hard: a closer look at subpopulation shift\.In*Proceedings of the 40th International Conference on Machine Learning*, 39584–39622\.
- Zhu et al\. \(2024\)Zhu, Y\.; Chen, Y\.; Li, X\.; Zhang, R\.; Xue, H\.; Tian, X\.; Jiang, R\.; Miao, B\.; Metaxas, D\. N\.; Neill, D\. B\.; et al\. 2024\.ImageNet\-OOD: Deciphering Modern Out\-of\-Distribution Detection Algorithms\.In*International Conference on Learning Representations*\.

## Appendix AAppendix

#### A\. Datasets: final sizes, splits, and construction\.

All five benchmarks share one recipe: a coarse*supertype*valid on both train and test, with its fine*subtypes*partitioned into*seen*\(training\) and*unseen*\(test\-only\); the model predicts the supertype\. We use a strictfour\-way split:*train*/*seen\-val*/*test\-seen*/*test\-unseen*\.*train*,*seen\-val*and*test\-seen*are disjoint sets of*images*carved from the same*seen*subtypes, so no subtype label is withheld from training;*seen\-val*is used both for model selection and for fitting the post\-hoc calibrators of RQ3a \(never fit on test\)\.*test\-unseen*holds the images of the*unseen*subtypes\.No images are copied: each split is a directory of symbolic links into a single source copy of the dataset, and the exact partition is frozen in a seededsplit\_manifest\.json, the open\-sourceable artifact: we release the subtype lists and the deterministic construction code \(reproducible from\-\-seedalone\), not the images\. Models \(AlexNet, ResNet\-18, ResNet\-50, ViT\-B/16 and ConvNeXt\-T; CIFAR\-100 uses only AlexNet and ResNet\-18\) are trained from scratch on the supertype label; all metrics are computed on a per\-supertype\-balanced test set with equal\-width ECE \(Appendix[5](https://arxiv.org/html/2608.00928#A1.T5)reports a second\-seed check\)\. Table[4](https://arxiv.org/html/2608.00928#A1.T4)gives the final sizes and Table[5](https://arxiv.org/html/2608.00928#A1.T5)the full training and evaluation recipe\.

Table 4:The five subtype\-shift benchmarks: construction, splits, and protocol\. One schema throughout: predict the coarse*supertype*, train only on*seen*subtypes, test on both seen and*unseen*subtypes of the same supertype\. “subt\. \(s/u\)” = fine subtypes seen/unseen; image counts are the built subset sizes\. “Who splits; protocol” gives the partition source and how the supertype classifier is built:*direct\-coarse*\(end\-to\-end on supertype labels, from scratch\) or*fine\-pretrained frozen*\(train on seen subtypes, freeze, learn a supertype head\)\. The IN\-25 split is deliberately imbalanced across supertypes \(e\.g\.dog117 vsfungus1 seen subtype\), inherited from ImageNet\-1K; we do not rebalance it and instead evaluate on a per\-supertype\-balanced test set\. All five benchmarks are built and the full train→\\tologits→\\tocalibration pipeline is validated end\-to\-end\. Train counts are the seen set after the val holdout; IN\-25’s train pool is⊂\\subset488,644 and all test sizes are the per\-supertype\-balanced evaluation sets\.Table 5:Training and evaluation recipe\. ResNet\-50 is the primary backbone; all five architectures are swept on IN\-25 and BREEDS, while CIFAR\-100 uses AlexNet and ResNet\-18\. “Init” and “Protocol” are the two training axes\. Hyper\-parameters marked “seen\-VAL” are tuned per dataset on the held\-out seen\-subtype validation split, not fixed a priori\. The calibration metrics \(proper scores, equal\-width ECE, per\-supertype balancing\) are defined in the Evaluation section\. Post\-hoc calibrators and detection scores are fit and applied only at evaluation time on saved logits\.
#### B\. Protocol and initialization controls\.

The main table fixes two axes \(direct\-coarse, from scratch\)\. We vary each as a control fit, like the main results, on a held\-out seen\-val\.Protocol is neutral\(Table[6](https://arxiv.org/html/2608.00928#A1.T6)\): across CIFAR\-100 and both BREEDS tasks, switching one\-stage direct\-coarse for two\-stage fine\-pretrained\-frozen leaves the unseen calibration gap essentially unchanged \(\|Δ​Δcal\|<6\|\\Delta\\Delta\_\{\\mathrm\{cal\}\}\|<6, small against gaps of2020–3030\) at matched accuracy, so the two\-stage protocol neither creates nor repairs silent overconfidence\.*Pretraining reduces but does not remove the gap*\(Table[7](https://arxiv.org/html/2608.00928#A1.T7)\): on IN\-25 an ImageNet\-1K\-pretrained frozen backbone lowersΔcal\\Delta\_\{\\mathrm\{cal\}\}\(19\.2→12\.819\.2\\\!\\to\\\!12\.8\) without closing it, and it does so by making confidence itself more responsive \(at essentially unchangedΔacc\\Delta\_\{\\mathrm\{acc\}\},Δconf\\Delta\_\{\\mathrm\{conf\}\}rises from11\.211\.2to14\.514\.5\), while it brings the subtype and generic\-corruption arms together \(theirΔconf/Δacc\\Delta\_\{\\mathrm\{conf\}\}/\\Delta\_\{\\mathrm\{acc\}\}ratios become close\) and makes unseen subtypes detectable \(Mahalanobis0\.45→0\.750\.45\\\!\\to\\\!0\.75\), because its 1K features displace the 22K\-only unseen synsets in feature space\. These controls unify: the from\-scratch coarse features are blind to in\-taxonomy novelty \(the dissociation holds and detection fails\), whereas an ImageNet\-pretrained backbone partly sees it: confidence now falls somewhat under subtype shift \(narrowing the dissociation\) and the unseen synsets become separable\. Yet the overconfidence is only reduced, not removed\. Pretraining is thus a partial mitigation, and the protocol axis changes nothing\.

#### Seed sensitivity\.

The seen/unseen subtype split, the data, and the per\-architecture recipe are fixed, so a model\-init seed changes only the weight initialization\. As a robustness check we retrained a second seed and found the calibration gaps unchanged in sign and closely matched in magnitude\.

Table 6:Protocol control: one\-stage direct\-coarse vs\. two\-stage fine\-pretrained\-frozen, at matched accuracy, across three datasets \(five architectures on each BREEDS task, the two small ones on CIFAR\-100\)\. All gaps are in percentage points\. Confidence stays as unresponsive under one protocol as the other, and the unseen calibration gap barely moves \(\|Δ​Δcal\|<6\|\\Delta\\Delta\_\{\\mathrm\{cal\}\}\|<6, against gaps of2020–3030\), so the protocol neither creates nor repairs silent overconfidence\.Δacc\\Delta\_\{\\mathrm\{acc\}\}Δconf\\Delta\_\{\\mathrm\{conf\}\}Δcal\\Delta\_\{\\mathrm\{cal\}\}DatasetModel1\-stage2\-stage1\-stage2\-stage1\-stage2\-stageΔ​Δcal\\Delta\\Delta\_\{\\mathrm\{cal\}\}CIFAR\-100AlexNet34\.833\.67\.26\.827\.626\.8−0\.8\-0\.8ResNet\-1839\.939\.912\.411\.627\.528\.3\+0\.8\+0\.8Liv\-17AlexNet33\.033\.89\.89\.422\.524\.4\+1\.9\+1\.9ResNet\-1828\.729\.78\.08\.821\.020\.3−0\.6\-0\.6ResNet\-5032\.431\.68\.99\.023\.122\.7−0\.4\-0\.4ViT\-B/1637\.234\.99\.29\.628\.025\.2−2\.8\-2\.8ConvNeXt\-T32\.132\.97\.39\.024\.723\.9−0\.7\-0\.7NL\-26AlexNet35\.938\.615\.815\.420\.023\.0\+3\.0\+3\.0ResNet\-1837\.638\.315\.416\.122\.022\.0−0\.0\-0\.0ResNet\-5038\.638\.216\.215\.922\.122\.0−0\.2\-0\.2ViT\-B/1632\.930\.28\.511\.124\.519\.1−5\.4\-5\.4ConvNeXt\-T38\.036\.213\.214\.724\.821\.4−3\.3\-3\.3Table 7:Initialization control \(IN\-25, ResNet\-50\): from\-scratch direct\-coarse vs\. ImageNet\-1K\-pretrained frozen\. Gaps are in percentage points\. Pretraining reduces but does not remove the unseen calibration gap \(Δcal\\Delta\_\{\\mathrm\{cal\}\}\), and it does so by making confidence more responsive: at essentially unchangedΔacc\\Delta\_\{\\mathrm\{acc\}\},Δconf\\Delta\_\{\\mathrm\{conf\}\}rises from11\.211\.2to14\.514\.5\. It also brings the subtype and generic\-corruption arms together \(theirΔconf/Δacc\\Delta\_\{\\mathrm\{conf\}\}/\\Delta\_\{\\mathrm\{acc\}\}ratios become close\) and makes unseen subtypes detectable \(Mahalanobis novelty\-detection AUROC\)\.
#### C\. The calibration gap is robust to the binning scheme\.

The standard equal\-width estimator\(Naeini, Cooper, and Hauskrecht[2015](https://arxiv.org/html/2608.00928#bib.bib34)\)partitions\[0,1\]\[0,1\]into bins\{Ib\}b=1B\\\{I\_\{b\}\\\}\_\{b=1\}^\{B\}withBb=\{i:c​\(xi\)∈Ib\}B\_\{b\}=\\\{i:c\(x\_\{i\}\)\\in I\_\{b\}\\\}; on a splitTT,

ECE^​\(T\)\\displaystyle\\widehat\{\\mathrm\{ECE\}\}\(T\)=∑b=1B\|Bb\|\|T\|​\|acc^​\(Bb\)−conf^​\(Bb\)\|,\\displaystyle=\\sum\_\{b=1\}^\{B\}\\frac\{\|B\_\{b\}\|\}\{\|T\|\}\\left\|\\widehat\{\\mathrm\{acc\}\}\(B\_\{b\}\)\-\\widehat\{\\mathrm\{conf\}\}\(B\_\{b\}\)\\right\|,\(1\)where empty bins contribute zero\. This estimator is biased, and the bias grows as samples\-per\-bin shrink, so we never read a calibration gap off ECE alone\. Our confidences are also heavily concentrated: on IN\-25,80%80\\%of seen\-split and50%50\\%of unseen\-split predictions fall in the single top equal\-width bin, exactly the regime in which equal\-width ECE is criticised as unstable\. Appendix[A](https://arxiv.org/html/2608.00928#A1.SS0.SSS0.Px4)therefore recomputes every gap under equal\-mass bins and against two binning\-free proper scores\.

#### D\. The calibration gap does not depend on the ECE histogram\.

Table[8](https://arxiv.org/html/2608.00928#A1.T8)recomputes every gap in the main table \(Table[1](https://arxiv.org/html/2608.00928#Sx4.T1)\) two ways\. First under equal\-mass \(adaptive\) bins, which place roughly equal counts per bin and so resolve the concentrated high\-confidence region finely: the two binning schemes agree to within0\.70\.7points on every row\. Second with two binning\-free proper scores\(Gneiting and Raftery[2007](https://arxiv.org/html/2608.00928#bib.bib10)\), the negative log\-likelihoodNLL=−1\|T\|​∑ilog⁡pg​\(yi\)​\(xi\)\\mathrm\{NLL\}=\-\\tfrac\{1\}\{\|T\|\}\\sum\_\{i\}\\log p\_\{g\(y\_\{i\}\)\}\(x\_\{i\}\)and the Brier score1\|T\|​∑i∑j\(pj​\(xi\)−𝟏​\{g​\(yi\)=j\}\)2\\tfrac\{1\}\{\|T\|\}\\sum\_\{i\}\\sum\_\{j\}\(p\_\{j\}\(x\_\{i\}\)\-\\mathbf\{1\}\\\{g\(y\_\{i\}\)=j\\\}\)^\{2\}; both rise from seen to unseen on all2222rows, and their rise tracksΔcal\\Delta\_\{\\mathrm\{cal\}\}closely \(Spearmanρ=0\.96\\rho=0\.96and0\.950\.95\)\. The gap is therefore a property of the predicted probabilities, not of where the bin edges fall\.

Table 8:The calibration gap under a second binning scheme and under two binning\-free proper scores\.Δcal\\Delta\_\{\\mathrm\{cal\}\}is in percentage points;Δ\\DeltaNLL andΔ\\DeltaBrier are the unseen minus seen rise in each score\. All three move together on every row\.
#### E\. Per\-dataset construction details\.

We detail here the construction of*iNaturalist*\(iNat\-25\)\(Van Horn et al\.[2021](https://arxiv.org/html/2608.00928#bib.bib38)\), whose real biological taxonomy broadens coverage along an axis the other main datasets cannot reach\. It is evaluated with the identical pipeline used in the main text: train a coarse \(supertype\) classifier on seen subtypes, then measure accuracy and calibration on held\-out \(unseen\) subtypes, so ECE/NLL/Brier/Δcal\\Delta\_\{\\mathrm\{cal\}\}are directly comparable\.

iNaturalist\.From iNat2021 \(10,00010\{,\}000species,2\.72\.7M images; we use the official5050\-image/speciestrain\_minisubset for training and the1010/speciesvalsplit for evaluation\) we take a biological rank as the supertype \(order: the2525orders with≥100\\geq\\\!100species, spanning three kingdoms; orfamilyfor finer, better\-balanced supertypes\) and split the species under each supertype into seen vs\. unseen\. The2525\-order subset covers≈6,854\\approx\\\!6\{,\}854species, giving≈171\\approx\\\!171K training images \(3,4273\{,\}427seen species×50\\times 50\) and≈34\\approx\\\!34K images per test split \(×10\\times 10\); order imbalance is handled by macro\-averaging rather than capping\. A ResNet\-50 is trained from scratch to predict the supertype on seen\-species images; held\-out species \(same supertypes\) form the test split\. Since the supertype is a true taxonomic node, the seen→\\tounseen*taxonomic distance*\(same genus / family / order\) is a controlled severity knob\. Ready\-made species partitions are available from prior work\(Wallin, Kahl, and Hammarstrand[2025](https://arxiv.org/html/2608.00928#bib.bib39),[2026](https://arxiv.org/html/2608.00928#bib.bib40)\)\.

Protocol\.ResNet\-50 from scratch to avoid pretraining leakage; per\-supertype\-balanced accuracy and ECE \(adaptive binning\); the two post\-hoc calibrators, seen\-side risk predictors, and confidence\-based detection computed exactly as in the main text\. A matched generic\-corruption \(covariate\-shift\) arm is run to confirm that the calibration failure follows semantic subtype novelty rather than appearance degradation\.

#### F\. Full reliability diagrams\.

Figure[6](https://arxiv.org/html/2608.00928#A1.F6)draws reliability curves \(binned confidence vs\. empirical accuracy\) for all five architectures on IN\-25, seen and unseen\. The seen curves track the diagonal; the unseen curves sit well below it \(overconfident\) on every architecture, so the gap is not an artifact of any one backbone\.

![Refer to caption](https://arxiv.org/html/2608.00928v1/x6.png)Figure 6:Reliability diagrams \(IN\-25, from scratch, direct\-coarse; ResNet\-50 and four other backbones\): the unseen curve falls below the diagonal \(overconfident\) on every architecture, and temperature scaling would slide each curve along, not onto, the diagonal\.
#### G\. Per\-sample discrimination: failure\-detection AUROC and risk–coverage AURC\.

Calibration is an aggregate property and does not say whether confidence still ranks individual predictions usefully, so we report two per\-sample measures on each split \(Table[9](https://arxiv.org/html/2608.00928#A1.T9)\)\. The*failure\-detection AUROC*asks whetherc​\(x\)c\(x\)separates correct from incorrect predictions within a split: it is the probability that a randomly drawn correct prediction is more confident than a randomly drawn incorrect one \(11= perfect,0\.50\.5= chance\)\. The AURC is the area under the risk–coverage curve obtained by answering only the most confident fraction of inputs and abstaining on the rest \(lower is better\), so a rise means the same confidence\-thresholded policy now incurs more error at equal coverage\.

Both degrade on unseen subtypes for every dataset, and the magnitude follows the severity of the shift: the drop is\+0\.15\+0\.15–0\.180\.18on CIFAR\-100, IN\-25 and BREEDS, whose accuracy gaps are2929–4040points, but only\+0\.05\+0\.05on iNat\-25, whose accuracy gap is12\.612\.6; AURC rises2\.3×2\.3\\times\(iNat\-25\) to18×18\\times\(IN\-25\)\. AUROC is rank\-based and therefore invariant to any monotone rescaling of confidence: unlike the calibration gap, this degradation cannot be repaired by temperature scaling\.

#### Novelty or error detection?

The novelty\-detection AUROCs of Table[3](https://arxiv.org/html/2608.00928#Sx4.T3)could in principle be trivial error detection: accuracy is lower on unseen subtypes, and confidence correlates with correctness, so a score that merely ranks errors would separate the two splits\. Restricting each score to inputs the model classifies correctly on both splits removes that confound\. The separation survives, falling by at most0\.040\.04: MSP0\.723→0\.6840\.723\\\!\\to\\\!0\.684\(IN\-25\),0\.723→0\.7010\.723\\\!\\to\\\!0\.701\(Liv\-17\),0\.740→0\.7140\.740\\\!\\to\\\!0\.714\(NL\-26\) and0\.563→0\.5380\.563\\\!\\to\\\!0\.538\(iNat\-25\), with energy behaving the same way \(0\.711→0\.6810\.711\\\!\\to\\\!0\.681,0\.698→0\.6800\.698\\\!\\to\\\!0\.680,0\.738→0\.7020\.738\\\!\\to\\\!0\.702,0\.563→0\.5450\.563\\\!\\to\\\!0\.545\)\. Mahalanobis is already at chance, so its within\-correct value is within noise and omitted\. The weak signal is therefore a genuine response to subtype novelty rather than an artifact of correctness\.

Table 9:Per\-sample discrimination \(ResNet\-50; ResNet\-18 for CIFAR\-100, matching the main table\), seen vs\. unseen\. Failure\-detection AUROC \(higher is better,0\.50\.5= chance\) and risk–coverage AURC \(lower is better\)\.
#### H\. Do the RQ3 conclusions depend on the architecture?

The main text reports RQ3a and RQ3b on ResNet\-50\. Since both only need saved logits, we can repeat them for all five architectures at no additional training cost \(Table[10](https://arxiv.org/html/2608.00928#A1.T10)\)\. The conclusions do not move\. Seen\-tuned temperature scaling reducesΔcal\\Delta\_\{\\mathrm\{cal\}\}on every one of the2020\(dataset, architecture\) pairs but never to zero, leaving3\.33\.3to19\.419\.4points; vector scaling leaves5\.15\.1to23\.423\.4\. Novelty detection stays weak everywhere, with the maximum softmax probability between0\.520\.52and0\.750\.75AUROC\. The one pattern the ResNet\-50 row hides is that ViT\-B/16 has the*largest*gap on all three ImageNet\-derived benchmarks, both before and after recalibration, so a more modern backbone is not the safer choice here\.

Table 10:RQ3 across architectures \(direct\-coarse from scratch\)\.Δcal\\Delta\_\{\\mathrm\{cal\}\}in percentage points under no calibration, temperature scaling and vector scaling; novelty\-detection AUROC for the two output\-based scores\.
#### I\. Is the gap repairable by a single temperature?

The standard fix for miscalibration under shift, re\-weighting the seen calibration set toward the unseen distribution, cannot help here: it borrows the corrective scale from calibration examples that resemble the test input, and an unseen subtype has almost none\. As a diagnostic rather than a deployable remedy, we therefore fit an*unseen\-oracle*temperatureT𝒰∗T^\{\*\}\_\{\\mathcal\{U\}\}directly on the unseen split, using its labels and hence unattainable in practice\. It is far larger than the seen\-fitT𝒮∗T^\{\*\}\_\{\\mathcal\{S\}\}\(T𝒰∗≈2T^\{\*\}\_\{\\mathcal\{U\}\}\\approx 2–33vs\.1\.21\.2–1\.71\.7\) and brings the unseen ECE down to22–55, so the aggregate overconfidence is a coherent single\-scale error that one temperature could repair in principle\. Two caveats keep this from being a remedy\. First, temperature is monotone, so even the oracle leaves the per\-sample discrimination loss of RQ1 untouched: it repairs the average confidence, not the broken ranking of correct against incorrect predictions\. Second, no single temperature serves both splits at once, since the unseen\-oracle scale over\-softens the seen inputs \(their ECE rises to≈10\{\\approx\}10\)\. It would have to be applied selectively, which requires knowing which inputs are unseen, and that is the detection question of RQ3b\.

#### J\. Per\-category heterogeneity\.

The collapse is highly uneven across coarse classes\. Table[11](https://arxiv.org/html/2608.00928#A1.T11)gives per\-supertype seen/unseen accuracy and ECE \(IN\-25, ResNet\-50\), sorted by accuracy drop: supertypes with few seen subtypes collapse almost completely \(big cat, accuracy96\.0%96\.0\\%seen vs\.5\.3%5\.3\\%unseen, ECE22vs\.8181\), while richly\-sampled ones stay calibrated \(dog, 117 subtypes, ECE1\.41\.4vs\.2\.22\.2\), mirroring thenjn\_\{j\}trend of Figure[5](https://arxiv.org/html/2608.00928#Sx4.F5)\.

Table 11:Per\-supertype collapse \(IN\-25, ResNet\-50\), sorted by accuracy drop\.njn\_\{j\}= number of seen subtypes; accuracy \(%\) and ECE on seen vs\. unseen\.
#### K\. Fine\-label diversity \(kk\-sweep\)\.

Table[12](https://arxiv.org/html/2608.00928#A1.T12)varies the number of seen fine subtypes per supertype,kk, on IN\-25 \(ResNet\-50, from scratch\) while holding the per\-class image budget fixed, so only fine\-label diversity changes\. The unseen calibration gap falls monotonically askkgrows \(Δcal\\Delta\_\{\\mathrm\{cal\}\}31→2331\\\!\\to\\\!23fromk=1k\{=\}1to44\)\. This upgrades the correlationalnjn\_\{j\}trend of the Analysis to a causal statement: training on more fine subtypes causes less overconfidence on unseen ones, holding image count fixed\. \(Subtypes are capped here, so the absolute gap is larger than the full\-data main\-table row\.\)

Table 12:Fine\-label diversitykk\-sweep \(IN\-25, ResNet\-50 from scratch; per\-class image budget fixed\)\. Raising the number of seen subtypes per supertype monotonically shrinks the unseen calibration gap\.