FALCON-Discover: Discovering Concentrated False-Confidence Regions for Calibration

# FALCON-Discover: Discovering Concentrated False-Confidence Regions for Calibration
Source: [https://arxiv.org/html/2607.18278](https://arxiv.org/html/2607.18278)
Filippo Cenacchi, Longbing Cao, and Runze Yang Macquarie University, Sydney, Australia filippo\.cenacchi@mq\.edu\.au, longbing\.cao@mq\.edu\.au, runze\.yang@hdr\.mq\.edu\.au

###### Abstract

Calibration is usually evaluated at the aggregate level: confidence is judged by whether it aligns, on average across samples, with empirical correctness\. This perspective is indispensable but incomplete, because the most operationally dangerous failures are often not average errors but a relatively small subset of predictions that remain highly confident despite being wrong\. We study this failure mode as its own empirical object and introduce*false\-confidence concentration*: the extent to which dangerous confident errors occupy a compact and discoverable slice of the prediction space\. This shifts the focus from population\-level calibration quality alone to the discovery of sample\-level regions in which confidence becomes systematically unreliable\. We present*FALCON\-Discover*, a post\-hoc discrepancy\-discovery framework that ranks predictions using signals derived from confidence, local support, neighborhood agreement, and perturbation stability under controlled support\-preserving perturbations\. The framework is model\-agnostic and does not alter the underlying classifier\. Across seven heterogeneous binary tabular datasets, with four seeds, five\-fold cross\-fitting, and strong base learners including XGBoost and CatBoost, we find that false\-confidence concentration is a recurrent but regime\-dependent property\. At the main confidence threshold, the discrepancy family substantially outperforms the strongest validation\-selected prior baseline in the strongest regimes, while raw confidence ranking recovers little dangerous\-error mass\. The strongest family member varies: learned discrepancy is strongest when multiple cues must be combined, whereas stability\-centered ranking is strongest when local decisional fragility dominates\. These results establish false\-confidence concentration as a family\-level discovery problem rather than a single\-score dominance claim, and motivate calibration strategies that explicitly target regions where confidence, support, and stability diverge\.

## 1Introduction

A deployed predictive model is not consumed through aggregate statistics alone\. In practice, people encounter one prediction, one confidence value, and one decision consequence at a time\. This simple fact creates a persistent tension in calibration research\. On one hand, aggregate calibration summaries such as Expected Calibration Error, Brier score, negative log\-likelihood, and reliability diagrams remain indispensable because they answer whether predicted confidence is aligned, at the population level, with observed correctness on average\(Guoet al\.,[2017](https://arxiv.org/html/2607.18278#bib.bib1); Platt,[1999](https://arxiv.org/html/2607.18278#bib.bib2); Zadrozny and Elkan,[2001](https://arxiv.org/html/2607.18278#bib.bib3),[2002](https://arxiv.org/html/2607.18278#bib.bib4); Vaicenaviciuset al\.,[2019](https://arxiv.org/html/2607.18278#bib.bib26); Mindereret al\.,[2021](https://arxiv.org/html/2607.18278#bib.bib27)\)\. On the other hand, these summaries do not directly answer the practically decisive question:*which predictions are dangerous to trust?*A model can appear well calibrated in aggregate while still containing a small but consequential subset of predictions that remain highly confident despite being wrong\. These are often the failures that matter most because they are least likely to be inspected, least likely to trigger caution, but most likely to propagate harmful downstream decisions\. The post\-hoc calibration literature has made major progress on score alignment, but it has largely retained a scalar view of the problem\. Temperature scaling\(Guoet al\.,[2017](https://arxiv.org/html/2607.18278#bib.bib1)\), Platt scaling\(Platt,[1999](https://arxiv.org/html/2607.18278#bib.bib2)\), isotonic regression\(Zadrozny and Elkan,[2001](https://arxiv.org/html/2607.18278#bib.bib3),[2002](https://arxiv.org/html/2607.18278#bib.bib4)\), beta calibration\(Kullet al\.,[2017](https://arxiv.org/html/2607.18278#bib.bib23),[2019](https://arxiv.org/html/2607.18278#bib.bib24)\), Bayesian binning\(Naeiniet al\.,[2015](https://arxiv.org/html/2607.18278#bib.bib22)\), and Dirichlet calibration all focus on improving a mapping from an existing score to a better calibrated probability estimate\. This body of work shows that strong discrimination does not imply strong calibration and that probability quality can often be improved substantially after training\(Niculescu\-Mizil and Caruana,[2005](https://arxiv.org/html/2607.18278#bib.bib21); Kuleshovet al\.,[2018](https://arxiv.org/html/2607.18278#bib.bib25)\)\. Recent work has also expanded the calibration landscape beyond the classical post\-hoc setting, including train\-time uncertainty–error alignment\(Mendeset al\.,[2025](https://arxiv.org/html/2607.18278#bib.bib40)\), confidence–calibration dynamics during model training\(Durai,[2025](https://arxiv.org/html/2607.18278#bib.bib42)\), and post\-hoc methods designed to reduce confidently incorrect predictions beyond standard aggregate calibration metrics\(Denoodt and Oramas,[2025](https://arxiv.org/html/2607.18278#bib.bib39); Gharounet al\.,[2025](https://arxiv.org/html/2607.18278#bib.bib43)\)\. Yet even in these newer directions, the central object usually remains a score distribution summarized globally, rather than a compact structural slice in which dangerous confident failures accumulate\(Denoodt and Oramas,[2025](https://arxiv.org/html/2607.18278#bib.bib39); Mendeset al\.,[2025](https://arxiv.org/html/2607.18278#bib.bib40); Durai,[2025](https://arxiv.org/html/2607.18278#bib.bib42); Gharounet al\.,[2025](https://arxiv.org/html/2607.18278#bib.bib43)\)\. A neighboring literature goes further by asking whether a model’s output should be trusted, whether a prediction should be abstained from, or whether an auxiliary score predicts correctness more effectively than native confidence\(Jianget al\.,[2018](https://arxiv.org/html/2607.18278#bib.bib5); Corbièreet al\.,[2019](https://arxiv.org/html/2607.18278#bib.bib28); Hendrycks and Gimpel,[2017](https://arxiv.org/html/2607.18278#bib.bib18); Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2607.18278#bib.bib7); El\-Yaniv and Wiener,[2010](https://arxiv.org/html/2607.18278#bib.bib29); Geifman and El\-Yaniv,[2019](https://arxiv.org/html/2607.18278#bib.bib8); Romanoet al\.,[2020](https://arxiv.org/html/2607.18278#bib.bib30)\)\. These directions are highly relevant because they restore decision\-time risk to the reliability problem\. However, even here the dominant framing remains mostly score\-centric:can one construct a better trust score, a better correctness predictor, or a better abstention signal than confidence alone?This paper starts from a different observation: dangerous overconfidence is often concentrated, so a small subset of samples can contain a disproportionate share of all high\-confidence errors\. This suggests a more useful empirical target: a compact, structurally identifiable slice of the prediction space whose review surfaces many failures that are unsafe to trust\. We therefore reframe calibration analysis as discrepancy discovery: rather than asking only whether confidence can be globally reshaped, we ask whether predictions ranked by disagreement between confidence, support, and structural stability recover more dangerous error mass than confidence alone\. This yields*FALCON\-Discover*, a post\-hoc, model\-agnostic framework that builds a discrepancy representation from native decision signals, local support, neighborhood agreement, and perturbation stability, then uses it to rank dangerous errors, localize discrepancy regions, and derive calibration\-facing weights\. Empirically, false\-confidence concentration emerges as a regime\-resolved phenomenon: it is strong and threshold\-stable in several benchmark regimes, mixed when stronger predictors absorb local structure, and boundary\-limited when high\-confidence error events become sparse\. The strongest detector varies by regime: the learned discrepancy ranker is strongest in some cases, while a simpler stability\-centered rule is stronger in others, showing that the phenomenon is broader than any single detector\. Accordingly, ranking is only one operational view; the broader contribution is a discrepancy representation for diagnosis, localization, and calibration refinement\. Our goal is not to replace aggregate calibration analysis, but to complement it with a sample\-level view of where confident predictions become structurally unsafe to trust\. We evaluate this claim on seven heterogeneous binary tabular datasets using four seeds and five\-fold cross\-fitting, fixingτ=0\.90\\tau=0\.90and review budgets in\{5%,10%,15%,20%\}\\\{5\\%,10\\%,15\\%,20\\%\\\}, and measuring concentration via FalseConf\-AUROC and Capture@20 against validation\-selected calibration and trust\-scoring baselines\.

The paper makes five contributions\. First, it formalizes*false\-confidence concentration*as a structural reliability object: the recoverability of compact high\-risk error slices hidden by aggregate calibration summaries\. Second, it introduces*prediction\-to\-structure learning*, a post\-hoc paradigm in which held\-out prediction behavior is lifted into a reusable discrepancy state rather than reduced to another scalar trust score\. Third, it characterizes when concentration should arise: high native certainty must coincide with weak local evidence or instability strongly enough to amplify dangerous\-error density inside a compact conflict set\. Fourth, it supplies controlled multi\-dataset evidence across thresholds and strong tabular backbones\. Fifth, it separates strong, mixed, and boundary regimes, making the claim explicitly family\-level rather than a universal dominance statement\.

## 2Related Work

Post\-hoc calibration is usually framed as aligning a scalar score with empirical correctness\. Temperature scaling remains a strong standard baseline because it is simple and architecture\-agnostic\(Guoet al\.,[2017](https://arxiv.org/html/2607.18278#bib.bib1)\)\. Platt scaling, isotonic regression, beta calibration, Bayesian binning, and Dirichlet calibration occupy nearby points in the same design space, transforming raw scores into probability estimates that better match observed frequencies\(Platt,[1999](https://arxiv.org/html/2607.18278#bib.bib2); Zadrozny and Elkan,[2001](https://arxiv.org/html/2607.18278#bib.bib3),[2002](https://arxiv.org/html/2607.18278#bib.bib4); Naeiniet al\.,[2015](https://arxiv.org/html/2607.18278#bib.bib22); Kullet al\.,[2017](https://arxiv.org/html/2607.18278#bib.bib23),[2019](https://arxiv.org/html/2607.18278#bib.bib24)\)\. The broader lesson is that discrimination and calibration are distinct, and that accurate classifiers can still be poor sources of probabilities\(Niculescu\-Mizil and Caruana,[2005](https://arxiv.org/html/2607.18278#bib.bib21); Kuleshovet al\.,[2018](https://arxiv.org/html/2607.18278#bib.bib25); Mindereret al\.,[2021](https://arxiv.org/html/2607.18278#bib.bib27)\)\. Recent work has extended this view through efficient post\-hoc uncertainty calibration, train\-time uncertainty–error alignment, and confidence–calibration dynamics in modern language models\(Denoodt and Oramas,[2025](https://arxiv.org/html/2607.18278#bib.bib39); Mendeset al\.,[2025](https://arxiv.org/html/2607.18278#bib.bib40); Durai,[2025](https://arxiv.org/html/2607.18278#bib.bib42)\)\. Our work addresses a complementary question left underemphasized by this literature: where dangerous confident failures accumulate, how concentrated they are, and whether a small review slice can recover them efficiently\. A nearby literature asks whether a model’s output should be trusted\. TrustScore replaces raw confidence with a class\-relative support signal\(Jianget al\.,[2018](https://arxiv.org/html/2607.18278#bib.bib5)\); failure\-prediction methods learn whether a model is likely to be wrong\(Corbièreet al\.,[2019](https://arxiv.org/html/2607.18278#bib.bib28); Hendrycks and Gimpel,[2017](https://arxiv.org/html/2607.18278#bib.bib18)\); and selective classification methods study reject, defer, or bounded\-error prediction under partial coverage\(El\-Yaniv and Wiener,[2010](https://arxiv.org/html/2607.18278#bib.bib29); Geifman and El\-Yaniv,[2017](https://arxiv.org/html/2607.18278#bib.bib7),[2019](https://arxiv.org/html/2607.18278#bib.bib8); Romanoet al\.,[2020](https://arxiv.org/html/2607.18278#bib.bib30)\)\. A broader uncertainty literature studies reliability under distribution shift, ensemble disagreement, Bayesian approximations, and energy\-based formulations\(Lakshminarayananet al\.,[2017](https://arxiv.org/html/2607.18278#bib.bib31); Gal and Ghahramani,[2016](https://arxiv.org/html/2607.18278#bib.bib12); Ovadiaet al\.,[2019](https://arxiv.org/html/2607.18278#bib.bib14); Ashukhaet al\.,[2020](https://arxiv.org/html/2607.18278#bib.bib32); Liuet al\.,[2020](https://arxiv.org/html/2607.18278#bib.bib33)\), while conformal prediction addresses set\-valued uncertainty with coverage guarantees\(Vovket al\.,[2005](https://arxiv.org/html/2607.18278#bib.bib35); Romanoet al\.,[2020](https://arxiv.org/html/2607.18278#bib.bib30)\)\. Structured reporting frameworks such as Model Cards and Datasheets add the complementary principle that failure modes should be made legible rather than hidden behind averages\(Mitchellet al\.,[2019](https://arxiv.org/html/2607.18278#bib.bib34); Gebruet al\.,[2021](https://arxiv.org/html/2607.18278#bib.bib11)\)\. These lines of work reinforce the same methodological intuition: confidence should not be interpreted in isolation\. What remains underdeveloped is a direct study of whether*confident*errors form recoverable local failure regions, whether those regions can be surfaced by reusable discrepancy states rather than by another calibrated probability or reject score, and whether the resulting structure can guide later calibration refinement\. This is the gap addressed\.

## 3Method

Figure[1](https://arxiv.org/html/2607.18278#S3.F1)summarizes the FALCON\-Discover pipeline\. The method is motivated by a simple hypothesis: dangerous overconfidence should be most visible when strong predictive certainty coexists with weak local support or instability under small, support\-preserving changes\. FALCON\-Discover therefore does not introduce one more scalar trust score in isolation\. Instead, it decomposes reliability into three complementary questions for each sample: how confident the model is, how well that sample is supported by nearby training data, and how stable the same prediction remains under small local perturbations\. These signals are combined into a shared discrepancy representation, which is then reused to rank dangerous samples, localize discrepancy regions, and derive calibration\-facing weights\. The central design choice is thus a structured discrepancy representation whose role is diagnostic and actionable rather than a collection of disconnected engineered scores\. The learning object is not a new classifier, but a post\-hoc*failure\-discovery representation*: the base model is fixed, while held\-out behavior is mapped into a discrepancy space that exposes where confidence, support, and stability conflict\. This distinguishes FALCON\-Discover from calibration maps, failure predictors, and abstention rules, which usually return a probability, correctness score, or reject decision rather than a reusable structure for ranking, localization, and calibration guidance\.

![Refer to caption](https://arxiv.org/html/2607.18278v1/diagram.png)Figure 1:Overview of FALCON\-Discover\.Held\-out predictions from a cross\-fitted base model are mapped to a discrepancy representationψ​\(x\)\\psi\(x\)combining certainty, local support/agreement, and perturbation stability, then reused to rank dangerous samples, localize discrepancy regions, and derive calibration\-facing weights\.FALCON\-Discover is instantiated for binary classification, matching all benchmark tasks\. Stage 1 produces held\-out scoresp​\(x\)p\(x\)and decisionsy^​\(x\)\\hat\{y\}\(x\); Stage 2 buildsψ​\(x\)\\psi\(x\)from certainty, support/agreement, and stability signals; and Stage 3 reusesψ​\(x\)\\psi\(x\)for ranking, region discovery, and calibration\-facing weights\. Multiclass extension is plausible but outside the present scope\. Let a trained binary classifier produce a positive\-class scorep​\(x\)∈\[0,1\]p\(x\)\\in\[0,1\]for samplexx, with hard decision

y^​\(x\)=𝕀​\[p​\(x\)≥0\.5\]\.\\tiny\\hat\{y\}\(x\)=\\mathbb\{I\}\[p\(x\)\\geq 0\.5\]\.\(1\)We define confidence as

conf​\(x\)=max⁡\{p​\(x\),1−p​\(x\)\}\.\\tiny\\mathrm\{conf\}\(x\)=\\max\\\{p\(x\),1\-p\(x\)\\\}\.\(2\)For a user\-selected confidence thresholdτ∈\(0,1\)\\tau\\in\(0,1\), we define the false\-confidence event

FCτ​\(x\)=𝕀​\[y^​\(x\)≠y⏟prediction is wrong∧conf​\(x\)≥τ⏟model is highly confident\]\.\\tiny\\mathrm\{FC\}\_\{\\tau\}\(x\)=\\mathbb\{I\}\\\!\\left\[\\underbrace\{\\hat\{y\}\(x\)\\neq y\}\_\{\\text\{prediction is wrong\}\}\\;\\land\\;\\underbrace\{\\mathrm\{conf\}\(x\)\\geq\\tau\}\_\{\\text\{model is highly confident\}\}\\right\]\.\(3\)FCτ​\(x\)\\mathrm\{FC\}\_\{\\tau\}\(x\)isolates the failure cases most likely to evade manual skepticism because they combine incorrectness with high apparent certainty\. Given a ranking rules​\(x\)s\(x\)and a review budgetα∈\(0,1\)\\alpha\\in\(0,1\), letTopα​\(s\)\\mathrm\{Top\}\_\{\\alpha\}\(s\)denote the topα\\alphafraction of test samples sorted by descendings​\(x\)s\(x\)\. We evaluate concentration through

Capture​@​α​\(s\)=∑x∈Topα​\(s\)FCτ​\(x\)∑xFCτ​\(x\)\.\\mathrm\{Capture\}@\\alpha\(s\)=\\frac\{\\sum\_\{x\\in\\mathrm\{Top\}\_\{\\alpha\}\(s\)\}\\mathrm\{FC\}\_\{\\tau\}\(x\)\}\{\\sum\_\{x\}\\mathrm\{FC\}\_\{\\tau\}\(x\)\}\.\(4\)Equation \([4](https://arxiv.org/html/2607.18278#S3.E4)\) formalizes the paper’s operational target: under a fixed review budget, how much dangerous error mass can be surfaced? To state more precisely when false\-confidence concentration should arise, define the*conflict\-amplification gap*

Γτ​\(c,e,s\)=Pr⁡\(FCτ​\(x\)=1∣conf​\(x\)≥c,ℰ​\(x\)≤e,𝒮​\(x\)≥s\)−Pr⁡\(FCτ​\(x\)=1∣conf​\(x\)≥c\),\\Gamma\_\{\\tau\}\(c,e,s\)=\\Pr\\\!\\big\(\\mathrm\{FC\}\_\{\\tau\}\(x\)=1\\mid\\mathrm\{conf\}\(x\)\\geq c,\\ \\mathcal\{E\}\(x\)\\leq e,\\ \\mathcal\{S\}\(x\)\\geq s\\big\)\-\\Pr\\\!\\big\(\\mathrm\{FC\}\_\{\\tau\}\(x\)=1\\mid\\mathrm\{conf\}\(x\)\\geq c\\big\),\(5\)together with the corresponding slice mass

M​\(c,e,s\)=Pr⁡\(conf​\(x\)≥c,ℰ​\(x\)≤e,𝒮​\(x\)≥s\)\.M\(c,e,s\)=\\Pr\\\!\\big\(\\mathrm\{conf\}\(x\)\\geq c,\\ \\mathcal\{E\}\(x\)\\leq e,\\ \\mathcal\{S\}\(x\)\\geq s\\big\)\.\(6\)Hereℰ​\(x\)\\mathcal\{E\}\(x\)denotes local evidence quantities such as support or neighborhood agreement, and𝒮​\(x\)\\mathcal\{S\}\(x\)denotes instability quantities such asΔmean​\(x\)\\Delta\_\{\\mathrm\{mean\}\}\(x\)or1−LC​\(x\)1\-\\mathrm\{LC\}\(x\)\. False\-confidence concentration should be strongest whenΓτ​\(c,e,s\)\\Gamma\_\{\\tau\}\(c,e,s\)is clearly positive on a slice whose mass is non\-trivial but not diffuse: in that case, conditioning on structural conflict raises dangerous\-error density relative to high confidence alone while keeping the region compact enough to be recoverable under a small review budget\. By contrast, when confidence, support, and stability remain aligned,Γτ​\(c,e,s\)\\Gamma\_\{\\tau\}\(c,e,s\)becomes small and discrepancy\-aware ranking should collapse toward confidence ranking\. This yields the three empirical regimes used later in the paper:*strong*regimes exhibit clear conflict amplification on recoverable slices,*mixed*regimes exhibit weaker or backbone\-dependent amplification, and*boundary*regimes have event spaces too sparse for reliable concentration estimates\. The characterization places the paper between global calibration and selective prediction: calibration asks whether confidence matches frequency on average\(Guoet al\.,[2017](https://arxiv.org/html/2607.18278#bib.bib1); Vaicenaviciuset al\.,[2019](https://arxiv.org/html/2607.18278#bib.bib26); Mindereret al\.,[2021](https://arxiv.org/html/2607.18278#bib.bib27)\), while trust scoring and selective prediction ask for a better accept/reject signal\(Jianget al\.,[2018](https://arxiv.org/html/2607.18278#bib.bib5); El\-Yaniv and Wiener,[2010](https://arxiv.org/html/2607.18278#bib.bib29); Romanoet al\.,[2020](https://arxiv.org/html/2607.18278#bib.bib30)\); our target is the intermediate structural question of whether dangerous high\-confidence error mass occupies a compact conflict region at all\. The contribution is therefore structural rather than distribution\-free: without additional assumptions linking event prevalence, conflict amplification, and slice mass, no non\-vacuous guarantee can identify concentrated false\-confidence regions from finite held\-out data alone\. We therefore treat concentration as a measurable property of a trained model and dataset, and evaluate it through held\-out ranking, threshold robustness, bootstrap intervals, fixed\-backbone sensitivity, and regime analysis\. The method then instantiates a family of discrepancy\-aware ranking rules and tests whether a small ranked slice captures a disproportionate share of false\-confidence events\. In this sense, the design is narrower than an end\-to\-end abstention architecture and more operational than a purely global calibration map, because it targets concentrated reliability failure directly and translates the discovered structure into region summaries and calibration\-facing weights\. Strict cross\-fitting is essential\. Let𝒟train\\mathcal\{D\}\_\{\\mathrm\{train\}\},𝒟val\\mathcal\{D\}\_\{\\mathrm\{val\}\}, and𝒟test\\mathcal\{D\}\_\{\\mathrm\{test\}\}denote disjoint train, validation, and test partitions\. For the current benchmark, the base classifier is selected from a candidate library containing logistic regression, random forests, extra trees, HistGradientBoosting, multilayer perceptrons, XGBoost, and CatBoost\. For each candidate in this pool, we produce out\-of\-fold predictions on𝒟train\\mathcal\{D\}\_\{\\mathrm\{train\}\}using stratifiedKK\-fold cross\-fitting and select the model with the strongest out\-of\-fold label AUROC rather than raw accuracy\. The chosen base learner is then refit on the full training partition and evaluated on validation and test splits\. All downstream discrepancy features are therefore constructed from out\-of\-fold or held\-out predictions, not from predictions contaminated by training leakage\. The three signal families play distinct roles: certainty captures how strongly the model commits to its decision, support captures whether that decision is locally grounded in the training data, and stability captures whether the same decision persists under small local changes that preserve coarse semantic identity\. For each samplexx, we construct a structured signal familyψ​\(x\)\\psi\(x\)from these three signal families\. The first family captures native certainty:

margin​\(x\)=\|2​p​\(x\)−1\|,\\tiny\\mathrm\{margin\}\(x\)=\|2p\(x\)\-1\|,\(7\)ent​\(x\)=−p​\(x\)​log⁡p​\(x\)−\(1−p​\(x\)\)​log⁡\(1−p​\(x\)\)\.\\tiny\\mathrm\{ent\}\(x\)=\-p\(x\)\\log p\(x\)\-\(1\-p\(x\)\)\\log\(1\-p\(x\)\)\.\(8\)The second family captures support and local agreement\. Letx~\\tilde\{x\}denote the transformed and standardized representation ofxx\. We define a normalized support score

supp​\(x\)=1−clip​\(dsup​\(x\)−q0\.50q0\.95−q0\.50,0,1\),\\tiny\\mathrm\{supp\}\(x\)=1\-\\mathrm\{clip\}\\\!\\left\(\\frac\{d\_\{\\mathrm\{sup\}\}\(x\)\-q\_\{0\.50\}\}\{q\_\{0\.95\}\-q\_\{0\.50\}\},\\,0,\\,1\\right\),\(9\)where

dsup​\(x\)=1d​∑j=1d\(x~j−μj\)2σj2\+ε⏟normalized distance from the transformed training manifold\\tiny d\_\{\\mathrm\{sup\}\}\(x\)=\\underbrace\{\\frac\{1\}\{d\}\\sum\_\{j=1\}^\{d\}\\frac\{\(\\tilde\{x\}\_\{j\}\-\\mu\_\{j\}\)^\{2\}\}\{\\sigma\_\{j\}^\{2\}\+\\varepsilon\}\}\_\{\\text\{normalized distance from the transformed training manifold\}\}\(10\)andq0\.50,q0\.95q\_\{0\.50\},q\_\{0\.95\}are the training\-set median and upper quantile\. We also define neighborhood agreement in transformed space\. Let𝒩k​\(x\)\\mathcal\{N\}\_\{k\}\(x\)be thekknearest neighbors ofxxamong the cross\-fitted training embeddings\. Then

agrlabel​\(x\)=max⁡\{1k​∑x′∈𝒩k​\(x\)𝕀​\[y​\(x′\)=1\],1k​∑x′∈𝒩k​\(x\)𝕀​\[y​\(x′\)=0\]\},\\tiny\\mathrm\{agr\}\_\{\\mathrm\{label\}\}\(x\)=\\max\\\!\\left\\\{\\frac\{1\}\{k\}\\sum\_\{x^\{\\prime\}\\in\\mathcal\{N\}\_\{k\}\(x\)\}\\mathbb\{I\}\[y\(x^\{\\prime\}\)=1\],\\,\\frac\{1\}\{k\}\\sum\_\{x^\{\\prime\}\\in\\mathcal\{N\}\_\{k\}\(x\)\}\\mathbb\{I\}\[y\(x^\{\\prime\}\)=0\]\\right\\\},\(11\)
The third family captures perturbation stability\. A central innovation of the framework is to treat local instability as a first\-class component of dangerous overconfidence\. For each transformed samplex~\\tilde\{x\}, we generate a perturbation set𝒫​\(x\)\\mathcal\{P\}\(x\)by partially mixing it with nearby support neighbors:

x~′=\(1−λ\)​x~\+λ​x~nbr,λ∈\{0\.10,0\.20,0\.30\}\.\\tiny\\tilde\{x\}^\{\\,\\prime\}=\(1\-\\lambda\)\\tilde\{x\}\+\\lambda\\tilde\{x\}\_\{\\mathrm\{nbr\}\},\\qquad\\lambda\\in\\\{0\.10,0\.20,0\.30\\\}\.\(12\)These perturbations preserve local identity while probing prediction stability\. We then define

Δmean​\(x\)=1\|𝒫​\(x\)\|​∑x′∈𝒫​\(x\)\|p​\(x′\)−p​\(x\)\|,\\tiny\\Delta\_\{\\mathrm\{mean\}\}\(x\)=\\frac\{1\}\{\|\\mathcal\{P\}\(x\)\|\}\\sum\_\{x^\{\\prime\}\\in\\mathcal\{P\}\(x\)\}\|p\(x^\{\\prime\}\)\-p\(x\)\|,\(13\)Δmax​\(x\)=maxx′∈𝒫​\(x\)⁡\|p​\(x′\)−p​\(x\)\|,\\tiny\\Delta\_\{\\mathrm\{max\}\}\(x\)=\\max\_\{x^\{\\prime\}\\in\\mathcal\{P\}\(x\)\}\|p\(x^\{\\prime\}\)\-p\(x\)\|,\(14\)LC​\(x\)=1\|𝒫​\(x\)\|​∑x′∈𝒫​\(x\)𝕀​\[y^​\(x′\)=y^​\(x\)\],\\tiny\\mathrm\{LC\}\(x\)=\\frac\{1\}\{\|\\mathcal\{P\}\(x\)\|\}\\sum\_\{x^\{\\prime\}\\in\\mathcal\{P\}\(x\)\}\\mathbb\{I\}\[\\hat\{y\}\(x^\{\\prime\}\)=\\hat\{y\}\(x\)\],\(15\)Varlogit​\(x\)=Var​\(log⁡p​\(x′\)1−p​\(x′\)\)x′∈𝒫​\(x\)\.\\tiny\\mathrm\{Var\}\_\{\\mathrm\{logit\}\}\(x\)=\\mathrm\{Var\}\\\!\\left\(\\log\\frac\{p\(x^\{\\prime\}\)\}\{1\-p\(x^\{\\prime\}\)\}\\right\)\_\{x^\{\\prime\}\\in\\mathcal\{P\}\(x\)\}\.\(16\)The final structured signal family is

ψ​\(x\)=\\displaystyle\\psi\(x\)=\{\}\[conf​\(x\),margin​\(x\),ent​\(x\)⏟native certainty,\\displaystyle\\big\[\\underbrace\{\\mathrm\{conf\}\(x\),\\mathrm\{margin\}\(x\),\\mathrm\{ent\}\(x\)\}\_\{\\text\{native certainty\}\},\(17\)supp​\(x\),agrlabel​\(x\),agrpred​\(x\)⏟local support and agreement,\\displaystyle\\underbrace\{\\mathrm\{supp\}\(x\),\\mathrm\{agr\}\_\{\\mathrm\{label\}\}\(x\),\\mathrm\{agr\}\_\{\\mathrm\{pred\}\}\(x\)\}\_\{\\text\{local support and agreement\}\},Δmean​\(x\),Δmax​\(x\),LC​\(x\),Varlogit​\(x\)⏟structural stability\]\.\\displaystyle\\underbrace\{\\Delta\_\{\\mathrm\{mean\}\}\(x\),\\Delta\_\{\\mathrm\{max\}\}\(x\),\\mathrm\{LC\}\(x\),\\mathrm\{Var\}\_\{\\mathrm\{logit\}\}\(x\)\}\_\{\\text\{structural stability\}\}\\big\]\.
Equation \([17](https://arxiv.org/html/2607.18278#S3.E17)\) is the key organizational step of the method\. The paper does not treat reliability as a scalar attribute attached to a prediction, but as a*structured conflict state*in which certainty, support, agreement, and stability can align or disagree\. This is the paper’s learning paradigm:*prediction\-to\-structure learning*, where held\-out prediction behavior is first mapped into a discrepancy state and only then queried through ranking, localization, or calibration\-facing actions\. The representation is therefore the claim\-bearing object; the downstream detectors are operational views of the same state rather than independent engineered scores\. This framing also clarifies why concentration is regime\-dependent\. In certainty\-dominated regimes the upper tail is initiated by high confidence and then sharpened by structural evidence; in stability\-dominated regimes local perturbation sensitivity becomes the main separating cue; and in boundary regimes sparse false\-confidence events weaken all views simultaneously\. The scientific question is therefore not whether one detector is universally best, but whether concentrated failure structure is recoverable once prediction behavior is lifted into a discrepancy state\. We evaluate random ranking and confidence\-only ranking as minimal baselines\. The stability\-centered rule is intentionally simple and analytic: the larger coefficient is placed on average probability drift because sustained score movement is a stronger indicator of local unreliability than a single label flip alone, while label inconsistency remains a secondary corroborating signal\. We then define a stability\-centered score:

sstab​\(x\)=0\.7​Δmean​\(x\)⏟probability drift\+0\.3​\(1−LC​\(x\)\)⏟label inconsistency\.\\tiny s\_\{\\mathrm\{stab\}\}\(x\)=\\underbrace\{0\.7\\,\\Delta\_\{\\mathrm\{mean\}\}\(x\)\}\_\{\\text\{probability drift\}\}\+\\underbrace\{0\.3\\,\(1\-\\mathrm\{LC\}\(x\)\)\}\_\{\\text\{label inconsistency\}\}\.\(18\)and a fixed analytic discrepancy score built from a coarse priority ladder,

sdisc​\(x\)\\displaystyle s\_\{\\mathrm\{disc\}\}\(x\)=1\.30​conf​\(x\)⏟high native certainty\+1\.00​Δmean​\(x\)⏟average instability\+0\.80​Δmax​\(x\)⏟worst\-case instability\\displaystyle=\\underbrace\{1\.30\\,\\mathrm\{conf\}\(x\)\}\_\{\\text\{high native certainty\}\}\+\\underbrace\{1\.00\\,\\Delta\_\{\\mathrm\{mean\}\}\(x\)\}\_\{\\text\{average instability\}\}\+\\underbrace\{0\.80\\,\\Delta\_\{\\mathrm\{max\}\}\(x\)\}\_\{\\text\{worst\-case instability\}\}\+1\.00​\(1−LC​\(x\)\)⏟label inconsistency\+0\.80​\(1−supp​\(x\)\)⏟support collapse\+0\.70​\(1−agrlabel​\(x\)\)⏟weak local agreement\.\\displaystyle\\qquad\+\\underbrace\{1\.00\\,\(1\-\\mathrm\{LC\}\(x\)\)\}\_\{\\text\{label inconsistency\}\}\+\\underbrace\{0\.80\\,\(1\-\\mathrm\{supp\}\(x\)\)\}\_\{\\text\{support collapse\}\}\+\\underbrace\{0\.70\\,\(1\-\\mathrm\{agr\}\_\{\\mathrm\{label\}\}\(x\)\)\}\_\{\\text\{weak local agreement\}\}\.\(19\)
Equation \([19](https://arxiv.org/html/2607.18278#S3.E19)\) is not intended as the paper’s uniquely best detector\. Its role is methodological and falsification\-oriented\. If false\-confidence concentration were visible only under a trained detector, the claim would be weaker because it could be attributed to detector flexibility rather than to the structure of the discrepancy state itself\. By contrast, if the same phenomenon is already exposed by a coarse monotone aggregation of conflict signals, the claim is stronger: the state is informative before optimization\. For that reason the coefficients are ordinal rather than tuned\. They encode a transparent priority ladder among primary and corroborating conflict cues while deliberately avoiding the stronger claim that one fixed numeric weighting is universally optimal\. Positive affine rescalings are immaterial, and the paper does not interpret the absolute value ofsdisc​\(x\)s\_\{\\mathrm\{disc\}\}\(x\); only the induced ranking matters\. The empirical burden therefore lies on the state\-level phenomenon and the family\-level pattern, not on Equation \([19](https://arxiv.org/html/2607.18278#S3.E19)\) as a uniquely privileged formula\. The learned discrepancy ranker is likewise introduced as a*witness detector*over the standardized discrepancy state rather than as the paper’s primary source of capacity\. The methodological claim is that held\-out prediction behavior can be lifted into a structured conflict state whose separating information is already present before any powerful detector is applied\. For that reason we intentionally use a weighted linear logit as the smallest transparent learned rule: if concentration is real, a minimal detector should expose it; if it is absent, expressive nonlinear fitting should not be allowed to manufacture it\. The scientific burden is therefore placed onψ~​\(x\)\\tilde\{\\psi\}\(x\)rather than on detector complexity\. This clarifies the logic of the discrepancy family: the fixed rules act as analytic witness scores, while the learned detector acts as a data\-adaptive witness score over the same state\. If these views agree in the strongest regimes, the evidence favors the discrepancy state itself rather than any one detector implementation\. The paper’s methodological novelty therefore lies in the structured state and the prediction\-to\-structure learning paradigm, not in proposing a new high\-capacity classifier\.

qθ​\(x\)=σ​\(w⊤​ψ~​\(x\)\+b⏟learned combination of discrepancy signals\)\.\\scriptsize q\_\{\\theta\}\(x\)=\\sigma\\\!\\left\(\\underbrace\{w^\{\\top\}\\tilde\{\\psi\}\(x\)\+b\}\_\{\\text\{learned combination of discrepancy signals\}\}\\right\)\.\(20\)
with weighted objective

ℒ​\(θ\)\\displaystyle\\mathcal\{L\}\(\\theta\)=−∑i=1nωi⏟false\-confidence emphasis​\[zi​log⁡qθ​\(xi\)\+\(1−zi\)​log⁡\(1−qθ​\(xi\)\)\],\\displaystyle=\-\\sum\_\{i=1\}^\{n\}\\underbrace\{\\omega\_\{i\}\}\_\{\\text\{false\-confidence emphasis\}\}\\Big\[z\_\{i\}\\log q\_\{\\theta\}\(x\_\{i\}\)\+\(1\-z\_\{i\}\)\\log\(1\-q\_\{\\theta\}\(x\_\{i\}\)\)\\Big\],\(21\)
wherezi=FCτ​\(xi\)z\_\{i\}=\\mathrm\{FC\}\_\{\\tau\}\(x\_\{i\}\)and

ωi=1\+η​FCτ​\(xi\)\.\\omega\_\{i\}=1\+\\eta\\,\\mathrm\{FC\}\_\{\\tau\}\(x\_\{i\}\)\.\(22\)This objective is fitted only on validation/calibration folds with observed labels; test labels are used exclusively for final reporting, so neither the learned ranker norwcalw\_\{\\mathrm\{cal\}\}accesses test outcomes\. To move from ranking to structure, we cluster the held\-out discrepancy representation jointly with the learned discrepancy score, support, local agreement, and stability quantities; this localization step uses label\-free discrepancy features and is never used to select the reported ranking rule\. Finally, we derive a discrepancy\-aware weighting profile for future calibration training\. The coefficients are chosen to reflect an explicit priority ordering rather than numerical optimization: the profile should upweight samples most strongly when they are already false\-confidence events, then when they are highly confident or locally unstable, and then when they show weaker but still informative structural warning signs such as weak support, weak neighborhood agreement, or label inconsistency\. Because this profile is intended for later calibration refinement rather than for test\-time prediction, its purpose is to allocate more training emphasis to the most operationally dangerous and structurally suspicious samples\.

wcal​\(x\)=1\.2​conf​\(x\)⏟high certainty\+1\.0​\(1−supp​\(x\)\)⏟weak support\+0\.9​\(1−agrlabel​\(x\)\)⏟weak local agreement\+1\.2​Δmean​\(x\)⏟instability\+1\.0​\(1−LC​\(x\)\)⏟label inconsistency\+2\.0​FCτ​\(x\)⏟false\-confidence emphasis\\tiny w\_\{\\mathrm\{cal\}\}\(x\)=\\underbrace\{1\.2\\,\\mathrm\{conf\}\(x\)\}\_\{\\text\{high certainty\}\}\+\\underbrace\{1\.0\\,\(1\-\\mathrm\{supp\}\(x\)\)\}\_\{\\text\{weak support\}\}\+\\underbrace\{0\.9\\,\(1\-\\mathrm\{agr\}\_\{\\mathrm\{label\}\}\(x\)\)\}\_\{\\text\{weak local agreement\}\}\+\\underbrace\{1\.2\\,\\Delta\_\{\\mathrm\{mean\}\}\(x\)\}\_\{\\text\{instability\}\}\+\\underbrace\{1\.0\\,\(1\-\\mathrm\{LC\}\(x\)\)\}\_\{\\text\{label inconsistency\}\}\+\\underbrace\{2\.0\\,\\mathrm\{FC\}\_\{\\tau\}\(x\)\}\_\{\\text\{false\-confidence emphasis\}\}\(23\)The coefficient ordering in Equation \([23](https://arxiv.org/html/2607.18278#S3.E23)\) follows the operational logic of the method\. The largest coefficient is assigned toFCτ​\(x\)\\mathrm\{FC\}\_\{\\tau\}\(x\)because samples that are already high\-confidence errors are, by definition, the most urgent targets for later calibration refinement\. The next\-largest weights are assigned toconf​\(x\)\\mathrm\{conf\}\(x\)andΔmean​\(x\)\\Delta\_\{\\mathrm\{mean\}\}\(x\)because the target failure mode is specifically*confident*error under local instability: high certainty makes the prediction dangerous to trust, while instability indicates that this certainty is poorly supported\. The remaining terms receive slightly smaller but still positive weights because they act as corroborating structural cues\. Weak support indicates that the sample lies in a poorly grounded region of the training manifold, weak neighborhood agreement indicates that nearby evidence is inconsistent, and label inconsistency captures local decisional fragility\. The exact numeric values are therefore intended as transparent analytic priorities that encode this ordering, not as claims of universal optimality\. The stronger empirical claim of the paper lies in the discrepancy representation and the family\-level pattern it exposes, rather than in one uniquely best fixed weighting scheme\. The novelty is therefore representational rather than architectural: familiar statistical signals are organized into a single discrepancy space that supports three operations usually treated separately, failure discovery, region localization, and calibration guidance\.

## 4Experimental Design

The evaluation targets concentration rather than global accuracy\. We establish the phenomenon in seven public, heterogeneous binary tabular datasets \(Adult, Bank Marketing, Spambase, MiniBooNE, Magic Telescope, Nomao, Phoneme\), selected a priori because they provide a controlled but non\-uniform testbed for the paper’s structural claim: they vary in size, class balance, feature geometry, and false\-confidence prevalence while preserving a clean binary setting in which confidence, local support, neighborhood agreement, and support\-preserving perturbation stability can be defined transparently\. This is not a convenience choice but a deliberate first laboratory for testing whether compact false\-confidence slices exist before exporting the paradigm to multiclass, sequential, vision, language, or multimodal settings, where local evidence and perturbation semantics are harder to define cleanly\. Strong tree\-based models are included because recent tabular benchmarks identify them as highly competitive reference points\(Grinsztajnet al\.,[2022](https://arxiv.org/html/2607.18278#bib.bib36); Holzmülleret al\.,[2024](https://arxiv.org/html/2607.18278#bib.bib37); McElfreshet al\.,[2023](https://arxiv.org/html/2607.18278#bib.bib38)\)\. The final reported results aggregate across four seeds and use five\-fold cross\-fitting\. The main threshold is fixed atτ=0\.90\\tau=0\.90, and review budgets are fixed to\{5%,10%,15%,20%\}\\\{5\\%,10\\%,15\\%,20\\%\\\}for comparability\. The prior baseline family consists of confidence\-only ranking, temperature scaling, Platt scaling, isotonic calibration, beta calibration, and TrustScore, chosen a priori to span the canonical post\-hoc calibration and trust\-scoring families most directly relevant to the claim: score reshaping, non\-parametric calibration, distributional calibration, confidence\-only ranking, and local\-support trust estimation\. In addition, Appendix Table[8](https://arxiv.org/html/2607.18278#A18.T8)reports LightGBM, FT\-Transformer, RealMLP, and SAINT comparator runs so that the discovery claim is tested not only against standard calibration baselines but also against strong modern tabular predictors\. Experiments were run on a workstation with two NVIDIA RTX A6000 GPUs\. For each dataset, seed, and threshold, prior baselines are tuned on the validation split only; the strongest validation baseline is then frozen and evaluated once on the held\-out test split, preventing test\-set cherry\-picking and preserving the same information budget for all methods\. Appendix Sections[A](https://arxiv.org/html/2607.18278#A1)and[Z](https://arxiv.org/html/2607.18278#A26)summarize the reporting axes, metric families, success criterion, and anonymous review artifact used to evaluate and reproduce concentration beyond raw accuracy\. Two evaluation choices deserve emphasis\. First, the20%20\\%review slice reported in the main tables is an operationally interpretable reporting point rather than a cherry\-picked budget: all operating\-point summaries are computed over\{5%,10%,15%,20%\}\\\{5\\%,10\\%,15\\%,20\\%\\\}, and the appendix visualizes the full budget sweep\. Second, discrepancy regions are evaluated functionally rather than by raw cluster identity across seeds\. Because cluster labels are non\-identifiable up to permutation, stability is read through seed\-aggregated region statistics—top\-1 and top\-2 false\-confidence mass, mean support, mean instability, and correctness rate—rather than through literal cluster\-label matching\. For a discovery paper, the stable object is therefore the recurrence of the same*type*of discrepancy\-heavy slice, not an exact unsupervised label identity\. Appendix Section[M](https://arxiv.org/html/2607.18278#A13)makes this notion of region stability explicit\.

## 5Results

Table 1:Main multi\-dataset result atτ=0\.90\\tau=0\.90\.For each dataset we report the strongest*validation\-selected*prior baseline, the strongest discrepancy\-family rule, and the resulting gains on the test split\.Table 2:Operational impact and threshold robustness\.Counts are means across seeds atτ=0\.90\\tau=0\.90; last columns reportΔ\\DeltaCap@20\.
Table 3:Discrepancy\-family ablation atτ=0\.90\\tau=0\.90\.Values are meanCapture@20seeds\.

The central empirical question is not whether a new model improves global predictive accuracy, but whether false\-confidence concentration is recoverable and regime\-consistent with the structural characterization above\. We read the evidence along four axes: performance against standard baselines, agreement across concentration\-oriented measures, sensitivity to dataset characteristics, and robustness across strong tabular backbones\. Adult, Bank Marketing, and MiniBooNE are the strongest regimes: the strongest prior baselines recover only a modest share of false\-confidence mass, whereas the discrepancy family surfaces a large majority under the same review budget\. The identity of the strongest family member is informative rather than incidental: learned discrepancy is strongest when multiple conflict cues must be combined jointly, while stability\-centered ranking is strongest on Nomao and Spambase, where perturbation fragility carries more separating signal\. Magic Telescope is mixed because stronger modern tabular predictors absorb part of the relevant local structure into the base predictor, reducing residual headroom for post\-hoc discovery\. Phoneme is treated as a boundary regime because sparse false\-confidence events reduce the evidential strength of bothFalseConf\-AUROCandCapture@20\. Table[3](https://arxiv.org/html/2607.18278#S5.T3)should therefore be read mechanistically: raw confidence ranking is a weak baseline, whereas supervised certainty features reveal that confidence initiates the upper tail in the strongest regimes; support and stability provide the state variables needed for localization, regime transfer, and stability\-dominated cases\. Appendix Tables[5](https://arxiv.org/html/2607.18278#S5.T5)–[5](https://arxiv.org/html/2607.18278#S5.T5)report the corresponding signal\-family, fixed\-backbone, perturbation, uncertainty, downstream calibration, region\-quality, and stronger\-baseline checks\. Appendix Sections[N](https://arxiv.org/html/2607.18278#A14)–[P](https://arxiv.org/html/2607.18278#A16)further interpret the driver structure in the strongest regimes, the mixed\-regime behavior on Magic Telescope, and the falsification\-oriented role of the fixed analytic score\. Appendix Table[14](https://arxiv.org/html/2607.18278#A20.T14)reports paired seed\-level uncertainty for the same dataset–seed records used in Table[1](https://arxiv.org/html/2607.18278#S5.T1), and Appendix Table[15](https://arxiv.org/html/2607.18278#A21.T15)\(null concentration\) compares the observedFamily Cap@20values against random 20% review slices; the anonymous review artifact in Appendix Section[Z](https://arxiv.org/html/2607.18278#A26)maps these empirical objects to released scripts and stored outputs\. This null\-separation matters because it shows that the main effect is not a generic review\-budget artifact: in the strongest regimes, discrepancy\-aware ranking recovers far more false\-confidence mass than a prevalence\-matched random slice, so the reported gains reflect structured localization rather than merely inspecting more cases\.

Table 4:Extended validation\.Values atτ=0\.90\\tau=0\.90\. Panels A–B reportCap@20; Panel C reportsΔ\\DeltaCap@20, except the last row\.
Table 5:Consequences and baselines\.Calibration, region quality, and stronger comparator checks on the three strongest datasets\.

## 6Discussion and Limitations

The principal result is not that one new score dominates every dataset, but that calibration failure can be treated as a discoverable structural object\. FALCON\-Discover shows that high\-confidence errors can concentrate in compact regions of prediction space, that these regions can be recovered under fixed review budgets, and that the recovered regions can be described through a discrepancy state rather than only through a scalar confidence value\. This changes the target of calibration analysis: instead of fitting only a global probability map, one can first identify where confidence is misleading and then allocate monitoring, abstention, or calibration effort selectively\. The contribution is therefore methodological in a precise sense\. The paper does not propose a new end\-to\-end classifier, but a new post\-hoc learning paradigm—prediction\-to\-structure learning—in which held\-out prediction behavior is lifted into a structured conflict state and only then queried through ranking, localization, or calibration\-facing actions\. The witness scores, learned witness detector, and downstream weighting profile are all operational views of this same state, so the novelty lies not in architectural capacity but in the introduction of a new reliability object together with a reusable state\-based interface for acting on it\. Although a global calibrator optimized for average probability quality and a discrepancy\-aware weighting scheme emphasize different objectives, they are not in conflict in the present benchmark: the weighted stage improves both calibration quality and dangerous\-error recovery\. Discovery therefore remains the main contribution, while calibration\-facing weighting shows that the discovered discrepancy structure is operationally actionable\. The study deliberately starts from binary tabular benchmarks, where confidence, support, neighborhood agreement, and support\-preserving perturbations can be defined without modality\-specific assumptions\. This is a controlled identification setting rather than a claim of completed generality: the contribution is the reliability object and prediction\-to\-structure interface, not a universal perturbation recipe for every modality\. Because the strongest discrepancy\-family member varies across datasets, FALCON\-Discover should be read as a family\-level discovery framework rather than a single universally dominant score\. Extending the same principle to multiclass, sequential, vision, language, and multimodal systems requires modality\-specific definitions of support and stability, and is the natural next test of generality\.

## 7Conclusion

This paper introduced false\-confidence concentration as a structural object for calibration research and presented FALCON\-Discover as a post\-hoc prediction\-to\-structure framework for making that object measurable, localizable, and actionable\. The central finding is that dangerous overconfidence can be concentrated in compact, discoverable subsets of predictions rather than diffusely distributed across the test set\. This shifts calibration from global score alignment alone toward targeted reliability discovery: identifying where confidence becomes unsafe to trust\. Empirically, concentration is strong in several benchmark regimes, mixed when stronger predictors absorb local structure, and weakly interpretable when false\-confidence events become sparse\. Methodologically, the results separate raw confidence ranking from learned certainty structure and from the broader discrepancy state: confidence can initiate the upper tail, while support and stability make the failure region interpretable, transferable across regimes, and actionable for calibration\-facing weighting\. The strongest claim is therefore family\-level: reliability failure is structured, not merely scalar\. Calibration should not only align scores with frequencies, but also reveal where predictions should not be trusted\.

## References

- \[1\]\(2020\)Pitfalls of in\-domain uncertainty estimation and ensembling in deep learning\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[2\]C\. Corbière, N\. Thome, A\. Bar\-Hen, M\. Cord, and P\. Pérez\(2019\)Addressing failure prediction by learning model confidence\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[3\]F\. Denoodt and J\. Oramas\(2025\)Efficient post\-hoc uncertainty calibration via variance\-based smoothing\.arXiv preprint arXiv:2503\.15583\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2503.15583)Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[4\]A\. Durai\(2025\)Phases of uncertainty: confidence–calibration dynamics in language model training\.InProceedings of the 2nd Workshop on Uncertainty\-Aware NLP \(UncertaiNLP 2025\),Suzhou, China,pp\. 11–16\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.uncertainlp-main.2),[Link](https://aclanthology.org/2025.uncertainlp-main.2/)Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[5\]R\. El\-Yaniv and Y\. Wiener\(2010\)On the foundations of noise\-free selective classification\.Journal of Machine Learning Research11\(5\),pp\. 1605–1641\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1),[§3](https://arxiv.org/html/2607.18278#S3.p2.26)\.
- \[6\]Y\. Gal and Z\. Ghahramani\(2016\)Dropout as a bayesian approximation: representing model uncertainty in deep learning\.InProceedings of the 33rd International Conference on Machine Learning,pp\. 1050–1059\.Cited by:[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[7\]T\. Gebru, J\. Morgenstern, B\. Vecchione, J\. W\. Vaughan, H\. Wallach, H\. Daum’e III, and K\. Crawford\(2021\)Datasheets for datasets\.Communications of the ACM64\(12\),pp\. 86–92\.Cited by:[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[8\]Y\. Geifman and R\. El\-Yaniv\(2017\)Selective classification for deep neural networks\.arXiv preprint arXiv:1705\.08500\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[9\]Y\. Geifman and R\. El\-Yaniv\(2019\)SelectiveNet: a deep neural network with an integrated reject option\.InProceedings of the 36th International Conference on Machine Learning,pp\. 2151–2159\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[10\]H\. Gharoun, M\. S\. Khorshidi, K\. Ranjbarigderi, F\. Chen, and A\. H\. Gandomi\(2025\)Uncertainty\-aware post\-hoc calibration: mitigating confidently incorrect predictions beyond calibration metrics\.arXiv preprint arXiv:2510\.17915\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2510.17915)Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2)\.
- \[11\]L\. Grinsztajn, E\. Oyallon, and G\. Varoquaux\(2022\)Why do tree\-based models still outperform deep learning on typical tabular data?\.InAdvances in Neural Information Processing Systems 35 \(NeurIPS 2022\),External Links:[Link](https://arxiv.org/abs/2207.08815)Cited by:[§4](https://arxiv.org/html/2607.18278#S4.p1.4)\.
- \[12\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning,pp\. 1321–1330\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1),[§3](https://arxiv.org/html/2607.18278#S3.p2.26)\.
- \[13\]D\. Hendrycks and K\. Gimpel\(2017\)A baseline for detecting misclassified and out\-of\-distribution examples in neural networks\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[14\]D\. Holzmüller, L\. Grinsztajn, and I\. Steinwart\(2024\)Better by default: strong pre\-tuned MLPs and boosted trees on tabular data\.InAdvances in Neural Information Processing Systems 37 \(NeurIPS 2024\),External Links:[Link](https://arxiv.org/abs/2407.04491)Cited by:[§4](https://arxiv.org/html/2607.18278#S4.p1.4)\.
- \[15\]H\. Jiang, B\. Kim, M\. Y\. Guan, and M\. Gupta\(2018\)To trust or not to trust a classifier\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1),[§3](https://arxiv.org/html/2607.18278#S3.p2.26)\.
- \[16\]V\. Kuleshov, N\. Fenner, and S\. Ermon\(2018\)Accurate uncertainties for deep learning using calibrated regression\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 2796–2804\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[17\]M\. Kull, M\. Perello\-Nieto, M\. Kängsepp, T\. Silva Filho, H\. Song, and P\. Flach\(2019\)Beyond temperature scaling: obtaining well\-calibrated multiclass probabilities with dirichlet calibration\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[18\]M\. Kull, T\. Silva Filho, and P\. Flach\(2017\)Beta calibration: a well\-founded and easily implemented improvement on logistic calibration for binary classifiers\.InProceedings of the 20th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.54,pp\. 623–631\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[19\]B\. Lakshminarayanan, A\. Pritzel, and C\. Blundell\(2017\)Simple and scalable predictive uncertainty estimation using deep ensembles\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[20\]W\. Liu, X\. Wang, J\. D\. Owens, and Y\. Li\(2020\)Energy\-based out\-of\-distribution detection\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 21464–21475\.Cited by:[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[21\]D\. McElfresh, S\. Khandagale, J\. Valverde, V\. P\. C, B\. Feuer, C\. Hegde, G\. Ramakrishnan, M\. Goldblum, and C\. White\(2023\)When do neural nets outperform boosted trees on tabular data?\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\),External Links:[Link](https://arxiv.org/abs/2305.02997)Cited by:[§4](https://arxiv.org/html/2607.18278#S4.p1.4)\.
- \[22\]P\. Mendes, P\. Romano, and D\. Garlan\(2025\)CLUE: neural networks calibration via learning uncertainty\-error alignment\.arXiv preprint arXiv:2505\.22803\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2505.22803)Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[23\]M\. Minderer, J\. Djolonga, R\. Romijnders, F\. Hubis, X\. Zhai, N\. Houlsby, L\. Beyer, and M\. Lucic\(2021\)Revisiting the calibration of modern neural networks\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 15682–15694\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1),[§3](https://arxiv.org/html/2607.18278#S3.p2.26)\.
- \[24\]M\. Mitchell, S\. Wu, A\. Zaldivar, P\. Barnes, L\. Vasserman, B\. Hutchinson, E\. Spitzer, I\. D\. Raji, and T\. Gebru\(2019\)Model cards for model reporting\.InProceedings of the Conference on Fairness, Accountability, and Transparency,pp\. 220–229\.Cited by:[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[25\]M\. P\. Naeini, G\. F\. Cooper, and M\. Hauskrecht\(2015\)Obtaining well calibrated probabilities using bayesian binning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.29\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[26\]A\. Niculescu\-Mizil and R\. Caruana\(2005\)Predicting good probabilities with supervised learning\.InProceedings of the 22nd International Conference on Machine Learning,pp\. 625–632\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[27\]Y\. Ovadia, E\. Fertig, J\. Ren, Z\. Nado, D\. Sculley, S\. Nowozin, J\. Dillon, B\. Lakshminarayanan, and J\. Snoek\(2019\)Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[28\]J\. C\. Platt\(1999\)Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods\.Advances in Large Margin Classifiers10\(3\),pp\. 61–74\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[29\]Y\. Romano, M\. Sesia, and E\. J\. Candès\(2020\)Classification with valid and adaptive coverage\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 3581–3591\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1),[§3](https://arxiv.org/html/2607.18278#S3.p2.26)\.
- \[30\]J\. Vaicenavicius, D\. Widmann, C\. Andersson, F\. Lindsten, J\. Roll, and T\. B\. Schön\(2019\)Evaluating model calibration in classification\.InProceedings of the 22nd International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.89,pp\. 3459–3467\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§3](https://arxiv.org/html/2607.18278#S3.p2.26)\.
- \[31\]V\. Vovk, A\. Gammerman, and G\. Shafer\(2005\)Algorithmic learning in a random world\.Springer\.Cited by:[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[32\]B\. Zadrozny and C\. Elkan\(2001\)Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers\.InProceedings of the Eighteenth International Conference on Machine Learning,pp\. 609–616\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.
- \[33\]B\. Zadrozny and C\. Elkan\(2002\)Transforming classifier scores into accurate multiclass probability estimates\.InProceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 694–699\.Cited by:[§1](https://arxiv.org/html/2607.18278#S1.p1.2),[§2](https://arxiv.org/html/2607.18278#S2.p1.1)\.

## Appendix AEvaluation Reporting Details

The evidence is organized around four aspects: comparison with standard aggregate calibration and trust\-scoring baselines, comparison across complementary concentration\-oriented views, sensitivity to data characteristics, and sensitivity to classifier characteristics\. Concretely, the empirical evidence is reported through six quantities: FalseConf\-AUROC, Capture@20, dataset\-level operating\-point summaries, absolute event\-count summaries on the strongest datasets, threshold robustness underτ∈\{0\.85,0\.90,0\.95\}\\tau\\in\\\{0\.85,0\.90,0\.95\\\}, and benchmark\-level synthesis across datasets and detector variants, rather than raw accuracy alone\. Standard aggregate calibration measures such as ECE, Brier score, NLL, and reliability diagrams remain important reference quantities, but they do not directly answer the paper’s operational question of whether dangerous high\-confidence errors are concentrated and recoverable within a small review slice\. This combination is deliberate: it ensures that the claims do not rely on one metric family alone and that the core empirical story is supported simultaneously by ranking quality, operational impact, visual evidence, robustness analysis, sensitivity to event prevalence and class balance, and detector\-family ablation across strong tabular backbones\. Because the contribution is empirical discovery rather than universal score dominance, the success criterion is precise: the method succeeds if discrepancy\-family ranking captures substantially more false\-confidence mass than confidence\-only and the strongest validation\-selected prior baseline, ideally with large effect sizes, stable threshold behavior, and interpretable discrepancy regions\.

## Appendix BAdditional Visual Evidence

This section provides representative operating\-point summaries and discrepancy\-region visualizations for the strongest datasets, complementing the quantitative results in the main text\. These figures are included as diagnostic visual evidence rather than primary quantitative evidence; the main claims are supported by the tables, bootstrap intervals, threshold sweeps, and backbone checks\.

![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_adult/fig1_global_metrics.png)\(a\)Adult: operating\-point summary
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_bank_marketing/fig1_global_metrics.png)\(b\)Bank Marketing: operating\-point summary
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_miniboone/fig1_global_metrics.png)\(c\)MiniBooNE: operating\-point summary
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_nomao/fig2_discrepancy_clusters.png)\(d\)Nomao: discrepancy\-region structure

Figure 2:Additional visual evidence\.Discrepancy\-family ranking recovers substantially more high\-confidence errors than prior baselines on the strongest datasets, while discrepancy regions are structurally organized rather than diffuse\.
## Appendix CPer\-dataset operating\-point summaries

This appendix contains the full per\-dataset operating\-point summaries, the full per\-dataset discrepancy\-region visualizations, and the full per\-dataset discrepancy\-aware weighting profiles\.

![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_adult/fig1_global_metrics.png)\(a\)Adult
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_bank_marketing/fig1_global_metrics.png)\(b\)Bank Marketing
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_miniboone/fig1_global_metrics.png)\(c\)MiniBooNE
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_magic_telescope/fig1_global_metrics.png)\(d\)Magic Telescope

![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_nomao/fig1_global_metrics.png)\(a\)Nomao
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_spambase/fig1_global_metrics.png)\(b\)Spambase
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_phoneme/fig1_global_metrics.png)\(c\)Phoneme

Figure 4:Full per\-dataset operating\-point summaries available in the anonymous review artifact\.
## Appendix DPer\-dataset discrepancy\-region visualizations

![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_adult/fig2_discrepancy_clusters.png)\(a\)Adult
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_bank_marketing/fig2_discrepancy_clusters.png)\(b\)Bank Marketing
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_miniboone/fig2_discrepancy_clusters.png)\(c\)MiniBooNE
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_magic_telescope/fig2_discrepancy_clusters.png)\(d\)Magic Telescope

![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_nomao/fig2_discrepancy_clusters.png)\(a\)Nomao
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_spambase/fig2_discrepancy_clusters.png)\(b\)Spambase
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_phoneme/fig2_discrepancy_clusters.png)\(c\)Phoneme

Figure 6:Full per\-dataset discrepancy\-region visualizations available in the anonymous review artifact\.
## Appendix EInterpreting the Signal\-Family Ablation

Table[5](https://arxiv.org/html/2607.18278#S5.T5)is intended as a mechanistic analysis rather than a claim that every signal family is equally necessary on every dataset\. The ablation separates three distinct objects that are easy to conflate\. First,*raw confidence ranking*is the minimal deployment baseline: it sorts predictions directly by native confidence and asks whether the model’s own confidence is sufficient to expose dangerous errors\. This baseline is weak in the strongest regimes, showing that the model’s native confidence ordering does not by itself recover the false\-confidence mass efficiently\. Second,*certainty\-only learning*is not the same object as raw confidence ranking\. It is a supervised witness detector trained on calibration folds to recognize the false\-confidence event using certainty\-derived features\. Its strength in Adult, Bank Marketing, and MiniBooNE shows that false\-confidence concentration often begins in the high\-certainty tail\. This supports, rather than contradicts, the concentration hypothesis: the dangerous region is not uniformly spread across the prediction space\. Third, the*full discrepancy state*is the paper’s claim\-bearing object\. Its purpose is not only to maximize one Capture@20 number on the strongest datasets, but to provide a reusable state for ranking, localization, regime interpretation, and calibration\-facing weighting\. Support and stability are therefore evaluated by their role in explaining and transporting the discovered failure structure, not only by whether their removal lowers Capture@20 in every certainty\-dominated regime\. This distinction is important because the strongest signal family changes across regimes: certainty dominates the strongest datasets, whereas perturbation stability is decisive in Nomao and Spambase\. The ablation should therefore be read as follows\. Certainty identifies where the upper tail begins; support and agreement indicate whether that tail is locally grounded; stability indicates whether the decision is fragile under support\-preserving perturbations\. FALCON\-Discover combines these views because false\-confidence concentration is a family\-level reliability phenomenon, not a universal single\-score mechanism\.

## Appendix FSplit Discipline and Label Access

All learned and weighted components follow a strict split discipline\. The base predictor is selected using out\-of\-fold label AUROC on the training partition, then refit on the full training partition and evaluated on validation and test partitions\. Discrepancy features for held\-out examples are computed from held\-out predictions and training\-derived support statistics\. The learned discrepancy ranker is fitted only on validation/calibration folds where labels are available\. Its target iszi=FCτ​\(xi\)z\_\{i\}=\\mathrm\{FC\}\_\{\\tau\}\(x\_\{i\}\)on those folds\. The held\-out test labels are never used to fit the ranker, choose the ranker, tune the analytic score, set the perturbation parameters, select the confidence threshold, or constructwcalw\_\{\\mathrm\{cal\}\}\. Test labels are used only once: to compute final reporting quantities such asFalseConf\-AUROC,Capture@20, and recovered false\-confidence counts\. The weighting profile in Equation \([23](https://arxiv.org/html/2607.18278#S3.E23)\) is therefore a calibration\-training object, not a test\-time scoring rule requiring unknown labels\. In a deployment pipeline,FCτ​\(x\)\\mathrm\{FC\}\_\{\\tau\}\(x\)is available only for calibration data with observed outcomes; for unlabeled future examples, the deployable quantities are the label\-free components ofψ​\(x\)\\psi\(x\)and the fitted ranker learned from calibration data\.

## Appendix GDistinction from Failure Prediction, Selective Classification, and Trust Scoring

FALCON\-Discover differs from standard failure prediction, selective classification, and trust scoring in its target object and output interface\. Failure prediction usually asks whether an auxiliary model can predict correctness\. Selective classification asks whether a model should accept, reject, or defer a prediction under a desired coverage\-risk tradeoff\. Trust scoring asks whether an alternative scalar score better reflects local support than native confidence\. These are important but score\-centric formulations\. FALCON\-Discover instead asks whether high\-confidence errors occupy a compact, recoverable region of prediction space\. The output is therefore not only a scalar trust score or a reject decision\. The method constructs a discrepancy state that can be queried in multiple ways: ranking false\-confidence events under a review budget, localizing discrepancy\-heavy regions, interpreting which signal family drives the failure mode, and deriving calibration\-facing weights\. This makes false\-confidence concentration a structural discovery problem rather than only a correctness\-prediction problem\. This distinction is also reflected in the evaluation\. The main outcome is not global accuracy, ECE alone, or selective risk alone, but the recoverability of dangerous high\-confidence error mass under fixed review budgets\.Capture@20andFalseConf\-AUROCtherefore measure whether the false\-confidence event is concentrated and discoverable, not whether a model is globally better calibrated on average\.

## Appendix HWhy Binary Tabular Benchmarks Are the First Testbed

The binary tabular setting is used as a controlled first testbed because all components of the discrepancy state can be defined without modality\-specific assumptions\. Confidence is unambiguous, local support can be measured in transformed feature space, neighborhood agreement can be computed directly, and support\-preserving perturbations can be implemented as local neighbor mixing\. This makes the setting appropriate for testing the core structural claim: whether dangerous high\-confidence errors are concentrated and recoverable\. The paper does not claim that the same perturbation operator should transfer unchanged to vision, language, sequential, or multimodal settings\. In those settings, support and stability must be redefined using modality\-appropriate neighborhoods and perturbations\. For example, vision systems may require augmentation\-preserving neighborhoods, language systems may require semantic\-preserving paraphrase neighborhoods, and sequential systems may require temporally constrained perturbations\. The general claim is therefore not tied to tabular interpolation; it is that confidence should be evaluated together with local evidence and stability to discover concentrated false\-confidence regions\.

## Appendix ISparse False\-Confidence Events and Boundary Regimes

False\-confidence concentration metrics become less stable when the number of false\-confidence events is very small\. In such cases, bothFalseConf\-AUROCandCapture@20have reduced evidential strength: a small change in the identity of a few high\-confidence errors can noticeably alter the measured recovery fraction\. For this reason, Phoneme is treated as a boundary regime rather than as decisive counterevidence\. This interpretation is stricter than simply excluding the dataset\. The dataset remains reported in the main table, but its role is diagnostic: it shows where the proposed concentration view becomes weakly identifiable because the event space itself is sparse\. The paper’s main empirical claim is therefore not universal dominance across every dataset, but a regime\-resolved finding: false\-confidence concentration is strong and actionable when the event space is sufficiently populated, mixed when the base predictor absorbs part of the local structure, and boundary\-limited when too few false\-confidence events remain to support stable discovery\.

## Appendix JPerturbation and Neighborhood Sensitivity

The perturbation and neighborhood parameters are fixed before test evaluation and are not selected on the test split\. Their role is to define a local stability probe rather than to optimize final performance\. The sensitivity results show that the method is not dependent on a single pathological perturbation setting: performance changes smoothly asλ\\lambdaandkkvary, with stronger recovery when perturbations are large enough to expose local instability but still small enough to remain support\-preserving\. The important object is therefore not a uniquely optimal value ofλ\\lambdaorkk, but the recurrence of the same qualitative pattern: local stability information provides an independent view of false\-confidence structure, and this view becomes decisive in stability\-dominated regimes\. The fixed settings used in the main experiments are chosen to balance locality and sensitivity, while the appendix reports the broader sweep to make this design choice auditable\.

## Appendix KBenchmark\-level empirical synthesis

Table[6](https://arxiv.org/html/2607.18278#A11.T6)aggregates the strongest reviewer\-oriented empirical checks derived from Tables[1](https://arxiv.org/html/2607.18278#S5.T1)–[3](https://arxiv.org/html/2607.18278#S5.T3)\. It answers three questions directly\. First, how often does the discrepancy family beat the strongest prior baseline? Second, how stable are the gains across thresholds on the strongest datasets? Third, does the overall story depend on a single detector, or does it hold at the family level? The table shows that the paper’s empirical claim is supported simultaneously at the dataset, threshold, and detector\-selection levels\.

Table 6:Benchmark\-level empirical synthesis derived from the main result tables\.Two summary points are especially important\. First, the discrepancy family does not rely on a single unusually favorable dataset: it achieves positive AUROC gains on six of seven datasets and positiveCapture@20gains on five of seven datasets\. Second, the strongest regimes are stable under threshold variation: all nine threshold settings for Adult, Bank Marketing, and MiniBooNE remain positive\. This makes the empirical case materially stronger than a one\-threshold, one\-detector, or one\-dataset story\.

## Appendix LDataset\-Regime Interpretation

This section provides a concise summary of how the main evaluation metrics should be interpreted across different empirical regimes in the benchmark\. The regimes reflect differences in false\-confidence event prevalence, class balance, and structural complexity\. This table is referenced in the main paper to guide the interpretation ofFalseConf\-AUROCandCapture@20under varying data conditions\.

Table 7:Dataset\-regime interpretation\.Summary of how the main evaluation quantities should be read across event prevalence, class balance, and sparse\-regime boundary cases\.
## Appendix MFunctional Stability of Discrepancy Regions

A natural question is whether the discovered discrepancy regions are stable across seeds\. For this paper, the relevant notion of stability is*functional*rather than label\-identitarian\. Because unsupervised cluster labels are permutation\-invariant, raw cluster identities are not meaningful objects to compare across random seeds\. The stable scientific object is instead the recurrence of the same type of discrepancy\-heavy slice: regions that repeatedly capture large false\-confidence mass while exhibiting the expected structural profile\. For that reason the appendix evaluates region stability through seed\-aggregated top\-1 and top\-2 false\-confidence mass, mean support, mean instability, and correctness rate rather than through literal cluster\-label matching\. Under this reading, the region summaries in Table[11](https://arxiv.org/html/2607.18278#A18.T11)are not an auxiliary visualization artifact; they are the functional stability report for the localization component of the method\.

## Appendix NWhich Signals Drive Concentration?

The paper should not be read as claiming one universal driver of false\-confidence concentration\. In the strongest datasets, certainty initiates the upper tail, as reflected by the strong certainty\-only rows in Table[5](https://arxiv.org/html/2607.18278#S5.T5); support\-only and stability\-only are much weaker on their own\. The scientific role of structural signals is therefore not to replace confidence, but to explain, localize, and transfer the phenomenon across detectors, thresholds, and backbone families\. When stability\-centered ranking becomes strongest, as on Nomao and Spambase, the relevant conflict is less about raw confidence alone and more about decisional fragility under local perturbation\. This regime\-dependence is exactly why the paper advances a discrepancy\-family claim rather than a single\-score claim\.

## Appendix OInterpretation of the Mixed Regime on Magic Telescope

Magic Telescope should not be read as evidence against false\-confidence concentration\. The paper’s claim is two\-stage\. First, the discrepancy family is compared against the original calibration and trust\-scoring baselines; under that comparison Magic remains positive\. Second, stronger modern tabular predictors are added to test how much residual headroom remains once a more expressive backbone is allowed to internalize local structure\. Magic is mixed because the second test is stringent: SAINT exceeds both the strongest original prior baseline and the discrepancy family there\. This is consistent with a regime in which part of the relevant confidence–support interaction is already absorbed into the predictor itself, reducing the remaining benefit available to post\-hoc discrepancy discovery without invalidating the concentration phenomenon against standard baselines\.

## Appendix PWhy the Analytic Score is Included

Equation \([19](https://arxiv.org/html/2607.18278#S3.E19)\) should be read as a transparent reference map, not as the paper’s uniquely best detector\. Its role is to strengthen the scientific claim by showing that concentration is not visible only under a learned detector\. If a coarse monotone aggregation of conflict signals already surfaces the same phenomenon, then the claim is about the discrepancy state itself rather than about one optimized predictor\. The exact coefficient values are therefore ordinal rather than tuned: they express a priority ladder among primary conflict cues and corroborating cues, while leaving the paper’s empirical burden on the family\-level pattern rather than on one particular fixed formula\.

## Appendix QWhy Binary Tabular is the First Testbed

The present paper should not be read as claiming that false\-confidence concentration is unique to binary tabular prediction\. Rather, binary tabular data is the cleanest first environment in which the discrepancy state can be defined transparently and audited component\-by\-component\. Confidence is available directly from the classifier, local evidence can be measured through transformed\-space support and neighborhood agreement, and support\-preserving perturbations can be specified by controlled interpolation with nearby neighbors\. These design choices make it possible to study the structural object itself before the same paradigm is transported to richer modalities where the semantics of local evidence and perturbation are much less settled\. In that sense, the binary\-tabular setting is the paper’s controlled laboratory, not its intended ceiling\.

## Appendix RFull baseline comparison

Table[8](https://arxiv.org/html/2607.18278#A18.T8)reports the full comparator landscape atτ=0\.90\\tau=0\.90\. The main paper reports only the strongest validation\-selected prior baseline from the original calibration and trust\-scoring family for compactness, but the appendix expands the comparison to strong recent tabular backbones including LightGBM, FT\-Transformer, RealMLP, and SAINT\. This makes it possible to separate two questions cleanly: whether discrepancy\-family ranking beats the original post\-hoc prior family, and how it compares to stronger modern tabular predictors\.

Table 8:Expanded comparator summary atτ=0\.90\\tau=0\.90\.Values are mean test\-setCapture@20across seeds\. The table reports the original comparator family together with the added strong tabular backbones\. Red marks the best value in each dataset column and blue marks the second\-best\.RealMLP is one of the strongest comparator on six of the seven datasets, while SAINT is strongest on Magic Telescope\. In the presented run’s results TabPFN is not reported because all seven train splits lie outside its intended small\-data regime\.

Table 9:Backbone robustness across all seven datasets\.Entries report the mean gain of the best discrepancy\-family rule over the strongest prior baseline under the base\-model backbones\. Higher is better for bothΔ\\DeltaCap@20andΔ\\DeltaFalseConf\-AUROC\.Table 10:Bootstrap interval summary against the strongest prior baseline\.Δ\\Deltadenotes the mean improvement of the best discrepancy\-family rule over the strongest prior comparator available for that dataset\. Lower empiricalP​\(Δ≤0\)P\(\\Delta\\leq 0\)is stronger evidence of a stable positive gain\.Table 11:Region\-quality summary across all seven datasets\.Higher top\-1 and top\-2 false\-confidence mass indicate stronger concentration inside the discovered discrepancy regions\. Lower support and higher instability are more discrepancy\-consistent\.Table 12:Weighted\-vs\-unweighted Platt calibration across all seven datasets\.Lower ECE is better\. NegativeΔ\\DeltaECEindicates improved calibration under discrepancy\-aware weighting\. PositiveΔ\\DeltaCap@20means discrepancy\-weighted calibration recovers more dangerous confident errors at the same review budget\.Table 13:Perturbation\-sensitivity summary across all seven datasets\.For each dataset, the table reports the strongestλ\\lambdasetting and strongest neighborhood sizekkin terms ofΔ\\DeltaCap@20over the strongest prior baseline\.
## Appendix SEvaluation Contract: What Would Confirm or Falsify the Claim

The empirical claim of FALCON\-Discover is intentionally narrower than universal calibration dominance\. The paper tests the following falsifiable statement: for a trained classifier and a fixed confidence thresholdτ\\tau, high\-confidence errors may form a compact, recoverable region of prediction space, and a discrepancy state combining certainty, local evidence, and stability should recover more of that dangerous\-error mass under a fixed review budget than standard confidence, calibration, or trust\-scoring baselines\. The claim is supported when four conditions hold jointly\. First,FalseConf\-AUROCimproves over the strongest validation\-selected prior baseline, showing that the ranking is globally better at separating false\-confidence events\. Second,Capture@20improves, showing that the gain is operational under a small review budget\. Third, the gain is stable across nearby thresholds, showing that the result is not an artifact of one arbitraryτ\\tau\. Fourth, the discovered region has an interpretable discrepancy profile, showing that the method has localized a structural failure region rather than merely fitted another opaque score\. The claim would be weakened or falsified if false\-confidence events were diffuse, if random or raw\-confidence rankings recovered comparable mass, if gains disappeared under threshold changes, or if discovered regions had no stable discrepancy profile\. This evaluation contract is why the paper reports ranking metrics, absolute recovery counts, threshold sweeps, signal\-family ablations, region summaries, and stronger comparator checks rather than relying on a single aggregate calibration score\.

## Appendix TAll\-Dataset Uncertainty for the Headline Result

Table[14](https://arxiv.org/html/2607.18278#A20.T14)reports uncertainty for the main result at the same granularity as Table[1](https://arxiv.org/html/2607.18278#S5.T1)\. Intervals are computed across seeds using the same held\-out splits as the main evaluation\. This table is included to ensure that the main claim is not read from means alone\.

Table 14:All\-dataset uncertainty for the main result atτ=0\.90\\tau=0\.90\.Values are mean and 95% bootstrap interval across seeds for the strongest validation\-selected prior baseline and the strongest discrepancy\-family rule\. Means match Table[1](https://arxiv.org/html/2607.18278#S5.T1)\.
## Appendix UNull Concentration Test

To test whether the observed recovery is larger than expected from a generic ranked subset, we construct a null distribution forCapture@20\. For each dataset and seed, we preserve the number of false\-confidence events and the review budget, but sample random top\-20% subsets without using the discrepancy score\. We repeat this procedureB=10,000B=10\{,\}000times and compute the empirical null distribution of recovered false\-confidence mass\. The null mean equals the review budget in expectation, but finite\-sample variation can be substantial when false\-confidence events are sparse\. We therefore report both the observedCapture@20and the empirical percentile of the observed score under the null\.

Table 15:Null concentration test atτ=0\.90\\tau=0\.90\.The observed familyCapture@20is compared against random 20% review subsets that preserve dataset size and false\-confidence prevalence\. Observed values match Table[1](https://arxiv.org/html/2607.18278#S5.T1)\.This null test separates false\-confidence concentration from a trivial review\-budget effect\. A method that merely samples a 20% slice should recover approximately 20% of false\-confidence events in expectation\. Adult, Bank Marketing, MiniBooNE, Nomao, and Spambase recover far more false\-confidence mass than this random\-review null\. Magic Telescope is close to the null and should be interpreted as a mixed regime, while Phoneme remains a boundary regime where sparse false\-confidence events make the concentration estimate weakly identifiable\.

## Appendix VArtifact Reproduction Map

Table[16](https://arxiv.org/html/2607.18278#A22.T16)maps every empirical object in the paper to the script, input files, and stored outputs needed to reproduce it from the anonymous review artifact introduced in Appendix Section[Z](https://arxiv.org/html/2607.18278#A26)\. The artifact is organized around table\-level reproduction: each table can be regenerated independently from stored split IDs, model predictions, false\-confidence labels, discrepancy features, and ranking outputs\.

Table 16:Artifact reproduction map\.Each empirical claim is mapped to the script and stored output required for reproduction\.
## Appendix WAlgorithmic Summary

Algorithm 1FALCON\-Discover1:Training partition

𝒟train\\mathcal\{D\}\_\{\\mathrm\{train\}\}, calibration partition

𝒟val\\mathcal\{D\}\_\{\\mathrm\{val\}\}, test partition

𝒟test\\mathcal\{D\}\_\{\\mathrm\{test\}\}, confidence threshold

τ\\tau, review budget

α\\alpha
2:Select base classifier by out\-of\-fold AUROC on

𝒟train\\mathcal\{D\}\_\{\\mathrm\{train\}\}
3:Refit selected classifier on

𝒟train\\mathcal\{D\}\_\{\\mathrm\{train\}\}
4:Compute held\-out scores

p​\(x\)p\(x\)and predictions

y^​\(x\)\\hat\{y\}\(x\)on

𝒟val\\mathcal\{D\}\_\{\\mathrm\{val\}\}and

𝒟test\\mathcal\{D\}\_\{\\mathrm\{test\}\}
5:Construct certainty features from confidence, margin, and entropy

6:Construct local\-evidence features from support and neighborhood agreement

7:Construct stability features from support\-preserving perturbations

8:Form discrepancy state

ψ​\(x\)\\psi\(x\)
9:Fit learned witness ranker

qθ​\(x\)q\_\{\\theta\}\(x\)on calibration data using

zi=FCτ​\(xi\)z\_\{i\}=\\mathrm\{FC\}\_\{\\tau\}\(x\_\{i\}\)
10:Rank test examples using the validation\-selected discrepancy\-family rule

11:ReportFalseConf\-AUROC,Capture@α\\alpha, recovered false\-confidence counts, and region summaries

12:Use calibration\-only labels to derive

wcalw\_\{\\mathrm\{cal\}\}for future calibration training

## Appendix XExact Evaluation Formulas for Tables[1](https://arxiv.org/html/2607.18278#S5.T1)and[3](https://arxiv.org/html/2607.18278#S5.T3)

This section gives the exact finite\-sample definitions used for every numeric column reported in Tables[1](https://arxiv.org/html/2607.18278#S5.T1)and[3](https://arxiv.org/html/2607.18278#S5.T3)\. Because false\-confidence concentration is the paper’s central empirical object, we make these reporting formulas explicit\.

Let𝒯d,r=\{\(xi,yi\)\}i=1nd,r\\mathcal\{T\}\_\{d,r\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\_\{d,r\}\}denote the held\-out test split for datasetddand seedrr, with sizend,r=\|𝒯d,r\|n\_\{d,r\}=\|\\mathcal\{T\}\_\{d,r\}\|\. For a fixed confidence thresholdτ\\tau, define

zi\(τ\)=FCτ​\(xi\)∈\{0,1\},z\_\{i\}^\{\(\\tau\)\}=\\mathrm\{FC\}\_\{\\tau\}\(x\_\{i\}\)\\in\\\{0,1\\\},\(24\)whereFCτ​\(xi\)\\mathrm\{FC\}\_\{\\tau\}\(x\_\{i\}\)is given by Equation \([3](https://arxiv.org/html/2607.18278#S3.E3)\)\. Lets​\(xi\)s\(x\_\{i\}\)be any ranking score, and let

kα\(d,r\)=⌈α​nd,r⌉k\_\{\\alpha\}^\{\(d,r\)\}=\\left\\lceil\\alpha\\,n\_\{d,r\}\\right\\rceil\(25\)denote the review budget in number of test samples\. We writeTopα​\(s;𝒯d,r\)\\mathrm\{Top\}\_\{\\alpha\}\(s;\\mathcal\{T\}\_\{d,r\}\)for the index set of the topkα\(d,r\)k\_\{\\alpha\}^\{\(d,r\)\}test samples ranked by descendings​\(x\)s\(x\)\.

### False\-confidence AUROC\.

For any ranking ruless, the*FalseConf\-AUROC*used in Table[1](https://arxiv.org/html/2607.18278#S5.T1)is the AUROC obtained when the binary target is the false\-confidence eventzi\(τ\)z\_\{i\}^\{\(\\tau\)\}and the scoring variable iss​\(xi\)s\(x\_\{i\}\):

FC​\-​AUROCτ​\(s;𝒯d,r\)=1N1\(d,r,τ\)​N0\(d,r,τ\)​∑i:zi\(τ\)=1∑j:zj\(τ\)=0\[𝕀​\(s​\(xi\)\>s​\(xj\)\)\+12​𝕀​\(s​\(xi\)=s​\(xj\)\)\],\\mathrm\{FC\\mbox\{\-\}AUROC\}\_\{\\tau\}\(s;\\mathcal\{T\}\_\{d,r\}\)=\\frac\{1\}\{N\_\{1\}^\{\(d,r,\\tau\)\}N\_\{0\}^\{\(d,r,\\tau\)\}\}\\sum\_\{i:z\_\{i\}^\{\(\\tau\)\}=1\}\\sum\_\{j:z\_\{j\}^\{\(\\tau\)\}=0\}\\left\[\\mathbb\{I\}\\\!\\big\(s\(x\_\{i\}\)\>s\(x\_\{j\}\)\\big\)\+\\frac\{1\}\{2\}\\mathbb\{I\}\\\!\\big\(s\(x\_\{i\}\)=s\(x\_\{j\}\)\\big\)\\right\],\(26\)where

N1\(d,r,τ\)=∑i=1nd,rzi\(τ\),N0\(d,r,τ\)=nd,r−N1\(d,r,τ\)\.N\_\{1\}^\{\(d,r,\\tau\)\}=\\sum\_\{i=1\}^\{n\_\{d,r\}\}z\_\{i\}^\{\(\\tau\)\},\\qquad N\_\{0\}^\{\(d,r,\\tau\)\}=n\_\{d,r\}\-N\_\{1\}^\{\(d,r,\\tau\)\}\.\(27\)This is a standard AUROC computation, but with the positive class redefined as the set of dangerous high\-confidence errors\.

### Absolute recovered false\-confidence count\.

For any scoress, define the number of false\-confidence events recovered inside the top\-α\\alphareview slice as

FCCount​@​α​\(s;𝒯d,r,τ\)=∑i∈Topα​\(s;𝒯d,r\)zi\(τ\)\.\\mathrm\{FCCount\}@\\alpha\(s;\\mathcal\{T\}\_\{d,r\},\\tau\)=\\sum\_\{i\\in\\mathrm\{Top\}\_\{\\alpha\}\(s;\\mathcal\{T\}\_\{d,r\}\)\}z\_\{i\}^\{\(\\tau\)\}\.\(28\)

### Capture@α\\alpha\.

The normalized recovery fraction is

Capture​@​α​\(s;𝒯d,r,τ\)=FCCount​@​α​\(s;𝒯d,r,τ\)N1\(d,r,τ\)\.\\mathrm\{Capture\}@\\alpha\(s;\\mathcal\{T\}\_\{d,r\},\\tau\)=\\frac\{\\mathrm\{FCCount\}@\\alpha\(s;\\mathcal\{T\}\_\{d,r\},\\tau\)\}\{N\_\{1\}^\{\(d,r,\\tau\)\}\}\.\(29\)Equation \([29](https://arxiv.org/html/2607.18278#A24.E29)\) is the finite\-sample version of Equation \([4](https://arxiv.org/html/2607.18278#S3.E4)\)\.

### Reported score identities\.

For each datasetddand seedrr, letsprior\(d,r\)s\_\{\\mathrm\{prior\}\}^\{\(d,r\)\}denote the validation\-selected prior baseline score reported in the*Best prior*column of Table[1](https://arxiv.org/html/2607.18278#S5.T1), and letsfam\(d,r\)s\_\{\\mathrm\{fam\}\}^\{\(d,r\)\}denote the discrepancy\-family score reported in the*Best family*column\. The method\-name columns themselves are identifiers rather than numeric metrics; all numeric columns below are computed from these two scores\.

### Per\-seed quantities for Table[1](https://arxiv.org/html/2607.18278#S5.T1)\.

At the main thresholdτ=0\.90\\tau=0\.90, the per\-seed quantities underlying the numeric columns are

Aprior\(d,r\)\\displaystyle A\_\{\\mathrm\{prior\}\}^\{\(d,r\)\}=FC​\-​AUROC0\.90​\(sprior\(d,r\);𝒯d,r\),\\displaystyle=\\mathrm\{FC\\mbox\{\-\}AUROC\}\_\{0\.90\}\\\!\\left\(s\_\{\\mathrm\{prior\}\}^\{\(d,r\)\};\\mathcal\{T\}\_\{d,r\}\\right\),\(30\)Afam\(d,r\)\\displaystyle A\_\{\\mathrm\{fam\}\}^\{\(d,r\)\}=FC​\-​AUROC0\.90​\(sfam\(d,r\);𝒯d,r\),\\displaystyle=\\mathrm\{FC\\mbox\{\-\}AUROC\}\_\{0\.90\}\\\!\\left\(s\_\{\\mathrm\{fam\}\}^\{\(d,r\)\};\\mathcal\{T\}\_\{d,r\}\\right\),\(31\)Cprior\(d,r\)\\displaystyle C\_\{\\mathrm\{prior\}\}^\{\(d,r\)\}=Capture​@​0\.20​\(sprior\(d,r\);𝒯d,r,0\.90\),\\displaystyle=\\mathrm\{Capture\}@0\.20\\\!\\left\(s\_\{\\mathrm\{prior\}\}^\{\(d,r\)\};\\mathcal\{T\}\_\{d,r\},0\.90\\right\),\(32\)Cfam\(d,r\)\\displaystyle C\_\{\\mathrm\{fam\}\}^\{\(d,r\)\}=Capture​@​0\.20​\(sfam\(d,r\);𝒯d,r,0\.90\),\\displaystyle=\\mathrm\{Capture\}@0\.20\\\!\\left\(s\_\{\\mathrm\{fam\}\}^\{\(d,r\)\};\\mathcal\{T\}\_\{d,r\},0\.90\\right\),\(33\)Δ​A\(d,r\)\\displaystyle\\Delta A^\{\(d,r\)\}=Afam\(d,r\)−Aprior\(d,r\),\\displaystyle=A\_\{\\mathrm\{fam\}\}^\{\(d,r\)\}\-A\_\{\\mathrm\{prior\}\}^\{\(d,r\)\},\(34\)Δ​C\(d,r\)\\displaystyle\\Delta C^\{\(d,r\)\}=Cfam\(d,r\)−Cprior\(d,r\)\.\\displaystyle=C\_\{\\mathrm\{fam\}\}^\{\(d,r\)\}\-C\_\{\\mathrm\{prior\}\}^\{\(d,r\)\}\.\(35\)

### Seed aggregation for Table[1](https://arxiv.org/html/2607.18278#S5.T1)\.

All reported numeric entries are means across the four seeds:

Q¯\(d\)=1R​∑r=1RQ\(d,r\),R=4\.\\overline\{Q\}^\{\(d\)\}=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}Q^\{\(d,r\)\},\\qquad R=4\.\(36\)Thus the Table[1](https://arxiv.org/html/2607.18278#S5.T1)columns are

Prior AUROC=Aprior¯\(d\),\\displaystyle=\\overline\{A\_\{\\mathrm\{prior\}\}\}^\{\(d\)\},\(37\)Family AUROC=Afam¯\(d\),\\displaystyle=\\overline\{A\_\{\\mathrm\{fam\}\}\}^\{\(d\)\},\(38\)Prior Cap@20=Cprior¯\(d\),\\displaystyle=\\overline\{C\_\{\\mathrm\{prior\}\}\}^\{\(d\)\},\(39\)Family Cap@20=Cfam¯\(d\),\\displaystyle=\\overline\{C\_\{\\mathrm\{fam\}\}\}^\{\(d\)\},\(40\)Δ​AUROC\\displaystyle\\Delta\\texttt\{AUROC\}=Δ​A¯\(d\),\\displaystyle=\\overline\{\\Delta A\}^\{\(d\)\},\(41\)Δ​Cap@20\\displaystyle\\Delta\\texttt\{Cap@20\}=Δ​C¯\(d\)\.\\displaystyle=\\overline\{\\Delta C\}^\{\(d\)\}\.\(42\)

### Per\-seed quantities for Table[3](https://arxiv.org/html/2607.18278#S5.T3)\.

For the operational table we report*absolute*false\-confidence recovery counts atτ=0\.90\\tau=0\.90, together with threshold\-specific gains in normalized recovery\. The per\-seed total number of false\-confidence events is

NFC\(d,r\)=N1\(d,r,0\.90\)=∑i=1nd,rzi\(0\.90\)\.N\_\{\\mathrm\{FC\}\}^\{\(d,r\)\}=N\_\{1\}^\{\(d,r,0\.90\)\}=\\sum\_\{i=1\}^\{n\_\{d,r\}\}z\_\{i\}^\{\(0\.90\)\}\.\(43\)The per\-seed absolute recovered counts at a20%20\\%review budget are

Nprior​@​20\(d,r\)\\displaystyle N\_\{\\mathrm\{prior\}@20\}^\{\(d,r\)\}=FCCount​@​0\.20​\(sprior\(d,r\);𝒯d,r,0\.90\),\\displaystyle=\\mathrm\{FCCount\}@0\.20\\\!\\left\(s\_\{\\mathrm\{prior\}\}^\{\(d,r\)\};\\mathcal\{T\}\_\{d,r\},0\.90\\right\),\(44\)Nfam​@​20\(d,r\)\\displaystyle N\_\{\\mathrm\{fam\}@20\}^\{\(d,r\)\}=FCCount​@​0\.20​\(sfam\(d,r\);𝒯d,r,0\.90\)\.\\displaystyle=\\mathrm\{FCCount\}@0\.20\\\!\\left\(s\_\{\\mathrm\{fam\}\}^\{\(d,r\)\};\\mathcal\{T\}\_\{d,r\},0\.90\\right\)\.\(45\)The threshold\-robust gain columns are defined, for eachτ∈\{0\.85,0\.90,0\.95\}\\tau\\in\\\{0\.85,0\.90,0\.95\\\}, by

Δ​Capture​@​20τ\(d,r\)=Capture​@​0\.20​\(sfam\(d,r\);𝒯d,r,τ\)−Capture​@​0\.20​\(sprior\(d,r\);𝒯d,r,τ\)\.\\Delta\\mathrm\{Capture\}@20\_\{\\tau\}^\{\(d,r\)\}=\\mathrm\{Capture\}@0\.20\\\!\\left\(s\_\{\\mathrm\{fam\}\}^\{\(d,r\)\};\\mathcal\{T\}\_\{d,r\},\\tau\\right\)\-\\mathrm\{Capture\}@0\.20\\\!\\left\(s\_\{\\mathrm\{prior\}\}^\{\(d,r\)\};\\mathcal\{T\}\_\{d,r\},\\tau\\right\)\.\(46\)

### Seed aggregation for Table[3](https://arxiv.org/html/2607.18278#S5.T3)\.

The Table[3](https://arxiv.org/html/2607.18278#S5.T3)entries are again mean values across seeds:

FC=1R​∑r=1RNFC\(d,r\),\\displaystyle=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}N\_\{\\mathrm\{FC\}\}^\{\(d,r\)\},\(47\)Prior@20=1R​∑r=1RNprior​@​20\(d,r\),\\displaystyle=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}N\_\{\\mathrm\{prior\}@20\}^\{\(d,r\)\},\(48\)Fam\.@20=1R​∑r=1RNfam​@​20\(d,r\),\\displaystyle=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}N\_\{\\mathrm\{fam\}@20\}^\{\(d,r\)\},\(49\)@\.85=1R​∑r=1RΔ​Capture​@​200\.85\(d,r\),\\displaystyle=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}\\Delta\\mathrm\{Capture\}@20\_\{0\.85\}^\{\(d,r\)\},\(50\)@\.90=1R​∑r=1RΔ​Capture​@​200\.90\(d,r\),\\displaystyle=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}\\Delta\\mathrm\{Capture\}@20\_\{0\.90\}^\{\(d,r\)\},\(51\)@\.95=1R​∑r=1RΔ​Capture​@​200\.95\(d,r\)\.\\displaystyle=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}\\Delta\\mathrm\{Capture\}@20\_\{0\.95\}^\{\(d,r\)\}\.\(52\)

### Interpretation\.

Table[1](https://arxiv.org/html/2607.18278#S5.T1)therefore reports*normalized ranking quality*against the false\-confidence target, while Table[3](https://arxiv.org/html/2607.18278#S5.T3)reports*absolute operational recovery*and*threshold\-stability of the recovery gain*\. The first table answers whether discrepancy\-aware ranking separates dangerous confident errors better than the strongest prior baseline; the second answers how many such events are actually surfaced under a fixed review budget and how stable that advantage remains as the confidence threshold changes\.

## Appendix YPer\-dataset discrepancy\-aware weighting profiles

![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_adult/fig4_weight_profile.png)\(a\)Adult
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_bank_marketing/fig4_weight_profile.png)\(b\)Bank Marketing
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_miniboone/fig4_weight_profile.png)\(c\)MiniBooNE
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_magic_telescope/fig4_weight_profile.png)\(d\)Magic Telescope

![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_nomao/fig4_weight_profile.png)\(a\)Nomao
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_spambase/fig4_weight_profile.png)\(b\)Spambase
![Refer to caption](https://arxiv.org/html/2607.18278v1/figures/openml_phoneme/fig4_weight_profile.png)\(c\)Phoneme

Figure 8:Full per\-dataset discrepancy\-aware weighting profiles available in the anonymous review artifact\.
## Appendix ZAnonymous Artifact Availability

An anonymous review artifact for this submission is available at:

https://anonymous\.4open\.science/r/FALCON\-Discover\-876E/README\.md

The artifact provides a review\-safe release of the code and outputs used in this paper, including a compact reference implementation, reproduction scripts, precomputed result tables, figures, and an artifact card\. It is intentionally structured around table\-level and figure\-level reproduction rather than the full private development codebase, so that reviewers can inspect the core discrepancy\-discovery pipeline, reproduce the reported empirical objects, and verify the mapping between paper claims and released assets without exposing non\-shareable project infrastructure\.

The artifact is organized to support the main empirical objects in the paper: the main multi\-dataset result, operational impact and threshold robustness, discrepancy\-family ablation, extended validation, uncertainty and null\-concentration checks, and per\-dataset visual summaries\. The next appendix section provides the explicit reproduction map from each paper object to the corresponding script, inputs, and stored outputs\.

## NeurIPS Paper Checklist

1. 1\.Claims
2. Answer:\[Yes\]
3. Justification: The claims are restricted to discrepancy discovery and avoid unsupported universal superiority language\.
4. 2\.Limitations
5. Answer:\[Yes\]
6. Justification: The paper explicitly states the benchmark\-size limitation and the conditional nature of the empirical gains\.
7. 3\.Theory assumptions and proofs
8. Answer:\[N/A\]
9. Justification: The contribution is methodological and empirical rather than theorem\-driven\.
10. 4\.Experimental result reproducibility
11. Answer:\[Yes\]
12. Justification: The anonymous review artifact provides the code, dataset protocol, seed aggregation logic, reproduction scripts, and stored outputs needed to reproduce the reported empirical objects\.
13. 5\.Open access to data and code
14. Answer:\[Yes\]
15. Justification: The datasets are public and an anonymous 4open review artifact is provided for code, figures, tables, and reproduction materials\.
16. 6\.Experimental setting/details
17. Answer:\[Yes\]
18. Justification: Datasets, base learners, ranking rules, concentration metrics, and perturbation procedures are documented\.
19. 7\.Experiment statistical significance
20. Answer:\[Yes\]
21. Justification: The paper reports bootstrap deltas, intervals, and empirical probabilities for non\-positive gains\.
22. 8\.Experiments compute resources
23. Answer:\[Yes\]
24. Justification: The five\-fold, four\-seed run with XGBoost and CatBoost is reported and the released implementation is reproducible\.
25. 9\.Code of ethics
26. Answer:\[Yes\]
27. Justification: The work analyzes reliability on public datasets and does not claim autonomous deployment readiness\.
28. 10\.Broader impacts
29. Answer:\[Yes\]
30. Justification: Concentrated false\-confidence analysis can improve monitoring and escalation but also requires careful interpretation\.
31. 11\.Safeguards
32. Answer:\[Yes\]
33. Justification: The method is positioned as an analysis and training\-support tool rather than a decision substitute\.
34. 12\.Licenses for existing assets
35. Answer:\[Yes\]
36. Justification: Existing datasets and methods are cited, and the package is designed to include license information\.
37. 13\.New assets
38. Answer:\[Yes\]
39. Justification: The new assets are the discrepancy\-discovery code, plots, tables, and weighting outputs\.
40. 14\.Crowdsourcing and Research with Human Subjects
41. Answer:\[N/A\]
42. Justification: No new human\-subject data collection is involved\.
43. 15\.IRB approvals
44. Answer:\[N/A\]
45. Justification: No human\-subject study was conducted\.
46. 16\.Declaration of LLM usage
47. Answer:\[No\]
48. Justification: No LLM is part of the scientific method\.