PLSP (Pre-hoc Liminal Space Profiling): OOD Prediction over Detection -- An Anticipatory Approach for Machine Learning Model Reliability

# PLSP (Pre-hoc Liminal Space Profiling): OOD Prediction over Detection – An Anticipatory Approach for Machine Learning Model Reliability
Source: [https://arxiv.org/html/2609.12225](https://arxiv.org/html/2609.12225)
Himanshu Buckchash††thanks:Correspondence: himanshu\.buckchash@imc\.ac\.atAffiliation:IMC University of Applied SciencesAffiliation:Krems, AustriaBalasubramanian RamanAffiliation:Indian Institute of Technology RoorkeeAffiliation:Roorkee, IndiaDeepak DhunganaAffiliation:IMC University of Applied SciencesAffiliation:Krems, Austria

###### Abstract

Out\-of\-Distribution \(OOD\) data poses a significant threat to machine learning models, often leading to model failure during deployment\. All existing OOD detection methods are post\-hoc, relying on evaluation metrics such as accuracy and AUC\-ROC during inference to indirectly assess the model’s response to OOD data by measuring deviations\. In contrast to existing approaches, the proposed work shifts the paradigm fromOOD detection to OOD predictionby proposing a pre\-hoc anticipatory framework called PLSP for OOD prediction\. We make several key contributions: \(a\) a dataset\-independent metric called the CREDibility Score \(CREDS\) is proposed for OOD prediction; \(b\) credibility curves are introduced to study the maximum credibility a model can attain; and \(c\) credibility heat maps \(and volume under surface\) are introduced to characterize pre\-hoc model behavior across different datasets\. This work provides a novel perspective on signal processing under distributional shifts\. Experiments across multiple datasets demonstrate that the proposed metric serves as a valuable measure for improving the robustness of machine learning models toward OOD prediction\.

*K*eywordsout\-of\-distribution detection⋅\\cdotOOD prediction⋅\\cdotmodel robustness⋅\\cdotdistribution shift⋅\\cdotmachine learning reliability⋅\\cdotcredibility metrics

## 1Introduction

Many real world critical applications of machine learning such as medical image analysis or anomaly detection, share the common challenge of \(inference\-time\) distribution shift in various degrees\[[1](https://arxiv.org/html/2609.12225#bib.bib1)\]\. This problem arises due to the presence of OOD data during inference due to regional or environmental changes\[[2](https://arxiv.org/html/2609.12225#bib.bib2),[3](https://arxiv.org/html/2609.12225#bib.bib3)\]\. This leads to reduced performance, increased safety risk, reduced customer satisfaction, and revenue and trust loss to the service provider\[[4](https://arxiv.org/html/2609.12225#bib.bib4)\]\.

Main reason behind this challenge is the inability of machine learning methods to exhaustively model the underlying data distribution\[[5](https://arxiv.org/html/2609.12225#bib.bib5),[6](https://arxiv.org/html/2609.12225#bib.bib6),[7](https://arxiv.org/html/2609.12225#bib.bib7)\]\. Supervised models are often trained by minimizing the generalization error with the assumption that the train and test samples are drawn from an i\.i\.d\. distribution\[[8](https://arxiv.org/html/2609.12225#bib.bib8)\]\. This assumption however, does not hold on the real world data due to the presence of out\-of\-distribution \(OOD\) samples\[[4](https://arxiv.org/html/2609.12225#bib.bib4),[7](https://arxiv.org/html/2609.12225#bib.bib7),[9](https://arxiv.org/html/2609.12225#bib.bib9),[10](https://arxiv.org/html/2609.12225#bib.bib10),[11](https://arxiv.org/html/2609.12225#bib.bib11)\]\.

In the existing literature, the generalization performance is measured either using metrics like average or worst\-case accuracy over multiple OOD datasets\[[12](https://arxiv.org/html/2609.12225#bib.bib12),[13](https://arxiv.org/html/2609.12225#bib.bib13),[14](https://arxiv.org/html/2609.12225#bib.bib14),[15](https://arxiv.org/html/2609.12225#bib.bib15)\], or through a distance metric over a single OOD dataset\[[4](https://arxiv.org/html/2609.12225#bib.bib4),[9](https://arxiv.org/html/2609.12225#bib.bib9),[11](https://arxiv.org/html/2609.12225#bib.bib11),[16](https://arxiv.org/html/2609.12225#bib.bib16),[10](https://arxiv.org/html/2609.12225#bib.bib10),[17](https://arxiv.org/html/2609.12225#bib.bib17),[18](https://arxiv.org/html/2609.12225#bib.bib18)\]\. However, these approaches, do not directly perceive the importance of features during OOD data evaluation\. Due to this, they are not able to assess the impact of changes on the model for any given OOD data sample\[[19](https://arxiv.org/html/2609.12225#bib.bib19),[20](https://arxiv.org/html/2609.12225#bib.bib20),[21](https://arxiv.org/html/2609.12225#bib.bib21),[22](https://arxiv.org/html/2609.12225#bib.bib22)\]\. These approaches also lack the ability to measure generalization of the model for any potential change in OOD input\. However, the proposedCREDSmetric perceives bothfeature level understandingof a model anddistributional changes\. It enables the assessment of the maximum possible credibility in case of variation in the form of a given OOD dataset and also in absence of an OOD dataset\.

In this work,Credibilityis defined as the measure of generalizability of a model\. High credibility implies higher generalization ability of the model, similarly, low credibility implies low generalization capacity of the model\.

![Refer to caption](https://arxiv.org/html/2609.12225v1/main.png)Figure 1:Direct correlation between CREDS and accuracy across increasing noise levels \(0\-50%\) for different datasets, demonstrating CREDS as an effective competitor to post\-hoc metrics\.Our findings reveal that the proposed pre\-hoc CREDS metric provides estimates comparable to the post\-hoc accuracy metric, as evidenced by the monotonous decline in both measures with increasing noise across all datasets \(Fig\.[1](https://arxiv.org/html/2609.12225#S1.F1)\)\. This correlation demonstrates that CREDS serves as a valid measure of model trustworthiness and generalizability\. Critically, unlike existing post\-hoc metrics, CREDS can be evaluatedin the absence of any OOD dataset\(s\), offering the distinct advantage that model behavior under distributional shifts can be studied and adjusted during the development environment before deployment, thereby preventing catastrophic failures in production settings\.

CREDS operatesby first measuring how much the statistical distribution of each input feature in new data deviates from its training\-data counterpart, for each specific class\. This deviation is converted into a foundational credibility score for the feature and class, subsequently weighted by the model’s reliance on that feature\. Finally, these model\-aware feature credibilities are aggregated across all features and then averaged over all classes to produce the overall CREDS score\.

The main contributions of this work are:

- •To the best of our knowledge, we are the first to propose a direct measure \(CREDS\) of a machine learning model’s generalization ability and show its application\. It is an approximate formulation, under assumptions, since a closed form solution is not possilbe\.
- •We also propose a method to assess possible changes in the credibility of a model upon distributional changes due to OOD data during inference\. The proposed AUC metric for change in credibility, helps us understand generalization of a model for a particular OOD dataset\. We also propose a new Volume Under Surface \(VUS\) metric to measure the extent of model generalization for changes in input distribution, given we don’t have any OOD data to evaluate our model\.
- •We conduct experiments on multiple datasets to show the usage of credibility scores and associated methods to evaluate models\. We also study the impact of noise on credibility and generalization of a model\.

## 2Credibility Based OOD Evaluation

In Section[2\.1](https://arxiv.org/html/2609.12225#S2.SS1), we mathematically introduce CREDS that helps us to evaluate how the model sees an input distribution\. It is an approximate formulation \(under stated assumptions\) which is empirically validated to work\. In Section[2\.2](https://arxiv.org/html/2609.12225#S2.SS2), we then introduce credibility curves to understand the change in credibility score with changing mean and deviation\. From these curves, we find the maximum possible credibility score, a model can achieve, given changes in an input data stream\. Lastly, In Section[2\.3](https://arxiv.org/html/2609.12225#S2.SS3), we evaluate the overall vulnerability of the model for OOD data\.

### 2\.1Credibility Score \(CREDS\)

The way a model is affected by OOD data depends mainly on two factors\. Firstly,how does the distribution of OOD data vary w\.r\.t\. the training data?Secondly,how the changes in feature distribution are perceived by the model?To understand this, we mathematically define credibility \(CREDS\),Ck,j\{C\}\_\{k,j\}for thekt​hk^\{th\}feature, given a class labeljjas:

Ck,j=e−DK​L\(p\(Xvk\|Y=j\)\|\|q\(Xtk\|Y=j\)\)=e−Ep⁡\(Xkv\|Y=j\)​\[log⁡\(p⁡\(Xkt\|Y=j\)OPENq⁡\(Xkv\|Y=j\)\)\)\]\\begin\{split\}C\_\{k,j\}&=e^\{\-D\_\{KL\}\(p\(X^\{v\}\_\{k\}\|Y=j\)\|\|q\(X^\{t\}\_\{k\}\|Y=j\)\)\}\\\\ &=e^\{\-\\mathrm\{E\}\_\{p\(X^\{v\}\_\{k\}\|Y=j\)\}\\Big\[\\log\\big\(\\frac\{p\(X^\{t\}\_\{k\}\|Y=j\)\}\{q\(X^\{v\}\_\{k\}\|Y=j\)\)\}\\big\)\\Big\]\}\\\\ \\end\{split\}\(1\)WhereXktX^\{t\}\_\{k\}andXkvX^\{v\}\_\{k\}arekt​hk^\{th\}feature for training and OOD dataset respectively, andqqandppare the respective probability distributions\. Here,DK​LD\_\{KL\}represents the KL divergence between the two distributions\[[23](https://arxiv.org/html/2609.12225#bib.bib23),[24](https://arxiv.org/html/2609.12225#bib.bib24)\]\. If we considerppandqqas multivariate normal distributions with meanμv,k\\mu\_\{v,k\}andμt,k\\mu\_\{t,k\}and covariance matrixΣv,k\\Sigma\_\{v,k\}andΣt,k\\Sigma\_\{t,k\}respectively with dimensionsλ\\lambda, then we can compute:

DK​L\(p\|\|q\)=12\{tr\{Σt,k−1Σv,k\}\+\(μt,k−μv,k\)T​Σt,k−1​\(μt,k−μv,k\)−λ\+ln\|Σt,k\|\|Σv,k\|\}\\begin\{split\}D\_\{KL\}\(p\|\|q\)&=\\frac\{1\}\{2\}\\Big\\\{tr\\\{\\Sigma\_\{t,k\}^\{\-1\}\\Sigma\_\{v,k\}\\\}\\\\ &\+\(\\mu\_\{t,k\}\-\\mu\_\{v,k\}\)^\{T\}\\Sigma\_\{t,k\}^\{\-1\}\(\\mu\_\{t,k\}\-\\mu\_\{v,k\}\)\\\\ &\-\\lambda\+\\ln\{\\frac\{\|\\Sigma\_\{t,k\}\|\}\{\|\\Sigma\_\{v,k\}\|\}\}\\Big\\\}\\end\{split\}\(2\)For simplicity of calculation \(for tractability and feasibility\) we can estimate deviationσv,k\\sigma\_\{v,k\}for the dataset instead of covarianceΣv,k\\Sigma\_\{v,k\}\. Assuming a normal distribution, we can rewriteDK​LD\_\{KL\}as:

DK​L=\[ln⁡\(σt,kσv,k\)\+σv,k2\+\(μv,k−μt,k\)22​σt,k2−12\]D\_\{KL\}=\\Big\[\\ln\\big\(\\frac\{\\sigma\_\{t,k\}\}\{\\sigma\_\{v,k\}\}\\big\)\+\\frac\{\\sigma\_\{v,k\}^\{2\}\+\(\\mu\_\{v,k\}\-\\mu\_\{t,k\}\)^\{2\}\}\{2\\sigma\_\{t,k\}^\{2\}\}\-\\frac\{1\}\{2\}\\Big\]\(3\)
Even if the true distributions aren’t perfectly Gaussian, changes in their central tendency \(mean\) and dispersion \(variance\) are often the most significant and impactful types of shifts for many ML models\. The GaussianDK​LD\_\{KL\}formula is sensitive to exactly these\. Moreover, by calculatingCk,jC\_\{k,j\}for each feature and class, CREDS can pinpoint where the distributional shift is occurring at a granular level\.

The credibilityCk,jC\_\{k,j\}just accounts for distribution difference in between OOD data and training data\. To incorporate how model perceives the distributional changes, we take feature importancefkf\_\{k\}into account to calculate class credibilityCjC\_\{j\}as:Cj=∑kfk​Ck,jC\_\{j\}=\\sum\_\{k\}f\_\{k\}C\_\{k,j\}\. To estimate a feature’s overall effects on the results, we can also estimateCkC\_\{k\}for a given feature as:Ck=∑jnj∑jnj​Ck,jC\_\{k\}=\\sum\_\{j\}\\frac\{n\_\{j\}\}\{\\sum\_\{j\}n\_\{j\}\}C\_\{k,j\}\. Wherenjn\_\{j\}are the number of samples in classjjof training dataset\. We can further compute the overall credibility scoreCCof the model as:

C=∑j∑knj∑jnj​fk​Ck,jC=\\sum\_\{j\}\\sum\_\{k\}\\frac\{n\_\{j\}\}\{\\sum\_\{j\}n\_\{j\}\}f\_\{k\}C\_\{k,j\}\(4\)Overall, credibilityCCis a collective measure of: “How model observes input distribution weighted over all the classes and features"\.

The credibility score \(CREDS\) isnot a performance scorelike Accuracy or AUC\-ROC\. Instead it tells us about feasibility to utilize our model to predict on given test data\. The metric helps to understand tendency to generalize under unknown distributions\. This includes both data and model’s perspective on perceiving the data\.

### 2\.2Credibility Curves

To evaluate the changes observed in credibility score with mean and deviation of input data stream we propose utilization of credibility curves\. These curves show changes inCjC\_\{j\}for a classjjby fixing eitherμv,k\\mu\_\{v,k\}orσv,k\\sigma\_\{v,k\}for allkkfeatures\.

To assess generalization of credibility for a classjjwith given set ofσv,k\\sigma\_\{v,k\}, we vary the value ofμv,k\\mu\_\{v,k\}for each feature in a range−ϵ\-\\epsilonto\+ϵ\+\\epsilongivenϵ\>0\\epsilon\>0\. We calculate Area Under theCjC\_\{j\}vsμv\\mu\_\{v\}curve for a given classkkto estimate the potential of network as:

A​U​C​μvjϵ=∫−ϵϵ∑kfk​Ck,j​\(μt,k,μv,σt,k,σv,k\)​d​μvAUC\{\\mu\_\{v\}\}^\{\\epsilon\}\_\{j\}=\\int\_\{\-\\epsilon\}^\{\\epsilon\}\\sum\_\{k\}f\_\{k\}C\_\{k,j\}\(\\mu\_\{t,k\},\\mu\_\{v\},\\sigma\_\{t,k\},\\sigma\_\{v,k\}\)d\\mu\_\{v\}\(5\)
A​U​C​μvϵ=∑jnj∑jnj​A​U​C​μvjϵAUC\{\\mu\_\{v\}\}^\{\\epsilon\}=\\sum\_\{j\}\\frac\{n\_\{j\}\}\{\\sum\_\{j\}n\_\{j\}\}AUC\{\\mu\_\{v\}\}^\{\\epsilon\}\_\{j\}\(6\)
whereCk,j\(\.\)C\_\{k,j\}\(\.\)function gives estimate ofCk,jC\_\{k,j\}for a set of parameters\. Similarly, we calculate Area Under theCjC\_\{j\}vsσv\\sigma\_\{v\}curve for a given classkkto estimate the potential of network as:

A​U​C​σvjϵ=∫0ϵ∑kfk​Ck,j​\(μt,k,μv,k,σt,k,σv\)​d​σvAUC\{\\sigma\_\{v\}\}^\{\\epsilon\}\_\{j\}=\\int\_\{0\}^\{\\epsilon\}\\sum\_\{k\}f\_\{k\}C\_\{k,j\}\(\\mu\_\{t,k\},\\mu\_\{v,k\},\\sigma\_\{t,k\},\\sigma\_\{v\}\)d\\sigma\_\{v\}\(7\)A​U​C​σvϵ=∑jnj∑jnj​A​U​C​σvjϵAUC\{\\sigma\_\{v\}\}^\{\\epsilon\}=\\sum\_\{j\}\\frac\{n\_\{j\}\}\{\\sum\_\{j\}n\_\{j\}\}AUC\{\\sigma\_\{v\}\}^\{\\epsilon\}\_\{j\}\(8\)
Higher the value ofA​U​C​μvjϵAUC\{\\mu\_\{v\}\}^\{\\epsilon\}\_\{j\}orA​U​C​σvjϵAUC\{\\sigma\_\{v\}\}^\{\\epsilon\}\_\{j\}, more credibility one model can sustain for each classjjon potential changes inμv\\mu\_\{v\}orσv\\sigma\_\{v\}\. We also estimate the maximum credibilityCj,m​a​xμC^\{\\mu\}\_\{j,max\}andCj,m​a​xσC^\{\\sigma\}\_\{j,max\}for variation ofμv\\mu\_\{v\}andσv\\sigma\_\{v\}respectively\. For that we estimate∂Ck,j/∂μv\\partial C\_\{k,j\}/\\partial\\mu\_\{v\}and∂Ck,j/∂σv\\partial C\_\{k,j\}/\\partial\\sigma\_\{v\}as:

∂Cj∂μv=∑kfk∂Ck,j∂μv=−∑kfk\(μv−μt,k\)σt,k2Ck,j\\frac\{\\partial C\_\{j\}\}\{\\partial\\mu\_\{v\}\}=\\sum\_\{k\}f\_\{k\}\\frac\{\\partial C\_\{k,j\}\}\{\\partial\\mu\_\{v\}\}=\-\\sum\_\{k\}f\_\{k\}\\frac\{\(\\mu\_\{v\}\-\\mu\_\{t,k\}\)\}\{\\sigma\_\{t,k\}^\{2\}\}C\_\{k,j\}\(9\)
∂Cj∂σv=∑kfk​∂Ck,j∂σv=∑kfk​\[1σv−σvσt,k2\]​Ck,j\\frac\{\\partial C\_\{j\}\}\{\\partial\\sigma\_\{v\}\}=\\sum\_\{k\}f\_\{k\}\\frac\{\\partial C\_\{k,j\}\}\{\\partial\\sigma\_\{v\}\}=\\sum\_\{k\}f\_\{k\}\\Big\[\\frac\{1\}\{\\sigma\_\{v\}\}\-\\frac\{\\sigma\_\{v\}\}\{\\sigma\_\{t,k\}^\{2\}\}\\Big\]C\_\{k,j\}\(10\)For a maxima we estimateCjC\_\{j\}such that∂Cj/∂μv=0\\partial C\_\{j\}/\\partial\\mu\_\{v\}=0and∂2Cj/∂2μv<0\\partial^\{2\}C\_\{j\}/\\partial^\{2\}\\mu\_\{v\}<0and∂Cj/∂σv=0\\partial C\_\{j\}/\\partial\\sigma\_\{v\}=0and∂2Cj/∂2σv<0\\partial^\{2\}C\_\{j\}/\\partial^\{2\}\\sigma\_\{v\}<0\. For the case, givenμt,k=μt​∀k\\mu\_\{t,k\}=\\mu\_\{t\}\\forall k, one possible solution for same isμv=μt\\mu\_\{v\}=\\mu\_\{t\}\. Other possible way to find a maxima, is to generate samples for range of values forμv\\mu\_\{v\}andσv\\sigma\_\{v\}and estimateCj,m​a​xμC^\{\\mu\}\_\{j,max\}andCj,m​a​xσC^\{\\sigma\}\_\{j,max\}\. These curves help to see how change in current distribution affects the credibility score of the potential data\.

### 2\.3Model Vulnerability Analysis

If we don’t have an available OOD dataset, we would still like to see how our model behave to potential OOD distributions\. To do so we plot a heatmap ofCjC\_\{j\}with variation in bothμv,k\\mu\_\{v,k\}andσv,k\\sigma\_\{v,k\}such thatμv,k=μv​∀k\\mu\_\{v,k\}=\\mu\_\{v\}\\forall kandσv,k=σv​∀k\\sigma\_\{v,k\}=\\sigma\_\{v\}\\forall k\. These plots help us to understand variation of credibility for a class with distribution of data\.

Similar toA​U​Cμv​jϵAUC\_\{\\mu\_\{v\}j\}^\{\\epsilon\}andA​U​Cσv​jϵAUC\_\{\\sigma\_\{v\}j\}^\{\\epsilon\}we can compute the Volume Under the Surface \(VUS\) of these plots betweenCjC\_\{j\},μv\\mu\_\{v\}andσv\\sigma\_\{v\}to see how well the model performs for a class when there is a change in distribution\. Mathematically we can define it as:

V​U​Sjα,β=∫0α∫−ββ∑kfk​Ck,j​\(μt,k,μv,σt,k,σv\)​d​μv​d​σvVUS\_\{j\}^\{\\alpha,\\beta\}=\\int\_\{0\}^\{\\alpha\}\\int\_\{\-\\beta\}^\{\\beta\}\\sum\_\{k\}f\_\{k\}C\_\{k,j\}\(\\mu\_\{t,k\},\\mu\_\{v\},\\sigma\_\{t,k\},\\sigma\_\{v\}\)d\\mu\_\{v\}d\\sigma\_\{v\}\(11\)V​U​Sα,β=∑jnj∑jnj​V​U​Sjα,βVUS^\{\\alpha,\\beta\}=\\sum\_\{j\}\\frac\{n\_\{j\}\}\{\\sum\_\{j\}n\_\{j\}\}VUS\_\{j\}^\{\\alpha,\\beta\}\(12\)WhereV​U​Sα,βVUS^\{\\alpha,\\beta\}gives an overall metrics for a model\. Higher the value ofV​U​Sjα,βVUS\_\{j\}^\{\\alpha,\\beta\}, more chances for a class to achieve better credibility for potential OOD data\. If all the features of the dataset have aμt,k=0\\mu\_\{t,k\}=0andσt,k=1\\sigma\_\{t,k\}=1, then there is atleast one point for whichCjC\_\{j\}is exactly equal to 1\.

## 3Experiments

This section demonstrates how these methods can be utilized for evaluating various cases\. We evaluate on five benchmark datasets spanning handwritten digits \(MNIST\[[25](https://arxiv.org/html/2609.12225#bib.bib25)\], SVHN\[[26](https://arxiv.org/html/2609.12225#bib.bib26)\]\) and natural images \(CIFAR\-10\[[27](https://arxiv.org/html/2609.12225#bib.bib27)\], CINIC\-10\[[28](https://arxiv.org/html/2609.12225#bib.bib28)\], STL10\[[29](https://arxiv.org/html/2609.12225#bib.bib29)\]\), with training samples ranging from 500 to 90K\. All images are preprocessed into feature vectors of size 49 \(digit datasets\) or 64 \(natural image datasets\) through dimensionality reduction\. This diverse collection enables comprehensive evaluation across different data distributions and complexities\. Additionally,Noisy Datasetsare created to see how increasing noise impacts the model credibility\. Four noise levels are used:10%10\\%,20%20\\%,30%30\\%,and40%40\\%\.

#### 3\.0\.1Training Details

We run experiments by pairing MNIST with SVHN and CIFAR\-10 with CINIC\-10 and STL\-10\. The dataset MNIST and SVHN form a pair of dataset which are out of distribution for each other\. CIFAR\-10, CINIC\-10 and STL\-10 forms a group datasets which is partially different from each other\. We train a random forest\[[30](https://arxiv.org/html/2609.12225#bib.bib30)\]model on features and utilize the feature importance received from the model to calculate credibility scores \(CREDS\),CC\. We further estimate credibility curves by changingμv\\mu\_\{v\}andσv\\sigma\_\{v\}and estimateCm​a​xμC^\{\\mu\}\_\{max\}andCm​a​xσC^\{\\sigma\}\_\{max\}for the dataset and also estimateA​U​Cμv5AUC^\{5\}\_\{\\mu\_\{v\}\}andA​U​Cσv5AUC^\{5\}\_\{\\sigma\_\{v\}\}\. For these datasets we also estimateV​U​S5,5VUS^\{5,5\}\.

Table 1:Overall CREDS results \(with mean and deviation\) for MNIST and SVHN\. Credibility is estimated for all the permutations of SVHN and MNIST dataset\. Additionally, noise was varied in each dataset from 0% to 50 % to note the change in CREDS\.
#### 3\.0\.2Credibility on MNIST & SVHN

We train our model on MNIST and test on SVHN and vice versa and observe the credibility score as given in Table[1](https://arxiv.org/html/2609.12225#S3.T1)\. Intra\-dataset training and testing on MNIST & SVHN give high CREDS values of0\.9710\.971&0\.9860\.986respectively\. Higher CREDS on intra\-dataset evaluation, show that the test data acts like an In\-Distribution for the model\. This is becauseμv,k→μt,k\\mu\_\{v,k\}\\rightarrow\\mu\_\{t,k\}andσv,k→σt,k\\sigma\_\{v,k\}\\rightarrow\\sigma\_\{t,k\}\. This bringsDK​LD\_\{KL\}\(refer \(Equation[3](https://arxiv.org/html/2609.12225#S2.E3)\)\) tend to00, in turn bringingCk,jC\_\{k,j\}\(refer \(Equation[1](https://arxiv.org/html/2609.12225#S2.E1)\)\) close to11\. When training on MNIST and testing on SVHN, CREDS is0\.45810\.4581, indicating significant dataset differences\. The reverse configuration yields0\.44460\.4446, slightly lower due to differences in prior distributionsp⁡\(Xkt/Y=j\)p\(X^\{t\}\_\{k\}/Y=j\), feature importancesfkf\_\{k\}, and class distributionsnj/∑jnjn\_\{j\}/\\sum\_\{j\}n\_\{j\}\. Note thatCk,jC\_\{k,j\}, being exponential inDK​LD\_\{KL\}, captures larger variations at higherDK​LD\_\{KL\}values despite small changes in credibility\.

![Refer to caption](https://arxiv.org/html/2609.12225v1/credibility_curves.png)Figure 2:Credibility Curves for variation inμv\\mu\_\{v\}andσv\\sigma\_\{v\}for \(a\) model trained on MNIST and tested on SVHN \(b\) model trained on SVHN and tested on MNIST \(c\) model trained on CINIC\-10 and tested on CIFAR\-10 \(d\) model trained on CIFAR\-10 and tested on CINIC\-10\.Table 2:Credibility results \(with mean and std\.\) for experiments on CIFAR\-10, STL\-10, and CINIC\-10\. Credibility is estimated for all the permutations of all datasets\. Noise was varied in each dataset from 0% to 50 % to note the change in CREDS\.
#### 3\.0\.3Credibility on CIFAR\-10, CINIC\-10 & STL\-10

Intra\-dataset CREDSs \(Table[2](https://arxiv.org/html/2609.12225#S3.T2)\) are high:0\.9980\.998,0\.9990\.999, and0\.9970\.997for CIFAR\-10, CINIC\-10, and STL\-10 respectively\. CIFAR\-10/CINIC\-10 cross\-evaluation yields∼\\sim0\.94, as CINIC\-10 contains CIFAR\-10 images, making prior probabilitiesp⁡\(Xkt/Y=j\)p\(X\_\{k\}^\{t\}/Y=j\)similar\. However, CIFAR\-10/STL\-10 cross\-evaluation drops to≈\\approx0\.89, indicating dissimilar feature distributions despite visual similarity\.

#### 3\.0\.4Impact of Noise on Credibility

Addition of Gaussian noiseN⁡\(0,1\)N\(0,1\)deviates the posterior probabilityp⁡\(Xkv\|Y=j\)p\(X^\{v\}\_\{k\}\|Y=j\)from the priorp⁡\(Xkt\|Y=j\)p\(X^\{t\}\_\{k\}\|Y=j\), decreasing credibilityCk,jC\_\{k,j\}\(Tables[1](https://arxiv.org/html/2609.12225#S3.T1)and[2](https://arxiv.org/html/2609.12225#S3.T2)\)\. MNIST exhibits larger credibility drops than SVHN due to simpler learned features being more susceptible to noise\.

Table 3:Cm​a​xμC^\{\\mu\}\_\{max\},Cm​a​xσC^\{\\sigma\}\_\{max\},A​U​Cμv5AUC\_\{\\mu\_\{v\}\}^\{5\}andA​U​Cσv5AUC\_\{\\sigma\_\{v\}\}^\{5\}on MNIST and SVHN \(with mean and std\. on multiple runs\)\.Cm​a​xμC^\{\\mu\}\_\{max\},Cm​a​xσC^\{\\sigma\}\_\{max\},A​U​Cμv5AUC\_\{\\mu\_\{v\}\}^\{5\}andA​U​Cσv5AUC\_\{\\sigma\_\{v\}\}^\{5\}are estimated for all the permutations of SVHN and MNIST\. Noise was varied from 0% to 50 % to observe how noise effectsCm​a​xμC^\{\\mu\}\_\{max\},Cm​a​xσC^\{\\sigma\}\_\{max\},A​U​Cμv5AUC\_\{\\mu\_\{v\}\}^\{5\}andA​U​Cσv5AUC\_\{\\sigma\_\{v\}\}^\{5\}\.
#### 3\.0\.5Credibility Curves

Fig\.[2](https://arxiv.org/html/2609.12225#S3.F2)shows credibility curves forμv\\mu\_\{v\}andσv\\sigma\_\{v\}variations\. MNIST\-trained models show lowerCj,m​a​xμC\_\{j,max\}^\{\\mu\}andCj,m​a​xσC\_\{j,max\}^\{\\sigma\}than SVHN\-trained models, indicating higher susceptibility to distributional changes\. CIFAR\-10 and CINIC\-10 exhibit similar credibility patterns, reflecting their dataset similarity\.

#### 3\.0\.6Maximum Credibility Potential of a Model

Tables[3](https://arxiv.org/html/2609.12225#S3.T3)and[4](https://arxiv.org/html/2609.12225#S3.T4)demonstrate models can achieve maximum potential credibilities \(Cm​a​xμC\_\{max\}^\{\\mu\},Cm​a​xσC\_\{max\}^\{\\sigma\}\) exceeding their baselineCCon OOD data \(e\.g\., MNIST→\\rightarrowSVHN baselineC=0\.4581C=0\.4581, butCm​a​xμ=0\.6323C\_\{max\}^\{\\mu\}=0\.6323\)\. Adding Gaussian noise minimally impactsCm​a​xμC\_\{max\}^\{\\mu\}but significantly degradesCm​a​xσC\_\{max\}^\{\\sigma\}, showing that noise limits model robustness primarily through variance shifts rather than mean shifts\.

#### 3\.0\.7Model Generalization on Varying OOD Dataset

We computeA​U​Cμv5AUC^\{5\}\_\{\\mu\_\{v\}\}andA​U​Cσv5AUC^\{5\}\_\{\\sigma\_\{v\}\}\(Tables[3](https://arxiv.org/html/2609.12225#S3.T3),[4](https://arxiv.org/html/2609.12225#S3.T4)\) to quantify generalization under parameter variations\. As expected, AUC values are higher for structurally similar test sets\. With increasing noise,A​U​Cμv5AUC^\{5\}\_\{\\mu\_\{v\}\}rises whileA​U​Cσv5AUC^\{5\}\_\{\\sigma\_\{v\}\}falls, suggesting model credibility remains more consistent under mean shifts than under deviation shifts in noisy environments\.

#### 3\.0\.8Overall Model Generalization Estimates

Credibility heatmaps \(Fig\.[3](https://arxiv.org/html/2609.12225#S3.F3)\) reveal class\-specific patterns inCjC\_\{j\}for simultaneousμv\\mu\_\{v\}andσv\\sigma\_\{v\}variations\.V​U​S5,5VUS^\{5,5\}results \(Table[5](https://arxiv.org/html/2609.12225#S3.T5)\) show SVHN models generalize better than MNIST, while CINIC\-10 slightly outperforms CIFAR\-10 and STL\-10 due to ImageNet augmentation and larger sample size\.

Table 4:Cm​a​xμC^\{\\mu\}\_\{max\},Cm​a​xσC^\{\\sigma\}\_\{max\},A​U​Cμv5AUC\_\{\\mu\_\{v\}\}^\{5\}andA​U​Cσv5AUC\_\{\\sigma\_\{v\}\}^\{5\}for various experiments on CIFAR\-10, STL\-10 and CINIC\-10 datasets\. Results are reported in the form of mean and deviation across multiple runs\.Cm​a​xμC^\{\\mu\}\_\{max\},Cm​a​xσC^\{\\sigma\}\_\{max\},A​U​Cμv5AUC\_\{\\mu\_\{v\}\}^\{5\}andA​U​Cσv5AUC\_\{\\sigma\_\{v\}\}^\{5\}are estimated for all the permutations of CIFAR\-10, STL\-10 and CINIC\-10 dataset\. Additionally, we increase noise level in each dataset from 0% to 50 % to observe how noise effectsCm​a​xμC^\{\\mu\}\_\{max\},Cm​a​xσC^\{\\sigma\}\_\{max\},A​U​Cμv5AUC\_\{\\mu\_\{v\}\}^\{5\}andA​U​Cσv5AUC\_\{\\sigma\_\{v\}\}^\{5\}\.![Refer to caption](https://arxiv.org/html/2609.12225v1/SVHN_MNIST_0_1.png)Figure 3:Credibility heatmaps for model trained on \(a\) MNIST and \(b\) SVHN for class labels 0 \(top\) and 1 \(bottom\)\.Table 5:V​U​S5,5VUS^\{5,5\}calculated for models trained on MNIST, SVHN, CIFAR\-10, STL\-10 and CINIC\-10, with mean and deviation across multiple runs\.

## 4Conclusion

This work introduces PLSP, a paradigm shift from post\-hoc OOD detection to pre\-hoc OOD prediction through the CREDibility Score \(CREDS\) metric\. Unlike existing approaches requiring OOD datasets during inference, CREDS enables anticipatory assessment of model behavior under distributional shifts during development\. Experiments across MNIST, SVHN, CIFAR\-10, CINIC\-10, and STL\-10 demonstrate that CREDS provides estimates comparable to post\-hoc accuracy metrics while offering evaluation in the absence of OOD data\. The introduced credibility curves, heatmaps, and VUS metrics provide comprehensive tools for analyzing model vulnerability, revealing differential sensitivity to mean versus variance shifts\. The strong correlation between CREDS and accuracy validates its utility as a pre\-deployment robustness indicator, enabling proactive adjustments to prevent catastrophic failures in production environments\.

## References

- \[1\]Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu\.Generalized out\-of\-distribution detection: A survey\.arXiv preprint arXiv:2110\.11334, 2021\.
- \[2\]Abhijit Guha Roy, Jie Ren, Shekoofeh Azizi, Aaron Loh, Vivek Natarajan, Basil Mustafa, Nick Pawlowski, Jan Freyberg, Yuan Liu, Zach Beaver, et al\.Does your dermatology classifier know what it doesn’t know? detecting the long\-tail of unseen conditions\.arXiv preprint arXiv:2104\.03829, 2021\.
- \[3\]Ishaan Gulrajani and David Lopez\-Paz\.In search of lost domain generalization\.InInternational Conference on Learning Representations, 2020\.
- \[4\]Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan\.Exploring the limits of out\-of\-distribution detection\.arXiv preprint arXiv:2106\.03004, 2021\.
- \[5\]Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton\.Imagenet classification with deep convolutional neural networks\.Advances in neural information processing systems, 25:1097–1105, 2012\.
- \[6\]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun\.Delving deep into rectifiers: Surpassing human\-level performance on imagenet classification\.InProceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015\.
- \[7\]Nick Drummond and Rob Shearer\.The open world assumption\.IneSI Workshop: The Closed World of Databases meets the Open World of the Semantic Web, volume 15, 2006\.
- \[8\]Yong Liu\.Unbiased estimate of generalization error and model selection in neural network\.Neural Networks, 8\(2\):215–219, 1995\.
- \[9\]Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Ledsam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al\.Contrastive training for improved out\-of\-distribution detection\.arXiv preprint arXiv:2007\.05566, 2020\.
- \[10\]Timo Milbich, Karsten Roth, Samarth Sinha, Ludwig Schmidt, Marzyeh Ghassemi, and Björn Ommer\.Characterizing generalization under out\-of\-distribution shifts in deep metric learning\.InAdvances in Neural Information Processing Systems, 2021\.
- \[11\]Hoang Thanh\-Tung and Truyen Tran\.Toward a generalization metric for deep generative models\.In”I Can’t Believe It’s Not Better\!”NeurIPS 2020 workshop, 2020\.
- \[12\]Charlie Frogner, Sebastian Claici, Edward Chien, and Justin Solomon\.Incorporating unlabeled data into distributionally robust learning\.Journal of Machine Learning Research, 22\(56\):1–46, 2021\.
- \[13\]Zheyan Shen, Peng Cui, Tong Zhang, and Kun Kunag\.Stable learning via sample reweighting\.InProceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5692–5699, 2020\.
- \[14\]Jiashuo Liu, Zheyuan Hu, Peng Cui, Bo Li, and Zheyan Shen\.Heterogeneous risk minimization\.arXiv preprint arXiv:2105\.03818, 2021\.
- \[15\]Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy\.Domain generalization: A survey\.arXiv preprint arXiv:2103\.02503, 2021\.
- \[16\]Haotian Ye, Chuanlong Xie, Yue Liu, and Zhenguo Li\.Out\-of\-distribution generalization analysis via influence function\.arXiv preprint arXiv:2101\.08521, 2021\.
- \[17\]Nanyang Ye, Kaican Li, Lanqing Hong, Haoyue Bai, Yiting Chen, Fengwei Zhou, and Zhenguo Li\.Ood\-bench: Benchmarking and understanding out\-of\-distribution generalization datasets and algorithms\.arXiv preprint arXiv:2106\.03721, 2021\.
- \[18\]Behnam Neyshabur, Srinadh Bhojanapalli, David Mcallester, and Nati Srebro\.Exploring generalization in deep learning\.Advances in Neural Information Processing Systems, 30:5947–5956, 2017\.
- \[19\]Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez\-Paz\.Invariant risk minimization\.stat, 1050:27, 2020\.
- \[20\]David Krueger, Ethan Caballero, Joern\-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville\.Out\-of\-distribution generalization via risk extrapolation \(rex\)\.InInternational Conference on Machine Learning, pages 5815–5826\. PMLR, 2021\.
- \[21\]Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang\.Adversarial domain adaptation with domain mixup\.InProceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6502–6509, 2020\.
- \[22\]Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren\.Improve unsupervised domain adaptation with mixup training\.arXiv preprint arXiv:2001\.00677, 2020\.
- \[23\]Solomon Kullback\.Information Theory and Statistics\.Wiley, New York, 1959\.
- \[24\]Yufeng Zhang, Wanwei Liu, Zhenbang Chen, Kenli Li, and Ji Wang\.On the properties of kullback\-leibler divergence between gaussians\.arXiv preprint arXiv:2102\.05485, 2021\.
- \[25\]Li Deng\.The mnist database of handwritten digit images for machine learning research \[best of the web\]\.IEEE Signal Processing Magazine, 29\(6\):141–142, 2012\.
- \[26\]Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y\. Ng\.Reading digits in natural images with unsupervised feature learning\.InNIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011\.
- \[27\]Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar\.Do cifar\-10 classifiers generalize to cifar\-10?arXiv preprint arXiv:1806\.00451, 2018\.
- \[28\]Luke N Darlow, Elliot J Crowley, Antreas Antoniou, and Amos J Storkey\.Cinic\-10 is not imagenet or cifar\-10\.arXiv e\-prints, pages arXiv–1810, 2018\.
- \[29\]Adam Coates, Honglak Lee, and Andrew Y\. Ng\.An analysis of single layer networks in unsupervised feature learning\.2011\.
- \[30\]Leo Breiman\.Random forests\.Machine learning, 45\(1\):5–32, 2001\.