Neural Feature Governance: Extending Atom Prevalence

# Neural Feature Governance: Extending Atom Prevalence
Source: [https://arxiv.org/html/2607.21671](https://arxiv.org/html/2607.21671)
Idris Karel Seunda EkweCorresponding author\. Email:idris\.seunda@aims\-cameroon\.orgAfrican Institute for Mathematical Sciences \(AIMS\), CameroonPatrick Tenga ShakoErnest Parfait FokouéCorresponding author\. Email:epfeqa@rit\.eduRochester Institute of Technology \(RIT\), US

###### Abstract

Neural network compression and interpretability remain open challenges in modern deep learning, where billion\-parameter architectures deliver impressive accuracy at the cost of transparency, computational efficiency, and reliable uncertainty quantification\. This paper introduces Neural Atom Prevalence \(NAP\), a principled Bayesian framework for structured node\-level model selection in feedforward neural networks\. NAP introduces the neural atom \(activation unit\) and functions as a hybrid method operating through a four\-phase pipeline: Bayesian Lottery Ticket \(BLT\) identification via Iterative Magnitude Pruning \(IMP\), soft variational training of the Spike and Slab Independent Gaussian \(SS\-IG\) model, Poisson\-Binomial \(PB\) optimal layer\-size selection, and Bayesian fine\-tuning to produce a sparse, stable, interpretable, and accurate model\. Extensive empirical validation across simulated nonlinear regression, two UCI benchmark datasets \(Concrete, YearPredictionMSD\), and the MNIST image classification task demonstrates that NAP achieves state\-of\-the\-art structural sparsity, reducing active nodes to as few as8%8\\%of the original dense architecture on MNIST, while well\-calibrated probabilistically: the aleatoric\-epistemic uncertainty decomposition reveals that model ignorance accounts for only 3 to4%4\\%of total predictive variance across all experiments, and regression reliability diagrams confirm a near\-nominal predictive interval coverage \(93\.4% observed against a 95% target\)\. These results establish NAP as a reliable, theoretically grounded, and computationally tractable solution to the simultaneous pursuit of sparsity, accuracy, interpretability, and uncertainty quantification in Bayesian neural networks\.

Keywords:Bayesian neural network, neural atom prevalence, spike and slab, uncertainty quantification, Poisson\-Binomial\.

## 1Introduction

The neural network is now gaining important interest in the machine learning environment due to their high flexibility and its capacity to approximate any function with high precision \(Horniket al\.\([1989](https://arxiv.org/html/2607.21671#bib.bib9)\)\)\. However, if having a model that achieves higher performance is great, understanding how it achieves this performance is crucial\. The Bayesian approach is known as an appropriate framework for construction of interpretable Deep learning models that allow quantification of uncertainty \(Blundellet al\.\([2015](https://arxiv.org/html/2607.21671#bib.bib10)\)\)\. But neural network models are characterized by a low level of interpretability \(usually called a black box\) and their complexity, related to their architecture, sometimes composed of networks of billions of weights and neurons which make it difficult to implement\. Additionally, although the Bayesian framework offers uncertainty quantification, it unfortunately has the drawback of making calculations heavier, thereby increasing computational performance demands for building models that are efficient, flexible, interpretable, and stable \(Blundellet al\.\([2015](https://arxiv.org/html/2607.21671#bib.bib10)\)\)\. To overcome these issues, the model compression has been found as a good strategy, trying to reduce model sizes and complexity while keeping acceptable performances, interpretability and stability\. Model compression can be achieve by improving the sparsity of models, feature selection or more generally model selection\. All these include the same concepts of, going from a given set of variables, atoms or neurons, eliminate those with nonsignificant or not positive contribution to the success of final machine learning task \(Hanet al\.\([2015](https://arxiv.org/html/2607.21671#bib.bib6)\); Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\)\.

Today, research on interpretable and sparse neural network architectures has achieved great progress with the development of an empirical framework such as the Lottery ticket hypothesis \(Frankle and Carbin \([2019](https://arxiv.org/html/2607.21671#bib.bib4)\)\) able to prune up to 80% of the network with at least the same performance as the dense network\. This frame has been converted into Bayesian to develop the Bayesian winning ticket \(Kuhnet al\.\([2026](https://arxiv.org/html/2607.21671#bib.bib5)\)\) for uncertainty quantification\. However, it performs at the weights level and is characterized by an unstructured pruning strategy \(Hanet al\.\([2015](https://arxiv.org/html/2607.21671#bib.bib6)\); Frankle and Carbin \([2019](https://arxiv.org/html/2607.21671#bib.bib4)\)\) or magnitude pruning, which rely more on the value of the weight than the probability to be included in the final model\.

More robust and theoretical frame has been developed, performing model compression at the neuron level and playing on the choice of the prior distribution of model parameters to force some neurons to be silent or completely pruned during training, allowing the model size to decrease progressively during the training \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\)\. But the compression level of the network is sometimes very small compared to the Bayesian lottery framework \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\); Kuhnet al\.\([2026](https://arxiv.org/html/2607.21671#bib.bib5)\)\)\. Also, premature pruning can in some cases result in a loss of accuracy due to nodes removed in earlier stages of the training that could contribute considerably by the end of the training\. These limitations highlight the need of a unified framework, which combines both the ability to compress efficiently by using the Bayesian Lottery Ticket approach, as well as the inclusion of neurons probabilistically, which results in efficient sparse networks\.

The main objective of this thesis is to develop a neural atom prevalence method for neural network model construction as an answer to the question: Can Fokoué’s atom prevalence principle \(Fokoué \([2008](https://arxiv.org/html/2607.21671#bib.bib3)\)\), based on identifying the most likely model size and selecting the top\-ranked atoms, be successfully extended to neural networks under a variational Bayesian framework?\. More precisely, it aims at defining a framework for neural network model construction through neuron selection that achieves a benchmark network compression level, preserving or improving model performance and interpretability\. To achieve this goal, we define the neural atom; Construct the theoretical framework of the neural atom prevalence method; Leverage the performance of Bayesian Lottery ticket and the Spike and Slab Independent Gaussian \(SS\-IG\) prior into a single unified frame for Bayesian neural network model construction; and validate the method empirically\.

The following part of the paper is organized into four sections\. Section[2](https://arxiv.org/html/2607.21671#S2)reviews the related work\. Section[3](https://arxiv.org/html/2607.21671#S3)presents the proposed neural atom prevalence method; Section[4](https://arxiv.org/html/2607.21671#S4)presents the experiments and empirical validation of the method and the final Section[5](https://arxiv.org/html/2607.21671#S5)is the conclusion\.

## 2Related Work

A model that cannot explain which inputs drove its predictions provides limited value to scientists, clinicians, or regulators who must justify decisions\. The goal of feature selection or model selection is therefore not only predictive accuracy but also identifying a sparse or compressed, stable, and interpretable subset of inputs or model components that are really necessary for prediction\. The following paragraphs review the flow of feature selection in Bayesian frame work, from the linear models to neural network models, and then anounce the paper proposed method\.

In the area of feature selection, one of the key contributions in the Bayesian framework, and the direct theoretical foundation of this paper, is the Atom Prevalence Method of FokouéFokoué \([2008](https://arxiv.org/html/2607.21671#bib.bib3)\)\. It came directly upon the limitation of the Median Probability Method \(MP\)Barbieri and Berger \([2004](https://arxiv.org/html/2607.21671#bib.bib32)\), which aims to identify the optimal model among a set of possible models, built on a set of atoms assimilated to features\. The MP method is based on a cut\-off posterior inclusion probabilityp=12p=\\frac\{1\}\{2\}in feature selection, which was successful in many cases but inefficient when no posterior inclusion was equal to or above12\\frac\{1\}\{2\}\. Fokoué solved this problem by introducing an adaptive cut\-off to guarantee the existence of the optimal model\. More specifically, he synthesized the exact sparsity of Mitchell and Beauchamp with the computational tractability of SSVS by using birth\-and\-death MCMC, a continuous\-time Markov process that adds \(birth\) or removes \(death\) variables from the active set without discrete accept/reject steps\. Under the full hierarchical model:

βi∣λi\\displaystyle\\beta\_\{i\}\\mid\\lambda\_\{i\}∼𝒩​\(βi;0,λi−1\),λi∣a,b∼Gamma​\(λi;a,b\),k∼Poisson​\(ω\)⋅𝟏​\[1≤k≤m\],\\displaystyle\\sim\\mathcal\{N\}\(\\beta\_\{i\};\\,0,\\,\\lambda\_\{i\}^\{\-1\}\),\\quad\\lambda\_\{i\}\\mid a,b\\sim\\text\{Gamma\}\(\\lambda\_\{i\};\\,a,b\),\\quad k\\sim\\text\{Poisson\}\(\\omega\)\\cdot\\mathbf\{1\}\[1\\leq k\\leq m\],vi∣π\\displaystyle v\_\{i\}\\mid\\pi∼Bernoulli​\(π\),π∼Beta​\(aπ,bπ\),σ−2∣c,d∼Gamma​\(σ−2;c,d\)\\displaystyle\\sim\\text\{Bernoulli\}\(\\pi\),\\quad\\pi\\sim\\text\{Beta\}\(a\_\{\\pi\},b\_\{\\pi\}\),\\quad\\sigma^\{\-2\}\\mid c,d\\sim\\text\{Gamma\}\(\\sigma^\{\-2\};\\,c,d\)the birth rate for an inactive variablejjgiven current active setAAis proportional to the Bayes factorp​\(𝐲∣A∪\{j\}\)/p​\(𝐲∣A\)p\(\\mathbf\{y\}\\mid A\\cup\\\{j\\\}\)/p\(\\mathbf\{y\}\\mid A\), and the death rate proportionally to its reciprocal\.

From the resulting samples, three key quantities are computed: \(1\) the inclusion probabilityp^j=Pr⁡\(vj=1∣𝐲\)\\hat\{p\}\_\{j\}=\\Pr\(v\_\{j\}=1\\mid\\mathbf\{y\}\)for each predictor; \(2\) the optimal model sizekopt=arg⁡maxk⁡Pr⁡\(\|A\|=k∣𝐲\)k^\{\\mathrm\{opt\}\}=\\arg\\max\_\{k\}\\Pr\(\|A\|=k\\mid\\mathbf\{y\}\), used find the adaptive cut off; and \(3\) the selected model, comprising thekoptk^\{\\mathrm\{opt\}\}predictors with highestp^j\\hat\{p\}\_\{j\}\. Atom Prevalence thereby delivers exact Bayesian sparsity, full uncertainty quantification, and scalability topp, addressing the limitations of the MP model and three preceding approaches\.

The above method is discussed for linear models \(The atoms are linear predictorsxjx\_\{j\}\), putting aside the large family of non\-linear models, among which are neural networks\. In a neural network, the natural atoms are neuron activationshj​\(𝐱\)h\_\{j\}\(\\mathbf\{x\}\), the non\-linear, learned representations\. Extending Atom Prevalence from predictors to neurons is the central contribution of this thesis\.

Neural networks violate the premise of linear variable selection: there is no fixed, interpretable set of features to select, and the parameter count vastly exceeds the sample size\. The canonical response to over\-parameterisation has been pruning, the removal of redundant weights or neurons after training\. LeCun, Denker, and Solla \(LeCunet al\.\([1989](https://arxiv.org/html/2607.21671#bib.bib19)\)\) first formalised this via a second\-order Taylor expansion of the training loss, removing weights whose removal caused the smallest increase in loss \(Optimal Brain Damage\)\. Han et al\. \(Hanet al\.\([2015](https://arxiv.org/html/2607.21671#bib.bib6)\)\) demonstrated the same principle at scale: magnitude pruning alone removes over 90% of weights from large convolutional networks without accuracy loss\. These results empirically establish that trained networks are massively over\-parameterised, but they provide no probabilistic characterization of which weights or neurons are structurally necessary\.

Frankle and Carlin \(Frankle and Carbin \([2019](https://arxiv.org/html/2607.21671#bib.bib4)\)\) transformed the pruning narrative with the Lottery Ticket Hypothesis \(LTH\): a randomly\-initialised dense network contains a sparse subnetwork \(a winning ticket\) that, when trained in isolation from the original initialisationθ0\\theta\_\{0\}, matches the full network’s accuracy in at most the same number of iterations\. The identification procedure is iterative magnitude pruning with weight resetting:

1. 1\.Randomly initialisef​\(𝐱;θ0\)f\(\\mathbf\{x\};\\,\\theta\_\{0\}\)and train to convergence, obtainingθT\\theta\_\{T\}\.
2. 2\.Prune thep%p\\%smallest\-magnitude weights, creating maskmm\.
3. 3\.Reset surviving weights toθ0\\theta\_\{0\}\(notθT\\theta\_\{T\}\)\.
4. 4\.Repeat until target sparsity is reached\.

The decisive control experiment is random reinitialization: the winning ticket’s maskmmapplied to a fresh random initialisationθ0′\\theta\_\{0\}^\{\\prime\}performs no better than random pruning, proving that the specific initial valuesm⊙θ0m\\odot\\theta\_\{0\}\(not merely the sparse architecture\) constitute the ticket\.

The LTH has two implications for this thesis\. First, it establishes that sparse subnetworks with specific structural properties exist a priori, a finding that is structurally analogous to the spike\-and\-slab assertion that a sparse subset of predictors is active\. Second, it shows that which neurons survive matters, not just how many\. Moving from a frequentist to probabilistic frame, Kuhn et al\. \(Kuhnet al\.\([2026](https://arxiv.org/html/2607.21671#bib.bib5)\)\) extend the LTH into the Bayesian framework by placing priors on weight\-level mask variables and using variational inference to approximate the posterior over active weights\. However, ifFrankle and Carbin \([2019](https://arxiv.org/html/2607.21671#bib.bib4)\)confirms that winning tickets generalise across architectures,Rachwanet al\.\([2022](https://arxiv.org/html/2607.21671#bib.bib8)\)acknowledges that the procedure is expensive and provides no principled criterion for choosing the sparsity level\. Moreover, this method was performed only at the weights level, not at the node level, and removing a neuron eliminates an entire learned feature representation\.

In Bayesian neural network, exact posterior inference in Bayesian neural networks is intractable\. However, two broad families of approximation have emerged to handle this\. Markov Chain Monte Carlo \(MCMC\) methods, pioneered by Neal \(Neal \([1996](https://arxiv.org/html/2607.21671#bib.bib21)\)\), draw samples from the true posterior and are asymptotically exact, but their computational cost scales poorly with network size, making them impractical for modern architectures\. Variational Inference \(VI\) offers a more scalable alternative: it reframes posterior computation as an optimisation problem, searching within a tractable family of distributions for the member closest to the true posterior in KL divergence\. Among variational families, the Mean\-Field family is the most widely adopted \(Xinget al\.\([2012](https://arxiv.org/html/2607.21671#bib.bib31)\)\)\.

Bayesian neural networks \(BNNs\) place prior distributions over weights and use Bayes’ theorem to compute posterior weight distributions, providing uncertainty quantification absent from standard networks\. Neal \(Neal \([1996](https://arxiv.org/html/2607.21671#bib.bib21)\)\) established the theoretical foundation but noted that exact posterior inference via MCMC is intractable for networks of practical size\. Modern BNNs circumvent this through variational inference: Blundell et al\. \(Blundellet al\.\([2015](https://arxiv.org/html/2607.21671#bib.bib10)\)\) introduced Bayes by Backprop, which minimises the KL divergence between a factorised Gaussian variational family and the true posterior\. The practical accessibility of BNNs was further broadened by Gal and Ghahramani \(Gal and Ghahramani \([2016](https://arxiv.org/html/2607.21671#bib.bib22)\)\), who proved that dropout training \(already standard practice\) is mathematically equivalent to approximate Bayesian inference in a deep Gaussian process\.

A limitation unites all these approaches: they use continuous weight priors\. Consequently, the posterior probability that any specific weight is exactly zero is zero\. Continuous priors cannot provide genuine sparsity\. Molchanov et al\. \(Molchanovet al\.\([2017](https://arxiv.org/html/2607.21671#bib.bib23)\)\) made partial progress by showing that variational dropout with an unbounded dropout rate naturally shrinks some weights toward zero, achieving effective sparsity\. However, the resulting zeros are approximate, and there is no principled mapping from dropout rates to posterior inclusion probabilities for individual neurons\.

The above challenges were also stated byMlambo \([2024](https://arxiv.org/html/2607.21671#bib.bib24)\), who highlighted that mean\-field variational methods don’t scale to deep architectures\. However,Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)developed a method like SS\-Independent Gaussian using a mean\-field variational family, applied at the neuron level, with demonstrated performance and consistency\. But no existing method uses Atom Prevalence as the generative framework for neural feature selection\.

This paper proposes a framework that bridges the gap identified at the end of the previous paragraph\. It treats each neuron as an atom in the sense ofFokoué \([2008](https://arxiv.org/html/2607.21671#bib.bib3)\), assigns it a posterior inclusion probability through a spike\-and\-slab variational posterior, and selects the final architecture by identifying the most likely layer size and retaining the most prevalent neurons\. Concretely, the spike\-and\-slab Independent Gaussian prior ofJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)provides the variational machinery for computing neuron\-level inclusion probabilities within a feedforward network, while the Bayesian Lottery Ticket initialises the procedure from a compressed, well\-conditioned subnetwork rather than a redundant dense one\. The Poisson\-Binomial distribution then replaces the fixed cut\-off of the Median Probability model with a data\-driven optimal layer size, mirroring the adaptive threshold that Fokoué introduced for linear atoms\. The result is a framework that simultaneously inherits the exact sparsity of spike\-and\-slab priors, the compression efficiency of iterative magnitude pruning, and the probabilistic selection logic of Atom Prevalence, extended for the first time to the nonlinear, layered representations of deep networks\.

## 3Proposed Method: Neural Atom Prevalence

The proposed method can be categorized as neuron model selection procedure in Bayesian Neural Network, based on the selection of neurons to form a BNN\. The principle or skeleton is similar to the Atom prevalence of Fokoué \(Fokoué \([2008](https://arxiv.org/html/2607.21671#bib.bib3)\)\) but applied in deep learning and with different ingredients\. The core idea of the method is to compress or reduce the size of a large network architecture while preserving the stability, the interpretability, and the performance in terms of accuracy\. The entire body of this chapter is dedicated to the construction of the method, covering the problem setup and notation, the theoretical development of the neural atom prevalence method, the posterior approximation, the neural atom selection rule, and finally, the neural atom prevalence method pipeline and the algorithm\. The structure of the chapter will follow the order of the above steps\.

### 3\.1Problem setup and notation

Consider the machine learning task consisting of building a model that explains a relationship between an output variableYYand a feature vectorX=\(x1,…​xd\)X=\(x\_\{1\},\.\.\.x\_\{d\}\)\(ddis the feature dimension\) using a data setD=\{\(X1,Y1\),…,\(Xn,Yn\)\}D=\\\{\(X\_\{1\},Y\_\{1\}\),\\dots,\(X\_\{n\},Y\_\{n\}\)\\\}\(nnis the dataset size\), observations which are drawn from the joint distributionPX​YP\_\{XY\}of\(X,Y\)\(X,Y\)\. This means finding the best approximation functionηθ\\eta\_\{\\theta\}\(a function depending on parametersθ\\theta\) of the true functionη0\\eta\_\{0\}mapping the featureXXto the outputYY\. In this work, the general form of the true model \(true functions\) is defined as follows:

Yi=ηo​\(Xi\)\+ϵi,i=1,…,nY\_\{i\}=\\eta\_\{o\}\(X\_\{i\}\)\+\\epsilon\_\{i\},\\quad i=1,\\dots,n\(1\)
whereϵi∼𝒩​\(0,σe2\)\\epsilon\_\{i\}\\sim\\mathcal\{N\}\(0,\\sigma\_\{e\}^\{2\}\)andηo​\(⋅\):ℝd⟶ℝ\\eta\_\{o\}\(\\cdot\):\\mathbb\{R\}^\{d\}\\longrightarrow\\mathbb\{R\}\. With this configuration, the conditional distribution ofYYgivenXXunder this true model is \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\):

fo​\(Y\|X=x\)=12​π​σe2​exp⁡\(−\(y−ηo​\(x\)\)22​σe2\)f\_\{o\}\(Y\|X=x\)=\\frac\{1\}\{\\sqrt\{2\\pi\\sigma\_\{e\}^\{2\}\}\}\\exp\\left\(\-\\frac\{\(y\-\\eta\_\{o\}\(x\)\)^\{2\}\}\{2\\sigma\_\{e\}^\{2\}\}\\right\)
Since we are working in the neural networks space, this work aims to construct the neural network modelηθ\\eta\_\{\\theta\}that gives the best approximationfθf\_\{\\theta\}of the functionf0f\_\{0\}where:

fθ​\(Y\|X=x\)=12​π​σe2​exp⁡\(−\(y−ηθ​\(x\)\)22​σe2\)f\_\{\\theta\}\(Y\|X=x\)=\\frac\{1\}\{\\sqrt\{2\\pi\\sigma\_\{e\}^\{2\}\}\}\\exp\\left\(\-\\frac\{\(y\-\\eta\_\{\\theta\}\(x\)\)^\{2\}\}\{2\\sigma\_\{e\}^\{2\}\}\\right\)and which is sparse, confident and interpretable\.

With the assumption of independence between the observations, the likelihood of the data under the model is:

Pθn=∏i=1nfθ​\(yi\|xi\)P\_\{\\theta\}^\{n\}=\\prod\_\{i=1\}^\{n\}f\_\{\\theta\}\(y\_\{i\}\|x\_\{i\}\)\(2\)
For high computational issues, as suggested by Jantre et al\. \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\), the following assumptions will be considered:PX=U​\[0,1\]P\_\{X\}=U\[0,1\],σe2=1\\sigma\_\{e\}^\{2\}=1, and that all activation functionsψl\\psi\_\{l\}in the work are 1\-Lipschitz continuous\.

#### 3\.1\.1Neural Network architecture

A large part of research on neural networks has focused on the weights, considering them as the basic unit for the composition of these architectures, and this is probably due to the importance or the place of the weights for traditional machine learning, like linear regression\. In the field of model size reduction, a lot of work has been carried out at the weight level, achieving important and very useful results\. However, the price of these achievements is heavy, especially in the field of deep learning, where the weight space dimension grows extremely faster as the network becomes deeper\. Shifting from weight space to neuron space can consistently reduce the dimensionality of the problem \(moving from dimension D = 27 in weight space to D=10 in the neuron space for a small\-sized network\)\. Since we are doing a research, we will first construct the method for the basis neural architecture, the Feedforward Neural Network \(FNN\)\. In that case, all the chapters will then be based on how to select neurons in a large FNN to achieve good dimension reduction \(or network compression\), model interpretability, level of stability and also a good accuracy\. Let define mathematical the FFNN:

let consider a FNN withLLhidden layers, each whichKlK\_\{l\}\(l∈\{1,2,…,L\}l\\in\\\{1,2,\.\.\.,L\\\}\) neurons, an input layer withK0K\_\{0\}input variables \(X=xi,i∈\{1,2,…,d\}X=\{x\_\{i\},i\\in\\\{1,2,\.\.\.,d\\\}\}\) and finally an output layer withKL\+1K\_\{L\+1\}dependent variables \(yi,i∈\{1,2,…,KL\+1\}\{y\_\{i\},i\\in\\\{1,2,\.\.\.,K\_\{L\+1\}\\\}\}\)\. Let denotes byWl=\(wi​jl,1≤i≤\{1,2,…,Kl\},j∈\{1,2,…,Kl\+1\}\)W^\{l\}=\(w^\{l\}\_\{ij\},1\\leq i\\leq\\\{1,2,\.\.\.,K\_\{l\}\\\},j\\in\\\{1,2,\.\.\.,K\_\{l\+1\}\\\}\)for all0≤l≤L0\\leq l\\leq LandBl=\(b1l,…,bKl\+1l\)B^\{l\}=\(b^\{l\}\_\{1\},\.\.\.,b^\{l\}\_\{K\_\{l\+1\}\}\), respectively the matrix of weights and the vector of bias connecting layerllto layerl\+1l\+1\. Therefore the general form of the neural network function is given by the following equation:

ηθ​\(x\)=ψL\+1​\(BL\+WL​ψL​\(BL−1\+WL−1​ψL−1​\(…​\(B0\+W0​x\)\)\)\)\\eta\_\{\\theta\}\(x\)=\\psi\_\{L\+1\}\(B^\{L\}\+W^\{L\}\\psi\_\{L\}\(B^\{L\-1\}\+W^\{L\-1\}\\psi\_\{L\-1\}\(\\dots\(B^\{0\}\+W^\{0\}x\)\)\)\)
where the parameters isθ=\{Wl,Bl\}l=0L\\theta=\\\{W^\{l\},B^\{l\}\\\}\_\{l=0\}^\{L\}andψl\\psi\_\{l\}\(l=1,…,L\+1l=1,\\dots,L\+1\) the activation function in layerll\.

#### 3\.1\.2Neural atoms

The construction ofηθ\\eta\_\{\\theta\}while operating at the neuron or node level requires mathematically defining what a neuron is\. Operating at the neuron level implies considering neurons as the basic components of the neural model, the neural atom\. So, defining mathematically what a neuron is, is defining what a neural atom is\.

###### Definition 1\(Neural atom\)\.

A neural atom or an activation unit is the functionhjlh^\{l\}\_\{j\}defined for all1≤l≤L1\\leq l\\leq Land1≤j≤Kl1\\leq j\\leq K\_\{l\}by:

hjl​\(hl−1,Wjl−1,bl−1\)=\{ψ1​\(∑idwi​j0​xi\+b0\)if​l=1ψl​\(gl​j\)if​2≤l≤Lh^\{l\}\_\{j\}\(h^\{l\-1\},W^\{l\-1\}\_\{j\},b\_\{l\-1\}\)=\\begin\{cases\}\\psi\_\{1\}\(\\sum^\{d\}\_\{i\}w^\{0\}\_\{ij\}x\_\{i\}\+b\_\{0\}\)&\\text\{if \}l=1\\\\ \\psi\_\{l\}\(g\_\{lj\}\)&\\text\{if \}2\\leq l\\leq L\\end\{cases\}\(3\)where:

- •hl−1=\(h1l−1,…,hKl−1l−1\)h^\{l\-1\}=\(h^\{l\-1\}\_\{1\},\.\.\.,h^\{l\-1\}\_\{K\_\{l\-1\}\}\), the neural atoms of layerl−1l\-1;
- •Wjl−1=\(w1​jl−1,…,wKl−1​jl−1\)W^\{l\-1\}\_\{j\}=\(w^\{l\-1\}\_\{1j\},\.\.\.,w^\{l\-1\}\_\{K\_\{l\-1\}j\}\)the vector of weights incident to neural atomhjlh^\{l\}\_\{j\}so thatwi​jlw^\{l\}\_\{ij\}can be represented by\(hil−1,hjl\)\\left\(h\_\{i\}^\{l\-1\},h\_\{j\}^\{l\}\\right\);
- •ψl\\psi\_\{l\}is an activation function;
- •gl​jg\_\{lj\}is an preactivation unit define by: gl​j=Wjl−1×hl−1\+bl−1=∑iKl−1wi​jl−1​hil−1\+bl−1g\_\{lj\}=W^\{l\-1\}\_\{j\}\\times h^\{l\-1\}\+b\_\{l\-1\}=\\sum^\{K\_\{l\-1\}\}\_\{i\}w^\{l\-1\}\_\{ij\}h^\{l\-1\}\_\{i\}\+b\_\{l\-1\}\(4\)

With the above definition, it can be derived that the reduced model will be a function of some of the neural atoms, selected from the set𝒜=\{hjl,1≤j≤Kl,1≤l≤L\}\\mathcal\{A\}=\\\{h^\{l\}\_\{j\},\\quad 1\\leq j\\leq K\_\{l\},\\quad 1\\leq l\\leq L\\\}of all atoms\. So the Neural Atom Prevalence method aims to select the most prevalent neural atoms to constructηθ\\eta\_\{\\theta\}\.

Before moving, let’s do the following remark[1](https://arxiv.org/html/2607.21671#Thmremark1):

Since this research aims to develop a framework that starts from a dense neural network and selects a subset of neurons to form a smaller subnetwork, the remark[1](https://arxiv.org/html/2607.21671#Thmremark1)will guide us for the next section on the presentation of the NAP generative model\.

### 3\.2Theoretical development of the Neural Atom Prevalence Method: the NAP Generative Model

This method is built in full Bayesian\(for uncertainty quantification\) upon the work of Jantre et al\. \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\) and Fokoué \(Fokoué \([2008](https://arxiv.org/html/2607.21671#bib.bib3)\)\), which introduced the spike\-and\-slab prior for model parameter selection in multi\-linear regression models\. It is improved here for node selection by assigning a group prior to all weights incoming to a node, as inJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\. The rest of this section will present the full probabilistic model ofηθ\\eta\_\{\\theta\}before observing the data, which depends on two groups of parameters,θ\\thetaand the neuron indicatorVV, which will be defined below\.

#### 3\.2\.1Prior for Neural atom inclusion, model selection probability and model size distribution

A priori from the original atom paper, atom inclusion follows a Bernoulli distribution of parameterπi\\pi\_\{i\}, independently from atomj≠ij\\neq i\. The naive use of this cannot work here since the probability of a neuron in layerllis zero if all the neurons of the previous layer have been excluded\. Therefore, this remark[1](https://arxiv.org/html/2607.21671#Thmremark1)justifies the choice of this work to borrow the concept of adaptive layer selection fromJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\), by treating the hidden layer independently based on the fact that this layer will exist in the final model architecture\. In this consideration, the number of possible models is then:

N=∏l=1L\(2Kl−1\)N=\\prod\_\{l=1\}^\{L\}\(2^\{K\_\{l\}\}\-1\)

#### 3\.2\.2model selection probability

For every neuronhjlh^\{l\}\_\{j\}let denote byvjlv^\{l\}\_\{j\}\(vjl∈\{0,1\}\)v^\{l\}\_\{j\}\\in\\\{0,1\\\}\)the variable indicating the selection status ofhjlh^\{l\}\_\{j\}\. As inFokoué \([2008](https://arxiv.org/html/2607.21671#bib.bib3)\); Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)let consider that apriori allvjlv^\{l\}\_\{j\}independently follow Bernoulli distribution of parameterπl\\pi^\{l\}:

vjl∼Bernoulli​\(πl\),1≤j≤Kl1≤l≤Lv^\{l\}\_\{j\}\\sim\\text\{Bernoulli\}\(\\pi^\{l\}\),\\quad 1\\leq j\\leq K\_\{l\}\\quad 1\\leq l\\leq L
Then, for each layerl,1≤l≤Ll,\\quad 1\\leq l\\leq L:

Vl=\(v1l,…,vKll\)∼∏j=1KlBernoulli​\(πl\)V^\{l\}=\(v\_\{1\}^\{l\},\\dots,v\_\{K\_\{l\}\}^\{l\}\)\\sim\\prod\_\{j=1\}^\{K\_\{l\}\}\\text\{Bernoulli\}\(\\pi^\{l\}\)
such that for a modelMVM\_\{V\}represented by the vectorV=\(V1,…​VL\)V=\(V^\{1\},\.\.\.V^\{L\}\)indicating for each layer which nodes are selected to be part ofηθ\\eta\_\{\\theta\}, the priorP​\(V\)P\(V\)is given by:

P​\(V\)=∏k=1LP​\(Vl\)w​h​e​r​eVl∼∏i=1KlBernoulli​\(πl\)P\(V\)=\\prod\_\{k=1\}^\{L\}P\(V^\{l\}\)\\quad where\\quad V^\{l\}\\sim\\prod\_\{i=1\}^\{K\_\{l\}\}\\text\{Bernoulli\}\(\\pi^\{l\}\)
This is the prior probability for a given model \(subset of hidden neuron atoms\) to be selected\. More explicitly, from the aforementioned theorem[1](https://arxiv.org/html/2607.21671#Thmremark1)and the independence, we have that:

P​\(Vl\)\\displaystyle P\(V^\{l\}\)=∏j=1Kl\(πl\)vjl​\(1−πl\)1−vjl\\displaystyle=\\prod^\{K\_\{l\}\}\_\{j=1\}\(\\pi^\{l\}\)^\{v^\{l\}\_\{j\}\}\(1\-\\pi^\{l\}\)^\{1\-\{v^\{l\}\_\{j\}\}\}=\(πl\)∑j=1Klvjl​\(1−πl\)Kl−∑j=1Klvjl\\displaystyle=\(\\pi^\{l\}\)^\{\\sum^\{K\_\{l\}\}\_\{j=1\}\{v^\{l\}\_\{j\}\}\}\(1\-\\pi^\{l\}\)^\{K\_\{l\}\-\\sum^\{K\_\{l\}\}\_\{j=1\}\{v^\{l\}\_\{j\}\}\}such that the the detailed expression ofP​\(V\)P\(V\)is given by:

P​\(V\)=∏l=1L\(πl\)∑j=1Klvjl​\(1−πl\)Kl−∑j=1KlvjlP\(V\)=\\prod^\{L\}\_\{l=1\}\(\\pi^\{l\}\)^\{\\sum^\{K\_\{l\}\}\_\{j=1\}\{v^\{l\}\_\{j\}\}\}\(1\-\\pi^\{l\}\)^\{K\_\{l\}\-\\sum^\{K\_\{l\}\}\_\{j=1\}\{v^\{l\}\_\{j\}\}\}\(5\)
In the next section, the parameterπ=\(πl,\.\.,πL\)\\pi=\(\\pi^\{l\},\.\.,\\pi^\{L\}\)will be determined by the theoretical calibration formula derived inJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\.

#### 3\.2\.3Model Size Prior

Given that the method proposed assumes that the input layer and output layer are fixed from the beginning \(neural atoms are hidden nodes\), then the model size here is essentially determined by the size of the hidden layers\. So for a modelMVM\_\{V\}it size is given by the vectorKV=\(KV1,…,KVL\)K\_\{V\}=\(K\_\{V^\{1\}\},\.\.\.,K\_\{V^\{L\}\}\)where for1≤l≤L1\\leq l\\leq L,KVl=‖Vl‖0K\_\{V^\{l\}\}=\\\|V^\{l\}\\\|\_\{0\}and‖Vl‖0\\\|V^\{l\}\\\|\_\{0\}is the number of non\-zero components ofVlV^\{l\}, the number of atoms in layerll\.Under the prior defined in section[3\.2\.2](https://arxiv.org/html/2607.21671#S3.SS2.SSS2), we then have thatKVl=∑j=1KlvjlK\_\{V^\{l\}\}=\\sum^\{K\_\{l\}\}\_\{j=1\}v^\{l\}\_\{j\}is the sum of independent Bernoulli distributions with the same parameterπl\\pi^\{l\}, which is clearly easy to determine due to the common parameter for all nodes in that layer\.KVlK\_\{V^\{l\}\}Consequently, follow a Binomial Distribution of parameter\(Kl,πl\)\(K\_\{l\},\\pi^\{l\}\), so the explicit probability for a layer l to have size k is given by the following equation:

P​\(KVl=k\)=\(Klk\)​\(πl\)k​\(1−πl\)Kl−kP\(K\_\{V^\{l\}\}=k\)=\\binom\{K\_\{l\}\}\{k\}\(\{\\pi^\{l\}\}\)^\{k\}\(1\-\{\\pi^\{l\}\}\)^\{K\_\{l\}\-k\}\(6\)This is the prior distribution of the layer size for each hidden layer in the network, and its posterior, which will be used by NAP for atom selection, is the Poisson Binomial distribution, where each atomhjlh^\{l\}\_\{j\}has its own inclusion probabilityπ~jl\\tilde\{\\pi\}^\{l\}\_\{j\}, which will be defined in the following sections\.

#### 3\.2\.4Neural Atom Prior with Spike\-and\-Slab: Weights and Bias

This section is largely based on the work of Jantre et al\. \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\)\. In their work, the neural atom is treated via a latent variable\-based prior and it prior distribution in this work is the grouped distribution ofW¯l​j=\(Wjl−1,bl−1\)=\(w1​jl−1,…,wKl−1​jl−1,bl−1\)\\overline\{W\}\_\{lj\}=\(W^\{l\-1\}\_\{j\},b\_\{l\-1\}\)=\(w^\{l\-1\}\_\{1j\},\.\.\.,w^\{l\-1\}\_\{K\_\{l\-1\}j\},b\_\{l\-1\}\)\. This grouped distribution is the Spike and Slab distribution, where the spike is a Dirac spike, represented byvjlv^\{l\}\_\{j\}\. The slab is the standard Gaussian \(centered atμ0=0\\mu\_\{0\}=0with varianceσ02=1\\sigma^\{2\}\_\{0\}=1\) distributed random variable\. More formally, for an atomhjlh\_\{j\}^\{l\}, we have that:

W¯l​j\|vjl∼\(1−vjl\)​δ0\+vjl​𝒩​\(0,σ02​I\)\\overline\{W\}\_\{lj\}\|v\_\{j\}^\{l\}\\sim\(1\-v\_\{j\}^\{l\}\)\\delta\_\{0\}\+v\_\{j\}^\{l\}\\mathcal\{N\}\(0,\\sigma\_\{0\}^\{2\}I\)\(7\)whereW¯l​j\\overline\{W\}\_\{lj\}is the weight and bias vector that connects the layerl−1l\-1to the nodejjin layerll\.δ0\\delta\_\{0\}denotes the Dirac measure concentrated at zero, applied component\-wise to each element ofW¯l​j\\overline\{W\}\_\{lj\}\. From the independence of components ofW¯l​j\\overline\{W\}\_\{lj\}and the Gaussian Slab result, the SS\-IG \(Spike and Slab Independent Gaussian\) prior for neural atoms\.

Whenvjl=0v\_\{j\}^\{l\}=0, the atom is inactive, and all its incoming weights and bias are forced to exactly zero by the Dirac spikeδ0\\delta\_\{0\}\. Whenvjl=1v\_\{j\}^\{l\}=1, the neuron is active, and its weights are given a centred Gaussian slab with varianceσ02\\sigma\_\{0\}^\{2\}, encoding the prior belief that active weights are non\-negligible but not systematically large\.

Finally, under the independence assumption across atoms, the full weight prior is:

P​\(θ\)=∑VP​\(θ,V\)=∑VP​\(θ\|V\)​P​\(V\)=∑V∏l=1L∏j=1KlP​\(W¯l​j\|vjl\)​P​\(vjl\)P\(\\theta\)=\\sum\_\{V\}P\(\\theta,V\)=\\sum\_\{V\}P\(\\theta\|V\)P\(V\)=\\sum\_\{V\}\\prod\_\{l=1\}^\{L\}\\prod^\{K\_\{l\}\}\_\{j=1\}P\(\\overline\{W\}\_\{lj\}\|v\_\{j\}^\{l\}\)P\(v\_\{j\}^\{l\}\)\(8\)
And then for the full joint generative model, the full joint distribution of \(X, Y\) is:

PX​Y=∑θ,VP​\(X,Y\|θ,V\)=∑θ,VP​\(Y\|X,θ,V\)​P​\(θ\|V\)​P​\(V\)P\_\{XY\}=\\sum\_\{\\theta,V\}P\(X,Y\|\\theta,V\)=\\sum\_\{\\theta,V\}P\(Y\|X,\\theta,V\)P\(\\theta\|V\)P\(V\)\(9\)
We finally fully specify the NAP generative model through the joint priorP​\(θ,V\)P\(\\theta,V\), which is composed by two main components, theP​\(V\)P\(V\), indicating the prior beliefs about sparsity, derived from independent Bernoulli andP​\(θ\|V\)P\(\\theta\|V\)the prior of weights\. The next section of this work will focus on deriving the posteriorP​\(θ,V\|D\)P\(\\theta,V\|D\)through Variational Inference due to its tractability compared to MCMC and also as recommended byJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\.

### 3\.3Posterior approximation via Variational Inference

Since the work is carried out at the node level, but driven by the weight distribution, the posterior of an activation unithjlh\_\{j\}^\{l\}is given by the marginal joint posterior of\(W¯l​j,vjl\)\(\\overline\{W\}\_\{lj\},v\_\{j\}^\{l\}\)\. This is considered so that posterior of all activation units are driven by the posterior of all the weights, biasW¯\\overline\{W\}andVV,P​\(θ,V∣D\)P\(\\theta,V\\mid D\), whereθ\\thetais the set of parameters andVVis the whole model indicator\. This posterior has been derived by Jantre et al\.Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)and is:

P​\(θ,V∣D\)=Pθn​P​\(θ∣V\)​P​\(V\)∑V∫Pθn​P​\(θ∣V\)​P​\(V\)​𝑑θP\(\\theta,V\\mid D\)=\\frac\{P\_\{\\theta\}^\{n\}\\;P\(\\theta\\mid V\)\\;P\(V\)\}\{\\displaystyle\\sum\_\{V\}\\int P\_\{\\theta\}^\{n\}\\;P\(\\theta\\mid V\)\\;P\(V\)\\;d\\theta\}
As is widely known in BNN, this posterior is not tractable, especially in deep architectures\. To move forward with this, we will apply variational inference to choose a posterior distribution close to the true posterior and that is more tractable\. The method is based on identifying a posterior family calling variational family, especially the Mean\-Field\-Family \(Xinget al\.\([2012](https://arxiv.org/html/2607.21671#bib.bib31)\)\), which will serve as a space for running an optimization algorithm to choose the variational distributionq∗q^\{\*\}that minimizedK​L\(q,P\(\.\|D\)\)d\_\{KL\}\(q,P\(\.\|D\)\), which denotes the KL\-distance between q and the true posteriorP\(\.\|D\)P\(\.\|D\)\.

#### 3\.3\.1Variational Family Method for Posterior: Mean Field family\.

The variational family considered in this study is an adapted version of the Mean Field family of Jantre et al\. \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\), adjusted to take into account the variational posterior of the model size distribution\. This is defined as follows:

QM​F=\{W¯l​j∣vjl​∼i\.d​\[\(1−vjl\)​δ0\+vjl​𝒩​\(μjl,diag​\(\(σjl\)2\)\)\],KV∼∑V∈\{V,‖v‖0=KV\}q​\(V\),vjl​∼i\.d​Ber​\(πj~l\)\}Q^\{MF\}=\\left\\\{\\overline\{W\}\_\{lj\}\\mid v\_\{j\}^\{l\}\\overset\{i\.d\}\{\\sim\}\\left\[\(1\-v\_\{j\}^\{l\}\)\\,\\delta\_\{0\}\+v\_\{j\}^\{l\}\\;\\mathcal\{N\}\\\!\\left\(\\mu\_\{j\}^\{l\},\\,\\mathrm\{diag\}\\\!\\left\(\(\\sigma^\{l\}\_\{j\}\)^\{2\}\\right\)\\right\)\\right\],\\;K\_\{V\}\\sim\\sum\_\{V\\in\\\{V,\\;\\\|v\\\|\_\{0\}=K\_\{V\}\\\}\}q\(V\),\\;v\_\{j\}^\{l\}\\overset\{i\.d\}\{\\sim\}\\mathrm\{Ber\}\(\\tilde\{\\pi\_\{j\}\}^\{l\}\)\\right\\\}\(10\)
whereμjl=\(μ1​jl,…,μKl−1​jl,μKl−1\+1​jl\)\\mu\_\{j\}^\{l\}=\\left\(\\mu\_\{1j\}^\{l\},\\ldots,\\mu\_\{K\_\{l\-1\}j\}^\{l\},\\mu\_\{K\_\{l\-1\}\+1j\}^\{l\}\\right\)and\(σjl\)2=\{\(σ1​jl\)2,…​\(σKl−1​jl\)2,\(σKl−1\+1​jl\)2\}\\left\(\\sigma\_\{j\}^\{l\}\\right\)^\{2\}=\\left\\\{\\left\(\\sigma\_\{1j\}^\{l\}\\right\)^\{2\},\\ldots\\left\(\\sigma\_\{K\_\{l\-1\}j\}^\{l\}\\right\)^\{2\},\\left\(\\sigma\_\{K\_\{l\-1\}\+1j\}^\{l\}\\right\)^\{2\}\\right\\\}are respectively the vectors of variational mean and standard deviation parameters of weights and biases incident on nodejjin layerll\.

Given all this, the approximated posterior is given by:

q∗​\(θ,V\)\\displaystyle q^\{\*\}\(\\theta,V\)=arg⁡minq∈QM​F​∑V∫\[log⁡q​\(θ,V\)−log⁡P​\(θ,V∣D\)\]​q​\(θ,V\)​𝑑θ\\displaystyle=\\underset\{q\\in Q^\{MF\}\}\{\\arg\\min\}\\sum\_\{V\}\\int\\left\[\\log q\(\\theta,V\)\-\\log P\(\\theta,V\\mid D\)\\right\]q\(\\theta,V\)\\,d\\theta=arg⁡minq∈QM​F​\(∑V∫\[log⁡q​\(θ,V\)−log⁡P​\(θ,V,D\)\]​q​\(θ,V\)​𝑑θ\+log⁡\(∑V∫Pθn​P​\(θ∣V\)​P​\(V\)​𝑑θ\)\)\\displaystyle=\\underset\{q\\in Q^\{MF\}\}\{\\arg\\min\}\\left\(\\sum\_\{V\}\\int\\left\[\\log q\(\\theta,V\)\-\\log P\(\\theta,V,D\)\\right\]q\(\\theta,V\)\\,d\\theta\+\\log\\\!\\left\(\\sum\_\{V\}\\int P\_\{\\theta\}^\{n\}\\,P\(\\theta\\mid V\)P\(V\)\\,d\\theta\\right\)\\right\)=arg⁡minq∈QM​F\[−ELBO\(q,P\(⋅∣D\)\)\]\+log\(∑V∫PθnP\(θ∣V\)P\(V\)dθ\)\\displaystyle=\\underset\{q\\in Q^\{MF\}\}\{\\arg\\min\}\\left\[\-\\mathrm\{ELBO\}\(q,\\,P\(\\cdot\\mid D\)\)\\right\]\+\\log\\\!\\left\(\\sum\_\{V\}\\int P\_\{\\theta\}^\{n\}\\,P\(\\theta\\mid V\)P\(V\)\\,d\\theta\\right\)=arg⁡maxq∈QM​FELBO\(q,P\(⋅∣D\)\)\\displaystyle=\\underset\{q\\in Q^\{MF\}\}\{\\arg\\max\}\\;\\mathrm\{ELBO\}\(q,\\,P\(\\cdot\\mid D\)\)where

ELBO\(q,P\(⋅∣D\)\)=∑V∫\[logP\(θ,V,D\)−log\(q\(θ,V\)\)\]q\(θ,V\)dθ\\mathrm\{ELBO\}\(q,P\(\\cdot\\mid D\)\)=\\sum\_\{V\}\\int\\left\[\\log P\(\\theta,V,D\)\-\\log\(q\(\\theta,V\)\)\\right\]q\(\\theta,V\)\\,d\\theta\(11\)
The resolution of the above optimization problem will produce the marginal variational posterior ofθ\\theta,q∗q^\{\*\}, such that the variational posterior of the activation units isq∗​\(θ∣V\)q^\{\*\}\(\\theta\\mid V\)\. Just as in Jantre et al\. \(2022\) let us denote byQ∗Q^\{\*\}the probability distribution function corresponding toq∗q^\{\*\}, where:

Q∗​\(A\)=∫Aq∗​\(θ\)​𝑑θ=∑V∫Aq∗​\(θ∣V\)​P​\(V\)​𝑑θQ^\{\*\}\(A\)=\\int\_\{A\}q^\{\*\}\(\\theta\)\\,d\\theta=\\sum\_\{V\}\\int\_\{A\}q^\{\*\}\(\\theta\\mid V\)P\(V\)\\,d\\theta\(12\)

#### 3\.3\.2Consistency of the method

This probability distribution function is consistent\. In fact, by allowing different numbers of activation units and different weight norm magnitudes per layer in the network, and by introducing a more flexible sparsity parameter \(different regarding the layer\), Jantre et al\. provide a theoretical proof of consistency of the variational probability distributionQ∗Q^\{\*\}\. More especially, they show thatQ∗Q^\{\*\}converges to the true posteriorP\(⋅∣D\)P\(\\cdot\\mid D\)asnn\(data size\) goes to infinity under the following keys conditions established inJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\):

- •the activation function must be11\-Lipschitz continuous \(satisfied by swish, ReLU, tanh, and sigmoid\);
- •network width and weight magnitudes are controlled and governed respectively by layer\-dependent upper boundsSl=KlS\_\{l\}=K\_\{l\}rather than a uniform node count, accommodating the non\-uniform topologies that arise after pruning and by per\-layerℓ1\\ell\_\{1\}boundsBlB\_\{l\}on incident weights rather than a single global bound, permitting sparse layers with heavier surviving weights;
- •prior inclusion probabilitiesπl\\pi^\{l\}are specified layer\-wise, withπL\+1=1\\pi^\{L\+1\}=1fixed at the output, ensuring optimal node recovery per layer and valid contraction rates\.

Together, these conditions guarantee thatQ∗Q^\{\*\}contracts around the true regression functionη0\\eta\_\{0\}at the minimax\-optimal rate asn→∞n\\to\\infty, directly justifying the consistency claim for the NAP sparse architecture whose topology varies across the IMP, SS\-IG, and fine\-tune phases\.

With this notation \([12](https://arxiv.org/html/2607.21671#S3.E12)\), we have that:

ELBO\(q,P\(⋅\|D\)\)\\displaystyle\\text\{ELBO\}\(q,\\,P\(\\cdot\|D\)\)=∑V−∫\[log⁡q​\(θ,V\)−log⁡P​\(θ,V,D\)\]​q​\(θ,V\)​𝑑θ\\displaystyle=\\sum\_\{V\}\-\\int\\left\[\\log q\(\\theta,V\)\-\\log P\(\\theta,V,D\)\\right\]q\(\\theta,V\)\\,d\\theta=−∑V∫log⁡\[q​\(θ,V\)\]​q​\(θ,V\)​𝑑θ\+∑V∫log⁡\(P​\(θ,V,D\)\)​q​\(θ,V\)​𝑑θ\\displaystyle=\-\\sum\_\{V\}\\int\\log\[q\(\\theta,V\)\]\\,q\(\\theta,V\)\\,d\\theta\+\\sum\_\{V\}\\int\\log\(P\(\\theta,V,D\)\)\\,q\(\\theta,V\)\\,d\\theta=−∑V∫log⁡\(q​\(θ,V\)\)​q​\(θ,V\)​𝑑θ\+∑V∫log⁡\(P​\(D\|θ,V\)​P​\(θ,V\)\)​q​\(θ,V\)​𝑑θ\\displaystyle=\-\\sum\_\{V\}\\int\\log\(q\(\\theta,V\)\)\\,q\(\\theta,V\)\\,d\\theta\+\\sum\_\{V\}\\int\\log\(P\(D\|\\theta,V\)\\,P\(\\theta,V\)\)q\(\\theta,V\)\\,d\\theta=−∑V∫log⁡\(q​\(θ,V\)\)​q​\(θ,V\)​𝑑θ\+∑V∫log⁡\(Pθn\)​q​\(θ,V\)​𝑑θ\+∑V∫log⁡\(P​\(θ,V\)\)​q​\(θ,V\)​𝑑θ\\displaystyle=\-\\sum\_\{V\}\\int\\log\(q\(\\theta,V\)\)\\,q\(\\theta,V\)\\,d\\theta\+\\sum\_\{V\}\\int\\log\\\!\\left\(P\_\{\\theta\}^\{n\}\\right\)q\(\\theta,V\)\\,d\\theta\+\\sum\_\{V\}\\int\\log\(P\(\\theta,V\)\)\\,q\(\\theta,V\)\\,d\\theta=∑V∫log⁡\(Pθn\)​q​\(θ,V\)​𝑑θ−\[−∑V∫log⁡\(P​\(θ,V\)\)​q​\(θ,V\)​𝑑θ\+∑V∫log⁡\(q​\(θ,V\)\)​q​\(θ,V\)​𝑑θ\]\\displaystyle=\\sum\_\{V\}\\int\\log\\\!\\left\(P\_\{\\theta\}^\{n\}\\right\)q\(\\theta,V\)\\,d\\theta\-\\left\[\-\\sum\_\{V\}\\int\\log\(P\(\\theta,V\)\)\\,q\(\\theta,V\)\\,d\\theta\+\\sum\_\{V\}\\int\\log\(q\(\\theta,V\)\)\\,q\(\\theta,V\)\\,d\\theta\\right\]=∑V∫log⁡\(Pθn\)​q​\(θ,V\)​𝑑θ−∑V∫\[log⁡\(q​\(θ,V\)\)−log⁡\(P​\(θ,V\)\)\]​q​\(θ,V\)​𝑑θ\\displaystyle=\\sum\_\{V\}\\int\\log\\\!\\left\(P\_\{\\theta\}^\{n\}\\right\)q\(\\theta,V\)\\,d\\theta\-\\sum\_\{V\}\\int\\left\[\\log\(q\(\\theta,V\)\)\-\\log\(P\(\\theta,V\)\)\\right\]\\,q\(\\theta,V\)\\,d\\theta=𝔼q​\(log⁡\(Pθn\)\)−∑V∫q​\(θ,V\)​log⁡\(q​\(θ,V\)P​\(θ,V\)\)​𝑑θ\\displaystyle=\\mathbb\{E\}\_\{q\}\\\!\\left\(\\log\(P\_\{\\theta\}^\{n\}\)\\right\)\-\\sum\_\{V\}\\int q\(\\theta,V\)\\log\\\!\\left\(\\frac\{q\(\\theta,V\)\}\{P\(\\theta,V\)\}\\right\)d\\theta=𝔼q​\[log⁡Pθn\]−dK​L​\(q,P\)\\displaystyle=\\mathbb\{E\}\_\{q\}\\\!\\left\[\\log P\_\{\\theta\}^\{n\}\\right\]\-d\_\{KL\}\(q\\,,P\)=𝔼q​\(θ\|V\)​q​\(V\)\[logPθn\]−dK​L\(q\(θ\|V\)q\(V\)∥P\(θ\|V\)P\(V\)\)Over all possibleV\\displaystyle=\\mathbb\{E\}\_\{q\(\\theta\|V\)q\(V\)\}\\\!\\left\[\\log P\_\{\\theta\}^\{n\}\\right\]\-d\_\{KL\}\\\!\\left\(q\(\\theta\|V\)q\(V\)\\,\\\|\\,P\(\\theta\|V\)P\(V\)\\right\)\\quad\\text\{Over all possible $V$\}
From chain rule ofdK​Ld\_\{KL\}:

ELBO\(q,P\(⋅\|D\)\)=𝔼q​\(θ\|V\)​q​\(V\)​\[log⁡Pθn\]⏟a−\[dK​L​\(q​\(V\)∥P​\(V\)\)⏟b\+𝔼q​\(V\)\[dK​L\(q\(θ\|V\)∥P\(θ\|V\)\)\]⏟c\]\\text\{ELBO\}\(q,\\,P\(\\cdot\|D\)\)=\\underbrace\{\\mathbb\{E\}\_\{q\(\\theta\|V\)q\(V\)\}\\\!\\left\[\\log P\_\{\\theta\}^\{n\}\\right\]\}\_\{a\}\-\\left\[\\underbrace\{d\_\{KL\}\(q\(V\)\\\|P\(V\)\)\}\_\{b\}\+\\underbrace\{\\mathbb\{E\}\_\{q\(V\)\}\\\!\\left\[d\_\{KL\}\(q\(\\theta\|V\)\\\|P\(\\theta\|V\)\)\\right\]\}\_\{c\}\\right\]
Let’s continue the development by extending first Componentsbbandccof ELBO\. This will let us recall the following formulas: Starting with the variational distribution, we have

q​\(θ\|V\)=∏l=1Lq​\(Wl,Bl\|Vl\)=∏l=1L∏j=1Klq​\(W¯l​j\|vjl\)q\(\\theta\|V\)=\\prod\_\{l=1\}^\{L\}q\(W^\{l\},B^\{l\}\|V^\{l\}\)=\\prod\_\{l=1\}^\{L\}\\prod\_\{j=1\}^\{K\_\{l\}\}q\(\\overline\{W\}\_\{lj\}\\,\|\\,\{v\}\_\{j\}^\{l\}\)q​\(V\)=∏l=1L∏j=1KlBernoulli​\(π~jl\)q\(V\)=\\prod\_\{l=1\}^\{L\}\\prod\_\{j=1\}^\{K\_\{l\}\}\\text\{Bernoulli\}\\\!\\left\(\\tilde\{\\pi\}\_\{j\}^\{l\}\\right\)
AndP​\(θ\|V\)P\(\\theta\|V\)andP​\(V\)P\(V\)are defined analogously and refer to prior distribution, withθ\|V\\theta\|Vdistributed as in[7](https://arxiv.org/html/2607.21671#S3.E7), andVVas in[5](https://arxiv.org/html/2607.21671#S3.E5), based on the same prior inclusion probability per layer\.

We can now extend the Componentbbas follows:

b\\displaystyle b=dK​L​\(q​\(V\)∥P​\(V\)\)\\displaystyle=d\_\{KL\}\(q\(V\)\\\|P\(V\)\)\(13\)=∑l​j∫q​\(vjl\)​log⁡\[q​\(vjl\)P​\(vjl\)\]​𝑑vjl\\displaystyle=\\sum\_\{lj\}\\int q\(v\_\{j\}^\{l\}\)\\log\\\!\\left\[\\frac\{q\(v\_\{j\}^\{l\}\)\}\{P\(v\_\{j\}^\{l\}\)\}\\right\]dv\_\{j\}^\{l\}\(14\)=∑l​jdK​L​\(q​\(vjl\)∥P​\(vjl\)\)\\displaystyle=\\sum\_\{lj\}d\_\{KL\}\(q\(v\_\{j\}^\{l\}\)\\\|P\(v\_\{j\}^\{l\}\)\)\(15\)=∑l​j\[π~jl​log⁡π~jlπl\+\(1−π~jl\)​log⁡\(1−π~jl\)\(1−πl\)\]\\displaystyle=\\sum\_\{lj\}\\left\[\\tilde\{\\pi\}\_\{j\}^\{l\}\\log\\frac\{\\tilde\{\\pi\}\_\{j\}^\{l\}\}\{\\pi^\{l\}\}\+\(1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\)\\log\\frac\{\(1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\)\}\{\(1\-\\pi^\{l\}\)\}\\right\]\(16\)
Which is the sum of the KL distance between the variational distributionqqand the priorPPfor every inclusion indicator variable for all the neural atoms\. This quantity is differentiable with respect toπ~jl\\tilde\{\\pi\}\_\{j\}^\{l\}, which is important for training\.

Concerning the Componentccwe have:

c\\displaystyle c=𝔼q​\(V\)\[dK​L\(q\(θ\|V\)∥P\(θ\|V\)\)\]\\displaystyle=\\mathbb\{E\}\_\{q\(V\)\}\\\!\\left\[d\_\{KL\}\(q\(\\theta\|V\)\\\|P\(\\theta\|V\)\)\\right\]\(17\)=𝔼q​\(V\)\[∑l​jdK​L\(q\(W¯l​j\|vjl\)\|P\(W¯l​j\|vjl\)\)\]\\displaystyle=\\mathbb\{E\}\_\{q\(V\)\}\\\!\\left\[\\sum\_\{lj\}d\_\{KL\}\\\!\\left\(q\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}\)\\,\\middle\|\\,P\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}\)\\right\)\\right\]\(18\)=∑l​j𝔼q​\(vjl\)\[dK​L\(q\(W¯l​j\|vjl\)\|P\(W¯l​j\|vjl\)\)\]\\displaystyle=\\sum\_\{lj\}\\mathbb\{E\}\_\{q\(v\_\{j\}^\{l\}\)\}\\\!\\left\[d\_\{KL\}\\\!\\left\(q\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}\)\\,\\middle\|\\,P\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}\)\\right\)\\right\]\(19\)
Sincevjl∈\{0,1\}v\_\{j\}^\{l\}\\in\\\{0,1\\\}we have:

c=∑l​jq\(vjl=1\)dK​L\(q\(W¯l​j\|vjl=1\)\|P\(W¯l​j\|vjl=1\)\)\\displaystyle c=\\sum\_\{lj\}q\(v\_\{j\}^\{l\}=1\)\\,d\_\{KL\}\\\!\\left\(q\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}=1\)\\,\\middle\|\\,P\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}=1\)\\right\)\+\\displaystyle\+∑l​jq\(vjl=0\)dK​L\(q\(W¯l​j\|vjl=0\)\|P\(W¯l​j\|vjl=0\)\)\\displaystyle\\sum\_\{lj\}q\(v\_\{j\}^\{l\}=0\)\\,d\_\{KL\}\\\!\\left\(q\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}=0\)\\,\\middle\|\\,P\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}=0\)\\right\)
Sinceq​\(vjl=0\)=1−π~jlq\(v\_\{j\}^\{l\}=0\)=1\-\\tilde\{\\pi\}\_\{j\}^\{l\}and that undervjl=0v\_\{j\}^\{l\}=0

q​\(W¯l​j\|vjl=0\)=δ0=P​\(W¯l​j\|vjl=0\)\.q\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}=0\)=\\delta\_\{0\}=P\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}=0\)\.

then

dK​L\(q\(W¯l​j\|vjl=0\)\|P\(W¯l​j\|vjl=0\)\)=0d\_\{KL\}\\\!\\left\(q\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}=0\)\\,\\middle\|\\,P\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}=0\)\\right\)=0
So

c=∑l​jπ~jldK​L\(q\(W¯l​j\|vjl=1\)\|P\(W¯l​j\|vjl=1\)\)c=\\sum\_\{lj\}\\tilde\{\\pi\}\_\{j\}^\{l\}\\,d\_\{KL\}\\\!\\left\(q\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}=1\)\\,\\middle\|\\,P\(\\overline\{W\}\_\{lj\}\\,\|\\,v\_\{j\}^\{l\}=1\)\\right\)
=∑l​jπ~jldK​L\(𝒩\(μjl,diag\(σjl​2\)\)\|𝒩\(0,σ02I\)\)=\\sum\_\{lj\}\\tilde\{\\pi\}\_\{j\}^\{l\}\\,d\_\{KL\}\\\!\\left\(\\mathcal\{N\}\(\\mu\_\{j\}^\{l\},\\,\\text\{diag\}\(\\sigma\_\{j\}^\{l2\}\)\)\\,\\middle\|\\,\\mathcal\{N\}\(0,\\,\\sigma\_\{0\}^\{2\}I\)\\right\)
This result from the variational family and prior ofθ\\theta\. This quantity is also differentiable regardingμjl\\mu\_\{j\}^\{l\}andσjl\\sigma\_\{j\}^\{l\}\.

If Componentsbbandccof ELBO are differentiable regarding the parameters, it’s not the case of the data\-fit parameter Componentaa\. Authors have noticed that and Jantre et al as well\. This challenge is due to the discrete character of avjlv\_\{j\}^\{l\}\. As with previous research \(Jang et al 2017, Maddison et al 2017,Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\); Janget al\.\([2017](https://arxiv.org/html/2607.21671#bib.bib28)\); Maddisonet al\.\([2017](https://arxiv.org/html/2607.21671#bib.bib29)\)\) we will consider the Gumbel\-Softmax approximation that approximate the distributionq​\(vjl\)∼Bernoulli​\(π~jl\)q\(v\_\{j\}^\{l\}\)\\sim\\text\{Bernoulli\}\(\\tilde\{\\pi\}\_\{j\}^\{l\}\)by the Gumbel\-Softmax distribution:q​\(v~jl\)∼GS​\(π~jl,τ\)q\(\\tilde\{v\}\_\{j\}^\{l\}\)\\sim\\text\{GS\}\(\\tilde\{\\pi\}\_\{j\}^\{l\},\\tau\)\. The appendix[A](https://arxiv.org/html/2607.21671#A1)provides more details about this approximation, showing how it works practically and also provides weights posterior hyperparameter specification to complete the SS\-IG frame\.

### 3\.4The Neural Atom Prevalence selection rule

In the previous section we specify step by step how to compute the variational posterior distribution of weights, bias and also the neural atom inclusion posterior probabilityπj~l\\tilde\{\\pi\_\{j\}\}^\{l\}for each neural atomhjlh^\{l\}\_\{j\}\. This inclusion posterior tell how likely the atom is to be selected for the modelηθ\\eta\_\{\\theta\}\. An this last posterior depend on the prior inclusion per layerπl\\pi^\{l\}, but we haven yet specify hoe it’s obtain\.

#### 3\.4\.1πl\\pi^\{l\}calibration from the Bayesian Lottery Ticket

To complete the method of Jantre et al\. \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\) we have to specify how to obtain or chooseπl\\pi^\{l\}andτ\\tau\.

Let’s start withτ\\tauwhose the most suitable value, serving as trade off between discretevjlv\_\{j\}^\{l\}and the continuous approximatev~jl\\tilde\{v\}\_\{j\}^\{l\}, isτ=0\.5\\tau=0\.5\.

Forπl\\pi^\{l\}, the paper by Jantre et al\. \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\), through it Corollary 4\.5, achieves the proofs of the consistency of the method and the resulting variation posteriors, stating that the optimal value ofπl\\pi\_\{l\}is giving by the following equation[20](https://arxiv.org/html/2607.21671#S3.E20):

−log⁡\(πl\)=log⁡\(kl\+1\)\+Cl​\(kl\+1\)​𝒱l\-\\log\(\\pi^\{l\}\)=\\log\(k\_\{l\+1\}\)\+C\_\{l\}\(k\_\{l\}\+1\)\\mathcal\{V\}\_\{l\}\(20\)where

𝒱l=Bl2/\(kl\+1\)\+∑m=0m≠lLlog⁡\(Bm\)\+L\+log⁡\(kl\+1\)\+log⁡\(n\)\+log⁡\(∑m=0L𝒰m\)\\mathcal\{V\}\_\{l\}=B\_\{l\}^\{2\}/\(k\_\{l\}\+1\)\+\\sum\_\{\\begin\{subarray\}\{c\}m=0\\\\ m\\neq l\\end\{subarray\}\}^\{L\}\\log\(B\_\{m\}\)\+L\+\\log\(k\_\{l\}\+1\)\+\\log\(n\)\+\\log\\\!\\left\(\\sum\_\{m=0\}^\{L\}\\mathcal\{U\}\_\{m\}\\right\)𝒰m=\(L\+1\)2​\[log⁡n\+log⁡\(L\+1\)\+log⁡\(kl\+1\)\+log⁡\(kl\+1\)\]\\mathcal\{U\}\_\{m\}=\(L\+1\)^\{2\}\\\!\\left\[\\log n\+\\log\(L\+1\)\+\\log\(k\_\{l\}\+1\)\+\\log\(k\_\{l\+1\}\)\\right\]‖\(‖W¯l​1‖1,…,‖W¯l​Kl‖1\)‖∞≤Bl\\\|\(\\\|\\overline\{W\}\_\{l1\}\\\|\_\{1\},\.\.\.,\\\|\\overline\{W\}\_\{l\{K\_\{l\}\}\}\\\|\_\{1\}\)\\\|\_\{\\infty\}\\leq B\_\{l\}where theBl=Kl−1\+1B\_\{l\}=K\_\{l\-1\}\+1inJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)they selectClC\_\{l\}in the order of10−510^\{\-5\}to avoid havingπl<10−50\\pi^\{l\}<10^\{\-50\}\. This work take a different approach to have a more realisticπl\\pi^\{l\}\. The Bayesian Lottery ticket\(the winning ticket\) is a good departure for improving the model sparsity while preserving prediction preformance\. According toKuhnet al\.\([2026](https://arxiv.org/html/2607.21671#bib.bib5)\)BLT can prune up to 80% of model weights without harming the prediction performance\. To take advantage of this fact, the NAP method computeBl=Bw​tB\_\{l\}=B\_\{wt\}recorded on the winning ticket andClC\_\{l\}is calculated such that10−50≤πl≤Sw​t10^\{\-50\}\\leq\\pi^\{l\}\\leq S\_\{wt\}wereSw​tS\_\{wt\}is given as follow:

Sw​t=‖\(‖W¯l​1‖1,…,‖W¯l​Kl‖1\)‖0S\_\{wt\}=\\\|\(\\\|\\overline\{W\}\_\{l1\}\\\|\_\{1\},\.\.\.,\\\|\\overline\{W\}\_\{l\{K\_\{l\}\}\}\\\|\_\{1\}\)\\\|\_\{0\}calculated on the winning ticket\.

#### 3\.4\.2Posterior distribution of the Layer SizeKVlK\_\{V^\{l\}\}:

The first motivation of the adjustment above is to obtain the inclusion posterior probability ofπ~jl\\tilde\{\\pi\}\_\{j\}^\{l\}for each neural atom\.

Recalling that for each layerll:

KVl=∑j=1Klvjl\(vjl∼Bernoulli​\(π~jl\)\)K\_\{V^\{l\}\}=\\sum\_\{j=1\}^\{K\_\{l\}\}v\_\{j\}^\{l\}\\quad\(v\_\{j\}^\{l\}\\sim\\text\{Bernoulli\}\(\\tilde\{\\pi\}\_\{j\}^\{l\}\)\)
The variableKVlK\_\{V^\{l\}\}follows a Poisson\-binomial distribution \(Hong \([2013](https://arxiv.org/html/2607.21671#bib.bib1)\)\):

P​\(KVl=k\)=∑Vl‖Vl‖0=k∏j=1Kl\(π~jl\)vjl​\(1−π~jl\)1−vjlP\(K\_\{V^\{l\}\}=k\)=\\sum\_\{\\begin\{subarray\}\{c\}V^\{l\}\\\\ \\\|V^\{l\}\\\|\_\{0\}=k\\end\{subarray\}\}\\prod\_\{j=1\}^\{K\_\{l\}\}\(\{\\tilde\{\\pi\}\_\{j\}^\{l\}\}\)^\{v\_\{j\}^\{l\}\}\(1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\)^\{1\-v\_\{j\}^\{l\}\}
but for very dense networks this distribution is intractable \(Hong \([2013](https://arxiv.org/html/2607.21671#bib.bib1)\)\)\. Many algorithm has been develop to overcome this issue among which the Dynamic Programming Recursion \(Hong \([2013](https://arxiv.org/html/2607.21671#bib.bib1)\)\) with which is give as follow:

Forkk: LetKVl\(j\)K\_\{V^\{l\}\}^\{\(j\)\}denote the number of active neurons among the firstjjneurons in layerll, with the conventionKVl\(0\)=0K\_\{V^\{l\}\}^\{\(0\)\}=0\. The distribution of the total active node countKVl=KVl\(Kl\)K\_\{V^\{l\}\}=K\_\{V^\{l\}\}^\{\(K\_\{l\}\)\}is then computed via the recursion

P​\(KVl\(j\)=k\)=P​\(KVl\(j−1\)=k\)​\(1−π~jl\)\+P​\(KVl\(j−1\)=k−1\)​π~jl,P\\\!\\left\(K\_\{V^\{l\}\}^\{\(j\)\}=k\\right\)=P\\\!\\left\(K\_\{V^\{l\}\}^\{\(j\-1\)\}=k\\right\)\\left\(1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\\right\)\+P\\\!\\left\(K\_\{V^\{l\}\}^\{\(j\-1\)\}=k\-1\\right\)\\tilde\{\\pi\}\_\{j\}^\{l\},
initialised atP​\(KVl\(0\)=0\)=1P\(K\_\{V^\{l\}\}^\{\(0\)\}=0\)=1andP​\(KVl\(0\)=k\)=0P\(K\_\{V^\{l\}\}^\{\(0\)\}=k\)=0fork\>0k\>0\. This forward recursion has complexity𝒪​\(Kmax2\)\\mathcal\{O\}\(K\_\{\\max\}^\{2\}\)and is considerably more tractable than enumerating all2Kl2^\{K\_\{l\}\}node subsets directly\.

#### 3\.4\.3The most likely layer sizeKlo​pK\_\{l\}^\{op\}

With the posterior distribution of the layerll, the most likely layer size of the distribution is given by:

Klo​p=arg maxk​P​\(KVl=k\)K\_\{l\}^\{op\}=\\text\{arg max\}\_\{k\}P\(K\_\{V^\{l\}\}=k\)

#### 3\.4\.4Selection of the most prevalent Neural atom

The selection is performed by simply rank the atom in each layer according to their inclusion posteriorπ~jl\\tilde\{\\pi\}\_\{j\}^\{l\}

Then Atomhjlh\_\{j\}^\{l\}is selected if

ℛ​\(π~jl\)≤Klo​p\\mathcal\{R\}\(\\tilde\{\\pi\}\_\{j\}^\{l\}\)\\leq K\_\{l\}^\{op\}
whereℛ​\(π~jl\)\\mathcal\{R\}\(\\tilde\{\\pi\}\_\{j\}^\{l\}\)is the rank ofπ~jl\\tilde\{\\pi\}\_\{j\}^\{l\}in the sorted list ofπ~jl\\tilde\{\\pi\}\_\{j\}^\{l\}in layerll\.

### 3\.5Neural Atom Prevalence method pipeline and Algorithm

The final NAP method combines successively the BLT through Iterative Magnitude Pruning and, the Soft SS\-IG and PB selection and then the fine\-tuning of the selected network\. To finalise the pipeline, the following adjustments need to be made from the original SS\-IG byJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\. Avoid pruning during training: To achieve this, the soft variablev~jl\\tilde\{v\}\_\{j\}^\{l\}will be used in both the forward and backward pass during training\. This will keep all the neurons until the end of the training, allowing to compute both the inclusion posterior and the model size distribution\. That is why we are talking about soft SS\-IG on winning ticket\.

#### 3\.5\.1Method Algorithm: Neural atom prevalence method

The following table[1](https://arxiv.org/html/2607.21671#S3.T1)provides a detailed algorithm of the Neural Atom Prevalence pipeline\.

Table 1:Neural Atom Prevalence \(NAP\) Algorithm StructureAlgorithm: Neural Atom Prevalence \(NAP\)Input: Training data D, dense architecture, optimizer, hyperparametersOutput: Sparse BNN with selected neurons1\. Train Bayesian Lottery Ticket \(BLT\) on dense network2\. Extract winning ticket weightsWlW^\{l\}and biasesBlB^\{l\}3\. Compute layer\-wise prior inclusion probabilitiesπl\\pi^\{l\}using Corollary 4\.5 of Jantre et al\.substituting dense network statistics with winning ticket statistics \(Sw​tS\_\{wt\},Bw​tB\_\{wt\}\)4\. Initialize SS\-IG on winning ticket architecture with softvjlv\_\{j\}^\{l\}in forward/backward passes5\. Train SS\-IG forEEepochs, tracking posterior inclusion probabilitiesπ~jl\\tilde\{\\pi\}\_\{j\}^\{l\}6\. For each layerll, compute layer size distributionP​\(KVl=k\)P\(K\_\{V^\{l\}\}=k\)via Poisson\-binomial DP recursion7\. Find optimal layer sizeKlo​p=argmaxk​P​\(KVl=k\)K\_\{l\}^\{op\}=\\text\{argmax\}\_\{k\}P\(K\_\{V^\{l\}\}=k\)8\. In each layer, select theKlo​pK\_\{l\}^\{op\}neurons with highestπ~jl\\tilde\{\\pi\}\_\{j\}^\{l\}9\. Prune unselected neurons and their incident weights10\. Fine\-tune the sparse BNN from epoch\-0 initialization of the original SS\-IG11\. Return final sparse networkWe proposed a neural atom prevalence method for model selection, based on the prevalence of neural atom\. Our method is an adjustment of the spike and slab independent gaussian method ofJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\. We modified the training process of their method and the determination of the model prior inclusion distribution probability using Bayesian winning ticket network architecture\. This change allows us to transpose the Fokoue’s atom prevalence method to neural network by computing the posterior inclusion probability for all the neural atom, the a posteriory most likely layer size for each layer in the network and then construct the final model by selecting the most prevalent atoms\. this model as shown to be consistent, producing a model, that, converge to the true model as the dataset become larger\. The next chapter of this work will consist of experimenting the method with the aims of proving the quality of the model and empirically comparing it with the baseline SS\-IG and the dense VBNN\.

## 4Empirical Validation of Neural Atom Prevalence Method

This chapter serves as empirical validation of the method and algorithm developed in the previous chapter\. Our framework is designed to be applicable to both regression and classification problems\. To ensure scientific rigor, experiments are conducted on both real\-world and simulated datasets\. This is important since simulated data provide a known ground truth against which results can be exactly assessed\. Two other models will be used in comparative analysis, the original SS\-IG by Jantre et al\.Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\(baseline\) and the dense VBNN as described inJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\. The chapter is organized as follows: The first part will focus on the setup of the experiment, presenting the experiment environment from the selection of the data set to the specification of hyperparameters for all types of data \(simulated, regression and classification\)\. The second and third parts will present the results and discussions, showing the performance of the proposed method compared to the baseline and the winning ticket\.

### 4\.1Experimental Setup

All experiments are conducted on a local Jupyter notebook equipped with an NVIDIA 6 GB VRAM GPU, running with Python 3\.11 and CUDA\. Random seeds are fixed at 42 across all libraries \(PyTorch, NumPy\) to ensure reproducibility\. The Adam optimizer is used throughout with gradient clipping at norm 10\.0\. Three methods are compared in all experiments:

- •The proposed Neural Atom Prevalence Method: Applying SS\-IG with softvj~l\\tilde\{v\_\{j\}\}^\{l\}in both forward and backward passes on the winning Ticket network with weights\. This is followed by the Poisson Binomial dynamic programming algorithm, applied to compute the optimal network size, which allows node selection to be performed\. Finally, the selected network is then fine\-tune in a Bayesian framework from the epoch0 of SS\-IG initialization;
- •The original SS\-IG \(by Jantre et al\. 2022\): The original Spike and Slab Independent Gaussian Bayesian neural network, trained on the full dense architecture using a hard Gumbel\-Softmax relaxation \(straight\-through estimator\) with layer\-wise prior inclusion probabilities from Corollary 4\.5 ofJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\);
- •The VBNN as presented in \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\)\.

At the end of all experiments, the sparsity is reported as the fraction of nodes removed relative to the original dense architecture in each layer\. All accuracy and Root Mean Squared Error \(RMSE\) values are reported on training, validation, and test sets across training epochs\.

#### 4\.1\.1Simulated Data

The simulation study follows the design of Jantre et al\. \(2022, section 6\.2\) to allow for direct comparison\. Data are generated from the nonlinear regression model:

Y=7​x21\+x12\+sin⁡\(x3​x4\)\+2​x5\+ε,ε∼𝒩​\(0,1\)Y=\\frac\{7x\_\{2\}\}\{1\+x\_\{1\}^\{2\}\}\+\\sin\(x\_\{3\}x\_\{4\}\)\+2x\_\{5\}\+\\varepsilon,\\quad\\varepsilon\\sim\\mathcal\{N\}\(0,1\)
Where all covariatesx1,…,x5x\_\{1\},\\ldots,x\_\{5\}are i\.i\.d\.𝒩​\(0,1\)\\mathcal\{N\}\(0,1\)and independent ofε\\varepsilon\. A training set of 3000 observations and a test set of 1000 observations are generated\. Regarding the network architecture, a two\-hidden\-layer network with 20 neurons per layer is used, exactly as inJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\. Swish activation is used for all three methods for consistency withJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\. The model is trained for 10000 epochs with a learning rate of5×10−35\\times 10^\{\-3\}and full batch gradient descent\. The evaluation metric is Root Mean Squared Error \(RMSE\) on the test set, along with layer\-wise node sparsity\.

The hyperparameters for the simulated data experiment are summarized in Table 4\.1\.

Table 2:Simulated Data Hyperparameters

### 4\.2UCI Regression Datasets

Two real\-world regression datasets from the UCI Machine Learning RepositoryDuaet al\.\([2017](https://arxiv.org/html/2607.21671#bib.bib25)\)are used, following the experimental protocol inJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\. Each dataset is split randomly into a 90% training set and a 10% test set\. For the smaller dataset \(Concrete\), this procedure is repeated over 20 independent random splits, and results are reported as the mean±\\pmstandard deviation of test RMSE\. For the Year dataset, one split is used, following the original protocol\.

A single hidden layer network with 50 neurons is used for Concrete, and 100 neurons for Year, with Swish activation throughout\. All models are trained with the Adam optimizer at learning rate10−310^\{\-3\}and batch size 128 \(256 for Year\)\.

Table[3](https://arxiv.org/html/2607.21671#S4.T3)summarizes the UCI regression datasets and their respective configurations\.

Table 3:UCI Regression Dataset Summary#### 4\.2\.1Prior Calibration and Pruning

The layer\-wise prior inclusion probabilitiesπ1\\pi^\{1\}for SS\-IG and NAP are computed from Corollary 4\.5 ofJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\. For NAP specifically,π1\\pi^\{1\}is computed using the surviving node countSw​tS\_\{wt\}and the maximum L1 weight normBw​tB\_\{wt\}of the winning ticket, rather than the full dense architecture, calibrating the prior to the compressed topology\. For the SS\-IG baseline,π1\\pi^\{1\}is computed using the full dense architecture dimensions andB1=k1\+1B\_\{1\}=k\_\{1\}\+1as recommended in Appendix B\.4 ofJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\.

The IMP phase uses 5 pruning rounds with a pruning fraction of 0\.35 per round\. The evaluation metric is test RMSE in original \(unscaled\) units, obtained by inverting theStandardScalerapplied during preprocessing\.

#### 4\.2\.2Image Classification Datasets

The same standard image classification benchmark is used as inJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\): MNIST\. This dataset consists of 60,000 training images and 10,000 test images of size28×2828\\times 28pixels across 10 classes\. Pixel values are divided by 126, following the exact protocol ofJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\. Images are flattened to vectors of dimension 784 for the MLP architecture\.

#### 4\.2\.3Architecture

A two\-hidden\-layer MLP with 400 neurons per layer and 10 output neurons is used \(784→400→400→10784\\rightarrow 400\\rightarrow 400\\rightarrow 10\)\. The Swish activation function \(f​\(x\)=x⋅σ​\(x\)f\(x\)=x\\cdot\\sigma\(x\), also known as SiLU\) is used for all three methods, as recommended by the original paper to avoid the dying neuron problem in large networks \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\)\.

#### 4\.2\.4Training Configuration

All methods are trained for 1,200 epochs with batch size 1,024 and learning rate10−310^\{\-3\}, matching the paper’s classification setup\. The KL term is annealed linearly from 0 to its maximum value over the first 400 epochs \(one third of training\), following the original implementation of the original SS\-IG\. For the IMP phase, 5 pruning rounds are used with a pruning fraction of 0\.35 per round, and 500 epochs of training per round\. The fine\-tuning phase of NAP runs for an additional 1,200 epochs on the pruned BNN topology\.

#### 4\.2\.5Hyperparameters

Table[4](https://arxiv.org/html/2607.21671#S4.T4)lists the hyperparameters for the classification experiments\.

Table 4:Classification Experiment HyperparametersHereSlS\_\{l\}denotes the number of surviving nodes in layerllafter IMP andKlo​pK^\{op\}\_\{l\}denotes the Poisson\-Binomial optimal node count selected by NAP in layerll\.

#### 4\.2\.6Metrics

Test accuracy is the primary performance metric\. Node sparsity per layer is computed as the fraction of nodes removed relative to the original dense architecture \(400 nodes per layer\)\. For NAP, sparsity is fixed after PB selection and remains constant during fine\-tuning\. The compression ratio is defined as the ratio of active parameters in the compressed network to total parameters in the original dense network\.

#### 4\.2\.7Weight Initialization Strategy

A key design choice of the proposed method, directly analogous to the Lottery Ticket Hypothesis, concerns the initialization of the final pruned BNN\. In the LTH, after IMP identifies the winning ticket topology, the surviving weights are rewound to their epoch\-0 values \(the original random initialization of the dense network\) before final training\. The NAP method adopts the same principle for the Bayesian setting: after PB selection identifies the optimalKo​pK^\{op\}node topology from the trained SS\-IG posteriors, the weights of the final BNN are initialized from the epoch\-0 state of the original SS\-IG that is, the Kaiming uniform initialization ofμ\\mu, the constant initialization ofρ=−6\\rho=\-6\(givingσ=eρ/2=e−3≈0\.05\\sigma=e^\{\\rho/2\}=e^\{\-3\}\\approx 0\.05\), and the prior inclusion probabilityπ=0\.99\\pi=0\.99, rendering all nodes active\. Only the node positions identified by PB are retained; all other nodes are discarded\. This rewinding strategy ensures that the fine\-tuning phase benefits from a clean, unbiased starting point calibrated to the selected topology, rather than inheriting potentially over\-fitted posteriors from the full training run\.

### 4\.3Results and Discussion

#### 4\.3\.1Simulated Data experiment

The simulated data experiment evaluates the Neural Atom Prevalence \(NAP\) under known ground\-truth conditions\. As detailed in the figure[1](https://arxiv.org/html/2607.21671#S4.F1), combining Bayesian Lottery Ticket, SS\-IG on ticket, binomial selection, and fine\-tuning results in a sparse network, pruning 14 out of 20 nodes per layer\. On test data, the pipeline stabilizes RMSE around1\.201\.20\(Table[5](https://arxiv.org/html/2607.21671#S4.T5)\)\.

As shown in Panel A \(Figure[1](https://arxiv.org/html/2607.21671#S4.F1)\), the test RMSE stabilizes at∼1\.15\\sim 1\.15during the SS\-IG phase on the winning ticket and NAP only marginally nudges this to∼1\.20\\sim 1\.20a negligible performance penalty for the massive gains in sparsity\. And the compression is important, as highlighted by Panel B \(Figure[1](https://arxiv.org/html/2607.21671#S4.F1)\), from the dense network, through the winning ticket, to the final NAP model, the layer size drop from 20 to 6 neurons this drop is meanly due to the Bayesian IMP as suggested by Panels C \(Figure[1](https://arxiv.org/html/2607.21671#S4.F1)\)\. Panel A \(Figure[1](https://arxiv.org/html/2607.21671#S4.F1)\) also highlights a gap between train and test accuracy curves, suggesting a possible overfitting situation\.

However, although this compression and RMSE appear to slightly underperform the results obtained by Jantre et al\. on the same experiment, the uncertainty analysis provides powerful information supporting the high quality of the NAP quantification\. The uncertainty decomposition in Panel D \(Figure[1](https://arxiv.org/html/2607.21671#S4.F1)\) reveals a clean separation with the aleatoric mean: settles at 1\.2456 while the epistemic mean drops to just 0\.0387, leaving an epistemic fraction of only∼3%\\sim 3\\%\. While this mirrors our real\-world year\-prediction results, it is even more compelling here because the ground\-truth noise is an explicitly known variable\. So, the uncertainty of the model is largely due to the noise introduced in data rather than model failure\.

NAP model coverage report shows that the 95% prediction intervals achieve a 93\.4% observed coverage\. This represents a small undercoverage of about 1\.6%, which is well within the expected margins for approximate Bayesian inference and it’s similar to the literature\(Kendall and Gal \([2017](https://arxiv.org/html/2607.21671#bib.bib26)\)reported similar 1–3% \)\.

Finally, the NAP method is a clear baseline for the sparsity\-accuracy trade\-off, With the optimal RMSE locked in, the exact performance cost of pruning can be precisely quantified, which in this case is a minor∼0\.04\\sim 0\.04RMSE increase\. An in the real\-world setting it can be confirmed that an observed coverage of 93\.4% is genuinely close to the nominal 95% target\. This experiment demonstrated that although the NAP pipeline slightly underperforms the SS\-IG by Jantre et al\., it is a highly reliable and well\-calibrated method, providing a good representation of reality when it comes to uncertainty quantification and sparsity\.

![Refer to caption](https://arxiv.org/html/2607.21671v1/Figures/nap_simulation_II_results1.png)Figure 1:Results of experiment on simulated data:\(A\) RMSE vs epoch, NAP vs baseline\. \(B\) Node sparsity vs epoch\. \(C\) Node sparsity vs IMP round with test RMSE\. \(D\) Epistemic vs aleatoric uncertainty scatter\. \(E\) Regression reliability/calibration diagram\.
#### 4\.3\.2Regression experiment on UCI dataset

After analyzing the performance of the NAP pipeline on simulated data, the experiments with real\-world datasets from UCI confirm that the pipeline is consistent in real\-world settings\. As noted with the preceding analysis, the NAP framework shows a balance between shrinking the model and keeping its predictions accurate\.

#### 4\.3\.3YearPredictionMSD dataset

On the YearPredictionMSD dataset, NAP recorded a test RMSE of8\.828\.82years \(Table[5](https://arxiv.org/html/2607.21671#S4.T5)\), which is consistent with the literature \(8\.688\.68for SS\-IG\)\. However, this small increase of0\.140\.14years is compensated by a higher sparsity level\. NAP used only44%44\\%\(Table[5](https://arxiv.org/html/2607.21671#S4.T5)\) of the dense network, while SS\-IG and SV\-BNN used at least71%71\\%\(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\)\. This highlights the good compression performance of the method and this 8\.8\-year mark effectively represents the irreducible error floor for this prediction task\. These results hold up remarkably well against classic sparse regression benchmarks\. For context, the Lottery Ticket Hypothesis \(Frankle and Carbin \([2019](https://arxiv.org/html/2607.21671#bib.bib4)\)\) typically retains 80 to 90% dense precision at a weight fraction of 10 to 20% and NAP achieves a∼1\.7%\\sim 1\.7\\%relative RMSE increase \(compared to an estimated dense VBNN baseline of∼8\.67\\sim 8\.67years\) is right in line with this expectation at such high sparsity levels\. Even better, the training dynamics remain incredibly stable\(Figure[2](https://arxiv.org/html/2607.21671#S4.F2)\)\. The narrow±1​σ\\pm 1\\sigmabands in Panels A and B show minimal run\-to\-run variance, proving that NAP avoids the instability often triggered by unstructured pruning\. Furthermore, the model delivers extreme sparsity without sacrificing robust convergence\.

Our uncertainty analysis reveals a model that is highly confident but constrained by intrinsically noisy data\. As shown in Panel E of Figure[2](https://arxiv.org/html/2607.21671#S4.F2), aleatoric uncertainty strongly dominates the total predictive variance \(meanσ=8\.213\\sigma=8\.213years\), while epistemic uncertainty represents a mere 0\.337 years, just 4\.1% of the total\. Specifically, Panel E\(Figure[2](https://arxiv.org/html/2607.21671#S4.F2)\) shows a clear separation between epistemic and aleatoric uncertainty, indicating that the model is entirely sure of its own parameters; it simply cannot overcome the raw noise built into the year\-prediction signal itself\. The reliability diagram \(Panel F, Figure[2](https://arxiv.org/html/2607.21671#S4.F2)\) supports this, showing excellent probabilistic calibration\. From Panel F, we can see that the coverage is very close to the diagonal line, with a Calibration Error=0\.0310=0\.0310, indicating that the uncertainty estimates deviate by only3\.1%3\.1\\%, well below the standard 0\.05 threshold for "good" calibration \(Guoet al\.\([2017](https://arxiv.org/html/2607.21671#bib.bib27)\)\)\. Unlike typical, overconfident neural networks, the NAP model yields trustworthy prediction intervals\.

![Refer to caption](https://arxiv.org/html/2607.21671v1/Figures/nap_year_training_curves.png)Figure 2:Results of experiment on YearPredictionMSD dataset: \(A\) RMSE vs epoch, SS\-IG phase\. \(B\) RMSE vs epoch, fine\-tune phase\. \(C\) Surviving nodes per IMP round\. \(D\) Fraction of active neurons vs epoch\. \(E\) Uncertainty decomposition histogram\. \(F\) Regression reliability/calibration diagram\.
#### 4\.3\.4Concrete dataset

On the Concrete dataset, the NAP method exhibits impressive performance, achieving better compression compared to SS\-IG \(75%75\\%vs58%58\\%\)\. Moreover, this compression clearly improves the model performance with an RMSE of6\.3176\.317, compared to7\.927\.92for SS\-IG\. The prediction interval of NAP is also smaller than that of SS\-IG \(Table[5](https://arxiv.org/html/2607.21671#S4.T5)\)\.

Regarding uncertainty quantification and the reliability of the method, panels C and D of Figure[3](https://arxiv.org/html/2607.21671#S4.F3)indicate that almost all total uncertainty of the model produced by the method is driven by data quality\. This is quantified by the value of the epistemic/total uncertainty ratio\. Even if this result is not as perfect as the one recorded on the year data, Panel D suggests that the uncertainty of the model is very well\-calibrated, with a calibration error of0\.02730\.0273\.

Over the two experiments, it comes out that the NAP method in a regression setting and in a real\-world setting produces a trustworthy, well\-calibrated, and confident model which achieves good performance with a benchmark sparsity\. These experiments successfully validate the NAP method for model compression in neural networks and uncertainty quantification\.

![Refer to caption](https://arxiv.org/html/2607.21671v1/Figures/nap_concrete_curves.png)Figure 3:Results of experiment on Concrete Dataset:\(A\) RMSE vs epoch, SS\-IG and fine\-tune\. \(B\) Fraction of active neurons vs epoch\. \(C\) Uncertainty decomposition histogram \(aleatoric vs epistemic\)\. \(D\) Regression reliability/calibration diagram\.

### 4\.4Classification experiment on MNIST

The experiment in the case of the classification task has completed the evaluation and validation of the NAP method, providing the behavior of the pipeline on the image dataset MNIST\. The results don’t deviate from the first two experiments, if it’s not to give better performance\. The NAP method achieves an accuracy of∼97%\\sim 97\\%with the higher sparsity level, using only 8% of the initial dense network, While demonstrating good stability in the earlier epochs \(epoch∼50\\sim~50\), Well quantifying uncertainty and reliable \(Figure[4](https://arxiv.org/html/2607.21671#S4.F4)\)\. The work also show the performance of the modified SS\-IG running from the winning ticket which has similar results to its original version \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\)\.

![Refer to caption](https://arxiv.org/html/2607.21671v1/Figures/nap_mnist_training_curves.png)Figure 4:Results of experiment on MNIST dataset:\(A\) Accuracy vs epoch, SS\-IG phase\. \(B\) Accuracy vs epoch, fine\-tune phase\. \(C\) Surviving nodes per IMP round\. \(D\) Fraction of active neurons vs epoch\. \(E\) Predictive entropy distribution \(correct vs incorrect\)\. \(F\) Reliability/calibration diagram\.#### 4\.4\.1Model Sparsity, Accuracy, and Convergence Dynamics on MNIST

More detail analysis of the results indicates that, as shown in Panel A of Figure[4](https://arxiv.org/html/2607.21671#S4.F4), SS\-IG \(softvj~l\\tilde\{v\_\{j\}\}^\{l\}only, on winning ticket\) test accuracy rapidly climbs and stabilizes above 90% within the first 100 epochs, maintaining a tight±1​σ\\pm 1\\sigmaMonte Carlo \(MC\) confidence band throughout the 1200\-epoch run\. The NAP Model\(Fine\-tune\) presents a similar stable trajectory during fine\-tuning \(Panel B, Figure[4](https://arxiv.org/html/2607.21671#S4.F4)\)\.

The structural compression driving these results is detailed in Panels C and D of Figure[4](https://arxiv.org/html/2607.21671#S4.F4)\. Panel C tracks the surviving nodes across 5 Iterative Magnitude Pruning \(IMP\) rounds for both layers\. While Layer 2 \(teal\) maintains its node count at 400 across almost all rounds, Layer 1 \(purple\) experiences targeted pruning, dropping to 380 nodes by Round 5\. Moving to the next step \(soft SS\-IG plus PB selection model the compression is captured in Panel D, which plots the fraction of active neurons over time\. We observe a dramatic, rapid drop in active neurons within the first 150 epochs, with Layer 1 \(solid blue\) flattening out below 10% active capacity, and Layer 2 \(dashed blue\) stabilizing around 15%, consistent withJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\. The narrow variance bands for both layers indicate that the pruning trajectory is highly predictable and free from the erratic convergence spikes often seen in unstructured pruning schemes\. The constant and horizontal line showing the sparsity of the NAP model \(Fine\-tune\) simply reflects the fact that after the Poisson Binomial selection \(Ko​p=\(20n​o​d​e​s,43n​o​d​e​s\)K\_\{op\}=\(20\\quad nodes,\\quad 43\\quad nodes\)\), the selected network is just fine\-tuned; there is no more compression\.

#### 4\.4\.2Predictive Uncertainty Decomposition and Calibration Behavior

The model’s probabilistic health is illustrated via the uncertainty decomposition and calibration diagnostics in Panels E and F of Figure[4](https://arxiv.org/html/2607.21671#S4.F4)\. Panel E displays the predictive entropy distribution for the final model, revealing a distinct behavioral split between correct and incorrect predictions\. For correct predictions \(the sharp blue peak\), the model is highly confident, yielding a mean entropy of 0\.150\. Conversely, incorrect predictions \(the wide, flat red distribution\) exhibit significantly higher uncertainty, stretching across an entropy range from 0\.5 to over 2\.0 with a mean of 0\.899\. The inset scatter plot confirms that while epistemic uncertainty remains generally low \(∼0\.038\\sim 0\.038\), aleatoric uncertainty dominates the total variance, scaling proportionally with incorrect classifications\. This proves that the model is naturally confident about what it knows and what it does not know\.

However, the reliability diagram in Panel F introduces a red flag regarding real\-world deployment, a calibration not good for the preceding experiments\. The figure presents an overconfident NAP model, particularly in the lower confidence bins \(0\.20\.2to0\.60\.6\), where the actual accuracy falls well short of the perfect calibration line \(the dashed diagonal\)\. This is reflected in an Expected Calibration Error \(ECE\) of 0\.1005 and a Maximum Calibration Error \(MCE\) of 0\.3603\. While the network remains highly accurate on the whole, these results indicate the need for post\-hoc calibration adjustments\.

In summary the NAP in both regression and classification settings remains highly accurate, stable, and reliable\. To definitively assess the NAP frame globally, the next section present the comparison analysis on NAP with the baseline for this work, the SS\-IG \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\)\.

### 4\.5Model comparison

To contextualize the performance of the proposed Neural Atom Prevalence \(NAP\) framework, we conduct a comparative analysis against the layer\-adaptive node selection methodology \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\) and the Variational Bayesian neural network\(VBNN\), a dense network without pruning \(Blundellet al\.\([2015](https://arxiv.org/html/2607.21671#bib.bib10)\)\)\. Both frameworks \(NAP and SS\-IG Method\) share the objective of inducing structural \(node\-level\) sparsity within a Bayesian Neural Network \(BNN\) environment using Variational Inference\. However, they differ fundamentally in their pruning mechanics and architectural execution\. We evaluate these differences across three distinct axes: model sparsity, predictive precision, and uncertainty quantification \(UQ\)\.

#### 4\.5\.1Model Sparsity Dynamics

The primary point of divergence between the two approaches lies in how node pruning is executed and sustained\. Jantre et al\. \(2022\) utilize a continuous variational formulation governed by spike\-and\-slab Gaussian priors\. This formulation allows the network to automatically suppress redundant nodes during training by driving their inclusion probabilities near zero, eliminating the need for hard, ad\-hoc thresholding rules\.

In contrast, the NAP pipeline employs a hybrid, multi\-phase trajectory \(BLT→\\rightarrowSS\-IG→\\rightarrowPB\-selection→\\rightarrowfine\-tune\)\. Instead of relying solely on smooth prior contraction, NAP enforces structural compression explicitly via Iterative Magnitude Pruning \(IMP\) loops\. As observed in our empirical results \(Table[5](https://arxiv.org/html/2607.21671#S4.T5)\), this strategy yields the best sparsity and a highly stable structural compression, reducing active node fractions down to∼\\sim15\-20% in non\-linear settings and scaling down to∼\\sim10% in Layer 1 on complex tasks like MNIST\. While Jantre et al\. demonstrate elegant asymptotic guarantees for posterior node recovery, NAP provides more explicit, rigid control over the target compression rate across individual layers, backed by tight±1​σ\\pm 1\\sigmavariance bands that prevent the convergence instabilities typical of unstructured pruning\.

Table 5:Comparison table of NAP with SS\-IG and VBNN
#### 4\.5\.2Model Precision \(RMSE and Accuracy\)

On the precision axis, NAP and SS\-IG models successfully challenge the traditional assumption that extreme model compression requires a steep sacrifice in predictive power\. Globally, in terms of accuracy and RMSE, NAP and SS\-IG are extremely close to the dense VBNN\. However, if VBNN is better than SS\-IG for all the experiments, Table[5](https://arxiv.org/html/2607.21671#S4.T5)shows that NAP outperforms both VBNN and SS\-IG on the concrete dataset with a difference in RMSE of−1\.603\-1\.603compared to SS\-IG and−1\.023\-1\.023compared to VBNN\. On classification setting with MNIST, NAP performs at least as well as SS\-IG, giving an accuracy around97%97\\%,2%2\\%below the99%99\\%of VBNN\. For the two other data sets, SS\-IG achieves better performance than NAP, but this difference is very small, with respectively0\.00630\.0063and0\.1220\.122for simulated data and YearPredictionMSD data\.

The performance of NAP results from it hybrid structure\. More precisely, the final fine\-tuning phase successfully stabilizes the network, allowing it to lock onto the irreducible error floor of the task\. And NAP’s precision benefits heavily from the "winning ticket" initialization, meaning that even at high sparsity levels, the preserved sub\-network maintains a clear path to optimal convergence without overfitting\.

#### 4\.5\.3Uncertainty Quantification \(UQ\)

The final, and perhaps most critical, axis of comparison is the fidelity of the predictive uncertainty\. BecauseJantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)lean on a pure variational framework to approximate the posterior, their model naturally captures predictive uncertainty but inherits the classic limitations of Mean\-Field Variational Inference \(MFVI\), namely, a tendency to underestimate the true posterior variance, which can lead to overconfident predictions in out\-of\-distribution \(OOD\) scenarios\.

The NAP framework provides a much more detailed breakdown of this behavior by explicitly decoupling aleatoric and epistemic components\. In controlled settings, NAP shows good calibration, isolating the irreducible data noise \(aleatoric\) while driving model ignorance \(epistemic\) down to an efficient fraction \(∼\\sim3\-4%\)\. This produces highly trustworthy, perfectly calibrated prediction intervals under in\-distribution testing as with the experiment on simulated data where NAP shows an empirical coverage of 93\.4% against a nominal 95% target\.

However, the diagnostic analysis also uncovers a structural vulnerability shared by NAP and SS\-IG the near\-zero epistemic uncertainty induced by aggressive node pruning means that deterministic uncertainty approximations \(such as MC Dropout overlays or MFVI variants\) run a severe risk of overconfidence when exposed to OOD data\. Consequently, while NAP provides excellent, highly calibrated confidence boundaries for its native test set, migrating it to production requires an explicit OOD detection layer, a practical safeguard that builds upon the theoretical UQ foundations laid out in spike\-and\-slab architectures\.

In summary, this chapter provides a detailed empirical evaluation and validation of the NAP framework\. Through different settings, the chapter presents step\-by\-step the behavior of the NAP method on simulated dataset, in a regression setting with UCI year and concrete datasets, and in classification setting, on MNIST data\. The results show that, in a hybrid approach combining the higher and controlled compression potential of Bayesian Lottery Ticket through \(IMP\), the consistency of layer\-adaptive node selection through SS\-IG and the stability of the Poisson\-Binomial selection and fine\-tuning, we produce sparse, stable, confident, and accurate neural network models\. The uncertainty quantification analysis reveals that the model produced by the NAP pipeline is globally reliable and highly confident about what it knows and does not know\. We also demonstrate that the NAP models achieve better sparsity performance than SS\-IG and, in some cases, also outperform both SS\-IG and the dense VBNN in terms RMSE\. The analysis finally highlights that an additional analysis of NAP’s behavior when exposed to out\-of\-distribution data is needed to guard the model against overconfidence and to definitively strengthen the pipeline\. We finally construct bloc by bloc the Neural Atom Prevalence Method, that reproduce the the Fokoue’s Atom Prevalence MethodFokoué \([2008](https://arxiv.org/html/2607.21671#bib.bib3)\)and then answers the core research question of this thesis\.

## 5Conclusion

This thesis aimed to develop a method for neural model construction that produces sparse, accurate, interpretable, stable, confident, and reliable, by extending Fokoué’s Atom Prevalence principle to neural networks under a variational Bayesian framework\. The Neural Atom Prevalence \(NAP\) framework developed in this work is a hybrid pipeline combining the performance of Bayesian Lottery ticket Hypothesis principle\(through IMP\), an adaptive SS\-IG\(without pruning\), PB selection, and fine\-tuning to produce highly sparse, confident, reliable and accurate neural network model\. This constitutes a coherent and novel contribution to the intersection of Bayesian model selection, network compression, and uncertainty\-aware deep learning\.

The principal contributions of this thesis are the following\.

- •Neural Atom formalization: The thesis provides a precise mathematical definition of the neural atom as the activation unithjl=ψl​\(gjl\)h\_\{j\}^\{l\}=\\psi\_\{l\}\(g\_\{j\}^\{l\}\), shifting the fundamental unit of network analysis from the weight to the neuron with the benefit of reducing the dimensionality of the model selection problem from the weight space to node space\. phases \(IMP, SS\-IG, PB selection and fine\-tuning\)\.
- •Algorithmic: In the NAP pipeline \(BLT→\\rightarrowSS\-IG on ticket→\\rightarrowPB selection→\\rightarrowBayesian fine\-tuning\), the IMP phase inherits the stable and high compression potential of the Bayesian Lottery Ticket Hypothesis; the soft Gumbel\-Softmax relaxation applied in both forward and backward passes during the SS\-IG phase prevents premature pruning; the Poisson\-Binomial dynamic programming recursion provides a computationally tractable \(O​\(kmax2\)O\(k\_\{\\max\}^\{2\}\)\) mechanism for computing the exact posterior distribution of the layer size and finally the optimal sizeKlop=arg⁡maxk⁡P​\(Kl=k\)K\_\{l\}^\{\\text\{op\}\}=\\arg\\max\_\{k\}P\(K\_\{l\}=k\)is then used to select theKlopK\_\{l\}^\{\\text\{op\}\}most prevalent atoms per layer, directly transposing the atom prevalence mechanism in the neural setting\.

Empirical validation in four distinct experimental settings \(simulated nonlinear regression, the YearPredictionMSD dataset, the Concrete compressive strength dataset, and the MNIST classification benchmark\) demonstrates interesting performance of NAP method\. It consistently achieves:

- •Benchmark sparsity: 70% node removal on simulated data, 75% on Concrete, 56% on YearPredictionMSD, and 92% on MNIST, always computed relative to the original dense architecture\.
- •Competitive accuracy: RMSE of 1\.20 \(simulated\),6\.31±0\.446\.31\\pm 0\.44MPa \(Concrete\), 8\.82 years \(Year\), and 97% test accuracy \(MNIST\), all within a narrow margin of the denser SS\-IG baseline and the unpruned variational BNN\.
- •Reliable uncertainty quantification: Epistemic uncertainty constitutes only33to4%4\\%of total predictive variance in regression tasks\. The 95% predictive interval achieves 93\.4% empirical coverage on simulated data \(MACE = 0\.0131\) and calibration errors from0\.02730\.0273to0\.03100\.0310on real datasets, well within the 0\.05 threshold widely accepted as indicating good calibration\.
- •Stable training dynamics: Tight±1​σ\\pm 1\\sigmaMonte Carlo variance bands across all epoch trajectories confirm that the hybrid NAP approach avoids the convergence instabilities characteristic of unstructured magnitude pruning\.

The NAP pipeline outperforms baselines on Concrete, achieving a test RMSE of6\.31±0\.446\.31\\pm 0\.44MPa on the Concrete dataset, compared to7\.92±0\.687\.92\\pm 0\.68of the SS\-IG baseline and7\.30±0\.607\.30\\pm 0\.60of the dense variational BNN, while simultaneously achieving higher sparsity \(75% vs\. 59%\)\. This result, replicated over 20 independent data splits, suggests that the winning\-ticket initialisation and layer\-size optimisation embedded in the NAP pipeline provide an implicit regularisation effect that benefits generalisation beyond what is achievable through variational training alone on the full architecture\.

NAP achieves high compression through three complementary forces: IMP produces a stable winning ticket, soft SS\-IG estimates principled posterior inclusion probabilities without premature pruning, and PB selection identifies the statistically optimal layer size from the full posterior distribution ofKVlK\_\{V^\{l\}\}\. The dominance of aleatoric over epistemic uncertainty across regression tasks confirms that NAP saturates its representational capacity on simulated data, the aleatoric mean \(1\.2456\) aligns closely with the true noise varianceϵ∼𝒩​\(0,1\)\\epsilon\\sim\\mathcal\{N\}\(0,1\), while epistemic uncertainty is negligible \(0\.0387\), providing stronger validation than test RMSE alone\. The MNIST results \(97%97\\%accuracy,92%92\\%sparsity\) reveal moderate miscalibration \(ECE = 0\.1005, MCE = 0\.3603\), with systematic overconfidence in low\-to\-mid confidence bins that reflects known Mean\-Field Variational Inference limitations rather than a failure of the NAP methodology itself\. Finally, the PB selection component of the NAP pipeline characterized its similarity with Fokoué’s Atom prevalence method, and then giving the yes answer to the research question of this study: Theoretical argument and empirical evidence support that NAP transfers Fokoué’s Atom prevalence method principle on the Bayesian neural network in the variational framework\.

Despite its strong performance, the NAP framework faces non\-negligible limitations that when solved, will definitely legitimize the high potential of NAP in neural network compression and uncertainty quantification\.

- •Mean\-Field Variational posterior quality: The mean\-field assumption systematically underestimates posterior variance, producing overconfident predictions, directly visible in MNIST calibration \(ECE = 0\.1005\)\. In multi\-modal loss landscapes, the solution may collapse to a suboptimal mode, misidentifying true prevalent atoms\.
- •Out\-of\-distribution generalization: Aggressive pruning drives epistemic uncertainty near zero for in\-distribution data, meaning the model cannot signal unfamiliar inputs\. This is a structural vulnerability for safetycritical applications, yet no OOD evaluation or mitigation is provided\.
- •Architectural restriction to feedforward networks: NAP does not extend to CNNs, transformers, RNNs, or residual connections, precluding application to virtually all modern large\-scale architectures\.
- •Computational cost: The four\-phase pipeline is substantially more expensive than single\-phase variational training \(10,000 epochs on simulated data; 5 IMP rounds×\\times500 epochs \+ 1,200 fine\-tuning epochs on MNIST\)\.

Future work will explore solutions of the above limitations, and extend the method to convolutional neural networks \(CNN\) and transformers\.

## References

- M\. M\. Barbieri and J\. O\. Berger \(2004\)Optimal predictive model selection\.The Annals of Statistics\.Cited by:[§2](https://arxiv.org/html/2607.21671#S2.p2.2)\.
- C\. Blundell, J\. Cornebise, K\. Kavukcuoglu, and D\. Wierstra \(2015\)Weight uncertainty in neural networks\.InInternational Conference on Machine Learning,pp\. 1613–1622\.Cited by:[§1](https://arxiv.org/html/2607.21671#S1.p1.1),[§2](https://arxiv.org/html/2607.21671#S2.p9.1),[§4\.5](https://arxiv.org/html/2607.21671#S4.SS5.p1.1)\.
- D\. Dua, C\. Graff,et al\.\(2017\)UCI machine learning repository, 2017\.URL http://archive\. ics\. uci\. edu/ml7\(1\),pp\. 62\.Cited by:[§4\.2](https://arxiv.org/html/2607.21671#S4.SS2.p1.1)\.
- E\. P\. Fokoué \(2008\)Estimation of atom prevalence for optimal prediction\.Contemporary Mathematics443,pp\. 103–129\.Cited by:[§1](https://arxiv.org/html/2607.21671#S1.p4.1),[§2](https://arxiv.org/html/2607.21671#S2.p12.1),[§2](https://arxiv.org/html/2607.21671#S2.p2.2),[§3\.2\.2](https://arxiv.org/html/2607.21671#S3.SS2.SSS2.p1.6),[§3\.2](https://arxiv.org/html/2607.21671#S3.SS2.p1.3),[§3](https://arxiv.org/html/2607.21671#S3.p1.1),[§4\.5\.3](https://arxiv.org/html/2607.21671#S4.SS5.SSS3.p4.1)\.
- J\. Frankle and M\. Carbin \(2019\)The lottery ticket hypothesis: finding sparse, trainable neural networks\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[§1](https://arxiv.org/html/2607.21671#S1.p2.1),[§2](https://arxiv.org/html/2607.21671#S2.p6.1),[§2](https://arxiv.org/html/2607.21671#S2.p7.1),[§4\.3\.3](https://arxiv.org/html/2607.21671#S4.SS3.SSS3.p1.8)\.
- Y\. Gal and Z\. Ghahramani \(2016\)Dropout as a bayesian approximation: representing model uncertainty in deep learning\.InInternational Conference on Machine Learning \(ICML\),pp\. 1050–1059\.Cited by:[§2](https://arxiv.org/html/2607.21671#S2.p9.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InInternational conference on machine learning,pp\. 1321–1330\.Cited by:[§4\.3\.3](https://arxiv.org/html/2607.21671#S4.SS3.SSS3.p2.3)\.
- S\. Han, J\. Pool, J\. Tran, and W\. J\. Dally \(2015\)Learning both weights and connections for efficient neural networks\.InAdvances in Neural Information Processing Systems \(NIPS\),Vol\.28,pp\. 1135–1143\.Cited by:[§1](https://arxiv.org/html/2607.21671#S1.p1.1),[§1](https://arxiv.org/html/2607.21671#S1.p2.1),[§2](https://arxiv.org/html/2607.21671#S2.p5.1)\.
- Y\. Hong \(2013\)On computing the distribution function for the poisson binomial distribution\.Computational Statistics & Data Analysis59,pp\. 41–51\.Cited by:[§3\.4\.2](https://arxiv.org/html/2607.21671#S3.SS4.SSS2.p3.1),[§3\.4\.2](https://arxiv.org/html/2607.21671#S3.SS4.SSS2.p4.1)\.
- K\. Hornik, M\. Stinchcombe, and H\. White \(1989\)Multilayer feedforward networks are universal approximators\.Neural Networks2\(5\),pp\. 359–366\.Cited by:[§1](https://arxiv.org/html/2607.21671#S1.p1.1)\.
- E\. Jang, S\. Gu, and B\. Poole \(2017\)Categorical reparameterization with gumbel\-softmax\.In5th International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=rkE3y85ee)Cited by:[§3\.3\.2](https://arxiv.org/html/2607.21671#S3.SS3.SSS2.p18.6)\.
- S\. Jantre, S\. Bhattacharya, and T\. Maiti \(2023\)Layer adaptive node selection in bayesian neural networks: statistical guarantees and implementation details\.Neural Networks167,pp\. 309–330\.External Links:ISSN 0893\-6080,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.neunet.2023.08.029),[Link](https://www.sciencedirect.com/science/article/pii/S0893608023004495)Cited by:[§A\.1](https://arxiv.org/html/2607.21671#A1.SS1.p1.4),[§1](https://arxiv.org/html/2607.21671#S1.p1.1),[§1](https://arxiv.org/html/2607.21671#S1.p3.1),[§2](https://arxiv.org/html/2607.21671#S2.p11.1),[§2](https://arxiv.org/html/2607.21671#S2.p12.1),[§3\.1](https://arxiv.org/html/2607.21671#S3.SS1.p2.4),[§3\.1](https://arxiv.org/html/2607.21671#S3.SS1.p5.3),[§3\.2\.1](https://arxiv.org/html/2607.21671#S3.SS2.SSS1.p1.3),[§3\.2\.2](https://arxiv.org/html/2607.21671#S3.SS2.SSS2.p1.6),[§3\.2\.2](https://arxiv.org/html/2607.21671#S3.SS2.SSS2.p8.1),[§3\.2\.4](https://arxiv.org/html/2607.21671#S3.SS2.SSS4.p1.5),[§3\.2\.4](https://arxiv.org/html/2607.21671#S3.SS2.SSS4.p5.4),[§3\.2](https://arxiv.org/html/2607.21671#S3.SS2.p1.3),[§3\.3\.1](https://arxiv.org/html/2607.21671#S3.SS3.SSS1.p1.1),[§3\.3\.2](https://arxiv.org/html/2607.21671#S3.SS3.SSS2.p1.4),[§3\.3\.2](https://arxiv.org/html/2607.21671#S3.SS3.SSS2.p18.6),[§3\.3](https://arxiv.org/html/2607.21671#S3.SS3.p1.7),[§3\.4\.1](https://arxiv.org/html/2607.21671#S3.SS4.SSS1.p1.2),[§3\.4\.1](https://arxiv.org/html/2607.21671#S3.SS4.SSS1.p3.2),[§3\.4\.1](https://arxiv.org/html/2607.21671#S3.SS4.SSS1.p4.9),[§3\.5\.1](https://arxiv.org/html/2607.21671#S3.SS5.SSS1.p2.1),[§3\.5](https://arxiv.org/html/2607.21671#S3.SS5.p1.1),[2nd item](https://arxiv.org/html/2607.21671#S4.I1.i2.p1.1),[3rd item](https://arxiv.org/html/2607.21671#S4.I1.i3.p1.1),[§4\.1\.1](https://arxiv.org/html/2607.21671#S4.SS1.SSS1.p2.4),[§4\.2\.1](https://arxiv.org/html/2607.21671#S4.SS2.SSS1.p1.6),[§4\.2\.2](https://arxiv.org/html/2607.21671#S4.SS2.SSS2.p1.1),[§4\.2\.3](https://arxiv.org/html/2607.21671#S4.SS2.SSS3.p1.2),[§4\.2](https://arxiv.org/html/2607.21671#S4.SS2.p1.1),[§4\.3\.3](https://arxiv.org/html/2607.21671#S4.SS3.SSS3.p1.8),[§4\.4\.1](https://arxiv.org/html/2607.21671#S4.SS4.SSS1.p2.1),[§4\.4\.2](https://arxiv.org/html/2607.21671#S4.SS4.SSS2.p3.1),[§4\.4](https://arxiv.org/html/2607.21671#S4.SS4.p1.2),[§4\.5\.3](https://arxiv.org/html/2607.21671#S4.SS5.SSS3.p1.1),[§4\.5](https://arxiv.org/html/2607.21671#S4.SS5.p1.1),[§4](https://arxiv.org/html/2607.21671#S4.p1.1)\.
- A\. Kendall and Y\. Gal \(2017\)What uncertainties do we need in bayesian deep learning for computer vision?\.Advances in neural information processing systems30\.Cited by:[§4\.3\.1](https://arxiv.org/html/2607.21671#S4.SS3.SSS1.p4.1)\.
- N\. Kuhn, A\. Weyrauch, L\. Heyen, A\. Streit, M\. Götz, and C\. Debus \(2026\)Bayesian lottery ticket hypothesis\.arXiv preprint arXiv:2602\.18825\.Cited by:[§1](https://arxiv.org/html/2607.21671#S1.p2.1),[§1](https://arxiv.org/html/2607.21671#S1.p3.1),[§2](https://arxiv.org/html/2607.21671#S2.p7.1),[§3\.4\.1](https://arxiv.org/html/2607.21671#S3.SS4.SSS1.p4.9)\.
- Y\. LeCun, J\. Denker, and S\. Solla \(1989\)Optimal brain damage\.InAdvances in Neural Information Processing Systems \(NIPS\),Vol\.2,pp\. 598–605\.Cited by:[§2](https://arxiv.org/html/2607.21671#S2.p5.1)\.
- C\. J\. Maddison, A\. Mnih, and Y\. W\. Teh \(2017\)The concrete distribution: a continuous relaxation of discrete random variables\.In5th International Conference on Learning Representations \(ICLR\),External Links:[Link](https://openreview.net/forum?id=S1j7snclg)Cited by:[§3\.3\.2](https://arxiv.org/html/2607.21671#S3.SS3.SSS2.p18.6)\.
- F\. Mlambo \(2024\)What are bayesian neural networks?\.Note:Preprint on ResearchGateCited by:[§2](https://arxiv.org/html/2607.21671#S2.p11.1)\.
- D\. Molchanov, A\. Ashukha, and D\. Vetrov \(2017\)Variational dropout sparsifies deep neural networks\.InInternational Conference on Machine Learning \(ICML\),pp\. 2498–2507\.Cited by:[§2](https://arxiv.org/html/2607.21671#S2.p10.1)\.
- R\. M\. Neal \(1996\)Bayesian learning for neural networks\.Vol\.118,Springer Science & Business Media\.Cited by:[§2](https://arxiv.org/html/2607.21671#S2.p8.1),[§2](https://arxiv.org/html/2607.21671#S2.p9.1)\.
- J\. Rachwan, D\. Zügner, B\. Charpentier, S\. Geisler, M\. Ayle, and S\. Günnemann \(2022\)Winning the lottery ahead of time: efficient early network pruning\.InInternational Conference on Machine Learning,pp\. 18293–18309\.Cited by:[§2](https://arxiv.org/html/2607.21671#S2.p7.1)\.
- E\. P\. Xing, M\. I\. Jordan, and S\. Russell \(2012\)A generalized mean field algorithm for variational inference in exponential families\.arXiv preprint arXiv:1212\.2512\.Cited by:[§2](https://arxiv.org/html/2607.21671#S2.p8.1),[§3\.3](https://arxiv.org/html/2607.21671#S3.SS3.p3.3)\.

## Appendix AGumbel\-Softmax approximation and weights hyperparameter specification

The approximation of the neural atom selection indicatorvjlv^\{l\}\_\{j\}by the Gumbel\-Softmax approximationv​j~l\\tilde\{vj\}^\{l\}whose distribution isq​\(v~jl\)∼GS​\(π~jl,τ\)q\(\\tilde\{v\}\_\{j\}^\{l\}\)\\sim\\text\{GS\}\(\\tilde\{\\pi\}\_\{j\}^\{l\},\\tau\)is such that:

v~jl=11\+exp⁡\(ηl​jτ\)=σ​\(ηl​jτ\)\\tilde\{v\}\_\{j\}^\{l\}=\\frac\{1\}\{1\+\\exp\\\!\\left\(\\frac\{\\eta\_\{lj\}\}\{\\tau\}\\right\)\}=\\sigma\\\!\\left\(\\frac\{\\eta\_\{lj\}\}\{\\tau\}\\right\)
ηl​j=log⁡π~jl1−π~jl\+log⁡αl​j1−αl​j\\eta\_\{lj\}=\\log\\frac\{\\tilde\{\\pi\}\_\{j\}^\{l\}\}\{1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\}\+\\log\\frac\{\\alpha\_\{lj\}\}\{1\-\\alpha\_\{lj\}\}
αl​j∼𝒰​\(0,1\)\\alpha\_\{lj\}\\sim\\mathcal\{U\}\(0,1\)
This approximation keep unchanged the role ofvjlv\_\{j\}^\{l\}while guaranteeing continuity for easier gradient calculations\. In fact:

v~jl⟶\{0if​ηl​j<01if​ηl​j\>0τ⟶0v~jl⟶12,τ⟶∞\\tilde\{v\}\_\{j\}^\{l\}\\longrightarrow\\begin\{cases\}0&\\text\{if \}\\eta\_\{lj\}<0\\\\ 1&\\text\{if \}\\eta\_\{lj\}\>0\\end\{cases\}\\quad\\tau\\longrightarrow 0\\qquad\\qquad\\tilde\{v\}\_\{j\}^\{l\}\\longrightarrow\\frac\{1\}\{2\},\\quad\\tau\\longrightarrow\\infty
More interestingly, the thing is that:

v~jl\>0\.5\\displaystyle\\tilde\{v\}^\{l\}\_\{j\}\>0\.5⇔σ​\(ηi​jτ\)\>0\.5\\displaystyle\\iff\\sigma\(\\frac\{\\eta\_\{ij\}\}\{\\tau\}\)\>0\.5⇔ηi​jτ\>0\\displaystyle\\iff\\frac\{\\eta\_\{ij\}\}\{\\tau\}\>0⇔ηi​j\>0\\displaystyle\\iff\\eta\_\{ij\}\>0⇔log⁡π~jl1−π~jl\+log⁡αl​j1−αl​j\>0\\displaystyle\\iff\\log\\frac\{\\tilde\{\\pi\}\_\{j\}^\{l\}\}\{1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\}\+\\log\\frac\{\\alpha\_\{lj\}\}\{1\-\\alpha\_\{lj\}\}\>0⇔π~jl​αl​j\(1−π~jl\)​\(1−αl​j\)\>1\\displaystyle\\iff\\frac\{\\tilde\{\\pi\}\_\{j\}^\{l\}\\,\\alpha\_\{lj\}\}\{\(1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\)\(1\-\\alpha\_\{lj\}\)\}\>1⇔π~jl​αl​j\>1−π~jl−αl​j\+π~jl​αl​j\\displaystyle\\iff\\tilde\{\\pi\}\_\{j\}^\{l\}\\,\\alpha\_\{lj\}\>1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\-\\alpha\_\{lj\}\+\\tilde\{\\pi\}\_\{j\}^\{l\}\\,\\alpha\_\{lj\}⇔0\>1−π~jl−αl​j\\displaystyle\\iff 0\>1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\-\\alpha\_\{lj\}⇔αl​j\>1−π~jl\\displaystyle\\iff\\alpha\_\{lj\}\>1\-\\tilde\{\\pi\}\_\{j\}^\{l\}And, sinceαl​j∼𝒰​\(0,1\)\\alpha\_\{lj\}\\sim\\mathcal\{U\}\(0,1\), then:

P​\(v~jl\>0\.5\)\\displaystyle P\(\\tilde\{v\}\_\{j\}^\{l\}\>0\.5\)=P​\(αl​j≥1−π~jl\)\\displaystyle=P\(\\alpha\_\{lj\}\\geq 1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\)=1−P​\(αl​j≤1−π~jl\)\\displaystyle=1\-P\(\\alpha\_\{lj\}\\leq 1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\)=\[1−0−\(1−π~jl\)1−0\]\\displaystyle=\\left\[1\-\\frac\{0\-\(1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\)\}\{1\-0\}\\right\]=1−\(1−π~jl\)−01−0\\displaystyle=1\-\\frac\{\(1\-\\tilde\{\\pi\}\_\{j\}^\{l\}\)\-0\}\{1\-0\}=π~jl\\displaystyle=\\tilde\{\\pi\}\_\{j\}^\{l\}
Sov~jl\>0\.5\\tilde\{v\}\_\{j\}^\{l\}\>0\.5is equivalent tovjl=1v\_\{j\}^\{l\}=1indicating an activated neural atom\.

### A\.1Spike and Slab hyperparameters

To compute the posterior of weight and bias, the method is finalized by considering the parametrization of𝒩​\(μjl,diag​\(σjl2\)\)\\mathcal\{N\}\(\\mu\_\{j\}^\{l\},\\text\{diag\}\(\{\\sigma\_\{j\}^\{l\}\}^\{2\}\)\)byμjl\+σjl⊙ℰjl\\mu\_\{j\}^\{l\}\+\\sigma\_\{j\}^\{l\}\\odot\\mathcal\{E\}\_\{j\}^\{l\}whereℰjl∼𝒩​\(0,I\)\\mathcal\{E\}\_\{j\}^\{l\}\\sim\\mathcal\{N\}\(0,I\)and⊙\\odotthe Hadamard product \(Jantreet al\.\([2023](https://arxiv.org/html/2607.21671#bib.bib2)\)\)\.