S-GAI: Spectral Geometry-Aware Initialization for Sigmoidal MLPs -- From Dataset Geometry to Network Weights

# Spectral Geometry-Aware Initialization for Sigmoidal MLPs From Dataset Geometry to Network Weights
Source: [https://arxiv.org/html/2606.28444](https://arxiv.org/html/2606.28444)
11institutetext:easonchu@stat\.sinica\.edu\.tw
Institute of Statistical Science, Academia Sinica, Taipei, Taiwan \(R\.O\.C\.\)###### Abstract

Classical universal approximation theorems establish the expressive power of sigmoidal multilayer perceptrons, but they do not prescribe how initial weights should encode the geometry of a data distribution\. We propose S\-GAI, a spectral geometry\-aware initialization framework for one\-hidden\-layer sigmoidal MLPs\. Starting from the constructive idea that sigmoid units can act as smooth half\-space gates, we move from hand\-specified planar geometry to class\-wise spectral geometry estimated from image data\. For each class, SVD provides a mean, principal directions, and spectral scales\. An energy threshold selects the retained directions, and each retained direction is represented by two sigmoid gates\. These class\-specific gates form a shared hidden layer initialized directly from the training set\. We also formulate a SVD\-based subspace classifier as a non\-neural geometric reference, which tests whether the estimated spectral class geometry is already discriminative before being embedded into the MLP\. Experiments on MNIST, Fashion\-MNIST, and a more challenging CIFAR\-10 test show that the S\-GAI\-initialized MLP starts from a substantially more informative hidden state than Xavier initialization and reaches comparable final accuracy under full training\. When the hidden layer is frozen, training only the output layer still gives stronger performance than frozen random gates, providing evidence that S\-GAI effectively embeds class\-wise spectral geometry into the MLP\.

## 1Introduction

The universal approximation theorem \(UAT\)\[[3](https://arxiv.org/html/2606.28444#bib.bib3)\]is often read as an expressive\-power statement: with enough hidden units, a sigmoidal MLP can approximate any continuous target on a compact domain\. In its classical form, the approximant is a finite sum

fN​\(x\)=∑j=1Nαj​σ​\(wj⊤​x\+bj\),x∈K⊂ℝd,f\_\{N\}\(x\)=\\sum\_\{j=1\}^\{N\}\\alpha\_\{j\}\\sigma\(w\_\{j\}^\{\\top\}x\+b\_\{j\}\),\\qquad x\\in K\\subset\\mathbb\{R\}^\{d\},\(1\)where the parameters are allowed to vary freely\. This form is powerful, but it leaves open a practical question: if the data occupy a structured region of space, can the initial weights already encode part of that geometry?

Most standard initializations, including Xavier/Glorot initialization\[[6](https://arxiv.org/html/2606.28444#bib.bib6)\], are designed to stabilize signal propagation and gradient scale\. They are not intended to represent class geometry before training\. For image classification, however, even a simple benchmark such as MNIST has visible class\-dependent structure: each digit class has a mean shape, dominant deformation directions, and residual directions that are less stable\. The central idea of this work is that initialization need not be blind to such structure\.

This paper connects two levels of geometry\. The first is a compact constructive background for sigmoidal networks\. This work builds on earlier work by Chu and Kuo\[[2](https://arxiv.org/html/2606.28444#bib.bib2)\], which developed a planar sigmoidal construction for organizing half\-spaces, polytopes, and finite covers through a boundary\-first, tropical\-geometry\-inspired viewpoint\. We use this component only as geometric motivation: it explains how prescribed regions can be translated into network weights when the geometry is known explicitly\.

The main contribution of this paper is the second level: a data\-driven high\-dimensional instantiation for image data\. Instead of assuming specified planar covers, we estimate class geometry from samples\. For each class, we compute a centered SVD/PCA basis\[[10](https://arxiv.org/html/2606.28444#bib.bib10)\], select an energy\-adaptive rank, and formulate a SVD\-based subspace classifier\. This classifier is not the neural model; it serves as a geometric reference that tests whether class\-wise spectral coordinates already separate the classes\.

We then compile the same spectral geometry into a one\-hidden\-layer sigmoidal MLP\. Each retained SVD direction defines a normalized coordinate around a class mean\. Compatibility with the class is expressed as a slab constraint on this coordinate, and each slab is represented by two sigmoid gates, one for each side of the interval\. Thus the hidden layer is partitioned into class\-specific groups, with two hidden units per retained direction, and the output layer initially aggregates the gates belonging to each class\. This gives S\-GAI, an explicit spectral geometry\-aware initializer whose width is controlled by the retained spectral ranks\.

Our contributions are:

- •We provide a concise bridge from UAT finite sums to smooth half\-space and cover\-based sigmoid gates, building on the planar sigmoidal construction of\[[2](https://arxiv.org/html/2606.28444#bib.bib2)\];
- •We introduce a SVD\-based subspace classifier that serves as a non\-neural reference for testing whether class\-wise spectral geometry is discriminative and selecting the proper parameters for S\-GAI;
- •We propose S\-GAI, a data\-driven spectral geometry\-aware initializer for MLPs, instantiated with class\-wise SVD; each retained class direction contributes two hidden units, and the hidden width is determined by an energy thresholdτ\\tau;
- •We evaluate S\-GAI through matched comparisons with Xavier initialization on MNIST, Fashion\-MNIST, and CIFAR\-10\. Under zero\-epoch and frozen\-hidden protocols, S\-GAI slab gates provide substantially more informative representations than matched random gates, while full training reaches comparable final accuracy under the same architecture and optimizer\.

![Refer to caption](https://arxiv.org/html/2606.28444v1/figures/flowchart.png)Figure 1:Method flow\. The top row summarizes the constructive view: finite\-sum sigmoidal networks can be organized as half\-space gates and then as polytope\- or cover\-based initializers\. The bottom row shows the data\-driven instantiation used for high\-dimensional images\. Class\-wise SVD estimates the mean, directions, and scales; an energy threshold selects the retained rankmk​\(τ\)m\_\{k\}\(\\tau\); each retained direction is compiled into two slab gateshk,i\+h^\{\+\}\_\{k,i\}andhk,i−h^\{\-\}\_\{k,i\}; and the resulting gates are placed in a class\-grouped hidden layer of widthH=2​∑kmkH=2\\sum\_\{k\}m\_\{k\}\. The SVD\-based subspace classifier is shown only as a non\-neural geometric reference for the selection of energy thresholdτ\\tau\.
## 2From Finite Sums to Geometric Gates

LetK⊂ℝdK\\subset\\mathbb\{R\}^\{d\}be compact and letσ​\(t\)=\(1\+e−t\)−1\\sigma\(t\)=\(1\+e^\{\-t\}\)^\{\-1\}\. Cybenko’s theorem states that finite sums of the form \([1](https://arxiv.org/html/2606.28444#S1.E1)\) are dense inC​\(K\)C\(K\)for sigmoidal activations\[[3](https://arxiv.org/html/2606.28444#bib.bib3)\]; related and extended results were given by Funahashi\[[5](https://arxiv.org/html/2606.28444#bib.bib5)\], Hornik*et al\.*\[[9](https://arxiv.org/html/2606.28444#bib.bib9)\], Hornik\[[8](https://arxiv.org/html/2606.28444#bib.bib8)\], Leshno*et al\.*\[[15](https://arxiv.org/html/2606.28444#bib.bib15)\], and Barron\[[1](https://arxiv.org/html/2606.28444#bib.bib1)\]; see also Pinkus\[[17](https://arxiv.org/html/2606.28444#bib.bib17)\]\. These theorems ensure representability, but they do not specify an initialization tied to a particular decision region\.

Following the planar construction in\[[2](https://arxiv.org/html/2606.28444#bib.bib2)\], we use the following elementary gate as the geometric primitive\. Fora∈ℝda\\in\\mathbb\{R\}^\{d\},c∈ℝc\\in\\mathbb\{R\}, and sharpnessκ\>0\\kappa\>0, define

ga,c,κ​\(x\)=σ​\(κ​\(c−a⊤​x\)\)\.g\_\{a,c,\\kappa\}\(x\)=\\sigma\\bigl\(\\kappa\(c\-a^\{\\top\}x\)\\bigr\)\.\(2\)This unit is close to one inside the half\-spacea⊤​x≤ca^\{\\top\}x\\leq cand close to zero outside it\.

###### Lemma 1\(Half\-space gate\)

Letδ\>0\\delta\>0\. Ifc−a⊤​x≥δc\-a^\{\\top\}x\\geq\\delta, thenga,c,κ​\(x\)≥1−exp⁡\(−κ​δ\)g\_\{a,c,\\kappa\}\(x\)\\geq 1\-\\exp\(\-\\kappa\\delta\)\. Ifa⊤​x−c≥δa^\{\\top\}x\-c\\geq\\delta, thenga,c,κ​\(x\)≤exp⁡\(−κ​δ\)g\_\{a,c,\\kappa\}\(x\)\\leq\\exp\(\-\\kappa\\delta\)\.

###### Proof

Fort≥0t\\geq 0,σ​\(t\)=1/\(1\+e−t\)≥1−e−t\\sigma\(t\)=1/\(1\+e^\{\-t\}\)\\geq 1\-e^\{\-t\}\. Fort≤0t\\leq 0,σ​\(t\)≤et\\sigma\(t\)\\leq e^\{t\}\. Apply these inequalities tot=κ​\(c−a⊤​x\)t=\\kappa\(c\-a^\{\\top\}x\)\.

Let a polytope be written as

P=⋂ℓ=1m\{x:aℓ⊤​x≤cℓ\}\.P=\\bigcap\_\{\\ell=1\}^\{m\}\\\{x:a\_\{\\ell\}^\{\\top\}x\\leq c\_\{\\ell\}\\\}\.\(3\)Using one gate per supporting half\-space, define

GP​\(x\)=∑ℓ=1mσ​\(κ​\(cℓ−aℓ⊤​x\)\)−\(m−12\)\.G\_\{P\}\(x\)=\\sum\_\{\\ell=1\}^\{m\}\\sigma\\bigl\(\\kappa\(c\_\{\\ell\}\-a\_\{\\ell\}^\{\\top\}x\)\\bigr\)\-\\left\(m\-\\frac\{1\}\{2\}\\right\)\.\(4\)
###### Proposition 1\(Robust polytope classifier\)

Assumem​exp⁡\(−κ​δ\)<1/2m\\exp\(\-\\kappa\\delta\)<1/2\. Ifxxsatisfies all inequalities with margin at leastδ\\delta, thenGP​\(x\)\>0G\_\{P\}\(x\)\>0\. Ifxxviolates at least one inequality with margin at leastδ\\delta, thenGP​\(x\)<0G\_\{P\}\(x\)<0\.

###### Proof

InsidePPwith marginδ\\delta, Lemma[1](https://arxiv.org/html/2606.28444#Thmlemma1)gives

GP​\(x\)≥m​\(1−e−κ​δ\)−\(m−12\)\>0\.G\_\{P\}\(x\)\\geq m\(1\-e^\{\-\\kappa\\delta\}\)\-\\left\(m\-\\frac\{1\}\{2\}\\right\)\>0\.If one constraint is violated by marginδ\\delta, then the corresponding gate is at moste−κ​δe^\{\-\\kappa\\delta\}, while all other gates are at most one\. Thus

GP​\(x\)≤\(m−1\)\+e−κ​δ−\(m−12\)<0\.G\_\{P\}\(x\)\\leq\(m\-1\)\+e^\{\-\\kappa\\delta\}\-\\left\(m\-\\frac\{1\}\{2\}\\right\)<0\.

Finite unions are obtained by a second sigmoidal aggregation\. ForC=⋃r=1RPrC=\\bigcup\_\{r=1\}^\{R\}P\_\{r\}, define

FC​\(x\)=∑r=1Rσ​\(λ​GPr​\(x\)\)−12\.F\_\{C\}\(x\)=\\sum\_\{r=1\}^\{R\}\\sigma\\bigl\(\\lambda G\_\{P\_\{r\}\}\(x\)\\bigr\)\-\\frac\{1\}\{2\}\.\(5\)Ifxxlies robustly inside one component, one term is close to one; ifxxlies robustly outside all components, all terms are close to zero for sufficiently largeλ\\lambda\. This gives a sigmoidal network in the same finite\-sum spirit as UAT, but with weights chosen from an explicit geometric description\.

The construction is inspired by the way tropical geometry represents polyhedral structure through max\-plus affine pieces\[[16](https://arxiv.org/html/2606.28444#bib.bib16)\]\. ReLU networks inherit closely related piecewise\-linear subdivisions and have been studied through tropical rational maps\[[19](https://arxiv.org/html/2606.28444#bib.bib19)\]\. We do not use ReLU networks; the point is instead to keep a smooth sigmoid activation while borrowing a boundary\-first design principle: identify geometric boundary primitives first, then compile them into weights\.

For a general compact target region in the plane, one may approximate the region by a finite cover and polygonal components, then apply \([4](https://arxiv.org/html/2606.28444#S2.E4)\) and \([5](https://arxiv.org/html/2606.28444#S2.E5)\)\. The planar examples and visual demonstrations are reported in prior work\[[2](https://arxiv.org/html/2606.28444#bib.bib2)\]\. In this paper, the role of this section is to motivate the high\-dimensional replacement: when the geometry is not hand\-specified, estimate it from data\.

## 3Spectral Class Geometry and SVD\-Referenced Subspace Classifier

For image data, class regions are not given as planar covers\. We therefore estimate class geometry from the training set\. For classkk, letμk∈ℝd\\mu\_\{k\}\\in\\mathbb\{R\}^\{d\}be the class mean and letXk∈ℝnk×dX\_\{k\}\\in\\mathbb\{R\}^\{n\_\{k\}\\times d\}contain centered samplesxi−μkx\_\{i\}\-\\mu\_\{k\}\. Compute

Xk=Uk​Sk​Vk⊤,Vk=\[vk,1,…,vk,d\],X\_\{k\}=U\_\{k\}S\_\{k\}V\_\{k\}^\{\\top\},\\qquad V\_\{k\}=\[v\_\{k,1\},\\ldots,v\_\{k,d\}\],\(6\)whereSk=diag⁡\(sk,1,…\)S\_\{k\}=\\operatorname\{diag\}\(s\_\{k,1\},\\ldots\)\. The empirical standard deviation along directionvk,iv\_\{k,i\}is

σ^k,i=sk,i/nk−1\.\\widehat\{\\sigma\}\_\{k,i\}=s\_\{k,i\}/\\sqrt\{n\_\{k\}\-1\}\.\(7\)For an energy thresholdτ∈\(0,1\)\\tau\\in\(0,1\), choose

mk​\(τ\)=min⁡\{m:∑i=1msk,i2∑isk,i2≥τ\}\.m\_\{k\}\(\\tau\)=\\min\\left\\\{m:\\frac\{\\sum\_\{i=1\}^\{m\}s\_\{k,i\}^\{2\}\}\{\\sum\_\{i\}s\_\{k,i\}^\{2\}\}\\geq\\tau\\right\\\}\.\(8\)
Withmk=mk​\(τ\)m\_\{k\}=m\_\{k\}\(\\tau\), the SVD\-Mahalanobis reference score is

dk​\(x\)=∑i=1mk\(vk,i⊤​\(x−μk\)σ^k,i\)2\+λ​‖\(I−Vk,mk​Vk,mk⊤\)​\(x−μk\)‖22σ^k,mk2,\\begin\{split\}d\_\{k\}\(x\)&=\\sum\_\{i=1\}^\{m\_\{k\}\}\\left\(\\frac\{v\_\{k,i\}^\{\\top\}\(x\-\\mu\_\{k\}\)\}\{\\widehat\{\\sigma\}\_\{k,i\}\}\\right\)^\{2\}\\\\ &\\quad\+\\lambda\\frac\{\\left\\\|\(I\-V\_\{k,m\_\{k\}\}V\_\{k,m\_\{k\}\}^\{\\top\}\)\(x\-\\mu\_\{k\}\)\\right\\\|\_\{2\}^\{2\}\}\{\\widehat\{\\sigma\}\_\{k,m\_\{k\}\}^\{2\}\},\\end\{split\}\(9\)whereVk,mkV\_\{k,m\_\{k\}\}contains the retained directions andλ≥0\\lambda\\geq 0penalizes residual energy outside the retained subspace\.

This score defines a subspace classifier based on the estimated SVD geometry of each class\. The first term is a truncated Mahalanobis distance inside the retained class subspace: deviations along high\-variance class directions are normalized by their empirical spectral scales\. The second term measures the normalized residual energy outside the retained subspace\. This complement penalty is important because a sample may have moderate coordinates along the retained directions of a class while still being poorly explained by that class subspace as a whole\. The residual term therefore acts as a rejection mechanism for samples that lie far from the retained spectral model\. Empirically, removing this term by settingλ=0\\lambda=0leads to a large accuracy drop of more than 13 percentage points, as shown in Fig\.[2](https://arxiv.org/html/2606.28444#S3.F2)\.

Classification is by

y^​\(x\)=arg​minkdk​\(x\)\.\\widehat\{y\}\(x\)=\\mathop\{\\rm arg\\,min\}\_\{k\}d\_\{k\}\(x\)\.\(10\)
The role of this classifier is twofold in this work\. First, it provides a non\-neural reference for testing whether class\-wise spectral geometry is already discriminative\. Second, it identifies the retained directions, ranks, and spectral scales that will later be compiled into sigmoid slab gates\. In this sense, the classifier is not a competing neural architecture, but a geometric intermediate between the estimated class subspaces and the initialized MLP\.

![Refer to caption](https://arxiv.org/html/2606.28444v1/x1.png)

![Refer to caption](https://arxiv.org/html/2606.28444v1/x2.png)

Figure 2:SVD\-based Subspace Classifier reference for MNIST\. The energy threshold selects class\-dependent ranks, and the complement penalty rejects samples that are not well explained by a class subspace\. This reference is non\-neural; it is used to validate the spectral geometry before compiling it into an MLP\.
## 4Compiling Spectral Slabs into a Sigmoid MLP

We now compile the class\-wise SVD geometry into a one\-hidden\-layer sigmoid MLP\. We refer to the resulting procedure as S\-GAI, short for Spectral Geometry\-Aware Initialization\. For classkkand retained directionii, define the normalized spectral coordinate

zk,i​\(x\)=vk,i⊤​\(x−μk\)σ^k,i\.z\_\{k,i\}\(x\)=\\frac\{v\_\{k,i\}^\{\\top\}\(x\-\\mu\_\{k\}\)\}\{\\widehat\{\\sigma\}\_\{k,i\}\}\.\(11\)A sample compatible with classkkshould have moderate values ofzk,i​\(x\)z\_\{k,i\}\(x\)along the retained directions\. We model the interval\[−ρ,ρ\]\[\-\\rho,\\rho\]by two sigmoid slab gates:

hk,i\+​\(x\)=σ​\(β​\(ρ−zk,i​\(x\)\)\),hk,i−​\(x\)=σ​\(β​\(ρ\+zk,i​\(x\)\)\),h\_\{k,i\}^\{\+\}\(x\)=\\sigma\\bigl\(\\beta\(\\rho\-z\_\{k,i\}\(x\)\)\\bigr\),\\qquad h\_\{k,i\}^\{\-\}\(x\)=\\sigma\\bigl\(\\beta\(\\rho\+z\_\{k,i\}\(x\)\)\\bigr\),\(12\)whereρ\>0\\rho\>0is the slab half\-width andβ\>0\\beta\>0controls sharpness\. Both gates are near one whenzk,i​\(x\)∈\[−ρ,ρ\]z\_\{k,i\}\(x\)\\in\[\-\\rho,\\rho\]\.

Equation \([12](https://arxiv.org/html/2606.28444#S4.E12)\) gives explicit hidden\-layer weights:

wk,i\+=−β​vk,i/σ^k,i,bk,i\+=β​ρ\+β​vk,i⊤​μk/σ^k,i,wk,i−=β​vk,i/σ^k,i,bk,i−=β​ρ−β​vk,i⊤​μk/σ^k,i\.\\begin\{array\}\[\]\{ll\}w\_\{k,i\}^\{\+\}=\-\\beta v\_\{k,i\}/\\widehat\{\\sigma\}\_\{k,i\},&b\_\{k,i\}^\{\+\}=\\beta\\rho\+\\beta v\_\{k,i\}^\{\\top\}\\mu\_\{k\}/\\widehat\{\\sigma\}\_\{k,i\},\\\\\[5\.69054pt\] w\_\{k,i\}^\{\-\}=\\phantom\{\-\}\\beta v\_\{k,i\}/\\widehat\{\\sigma\}\_\{k,i\},&b\_\{k,i\}^\{\-\}=\\beta\\rho\-\\beta v\_\{k,i\}^\{\\top\}\\mu\_\{k\}/\\widehat\{\\sigma\}\_\{k,i\}\.\\end\{array\}\(13\)Thus each retained SVD direction contributes two hidden units: one gate for the upper half\-space constraintzk,i≤ρz\_\{k,i\}\\leq\\rhoand one gate for the lower half\-space constraintzk,i≥−ρz\_\{k,i\}\\geq\-\\rho\. The hidden layer is partitioned into class\-specific groups,

ℋ=ℋ0​\|ℋ1\|​⋯\|ℋ9,\|ℋk\|=2​mk​\(τ\),\\mathcal\{H\}=\\mathcal\{H\}\_\{0\}\\,\|\\,\\mathcal\{H\}\_\{1\}\\,\|\\,\\cdots\\,\|\\,\\mathcal\{H\}\_\{9\},\\qquad\|\\mathcal\{H\}\_\{k\}\|=2m\_\{k\}\(\\tau\),and its total width is

H​\(τ\)=2​∑k=09mk​\(τ\)\.H\(\\tau\)=2\\sum\_\{k=0\}^\{9\}m\_\{k\}\(\\tau\)\.\(14\)The initialized class logit is the normalized average of the class slab responses:

ℓk\(0\)​\(x\)=γ2​mk​∑i=1mk\(hk,i\+​\(x\)\+hk,i−​\(x\)\),\\ell\_\{k\}^\{\(0\)\}\(x\)=\\frac\{\\gamma\}\{2m\_\{k\}\}\\sum\_\{i=1\}^\{m\_\{k\}\}\\bigl\(h\_\{k,i\}^\{\+\}\(x\)\+h\_\{k,i\}^\{\-\}\(x\)\\bigr\),\(15\)with scaleγ\>0\\gamma\>0\.

### 4\.1Experiment Protocols

After initialization, we study two protocols:*full training*, in which all weights are updated, and*frozen\-hidden training*, in which the S\-GAI slab gates \(the hidden layer\) are fixed and only the output layer is trained\.

For each dataset and each thresholdτ\\tau, the Xavier baseline uses exactly the same architecture and hidden widthH​\(τ\)H\(\\tau\)\. The only difference is the initial value of the weights\. In the*full training*protocol, both hidden and output weights are trainable\. In the*frozen\-hidden training*protocol, the randomly initialized hidden layer is fixed and only the output layer is trained\. This isolates the effect of S\-GAI’s data\-driven spectral geometry from parameter count\.

For a one\-hidden\-layer sigmoid MLP with input dimensiondd, hidden widthHH, and1010outputs, the total parameter count is

\(d​H\+H\)\+\(10​H\+10\)=\(d\+11\)​H\+10\.\(dH\+H\)\+\(10H\+10\)=\(d\+11\)H\+10\.\(16\)For MNIST and Fashion\-MNIST,d=784d=784; for CIFAR\-10,d=3072d=3072\. Under frozen\-hidden training, the trainable parameter count is only

## 5Experiments

### 5\.1Datasets and Experimental Setup

We evaluate S\-GAI on three image\-classification datasets\. The main controlled study uses MNIST handwritten digits\[[13](https://arxiv.org/html/2606.28444#bib.bib13),[14](https://arxiv.org/html/2606.28444#bib.bib14)\], flattened tox∈ℝ784x\\in\\mathbb\{R\}^\{784\}\. To test whether the effect is specific to digit shapes, we also evaluate the same protocol on Fashion\-MNIST\[[18](https://arxiv.org/html/2606.28444#bib.bib18)\], which has the same image size and number of classes as MNIST but contains clothing categories\. Finally, we include CIFAR\-10\[[12](https://arxiv.org/html/2606.28444#bib.bib12)\]as a more challenging natural\-image stress test\. For CIFAR\-10, we deliberately apply the method directly to flattened raw RGB pixels,x∈ℝ3072x\\in\\mathbb\{R\}^\{3072\}, without convolutional inductive bias, data augmentation, or pretrained features\.

All neural experiments use one\-hidden\-layer sigmoidal MLPs trained for 20 epochs with Adam\[[11](https://arxiv.org/html/2606.28444#bib.bib11)\]\. We repeat all experiments over multiple random seeds and test the energy thresholdsτ∈\{0\.75,0\.86,0\.90,0\.95\}\\tau\\in\\\{0\.75,0\.86,0\.90,0\.95\\\}\. For each dataset and each threshold, S\-GAI and Xavier models use the same hidden widthH​\(τ\)H\(\\tau\), optimizer, number of epochs, and train/freeze protocol\. The highlight threshold isτ=0\.95\\tau=0\.95for MNIST and Fashion\-MNIST\. For CIFAR\-10, we highlightτ=0\.90\\tau=0\.90, since this setting gives the strongest frozen\-hidden representation in our experiments\.

Tables[1](https://arxiv.org/html/2606.28444#S5.T1)–[4](https://arxiv.org/html/2606.28444#S5.T4)summarize the experimental evidence\. Table[1](https://arxiv.org/html/2606.28444#S5.T1)reports the retained ranks and induced hidden widths\. Table[2](https://arxiv.org/html/2606.28444#S5.T2)compares the non\-neural SVD\-Mahalanobis reference with the zero\-epoch initialized MLP\. Tables[3](https://arxiv.org/html/2606.28444#S5.T3)and[4](https://arxiv.org/html/2606.28444#S5.T4)report the matched train/freeze comparisons at the highlight threshold and across thresholds, respectively\.

Table 1:Energy thresholds determine the class\-dependent SVD ranks used by both the SVD\-based subspace classifier and the S\-GAI initializer\. The hidden width isH=2​∑kmkH=2\\sum\_\{k\}m\_\{k\}because each retained direction gives two sigmoid slab gates\. MNIST and Fashion\-MNIST are flattened toℝ784\\mathbb\{R\}^\{784\}, while CIFAR\-10 is flattened toℝ3072\\mathbb\{R\}^\{3072\}\.Table 2:Geometry\-only performance before gradient updates\. The SVD\-Mahalanobis column reports the non\-neural SVD\-based subspace reference classifier using Eq\. \([9](https://arxiv.org/html/2606.28444#S3.E9)\)\. S\-GAI init reports the zero\-epoch test accuracy of the S\-GAI\-initialized MLP\.Table 3:Matched MLP comparison at the highlight thresholdτ=0\.95\\tau=0\.95for MNIST and Fashion\-MNIST, andτ=0\.90\\tau=0\.90for CIFAR\-10\. Within each dataset, S\-GAI and Xavier models use the same architecture and hidden width; they differ only in initialization and whether the hidden layer is trainable\. The frozen rows test the quality of the initialized representation itself\.Table 4:Effect of the energy threshold on matched\-protocol accuracy across datasets\. Fully trainable models reach similar final accuracy, while the frozen\-hidden comparison consistently favors S\-GAI slab gates over frozen random gates\. CIFAR\-10 is evaluated directly on flattened raw pixels and therefore serves as a natural\-image stress test\.
### 5\.2Results

The key comparison is not S\-GAI against a smaller or larger network; it is S\-GAI against a matched Xavier network with the same architecture\. Across datasets, S\-GAI starts from a substantially more informative state than Xavier initialization\. Atτ=0\.95\\tau=0\.95, the zero\-epoch S\-GAI\-initialized model reaches87\.41%87\.41\\%test accuracy on MNIST and71\.32%71\.32\\%on Fashion\-MNIST, while the matched Xavier models remain near chance level\. On CIFAR\-10, where we highlightτ=0\.90\\tau=0\.90for the raw\-pixel stress test, the S\-GAI\-initialized model reaches20\.32%20\.32\\%before training, compared with9\.97%9\.97\\%for Xavier\.

Under full training, S\-GAI\-initialized and Xavier\-initialized models reach similar final accuracy\. This suggests that S\-GAI is not simply increasing capacity; rather, it changes the starting representation while keeping the architecture fixed\. The frozen\-hidden setting is more diagnostic: when the hidden layer cannot be trained, strong performance indicates that the initialized S\-GAI slab gates already form useful features\. Atτ=0\.95\\tau=0\.95, frozen S\-GAI slab gates reach96\.73%96\.73\\%on MNIST and86\.36%86\.36\\%on Fashion\-MNIST, compared with89\.31%89\.31\\%and80\.52%80\.52\\%for frozen random gates\. On CIFAR\-10, the strongest frozen\-hidden advantage appears atτ=0\.90\\tau=0\.90, where S\-GAI \(frozen\) reaches44\.36%44\.36\\%, compared with36\.53%36\.53\\%for frozen random gates\. This is consistent with the interpretation in\[[2](https://arxiv.org/html/2606.28444#bib.bib2)\]that geometry\-informed initialization can shift part of training from discovering decision boundaries from scratch toward calibrating and refining an already structured representation\.

![Refer to caption](https://arxiv.org/html/2606.28444v1/x3.png)

![Refer to caption](https://arxiv.org/html/2606.28444v1/x4.png)

![Refer to caption](https://arxiv.org/html/2606.28444v1/x5.png)

Figure 3:Accuracy versus trainable parameter count on MNIST, Fashion\-MNIST, and CIFAR\-10, shown from left to right\. S\-GAI and Xavier use matched hidden width at each energy threshold\. Fully trainable models reach similar final accuracy, while the frozen\-hidden comparison evaluates the quality of the fixed hidden representation\.![Refer to caption](https://arxiv.org/html/2606.28444v1/x6.png)

![Refer to caption](https://arxiv.org/html/2606.28444v1/x7.png)

![Refer to caption](https://arxiv.org/html/2606.28444v1/x8.png)

![Refer to caption](https://arxiv.org/html/2606.28444v1/x9.png)

![Refer to caption](https://arxiv.org/html/2606.28444v1/x10.png)

![Refer to caption](https://arxiv.org/html/2606.28444v1/x11.png)

Figure 4:Energy\-rank analysis on MNIST, Fashion\-MNIST, and CIFAR\-10, shown from left to right\. Top: before training, S\-GAI is already more informative than random initialization\. Bottom: after training, fully trainable models are close, while the frozen\-hidden setting reveals the representation value of the S\-GAI slab gates\.![Refer to caption](https://arxiv.org/html/2606.28444v1/x12.png)

![Refer to caption](https://arxiv.org/html/2606.28444v1/x13.png)

![Refer to caption](https://arxiv.org/html/2606.28444v1/x14.png)

Figure 5:Training curves on MNIST, Fashion\-MNIST, and CIFAR\-10, shown from left to right\. MNIST and Fashion\-MNIST useτ=0\.95\\tau=0\.95, while CIFAR\-10 usesτ=0\.90\\tau=0\.90as a raw\-pixel stress test\. Across datasets, S\-GAI starts from a more informative state than the matched Xavier initialization\. Under full training, the models reach similar final accuracy, while the frozen\-hidden protocol highlights the representation quality of S\-GAI\.Figs\.[3](https://arxiv.org/html/2606.28444#S5.F3)–[5](https://arxiv.org/html/2606.28444#S5.F5)summarize the same behavior from complementary views\. The parameter\-count and energy\-rank plots show that fully trainable models are comparable after optimization, while the frozen\-hidden comparison consistently favors S\-GAI\. Fig\.[6](https://arxiv.org/html/2606.28444#S5.F6)provides a direct diagnostic view through confusion matrices and error galleries\. The remaining errors are mostly plausible class confusions, and the before\-training confusion matrices show that S\-GAI already induces structured predictions before gradient updates on MNIST and Fashion\-MNIST\. On the more challenging CIFAR\-10 benchmark, this structure is weaker, but the initialized model remains above chance and avoids the degenerate single\-class prediction pattern observed under Xavier initialization\. After full training, both initializations reach comparable accuracy as in the other cases\.

![Refer to caption](https://arxiv.org/html/2606.28444v1/x15.png)

![Refer to caption](https://arxiv.org/html/2606.28444v1/x16.png)

![Refer to caption](https://arxiv.org/html/2606.28444v1/x17.png)

Figure 6:Diagnostic comparison on MNIST, Fashion\-MNIST, and CIFAR\-10, shown from top to bottom\. The MNIST and Fashion\-MNIST results useτ=0\.95\\tau=0\.95, while CIFAR\-10 usesτ=0\.90\\tau=0\.90, where the frozen\-hidden representation is strongest\. For each dataset, the four confusion matrices compare S\-GAI and the matched Xavier baseline before training and after full training\. The error gallery shows representative misclassified test images from S\-GAI after full training\. Across datasets, fully trained S\-GAI\-initialized and Xavier\-initialized models reach similar final accuracy, while the initialized S\-GAI slab gates already induce structured predictions before optimization, especially on MNIST and Fashion\-MNIST\.

## 6Discussion and Limitations

The proposed construction connects two levels of geometry\. At the UAT level, half\-space gates and finite covers show how a sigmoidal MLP can be programmed from a target region\. At the practical image level, class\-wise SVD estimates dominant directions and spectral scales directly from samples\. The energy thresholdτ\\taubecomes a practical complexity knob: smallτ\\taugives a coarse class model and fewer gates, while largeτ\\tauretains finer variation and uses more parameters\.

The cross\-dataset results support this interpretation\. On MNIST and Fashion\-MNIST, S\-GAI\-initialized networks are already strongly discriminative before training, and the frozen\-hidden protocol shows a clear advantage over frozen random gates\. On CIFAR\-10, the effect is weaker but still informative under the flattened raw\-pixel setting\.

The non\-neural SVD\-Mahalanobis subspace classifier also provides a useful diagnostic for the applicability of S\-GAI\. It uses the same class\-wise means, spectral directions, spectral scales, and retained ranks before they are compiled into sigmoid slab gates\. Therefore, its performance indicates whether the estimated input\-space spectral geometry is already discriminative\. This helps explain the cross\-dataset trend: the method is strongest on MNIST and Fashion\-MNIST, where the spectral reference is already informative, and weaker on CIFAR\-10, where class identity depends more on local texture, pose, background, and higher\-level visual features\. The CIFAR\-10 result therefore delineates the current regime of the method rather than simply serving as a failure case\.

There are also clear limitations\. The SVD geometry is class\-wise and linear; it does not model nonlinear manifolds inside each class\. The slab construction treats retained directions independently and therefore approximates an axis\-aligned spectral polytope in each class basis rather than the full data distribution\. Finally, all experiments use low\-resolution benchmarks and one\-hidden\-layer sigmoidal MLPs\. A natural next step is to transfer the initialization principle to learned feature spaces and architectures with stronger visual inductive bias, such as convolutional neural networks\[[13](https://arxiv.org/html/2606.28444#bib.bib13),[7](https://arxiv.org/html/2606.28444#bib.bib7)\]and Vision Transformers\[[4](https://arxiv.org/html/2606.28444#bib.bib4)\]\.

## 7Conclusion

We presented a geometry\-aware initialization framework for sigmoidal MLPs\. Starting from the original finite\-sum form of UAT, we summarized how smooth half\-space and cover gates can be constructed from target geometry\. For high\-dimensional image data, we estimated class geometry with SVD, formulated a SVD\-Mahalanobis subspace classifier as a non\-neural geometric reference, and compiled retained spectral directions into pairs of sigmoid slab gates\. This gives an explicit data\-geometry\-to\-weights procedure for initializing the hidden layer of a one\-hidden\-layer sigmoid network\.

The experiments compare S\-GAI against matched Xavier baselines across MNIST, Fashion\-MNIST, and CIFAR\-10\. Under full training, S\-GAI\-initialized and Xavier\-initialized models reach comparable final accuracy, so the main contribution is not a larger\-capacity classifier\. The clearest evidence appears before and under constrained training: S\-GAI starts from a much more informative state, and when the hidden layer is frozen, class\-wise spectral gates are consistently more useful than the matched frozen Xavier baseline\. These results support the view that empirical class geometry can be injected into a sigmoidal MLP before gradient\-based training, while the CIFAR\-10 stress test suggests that future work should extend the same principle to feature\-space and convolutional settings\.

## References

- \[1\]Barron, A\.R\.: Universal approximation bounds for superpositions of a sigmoidal function\. IEEE Transactions on Information Theory39\(3\), 930–945 \(1993\)\. https://doi\.org/10\.1109/18\.256500
- \[2\]Chu, Y\.S\., Kuo, Y\.C\.: From universal approximation theorem to tropical geometry of multi\-layer perceptrons \(2025\)\. https://doi\.org/10\.48550/arXiv\.2510\.15012,[https://arxiv\.org/abs/2510\.15012](https://arxiv.org/abs/2510.15012)
- \[3\]Cybenko, G\.: Approximation by superpositions of a sigmoidal function\. Mathematics of Control, Signals and Systems2\(4\), 303–314 \(1989\)\. https://doi\.org/10\.1007/BF02551274
- \[4\]Dosovitskiy, A\., Beyer, L\., Kolesnikov, A\., Weissenborn, D\., Zhai, X\., Unterthiner, T\., Dehghani, M\., Minderer, M\., Heigold, G\., Gelly, S\., Uszkoreit, J\., Houlsby, N\.: An image is worth 16x16 words: Transformers for image recognition at scale\. In: International Conference on Learning Representations \(2021\)
- \[5\]Funahashi, K\.I\.: On the approximate realization of continuous mappings by neural networks\. Neural Networks2\(3\), 183–192 \(1989\)\. https://doi\.org/10\.1016/0893\-6080\(89\)90003\-8
- \[6\]Glorot, X\., Bengio, Y\.: Understanding the difficulty of training deep feedforward neural networks\. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics\. JMLR Workshop and Conference Proceedings, vol\. 9, pp\. 249–256 \(2010\),[https://proceedings\.mlr\.press/v9/glorot10a\.html](https://proceedings.mlr.press/v9/glorot10a.html)
- \[7\]He, K\., Zhang, X\., Ren, S\., Sun, J\.: Deep residual learning for image recognition\. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition\. pp\. 770–778 \(2016\)
- \[8\]Hornik, K\.: Approximation capabilities of multilayer feedforward networks\. Neural Networks4\(2\), 251–257 \(1991\)\. https://doi\.org/10\.1016/0893\-6080\(91\)90009\-T
- \[9\]Hornik, K\., Stinchcombe, M\., White, H\.: Multilayer feedforward networks are universal approximators\. Neural Networks2\(5\), 359–366 \(1989\)\. https://doi\.org/10\.1016/0893\-6080\(89\)90020\-8
- \[10\]Jolliffe, I\.T\.: Principal Component Analysis\. Springer, 2 edn\. \(2002\)\. https://doi\.org/10\.1007/b98835
- \[11\]Kingma, D\.P\., Ba, J\.: Adam: A method for stochastic optimization\. In: International Conference on Learning Representations \(2015\),[https://arxiv\.org/abs/1412\.6980](https://arxiv.org/abs/1412.6980)
- \[12\]Krizhevsky, A\.: Learning multiple layers of features from tiny images\. Tech\. rep\., University of Toronto \(2009\),[https://www\.cs\.toronto\.edu/~kriz/learning\-features\-2009\-TR\.pdf](https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf)
- \[13\]LeCun, Y\., Bottou, L\., Bengio, Y\., Haffner, P\.: Gradient\-based learning applied to document recognition\. Proceedings of the IEEE86\(11\), 2278–2324 \(1998\)\. https://doi\.org/10\.1109/5\.726791
- \[14\]LeCun, Y\., Cortes, C\., Burges, C\.J\.C\.: MNIST handwritten digit database\.[http://yann\.lecun\.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/)\(2010\)
- \[15\]Leshno, M\., Lin, V\.Y\., Pinkus, A\., Schocken, S\.: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function\. Neural Networks6\(6\), 861–867 \(1993\)\. https://doi\.org/10\.1016/S0893\-6080\(05\)80131\-5
- \[16\]Maclagan, D\., Sturmfels, B\.: Introduction to Tropical Geometry, Graduate Studies in Mathematics, vol\. 161\. American Mathematical Society \(2015\)
- \[17\]Pinkus, A\.: Approximation theory of the MLP model in neural networks\. Acta Numerica8, 143–195 \(1999\)\. https://doi\.org/10\.1017/S0962492900002919
- \[18\]Xiao, H\., Rasul, K\., Vollgraf, R\.: Fashion\-mnist: a novel image dataset for benchmarking machine learning algorithms\. arXiv preprint arXiv:1708\.07747 \(2017\)
- \[19\]Zhang, L\., Naitzat, G\., Lim, L\.H\.: Tropical geometry of deep neural networks\. In: Proceedings of the 35th International Conference on Machine Learning\. Proceedings of Machine Learning Research, vol\. 80, pp\. 5824–5832 \(2018\),[https://proceedings\.mlr\.press/v80/zhang18i\.html](https://proceedings.mlr.press/v80/zhang18i.html)