Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests

# Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability–Observability Tests
Source: [https://arxiv.org/html/2607.05457](https://arxiv.org/html/2607.05457)
###### Abstract

Deep neural networks often contain substantial hidden\-state redundancy, but most compression methods operate directly on weights, neurons, or quantised representations without explicitly characterising the dynamical role of internal states\. This paper proposes a controllability–observability framework for empirical state\-order reduction of deep neural networks\. By viewing a trained network as a depth\-indexed nonlinear dynamical system, we construct data\-driven reachability, observability, and balanced Gramians from hidden\-state snapshots and output Jacobians\. The resulting A/B/C tests estimate layer\-wise reachable, observable, and jointly reachable–observable ranks\. These ranks are then used not only as diagnostic measures of hidden\-state redundancy, but also as actual compressed layer widths for realised reduced networks\. Experiments on MNIST and CIFAR\-10 compare the proposed balanced realisation against projection\-based reduction, unstructured pruning, structured pruning, low\-rank SVD, dynamic INT8 quantisation, and linear baselines\. On MNIST, a four\-layer SiLU DNN is reduced from state order10241024to277277, giving72\.95%72\.95\\%state compression and73\.48%73\.48\\%parameter compression, while maintaining95\.45%95\.45\\%accuracy compared with96\.60%96\.60\\%for the full model\. On CIFAR\-10, a larger SiLU DNN is reduced from state order46084608to13391339, giving70\.94%70\.94\\%state compression and83\.09%83\.09\\%parameter compression, while preserving accuracy from54\.45%54\.45\\%to54\.44%54\.44\\%and reducing CUDA inference latency by approximately3×3\\times\. The results show that balanced reachable–observable ranks provide a principled empirical minimal\-realisation criterion for designing compact neural architectures with little or no loss in accuracy\.

## IIntroduction

Deep neural networks \(DNNs\) have achieved strong performance across classification, perception, and representation\-learning tasks, but their high parameter counts and wide hidden layers often make them expensive to deploy on resource\-constrained hardware\. As a result, neural network compression has become an important research direction\. Existing methods include pruning, quantisation, low\-rank factorisation, and knowledge distillation\[[6](https://arxiv.org/html/2607.05457#bib.bib1),[8](https://arxiv.org/html/2607.05457#bib.bib2),[4](https://arxiv.org/html/2607.05457#bib.bib3),[9](https://arxiv.org/html/2607.05457#bib.bib4),[11](https://arxiv.org/html/2607.05457#bib.bib5)\]\. These approaches can significantly reduce storage or arithmetic cost, but they usually operate directly on weights, neurons, channels, or numerical precision\. They do not explicitly answer a state\-space question: which hidden\-state directions are actually excited by the data distribution and which of those directions are relevant to the output?

This paper proposes a control\-theoretic view of neural compression\. We interpret a trained feedforward DNN as a depth\-indexed nonlinear state\-space system,

hℓ\+1=fℓ​\(hℓ\),z=g​\(hL\),h\_\{\\ell\+1\}=f\_\{\\ell\}\(h\_\{\\ell\}\),\\qquad z=g\(h\_\{L\}\),\(1\)wherehℓh\_\{\\ell\}denotes the hidden state at layerℓ\\ellandzzdenotes the output logits\. This dynamical\-systems interpretation is closely related to prior views of residual networks, neural ordinary differential equations, stable deep architectures, and optimal\-control formulations of deep learning\[[7](https://arxiv.org/html/2607.05457#bib.bib8),[5](https://arxiv.org/html/2607.05457#bib.bib9),[3](https://arxiv.org/html/2607.05457#bib.bib10),[14](https://arxiv.org/html/2607.05457#bib.bib11),[2](https://arxiv.org/html/2607.05457#bib.bib12)\]\. However, instead of using this viewpoint primarily for stability or continuous\-depth modelling, we use it to estimate the useful hidden\-state order of a trained network\.

The key idea is that not every hidden direction in a wide DNN is equally important\. A direction may be excited by the input data but have little effect on the output logits\. Conversely, a direction may be output\-sensitive but rarely visited by the data distribution\. Inspired by controllability, observability, and balanced truncation in model reduction\[[13](https://arxiv.org/html/2607.05457#bib.bib13),[1](https://arxiv.org/html/2607.05457#bib.bib14),[10](https://arxiv.org/html/2607.05457#bib.bib15)\], we propose three empirical tests for each hidden layer\. Test A measures reachability using hidden\-state snapshot covariances\. Test B measures observability using output\-logit Jacobians with respect to hidden states\. Test C combines both effects through a balanced reachable–observable matrix\. The resulting layer\-wise C\-balanced ranks estimate the number of hidden directions that are simultaneously reached by the data and observable at the output\.

A central distinction in this work is between projected state reduction and realised neural compression\. A projected reduced model maps hidden states into lower\-dimensional coordinates using learned projection bases\. This is useful for diagnosing the existence of a low\-dimensional hidden representation, but it still stores the original full network\. In contrast, the proposed realised C\-balanced compression uses the C\-balanced ranks directly as hidden\-layer widths of a new compact DNN\. Thus, a full network with widths\[n1,n2,…,nL\]\[n\_\{1\},n\_\{2\},\\ldots,n\_\{L\}\]is replaced by a realised compressed architecture with widths\[r1,r2,…,rL\]\[r\_\{1\},r\_\{2\},\\ldots,r\_\{L\}\], whererℓ≪nℓr\_\{\\ell\}\\ll n\_\{\\ell\}\. This converts the empirical state\-order estimate into a true parameter\-compressed network\.

The proposed framework differs from standard compression methods in both criterion and interpretation\. Unstructured pruning removes individual weights but may not reduce dense inference latency\. Structured pruning removes neurons or channels but often relies on local importance scores\. Low\-rank factorisation compresses weight matrices but does not explicitly preserve the nonlinear hidden\-state geometry\. Quantisation reduces numerical precision but leaves the hidden\-state order unchanged\. In contrast, the proposed method is state\-based: it compresses the network according to directions that are jointly reachable from the data and observable through the classifier output\.

We evaluate the proposed method on MNIST and CIFAR\-10 using fully connected SiLU DNNs\. On MNIST, the C\-balanced test reduces the hidden\-state order from10241024to277277, corresponding to72\.95%72\.95\\%state compression\. The realised C\-balanced model reduces the parameter count from400,906400\{,\}906to106,323106\{,\}323, giving73\.48%73\.48\\%parameter compression, while maintaining95\.45%95\.45\\%accuracy compared with96\.60%96\.60\\%for the full model\. On CIFAR\-10, the C\-balanced test reduces the hidden\-state order from46084608to13391339, giving70\.94%70\.94\\%state compression\. The realised compressed model reduces the parameter count from9\.979\.97M to1\.691\.69M, giving83\.09%83\.09\\%parameter compression, while preserving accuracy from54\.45%54\.45\\%to54\.44%54\.44\\%and reducing CUDA inference latency by approximately3\.0×3\.0\\times\.

The main contributions of this paper are summarised as follows:

- •We formulate feedforward DNNs as depth\-indexed nonlinear state\-space systems and define hidden\-state order as a compression\-relevant quantity distinct from parameter count\.
- •We propose empirical reachability, observability, and balanced reachable–observable tests for estimating layer\-wise useful hidden\-state dimensions\.
- •We introduce realised C\-balanced neural compression, where C\-balanced ranks are used as actual hidden\-layer widths of compact networks\.
- •We compare the proposed method against projection\-based reduction, unstructured pruning, structured pruning, low\-rank SVD, dynamic INT8 quantisation, distillation, and linear baselines\.
- •We demonstrate substantial state\-order and parameter reductions on MNIST and CIFAR\-10 with little or no loss in accuracy\.

The remainder of the paper is organised as follows\. Section[II](https://arxiv.org/html/2607.05457#S2)reviews neural compression, dynamical\-systems views of DNNs, and empirical model reduction\. Section[III](https://arxiv.org/html/2607.05457#S3)formulates DNNs as nonlinear state\-space systems and defines the compression objective\. Section[IV](https://arxiv.org/html/2607.05457#S4)introduces the empirical A/B/C controllability–observability tests\. Section[V](https://arxiv.org/html/2607.05457#S5)presents the realised C\-balanced compression procedure\. Section[VI](https://arxiv.org/html/2607.05457#S6)reports the MNIST and CIFAR\-10 experiments\. Section[VII](https://arxiv.org/html/2607.05457#S7)discusses limitations and future extensions, and Section[VIII](https://arxiv.org/html/2607.05457#S8)concludes the paper\.

## IIRelated Work

This work connects neural network compression with control\-theoretic model reduction\. Existing compression methods usually reduce weights, neurons, channels, or numerical precision\. In contrast, the proposed approach estimates the useful hidden\-state order of a trained network using empirical reachability and observability tests\. This section reviews the most relevant literature and positions the proposed method relative to prior work\.

### II\-ANeural Network Compression

Neural network compression has been widely studied as a way to reduce memory footprint, computational cost, and inference latency\. A major class of methods is pruning, where redundant weights, neurons, filters, or channels are removed from a trained model\. Deep Compression combines pruning, trained quantisation, and entropy coding to significantly reduce the storage requirements of neural networks\[[6](https://arxiv.org/html/2607.05457#bib.bib1)\]\. Unstructured pruning removes individual weights and can achieve high sparsity, but it does not necessarily reduce dense matrix dimensions or hardware latency unless sparse kernels are used\. Structured pruning removes complete neurons, filters, or channels and is therefore more suitable for practical acceleration\.

Channel\-level pruning methods aim to produce thinner deployable networks\. Network Slimming uses sparsity regularisation on batch\-normalisation scale parameters to identify and remove insignificant channels\[[11](https://arxiv.org/html/2607.05457#bib.bib5)\]\. Other pruning studies have also shown that the final compressed architecture can be more important than inheriting the original pruned weights\. In particular, Liu*et al\.*argued that, for some structured pruning settings, training the pruned architecture from scratch can match or outperform fine\-tuning inherited weights\[[12](https://arxiv.org/html/2607.05457#bib.bib6)\]\. This observation is closely related to our realised compression strategy: the C\-balanced ranks are used to define the compressed architecture, which is then trained as a new compact model\.

Low\-rank factorisation provides another important compression direction\. Denton*et al\.*exploited approximate linear structure in convolutional networks by replacing large linear or convolutional operations with low\-rank approximations\[[4](https://arxiv.org/html/2607.05457#bib.bib3)\]\. Such methods reduce computation by factorising weight matrices or convolutional tensors\. However, low\-rank compression is usually layer\-local and weight\-centric\. It does not explicitly account for whether the retained directions are excited by the input distribution or observable at the output\. Our experiments show that naive layer\-wise SVD can substantially degrade accuracy when compared with the proposed state\-based C\-balanced realisation\.

Quantisation reduces the numerical precision of weights and activations\. Integer\-only quantisation schemes have been developed to enable efficient inference on low\-power and mobile hardware\[[9](https://arxiv.org/html/2607.05457#bib.bib4)\]\. Quantisation can reduce memory bandwidth and improve deployment efficiency, but it leaves the hidden\-state order and architecture width unchanged\. Therefore, quantisation is complementary to the proposed method: C\-balanced compression reduces the number of hidden coordinates, while quantisation reduces the numerical precision of the remaining computations\.

Knowledge distillation compresses a large teacher model into a smaller student model by matching softened output distributions\[[8](https://arxiv.org/html/2607.05457#bib.bib2)\]\. Distillation is often effective because teacher logits contain additional class\-similarity information beyond hard labels\. In this paper, distillation is used as an optional training strategy for the realised C\-balanced network\. However, the main novelty is not the distillation loss itself, but the way the student architecture is chosen: its layer widths are determined by empirical balanced reachable–observable ranks\.

### II\-BDynamical\-Systems Views of Deep Networks

A growing body of work interprets deep networks as dynamical systems evolving along the depth dimension\. Residual networks can be written as

hℓ\+1=hℓ\+ϕℓ​\(hℓ\),h\_\{\\ell\+1\}=h\_\{\\ell\}\+\\phi\_\{\\ell\}\(h\_\{\\ell\}\),\(2\)which resembles a forward\-Euler discretisation of a continuous\-time dynamical system\[[7](https://arxiv.org/html/2607.05457#bib.bib8)\]\. This interpretation motivated stable architectures for deep neural networks, where forward propagation is analysed using tools from ordinary differential equations and numerical stability\[[5](https://arxiv.org/html/2607.05457#bib.bib9)\]\. Neural ordinary differential equations make this connection explicit by replacing a finite sequence of layers with a learned continuous\-depth ODE\[[3](https://arxiv.org/html/2607.05457#bib.bib10)\]\. PDE\-inspired interpretations have also been used to motivate new convolutional and residual architectures\[[14](https://arxiv.org/html/2607.05457#bib.bib11)\]\. In addition, optimal\-control formulations of deep learning interpret training as a control problem over a depth\-indexed dynamical system\[[2](https://arxiv.org/html/2607.05457#bib.bib12)\]\.

These works mainly focus on stability, trainability, expressivity, or continuous\-depth modelling\. Our work uses the dynamical\-systems viewpoint for a different purpose: estimating the useful hidden\-state order of a trained network\. By treating hidden activations as states, we can ask whether each hidden direction is reachable from the data distribution and observable through the output logits\. This leads naturally to empirical controllability and observability tests\.

### II\-CClassical Model Reduction and Balanced Truncation

Model reduction is a classical topic in control theory and numerical linear algebra\. The goal is to approximate a high\-dimensional dynamical system by a lower\-dimensional system that preserves the important input–output behaviour\. Balanced truncation is a particularly influential method because it uses both controllability and observability Gramians to identify states that are simultaneously difficult to control and difficult to observe\[[13](https://arxiv.org/html/2607.05457#bib.bib13),[1](https://arxiv.org/html/2607.05457#bib.bib14)\]\. In linear systems, balanced coordinates reveal state directions ranked by Hankel singular values, and truncating weak balanced directions yields a reduced\-order model with input–output relevance\.

The philosophy of balanced truncation is closely aligned with the objective of neural compression\. A hidden direction should not be retained merely because it exists in a wide layer; it should be retained if it is both reached by the input distribution and relevant to the output\. However, standard balanced truncation is formulated for dynamical systems with explicit input and output maps, often in linear time\-invariant form\. Deep neural networks are nonlinear, depth\-indexed, data\-driven systems, and their “inputs” are samples from a data distribution rather than control signals in the classical sense\. Therefore, the classical theory cannot be applied directly without adaptation\.

### II\-DEmpirical Gramians and Nonlinear Model Reduction

For nonlinear systems, empirical Gramians provide a practical way to estimate controllability and observability properties from simulations or data\. Lall, Marsden, and Glavaski introduced empirical model reduction for controlled nonlinear systems using data\-driven approximations of balanced truncation\[[10](https://arxiv.org/html/2607.05457#bib.bib15)\]\. These methods extend the balanced\-reduction idea beyond purely linear systems by using sampled state responses and output sensitivities\.

The proposed method is inspired by this empirical\-Gramian viewpoint, but adapts it to trained neural networks\. Instead of simulating a physical nonlinear control system, we collect hidden\-state snapshots generated by a trained DNN on a representative dataset\. Reachability is approximated by the covariance of hidden activations, while observability is approximated by output\-logit Jacobians with respect to hidden states\. The balanced matrix combines these two quantities and produces a layer\-wise rank estimate\. This gives a data\-dependent estimate of the useful hidden\-state order of the trained network\.

### II\-EPositioning of This Work

The proposed framework differs from existing compression methods in three main ways\. First, it is state\-based rather than purely weight\-based\. Pruning, SVD, and quantisation operate mainly on weights or numerical representations, whereas the proposed method analyses hidden\-state directions\. Second, it combines data excitation and output relevance\. Reachability alone measures which directions are visited by the data, while observability alone measures which directions can affect the logits\. The C\-balanced test retains directions that satisfy both criteria\. Third, the method converts the estimated state ranks into realised compressed architectures\. Thus, the proposed C\-balanced ranks are not merely diagnostic; they define the layer widths of compact networks\.

In summary, prior work has extensively studied neural compression, dynamical interpretations of deep learning, and balanced model reduction\. The contribution of this paper is to connect these ideas through an empirical controllability–observability framework for neural compression\. The resulting method provides a principled way to estimate hidden\-state redundancy and to design compact DNN architectures with reduced state order, fewer parameters, and competitive accuracy\.

## IIIDNNs as Nonlinear State\-Space Systems

### III\-ADepth\-Indexed Dynamics

Deep neural networks can be interpreted as nonlinear dynamical systems evolving along the layer index\. This viewpoint is well established for residual networks, neural ordinary differential equations, and stable architectures for deep learning, where network depth is interpreted as a discrete or continuous evolution variable\[[7](https://arxiv.org/html/2607.05457#bib.bib8),[5](https://arxiv.org/html/2607.05457#bib.bib9),[3](https://arxiv.org/html/2607.05457#bib.bib10),[14](https://arxiv.org/html/2607.05457#bib.bib11),[2](https://arxiv.org/html/2607.05457#bib.bib12)\]\. In this work, we adopt this state\-space interpretation for empirical state\-order analysis and neural compression\.

Letx∈ℝdx\\in\\mathbb\{R\}^\{d\}denote an input sample and lethℓ∈ℝnℓh\_\{\\ell\}\\in\\mathbb\{R\}^\{n\_\{\\ell\}\}denote the hidden state at layerℓ\\ell\. The input is treated as the initial state,

The forward propagation of a feedforward DNN can then be written as the depth\-indexed nonlinear system

hℓ\+1=fℓ​\(hℓ\),ℓ=0,…,L−1,h\_\{\\ell\+1\}=f\_\{\\ell\}\(h\_\{\\ell\}\),\\qquad\\ell=0,\\ldots,L\-1,\(4\)wherefℓ:ℝnℓ→ℝnℓ\+1f\_\{\\ell\}:\\mathbb\{R\}^\{n\_\{\\ell\}\}\\rightarrow\\mathbb\{R\}^\{n\_\{\\ell\+1\}\}is the nonlinear transition map implemented by layerℓ\\ell\. For a standard fully connected DNN, this map is

fℓ​\(hℓ\)=σ​\(Wℓ​hℓ\+bℓ\),f\_\{\\ell\}\(h\_\{\\ell\}\)=\\sigma\(W\_\{\\ell\}h\_\{\\ell\}\+b\_\{\\ell\}\),\(5\)whereWℓW\_\{\\ell\}andbℓb\_\{\\ell\}are trainable weights and biases, andσ​\(⋅\)\\sigma\(\\cdot\)is a pointwise activation function such as ReLU, GELU, or SiLU\.

The network output is obtained from the final hidden state through an output map

wherez∈ℝCz\\in\\mathbb\{R\}^\{C\}denotes the output logits andCCis the number of classes\. For a linear classifier head,

g​\(hL\)=WL​hL\+bL\.g\(h\_\{L\}\)=W\_\{L\}h\_\{L\}\+b\_\{L\}\.\(7\)Thus, the full DNN can be represented compactly as

h0=x,hℓ\+1=fℓ​\(hℓ\),z=g​\(hL\)\.h\_\{0\}=x,\\qquad h\_\{\\ell\+1\}=f\_\{\\ell\}\(h\_\{\\ell\}\),\\qquad z=g\(h\_\{L\}\)\.\(8\)
This representation separates the hidden\-state trajectory from the parameterisation of the layer maps\. It therefore allows us to ask a control\-theoretic question: which hidden\-state directions are actually excited by the input distribution, and which of those directions are observable through the output logits?

### III\-BRelation to Residual and Continuous\-Depth Networks

The state\-space interpretation is especially direct for residual networks\. A residual block can be written as

hℓ\+1=hℓ\+ϕℓ​\(hℓ\),h\_\{\\ell\+1\}=h\_\{\\ell\}\+\\phi\_\{\\ell\}\(h\_\{\\ell\}\),\(9\)whereϕℓ\\phi\_\{\\ell\}denotes the residual transformation\. This resembles a forward\-Euler discretisation of a continuous\-time dynamical system,

d​h​\(t\)d​t=ϕ​\(h​\(t\),t\),\\frac\{dh\(t\)\}\{dt\}=\\phi\(h\(t\),t\),\(10\)with the layer index playing the role of a discretised time variable\[[5](https://arxiv.org/html/2607.05457#bib.bib9),[3](https://arxiv.org/html/2607.05457#bib.bib10),[14](https://arxiv.org/html/2607.05457#bib.bib11)\]\. Neural ODEs make this analogy explicit by replacing the discrete sequence of hidden layers with a learned differential equation whose solution defines the network output\[[3](https://arxiv.org/html/2607.05457#bib.bib10)\]\. Similarly, PDE\-inspired interpretations of CNNs and ResNets have been used to design parabolic and hyperbolic network architectures\[[14](https://arxiv.org/html/2607.05457#bib.bib11)\]\.

Although our experiments focus primarily on fully connected DNNs for controlled state\-order analysis, the proposed formulation applies more broadly\. For CNNs and ResNets, one may define the state at stageℓ\\ellas a pooled feature representation,

hℓ=GAP​\(Fℓ​\(x\)\),h\_\{\\ell\}=\\mathrm\{GAP\}\(F\_\{\\ell\}\(x\)\),\(11\)whereFℓ​\(x\)F\_\{\\ell\}\(x\)is the feature map produced by stageℓ\\ellandGAP​\(⋅\)\\mathrm\{GAP\}\(\\cdot\)denotes global average pooling\. In this case, the hidden\-state dimension corresponds to the number of channels at each stage, and state\-order reduction becomes channel\-width reduction\.

### III\-CHidden\-State Order

In this paper, the hidden\-state order of a DNN is defined as the total number of hidden coordinates propagated through the network\. For a network withLLhidden layers of widthsn1,…,nLn\_\{1\},\\ldots,n\_\{L\}, the full hidden\-state order is

nstate=∑ℓ=1Lnℓ\.n\_\{\\mathrm\{state\}\}=\\sum\_\{\\ell=1\}^\{L\}n\_\{\\ell\}\.\(12\)This quantity is distinct from the number of trainable parameters\. For a fully connected DNN with input dimensiondd, hidden widths\[n1,…,nL\]\[n\_\{1\},\\ldots,n\_\{L\}\], and output dimensionCC, the parameter count is

Nθ=d​n1\+n1\+∑ℓ=2L\(nℓ−1​nℓ\+nℓ\)\+nL​C\+C\.N\_\{\\theta\}=dn\_\{1\}\+n\_\{1\}\+\\sum\_\{\\ell=2\}^\{L\}\(n\_\{\\ell\-1\}n\_\{\\ell\}\+n\_\{\\ell\}\)\+n\_\{L\}C\+C\.\(13\)The state order measures the dimension of the internal representation, whereasNθN\_\{\\theta\}measures the number of stored trainable coefficients\. These two quantities are related but not identical\. A network may contain redundant hidden\-state directions even when its weight matrices are not obviously low rank\.

The proposed framework therefore focuses on hidden\-state redundancy rather than only weight redundancy\. A hidden direction is considered useful only if it is both excited by the data distribution and influential at the output\. This motivates the empirical reachability and observability tests introduced in the next section\.

### III\-DCompression Objective

Let the full network have hidden widths

\[n1,n2,…,nL\]\.\[n\_\{1\},n\_\{2\},\\ldots,n\_\{L\}\]\.\(14\)The objective is to identify reduced hidden widths

\[r1,r2,…,rL\],rℓ≪nℓ,\[r\_\{1\},r\_\{2\},\\ldots,r\_\{L\}\],\\qquad r\_\{\\ell\}\\ll n\_\{\\ell\},\(15\)such that the reduced model preserves the input–output function of the full network while reducing hidden\-state order, parameter count, and inference latency\.

The reduced hidden\-state order is

rstate=∑ℓ=1Lrℓ,r\_\{\\mathrm\{state\}\}=\\sum\_\{\\ell=1\}^\{L\}r\_\{\\ell\},\(16\)and the state\-compression ratio is

ηstate=1−rstatenstate\.\\eta\_\{\\mathrm\{state\}\}=1\-\\frac\{r\_\{\\mathrm\{state\}\}\}\{n\_\{\\mathrm\{state\}\}\}\.\(17\)For a realised compressed fully connected DNN with reduced widths\[r1,…,rL\]\[r\_\{1\},\\ldots,r\_\{L\}\], the parameter count becomes

Nθred=d​r1\+r1\+∑ℓ=2L\(rℓ−1​rℓ\+rℓ\)\+rL​C\+C\.N\_\{\\theta\}^\{\\mathrm\{red\}\}=dr\_\{1\}\+r\_\{1\}\+\\sum\_\{\\ell=2\}^\{L\}\(r\_\{\\ell\-1\}r\_\{\\ell\}\+r\_\{\\ell\}\)\+r\_\{L\}C\+C\.\(18\)The corresponding parameter\-compression ratio is

ηθ=1−NθredNθ\.\\eta\_\{\\theta\}=1\-\\frac\{N\_\{\\theta\}^\{\\mathrm\{red\}\}\}\{N\_\{\\theta\}\}\.\(19\)
The compression problem may therefore be written as

min\{rℓ\}ℓ=1L∑ℓ=1Lrℓ\\min\_\{\\\{r\_\{\\ell\}\\\}\_\{\\ell=1\}^\{L\}\}\\quad\\sum\_\{\\ell=1\}^\{L\}r\_\{\\ell\}\(20\)subject to

𝒜​\(Fred\)≥𝒜​\(Ffull\)−δ,\\mathcal\{A\}\(F\_\{\\mathrm\{red\}\}\)\\geq\\mathcal\{A\}\(F\_\{\\mathrm\{full\}\}\)\-\\delta,\(21\)where𝒜​\(⋅\)\\mathcal\{A\}\(\\cdot\)denotes task accuracy andδ\\deltais an acceptable accuracy degradation\. Directly solving this constrained problem is generally difficult\. Instead, we estimate the useful hidden\-state order at each layer using empirical controllability–observability tests\. The resulting ranks are then used either as projection dimensions or as actual hidden widths in a realised compressed architecture\.

### III\-EProjected Versus Realised State Reduction

The proposed framework distinguishes between projected state reduction and realised neural compression\. In projected reduction, the full hidden state is mapped to a lower\-dimensional coordinate

qℓ=UℓT​\(hℓ−μℓ\),qℓ∈ℝrℓ,q\_\{\\ell\}=U\_\{\\ell\}^\{T\}\(h\_\{\\ell\}\-\\mu\_\{\\ell\}\),\\qquad q\_\{\\ell\}\\in\\mathbb\{R\}^\{r\_\{\\ell\}\},\(22\)whereUℓ∈ℝnℓ×rℓU\_\{\\ell\}\\in\\mathbb\{R\}^\{n\_\{\\ell\}\\times r\_\{\\ell\}\}is a learned basis andμℓ\\mu\_\{\\ell\}is the empirical hidden\-state mean\. This representation is useful for diagnosing whether a low\-dimensional hidden\-state representation exists\. However, it still requires storing the original network and the projection bases\.

In realised reduction, the estimated ranksrℓr\_\{\\ell\}are used directly as the layer widths of a new compressed network:

h~ℓ\+1=f~ℓ​\(h~ℓ\),h~ℓ∈ℝrℓ\.\\tilde\{h\}\_\{\\ell\+1\}=\\tilde\{f\}\_\{\\ell\}\(\\tilde\{h\}\_\{\\ell\}\),\\qquad\\tilde\{h\}\_\{\\ell\}\\in\\mathbb\{R\}^\{r\_\{\\ell\}\}\.\(23\)This produces a genuinely smaller neural network with fewer hidden\-state coordinates and fewer trainable parameters\. The central question addressed in this paper is whether controllability–observability ranks estimated from the full network can provide reliable reduced widths for such realised compressed architectures\.

## IVEmpirical Controllability–Observability Tests

This section introduces the proposed empirical A/B/C tests for estimating the useful hidden\-state order of a trained DNN\. The tests are inspired by controllability, observability, and balanced truncation in dynamical systems, but are adapted to depth\-indexed neural network dynamics and computed directly from hidden\-state snapshots and output Jacobians\.

Let a trained DNN be written as

h0=x,hℓ\+1=fℓ​\(hℓ\),z=g​\(hL\),h\_\{0\}=x,\\qquad h\_\{\\ell\+1\}=f\_\{\\ell\}\(h\_\{\\ell\}\),\\qquad z=g\(h\_\{L\}\),\(24\)wherehℓ∈ℝnℓh\_\{\\ell\}\\in\\mathbb\{R\}^\{n\_\{\\ell\}\}is the hidden state at layerℓ\\ellandz∈ℝCz\\in\\mathbb\{R\}^\{C\}denotes the output logits\. Given a set ofNNinput samples\{xi\}i=1N\\\{x\_\{i\}\\\}\_\{i=1\}^\{N\}, the corresponding hidden states at layerℓ\\ellare collected as

Hℓ=\[hℓ​\(x1\)Thℓ​\(x2\)T⋮hℓ​\(xN\)T\]∈ℝN×nℓ\.H\_\{\\ell\}=\\begin\{bmatrix\}h\_\{\\ell\}\(x\_\{1\}\)^\{T\}\\\\ h\_\{\\ell\}\(x\_\{2\}\)^\{T\}\\\\ \\vdots\\\\ h\_\{\\ell\}\(x\_\{N\}\)^\{T\}\\end\{bmatrix\}\\in\\mathbb\{R\}^\{N\\times n\_\{\\ell\}\}\.\(25\)The empirical hidden\-state mean is

μℓ=1N​∑i=1Nhℓ​\(xi\),\\mu\_\{\\ell\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}h\_\{\\ell\}\(x\_\{i\}\),\(26\)and the centred snapshot matrix is

H¯ℓ=Hℓ−𝟏​μℓT,\\bar\{H\}\_\{\\ell\}=H\_\{\\ell\}\-\\mathbf\{1\}\\mu\_\{\\ell\}^\{T\},\(27\)where𝟏∈ℝN\\mathbf\{1\}\\in\\mathbb\{R\}^\{N\}is the all\-ones vector\.

### IV\-ATest A: Empirical Reachability

The first test measures which hidden\-state directions are excited by the data distribution\. We define the empirical reachability Gramian at layerℓ\\ellas the hidden\-state covariance

Gc,ℓ=1N​H¯ℓT​H¯ℓ∈ℝnℓ×nℓ\.G\_\{c,\\ell\}=\\frac\{1\}\{N\}\\bar\{H\}\_\{\\ell\}^\{T\}\\bar\{H\}\_\{\\ell\}\\in\\mathbb\{R\}^\{n\_\{\\ell\}\\times n\_\{\\ell\}\}\.\(28\)Large eigenvalues ofGc,ℓG\_\{c,\\ell\}correspond to directions in hidden\-state space that are strongly excited by the input data\. Let

Gc,ℓ=Vc,ℓ​Λc,ℓ​Vc,ℓT,G\_\{c,\\ell\}=V\_\{c,\\ell\}\\Lambda\_\{c,\\ell\}V\_\{c,\\ell\}^\{T\},\(29\)where the eigenvalues are sorted in descending order,

λc,ℓ,1≥λc,ℓ,2≥⋯≥λc,ℓ,nℓ≥0\.\\lambda\_\{c,\\ell,1\}\\geq\\lambda\_\{c,\\ell,2\}\\geq\\cdots\\geq\\lambda\_\{c,\\ell,n\_\{\\ell\}\}\\geq 0\.\(30\)The empirical reachable rankrc,ℓr\_\{c,\\ell\}is selected as the smallest integer satisfying

∑i=1rc,ℓλc,ℓ,i∑i=1nℓλc,ℓ,i≥1−ε\.\\frac\{\\sum\_\{i=1\}^\{r\_\{c,\\ell\}\}\\lambda\_\{c,\\ell,i\}\}\{\\sum\_\{i=1\}^\{n\_\{\\ell\}\}\\lambda\_\{c,\\ell,i\}\}\\geq 1\-\\varepsilon\.\(31\)The corresponding reachability basis is

Uc,ℓ=Vc,ℓ\(:,1:rc,ℓ\)\.U\_\{c,\\ell\}=V\_\{c,\\ell\}\(:,1:r\_\{c,\\ell\}\)\.\(32\)
Test A is therefore a data\-excitation test\. It identifies the low\-dimensional subspace that captures most of the hidden\-state variance induced by the input distribution\. However, a direction may be strongly excited by the data while having little effect on the output logits\. For this reason, reachability alone is not sufficient for task\-aware compression\.

### IV\-BTest B: Empirical Observability

The second test measures which hidden\-state directions influence the output logits\. For each input samplexix\_\{i\}, we define the output Jacobian with respect to the hidden state at layerℓ\\ell:

Jℓ​\(xi\)=∂z​\(xi\)∂hℓ​\(xi\)∈ℝC×nℓ\.J\_\{\\ell\}\(x\_\{i\}\)=\\frac\{\\partial z\(x\_\{i\}\)\}\{\\partial h\_\{\\ell\}\(x\_\{i\}\)\}\\in\\mathbb\{R\}^\{C\\times n\_\{\\ell\}\}\.\(33\)The empirical observability Gramian is then defined as

Go,ℓ=1N​∑i=1NJℓ​\(xi\)T​Jℓ​\(xi\)∈ℝnℓ×nℓ\.G\_\{o,\\ell\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}J\_\{\\ell\}\(x\_\{i\}\)^\{T\}J\_\{\\ell\}\(x\_\{i\}\)\\in\\mathbb\{R\}^\{n\_\{\\ell\}\\times n\_\{\\ell\}\}\.\(34\)For any directionv∈ℝnℓv\\in\\mathbb\{R\}^\{n\_\{\\ell\}\}, the quadratic form

vT​Go,ℓ​v=1N​∑i=1N‖Jℓ​\(xi\)​v‖22v^\{T\}G\_\{o,\\ell\}v=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\\|J\_\{\\ell\}\(x\_\{i\}\)v\\\|\_\{2\}^\{2\}\(35\)measures the average sensitivity of the output logits to perturbations in the hidden\-state directionvv\. Thus, directions associated with large eigenvalues ofGo,ℓG\_\{o,\\ell\}are highly observable at the output\.

Let

Go,ℓ=Vo,ℓ​Λo,ℓ​Vo,ℓT,G\_\{o,\\ell\}=V\_\{o,\\ell\}\\Lambda\_\{o,\\ell\}V\_\{o,\\ell\}^\{T\},\(36\)with eigenvalues sorted in descending order\. The empirical observable rankro,ℓr\_\{o,\\ell\}is selected as the smallest integer satisfying

∑i=1ro,ℓλo,ℓ,i∑i=1nℓλo,ℓ,i≥1−ε\.\\frac\{\\sum\_\{i=1\}^\{r\_\{o,\\ell\}\}\\lambda\_\{o,\\ell,i\}\}\{\\sum\_\{i=1\}^\{n\_\{\\ell\}\}\\lambda\_\{o,\\ell,i\}\}\\geq 1\-\\varepsilon\.\(37\)The corresponding observability basis is

Uo,ℓ=Vo,ℓ\(:,1:ro,ℓ\)\.U\_\{o,\\ell\}=V\_\{o,\\ell\}\(:,1:r\_\{o,\\ell\}\)\.\(38\)
Test B is task\-aware because it depends on the output map and the classifier logits\. However, observability alone can retain directions that are highly sensitive at the output but are rarely excited by the actual data distribution\. Therefore, a balanced criterion is needed to retain directions that are both data\-reachable and output\-observable\.

### IV\-CTest C: Balanced Reachable–Observable Rank

The third test combines reachability and observability\. For each layerℓ\\ell, we define the empirical balanced matrix

Bℓ=Gc,ℓ1/2​Go,ℓ​Gc,ℓ1/2∈ℝnℓ×nℓ,B\_\{\\ell\}=G\_\{c,\\ell\}^\{1/2\}G\_\{o,\\ell\}G\_\{c,\\ell\}^\{1/2\}\\in\\mathbb\{R\}^\{n\_\{\\ell\}\\times n\_\{\\ell\}\},\(39\)whereGc,ℓ1/2G\_\{c,\\ell\}^\{1/2\}denotes the symmetric positive semidefinite square root ofGc,ℓG\_\{c,\\ell\}\. For any directionw∈ℝnℓw\\in\\mathbb\{R\}^\{n\_\{\\ell\}\}, the energy induced byBℓB\_\{\\ell\}reflects the output sensitivity of directions weighted by how strongly they are excited by the data distribution\.

Let

Bℓ=Vb,ℓ​Λb,ℓ​Vb,ℓT,B\_\{\\ell\}=V\_\{b,\\ell\}\\Lambda\_\{b,\\ell\}V\_\{b,\\ell\}^\{T\},\(40\)where the eigenvalues

λb,ℓ,1≥λb,ℓ,2≥⋯≥λb,ℓ,nℓ≥0\\lambda\_\{b,\\ell,1\}\\geq\\lambda\_\{b,\\ell,2\}\\geq\\cdots\\geq\\lambda\_\{b,\\ell,n\_\{\\ell\}\}\\geq 0\(41\)are sorted in descending order\. The C\-balanced rankrb,ℓr\_\{b,\\ell\}is selected as the smallest integer satisfying

∑i=1rb,ℓλb,ℓ,i∑i=1nℓλb,ℓ,i≥1−ε\.\\frac\{\\sum\_\{i=1\}^\{r\_\{b,\\ell\}\}\\lambda\_\{b,\\ell,i\}\}\{\\sum\_\{i=1\}^\{n\_\{\\ell\}\}\\lambda\_\{b,\\ell,i\}\}\\geq 1\-\\varepsilon\.\(42\)
A corresponding state\-space basis can be formed as

Ub,ℓ=orth\(Gc,ℓ1/2Vb,ℓ\(:,1:rb,ℓ\)\),U\_\{b,\\ell\}=\\mathrm\{orth\}\\left\(G\_\{c,\\ell\}^\{1/2\}V\_\{b,\\ell\}\(:,1:r\_\{b,\\ell\}\)\\right\),\(43\)whereorth​\(⋅\)\\mathrm\{orth\}\(\\cdot\)denotes column orthonormalisation\. The projected C\-balanced state is then

qℓ=Ub,ℓT​\(hℓ−μℓ\),qℓ∈ℝrb,ℓ\.q\_\{\\ell\}=U\_\{b,\\ell\}^\{T\}\(h\_\{\\ell\}\-\\mu\_\{\\ell\}\),\\qquad q\_\{\\ell\}\\in\\mathbb\{R\}^\{r\_\{b,\\ell\}\}\.\(44\)
The C\-balanced test is the primary compression criterion used in this paper\. Unlike Test A, it does not retain directions solely because they are excited by the input\. Unlike Test B, it does not retain directions solely because they are locally output\-sensitive\. Instead, it keeps directions that are jointly reachable from the data distribution and observable through the classifier output\.

### IV\-DRank Selection and State\-Order Estimate

For each test, the same spectral energy criterion is used to select the layer\-wise rank\. Given a positive semidefinite matrixMℓM\_\{\\ell\}with eigenvalues

λℓ,1≥λℓ,2≥⋯≥λℓ,nℓ≥0,\\lambda\_\{\\ell,1\}\\geq\\lambda\_\{\\ell,2\}\\geq\\cdots\\geq\\lambda\_\{\\ell,n\_\{\\ell\}\}\\geq 0,\(45\)the selected rank is

rℓ=min⁡\{r:∑i=1rλℓ,i∑i=1nℓλℓ,i≥1−ε\}\.r\_\{\\ell\}=\\min\\left\\\{r:\\frac\{\\sum\_\{i=1\}^\{r\}\\lambda\_\{\\ell,i\}\}\{\\sum\_\{i=1\}^\{n\_\{\\ell\}\}\\lambda\_\{\\ell,i\}\}\\geq 1\-\\varepsilon\\right\\\}\.\(46\)The estimated reduced hidden\-state order is then

rstate=∑ℓ=1Lrℓ\.r\_\{\\mathrm\{state\}\}=\\sum\_\{\\ell=1\}^\{L\}r\_\{\\ell\}\.\(47\)The corresponding state\-compression ratio is

ηstate=1−rstatenstate\.\\eta\_\{\\mathrm\{state\}\}=1\-\\frac\{r\_\{\\mathrm\{state\}\}\}\{n\_\{\\mathrm\{state\}\}\}\.\(48\)
The toleranceε\\varepsiloncontrols the trade\-off between compression and fidelity\. Smaller values ofε\\varepsilonretain more spectral energy and therefore produce larger ranks, while larger values produce more aggressive compression\. In all reported experiments, we useε=10−4\\varepsilon=10^\{\-4\}unless otherwise stated\.

### IV\-EInterpretation of the A/B/C Tests

The three tests have complementary interpretations\. Test A measures data excitation\. Ifrc,ℓr\_\{c,\\ell\}is close tonℓn\_\{\\ell\}, then the hidden states at layerℓ\\elloccupy a high\-dimensional subspace under the data distribution\. Test B measures task sensitivity\. Ifro,ℓr\_\{o,\\ell\}is small, then only a small number of hidden\-state directions significantly affect the output logits\. Test C measures the dimension of the jointly reachable–observable subspace\. A small C\-balanced rank indicates that the layer contains many directions that are either weakly excited, weakly observable, or both\.

This distinction is important for neural compression\. Weight magnitude or matrix rank alone does not determine whether a hidden direction is useful for the input–output task\. A hidden direction should be preserved only if it is both activated by the data and relevant to the output\. The C\-balanced rank therefore provides an empirical minimal\-realisation\-style estimate of the useful hidden\-state order of the trained network\.

### IV\-FComputational Considerations

Computing the reachability Gramian requires only forward passes and hidden\-state snapshots\. Its cost is therefore dominated by the collection of hidden activations and the eigendecomposition of matrices of sizenℓ×nℓn\_\{\\ell\}\\times n\_\{\\ell\}\. Computing the observability Gramian is more expensive because it requires output Jacobians with respect to hidden states\. In the experiments, this is computed on a subset of samples to reduce cost\.

For classification withCCoutput logits, the observability Gramian can be written as

Go,ℓ=1N​∑i=1N∑c=1C∇hℓzc​\(xi\)​∇hℓzc​\(xi\)T\.G\_\{o,\\ell\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\sum\_\{c=1\}^\{C\}\\nabla\_\{h\_\{\\ell\}\}z\_\{c\}\(x\_\{i\}\)\\nabla\_\{h\_\{\\ell\}\}z\_\{c\}\(x\_\{i\}\)^\{T\}\.\(49\)SinceCCis small for MNIST and CIFAR\-10, exact Jacobian\-based observability is feasible\. For larger output spaces, one may use stochastic trace estimators, sampled logits, random output projections, or minibatch approximations\.

### IV\-GAlgorithm

Algorithm[1](https://arxiv.org/html/2607.05457#alg1)summarises the empirical A/B/C rank estimation procedure\. The output of the algorithm is a set of layer\-wise ranks for reachability, observability, and balanced compression\. In the following section, the C\-balanced ranks are used to construct realised compressed neural networks\.

Algorithm 1Empirical A/B/C Controllability–Observability Tests0:Trained network

FF, dataset

𝒟\\mathcal\{D\}, tolerance

ε\\varepsilon
0:Layer\-wise ranks

\{rc,ℓ,ro,ℓ,rb,ℓ\}ℓ=1L\\\{r\_\{c,\\ell\},r\_\{o,\\ell\},r\_\{b,\\ell\}\\\}\_\{\\ell=1\}^\{L\}
1:for

ℓ=1\\ell=1to

LLdo

2:Collect hidden states

\{hℓ​\(xi\)\}i=1N\\\{h\_\{\\ell\}\(x\_\{i\}\)\\\}\_\{i=1\}^\{N\}
3:Form centred snapshot matrix

H¯ℓ\\bar\{H\}\_\{\\ell\}
4:Compute reachability Gramian

Gc,ℓ=1N​H¯ℓT​H¯ℓG\_\{c,\\ell\}=\\frac\{1\}\{N\}\\bar\{H\}\_\{\\ell\}^\{T\}\\bar\{H\}\_\{\\ell\}
5:Compute output Jacobians

Jℓ​\(xi\)=∂z​\(xi\)∂hℓ​\(xi\)J\_\{\\ell\}\(x\_\{i\}\)=\\frac\{\\partial z\(x\_\{i\}\)\}\{\\partial h\_\{\\ell\}\(x\_\{i\}\)\}
6:Compute observability Gramian

Go,ℓ=1N​∑i=1NJℓ​\(xi\)T​Jℓ​\(xi\)G\_\{o,\\ell\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}J\_\{\\ell\}\(x\_\{i\}\)^\{T\}J\_\{\\ell\}\(x\_\{i\}\)
7:Form balanced matrix

Bℓ=Gc,ℓ1/2​Go,ℓ​Gc,ℓ1/2B\_\{\\ell\}=G\_\{c,\\ell\}^\{1/2\}G\_\{o,\\ell\}G\_\{c,\\ell\}^\{1/2\}
8:Select

rc,ℓr\_\{c,\\ell\}from eigenvalues of

Gc,ℓG\_\{c,\\ell\}
9:Select

ro,ℓr\_\{o,\\ell\}from eigenvalues of

Go,ℓG\_\{o,\\ell\}
10:Select

rb,ℓr\_\{b,\\ell\}from eigenvalues of

BℓB\_\{\\ell\}
11:endfor

12:return

\{rc,ℓ,ro,ℓ,rb,ℓ\}ℓ=1L\\\{r\_\{c,\\ell\},r\_\{o,\\ell\},r\_\{b,\\ell\}\\\}\_\{\\ell=1\}^\{L\}

## VRealised C\-Balanced Neural Compression

The A/B/C tests described in the previous section provide empirical estimates of the useful hidden\-state dimension at each layer\. In particular, the C\-balanced test identifies directions that are both reachable from the data distribution and observable through the output logits\. This section describes how these C\-balanced ranks are converted into actual compressed neural architectures\.

### V\-AProjected C\-Balanced Reduction

LetUℓ∈ℝnℓ×rℓU\_\{\\ell\}\\in\\mathbb\{R\}^\{n\_\{\\ell\}\\times r\_\{\\ell\}\}denote the C\-balanced projection basis at layerℓ\\ell, whererℓr\_\{\\ell\}is the selected C\-balanced rank\. A projected reduced hidden coordinate is defined as

qℓ=UℓT​\(hℓ−μℓ\),qℓ∈ℝrℓ,q\_\{\\ell\}=U\_\{\\ell\}^\{T\}\(h\_\{\\ell\}\-\\mu\_\{\\ell\}\),\\qquad q\_\{\\ell\}\\in\\mathbb\{R\}^\{r\_\{\\ell\}\},\(50\)whereμℓ\\mu\_\{\\ell\}is the empirical mean of the hidden states at layerℓ\\ell\. The corresponding approximate reconstruction is

h^ℓ=μℓ\+Uℓ​qℓ\.\\hat\{h\}\_\{\\ell\}=\\mu\_\{\\ell\}\+U\_\{\\ell\}q\_\{\\ell\}\.\(51\)
The projected model is useful for testing whether the full network admits a low\-dimensional hidden representation\. However, it is not a true parameter\-compressed model, because it still requires the original full network parameters and the additional projection bases\. Its storage cost is therefore approximately

Nproj=Nθ\+∑ℓ=1L\(nℓ​rℓ\+nℓ\),N\_\{\\mathrm\{proj\}\}=N\_\{\\theta\}\+\\sum\_\{\\ell=1\}^\{L\}\(n\_\{\\ell\}r\_\{\\ell\}\+n\_\{\\ell\}\),\(52\)whereNθN\_\{\\theta\}is the parameter count of the full network\. Thus, projected C\-balanced reduction diagnoses state redundancy but does not by itself guarantee a smaller deployable model\.

### V\-BRealised C\-Balanced Architecture

To obtain a genuinely compressed neural network, we use the C\-balanced ranks directly as hidden\-layer widths\. If the full network has hidden widths

\[n1,n2,…,nL\],\[n\_\{1\},n\_\{2\},\\ldots,n\_\{L\}\],\(53\)the realised compressed network is defined with widths

\[r1,r2,…,rL\],rℓ≪nℓ\.\[r\_\{1\},r\_\{2\},\\ldots,r\_\{L\}\],\\qquad r\_\{\\ell\}\\ll n\_\{\\ell\}\.\(54\)For a fully connected DNN, the realised compressed dynamics are

h~ℓ\+1=σ​\(W~ℓ​h~ℓ\+b~ℓ\),ℓ=0,…,L−1,\\tilde\{h\}\_\{\\ell\+1\}=\\sigma\(\\tilde\{W\}\_\{\\ell\}\\tilde\{h\}\_\{\\ell\}\+\\tilde\{b\}\_\{\\ell\}\),\\qquad\\ell=0,\\ldots,L\-1,\(55\)whereh~ℓ∈ℝrℓ\\tilde\{h\}\_\{\\ell\}\\in\\mathbb\{R\}^\{r\_\{\\ell\}\}forℓ≥1\\ell\\geq 1andh~0=x\\tilde\{h\}\_\{0\}=x\. The output logits are

z~=W~L​h~L\+b~L\.\\tilde\{z\}=\\tilde\{W\}\_\{L\}\\tilde\{h\}\_\{L\}\+\\tilde\{b\}\_\{L\}\.\(56\)
The realised C\-balanced model has hidden\-state order

rstate=∑ℓ=1Lrℓ,r\_\{\\mathrm\{state\}\}=\\sum\_\{\\ell=1\}^\{L\}r\_\{\\ell\},\(57\)and, for a fully connected architecture, parameter count

NθC=d​r1\+r1\+∑ℓ=2L\(rℓ−1​rℓ\+rℓ\)\+rL​C\+C,N\_\{\\theta\}^\{\\mathrm\{C\}\}=dr\_\{1\}\+r\_\{1\}\+\\sum\_\{\\ell=2\}^\{L\}\(r\_\{\\ell\-1\}r\_\{\\ell\}\+r\_\{\\ell\}\)\+r\_\{L\}C\+C,\(58\)whereddis the input dimension andCCis the number of output classes\. Unlike projected reduction, the realised C\-balanced model no longer stores the full hidden layers or projection bases\. Therefore, it provides a true reduction in both state order and parameter count\.

### V\-CTraining the Realised C\-Balanced Model

The realised C\-balanced model can be trained in two ways\. The first option is training from scratch using the standard supervised cross\-entropy loss

ℒCE=−1N​∑i=1Nlog⁡exp⁡\(z~i,yi\)∑c=1Cexp⁡\(z~i,c\)\.\\mathcal\{L\}\_\{\\mathrm\{CE\}\}=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\log\\frac\{\\exp\(\\tilde\{z\}\_\{i,y\_\{i\}\}\)\}\{\\sum\_\{c=1\}^\{C\}\\exp\(\\tilde\{z\}\_\{i,c\}\)\}\.\(59\)This tests whether the C\-balanced ranks define a compact architecture that is sufficiently expressive on its own\.

The second option is knowledge distillation from the full model\. LetzTz\_\{T\}andzSz\_\{S\}denote the logits of the teacher and student networks, respectively\. The distilled loss is

ℒdistill=αℒCE\+\(1−α\)T2DKL\(softmax\(zTT\)∥softmax\(zST\)\),\\mathcal\{L\}\_\{\\mathrm\{distill\}\}=\\alpha\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\+\(1\-\\alpha\)T^\{2\}D\_\{\\mathrm\{KL\}\}\\left\(\\mathrm\{softmax\}\\left\(\\frac\{z\_\{T\}\}\{T\}\\right\)\\middle\\\|\\mathrm\{softmax\}\\left\(\\frac\{z\_\{S\}\}\{T\}\\right\)\\right\),\(60\)whereTTis the distillation temperature andα∈\[0,1\]\\alpha\\in\[0,1\]controls the balance between hard labels and teacher soft targets\. Distillation encourages the compressed network to reproduce the smoother class\-probability structure learned by the full model\.

### V\-DAlgorithm

The overall procedure is summarised in Algorithm[2](https://arxiv.org/html/2607.05457#alg2)\. First, a full network is trained\. Then, hidden\-state snapshots and output Jacobians are collected on a representative data subset\. These are used to compute empirical reachability and observability Gramians\. The C\-balanced matrix is then formed for each layer, and the energy criterion is used to select the C\-balanced rank\. Finally, a realised compressed network is instantiated using these ranks as its hidden\-layer widths and trained either from scratch or by distillation\.

Algorithm 2Realised C\-Balanced Neural Compression0:Trained full network

FF, dataset

𝒟\\mathcal\{D\}, rank tolerance

ε\\varepsilon
0:Realised compressed network

FCF\_\{\\mathrm\{C\}\}
1:Collect hidden\-state snapshots

\{hℓ​\(xi\)\}i=1N\\\{h\_\{\\ell\}\(x\_\{i\}\)\\\}\_\{i=1\}^\{N\}for each layer

ℓ\\ell
2:Compute the empirical reachability Gramian

Gc,ℓ=1N​H¯ℓT​H¯ℓG\_\{c,\\ell\}=\\frac\{1\}\{N\}\\bar\{H\}\_\{\\ell\}^\{T\}\\bar\{H\}\_\{\\ell\}
3:Compute output Jacobians

Jℓ​\(xi\)=∂z​\(xi\)∂hℓ​\(xi\)J\_\{\\ell\}\(x\_\{i\}\)=\\frac\{\\partial z\(x\_\{i\}\)\}\{\\partial h\_\{\\ell\}\(x\_\{i\}\)\}
4:Compute the empirical observability Gramian

Go,ℓ=1N​∑i=1NJℓ​\(xi\)T​Jℓ​\(xi\)G\_\{o,\\ell\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}J\_\{\\ell\}\(x\_\{i\}\)^\{T\}J\_\{\\ell\}\(x\_\{i\}\)
5:Form the balanced matrix

Bℓ=Gc,ℓ1/2​Go,ℓ​Gc,ℓ1/2B\_\{\\ell\}=G\_\{c,\\ell\}^\{1/2\}G\_\{o,\\ell\}G\_\{c,\\ell\}^\{1/2\}
6:Select the smallest

rℓr\_\{\\ell\}satisfying

∑i=1rℓλi∑i=1nℓλi≥1−ε\\frac\{\\sum\_\{i=1\}^\{r\_\{\\ell\}\}\\lambda\_\{i\}\}\{\\sum\_\{i=1\}^\{n\_\{\\ell\}\}\\lambda\_\{i\}\}\\geq 1\-\\varepsilon
7:Instantiate

FCF\_\{\\mathrm\{C\}\}with hidden widths

\[r1,…,rL\]\[r\_\{1\},\\ldots,r\_\{L\}\]
8:Train

FCF\_\{\\mathrm\{C\}\}from scratch or using distillation from

FF
9:return

FCF\_\{\\mathrm\{C\}\}

### V\-EComparison with Standard Compression Methods

The realised C\-balanced approach differs from standard compression methods in its compression criterion\. Unstructured pruning removes individual weights, typically according to magnitude, but it does not necessarily reduce dense layer dimensions or hardware latency\. Structured pruning removes neurons or channels, but often requires a separate importance heuristic and fine\-tuning stage\. Low\-rank SVD factorises weight matrices, but it is a layer\-local operation and may not preserve the nonlinear input–output representation\. Quantisation reduces numerical precision but leaves the hidden\-state order unchanged\.

In contrast, C\-balanced compression is state\-based rather than weight\-based\. It estimates which hidden directions are jointly excited by the data and relevant to the output\. The resulting ranks define the layer widths of a compact architecture\. Therefore, the method reduces the hidden\-state order directly and can also reduce parameter count and latency when implemented as a realised network\.

### V\-FExtension to CNNs and ResNets

Although the main experiments in this paper focus on fully connected DNNs for controlled state\-order analysis, the same principle extends naturally to convolutional and residual networks\. For CNNs and ResNets, the state at stageℓ\\ellcan be defined as a global\-average\-pooled feature vector

hℓ=GAP​\(Fℓ​\(x\)\),h\_\{\\ell\}=\\mathrm\{GAP\}\(F\_\{\\ell\}\(x\)\),\(61\)whereFℓ​\(x\)F\_\{\\ell\}\(x\)is the feature map produced by stageℓ\\ell\. In this setting, the C\-balanced rankrℓr\_\{\\ell\}estimates the number of useful channels at stageℓ\\ell\. Thus, the realised C\-balanced compression rule becomes a channel\-width reduction rule:

\[C1,C2,…,CL\]→\[r1,r2,…,rL\]\.\[C\_\{1\},C\_\{2\},\\ldots,C\_\{L\}\]\\rightarrow\[r\_\{1\},r\_\{2\},\\ldots,r\_\{L\}\]\.\(62\)This provides a pathway for extending the proposed framework from dense hidden\-state compression to channel compression in CNN and ResNet architectures\.

### V\-GInterpretation

The realised C\-balanced model can be interpreted as an empirical minimal\-realisation approximation of the trained network\. The full network may contain many hidden directions that are reachable but weakly observable, or observable but rarely excited by the data distribution\. The C\-balanced criterion retains directions that are simultaneously reachable and observable\. Consequently, the realised compressed network is designed around the task\-relevant hidden\-state order rather than around the original over\-parameterised width\.

This distinction is important\. A low\-dimensional projection may show that a compact representation exists, but it does not necessarily produce a deployable compressed model\. The realised C\-balanced construction converts the empirical rank estimate into an actual architecture\. Therefore, it bridges the gap between control\-theoretic state reduction and practical neural network compression\.

## VIExperiments

This section evaluates the proposed controllability–observability state\-compression framework on image classification tasks\. The objective is not only to measure classification accuracy, but also to determine whether the empirical reachable–observable ranks can be used to design genuinely compressed neural networks with fewer hidden\-state dimensions and fewer parameters\. We evaluate the proposed A/B/C tests on MNIST and CIFAR\-10 and compare the resulting compressed models against standard compression baselines, including unstructured pruning, structured pruning, low\-rank singular value decomposition \(SVD\), dynamic INT8 quantisation, and a linear classifier baseline\.

### VI\-AExperimental Setup

We consider a trained neural network as a depth\-indexed nonlinear state\-space system\. For a fully connected DNN, the hidden\-state dynamics are written as

hℓ\+1=σ​\(Wℓ​hℓ\+bℓ\),ℓ=0,…,L−1,h\_\{\\ell\+1\}=\\sigma\(W\_\{\\ell\}h\_\{\\ell\}\+b\_\{\\ell\}\),\\qquad\\ell=0,\\ldots,L\-1,\(63\)whereh0=xh\_\{0\}=xis the input,hℓ∈ℝnℓh\_\{\\ell\}\\in\\mathbb\{R\}^\{n\_\{\\ell\}\}is the hidden state at layerℓ\\ell, andσ​\(⋅\)\\sigma\(\\cdot\)is the activation function\. The output logits are given by

z=WL​hL\+bL\.z=W\_\{L\}h\_\{L\}\+b\_\{L\}\.\(64\)The total hidden\-state order of the full model is defined as

nstate=∑ℓ=1Lnℓ\.n\_\{\\mathrm\{state\}\}=\\sum\_\{\\ell=1\}^\{L\}n\_\{\\ell\}\.\(65\)
For each trained full model, we compute three empirical rank tests\. Test A estimates reachability from the hidden\-state snapshot covariance

Gc,ℓ=1N​\(H¯ℓ\)T​H¯ℓ,G\_\{c,\\ell\}=\\frac\{1\}\{N\}\(\\bar\{H\}\_\{\\ell\}\)^\{T\}\\bar\{H\}\_\{\\ell\},\(66\)whereH¯ℓ\\bar\{H\}\_\{\\ell\}denotes the centred hidden\-state snapshot matrix at layerℓ\\ell\. Test B estimates observability from output Jacobians:

Go,ℓ=1N​∑i=1NJℓ​\(xi\)T​Jℓ​\(xi\),Jℓ​\(xi\)=∂z​\(xi\)∂hℓ​\(xi\)\.G\_\{o,\\ell\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}J\_\{\\ell\}\(x\_\{i\}\)^\{T\}J\_\{\\ell\}\(x\_\{i\}\),\\qquad J\_\{\\ell\}\(x\_\{i\}\)=\\frac\{\\partial z\(x\_\{i\}\)\}\{\\partial h\_\{\\ell\}\(x\_\{i\}\)\}\.\(67\)Test C combines both effects through the balanced reachable–observable matrix

Bℓ=Gc,ℓ1/2​Go,ℓ​Gc,ℓ1/2\.B\_\{\\ell\}=G\_\{c,\\ell\}^\{1/2\}G\_\{o,\\ell\}G\_\{c,\\ell\}^\{1/2\}\.\(68\)The layer\-wise reduced rank is selected as the smallestrℓr\_\{\\ell\}satisfying

∑i=1rℓλi∑i=1nℓλi≥1−ε,\\frac\{\\sum\_\{i=1\}^\{r\_\{\\ell\}\}\\lambda\_\{i\}\}\{\\sum\_\{i=1\}^\{n\_\{\\ell\}\}\\lambda\_\{i\}\}\\geq 1\-\\varepsilon,\(69\)whereλi\\lambda\_\{i\}are the eigenvalues of the corresponding Gramian or balanced matrix andε=10−4\\varepsilon=10^\{\-4\}in all reported experiments\.

TABLE I:Experimental setup for MNIST and CIFAR\-10\.We distinguish between two uses of the A/B/C ranks\. First, projected A/B/C models use the learned projection bases to simulate reduced hidden states while still storing the original full network weights\. These models are useful as reduced\-state diagnostics, but they are not true parameter\-compressed networks\. Second, realised C\-balanced models use the C\-balanced ranks directly as compressed hidden\-layer widths\. Thus, a full model with widths\[n1,n2,…,nL\]\[n\_\{1\},n\_\{2\},\\ldots,n\_\{L\}\]is replaced by a compact model with widths\[r1,r2,…,rL\]\[r\_\{1\},r\_\{2\},\\ldots,r\_\{L\}\]\. This realised model is trained either from scratch or with knowledge distillation from the full model\.

### VI\-BMNIST Results

TABLE II:Layer\-wise A/B/C ranks for the MNIST SiLU DNN\. Each full hidden layer has dimension256256\.TABLE III:MNIST compression results for the four\-layer SiLU DNN\. The full model uses hidden widths\[256,256,256,256\]\[256,256,256,256\], giving state order10241024and400,906400\{,\}906trainable parameters\. Projected A/B/C models reduce the internal state order but still store the full network and projection bases\. Realised C\-balanced models use the C\-balanced ranks as actual compressed hidden widths\.MethodAcc\. \(%\)CEStateState Comp\. \(%\)Params/StorageLatency \(s\)Full DNN96\.600\.199410240\.00400,9060\.000427Projected A, reachability96\.600\.19969982\.54657,4180\.000941Projected B, observability96\.600\.199466135\.45571,1460\.000938Projected C, balanced96\.000\.253927772\.95472,8420\.000946Realised C\-balanced95\.450\.237027772\.95106,3230\.000409Distilled realised C\-balanced94\.700\.245527772\.95106,3230\.000409Unstructured pruning,50%50\\%96\.100\.180510240\.00200,970 nonzero0\.000431Structured pruning to C ranks34\.502\.254427772\.95106,3230\.000408Structured pruning \+ fine\-tuning91\.750\.318627772\.95106,3230\.000408Low\-rank SVD, rank fraction0\.50\.585\.500\.507910240\.00332,0920\.000709Dynamic INT8 quantisation96\.500\.201210240\.00411,303 bytes0\.446602 CPULinear classifier90\.700\.33050–7,8500\.000092
CE denotes cross\-entropy\. Latency is reported in ms/sample\. INT8 latency is measured on CPU and is therefore not directly comparable with CUDA timings\.

On MNIST, the full SiLU DNN achieves96\.60%96\.60\\%accuracy with state order10241024and400,906400\{,\}906parameters\. Test A is conservative, producing ranks\[251,249,250,248\]\[251,249,250,248\]and reducing the state order only to998998\. This indicates that most hidden directions are excited by the MNIST input distribution\. Test B produces ranks\[218,218,215,10\]\[218,218,215,10\], reducing the order to661661while preserving the full\-model accuracy\. The final\-layer rank of1010agrees with the1010\-class output space\.

The balanced C test gives the strongest reduction, estimating ranks\[114,90,63,10\]\[114,90,63,10\]\. This reduces the hidden\-state order from10241024to277277, corresponding to72\.95%72\.95\\%state compression\. The projected C\-balanced model achieves96\.00%96\.00\\%accuracy, only0\.600\.60percentage points below the full model\. However, the projected model still stores the original full network together with the projection bases and state means, so its storage proxy is larger than the original model\.

The realised C\-balanced model uses\[114,90,63,10\]\[114,90,63,10\]as the actual hidden widths of a new compressed network\. This reduces the parameter count from400,906400\{,\}906to106,323106\{,\}323, corresponding to73\.48%73\.48\\%parameter compression or approximately3\.77×3\.77\\timesfewer parameters\. The realised model achieves95\.45%95\.45\\%accuracy, only1\.151\.15percentage points below the full model\. This demonstrates that the C\-balanced ranks can be converted into a genuinely compressed DNN architecture rather than merely a projection\-based diagnostic\.

### VI\-CCIFAR\-10 Results

TABLE IV:Layer\-wise A/B/C ranks for the CIFAR\-10 SiLU DNN\. The full hidden widths are\[2048,1024,1024,512\]\[2048,1024,1024,512\]\.TABLE V:CIFAR\-10 compression results for the full SiLU DNN\. The full model uses hidden widths\[2048,1024,1024,512\]\[2048,1024,1024,512\], giving state order46084608and9,971,2109\{,\}971\{,\}210trainable parameters\. Projected A/B/C models reduce the internal state order but still store the full network and projection bases\. Realised C\-balanced models use the C\-balanced ranks as actual compressed hidden widths\.MethodAcc\. \(%\)CEStateState Comp\. \(%\)Params/StorageLatency \(s\)Full DNN54\.451\.743746080\.009,971,2100\.001582Projected A, reachability54\.531\.744743196\.2716,224,7780\.003042Projected B, observability54\.521\.7423219952\.2813,005,8340\.002406Projected C, balanced53\.711\.9200133970\.9411,761,6740\.002061Realised C\-balanced54\.441\.5221133970\.941,685,9170\.000525Distilled realised C\-balanced54\.651\.5662133970\.941,685,9170\.000512Unstructured pruning,50%50\\%53\.561\.620746080\.004,987,914 nonzero0\.001585Structured pruning to C ranks11\.402\.2911133970\.941,685,9170\.000436Structured pruning \+ fine\-tuning55\.221\.4291133970\.941,685,9170\.000523Low\-rank SVD, rank fraction0\.50\.539\.062\.068346080\.008,264,7640\.001418Dynamic INT8 quantisation54\.061\.689746080\.009,992,359 bytes0\.448731 CPULinear classifier38\.711\.78240–30,7300\.000090
CE denotes cross\-entropy\. Latency is reported in ms/sample\. INT8 latency is measured on CPU and is therefore not directly comparable with CUDA timings\.

The CIFAR\-10 experiment is more challenging because the model operates on flattened images and therefore lacks the spatial inductive bias of convolutional architectures\. The full SiLU DNN achieves54\.45%54\.45\\%accuracy with state order46084608and9\.979\.97M parameters\. Although the absolute accuracy is below what would be expected from CNN or ResNet architectures, the experiment provides a useful setting for studying hidden\-state redundancy and compression in a high\-dimensional fully connected classifier\.

The reachability test A produces ranks\[2035,769,1012,503\]\[2035,769,1012,503\], reducing the state order only from46084608to43194319\. This corresponds to6\.27%6\.27\\%state compression and indicates that CIFAR\-10 excites a large fraction of the hidden\-state space\. The observability test B is more selective, producing ranks\[765,923,501,10\]\[765,923,501,10\]and reducing the state order to21992199, corresponding to52\.28%52\.28\\%compression\. This shows that many hidden directions are reachable but only weakly observable at the output\.

The balanced C test estimates ranks\[410,521,398,10\]\[410,521,398,10\], reducing the state order from46084608to13391339\. This corresponds to70\.94%70\.94\\%state compression\. When these ranks are used as actual hidden widths, the realised C\-balanced model reduces the parameter count from9,971,2109\{,\}971\{,\}210to1,685,9171\{,\}685\{,\}917, corresponding to83\.09%83\.09\\%parameter compression or approximately5\.91×5\.91\\timesfewer parameters\. Despite this large reduction, the realised C\-balanced model achieves54\.44%54\.44\\%accuracy, essentially matching the full model’s54\.45%54\.45\\%accuracy\. The CUDA latency is reduced from0\.0015820\.001582ms/sample to0\.0005250\.000525ms/sample, corresponding to approximately3\.0×3\.0\\timesfaster inference\. With distillation, the compressed model achieves54\.65%54\.65\\%accuracy using the same state order and parameter count\.

### VI\-DOverall Discussion

TABLE VI:Summary of the main C\-balanced realised compression results\.Across both MNIST and CIFAR\-10, the results show that the balanced controllability–observability test provides a meaningful empirical estimate of the useful hidden\-state order of a trained DNN\. The reachability test A is generally conservative because many hidden directions are excited by the input data\. The observability test B is more selective because it removes directions that weakly affect the output\. The balanced test C is the most useful for compression because it retains directions that are both reachable from the data distribution and observable through the classifier output\.

The experiments also highlight the difference between state\-order reduction and parameter compression\. Projected A/B/C models reduce hidden\-state dimension but do not necessarily reduce storage, since they retain the original weights and projection bases\. Realised C\-balanced models, in contrast, convert the C\-balanced ranks into actual compressed layer widths\. This turns the empirical minimal\-realisation estimate into a true weight\-compressed neural architecture\.

Overall, the proposed approach reduces the MNIST DNN from state order10241024to277277and from400,906400\{,\}906to106,323106\{,\}323parameters while maintaining high accuracy\. On CIFAR\-10, it reduces the full DNN from state order46084608to13391339and from9\.979\.97M to1\.691\.69M parameters while preserving essentially the same accuracy and reducing CUDA latency by approximately3\.0×3\.0\\times\. These results support the central claim that balanced reachable–observable ranks can be used as an empirical minimal\-realisation criterion for designing compact DNN architectures\.

## VIILimitations

Although the proposed controllability–observability framework provides a principled way to estimate hidden\-state redundancy and design compact DNN architectures, several limitations remain\.

### VII\-AEmpirical and Data\-Dependent Ranks

The proposed A/B/C ranks are empirical quantities estimated from a finite set of input samples\. Therefore, they characterise the reachable and observable hidden\-state directions with respect to the sampled data distribution, rather than providing a worst\-case guarantee over the entire input space\. If the evaluation distribution differs significantly from the data used to estimate the Gramians, the selected ranks may underestimate directions that become important under distribution shift\. This is a general limitation of data\-driven model reduction and should be addressed by using representative data, validation checks, and, where possible, rank\-sensitivity analysis\.

### VII\-BCost of Observability Estimation

The reachability Gramian is inexpensive to compute because it only requires hidden\-state snapshots\. In contrast, the observability Gramian requires output Jacobians with respect to hidden states:

Jℓ​\(xi\)=∂z​\(xi\)∂hℓ​\(xi\)\.J\_\{\\ell\}\(x\_\{i\}\)=\\frac\{\\partial z\(x\_\{i\}\)\}\{\\partial h\_\{\\ell\}\(x\_\{i\}\)\}\.\(70\)This can be computationally expensive for very large networks, large hidden dimensions, or tasks with high\-dimensional outputs\. In the reported classification experiments, the output dimension is small, and the observability Gramian is estimated using a subset of samples\. For larger\-scale models, more efficient approximations may be needed, such as stochastic trace estimation, random output projections, block\-wise Jacobian approximations, or layer\-wise sampling\.

### VII\-CNo Formal Worst\-Case Error Bound

Classical balanced truncation for linear time\-invariant systems admits strong input–output error guarantees\. The present method, however, is applied to nonlinear, depth\-indexed, data\-driven neural networks\. As a result, the C\-balanced ranks should be interpreted as empirical minimal\-realisation estimates rather than exact minimal realisations in the strict control\-theoretic sense\. The experiments demonstrate that these ranks are useful for neural compression, but a general nonlinear approximation\-error bound remains an open theoretical direction\.

### VII\-DProjection Does Not Equal Deployment Compression

Projected A/B/C models are useful for diagnosing whether a low\-dimensional hidden\-state representation exists\. However, projected reduction still stores the original full network weights, as well as the projection bases and state means\. Therefore, projected models reduce the hidden\-state dimension but do not necessarily reduce storage or deployment cost\. True compression is obtained only when the estimated ranks are converted into realised compressed architectures with reduced hidden widths\.

### VII\-EDependence on Retraining

The realised C\-balanced model uses the estimated C\-balanced ranks as the widths of a new compact network\. This model is then trained from scratch or with distillation\. Therefore, the proposed approach is not a purely post\-training compression method\. Its effectiveness depends on whether the reduced architecture can be successfully trained\. This is consistent with recent observations in structured pruning, where the compressed architecture itself can be as important as the inherited weights\. However, it also means that the method requires additional training time\.

### VII\-FCurrent Focus on Fully Connected DNNs

The main experiments use fully connected DNNs on MNIST and CIFAR\-10 in order to provide a controlled setting for analysing hidden\-state order\. For CIFAR\-10, this means that the model operates on flattened images and does not exploit the spatial inductive bias of convolutional networks\. Consequently, the absolute CIFAR\-10 accuracy is not intended to compete with modern CNN or ResNet architectures\. Instead, the experiment demonstrates that large hidden\-state and parameter reductions are possible even in high\-dimensional nonlinear classifiers\.

### VII\-GExtension to CNNs, ResNets, and Transformers

The proposed framework can be extended to CNNs and ResNets by defining the state at each stage as a global\-average\-pooled feature vector,

hℓ=GAP​\(Fℓ​\(x\)\)\.h\_\{\\ell\}=\\mathrm\{GAP\}\(F\_\{\\ell\}\(x\)\)\.\(71\)In this setting, the C\-balanced ranks correspond to compressed channel widths\. However, a fully channel\-consistent implementation must also handle convolutional kernels, batch\-normalisation layers, residual shortcuts, and stage\-wise compatibility constraints\. Extending the same idea to transformers would require defining states over token embeddings, attention heads, MLP blocks, or low\-dimensional pooled representations\. These extensions are promising but require additional architecture\-specific design\.

### VII\-HHardware\-Aware Compression

The proposed method reduces state order and parameter count when realised as a smaller architecture\. However, actual latency improvements depend on hardware, kernel efficiency, batch size, memory bandwidth, and implementation details\. For example, unstructured pruning can remove many weights without improving dense CUDA latency, whereas width reduction usually maps better to standard dense kernels\. A future hardware\-aware version of the method could include latency or energy directly in the rank\-selection criterion\.

### VII\-IRank\-Tolerance Selection

The rank\-selection toleranceε\\varepsiloncontrols the trade\-off between compression and accuracy\. A smaller tolerance retains more hidden directions and gives less compression, whereas a larger tolerance gives more aggressive compression but may degrade accuracy\. In this paper, a fixed tolerance is used for direct comparison across A/B/C tests\. A more complete deployment\-oriented method could selectε\\varepsilonautomatically using validation accuracy, latency constraints, or Pareto optimisation\.

### VII\-JSummary of Limitations

In summary, the proposed method should be understood as an empirical, data\-driven minimal\-realisation approach rather than an exact model\-reduction theorem for arbitrary DNNs\. Its main strength is that it provides a principled and interpretable way to estimate useful hidden\-state dimensions and convert them into compact architectures\. Its main limitations are the cost of observability estimation, the empirical nature of the ranks, the need for retraining, and the need for further architecture\-specific extensions to CNNs, ResNets, transformers, and hardware\-aware deployment\.

## VIIIConclusion

This paper proposed an empirical minimal\-realisation framework for compressing deep neural networks using controllability–observability tests\. By interpreting a trained DNN as a depth\-indexed nonlinear state\-space system, we defined empirical reachability, observability, and balanced reachable–observable Gramians from hidden\-state snapshots and output Jacobians\. The resulting A/B/C tests provide layer\-wise estimates of the hidden\-state directions that are excited by the data distribution, observable through the output logits, and jointly reachable and observable\.

The central contribution is the use of the C\-balanced ranks as a practical compression rule\. Unlike projection\-only reduction, which diagnoses low\-dimensional hidden representations but still stores the original full model, the realised C\-balanced approach converts the estimated ranks into actual compressed layer widths\. This produces compact DNN architectures with fewer hidden\-state dimensions and fewer trainable parameters\.

Experiments on MNIST and CIFAR\-10 demonstrate the effectiveness of the proposed approach\. On MNIST, the C\-balanced test reduced the hidden\-state order from10241024to277277, corresponding to72\.95%72\.95\\%state compression\. The realised C\-balanced DNN reduced the parameter count from400,906400\{,\}906to106,323106\{,\}323, giving73\.48%73\.48\\%parameter compression, while maintaining95\.45%95\.45\\%accuracy compared with96\.60%96\.60\\%for the full model\. On CIFAR\-10, the C\-balanced test reduced the hidden\-state order from46084608to13391339, corresponding to70\.94%70\.94\\%state compression\. The realised compressed DNN reduced the parameter count from9\.979\.97M to1\.691\.69M, giving83\.09%83\.09\\%parameter compression, while preserving accuracy from54\.45%54\.45\\%to54\.44%54\.44\\%and reducing CUDA inference latency by approximately3\.0×3\.0\\times\.

The results also show that the proposed state\-based criterion is complementary to standard compression methods\. Unstructured pruning can preserve accuracy but does not necessarily reduce dense inference latency\. Low\-rank SVD can reduce storage but may damage the nonlinear task representation\. Structured pruning benefits from the C\-balanced widths after fine\-tuning, suggesting that the proposed ranks provide useful target dimensions for architecture design and pruning\.

Overall, the proposed framework provides a principled bridge between control\-theoretic model reduction and neural network compression\. It shows that balanced reachable–observable ranks can serve as empirical minimal\-realisation estimates for trained DNNs and can be used to design compact architectures with little or no loss in accuracy\. Future work will extend the method to convolutional networks, ResNets, transformers, recurrent models, and hardware\-aware compression, and will investigate theoretical approximation bounds for nonlinear neural state\-space reductions\.

## References

- \[1\]A\. C\. Antoulas\(2005\)Approximation of large\-scale dynamical systems\.SIAM,Philadelphia, PA, USA\.Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p3.1),[§II\-C](https://arxiv.org/html/2607.05457#S2.SS3.p1.1)\.
- \[2\]M\. Benning, E\. Celledoni, M\. J\. Ehrhardt, B\. Owren, and C\. Schönlieb\(2019\)Deep learning as optimal control problems: models and numerical methods\.Journal of Computational Dynamics6\(2\),pp\. 171–198\.External Links:[Document](https://dx.doi.org/10.3934/jcd.2019009)Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p2.3),[§II\-B](https://arxiv.org/html/2607.05457#S2.SS2.p1.2),[§III\-A](https://arxiv.org/html/2607.05457#S3.SS1.p1.1)\.
- \[3\]R\. T\. Q\. Chen, Y\. Rubanova, J\. Bettencourt, and D\. Duvenaud\(2018\)Neural ordinary differential equations\.InAdvances in Neural Information Processing Systems,Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p2.3),[§II\-B](https://arxiv.org/html/2607.05457#S2.SS2.p1.2),[§III\-A](https://arxiv.org/html/2607.05457#S3.SS1.p1.1),[§III\-B](https://arxiv.org/html/2607.05457#S3.SS2.p1.3)\.
- \[4\]E\. L\. Denton, W\. Zaremba, J\. Bruna, Y\. LeCun, and R\. Fergus\(2014\)Exploiting linear structure within convolutional networks for efficient evaluation\.InAdvances in Neural Information Processing Systems,Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.05457#S2.SS1.p3.1)\.
- \[5\]E\. Haber and L\. Ruthotto\(2017\)Stable architectures for deep neural networks\.Inverse Problems34\(1\),pp\. 014004\.External Links:[Document](https://dx.doi.org/10.1088/1361-6420/aa9a90)Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p2.3),[§II\-B](https://arxiv.org/html/2607.05457#S2.SS2.p1.2),[§III\-A](https://arxiv.org/html/2607.05457#S3.SS1.p1.1),[§III\-B](https://arxiv.org/html/2607.05457#S3.SS2.p1.3)\.
- \[6\]S\. Han, H\. Mao, and W\. J\. Dally\(2015\)Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding\.arXiv preprint arXiv:1510\.00149\.External Links:1510\.00149Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.05457#S2.SS1.p1.1)\.
- \[7\]K\. He, X\. Zhang, S\. Ren, and J\. Sun\(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 770–778\.Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p2.3),[§II\-B](https://arxiv.org/html/2607.05457#S2.SS2.p1.2),[§III\-A](https://arxiv.org/html/2607.05457#S3.SS1.p1.1)\.
- \[8\]G\. Hinton, O\. Vinyals, and J\. Dean\(2015\)Distilling the knowledge in a neural network\.arXiv preprint arXiv:1503\.02531\.External Links:1503\.02531Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.05457#S2.SS1.p5.1)\.
- \[9\]B\. Jacob, S\. Kligys, B\. Chen, M\. Zhu, M\. Tang, A\. Howard, H\. Adam, and D\. Kalenichenko\(2018\)Quantization and training of neural networks for efficient integer\-arithmetic\-only inference\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 2704–2713\.Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.05457#S2.SS1.p4.1)\.
- \[10\]S\. Lall, J\. E\. Marsden, and S\. Glavaški\(2002\)A subspace approach to balanced truncation for model reduction of nonlinear control systems\.International Journal of Robust and Nonlinear Control12\(6\),pp\. 519–535\.External Links:[Document](https://dx.doi.org/10.1002/rnc.657)Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p3.1),[§II\-D](https://arxiv.org/html/2607.05457#S2.SS4.p1.1)\.
- \[11\]Z\. Liu, J\. Li, Z\. Shen, G\. Huang, S\. Yan, and C\. Zhang\(2017\)Learning efficient convolutional networks through network slimming\.InProceedings of the IEEE International Conference on Computer Vision,pp\. 2736–2744\.Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p1.1),[§II\-A](https://arxiv.org/html/2607.05457#S2.SS1.p2.1)\.
- \[12\]Z\. Liu, M\. Sun, T\. Zhou, G\. Huang, and T\. Darrell\(2019\)Rethinking the value of network pruning\.InInternational Conference on Learning Representations,Cited by:[§II\-A](https://arxiv.org/html/2607.05457#S2.SS1.p2.1)\.
- \[13\]B\. C\. Moore\(1981\)Principal component analysis in linear systems: controllability, observability, and model reduction\.IEEE Transactions on Automatic Control26\(1\),pp\. 17–32\.External Links:[Document](https://dx.doi.org/10.1109/TAC.1981.1102568)Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p3.1),[§II\-C](https://arxiv.org/html/2607.05457#S2.SS3.p1.1)\.
- \[14\]L\. Ruthotto and E\. Haber\(2020\)Deep neural networks motivated by partial differential equations\.Journal of Mathematical Imaging and Vision62,pp\. 352–364\.External Links:[Document](https://dx.doi.org/10.1007/s10851-019-00903-1)Cited by:[§I](https://arxiv.org/html/2607.05457#S1.p2.3),[§II\-B](https://arxiv.org/html/2607.05457#S2.SS2.p1.2),[§III\-A](https://arxiv.org/html/2607.05457#S3.SS1.p1.1),[§III\-B](https://arxiv.org/html/2607.05457#S3.SS2.p1.3)\.