Topological Simplification in Predictive Coding Networks

# Topological Simplification in Predictive Coding Networks
Source: [https://arxiv.org/html/2608.02816](https://arxiv.org/html/2608.02816)
\\theorembodyfont\\theoremheaderfont\\theorempostheader

:\\theoremsep \\jmlrvolume334\\jmlryear2026\\jmlrworkshopTopology, Algebra, and Geometry in Data Science

\\NameAdam Shaw\\Emailadshaw@usc\.edu \\NameJiayu Li\\Emailjli99757@usc\.edu \\NameMichael Sperling\\Emailmesperli@usc\.edu \\NameMichael Kim\\Emailmkim2763@usc\.edu \\NameAlvin Jin\\Emailalvinjin@usc\.edu \\addrUniversity of Southern California 3551 Trousdale ParkwayCA 90089

###### Abstract

We study the topology of learned representations in predictive coding networks \(PCNs\), a neuro\-inspired bidirectional architecture, using a quantitative layer\-wise persistent homology analysis\. We train well\-performing PCNs on a synthetic classification dataset \(≥99\.9%\\geq 99\.9\\%test accuracy\) and on MNIST \(≥95%\\geq 95\\%test accuracy\), and measure how topological features change across layers for different architectures and activation functions\. We find that smaller PCNs collapse connected components across layers earlier than larger models \(Spearmanρ∈\[0\.72,0\.79\]\\rho\\in\[0\.72,0\.79\]across activations\), with model size measured as the sum of hidden\-layer widths\. We also observe a strong negative correlation \(ρ=−0\.58\\rho=\-0\.58\) between the depth at which simplification occurs and reconstruction error; i\.e\., architectures that simplify later reconstruct better\. Finally, a seed\-level bootstrap comparison across architectures and activations shows that PCNs consistently collapse connected components later than matched MLPs, with an average difference of3\.63\.6layers\. These results suggest that persistent homology offers a useful quantitative lens on the compression–reconstruction tradeoff in PCNs, and that both model capacity and the recurrent, bidirectional dynamics of predictive coding inference shape when this tradeoff is resolved across layers\.

###### keywords:

Persistent Homology, Topological Simplification, Predictive Coding Networks, Representation Learning, Topological Data Analysis, Generative Models

## 1Introduction

Several studies have used persistent homology\(edelsbrunner2002persistence\)to analyze how topological structure evolves within deep neural network representations, showing that feedforward networks progressively simplify topology by collapsing connected components, loops, and other homological features\(Naitzat2020;Watanabe\_Yamana\_2021;suresh2024ph;ergen2024relunets\)\. Whether this phenomenon extends beyond feedforward architectures remains unclear\. In particular, little is known about how topology evolves in models that must simultaneously support inference and reconstruction, such as predictive coding networks \(PCNs\)\(rao1999predictive\)\.

In this work, we use persistent homology to study the evolution of representation topology in PCNs trained on both synthetic and real\-world datasets\. To this end, we introduce two metrics: one that quantifies when topological simplification occurs across depth, and one that measures reconstruction and inversion quality\.

Our results show that PCNs exhibit progressive topological simplification while keeping some ability for inversion\. Smaller models simplify topology earlier than larger models, and earlier simplification is associated with poorer reconstruction performance\. Together, these findings extend topological analyses of learned representations beyond feedforward networks and highlight a connection between topological simplification and reconstruction in generative architectures\.

## 2Background

### 2\.1Predictive Coding Networks

Predictive coding networks \(PCNs\) are hierarchical generative models that perform inference through iterative minimization of a global energy functional rather than a single feedforward computation\(rao1999predictive;whittington2017approximation;Stenlund2025\)\. Unlike conventional feedforward architectures, PCNs support approximate inversion of learned representations through the same recurrent dynamics used for inference\(rao1999predictive;sun2020inversion\)\. From a topological perspective, this makes PCNs a particularly informative test case: whereas feedforward classifiers may freely collapse topological structure to produce separable representations, predictive coding networks must preserve sufficient information to support reconstruction\.

PCNs have also attracted interest as a biologically\-plausible alternative to backpropagation\(backprop\)by avoiding theweight transport problem\(GROSSBERG198723\): While backpropagation requires error signals to pass through exact transposes of downstream weight matrices \(a biologically\-implausible operation\), PCNs use local interactions between neighboring layers\. Despite this locality constraint, predictive coding updates approximate backpropagation to a high degree\(whittington2017approximation\)\. We lay out more details regarding the PCN formulation we use in Appendix[A](https://arxiv.org/html/2608.02816#A1)\.

![Refer to caption](https://arxiv.org/html/2608.02816v1/x1.png)Figure 1:Left: Predictive Coding Network architecture; Right: PCN inference dynamics\(whittington2017approximation;rao1999predictive\)\.
### 2\.2Topological Simplification in Neural Network Representations

Persistent homology provides a framework for quantifying topological structure in finite point clouds by summarizing homological features across scales as a persistence barcode\(edelsbrunner2002persistence\)\. Persistent homology distinguishes robust topological structure from sampling noise by identifying features that persist across scales, making it particularly well\-suited for analyzing high\-dimensional neural representations inferred from finite samples\. In this work, we use persistent homology as a descriptive tool for tracking the evolution of topological complexity across layers of predictive coding networks\.

## 3Methodology

### 3\.1Objective

Our experiments test whether PCNs exhibit topological simplification and identify the architectural factors that govern this behavior\. We analyze how topological invariants evolve across layers in PCNs, comparing their behavior to standard feedforward networks to assess whether topological simplification may be a general property of deep representations or depends on specific architectural choices\. To isolate these effects, we systematically vary hidden\-layer width and activation function, measuring how these choices influence the timing and extent of Betti number decay\. We further examine how topological simplification relates to the approximate invertibility enabled by predictive coding dynamics, testing whether pressure to support reconstruction constrains excessive topological collapse\. All models are implemented using theJAX\-basedPCXlibrary\(jax;pinchetti2025pcx\)\.

### 3\.2Datasets

We conduct our experiments on both synthetic and real\-world datasets\. For synthetic data, we sample from a topologically\-complex, two\-dimensional manifold made up ofMaM\_\{a\}\(nine disjoint disks, i\.e\., withβ0=9,β1=0\\beta\_\{0\}=9,\\ \\beta\_\{1\}=0\) positioned withinMbM\_\{b\}\(a single connected region containing nine holes, i\.e\., withβ0=1,β1=9\\beta\_\{0\}=1,\\ \\beta\_\{1\}=9\); see\\figurereffig:D1\_picture\. This dataset follows the construction introduced byNaitzat2020, which we adopt because the classes are not linearly separable or even separable by any simple topological deformation, meaning any classifier must perform significant topological transformations on the data to pull the classes apart in feature space\. To complement this analysis, we also evaluate our approach on MNIST\(lecun1998mnist\), a widely used real\-world image dataset whose intrinsic topology is unknown and substantially more complex\.

\\floatconts

fig:D1\_picture

Figure 2:Visualization of the synthetic dataset, consisting of manifoldMaM\_\{a\}\(green\) embedded in manifoldMbM\_\{b\}\(red\), adapted fromNaitzat2020\.![Refer to caption](https://arxiv.org/html/2608.02816v1/x2.png)

### 3\.3Model Variants

For both the synthetic dataset and MNIST, we evaluate a range of architectures that vary in hidden\-layer width and in how that width progresses across layers, including uniform\-width networks, networks with an increasing/decreasing width, and networks with a bottleneck in the middle\. Despite this variety, all models share a fixed depth of eight hidden layers, so that topological simplification at each layer can be directly compared across architectures\. For each dataset, we train 30 independent instances of each model configured using distinct random seeds\. Synthetic data models were trained to at least99\.9%99\.9\\%test accuracy, while models trained on MNIST achieved at least95%95\\%test accuracy due to increased complexity\.

Throughout this paper, we refer to a model as havingLLhidden layers, with layer indicesℓ∈\{0,…,L\+1\}\\ell\\in\\\{0,\\ldots,L\+1\\\}including the input and output layers\. A complete list of all architectures used is provided in\\appendixrefapp:model\_details\.

### 3\.4Topology Tracking Procedure

We adopt and extend the topological analysis framework ofNaitzat2020, with help fromwheeler2021, to quantify how networks simplify representation topology across depth\. For each layerℓ\\ell, we collect the hidden representations of a fixed set of data points and construct a metric space using either a normalizedkk\-nearest neighbors \(kk\-NN\) graph with unweighted geodesic distance \(for synthetic data\) or a normalized Euclidean distance \(for real\-world data\), both of which mitigate arbitrary scaling differences across layers\. Persistent homology is then computed on this metric space, and Betti numbers are evaluated at a fixed filtration scaleη\\etato obtain per\-layer scalar summaries\. To avoid conflating the topologies of different classes, we restrict persistent homology computations to a single class per dataset: manifoldMaM\_\{a\}for the synthetic data, and the digit\-0class for MNIST\. Formal implementation details \(including the choice ofkkandη\\eta\) are provided in\\appendixrefsec:topology\_tracking\_appendix\.

### 3\.5Metrics

We define two new metrics: one to quantify timing of topological simplification, and another to quantify reconstruction fidelity\.

#### 3\.5\.1Center of Mass of Topological Simplification

LetB​\(ℓ\)=∑k∈𝒦wk​βk\(ℓ\)B\(\\ell\)=\\sum\_\{k\\in\\mathcal\{K\}\}w\_\{k\}\\beta\_\{k\}^\{\(\\ell\)\}be a weighted sum of Betti numbers at layerℓ\\ell, for some range of homological degrees \(e\.g\.,𝒦=\{0,1\}\\mathcal\{K\}=\\\{0,1\\\}forβ0\\beta\_\{0\}andβ1\\beta\_\{1\}\), and letB~​\(ℓ\)=minj≤ℓ⁡B​\(j\)\\tilde\{B\}\(\\ell\)=\\min\_\{j\\leq\\ell\}B\(j\)be its running minimum, so that simplification events are treated as irreversible\. The layer\-wise Betti dropΔ​\(ℓ\)=B~​\(ℓ−1\)−B~​\(ℓ\)\\Delta\(\\ell\)=\\tilde\{B\}\(\\ell\-1\)\-\\tilde\{B\}\(\\ell\)then defines a distribution over layersp​\(ℓ\)=Δ​\(ℓ\)/Dp\(\\ell\)=\\Delta\(\\ell\)/D, whereD=∑ℓΔ​\(ℓ\)D=\\sum\_\{\\ell\}\\Delta\(\\ell\)is the total simplification across all layers\. IfD=0D=0, the distributionp​\(ℓ\)p\(\\ell\)is undefined andCOM\\mathrm\{COM\}is left unassigned; this case did not arise in any of our experiments\.

###### Definition 3\.1\(COM\)\.

To quantify when simplification occurs, we define the*Center of Mass of Topological Simplification \(COM\)*as the expected depth of topological simplification under this distribution:

COM=∑ℓ=1L\+1ℓ​p​\(ℓ\)=∑ℓ=1L\+1ℓ​Δ​\(ℓ\)∑ℓ=1L\+1Δ​\(ℓ\)\.\\mathrm\{COM\}\\;=\\;\\sum\_\{\\ell=1\}^\{L\+1\}\\ell\\,p\(\\ell\)\\;=\\;\\frac\{\\sum\_\{\\ell=1\}^\{L\+1\}\\ell\\,\\Delta\(\\ell\)\}\{\\sum\_\{\\ell=1\}^\{L\+1\}\\Delta\(\\ell\)\}\.\(1\)

A lower COM indicates earlier topological collapse\. For conciseness, we writeCOM​\(β0\+β1\)\\mathrm\{COM\}\(\\beta\_\{0\}\+\\beta\_\{1\}\)as shorthand to indicate thatB​\(ℓ\)=β0\(ℓ\)\+β1\(ℓ\)B\(\\ell\)=\\beta\_\{0\}^\{\(\\ell\)\}\+\\beta\_\{1\}^\{\(\\ell\)\}, and similarly for other linear combinations of Betti numbers\.

#### 3\.5\.2Mean Reconstruction Distance

To quantify reconstruction fidelity, we measure how closely inverted outputs resemble valid same\-class inputs\. For each ofM=30M=30independently\-trained modelsmmof an architectureAA, we passN=1000N=1000synthetic one\-hot outputsyiy\_\{i\}through the inversion procedure \(see Appendix[A\.1](https://arxiv.org/html/2608.02816#A1.SS1)\), each yielding a reconstructionx^i\(m\)\\hat\{x\}\_\{i\}^\{\(m\)\}\. We then find its nearest Euclidean neighborxi∗\(m\)x\_\{i\}^\{\*\(m\)\}in the training set restricted to the same class\.

###### Definition 3\.2\(MRD\)\.

We define the*Mean Reconstruction Distance \(MRD\)*of an architectureAAto be the mean Euclidean distance between reconstructions and their nearest same\-class neighbors, averaged overNNreconstructions andMMmodel seeds:

MRD​\(A\)=1M​N​∑m=1M∑i=1N‖x^i\(m\)−xi∗\(m\)‖2\.\\mathrm\{MRD\}\(A\)=\\frac\{1\}\{MN\}\\sum\_\{m=1\}^\{M\}\\sum\_\{i=1\}^\{N\}\\left\\\|\\hat\{x\}\_\{i\}^\{\(m\)\}\-x\_\{i\}^\{\*\(m\)\}\\right\\\|\_\{2\}\.\(2\)

By comparing to the nearest same\-class neighbor rather than a fixed target, MRD captures whether inversions lie near the correct data manifold without penalizing valid novel reconstructions\.

See FigureLABEL:fig:recons\_comparisonfor a comparison of the inversion behavior of two models with substantially different MRD values\.

## 4Results

Across PCN architectures, we observe a consistent trend of topological simplification, with the timing and extent of this simplification varying systematically with architectural capacity and activation function, and correlating with reconstruction quality\. This analysis was conducted through a multitude of experiments available publicly on GitHub\.111https://github\.com/jiayuliusc/Topological\-Simplification\-in\-Predictive\-Coding\-Networks

\\floatconts

fig:betti\_trends

Figure 3:Zeroth Betti numbers \(β0\\beta\_\{0\}\) across layers for two eight\-layer ReLU architectures\. Each plot shows the meanβ0\\beta\_\{0\}at each layer, its±1\\pm 1SD band, and individual seeds\. The monotonic decline in the alternating\-width model \(right\) suggests genuine topological simplification rather than simple geometric compression and re\-expansion between layers\.\\subfigure

\[β0\\beta\_\{0\}for 30x4 \+ 18x4 ReLU\]![Refer to caption](https://arxiv.org/html/2608.02816v1/Figures/30x4_18x4_relu.png)\\subfigure\[β0\\beta\_\{0\}for Alternating \(30,​ 15\) ReLU\]![Refer to caption](https://arxiv.org/html/2608.02816v1/Figures/30_15x4.png)

### 4\.1Synthetic Data

##### Architectural Constraints

Our results indicate that smaller models exhibit topological simplification earlier, on average, than larger models\. To quantify this relationship, we examine the correlation between simplification timing \(COM\) and model size, measured as the sum of hidden\-layer widths\. As shown in FigureLABEL:fig:com\_vs\_hidden, simplification timing is strongly positively correlated with model size across activation functions, with Spearman correlation coefficients of0\.760\.76for ReLU,0\.790\.79for Leaky ReLU, and0\.720\.72fortanh\\tanh\. This stratification is necessary because the activation function is a confounding variable that influences both representation geometry and simplification behavior\. A likely explanation for these results is that larger models have the capacity to apply more complex geometric transformations while preserving the overall shape of the data manifold, whereas smaller models may be forced to collapse this structure earlier in order to produce a separable representation\.

\\floatconts

fig:com\_vs\_hidden

Figure 4:Model size versus simplification timing \(COM​\(β0\)\\mathrm\{COM\}\(\\beta\_\{0\}\)\) on the synthetic dataset, shown separately for each activation function\.\\subfigure

\[ReLU activations\]![Refer to caption](https://arxiv.org/html/2608.02816v1/Figures/all_relu_models_5to4_COM_vs_HIDDEN.png)\\subfigure\[Leaky ReLU activations\]![Refer to caption](https://arxiv.org/html/2608.02816v1/Figures/all_leaky_models_5to4_COM_vs_HIDDEN.png)\\subfigure\[tanh\\tanhactivations\]![Refer to caption](https://arxiv.org/html/2608.02816v1/Figures/all_tanh_models_5to4_COM_vs_HIDDEN.png)

##### Capacity Sensitivity by Activation Function

To further investigate this relationship across activation functions, we fit a separate linear model for each activation,

COMa​\(P\)=αa\+γa​P\+ε,\\mathrm\{COM\}\_\{a\}\(P\)=\\alpha\_\{a\}\+\\gamma\_\{a\}P\+\\varepsilon,\(3\)whereP=∑ℓ=1LnℓP=\\sum\_\{\\ell=1\}^\{L\}n\_\{\\ell\}is the total hidden width of the architecture andaadenotes the activation function\. The fitted slopes were positive for all three activations:100​γ^=1\.79100\\hat\{\\gamma\}=1\.79for ReLU,0\.960\.96for Leaky ReLU, and0\.910\.91fortanh\\tanh, with correspondingR2R^\{2\}values of0\.590\.59,0\.630\.63, and0\.410\.41, respectively\. Thus, while increasing capacity delays simplification for all activations, the effect is strongest for ReLU: its slope is nearly twice that of Leaky ReLU andtanh\\tanh\. This suggests that ReLU PCNs are more capacity\-sensitive, in the sense that added width is associated with a substantially greater delay in topological collapse than in PCNs using other activation functions\.

##### Simplification\-Reconstruction Tradeoff

We observe a strong negative correlation between COM and MRD \(ρ=−0\.58,p<10−3\\rho=\-0\.58,\\;p<10^\{\-3\}\), indicating that architectures that delay topological simplification preserve invertibility; see\\figurereffig:mrd\_com\. Intuitively, COM measures when irreversible geometric mergers occur in representation space, while MRD measures how much structure remains available for inversion\. Collapsing connected components corresponds to identifying previously distinct regions of the input manifold, so when this occurs early, information is lost, which harms reconstruction quality\.

\\floatconts

fig:mrd\_com

Figure 5:Graphical relationship and correlation between early topological simplification \(COM​\(β0\)\\mathrm\{COM\}\(\\beta\_\{0\}\)\) and reconstruction error \(MRD\) in architectures trained on the synthetic dataset\. The strong negative correlation highlights a clear association between earlier topological simplification and weaker reconstruction performance\.![Refer to caption](https://arxiv.org/html/2608.02816v1/Figures/all_models_MRD_vs_COM.png)

### 4\.2Consistency with Real\-World Data

We extend our analysis to the MNIST dataset of handwritten digits\(lecun1998mnist\)to evaluate whether the topological simplification phenomena observed in controlled synthetic settings persist in realistic, high\-dimensional data\. Due to the substantial computational cost of persistent homology estimation on high\-dimensional point clouds, we restrict our MNIST experiments to a set of representative architectures, detailed in\\tablereftab:mnist\_models\. Following the manifold view of image data\(fefferman2016testing;Naitzat2020\), we treat each digit class as a point cloud inℝ784\\mathbb\{R\}^\{784\}and trackβ0\\beta\_\{0\}andβ1\\beta\_\{1\}across layers of trained PCNs\. We use a Vietoris–Rips scale parameter acrossη∈\[0\.1,0\.9\]\\eta\\in\[0\.1,\\,0\.9\], but limit our focus toη∈\{0\.2,0\.3\}\\eta\\in\\\{0\.2,0\.3\\\}; the reason is that for smaller values ofη\\eta, the point clouds remain overly fragmented across most layers, obscuring meaningful trends in connectivity; conversely, for larger values ofη\\eta, the representations become rapidly connected, yielding trivial topological summaries\. The chosen range thus balances sensitivity to topological changes while enabling consistent comparison across layers and architectures\.

Because the MNIST analysis includes only a small number of representative architectures, correlation\-based analyses between simplification timing \(COM\) and model size are statistically underpowered\. Therefore, we adopt a distributional comparison approach, visualizing the COM values across models using violin plots \(\\figurereffig:com\_comparison\_mnist\)\. Despite the lack of a known ground\-truth topology for MNIST, we observe trends consistent with both our synthetic experiments and prior studies of feedforward networks\(Naitzat2020\)\. In particular, PCNs exhibit progressive topological simplification across layers, as reflected by a decrease inβ0\\beta\_\{0\}, with the onset and rate of simplification influenced by architectural constraints and activation functions\.

We do not report MNIST reconstructions\. PCN reconstructions on MNIST are known to be difficult\(orchard2019genpcn\), and in our runs the reconstructions were unstable or low quality\. We therefore restrict the MNIST study to representational topology\.

\\floatconts

fig:com\_comparison\_mnist

Figure 6:Violin plots ofCOM​\(β0\+β1\)\\mathrm\{COM\}\(\\beta\_\{0\}\+\\beta\_\{1\}\)for three MNIST architectures \(tanh\\tanhactivation\), ordered left to right by decreasing total hidden\-layer width, computed from3030independently trained networks per configuration\. Lower COM indicates earlier topological simplification\. For meaning of architecture names, refer to\\tablereftab:mnist\_models\. Model sizes are40964096,19201920, and14401440, respectively\.![Refer to caption](https://arxiv.org/html/2608.02816v1/Figures/three_tanh_models_8to5_B0_eta0.2.png)

### 4\.3Comparison with Feedforward Networks

We compare PCNs trained on the synthetic dataset with matched multilayer perceptrons \(MLPs\) trained to the same accuracy\. For each architecture, activation function, and trained seed/model, we compute a singleCOM​\(β0\)\\mathrm\{COM\}\(\\beta\_\{0\}\)value from the persistence diagrams\. We then bootstrap seed\-level means with replacement and compareΔCOM=COM¯PCN−COM¯MLP\\Delta\_\{\\mathrm\{COM\}\}=\\overline\{\\mathrm\{COM\}\}\_\{\\mathrm\{PCN\}\}\-\\overline\{\\mathrm\{COM\}\}\_\{\\mathrm\{MLP\}\}\.

Our results show a consistent positive COM gap: PCNs have larger COMs than matched MLPs for all architectures and activations tested\. The bootstrap analysis yielded a p\-value ofp<10−4p<10^\{\-4\}, rejecting the null hypothesis thatΔCOM≤0\\Delta\_\{\\mathrm\{COM\}\}\\leq 0\. The average COM differenceΔCOM\\Delta\_\{\\mathrm\{COM\}\}was3\.63\.6, meaning our PCNs simplified topology on average3\.63\.6layers later than an MLP of the same architecture and activation\. See\\figurereffig:bootstrap\_difference\_distributions for the complete results\.

## 5Conclusion

In this work, we use persistent homology to study how representation topology evolves across layers in predictive coding networks\. Our results show that topological simplification is not uniform across architectures: smaller models simplify earlier, while larger models \(trained to the same accuracy\) preserve topological structure deeper into the network\. We also find that earlier simplification is correlated with weaker inversion quality\. Models that simplify rapidly tend to discard structurally relevant information before it can be reconciled with top\-down predictions, resulting in poorer generative reconstructions\.

Together, these findings suggest a capacity–topology–reconstruction tradeoff in PCNs: increasing capacity can delay topological simplification, improving invertibility, whereas smaller models with earlier simplification produce more compressed but less reconstructible representations\. More broadly, our results demonstrate that persistent homology can serve as a useful quantitative probe of learned representations in recurrent, bidirectional architectures\. Future work can test whether the same trends hold across broader datasets and architectures, and can develop refined reconstruction\-aware topological metrics \(such as extensions of Mean Reconstruction Distance\) that better capture semantic fidelity in real\-world data\.

## References

## Appendix APredictive Coding Network Formulation

We implement our models using theJAX\-basedPCXlibrary\(pinchetti2025pcx\), which follows the formulation of PCNs introduced bywhittington2017approximation\.

Each layerℓ\\ellcontains neural activitiesxi\(ℓ\)x\_\{i\}^\{\(\\ell\)\}, wherei∈\{1,…,nℓ\}i\\in\\\{1,\\dots,n\_\{\\ell\}\\\}indexes units within the layer\. Top\-down predictions are defined via a generative model\. The predicted mean activityμi\(ℓ\)\\mu\_\{i\}^\{\(\\ell\)\}at layerℓ\\ellis given by

μi\(ℓ\)=∑j=1nℓ\+1θi,j\(ℓ\+1\)​f​\(xj\(ℓ\+1\)\),\\mu\_\{i\}^\{\(\\ell\)\}=\\sum\_\{j=1\}^\{n\_\{\\ell\+1\}\}\\theta\_\{i,j\}^\{\(\\ell\+1\)\}\\,f\\\!\\left\(x\_\{j\}^\{\(\\ell\+1\)\}\\right\),\(A\.1\)whereθi,j\(ℓ\+1\)\\theta\_\{i,j\}^\{\(\\ell\+1\)\}denotes the synaptic weight from unitjjin layerℓ\+1\\ell\+1to unitiiin layerℓ\\ell, andf​\(⋅\)f\(\\cdot\)is a \(possibly nonlinear\) activation function\.

Prediction errors are defined as variance\-normalized residuals

ϵi\(ℓ\)=xi\(ℓ\)−μi\(ℓ\)Σi\(ℓ\),\\epsilon\_\{i\}^\{\(\\ell\)\}=\\frac\{x\_\{i\}^\{\(\\ell\)\}\-\\mu\_\{i\}^\{\(\\ell\)\}\}\{\\Sigma\_\{i\}^\{\(\\ell\)\}\},\(A\.2\)whereΣi\(ℓ\)\\Sigma\_\{i\}^\{\(\\ell\)\}denotes the variance associated with unitiiat layerℓ\\ell\.

Under a Gaussian noise assumption, the variational free energy is given by

F=12​∑ℓ=0L∑i=1nℓ\(xi\(ℓ\)−μi\(ℓ\)\)2Σi\(ℓ\),F=\\frac\{1\}\{2\}\\sum\_\{\\ell=0\}^\{L\}\\sum\_\{i=1\}^\{n\_\{\\ell\}\}\\frac\{\\left\(x\_\{i\}^\{\(\\ell\)\}\-\\mu\_\{i\}^\{\(\\ell\)\}\\right\)^\{2\}\}\{\\Sigma\_\{i\}^\{\(\\ell\)\}\},\(A\.3\)where theLLthlayer is the last hidden layer\.

Inference corresponds to gradient descent on this free energy with respect to the neural activities\. The resulting continuous\-time dynamics for unitbbin layeraaare

x˙b\(a\)=−ϵb\(a\)\+∑i=1na−1ϵi\(a−1\)​θi,b\(a\)​f′​\(xb\(a\)\),\\dot\{x\}\_\{b\}^\{\(a\)\}=\-\\epsilon\_\{b\}^\{\(a\)\}\+\\sum\_\{i=1\}^\{n\_\{a\-1\}\}\\epsilon\_\{i\}^\{\(a\-1\)\}\\,\\theta\_\{i,b\}^\{\(a\)\}\\,f^\{\\prime\}\\\!\\left\(x\_\{b\}^\{\(a\)\}\\right\),\(A\.4\)wheref′​\(⋅\)f^\{\\prime\}\(\\cdot\)denotes the derivative of the activation function\.

Learning proceeds via gradient descent on the variational free energy with respect to the synaptic parameters\. The resulting gradient is given bywhittington2017approximation:

∂F∂θi,j\(ℓ\)=ϵi\(ℓ−1\)​f​\(xj\(ℓ\)\)\.\\frac\{\\partial F\}\{\\partial\\theta\_\{i,j\}^\{\(\\ell\)\}\}=\\epsilon\_\{i\}^\{\(\\ell\-1\)\}\\,f\\\!\\left\(x\_\{j\}^\{\(\\ell\)\}\\right\)\.\(A\.5\)This confirms that both inference and learning depend only on locally available pre\- and post\-synaptic signals\.

Because inference is defined by minimizing a shared energy functional rather than propagating signals through a fixed forward pathway, PCNs naturally admit approximate inversion by clamping higher\-level states and inferring lower\-level representations through the same dynamics\.

### A\.1Inversion

The recurrent, bidirectional dynamics of PCNs naturally enable what we refer to as*inversion*: the generation of input\-level representations from fixed higher\-level states\(rao1999predictive;sun2020inversion\)\. In our implementation, inversion is performed by clamping the output\-layer target and then directly optimizing the input through a top\-down reconstruction of the lower layers that best explains that imposed state under the learned generative parameters\. Concretely, we solve an energy\-minimization problem of the form

z^∈argmin𝑧​F​\(z,ytarget\),\\hat\{z\}\\in\\underset\{z\}\{\\mathrm\{argmin\}\}\\ F\(z,y\_\{\\text\{target\}\}\),\(A\.6\)whereytargety\_\{\\text\{target\}\}is a one\-hot class vector andFFis the variational free energy \(\\equationrefeq:pcn\_energy\)\.

To keep reconstructions bounded, we optimize this unconstrained auxiliary variablez^\\hat\{z\}and setx^=tanh⁡\(z^\)\\hat\{x\}=\\tanh\(\\hat\{z\}\)to be our “reconstruction”, ensuringx^∈\(−1,1\)d\\hat\{x\}\\in\(\-1,1\)^\{d\}; this matches the natural domain of our datasets\. Starting from a random initialization ofzz, we perform a fixed number of gradient descent steps on the energy functionFF\. This stochasticity allows the same target output to yield a distribution of plausible reconstructions rather than a single deterministic result\.

We leverage this inversion mechanism in\\sectionrefsec:mrd to assess the internal consistency and representational fidelity of the learned model; see\\figurereffig:recons\_comparison for a visualization\.

\\floatconts

fig:recons\_comparison\\subfigure\[30x8\_leakymodel\]![Refer to caption](https://arxiv.org/html/2608.02816v1/Figures/30x8_leaky_model_2.png)\\subfigure\[30x4\_10x4\_relumodel\]![Refer to caption](https://arxiv.org/html/2608.02816v1/Figures/30x4_10x4_relu_model_29.png)

Figure A\.1:Reconstructed inputs from two PCNs \(red\), overlaid on the true manifoldMaM\_\{a\}\(green\)\. Reconstructions by30x8\_leaky\(left\) remain localized within components, while those by30x4\_10x4\_relu\(right\) are substantially more scattered and disconnected, indicating weaker approximate invertibility, aligning with the former’s significantly lower MRD \(0\.0240\.024vs\.0\.1220\.122, respectively\)\.

## Appendix BModel Details

Below is a summary of the models used in our experiments, including activation functions used and the number of neurons in each layer\. Our implementation of Leaky ReLU weighs negative inputs by0\.20\.2\.

\\floatconts

tab:all\_models

Table B\.1:Summary of architectures trained on the synthetic dataset with their hidden layer widths and activation functions used\.\\floatconts

tab:mnist\_models

Table B\.2:Summary of architectures trained on MNIST with their hidden layer widths and activation functions used\.
## Appendix CTopology Tracking Procedure

### C\.1Synthetic Data

We adopt and extend the topological analysis framework ofNaitzat2020to study how the topology of a point\-cloud dataset evolves as it propagates through the hidden layers of a PCN\. All models are first trained to convergence, achieving at least99\.9%99\.9\\%test accuracy\.

As inNaitzat2020, we compute and track persistent homology on only one of the two manifolds in the synthetic dataset, denotedMaM\_\{a\}in\\sectionrefsec:datasets and\\figurereffig:D1\_picture\. Concretely, before extracting layer\-wise representations, we filter the dataset to retain only points belonging toMaM\_\{a\}\(i\.e\., a single class\) and then compute Betti numbers on this restricted point cloud\. This restriction is necessary because the full dataset is a union of two manifolds; computing homology on the union would conflate the topology of manifoldMaM\_\{a\}with that of manifoldMbM\_\{b\}and with inter\-manifold separations, obscuring the within\-manifold simplification behavior we aim to measure\.

To ensure comparability across layers with varying activation scales, we compute topology using the unweighted geodesic distance on akk\-NN graph constructed over each layer’s representations\. By assigning unit weight to allkk\-NN edges, this metric normalizes distances within each latent space, reducing sensitivity to absolute scale and enabling the use of a fixed persistent homology thresholdη=2\.5\\eta=2\.5across layers\.

The choice of thekk\-NN parameterkkis critical: akktoo small results in a disconnected graph \(β0\>ground truth\\beta\_\{0\}\>\\text\{ground truth\}\), while akktoo large induces “short\-circuiting” that collapses the manifold’s intrinsic structure\. To determine an optimalkkfor the synthetic dataset, we perform a Monte Carlo cross\-validation procedure:

1. 1\.Subsampling: In each ofn=1000n=1000trials, we randomly sample a fraction \(25%25\\%\) of the synthetic manifold to simulate varying densities\.
2. 2\.Iterative Search: For each subset, we iteratively increasekkfrom11to100100, constructing the corresponding geodesic distance matrix\.
3. 3\.Homological Matching: We compute the zeroth Betti number \(β0\\beta\_\{0\}\) using the distance matrix and identify the minimalkkrequired to recover the ground\-truth topology of manifoldMaM\_\{a\}\(β0=9\\beta\_\{0\}=9\)\.

The distribution converges to a mode ofk=14k=14\. This value consistently recovers the true connectivity of the manifold while remaining below the threshold of over\-connectivity\. To maintain consistency across all comparative experiments, we fixk=14k=14and utilize a single representative25%25\\%subset whose topology exactly matches the ground\-truth manifold signature\. This aligns with the methodology used inNaitzat2020\.

Persistent homology is then computed via Vietoris–Rips complexes using theRipserPython package\(bauer2021ripser\), itself relying on foundational work fromzomorodian2004phandedelsbrunner2002persistence\. A crucial result comes fromattali2013vrcomplex, showing that for a large enough sample size and at a sufficiently small scale, the topology of the VR\-complex of a manifold reflects the true topology of that manifold\.

### C\.2Real\-World Data

For real\-world datasets such as MNIST, the true topology of the underlying data manifold is unknown\. Consequently, unlike in synthetic settings, there is no way to select or validate akk\-NN construction by verifying recovery of the correct input\-layer topology, motivating the use of other distance metrics\.

The most immediate alternative is Euclidean distance; however, when computing persistent homology on layer\-wise activations, raw Euclidean distances are not directly comparable across layers due to arbitrary expansions or contractions induced by network weights\(Naitzat2020\)\. Without normalization, changes in persistent homology may therefore reflect trivial scaling effects rather than meaningful changes in representational structure\.

To address this issue, we followwheeler2021, who adopt a simple per\-layer metric normalization procedure: For a fixed layerℓ\\ell, letX\(ℓ\)=\{x1\(ℓ\),…,xn\(ℓ\)\}⊂ℝnℓX^\{\(\\ell\)\}=\\\{x\_\{1\}^\{\(\\ell\)\},\\dots,x\_\{n\}^\{\(\\ell\)\}\\\}\\subset\\mathbb\{R\}^\{n\_\{\\ell\}\}denote the activations ofnndistinct data points at that layer\. We compute the full pairwise Euclidean distance matrix and rescale it by its maximum entry, defining a normalized metric

dℓ​\(xi,xj\)=‖xi\(ℓ\)−xj\(ℓ\)‖2maxp,q⁡‖xp\(ℓ\)−xq\(ℓ\)‖2\.d\_\{\\ell\}\(x\_\{i\},x\_\{j\}\)\\;=\\;\\frac\{\\\|x\_\{i\}^\{\(\\ell\)\}\-x\_\{j\}^\{\(\\ell\)\}\\\|\_\{2\}\}\{\\max\_\{p,q\}\\\|x\_\{p\}^\{\(\\ell\)\}\-x\_\{q\}^\{\(\\ell\)\}\\\|\_\{2\}\}\.\(C\.1\)This normalization ensures that the diameter of the metric space\(X\(ℓ\),dℓ\)\(X^\{\(\\ell\)\},d\_\{\\ell\}\)is equal to one for every layer, thereby placing all layers on a common distance scale\.

edelsbrunner2010computationalshow that this per\-layer distance normalization does not alter the intrinsic topology recovered at any fixed layer, but merely rescales the filtration parameter, allowing Betti numbers to be compared across layers without confounding effects due to varying magnitudes of network weights\.

Persistent homology is otherwise calculated identically as in\\appendixrefsec:topology\_tracking\_d1\_appendix, with the digit\-0class being tracked\. For computational efficiency, we restrict attention to homology dimensionsk∈\{0,1\}k\\in\\\{0,1\\\}, which capture connected components and one\-dimensional loops, respectively\.

## Appendix DAdditional Figures

\\floatconts

fig:bootstrap\_difference\_distributions![Refer to caption](https://arxiv.org/html/2608.02816v1/x3.png)

Figure D\.1:Bootstrap distributions of COM mean differences between matched PCNs and MLPs\. Each panel showsΔCOM=COMPCN−COMMLP\\Delta\_\{\\mathrm\{COM\}\}=\\mathrm\{COM\}\_\{\\mathrm\{PCN\}\}\-\\mathrm\{COM\}\_\{\\mathrm\{MLP\}\}for one architecture–activation pair\. The dashed vertical line marks zero; all distributions lie to the right of zero, indicating larger COM for PCNs\.