On the Depth Scalability of Logic Gate Networks

# On the Depth Scalability of Logic Gate Networks
Source: [https://arxiv.org/html/2607.21633](https://arxiv.org/html/2607.21633)
###### Abstract

Logic Gate Networks \(LGNs\) implement computation through compositions of Boolean operations, yet unlike classical Boolean circuits, existing LGNs do not reliably benefit from increased depth\. We identify two distinct causes: optimization collapse in deep relaxed LGNs and a topology\-induced limitation that persists even when skip\-biased initialization and straight\-through estimation stabilize training\. Thus, trainability alone is insufficient; deeper layers must also receive information that supports useful computation\.

We introduce Input\-Anchored Logic Gate Networks \(IALGNs\), in which each gate combines an evolving hidden feature with a direct input anchor\. This topology preserves a computational spine while conditioning every layer on the original input\. We show that a depth\-DDpath can depend on up toD\+1D\+1input bits and establish a strict path\-wise depth hierarchy\. Random\-kkanchor relaxation further improves anchor selection without relaxing the spine\.

Across MNIST, CIFAR\-10, and CIFAR\-100, IALGNs achieve consistent fixed\-width depth–accuracy improvements beyond 100 layers, whereas alternative LGN topologies saturate or degrade\. Layer\-wise probes, topology ablations, and effective\-depth analysis show that input anchoring produces progressively more informative representations and preserves longer computational paths\. These results demonstrate that scalable depth in LGNs requires both stable optimization and an information\-access pattern that supports input\-conditioned refinement\.

![Refer to caption](https://arxiv.org/html/2607.21633v1/Figures/intro_homemade.png)Figure 1:Connectivity overview of the two architectures\. \(a\) In a randomly wired LGN \(RWLGN\), both input pins of a gate receive features from the previous hidden layer\. \(b\) In an input\-anchored LGN \(IALGN\), the spine pin receives a feature from the previous hidden layer, whereas the anchor pin receives a feature directly from the original input\. Thus, each IALGN layer remains conditioned on the input while refining its hidden representation\.## 1Introduction

Logic Gate Networks \(LGNs\) are neural architectures that implement computation through compositions of Boolean operations, closely resembling classical Boolean circuits\. By replacing conventional real\-valued neurons with logic gates, LGNs provide a structured and interpretable computational model with direct relevance to efficient discrete inference\.

From the perspective of Boolean circuit theory, depth is a fundamental computational resource\. Increasing circuit depth can strictly enlarge representational power, and broad classes of functions require exponentially larger circuits when restricted to insufficient depth\(Hastad[1986](https://arxiv.org/html/2607.21633#bib.bib1)\)\. These results suggest that scaling LGNs in depth should be both theoretically meaningful and practically beneficial\.

Constructing an LGN requires determining both the logical operation performed by each gate and the wiring that connects gates across layers\. Because this design space is discrete and combinatorial, differentiable LGNs replace discrete operation choices with continuous relaxations that can be optimized by gradient descent\(Petersenet al\.[2022](https://arxiv.org/html/2607.21633#bib.bib4); Kim[2023](https://arxiv.org/html/2607.21633#bib.bib11)\)\. Despite substantial progress in gate optimization, however, existing LGNs do not exhibit reliable depth–performance scaling\. Most reported architectures remain relatively shallow, and increasing depth frequently causes accuracy to saturate or decline\. Consequently, the computational benefits predicted by circuit theory remain largely unrealized in practical LGNs\.

This failure is commonly attributed to optimization difficulties\. Gradients propagated through many relaxed Boolean operations rapidly vanish, while discrepancies between relaxed training and discrete inference further destabilize learning\. Recent advances, including stochastic relaxations, straight\-through estimators \(STE\), and skip\-biased initialization, substantially improve trainability and stabilize optimization in deep LGNs\(Kim[2023](https://arxiv.org/html/2607.21633#bib.bib11); Petersenet al\.[2024](https://arxiv.org/html/2607.21633#bib.bib5); Yousefiet al\.[2025](https://arxiv.org/html/2607.21633#bib.bib6); Rüttgerset al\.[2025](https://arxiv.org/html/2607.21633#bib.bib7)\)\. Nevertheless, even when optimization remains stable, randomly wired LGNs still obtain little benefit from increasing depth\.

We argue that depth scalability requires satisfying two distinct conditions\. First, deeper layers must receive sufficient learning signals \(*optimization*\)\. Second, the network topology must enable each additional layer to perform useful computation rather than repeatedly recombining existing hidden representations \(*computation*\)\. Existing optimization techniques largely address the former but leave the latter unchanged\.

To address this limitation, we propose theInput\-Anchored Logic Gate Network\(IALGN\)\. Each gate combines one feature propagated through a fixed hidden computational spine with one feature directly sampled from the original input\. This simple modification allows every layer to refine accumulated computation while continually incorporating new input information\. We further show that the resulting topology induces a strict path\-wise depth hierarchy, separating the computational capability provided by the architecture from the trainability provided by optimization\.

Empirically, skip\-biased initialization and STE stabilize optimization in both randomly wired LGNs and IALGNs, yet only IALGNs exhibit systematic fixed\-width depth scaling\. Across MNIST, CIFAR\-10, and CIFAR\-100, layer\-wise probes show progressively improving representations, topology ablations demonstrate that input anchoring is essential, and effective\-depth analysis shows that IALGNs preserve longer computational paths throughout the network\.

Our contributions are summarized as follows:

- •We distinguish two obstacles to depth scalability in LGNs: optimization collapse and a topology\-induced limitation in progressive representation refinement\. We show that resolving optimization alone does not make additional depth useful\.
- •We propose IALGN, a minimal input\-anchored topology with random\-kkanchor relaxation, and establish a strict path\-wise depth hierarchy showing that increasing depth enlarges the computational capability of individual paths\.
- •We demonstrate consistent fixed\-width depth scaling on MNIST, CIFAR\-10, and CIFAR\-100\. Linear probes, topology ablations, and effective computational\-depth distributions connect the observed performance gains to progressively richer representations and longer surviving computational paths rather than generic wiring flexibility\.

## 2Related Work

### Differentiable Logic Gate Networks\.

Differentiable Logic Gate Networks \(DLGNs\) enable gradient\-based learning of Boolean circuits by relaxing the discrete selection of the 16 two\-input Boolean functions into a differentiable optimization problem\(Petersenet al\.[2022](https://arxiv.org/html/2607.21633#bib.bib4)\)\. The original formulation employs randomly wired feedforward connections together with skip\-biased initialization, establishing the foundation for subsequent LGN research\.

Recent work has improved DLGNs along several largely orthogonal directions\. Convolutional DLGNs extend logic\-gate computation to spatial vision tasks through structured logic kernels\(Petersenet al\.[2024](https://arxiv.org/html/2607.21633#bib.bib5)\)\. Other studies improve optimization by reducing the discretization gap or introducing more effective gate parameterizations, including stochastic relaxations, straight\-through estimators, and lightweight gate representations\(Kim[2023](https://arxiv.org/html/2607.21633#bib.bib11); Yousefiet al\.[2025](https://arxiv.org/html/2607.21633#bib.bib6); Rüttgerset al\.[2025](https://arxiv.org/html/2607.21633#bib.bib7)\)\. More recently, recurrent parameter sharing\(Bühreret al\.[2025](https://arxiv.org/html/2607.21633#bib.bib8)\), differentiable connectivity optimization\(Mommenet al\.[2026](https://arxiv.org/html/2607.21633#bib.bib14)\), and hardware\-oriented connectivity learning\(Fojciket al\.[2026](https://arxiv.org/html/2607.21633#bib.bib15)\)have further broadened the design space of DLGNs\. Unlike our work, however, these approaches primarily modify gate parameterization, optimization, or connectivity learning, rather than the information\-access pattern across layers\. Consequently, they do not explicitly investigate whether the inter\-layer topology itself limits depth scalability\.

### Topology and Information Flow in Deep Networks\.

In conventional deep neural networks, architectural connectivity plays a central role in making depth effective\. Residual and dense connections improve information propagation by creating short paths between early and late representations, allowing deep networks to progressively refine features rather than repeatedly overwrite them\(Veitet al\.[2016](https://arxiv.org/html/2607.21633#bib.bib13); Bhardwajet al\.[2019](https://arxiv.org/html/2607.21633#bib.bib12)\)\. Applying these principles to LGNs is non\-trivial because logic gates operate under strict two\-input constraints and saturating Boolean semantics, making standard additive skip connections inapplicable\. Instead of learning arbitrary hidden connectivity, we introduce a minimal topology in which every gate preserves a hidden\-layer spine while retaining direct access to the original input\. Our approach is therefore complementary to previous LGN research: rather than improving individual gates or their optimization, we investigate whether modifying the information\-access pattern itself enables scalable depth\.

![Refer to caption](https://arxiv.org/html/2607.21633v1/Figures/opt_collapse4.png)Figure 2:Optimization stability alone is insufficient for depth scalability\. \(a,b\) Layer\-wise update magnitudes under different initialization and training settings\. Skip\-biased initialization and STE substantially improve update propagation across deep networks\. \(c\) Despite stable optimization, RWLGN fails to benefit from increasing depth, whereas IALGN continues to improve, indicating that optimization alone does not explain depth scalability\. The IALGN curve in subfigure \(c\) uses Skip init \+ STE\.

## 3Why Deep DLGNs Do Not Scale

Optimization collapse is a well\-known obstacle to training deep Logic Gate Networks \(LGNs\)\. Because gradients must propagate through many relaxed Boolean operations, the learning signal reaching early layers rapidly diminishes as depth increases, often preventing deep models from converging\. Recent studies have substantially alleviated this problem through skip\-biased initialization and straight\-through estimators \(STE\), enabling stable optimization of considerably deeper LGNs\(Petersenet al\.[2022](https://arxiv.org/html/2607.21633#bib.bib4); Kim[2023](https://arxiv.org/html/2607.21633#bib.bib11); Yousefiet al\.[2025](https://arxiv.org/html/2607.21633#bib.bib6); Rüttgerset al\.[2025](https://arxiv.org/html/2607.21633#bib.bib7)\)\. This naturally raises an important question: if optimization can already be stabilized, why do randomly wired LGNs still fail to benefit from increasing depth?

To separate optimization from architectural effects, we compare randomly wired LGNs \(RWLGNs\) and the proposed IALGN under four representative training settings: random initialization, skip\-biased initialization, STE, and their combination\. Following\(Petersenet al\.[2022](https://arxiv.org/html/2607.21633#bib.bib4)\), we measure layer\-wise parameter update magnitudes rather than raw gradients, since adaptive optimizers internally rescale gradients before applying parameter updates\. Larger update magnitudes therefore indicate that a layer participates more actively in optimization\.

Figure[2](https://arxiv.org/html/2607.21633#S2.F2)\(a,b\) shows that modern optimization techniques substantially alleviate optimization collapse\. Skip\-biased initialization greatly improves optimization in RWLGNs, while STE provides particularly strong stabilization in IALGNs\. Under the strongest optimization setting \(Skip init \+ STE\), parameter updates remain consistently large throughout nearly all layers in both architectures, indicating that optimization collapse is largely mitigated\.

However, stable optimization alone does not make additional depth useful\. Figure[2](https://arxiv.org/html/2607.21633#S2.F2)\(c\) examines predictive performance as network depth increases under the same training settings\. Despite healthy parameter updates throughout the network, RWLGNs exhibit little or no benefit from increasing depth and eventually begin to degrade\. In contrast, IALGNs consistently improve as additional layers are introduced under the identical optimization protocol\.

These observations reveal that depth scalability requires satisfying two distinct conditions\. Optimization determines whether deep networks can be trained successfully, whereas network topology determines whether deeper layers perform additional useful computation\. Stable optimization is therefore necessary, but not sufficient, for scalable depth\. The remainder of this paper investigates how modifying the information available to each layer enables progressively more useful computation as depth increases\. In the next section, we show that providing every gate with direct access to the original input fundamentally changes the computation enabled by increasing depth\.

## 4Input\-Anchored Logic Gate Networks

![Refer to caption](https://arxiv.org/html/2607.21633v1/Figures/arch1.png)Figure 3:Input\-anchored Logic Gate Networks and random\-kkanchor relaxation\. \(a\) IALGN replaces the fully random inter\-layer wiring of standard LGNs with a two\-pin structure\. The*spine pin*receives the corresponding hidden feature from the previous layer, preserving a layer\-to\-layer computational path\. The*anchor pin*receives a feature directly from the original input, ensuring that every layer remains conditioned on the input\. \(b\) Random\-kkanchor relaxation sampleskkcandidate input anchors for each gate and learns a relaxed selection among them, improving anchor flexibility without searching over all possible input connections\.### Design Principle

The results in Section[3](https://arxiv.org/html/2607.21633#S3)suggest that the remaining limitation lies in the information available to each layer\. In conventional RWLGNs, every gate receives only hidden features produced by the preceding layer\. As depth increases, later gates repeatedly compose Boolean functions over features that already summarize previous computations, primarily increasing the complexity of existing feature interactions\. IALGN instead preserves the hidden computational pathway while continually exposing every layer to the original input\. Each layer therefore performs input\-conditioned refinement, updating accumulated computation using newly accessible input information\. Consequently, increasing depth expands the computation performed by the network rather than merely recombining earlier representations\.

One may view the proposed architecture as a collection of parallel computational chains\. However, each chain performs a depth\-dependent Boolean composition rather than an independent shallow prediction; the shared GroupSum readout aggregates the resulting deep features, not an ensemble of standalone classifiers\.

![Refer to caption](https://arxiv.org/html/2607.21633v1/Figures/rep_progess3.png)Figure 4:Progressive representation learning enabled by input anchoring\. \(a\) Layer\-wise linear\-probe accuracy on frozen hidden representations\. Unlike randomly wired DLGNs, IALGN exhibits progressively improving probe accuracy as depth increases on both CIFAR\-10 \(solid\) and CIFAR\-100 \(dashed\), indicating that deeper layers produce increasingly informative representations\. \(b\) Probe accuracy versus average binary activation entropy\. Although entropy provides only a supporting diagnostic of hidden\-state diversity, IALGN consistently achieves higher probe accuracy across comparable entropy levels, suggesting that its representational improvement cannot be explained solely by activation statistics\.
### Input\-Anchored Topology

Letx∈\{0,1\}dinx\\in\\\{0,1\\\}^\{d\_\{\\mathrm\{in\}\}\}denote the binarized input andhℓ∈\{0,1\}Wh\_\{\\ell\}\\in\\\{0,1\\\}^\{W\}the hidden representation at layerℓ\\ell\.

In a conventional randomly wired DLGN, each gate receives both of its inputs from the previous hidden layer,

hℓi=gℓi​\(hℓ−1aℓ,i,hℓ−1bℓ,i\),h\_\{\\ell\}^\{i\}=g\_\{\\ell\}^\{i\}\\left\(h\_\{\\ell\-1\}^\{a\_\{\\ell,i\}\},h\_\{\\ell\-1\}^\{b\_\{\\ell,i\}\}\\right\),\(1\)
wheregℓig\_\{\\ell\}^\{i\}denotes one of the sixteen Boolean operations andaℓ,i,bℓ,ia\_\{\\ell,i\},b\_\{\\ell,i\}are randomly sampled hidden indices\.

IALGN modifies only the source of one gate input,

hℓi=gℓi​\(hℓ−1i,xbℓ,i\)\.h\_\{\\ell\}^\{i\}=g\_\{\\ell\}^\{i\}\\left\(h\_\{\\ell\-1\}^\{i\},x^\{b\_\{\\ell,i\}\}\\right\)\.\(2\)
The first input, referred to as the*spine pin*, preserves a layer\-to\-layer computational path through the network\. The second input, referred to as the*anchor pin*, connects every layer directly to the original input\. For the first layer, both inputs are drawn from the binarized input\.

Importantly, this modification changes neither the gate operations nor the optimization algorithm\. The gate family, differentiable relaxation, loss function, and readout remain identical to standard DLGNs\. The only architectural change is the source of one gate input\. Nevertheless, this modification fundamentally changes the information available to every layer by allowing hidden computations to be continually refined using original input variables\.

### Computational Property

The architectural modification above admits a simple characterization of the computation performed by an individual computational path\.

###### Proposition 1\(Input Dependency\)

Any depth\-DDcomputational path in an IALGN depends on at mostD\+1D\+1distinct input bits\. Furthermore, this bound is tight\.

Proof\.\[Proof Sketch\] A depth\-0gate depends on one input bit\. Each subsequent layer receives one hidden feature from the previous layer and one direct input bit\. Since the hidden feature already summarizes all previously accessible inputs, each additional layer can introduce at most one previously unseen input bit\. Therefore, after depthDD, a computational path depends on at mostD\+1D\+1distinct input bits\. The bound is achieved by selecting a different input bit at every layer\.

Proposition 1 immediately yields a strict depth hierarchy\.

###### Theorem 1\(Strict Depth Hierarchy\)

For every integerD≥1D\\geq 1, there exists a Boolean function computable by a depth\-DDcomputational path in an IALGN that cannot be computed by any depth\-\(D−1\)\(D\-1\)computational path\.

Proof\.\[Proof Sketch\] Consider the parity function overD\+1D\+1input bits\. By Proposition 1, every depth\-\(D−1\)\(D\-1\)computational path depends on at mostDDinput bits and therefore cannot represent\(D\+1\)\(D\+1\)\-bit parity\. Conversely, because XOR is one of the primitive Boolean operations used in IALGN, a depth\-DDcomputational path can sequentially incorporate one new input bit per layer and compute parity over allD\+1D\+1inputs\. Hence, increasing depth strictly enlarges the class of Boolean functions representable by an individual computational path\.

Unlike the optimization analysis in Section[3](https://arxiv.org/html/2607.21633#S3), Proposition 1 and Theorem 1 characterize the computational capability induced by the network topology itself\. Stable optimization determines whether useful parameters can be learned, whereas the proposed topology determines what additional computation deeper layers are capable of performing\. This theoretical distinction motivates the empirical analyses in the following sections, which examine whether trained IALGNs actually exploit this additional computational capability\.

### Random\-kkAnchor Relaxation

A single fixed input anchor per gate is simple and efficient but may limit feature reuse when the input dimension is large\. To improve anchor flexibility without introducing combinatorial wiring search, IALGN sampleskkcandidate input anchors for each gate and learns a relaxed selection among them\. This relaxation preserves the architectural principle of input anchoring while allowing greater flexibility in selecting informative input variables\.

### Training and Inference

IALGN employs the same differentiable Boolean\-operation relaxation as standard DLGNs\. Each gate maintains learnable logits over the sixteen Boolean operations and is optimized using either raw relaxed gradients or the straight\-through estimator\. Unless otherwise specified, all experiments adopt the skip\-biased initialization proposed in previous LGN studies\.

During inference, both the Boolean operation and, when random\-kkrelaxation is used, the anchor selection are discretized using argmax\. Consequently, the resulting model remains an ordinary two\-input Boolean circuit with identical inference complexity to conventional LGNs\.

## 5Progressive Representation Learning

Section[4](https://arxiv.org/html/2607.21633#S4)established that input anchoring increases the computational capability of individual computational paths by continually exposing deeper layers to new input variables\. While this result characterizes the expressive capacity of the proposed architecture, it remains unclear whether trained networks actually exploit this additional computational capability\. We therefore investigate how hidden representations evolve as depth increases\.

### Linear Probe Analysis

To measure the information contained in intermediate representations, we train an independent linear classifier on the frozen hidden representation produced by each layer\. Since the probe itself introduces no additional nonlinear computation, its accuracy directly reflects the linear separability of the learned representation\.

Figure[4](https://arxiv.org/html/2607.21633#S4.F4)\(a\) reports the layer\-wise probe accuracy on CIFAR\-10 and CIFAR\-100\. Randomly wired DLGNs exhibit little improvement beyond the early layers, and probe accuracy gradually deteriorates as depth increases\. This behavior suggests that repeatedly recombining transformed hidden features provides limited representational benefit\.

In contrast, IALGN exhibits a steady improvement in probe accuracy throughout depth on both datasets\. As additional layers receive both the accumulated hidden computation and previously unseen input variables, intermediate representations become progressively easier to decode\. This empirical observation agrees with the theoretical analysis in Section 4: increased depth is effectively translated into increasingly expressive computation rather than merely deeper sequences of feature recombination\.

### Activation Entropy

To further examine whether the observed improvement is simply a consequence of changing activation statistics, Figure[4](https://arxiv.org/html/2607.21633#S4.F4)\(b\) compares probe accuracy with the average binary activation entropy of each layer\.

Although activation entropy alone is not a direct measure of representation quality, it provides a useful characterization of hidden\-state diversity\. The results show that IALGN consistently achieves substantially higher probe accuracy across comparable entropy levels than randomly wired DLGNs\. Consequently, the superior representations learned by IALGN cannot be attributed solely to differences in activation entropy, but instead reflect increasingly informative computations enabled by the proposed topology\.

Taken together, these results provide empirical evidence that the additional computational capability predicted by Section 4 is effectively utilized during learning\. Rather than merely stabilizing optimization, input anchoring enables progressively richer intermediate representations as network depth increases\.

Table 1:Fixed\-width depth scaling results\. Within each row, RWLGN and IALGN use identical width and depth and therefore the same number of deployed logic gates\. Results are mean±\\pmstandard deviation over five seeds\. LDLGN reimpl\. denotes our reimplementation under the same no\-augmentation protocol and should be interpreted as a within\-paper reference, not as a reproduction of the original reported results\. The best result in each row is shown in bold\.DatasetWidthDepthGatesRWLGNLDLGN reimpl\.IALGNCIFAR\-1012k448k50\.56±0\.5850\.56\\pm 0\.5850\.22±0\.1850\.22\\pm 0\.1853\.59±\\pm0\.04CIFAR\-1012k10120k50\.51±0\.6350\.51\\pm 0\.6350\.87±0\.3650\.87\\pm 0\.3654\.25±\\pm0\.06CIFAR\-1012k50600k50\.16±0\.3350\.16\\pm 0\.3345\.40±0\.3845\.40\\pm 0\.3856\.12±\\pm0\.05CIFAR\-1012k1001\.2M48\.73±0\.5048\.73\\pm 0\.5038\.85±0\.8738\.85\\pm 0\.8756\.41±\\pm0\.17CIFAR\-1012k1501\.8M47\.91±0\.0947\.91\\pm 0\.0922\.72±1\.3122\.72\\pm 1\.3156\.50±\\pm0\.11CIFAR\-10128k4512k57\.01±0\.4257\.01\\pm 0\.4254\.01±0\.1554\.01\\pm 0\.1557\.31±\\pm0\.47CIFAR\-10128k121\.536M56\.12±0\.2056\.12\\pm 0\.2054\.31±0\.2154\.31\\pm 0\.2158\.72±\\pm0\.58CIFAR\-10128k202\.560M55\.97±0\.0955\.97\\pm 0\.0953\.87±0\.3953\.87\\pm 0\.3959\.23±\\pm0\.27CIFAR\-10128k8010\.24M55\.66±0\.1755\.66\\pm 0\.1752\.05±0\.1152\.05\\pm 0\.1159\.61±\\pm0\.19CIFAR\-100128k4512k25\.77±0\.2525\.77\\pm 0\.2526\.43±0\.1926\.43\\pm 0\.1928\.84±\\pm0\.25CIFAR\-100128k121\.536M26\.65±0\.2726\.65\\pm 0\.2727\.00±0\.0827\.00\\pm 0\.0829\.46±\\pm0\.40CIFAR\-100128k202\.560M26\.31±0\.1426\.31\\pm 0\.1427\.04±0\.1627\.04\\pm 0\.1630\.29±\\pm0\.26CIFAR\-100128k8010\.24M26\.19±0\.2726\.19\\pm 0\.2725\.16±0\.2125\.16\\pm 0\.2130\.60±\\pm0\.06MNIST8k648k97\.29±0\.0697\.29\\pm 0\.0697\.54±0\.0397\.54\\pm 0\.0398\.22±\\pm0\.02MNIST8k100800k97\.12±0\.0697\.12\\pm 0\.0694\.31±0\.0894\.31\\pm 0\.0898\.33±\\pm0\.07

## 6Experiments

We evaluate IALGN from three complementary perspectives: \(i\) whether increasing depth improves predictive performance at fixed width, \(ii\) whether the observed gains arise from the proposed input\-anchored information\-access pattern rather than generic wiring flexibility, and \(iii\) whether the resulting architecture preserves longer effective computational paths that provide a mechanistic explanation for the progressively richer representations observed in Section[5](https://arxiv.org/html/2607.21633#S5)\. Additional sensitivity analyses on input binarization and the random\-kkanchor size are provided in the supplementary material\.

### Experimental Setup

We evaluate randomly wired LGNs \(RWLGNs\) and IALGNs on MNIST, CIFAR\-10, and CIFAR\-100\. Following prior LGN studies, real\-valued inputs are first converted into binary representations and then processed entirely through differentiable logic layers\. The outputs of the final layer are partitioned into class\-specific groups and aggregated using the standard GroupSum readout to produce classification logits\.

Our main analyses focus on CIFAR\-10 and CIFAR\-100, where the increased task difficulty makes the effect of depth more apparent\. MNIST is included as an additional benchmark to verify that the observed behavior is consistent across datasets\. Unless otherwise stated, each configuration is evaluated over five random seeds, and we report the mean and standard deviation across runs\. No data augmentation is used in the controlled comparisons\.

For the primary depth\-scaling experiment, the hidden width is fixed for each dataset while only the number of logic layers is varied\. Consequently, the total number of gates increases linearly with depth\. This experiment is designed to determine whether additional feedforward depth can be converted into improved predictive performance while preserving the per\-layer capacity, rather than to compare architectures under an identical total gate budget\.

To isolate architectural effects, both RWLGN and IALGN use the same logic operations, readout, optimization objective, skip\-biased initialization, and straight\-through estimator\. Their primary difference is therefore the wiring topology through which hidden and input features are supplied to each gate\. Raw\-gradient and random\-initialization variants are examined separately in Section[3](https://arxiv.org/html/2607.21633#S3)and in the supplementary material\.

We additionally include an LDLGN reimplementation trained under the same no\-augmentation protocol\. Because an official implementation is not publicly available, we treat this result as a controlled reference under our experimental setting rather than as an exact reproduction of the originally reported performance\.

### Depth Scaling

The primary objective of this work is to improve the scalability of Logic Gate Networks with respect to network depth while maintaining fixed per\-layer capacity\. We therefore compare IALGNs and randomly wired DLGNs under identical widths while varying the network depth\.

Table[1](https://arxiv.org/html/2607.21633#S5.T1)summarizes the classification accuracy on MNIST, CIFAR\-10, and CIFAR\-100\. Across all three benchmarks, RWLGNs fail to obtain consistent benefits from increasing depth and often degrade, with severe performance loss occurring in some deep configurations\. In contrast, IALGNs consistently benefit from additional depth, maintaining stable optimization and progressively improving classification accuracy\.

The improvement is particularly pronounced on more challenging datasets such as CIFAR\-10 and CIFAR\-100, where deeper IALGNs continue to gain accuracy while conventional DLGNs deteriorate\. These results demonstrate that the proposed input\-anchored topology effectively converts increased depth into additional computational capability rather than representation degradation, consistent with the theoretical analysis presented in Section[4](https://arxiv.org/html/2607.21633#S4)\. The following ablation investigates whether these gains arise from the proposed information\-access pattern itself rather than simply from increasing wiring flexibility\.

### Ablation: Input Anchors vs\. Hidden Wiring Relaxation

Table 2:Topology ablation on CIFAR\-10 atW=12W=12k andnb=16n\_\{b\}=16\. Hidden\-kkrelaxation increases wiring flexibility over hidden features, whereas input\-kkrelaxation preserves the input\-anchored topology\.Topology\(D=4\)Relaxed sourceAccuracyRWLGNnone50\.56±0\.5850\.56\\pm 0\.58LDLGN reimpl\.none50\.22±0\.1850\.22\\pm 0\.18IALGN \(kx=32,kh=1\{k\_\{x\}\}=32,\{k\_\{h\}\}=1\)input53\.59±0\.0453\.59\\pm 0\.04IALGN \(kx=32,kh=8\{k\_\{x\}\}=32,\{k\_\{h\}\}=8\)input, hidden53\.91±0\.2753\.91\\pm 0\.27IALGN \(kx=32,kh=16\{k\_\{x\}\}=32,\{k\_\{h\}\}=16\)input, hidden54\.11±\\pm0\.29IALGN \(kx=32,kh=32\{k\_\{x\}\}=32,\{k\_\{h\}\}=32\)input, hidden53\.73±0\.1153\.73\\pm 0\.11Topology\(D=50\)Relaxed sourceAccuracyRWLGNnone50\.16±0\.3350\.16\\pm 0\.33LDLGN reimpl\.none45\.40±0\.3845\.40\\pm 0\.38IALGN \(kx=32,kh=1\{k\_\{x\}\}=32,\{k\_\{h\}\}=1\)input56\.12±\\pm0\.05IALGN \(kx=32,kh=8\{k\_\{x\}\}=32,\{k\_\{h\}\}=8\)input, hidden55\.66±0\.1255\.66\\pm 0\.12IALGN \(kx=32,kh=16\{k\_\{x\}\}=32,\{k\_\{h\}\}=16\)input, hidden55\.23±0\.2355\.23\\pm 0\.23IALGN \(kx=32,kh=32\{k\_\{x\}\}=32,\{k\_\{h\}\}=32\)input, hidden55\.11±0\.1455\.11\\pm 0\.14We next ask whether the gain comes from input anchoring itself or simply from introducing more relaxed wiring choices\. To test this, we compare input\-anchor relaxation with hidden\-kkrelaxation, which also increases local wiring flexibility but relaxes hidden\-to\-hidden connections\. This comparison is deliberately stricter than varyingkkalone: both input\-kkand hidden\-kkintroduce local connection choices, but only input\-kkpreserves the fixed hidden spine\.

Table[2](https://arxiv.org/html/2607.21633#S6.T2)shows that hidden\-kkrelaxation provides at most a small benefit in shallow models, but degrades performance at larger depth\. In contrast, input\-kkrelaxation preserves the fixed spine path and improves anchor selection without disrupting the depth\-wise hidden pathway\. This supports the view that the key ingredient is not arbitrary wiring flexibility, but the input\-anchored inductive bias\.

### Effective Computational Path Depth

![Refer to caption](https://arxiv.org/html/2607.21633v1/Figures/exp4.png)Figure 5:Distribution of effective computational depth among final\-layer gates in 100\-layer networks\. Compared with RWLGNs, IALGNs consistently shift the distribution toward larger effective depths on both CIFAR\-10 and CIFAR\-100, indicating that input anchoring preserves longer computational paths throughout the network\.To better understand why input anchoring preserves deeper computational paths, consider how effective depth evolves during training\. In randomly wired DLGNs, repeated hidden\-to\-hidden compositions frequently collapse into skip\-like computations, preventing many final\-layer gates from accumulating long chains of non\-trivial Boolean operations\. Because every layer in IALGN instead combines the existing hidden computation with a newly accessible input variable, non\-trivial computations are more likely to survive across successive layers, resulting in substantially deeper effective computational paths\.

Figure 5 confirms this behavior empirically\. On both CIFAR\-10 and CIFAR\-100, IALGNs consistently produce effective\-depth distributions shifted toward larger values than RWLGNs, indicating that the proposed topology preserves longer computational paths throughout the network\. Combined with the linear\-probe analysis in Section[5](https://arxiv.org/html/2607.21633#S6.F5), these results suggest that input anchoring not only produces progressively more informative hidden representations but also allows those representations to emerge through substantially deeper computational pathways\.

## 7Conclusion

We showed that depth scalability in LGNs requires solving two distinct problems: optimization collapse and a topology\-induced limitation in progressive computation\. Even when skip\-biased initialization and straight\-through estimation stabilize deep randomly wired LGNs, additional layers do not reliably improve their representations or predictive performance\.

IALGN addresses the second limitation by combining a depth\-wise hidden spine with direct input anchors\. This topology induces a strict path\-wise depth hierarchy and, empirically, converts additional fixed\-width layers into improved accuracy on MNIST, CIFAR\-10, and CIFAR\-100\. Linear probes, topology ablations, and effective\-depth distributions further show that the gains are associated with progressively informative representations and longer computational paths rather than arbitrary wiring flexibility\.

Our study is limited to feedforward, flattened LGNs and does not establish superiority under every fixed total\-gate budget\. Future work may examine budget\-matched depth–width trade\-offs, structured anchor assignments, convolutional variants, and hardware\-aware implementations\. More broadly, our results suggest that realizing the value of depth in discrete neural architectures requires jointly designing optimization and information access\. We hope this perspective encourages future research on discrete neural architectures in which topology is designed not only for trainability but also for progressively expanding computation across depth\.

## References

- K\. Bhardwaj, G\. Li, and R\. Marculescu \(2019\)How does topology influence gradient propagation and model performance of deep networks with densenet\-type skip connections?\.arXiv\.External Links:1910\.00780Cited by:[§2](https://arxiv.org/html/2607.21633#S2.SSx2.p1.1)\.
- S\. Bührer, A\. Plesner, T\. Aczel, and R\. Wattenhofer \(2025\)Recurrent deep differentiable logic gate networks\.arXiv\.External Links:2508\.06097Cited by:[§2](https://arxiv.org/html/2607.21633#S2.SSx1.p2.1)\.
- K\. Fojcik, R\. Zioma, and J\. Armaitis \(2026\)LILogic net: compact logic gate networks with learnable connectivity for efficient hardware deployment\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\) Workshops,pp\. 3498–3507\.Cited by:[§2](https://arxiv.org/html/2607.21633#S2.SSx1.p2.1)\.
- J\. Hastad \(1986\)Almost optimal lower bounds for small depth circuits\.InProceedings of the Eighteenth Annual ACM Symposium on Theory of Computing,pp\. 6–20\.External Links:[Document](https://dx.doi.org/10.1145/12130.12132)Cited by:[§1](https://arxiv.org/html/2607.21633#S1.p2.1)\.
- Y\. Kim \(2023\)Deep stochastic logic gate networks\.IEEE Access11\(\),pp\. 122488–122501\.External Links:[Document](https://dx.doi.org/10.1109/ACCESS.2023.3328622)Cited by:[§1](https://arxiv.org/html/2607.21633#S1.p3.1),[§1](https://arxiv.org/html/2607.21633#S1.p4.1),[§2](https://arxiv.org/html/2607.21633#S2.SSx1.p2.1),[§3](https://arxiv.org/html/2607.21633#S3.p1.1)\.
- W\. Mommen, L\. Keuninckx, M\. Hartmann, W\. V\. Leekwijck, and P\. Wambacq \(2026\)Fully trainable deep differentiable logic gate networks and lookup table networks\.External Links:2607\.09399,[Link](https://arxiv.org/abs/2607.09399)Cited by:[§2](https://arxiv.org/html/2607.21633#S2.SSx1.p2.1)\.
- F\. Petersen, C\. Borgelt, H\. Kuehne, and O\. Deussen \(2022\)Deep differentiable logic gate networks\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 2006–2018\.Cited by:[§1](https://arxiv.org/html/2607.21633#S1.p3.1),[§2](https://arxiv.org/html/2607.21633#S2.SSx1.p1.1),[§3](https://arxiv.org/html/2607.21633#S3.p1.1),[§3](https://arxiv.org/html/2607.21633#S3.p2.1)\.
- F\. Petersen, H\. Kuehne, C\. Borgelt, J\. Welzel, and S\. Ermon \(2024\)Convolutional differentiable logic gate networks\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 121185–121203\.External Links:[Document](https://dx.doi.org/10.52202/079017-3851)Cited by:[§1](https://arxiv.org/html/2607.21633#S1.p4.1),[§2](https://arxiv.org/html/2607.21633#S2.SSx1.p2.1)\.
- L\. Rüttgers, T\. Aczel, A\. Plesner, and R\. Wattenhofer \(2025\)Light differentiable logic gate networks\.arXiv\.External Links:2510\.03250Cited by:[§1](https://arxiv.org/html/2607.21633#S1.p4.1),[§2](https://arxiv.org/html/2607.21633#S2.SSx1.p2.1),[§3](https://arxiv.org/html/2607.21633#S3.p1.1)\.
- A\. Veit, M\. Wilber, and S\. Belongie \(2016\)Residual networks behave like ensembles of relatively shallow networks\.arXiv\.External Links:1605\.06431Cited by:[§2](https://arxiv.org/html/2607.21633#S2.SSx2.p1.1)\.
- S\. Yousefi, A\. Plesner, T\. Aczel, and R\. Wattenhofer \(2025\)Mind the gap: removing the discretization gap in differentiable logic gate networks\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 121185–121203\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.48550/arXiv.2506.07500)Cited by:[§1](https://arxiv.org/html/2607.21633#S1.p4.1),[§2](https://arxiv.org/html/2607.21633#S2.SSx1.p2.1),[§3](https://arxiv.org/html/2607.21633#S3.p1.1)\.