Double-Scoring: Reliable Extraction of Strong Lottery Tickets

# Double-Scoring: Reliable Extraction of Strong Lottery Tickets
Source: [https://arxiv.org/html/2607.20555](https://arxiv.org/html/2607.20555)
Bryce A\. Christopherson Department of Mathematics & Statistics University of North Dakota Grand Forks, ND 58202\-8376 bryce\.christopherson@UND\.edu &Jack Baretz Department of Mathematics & Statistics University of North Dakota Grand Forks, ND 58202\-8376 jack\.baretz@UND\.edu Darian Colgrove Department of Mathematics & Statistics University of North Dakota Grand Forks, ND 58202\-8376 darian\.colgrove@UND\.edu &Salah Dandan Department of Mathematics & Statistics University of North Dakota Grand Forks, ND 58202\-8376 salah\.dandan@UND\.edu

###### Abstract

The lottery ticket hypothesis proposes that large random neural networks contain sparse subnetworks that can match the performance of dense models after comparable training\. A stronger version asserts that sufficiently overparameterized random networks contain subnetworks that are already accurate before any weight training\. Existing theory establishes that such strong lottery tickets exist, but reliable extraction remains difficult\. We revisitedge\-popup, a frozen\-weight score\-training method for extracting strong tickets, and identify layerwise sparsity selection as a central bottleneck\. We introducedouble\-scoring, an augmented score\-space parameterization that replaces a layerwise sparsity search with optimization over enlarged score tensors\. We prove that fixed\-density masking in an augmented score space preserves access to all original\-coordinate masks, and we show that the resulting method can be interpreted asedge\-popupon a zero\-augmented network\. In controlled experiments,double\-scoringsubstantially improves strong\-ticket extraction over fixed\-densityedge\-popupand pruning\-at\-initialization baselines, improves on the performance of rewound sparse\-training topologies, and exhibits markedly lower sensitivity to sparsity hyperparameters\. Ablations show that the gain is not merely due to additional trainable score parameters, but is tied to the augmented score\-space competition that induces the effective original sparsity\.

## 1Introduction

In 2019, Frankle and Carbin introduced the lottery ticket hypothesis\[[4](https://arxiv.org/html/2607.20555#bib.bib1)\], which posits that dense, randomly initialized neural networks contain sparse subnetworks \(“winning tickets”\) that, when trained in isolation, can match the performance of the original network\. Shortly thereafter, Ramanujan et al\.\[[15](https://arxiv.org/html/2607.20555#bib.bib2)\]proposed a stronger variant: sufficiently overparameterized random networks should contain subnetworks that already approximate a target network*without any training*\. These subnetworks are referred to as*strong lottery tickets*, whereas the subnetworks proposed in\[[4](https://arxiv.org/html/2607.20555#bib.bib1)\]are referred to asweak lottery tickets\.

This strong form was subsequently rigorously established in several works, precisely quantifying the necessary overparameterization\. Malach et al\.\[[9](https://arxiv.org/html/2607.20555#bib.bib3)\]proved existence under polynomial overparameterization, and later results\[[13](https://arxiv.org/html/2607.20555#bib.bib4),[14](https://arxiv.org/html/2607.20555#bib.bib16)\]significantly improved these bounds, showing that under appropriate weight distributions, only logarithmic growth in width is required\. These results collectively demonstrate that the existence of strong lottery tickets is no longer in doubt\. Thus, the central difficulty is not existence, but*extraction*\. That is, given a randomly initialized network known to contain a strong lottery ticket, how can one efficiently identify it?

A variety of pruning and masking strategies have been proposed for this purpose\. Some methods, including SNIP\[[8](https://arxiv.org/html/2607.20555#bib.bib6)\], GraSP\[[19](https://arxiv.org/html/2607.20555#bib.bib7)\], and SynFlow\[[18](https://arxiv.org/html/2607.20555#bib.bib8)\], identify sparse subnetworks at initialization using saliency criteria\. These methods are computationally attractive because they produce masks before full training\. Other sparse training methods such as Iterative Magnitude Pruning \(IMP\) use repeated train\-prune\-rewind cycles to identify sparse subnetworks that could train effectively from an early or original initialization\[[5](https://arxiv.org/html/2607.20555#bib.bib22)\]\. Likewise, SET\[[11](https://arxiv.org/html/2607.20555#bib.bib127)\], RigL\[[3](https://arxiv.org/html/2607.20555#bib.bib128)\], and movement pruning\[[16](https://arxiv.org/html/2607.20555#bib.bib129)\]learn sparse topologies while training weights\. These methods are not native strong\-ticket extraction algorithms, since their masks are discovered during weight optimization\. Nevertheless, they provide useful comparison points: after learning a topology, one can rewind the weights to the original initialization and test whether the topology itself defines a strong ticket\.

In this work, we revisit one of the earliest methods for strong ticket extraction: theedge\-popupalgorithm of Ramanujan et al\.\[[15](https://arxiv.org/html/2607.20555#bib.bib2)\], which freezes random weights and trains auxiliary scores whose top\-ranked entries determine the active mask\. We argue that the primary obstacle inedge\-popupis not its optimization dynamics, but rather its*parameterization of sparsity*\. The algorithm requires choosing layerwise sparsity levels in advance, yet existing theory provides no principled way to choose them\.

We show that this difficulty can be removed entirely\. First, we prove that for sufficiently sparse layers, the full class of masked subnetworks can be represented using a fixed mask density of12\\frac\{1\}\{2\}\. We then extend this idea to dense networks by doubling the number of trainable score parameters—that is, by enlarging the*score space*rather than the weight space\. This leads to a simple modification ofedge\-popup, which we calldouble\-scoring, that eliminates the need to tune layerwise sparsity parameters while preserving expressive power\. Empirically, this modification allows us to reliably extract strong lottery tickets using a procedure with essentially the same computational cost as standard training\. Moreover, the resulting subnetworks appear to be competitive with other weak lottery ticket extraction methods after training as well\.

##### Contributions\.

1. 1\.We identify layerwise sparsity selection as a practical bottleneck inedge\-popup\-style strong ticket extraction\.
2. 2\.We introducedouble\-scoring, an augmented score\-space parameterization that removes the need to tune a separate sparsity level for each layer\.
3. 3\.We prove that augmented score\-space masking preserves representational access to all original\-coordinate masks and is equivalent toedge\-popupon a zero\-augmented network under straight\-through gradients\.
4. 4\.In controlled FashionMNIST MLP experiments,double\-scoringsubstantially improves strong\-ticket extraction relative to fixed\-densityedge\-popup, random masks, SNIP, and GraSP, and remains competitive with rewound sparse\-training topologies\.
5. 5\.We provide stability and ablation studies showing that the gain is tied to augmented score\-space competition rather than simply to adding trainable score parameters\.

## 2Background and problem setup

We consider feed\-forward neural networks withℓ\\elllayers, widths\(n0,…,nℓ\)\(n\_\{0\},\\dots,n\_\{\\ell\}\), and activation functions\(σ1,…,σℓ−1\)\(\\sigma\_\{1\},\\dots,\\sigma\_\{\\ell\-1\}\)\. Such a network is a functionfW:ℝn0→ℝnℓf\_\{W\}:\\mathbb\{R\}^\{n\_\{0\}\}\\to\\mathbb\{R\}^\{n\_\{\\ell\}\}of the form

fW​\(x\)=Wℓ​σℓ−1​\(⋯​σ1​\(W1​x\+b1\)​⋯\)\+bℓ,f\_\{W\}\(x\)=W\_\{\\ell\}\\sigma\_\{\\ell\-1\}\\big\(\\cdots\\sigma\_\{1\}\(W\_\{1\}x\+b\_\{1\}\)\\cdots\\big\)\+b\_\{\\ell\},whereWi∈ℝni×ni−1W\_\{i\}\\in\\mathbb\{R\}^\{n\_\{i\}\\times n\_\{i\-1\}\}andbi∈ℝnib\_\{i\}\\in\\mathbb\{R\}^\{n\_\{i\}\}for eachii\.

The lottery ticket hypothesis\[[4](https://arxiv.org/html/2607.20555#bib.bib1)\]asserts that, for anyδ∈\(0,1\)\\delta\\in\(0,1\), there existsNNsuch that, with probability at leastδ\\delta, a randomly initialized feed\-forward network withmin⁡\{ni:1≤i≤ℓ−1\}≥N\\min\\\{n\_\{i\}:1\\leq i\\leq\\ell\-1\\\}\\geq Ncontains a maskHHsuch that the subnetworkfW⊙Hf\_\{W\\odot H\}, whereHHremains fixed, can achieve accuracy comparable to that of the original network after each undergoes comparable weight training\. The strong lottery ticket hypothesis\[[15](https://arxiv.org/html/2607.20555#bib.bib2)\]is effectively the same, but dispenses with the subsequent training requirement and instead asserts the maskHHsatisfies‖fW⊙H−g‖K,∞<ϵ\\\|f\_\{W\\odot H\}\-g\\\|\_\{K,\\infty\}<\\epsilonfor any chosen continuous functionggon a compact subsetKKof the domain\. Although a strong ticket is already performant at initialization, whether such a subnetwork also remains trainable as a weak ticket is an empirical question we examine in[Section˜6\.2](https://arxiv.org/html/2607.20555#S6.SS2)and again in[Section˜M\.1](https://arxiv.org/html/2607.20555#A13.SS1)\.

In the same paper in which Ramanujan et al\. initially posed the strong lottery ticket hypothesis, they also introducededge\-popup, a frozen\-weight score\-training method\. For a layer with fixed weightsWtW\_\{t\}, score tensorStS\_\{t\}, and densityktk\_\{t\}, defineHt=TopKMask⁡\(St;kt\)H\_\{t\}=\\operatorname\{TopKMask\}\(S\_\{t\};k\_\{t\}\)to retain the⌊kt​dt⌋\\lfloor k\_\{t\}d\_\{t\}\\rfloorentries of largest\|St\|\|S\_\{t\}\|, wheredtd\_\{t\}is the number of weights\. The forward pass usesWt⊙HtW\_\{t\}\\odot H\_\{t\}, while the backward pass uses a straight\-through estimator for the hard mask; gradients through the magnitude parameterization use the derivative of\|St\|\|S\_\{t\}\|away from zero\. Thus,edge\-popupoptimizes masks while keeping weights fixed\.

The difficulty in usingedge\-popupto extract strong lottery tickets from a network is that the algorithm assumes the correct choice of layerwise densitiesk1,…,kℓk\_\{1\},\\ldots,k\_\{\\ell\}are already known or somehow otherwise obtainable\. This is the bottleneck that preventsedge\-popupfrom extracting strong lottery tickets\.

## 3The sparsity selection bottleneck

Existence theorems for strong lottery tickets imply only that some good choice of layerwise densitiesk1,…,kℓk\_\{1\},\\ldots,k\_\{\\ell\}exists \(i\.e\. the densities of the masks for a strong ticket itself\); they do not identify that choice\. In practice, different layerwise density patterns can produce dramatically different performance, and there is no robust rule that predicts the correct tuple in advance\. Worse still, the correct choices provided by existence theorems are only viable for score initializations sufficiently near the desired target\. In many cases, it appears unlikely foranychoice of densities to produce a strong lottery ticket withedge\-popupfrom an unfortunate score initialization, since density selection and score initialization interact nontrivially, creating sensitivities not only to global sparsity but to the full layerwise density vector and the score initialization\. Since the correct vector is unknown, extracting a true strong ticket generally requires an expensive grid search overkk\-vectors and often some degree of luck\. This sensitivity is illustrated in a toy sine\-regression experiment in Appendix[H](https://arxiv.org/html/2607.20555#A8), where exhaustive sweeps over layerwise densities produce highly nonuniform loss landscapes\.

One can make this bottleneck explicit even at the level of counting\. Theiith weight matrix hasni−1​nin\_\{i\-1\}n\_\{i\}entries, and hence there are onlyni−1​ni\+1n\_\{i\-1\}n\_\{i\}\+1possible density values of the form0,1ni−1​ni,2ni−1​ni,…,ni−1​ni−1ni−1​ni,10,\\frac\{1\}\{n\_\{i\-1\}n\_\{i\}\},\\frac\{2\}\{n\_\{i\-1\}n\_\{i\}\},\\dots,\\frac\{n\_\{i\-1\}n\_\{i\}\-1\}\{n\_\{i\-1\}n\_\{i\}\},1\. Thus, if one wished to determine with certainty which layerwise density tuple is optimal, then, in principle, one would have to consider all combinations of these values across layers\. Without bias terms, the total number of possible layerwise density tuples is∏i=1ℓ\(ni−1​ni\+1\)\\prod\_\{i=1\}^\{\\ell\}\(n\_\{i\-1\}n\_\{i\}\+1\)\. Some of these tuples are obviously redundant—for instance, ifki=0k\_\{i\}=0, then the later valueski\+1,…,kℓk\_\{i\+1\},\\dots,k\_\{\\ell\}become irrelevant—but the search space still grows extremely rapidly\. For example, even a three\-layer network with weight matrices of shapes10×2010\\times 20,20×3020\\times 30, and30×530\\times 5\(only950950weights in total\) already has over1818million possible layerwise density tuples\.

## 4Double\-scoring method

Thedouble\-scoringmethod \([Algorithm˜1](https://arxiv.org/html/2607.20555#alg1)\) is a modification toedge\-popupthat functions by enlarging the*score space*of the model, allowing a dense layer to be treated as though it were embedded inside a half\-sparse augmented layer, without ever modifying the underlying weights \(the motivation for doing this is described along with the summary of our theoretical results in[Section˜5](https://arxiv.org/html/2607.20555#S5)\)\. The idea is to introduce two auxiliary score tensorsStS\_\{t\},TtT\_\{t\}of the same shape as each weight tensorWtW\_\{t\}, and to apply the masking operation to the concatenation of the two score tensorsS^t=\(St,Tt\)\\widehat\{S\}\_\{t\}=\(S\_\{t\},T\_\{t\}\)at fixed density12\\frac\{1\}\{2\}to produce a double\-width maskH^t=TopKMask​\(S^t;0\.5\)\\widehat\{H\}\_\{t\}=\\textrm\{TopKMask\}\(\\hat\{S\}\_\{t\};0\.5\), then yield the final mask by restricting the result back to the original coordinates; i\.e\.Ht=H^t\|origH\_\{t\}=\\widehat\{H\}\_\{t\}\|\_\{\\textrm\{orig\}\}\. Thus, one obtains the expressive flexibility of a half\-sparse augmented system without introducing any additional weight parameters\.

Algorithm 1double\-scoring1:Frozen randomly initialized weights

\{Wt,bt\}t=1L\\\{W\_\{t\},b\_\{t\}\\\}\_\{t=1\}^\{L\}, training data, loss function

ℒ\\mathcal\{L\}, learning rate

η\\eta
2:for

t=1,…,Lt=1,\\dots,Ldo

3:Initialize two score tensors

St,TtS\_\{t\},T\_\{t\}and bias score tensors

ft,gtf\_\{t\},g\_\{t\}with the same shapes as

WtW\_\{t\}and

btb\_\{t\}
4:foreach training iterationdo

5:for

t=1,…,Lt=1,\\dots,Ldo

6:

S^t←\(St,Tt\),f^t←\(ft,gt\)\\widehat\{S\}\_\{t\}\\leftarrow\(S\_\{t\},T\_\{t\}\),\\enskip\\widehat\{f\}\_\{t\}\\leftarrow\(f\_\{t\},g\_\{t\}\)
7:

H^t←TopKMask⁡\(S^t;1/2\),h^t←TopKMask⁡\(f^t;1/2\)\\widehat\{H\}\_\{t\}\\leftarrow\\operatorname\{TopKMask\}\(\\widehat\{S\}\_\{t\};1/2\),\\enskip\\widehat\{h\}\_\{t\}\\leftarrow\\operatorname\{TopKMask\}\(\\widehat\{f\}\_\{t\};1/2\)
8:

Ht←H^t\|orig,ht←h^t\|origH\_\{t\}\\leftarrow\\widehat\{H\}\_\{t\}\|\_\{\\mathrm\{orig\}\},\\enskip h\_\{t\}\\leftarrow\\widehat\{h\}\_\{t\}\|\_\{\\mathrm\{orig\}\}
9:

W~t←Wt⊙Ht,b~t←bt⊙ht\\widetilde\{W\}\_\{t\}\\leftarrow W\_\{t\}\\odot H\_\{t\},\\enskip\\widetilde\{b\}\_\{t\}\\leftarrow b\_\{t\}\\odot h\_\{t\}
10:Compute the network output using

\{\(W~t,b~t\)\}t=1L\\\{\(\\widetilde\{W\}\_\{t\},\\widetilde\{b\}\_\{t\}\)\\\}\_\{t=1\}^\{L\}, Compute loss

ℒ\\mathcal\{L\}
11:for

t=1,…,Lt=1,\\dots,Ldo

12:Update

St,Tt,ft,gtS\_\{t\},T\_\{t\},f\_\{t\},g\_\{t\}by gradient descent with straight\-through gradients

13:returnFinal masks

\{Ht,ht\}t=1L\\\{H\_\{t\},h\_\{t\}\\\}\_\{t=1\}^\{L\}

This construction fixes the density globally at12\\frac\{1\}\{2\}while still allowing the effective density on the original weights to be learned implicitly\. As we will show in[Section˜5](https://arxiv.org/html/2607.20555#S5), the resulting search space is expressive enough to represent all subnetworks obtainable via explicit layerwise density selection\. In this sense, the combinatorial search over density parameters is replaced by a continuous optimization over an enlarged score space, thereby eliminating the need to choose the brittle layerwise density hyperparameter during optimization without any detriment to the masked model’s expressive power\. We emphasize: these results are representational; they remove the expressive obstruction created by fixing density, but they do not by themselves guarantee that straight\-through score optimization will find the desired mask\.

### 4\.1Computational complexity

The proposeddouble\-scoringalgorithm increases the number of score parameters by a constant factor \(specifically, a factor of two per layer\), but does not alter the size of the weight tensors or the structure of the forward and backward passes through the network\. In particular, the dominant computational cost remains the evaluation of the network and its gradients with respect to the masked weights\. The additional overhead arises from computingTopKMask\\operatorname\{TopKMask\}on the augmented score tensors and updating the auxiliary scores\. Both operations scale linearly \(or log\-linearly, depending on the implementation of the top\-kkoperation\) in the number of parameters, and therefore introduce only a constant\-factor increase in runtime\.

Consequently, the overall asymptotic time complexity of thedouble\-scoringprocedure matches that of classicaledge\-popupand standard network training while eliminating the need for repeated runs over different sparsity configurations\. In practice, this leads to substantial computational savings relative to approaches that rely on exhaustive or heuristic search over layerwise density parameters, or methods that involve partial training and rewinding\.

## 5Theory summary

The mechanism ofdouble\-scoringrelies on an observation that is elementary but useful\. If a weight is already equal to zero, then changing the corresponding mask entry has no effect on the represented subnetwork\. Consequently, on a sufficiently sparse layer, one may alter mask entries on zero\-weight locations without changing the resulting masked tensor\. On a sufficiently sparse tensor, the apparent freedom in the density parameter inedge\-popupis largely illusory: one may trade a search over densities for a suitable choice of scored coordinates\.

###### Proposition 5\.1\.

LetW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}and suppose at least half of the entries ofWWare zero\. Letd=m​nd=mnand letr=⌊d2⌋r=\\lfloor\\frac\{d\}\{2\}\\rfloor\. Then, for every binary maskM∈\{0,1\}m×nM\\in\\\{0,1\\\}^\{m\\times n\}, there exists a binary maskM∗∈\{0,1\}m×nM^\{\\ast\}\\in\\\{0,1\\\}^\{m\\times n\}with exactlyrrones such that

W⊙M=W⊙M∗\.W\\odot M=W\\odot M^\{\\ast\}\.

###### Corollary 5\.2\.

LetNNbe a feed\-forward network, and suppose each layer ofNNcontains at least half zero weights\. Then every masked subnetwork ofNNcan be represented by choosing, in each layertt, a mask with exactly⌊dt2⌋\\lfloor\\frac\{d\_\{t\}\}\{2\}\\rfloorones, wheredtd\_\{t\}is the number of weights in layertt\.

Applied layerwise,[Corollary˜5\.2](https://arxiv.org/html/2607.20555#S5.Thmtheorem2)says that once each layer is at least half sparse, the entire search space of masked subnetworks can be represented using a single fixed density near12\\frac\{1\}\{2\}in every layer\. This does*not*mean thatedge\-popupatk=12k=\\frac\{1\}\{2\}will automatically converge to the best possible mask; optimization issues such as local minima may still remain, just as they do in ordinary training\. What it does show, however, is that the principal*expressive*obstruction disappears: in the half\-sparse regime, fixing the density at12\\frac\{1\}\{2\}is already rich enough to represent every masked subnetwork\.

If half\-sparse layers admit a fixed\-density search, then one can attempt to embed a dense layer into a larger layer whose additional entries are permanently zero\. Runningedge\-popupat fixed density1/21/2on this enlarged layer would then be expressive enough by[Corollary˜5\.2](https://arxiv.org/html/2607.20555#S5.Thmtheorem2)\. Naively enlarging the weight tensors themselves would be wasteful\. However, the additional zero entries do not require weights—they only require*scores*\. This leads to a simple but crucial observation: one can enlarge the*score space*without enlarging the weight space\.

###### Proposition 5\.3\.

Letw∈ℝdw\\in\\mathbb\{R\}^\{d\}and define the augmented vectorw^=\(w,0\)∈ℝ2​d\\widehat\{w\}=\(w,0\)\\in\\mathbb\{R\}^\{2d\}by adjoiningddzero coordinates\. Then for every binary maskm∈\{0,1\}dm\\in\\\{0,1\\\}^\{d\}there exists a binary maskm^∈\{0,1\}2​d\\widehat\{m\}\\in\\\{0,1\\\}^\{2d\}with exactlyddones such that the restriction ofw^⊙m^\\widehat\{w\}\\odot\\widehat\{m\}to the firstddcoordinates equalsw⊙mw\\odot m\.

That is,double\-scoringis not merely representationally equivalent toedge\-popupon an augmented half\-sparse system; the optimization dynamics agree exactly as well \(if this is not obvious,[Appendix˜H](https://arxiv.org/html/2607.20555#A8)contains formal verification of this fact\)\. These results show thatdouble\-scoringdoes not merely add parameters; it changes the parameterization so that fixed\-density score optimization can represent every effective sparsity pattern\.

## 6Experiments

In this section, we verify the performance ofdouble\-scoringexperimentally\. Note that the theoretical construction fixes the augmented score density at12\\frac\{1\}\{2\}, which shows that no layerwise density tuple is needed for representability\. In experiments, we also consider a targetable augmented variant in which a single global augmented\-space density is supplied to probe high\-sparsity regimes and an iterated variant which aggregates the half\-density masks over multiple runs\. We focus on the one\-shot augmented variant\. The iterated variant can reach different sparsity regimes but incurs a multiplicative runtime cost proportional to the number of rounds\. In both cases, the method removes the need to tune a separate density for each layer; the effective original\-coordinate sparsity is induced by competition with auxiliary score coordinates and is reported explicitly\. Throughout, it should be kept in mind thatdouble\-scoringshould be interpreted as an effective\-sparsity method: the requested augmented\-space density determines a competition rule, while the original\-coordinate sparsity is induced by the learned score geometry\. Therefore, the relevant quantities are both accuracy and achieved sparsity\.

Unless otherwise stated, experiments use a bias\-free ReLU MLP with three width\-256 hidden layers on FashionMNIST\. For each of three seeds, all methods share the same random initialization:50005000extraction/training examples,50005000validation examples, and the full test set\. Strong\-ticket accuracy is measured by extracting a mask, rewinding weights to the shared initialization and evaluating the masked network without further weight training\. Sparse\-training methods are included as rewound\-topology baselines\. As an additional architecture sanity check,[Appendix˜G](https://arxiv.org/html/2607.20555#A7)repeats the strong\-ticket comparison on CIFAR\-10 using a VGG\-style ConvNet without BatchNorm and with baselines matched to the achieved sparsity of DoubleScore\-Augmented\.

### 6\.1Strong ticket extraction

We first evaluate whether each method can extract a useful subnetwork while the weights remain frozen at their random initialization\. For each seed, we initialize a common linear MLP, allow each method to produce a binary mask, then rewind the weights to the shared initialization before evaluating the masked network\. This ensures that the reported accuracy measures strong\-ticket extraction rather than sparse training performance\.

Table 1:High\-sparsity strong\-ticket extraction results\. Entries report test accuracy of the extracted untrained subnetwork, with achieved sparsity shown separately\. Sparse\-training methods are evaluated as rewound\-topology baselines: their masks are learned during sparse training but evaluated after rewinding weights to the shared initialization\.[Table˜1](https://arxiv.org/html/2607.20555#S6.T1)summarizes the high\-sparsity regime on FashionMNIST\. These comparisons are not intended to claim same\-sparsity dominance in every row\. The double\-score variants induce an effective sparsity on the original coordinates, which can differ from the nominal requested sparsity\. We therefore report achieved sparsity throughout\. In[Appendix˜G](https://arxiv.org/html/2607.20555#A7), we include a matched\-sparsity CIFAR\-10 ConvNet sanity check in which baselines are run at the achieved sparsity induced by DoubleScore\-Augmented\. Fixed\-densityedge\-popupis a strong direct baseline, achieving77\.7%77\.7\\%accuracy at90\.0%90\.0\\%achieved sparsity, but its performance drops sharply at the more extreme95%95\\%requested sparsity, where it obtains60\.6%60\.6\\%\. In contrast, DoubleScore\-Augmented obtains82\.7%82\.7\\%accuracy in the90%90\\%requested\-sparsity setting and78\.0%78\.0\\%in the95%95\\%setting, achieving sparsities of81\.3%81\.3\\%and90%90\\%respectively\. DoubleScore\-Iterated is similarly strong, achieving82\.1%82\.1\\%and77\.1%77\.1\\%accuracy, respectively, at achieved sparsities of86\.6%86\.6\\%and90\.7%90\.7\\%\.

The comparison to pruning\-at\-initialization methods is especially stark\. SNIP and GraSP remain near chance in this benchmark, indicating that simple saliency\-at\-initialization criteria do not identify high\-performing strong tickets in this setting\. Rewound sparse\-training baselines provide a stronger comparison: movement pruning, in particular, discovers topologies that remain useful after rewinding\. Nevertheless, the double\-score variants outperform these rewound topologies in the high\-sparsity regime, while extracting masks with frozen weights\.

As in all experiments, we report achieved sparsity in addition to requested sparsity\. The augmented score\-space method does not enforce the requested original\-coordinate density exactly; rather, the effective sparsity emerges through competition with auxiliary score coordinates\. This is part of the mechanism studied in[Section˜6\.3](https://arxiv.org/html/2607.20555#S6.SS3)and[Section˜6\.4](https://arxiv.org/html/2607.20555#S6.SS4)\. The key conclusion from the baseline comparison is thatdouble\-scoringproduces high\-accuracy strong tickets in regimes where fixed\-densityedge\-popupand standard pruning\-at\-initialization baselines degrade substantially\. These results suggest that the main obstacle is not the existence of strong tickets, but their extraction: once the sparsity\-selection bottleneck is relaxed through augmented score\-space optimization, high\-performing strong tickets can be found reliably\.

### 6\.2Sparse training and weak tickets

As shown in the previous section,double\-scoringdominates in the strong\-ticket regime\. Here, we show that it also remains competitive after training\. For each method, we first extract a binary mask, rewind the weights to the shared random initialization, and then train only the surviving weights while keeping the mask fixed\. Thus this experiment separates the quality of the extracted topology at initialization from its trainability under standard sparse training\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_weak_ticket_main_dumbbell.png)Figure 1:Weak\-ticket training after strong\-ticket extraction\. Each method first extracts a binary mask, after which weights are rewound to the shared random initialization and trained with the mask fixed\. Points show test accuracy before and after masked weight training\. DoubleScore masks begin with high accuracy and remain competitive after training, whereas several baselines only become accurate after weight optimization\. Sparse\-training methods such as Movement are included as rewound\-topology baselines\.The results indicate that double\-score masks are not only strong at initialization, but also remain highly trainable\. At requested90%90\\%sparsity, DoubleScore\-Augmented begins at82\.5%82\.5\\%test accuracy and reaches83\.4%83\.4\\%after masked training, while DoubleScore\-Iterated begins at82\.1%82\.1\\%and reaches83\.5%83\.5\\%\. In contrast, pruning\-at\-initialization baselines such as SNIP and GraSP begin near chance but train to above83%83\\%\. Thus these baselines can produce trainable sparse networks, but they do not identify strong tickets\.

The comparison with sparse\-training methods further clarifies the distinction\. Movement pruning achieves the highest final trained accuracy, reaching84\.4%84\.4\\%at requested90%90\\%sparsity and84\.6%84\.6\\%at requested95%95\\%sparsity\. However, movement pruning learns its topology while training weights, whereas double\-score methods extract their masks with frozen weights\. The role ofdouble\-scoringis therefore not to replace all sparse\-training procedures, but to produce subnetworks that are already accurate at initialization and remain competitive after subsequent training\.

### 6\.3Hyperparameter stability

A central motivation fordouble\-scoringis thatedge\-popuprequires the user to specify a sparsity level, and in multilayer networks this choice is effectively a layerwise hyperparameter\. To test whether this tuning problem is merely cosmetic or whether it materially affects extraction quality, we performed a sparsity\-sensitivity experiment on the controlled FashionMNIST MLP benchmark\. We fixed the requested sparsity at90%90\\%, trainededge\-popupover a collection of scalar keep densities and randomly sampled layerwise keep\-density tuples, and evaluated each extracted mask after rewinding the weights to the same random initialization\. We compare this sweep to fixed target\-densityedge\-popup, random masks, and the two double\-score variants\. For theedge\-popupsweep, the reported range is taken over all tested scalar and layerwise keep\-density configurations across seeds\. We also report two validation\-oracleedge\-popupresults, obtained by selecting theedge\-popupconfiguration with the best validation accuracy for each seed both with and without a sparsity constraint\. The constrained oracle is the relevant same\-regime comparison\. The unconstrained oracle is included only to show that validation tuning overkkcan drift to a substantially denser regime\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_stability_main_combined.png)Figure 2:Hyperparameter stability at requested90%90\\%sparsity on FashionMNIST with a linear MLP\. Left:edge\-popupexhibits large variance across scalar and layerwise sparsity choices, while the double\-score variants are stable across seeds\. Right: the constrained oracle provides a same\-regime high\-sparsity reference, while the unconstrained validation\-oracleedge\-popupconfiguration achieves high accuracy only by selecting substantially denser masks, whereas the double\-score variants remain in the high\-sparsity regime\. All masks are evaluated after rewinding weights to the shared random initialization\.The results show thatedge\-popupis highly sensitive to the sparsity choice\. Across the sweep,edge\-popuptest accuracy ranges from18\.6%18\.6\\%to84\.4%84\.4\\%, with a standard deviation of19\.619\.6percentage points\. The fixed target\-densityedge\-popupbaseline is stable but substantially lower, achieving73\.5%73\.5\\%accuracy at90\.0%90\.0\\%achieved sparsity\. In contrast, DoubleScore\-Augmented achieves80\.1%80\.1\\%accuracy with standard deviation below0\.20\.2percentage points, and DoubleScore\-Iterated achieves78\.6%78\.6\\%accuracy with similarly low variance\. Thus the double\-score variants provide high\-performing strong\-ticket extraction without requiring a sparsity sweep\.

To separate hyperparameter sensitivity from the trivial capacity advantage of denser subnetworks, we report two validation\-selectededge\-popupreferences\. The first is a constrained oracle, which selects the best validation configuration amongedge\-popupmasks whose achieved sparsity lies in the high\-sparsity band85%−95%85\\%\-95\\%\. This separates high\-sparsity tuning from the capacity advantage of denser subnetworks and is the relevant same\-regime reference\. The unconstrained oracle is included only as a diagnostic: it can achieve higher accuracy by selecting substantially denser masks, and therefore is not a same\-sparsity comparison\. Thus the advantage ofdouble\-scoringis not that it dominates every possibleedge\-popupconfiguration, but that it avoids a fragile layerwise sparsity search while remaining in a high\-sparsity regime\.

### 6\.4Ablations

The previous experiments show that the augmented score\-space parameterization improves extraction and stabilizes sparsity selection\. We next isolate which components of the augmented score\-space parameterization are responsible for the observed gains\. A natural critique of thedouble\-scoringalgorithm is that the gains it produces are the result of doubling the number of trainable parameters\. Via a series of ablations, we will show that the gain is not merely from more trainable score parameters; the critical change is the minimal doubled score\-space parameterization\. We compare ordinary single\-scoreedge\-popup, signed\-score variants without the absolute\-value parameterization, augmented score spaces with different auxiliary widths, a frozen\-auxiliary variant, and a projected\-final variant\. In all cases, weights are frozen during mask extraction and the resulting mask is evaluated after rewinding to the shared random initialization\.

Table 2:Ablation study at requested90%90\\%sparsity\. All variants are evaluated as strong\-ticket extractors after rewinding weights to the shared initialization\. Score parameters are reported as trainable/total multiples relative to the original weight count\.The ablations show that the augmented score space is the critical ingredient for inducing high\-performing effective sparsity patterns\. At requested90%90\\%sparsity, ordinary single\-scoreedge\-popupwithabs​S\\mathrm\{abs\}\{S\}\-ranking achieves73\.9%73\.9\\%accuracy at90\.0%90\.0\\%achieved sparsity\. The canonical augmented variant, Aug\-x1\-Abs, improves to80\.3%80\.3\\%accuracy, while achieving82\.9%82\.9\\%sparsity\. Thus the augmented method substantially improves the extracted strong ticket, but it does so by allowing the effective original\-coordinate sparsity to emerge from competition with auxiliary score coordinates rather than by enforcing the requested density exactly\. SingleScore\-Abs is theedge\-popupbaseline using magnitude\-based score ranking\. The deviation in the result here from that of[Table˜1](https://arxiv.org/html/2607.20555#S6.T1)highlights the sensitivity of classicaledge\-popupto score initializations, as demonstrated in[Figure˜2](https://arxiv.org/html/2607.20555#S6.F2)\.

The frozen\-auxiliary ablation is especially informative\. Freezing the auxiliary scores barely changes performance: Aug\-x1\-Abs obtains80\.3%80\.3\\%accuracy, while Aug\-x1\-Abs\-FrozenAux obtains79\.9%79\.9\\%\. This indicates that the gain is not simply due to adding more trainable score parameters\. Instead, the auxiliary coordinates act as a learned or fixed competitive reservoir that changes the thresholding dynamics in the original coordinates\. The absolute\-value score parameterization is also consistently beneficial\. Removing it reduces performance for both single\-score and augmented variants\. At requested90%90\\%sparsity, SingleScore\-Abs outperforms SingleScore\-NoAbs, and Aug\-x1\-Abs outperforms Aug\-x1\-NoAbs\. This supports the use of magnitude\-based score ranking in the main method\.

Finally, the projected\-final variant clarifies the role of effective sparsity\. When the augmented method is trained normally but then projected back to exactly the requested original\-coordinate sparsity, performance drops substantially\. This suggests that the advantage ofdouble\-scoringis not simply that it learns better original\-coordinate saliency scores for a fixed target density\. Rather, the augmented score space improves extraction by jointly shaping the mask and its effective sparsity\. Consequently, all main comparisons report achieved sparsity alongside accuracy\.

## 7Limitations

Several limitations remain\. First, the theoretical results are representational and do not guarantee that straight\-through score optimization will find globally optimal masks\. Second,double\-scoringinduces an effective original\-coordinate sparsity rather than enforcing an exact sparsity budget\. This is useful when the goal is high\-performing sparse extraction, but applications requiring exact sparsity may require calibration of the nominal augmented density or a projection procedure, and our simple projected\-final ablation shows that naive projection can substantially reduce accuracy\. Third, the main experiments are controlled FashionMNIST MLP benchmarks\. We include a no\-BatchNorm CIFAR\-10 ConvNet sanity check, but large\-scale architectures, transformers, and modern high\-resolution benchmarks remain open due to compute constraints\. Finally, the iterated variant incurs substantially higher runtime than the one\-shot augmented variant, so we treat it as an exploratory extension rather than the main practical method\.

## 8Conclusion

We introduceddouble\-scoring, an augmented score\-space parameterization foredge\-popup\-style strong lottery ticket extraction\. The method replaces a brittle layerwise sparsity search with competition in an enlarged score space, while leaving the weight tensors unchanged\. Theoretical results show that the augmented parameterization preserves representational access to original\-coordinate masks, and controlled experiments show improved strong\-ticket extraction, strong hyperparameter stability, and robust trainability after mask extraction\. These results suggest that the practical barrier to strong lottery ticket extraction is not merely the existence of suitable subnetworks, but the parameterization used to search for them\.

## Acknowledgments and Disclosure of Funding

Use unnumbered first level headings for the acknowledgments\. All acknowledgments go at the end of the paper before the list of references\. Moreover, you are required to declare funding \(financial activities supporting the submitted work\) and competing interests \(related financial activities outside the submitted work\)\. More information about this disclosure can be found at:[https://neurips\.cc/Conferences/2026/PaperInformation/FundingDisclosure](https://neurips.cc/Conferences/2026/PaperInformation/FundingDisclosure)\.

## References

- \[1\]\(2024\)A survey on mixture of experts\.arXiv preprint arXiv:2407\.06204\.Cited by:[§M\.5](https://arxiv.org/html/2607.20555#A13.SS5.p1.1)\.
- \[2\]Z\. Chen, Y\. Deng, Y\. Wu, Q\. Gu, and Y\. Li\(2022\)Towards understanding the mixture\-of\-experts layer in deep learning\.Advances in neural information processing systems35,pp\. 23049–23062\.Cited by:[§M\.5](https://arxiv.org/html/2607.20555#A13.SS5.p1.1)\.
- \[3\]U\. Evci, T\. Gale, J\. Menick, P\. S\. Castro, and E\. Elsen\(2020\)Rigging the lottery: making all tickets winners\.InInternational conference on machine learning,pp\. 2943–2952\.Cited by:[§1](https://arxiv.org/html/2607.20555#S1.p3.1)\.
- \[4\]J\. Frankle and M\. Carbin\(2018\)The lottery ticket hypothesis: finding sparse, trainable neural networks\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.1803.03635),[Link](https://arxiv.org/abs/1803.03635)Cited by:[§1](https://arxiv.org/html/2607.20555#S1.p1.1),[§2](https://arxiv.org/html/2607.20555#S2.p2.11)\.
- \[5\]J\. Frankle, G\. K\. Dziugaite, D\. Roy, and M\. Carbin\(2020\-13–18 Jul\)Linear mode connectivity and the lottery ticket hypothesis\.InProceedings of the 37th International Conference on Machine Learning,H\. D\. III and A\. Singh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.119,pp\. 3259–3269\.External Links:[Link](https://proceedings.mlr.press/v119/frankle20a.html)Cited by:[§1](https://arxiv.org/html/2607.20555#S1.p3.1)\.
- \[6\]I\. C\. Gormley and S\. Frühwirth\-Schnatter\(2019\)Mixture of experts models\.InHandbook of mixture analysis,pp\. 271–307\.Cited by:[§M\.5](https://arxiv.org/html/2607.20555#A13.SS5.p1.1)\.
- \[7\]A\. Hosna, E\. Merry, J\. Gyalmo, Z\. Alom, Z\. Aung, and M\. A\. Azim\(2022\)Transfer learning: a friendly introduction\.Journal of Big Data9\(1\),pp\. 102\.Cited by:[§M\.5](https://arxiv.org/html/2607.20555#A13.SS5.p1.1.2)\.
- \[8\]N\. Lee, T\. Ajanthan, and P\. H\. Torr\(2018\)Snip: single\-shot network pruning based on connection sensitivity\.arXiv preprint arXiv:1810\.02340\.Cited by:[§1](https://arxiv.org/html/2607.20555#S1.p3.1)\.
- \[9\]E\. Malach, G\. Yehudai, S\. Shalev\-Shwartz, and O\. Shamir\(2020\)Proving the lottery ticket hypothesis: pruning is all you need\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.2002.00585),[Link](https://arxiv.org/abs/2002.00585)Cited by:[§1](https://arxiv.org/html/2607.20555#S1.p2.1)\.
- \[10\]S\. Masoudnia and R\. Ebrahimpour\(2014\)Mixture of experts: a literature survey\.Artificial Intelligence Review42,pp\. 275–293\.Cited by:[§M\.5](https://arxiv.org/html/2607.20555#A13.SS5.p1.1)\.
- \[11\]D\. C\. Mocanu, E\. Mocanu, P\. Stone, P\. H\. Nguyen, M\. Gibescu, and A\. Liotta\(2018\)Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science\.Nature communications9\(1\),pp\. 2383\.Cited by:[§1](https://arxiv.org/html/2607.20555#S1.p3.1)\.
- \[12\]H\. D\. Nguyen and F\. Chamroukhi\(2018\)Practical and theoretical aspects of mixture\-of\-experts modeling: an overview\.Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery8\(4\),pp\. e1246\.Cited by:[§M\.5](https://arxiv.org/html/2607.20555#A13.SS5.p1.1)\.
- \[13\]L\. Orseau, M\. Hutter, and O\. Rivasplata\(2020\)Logarithmic pruning is all you need\.Advances in Neural Information Processing Systems33,pp\. 2925–2934\.Cited by:[§1](https://arxiv.org/html/2607.20555#S1.p2.1)\.
- \[14\]A\. Pensia, S\. Rajput, A\. Nagle, H\. Vishwakarma, and D\. Papailiopoulos\(2020\)Optimal lottery tickets via subset sum: logarithmic over\-parameterization is sufficient\.Advances in neural information processing systems33,pp\. 2599–2610\.Cited by:[Figure 14](https://arxiv.org/html/2607.20555#A9.F14),[Appendix I](https://arxiv.org/html/2607.20555#A9.p1.1),[§1](https://arxiv.org/html/2607.20555#S1.p2.1)\.
- \[15\]V\. Ramanujan, M\. Wortsman, A\. Kembhavi, A\. Farhadi, and M\. Rastegari\(2019\)What’s hidden in a randomly weighted neural network?\.arXiv\.External Links:[Document](https://dx.doi.org/10.48550/ARXIV.1911.13299),[Link](https://arxiv.org/abs/1911.13299)Cited by:[Appendix K](https://arxiv.org/html/2607.20555#A11.p1.1),[Theorem H\.1](https://arxiv.org/html/2607.20555#A8.Thmtheorem1),[Remark H\.10](https://arxiv.org/html/2607.20555#A8.Thmtheorem10.p1.1),[§1](https://arxiv.org/html/2607.20555#S1.p1.1),[§1](https://arxiv.org/html/2607.20555#S1.p4.1),[§2](https://arxiv.org/html/2607.20555#S2.p2.11)\.
- \[16\]V\. Sanh, T\. Wolf, and A\. Rush\(2020\)Movement pruning: adaptive sparsity by fine\-tuning\.Advances in neural information processing systems33,pp\. 20378–20389\.Cited by:[§1](https://arxiv.org/html/2607.20555#S1.p3.1)\.
- \[17\]R\. Sutton\(2019\)The bitter lesson, 2019\.URL http://www\.incompleteideas\.net/IncIdeas/BitterLesson\.html\.Cited by:[§M\.10](https://arxiv.org/html/2607.20555#A13.SS10.p5.1)\.
- \[18\]H\. Tanaka, D\. Kunin, D\. L\. Yamins, and S\. Ganguli\(2020\)Pruning neural networks without any data by iteratively conserving synaptic flow\.Advances in Neural Information Processing Systems33,pp\. 6377–6389\.Cited by:[§1](https://arxiv.org/html/2607.20555#S1.p3.1)\.
- \[19\]C\. Wang, G\. Zhang, and R\. Grosse\(2020\)Picking winning tickets before training by preserving gradient flow\.arXiv preprint arXiv:2002\.07376\.Cited by:[§1](https://arxiv.org/html/2607.20555#S1.p3.1)\.
- \[20\]K\. Weiss, T\. M\. Khoshgoftaar, and D\. Wang\(2016\)A survey of transfer learning\.Journal of Big data3,pp\. 1–40\.Cited by:[§M\.5](https://arxiv.org/html/2607.20555#A13.SS5.p1.1.2)\.
- \[21\]J\. Yang, R\. Shi, and B\. Ni\(2021\)Medmnist classification decathlon: a lightweight automl benchmark for medical image analysis\.In2021 IEEE 18th International Symposium on Biomedical Imaging \(ISBI\),pp\. 191–195\.Cited by:[Figure 23](https://arxiv.org/html/2607.20555#A13.F23),[§M\.5](https://arxiv.org/html/2607.20555#A13.SS5.p2.1)\.
- \[22\]J\. Yang, R\. Shi, D\. Wei, Z\. Liu, L\. Zhao, B\. Ke, H\. Pfister, and B\. Ni\(2023\)Medmnist v2\-a large\-scale lightweight benchmark for 2d and 3d biomedical image classification\.Scientific Data10\(1\),pp\. 41\.Cited by:[Figure 23](https://arxiv.org/html/2607.20555#A13.F23),[§M\.5](https://arxiv.org/html/2607.20555#A13.SS5.p2.1)\.
- \[23\]M\. Yu, D\. Wang, Q\. Shan, C\. J\. Reed, and A\. Wan\(2024\)The super weight in large language models\.arXiv preprint arXiv:2411\.07191\.Cited by:[§M\.10](https://arxiv.org/html/2607.20555#A13.SS10.p3.1)\.
- \[24\]S\. E\. Yuksel, J\. N\. Wilson, and P\. D\. Gader\(2012\)Twenty years of mixture of experts\.IEEE transactions on neural networks and learning systems23\(8\),pp\. 1177–1193\.Cited by:[§M\.5](https://arxiv.org/html/2607.20555#A13.SS5.p1.1)\.
- \[25\]F\. Zhuang, Z\. Qi, K\. Duan, D\. Xi, Y\. Zhu, H\. Zhu, H\. Xiong, and Q\. He\(2020\)A comprehensive survey on transfer learning\.Proceedings of the IEEE109\(1\),pp\. 43–76\.Cited by:[§M\.5](https://arxiv.org/html/2607.20555#A13.SS5.p1.1.2)\.

## Appendix AReproducibility and code

All code necessary to reproduce the experiments is included in the anonymous supplementary material\.

## Appendix BExperimental details

All experiments in the main paper, with the exception of the toy experiment illustrating thekk\-selection bottleneck in[Figure˜10](https://arxiv.org/html/2607.20555#A8.F10), are conducted on FashionMNIST using a fully connected ReLU network\. The input images are flattened to dimension784784, and the network has three hidden layers of width256256, followed by a1010\-class output layer\. Unless otherwise stated, all linear layers are bias\-free\. We normalize FashionMNIST using mean0\.28600\.2860and standard deviation0\.35300\.3530\. For each random seed, we randomly shuffle the FashionMNIST training set and use5,0005\{,\}000examples for mask extraction or sparse training and5,0005\{,\}000disjoint examples for validation\. All reported test accuracies are computed on the full FashionMNIST test set\.

Weights are initialized once per seed and shared across methods\. We use the signed Kaiming uniform initialization

Wi​j∼Unif⁡\[−1k​gainfan​\_​in,1k​gainfan​\_​in\],W\_\{ij\}\\sim\\operatorname\{Unif\}\\\!\\left\[\-\\frac\{1\}\{\\sqrt\{k\}\}\\frac\{\\operatorname\{gain\}\}\{\\sqrt\{\\operatorname\{fan\\\_in\}\}\},\\frac\{1\}\{\\sqrt\{k\}\}\\frac\{\\operatorname\{gain\}\}\{\\sqrt\{\\operatorname\{fan\\\_in\}\}\}\\right\],withk=1/2k=1/2and ReLU gain\. After each method extracts a mask, we reload the original initialization before evaluating the masked network\. Thus the strong\-ticket experiments measure the quality of the extracted mask rather than the quality of trained weights\.

All methods use Adam unless otherwise stated, withβ=\(0\.9,0\.999\)\\beta=\(0\.9,0\.999\),ϵ=10−8\\epsilon=10^\{\-8\}, and weight decay0\. Score variables are trained with learning rate10−210^\{\-2\}, while weight\-training baselines use learning rate10−310^\{\-3\}\. The batch size is512512\. We set Python, NumPy, and PyTorch random seeds for each run and disable cuDNN benchmarking\.

##### Datasets and assets used\.

Table 3:External datasets and software assets used in the experiments\.
##### Mask densities and achieved sparsity\.

For a requested sparsityss, the corresponding keep density isk=1−sk=1\-s\. For methods that directly impose a density, masks are constructed by retaining the topkk\-fraction of scores or saliencies\. For double\-score methods, the effective sparsity on the original weight coordinates is induced by the augmented score\-space competition and need not equal the requested sparsity\. We therefore report the achieved sparsity

1−\#​\{active mask entries\}\#​\{total mask entries\}1\-\\frac\{\\\#\\\{\\text\{active mask entries\}\\\}\}\{\\\#\\\{\\text\{total mask entries\}\\\}\}for every method\.

##### Edge\-popup\.

For fixededge\-popup, weights are frozen and each weight tensor has an associated trainable score tensor of the same shape\. At each forward pass, we form a binary mask by retaining the entries with largest score magnitudes, i\.e\.

Ht=TopKMask⁡\(\|St\|;k\),H\_\{t\}=\\operatorname\{TopKMask\}\(\|S\_\{t\}\|;k\),and train only the scores using a straight\-through estimator for the hard top\-kkoperation\. The final mask is extracted from the learned score magnitudes\.

##### DoubleScore\-Augmented\.

For each layer with weight matrixWt∈ℝm×nW\_\{t\}\\in\\mathbb\{R\}^\{m\\times n\}, the augmented variant introduces a score matrix

S^t∈ℝm×\(n\+naux\),naux=n\\widehat\{S\}\_\{t\}\\in\\mathbb\{R\}^\{m\\times\(n\+n\_\{\\mathrm\{aux\}\}\)\},\\qquad n\_\{\\mathrm\{aux\}\}=nin the main experiments\. A hard top\-kkmask is computed in the augmented score space using\|S^t\|\|\\widehat\{S\}\_\{t\}\|, and the mask is then restricted to the originalnncoordinates\. The auxiliary score coordinates therefore compete with original coordinates for inclusion, allowing the effective original\-coordinate sparsity to emerge from optimization\. No additional weight parameters are introduced\.

##### DoubleScore\-Iterated\.

The iterated variant maintains a cumulative mask\. At each round, it applies a doubled score\-space mask only to currently active weights, multiplies the result into the cumulative mask, and stops once the achieved sparsity is within0\.020\.02of the requested sparsity or after at most88rounds\. The per\-round keep density is chosen as the ratio between the target keep density and the current keep density, capped at1/21/2\.

##### Pruning\-at\-initialization baselines\.

SNIP is implemented by introducing differentiable mask parameters initialized to one and ranking weights by the magnitude of the gradient of the loss with respect to these mask parameters\. We compute SNIP saliencies using55minibatches and then apply a global top\-kkthreshold\. GraSP is implemented using the standard Hessian\-gradient saliency structure\. We compute two independent collections of55minibatches, use temperature200200, form the Hessian\-gradient proxy, and rank weights by the signed score−θ​\(H​g\)θ\-\\theta\(Hg\)\_\{\\theta\}, followed by global top\-kkthresholding\.

##### Sparse\-training baselines\.

IMP, SET, RigL, and Movement pruning are included as rewound\-topology baselines\. These methods are allowed to train weights while discovering a sparse topology, but the resulting mask is evaluated only after rewinding the weights to the shared random initialization\. Thus their reported strong\-ticket accuracy measures whether the learned topology itself transfers back to initialization\.

IMP uses55pruning rounds and2020training epochs per round\. At each round, the model is reset to the original initialization, trained under the current mask, and globally magnitude\-pruned to the next density level\.

SET and RigL initialize a random global sparse mask at the requested keep density and train weights under the mask\. Every100100optimization steps,30%30\\%of active weights are pruned by magnitude and the same number of inactive weights are regrown\. SET regrows weights randomly\. RigL regrows using dense gradients computed on the current minibatch\.

Movement pruning trains weights and signed movement scores jointly\. It uses separate Adam parameter groups for weights and scores, with learning rates10−310^\{\-3\}and10−210^\{\-2\}, respectively\. The keep density follows a cubic schedule from dense to the requested density, with10%10\\%warmup and10%10\\%cooldown\. Movement scores are initialized to zero with10−610^\{\-6\}Gaussian noise for tie\-breaking, and the final mask is obtained from the signed movement scores without taking absolute values\.

##### Strong\-ticket extraction benchmark\.

For the main strong\-ticket comparison, we use seeds0,1,20,1,2, requested sparsities50%50\\%,90%90\\%, and95%95\\%, and train mask\-extraction methods for100100epochs\. The main paper reports the high\-sparsity90%90\\%and95%95\\%results, with the50%50\\%results included in the appendix\.

##### Weak\-ticket training benchmark\.

For the weak\-ticket experiment, each method first extracts a mask using the same extraction protocol\. We then rewind the weights to the shared initialization and train the surviving weights for4040epochs while keeping the mask fixed\. We report both the initial accuracy before masked weight training and the final accuracy after masked weight training\. A dense baseline is trained once per seed under the same optimizer settings\.

##### Hyperparameter\-stability benchmark\.

For the hyperparameter\-stability experiment, we fix requested sparsity at90%90\\%\. We compare fixed target\-densityedge\-popup, random masks, the double\-score variants, and a sweep ofedge\-popupconfigurations\. Theedge\-popupsweep includes scalar keep densities

0\.01,0\.02,0\.05,0\.075,0\.10,0\.15,0\.20,0\.30,0\.40,0\.50,0\.65,0\.800\.01,0\.02,0\.05,0\.075,0\.10,0\.15,0\.20,0\.30,0\.40,0\.50,0\.65,0\.80and88randomly sampled layerwise keep\-density tuples per seed drawn from the same pool\. We also report a validation\-oracleedge\-popupresult, which selects theedge\-popupconfiguration with highest validation accuracy for each seed\. The oracle is included only as an unconstrained tuning reference; it is not a same\-sparsity baseline\.

##### Ablation benchmark\.

For the ablation study, we use requested sparsities90%90\\%and95%95\\%with seeds0,1,20,1,2\. We compare ordinary single\-scoreedge\-popupwith and without magnitude ranking, augmented score spaces with auxiliary\-width multipliers1,2,1,2,and44, a frozen\-auxiliary variant, and a projected\-final variant\. In the frozen\-auxiliary variant, the auxiliary score coordinates are randomly initialized and held fixed, while the original score coordinates are trained\. In the projected\-final variant, the augmented method is trained normally, but the final mask is projected onto the original coordinates at the exact requested sparsity\. This tests whether the augmented method’s gains come from improved original\-coordinate saliency at fixed sparsity or from the effective sparsity induced by auxiliary score competition\.

##### Reported statistics\.

All tables report mean±\\pmstandard deviation over the stated random seeds\. We report test accuracy, validation accuracy when used for selection, achieved sparsity, and wall\-clock runtime\. Validation accuracy is used only for the validation\-oracleedge\-popupconfiguration in the hyperparameter\-stability experiment; all main comparisons are based on test accuracy after mask extraction and rewinding\.

##### Compute resources\.

All reported experiments were run on Google Colab GPU instances\. Runs used a single GPU when available, with PyTorch automatically falling back to CPU otherwise\. The main FashionMNIST MLP experiments require modest compute and can be reproduced on a single consumer GPU\. The CIFAR\-10 ConvNet sanity\-check experiments are more expensive but were also run on a single Colab GPU\. Each results table reports average wall\-clock time per method when relevant\. Because Colab assigns GPU types dynamically, the exact GPU model varied across runs; wall\-clock times are therefore reported as approximate reproduction guidance rather than hardware\-normalized benchmarks\. The total compute for the reported experiments was dominated by score\-training methods that require full extraction runs, especiallyedge\-popupsweeps, dynamic sparse\-training baselines, and iterateddouble\-scoringvariants\.

The reported experiments represent the final controlled runs used in the paper\. During development, additional exploratory and failed runs were performed to debug implementations, choose stable protocols, and test variants not included in the final paper\. These preliminary runs required additional compute but are not used to support the paper’s main claims\.

## Appendix CAdditional baseline experiments

This appendix contains additional experimental results omitted from the main paper for space\. Unless otherwise stated, all experiments use the setup described in Appendix[B](https://arxiv.org/html/2607.20555#A2): a bias\-free ReLU MLP with three hidden layers of width256256on FashionMNIST, shared random initializations across methods,5,0005\{,\}000extraction/training examples,5,0005\{,\}000validation examples, and the full test set\. All reported values are mean±\\pmstandard deviation over the stated random seeds\.

### C\.1Full strong\-ticket extraction results

The main paper reports the high\-sparsity90%90\\%and95%95\\%strong\-ticket extraction results\.[Table˜4](https://arxiv.org/html/2607.20555#A3.T4)gives the full strong\-ticket table, including the50%50\\%requested\-sparsity setting and wall\-clock time\. Each method first extracts a mask, after which the weights are rewound to the shared random initialization before evaluation\. Sparse\-training methods are therefore evaluated as rewound\-topology baselines\.

Table 4:Full strong\-ticket extraction results\. Each entry reports test accuracy of the extracted untrained subnetwork, achieved sparsity, and average runtime\. All masks are evaluated after rewinding weights to the shared random initialization\.![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_strong_ticket_linear.png)Figure 3:Strong\-ticket extraction accuracy across requested sparsity levels\. DoubleScore\-Augmented and DoubleScore\-Iterated remain accurate in the high\-sparsity regime where fixed\-densityedge\-popupand pruning\-at\-initialization baselines degrade\.![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_achieved_sparsity_linear.png)Figure 4:Achieved sparsity of extracted masks\. Methods that impose a fixed global density match the requested sparsity by construction\. The double\-score variants induce an effective original\-coordinate sparsity through competition with auxiliary score coordinates, so their achieved sparsity can differ from the requested value\.These full results reinforce the main\-paper conclusion\. At50%50\\%requested sparsity, fixed\-densityedge\-popupand DoubleScore\-Augmented perform similarly\. The separation becomes more pronounced in the high\-sparsity regime: at90%90\\%and95%95\\%requested sparsity, the double\-score variants retain substantially higher strong\-ticket accuracy than fixed\-densityedge\-popup, random masks, SNIP, GraSP, SET, RigL, and IMP\. Movement pruning is the strongest rewound sparse\-training baseline, but the double\-score methods obtain higher strong\-ticket accuracy in the high\-sparsity settings while extracting masks with frozen weights\.

## Appendix DWeak ticket training details

The weak\-ticket experiment asks whether the masks extracted in the strong\-ticket setting remain trainable after ordinary masked weight training\. For each method, we first extract a mask, rewind the weights to the shared initialization, and then train only the surviving weights while keeping the mask fixed\. The main paper reports a dumbbell plot at90%90\\%and95%95\\%requested sparsity\. Here we provide the full final\-accuracy, achieved\-sparsity, and training\-gain tables\.

Table 5:Final test accuracy after masked weight training\. Dense denotes the fully dense baseline trained under the same optimizer settings\. Several masks that are poor strong tickets, such as random, SNIP, and GraSP masks, become trainable weak tickets after weight optimization\.Table 6:Achieved sparsity for the weak\-ticket experiment\. The double\-score variants do not necessarily match the requested original\-coordinate sparsity, since their effective sparsity is induced by auxiliary score competition\.Table 7:Training gain in the weak\-ticket experiment, measured as final test accuracy after masked training minus initial test accuracy immediately after mask extraction and rewinding\. Large gains for random, SNIP, and GraSP masks show that these masks can be trainable weak tickets despite not being strong tickets\. The double\-score variants exhibit small gains because they already begin with high accuracy at initialization\.![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_weak_ticket_training_curves_90_clean.png)Figure 5:Masked weight\-training curves at requested90%90\\%sparsity\. Double\-score masks start near their final trained accuracy, while several baselines begin near chance and only become competitive after weight optimization\.The full weak\-ticket results clarify the distinction between strong\-ticket extraction and weak\-ticket trainability\. Random, SNIP, and GraSP masks are generally poor strong tickets, but after masked weight training they can achieve final test accuracies comparable to other sparse networks\. In contrast, the double\-score masks begin with high accuracy and therefore require little additional optimization to reach their final trained performance\. Movement pruning achieves the highest final trained accuracy in several settings, consistent with its role as a sparse\-training method rather than a frozen\-weight strong\-ticket extractor\.

## Appendix EHyperparameter stability details

The main text reports the full hyperparameter\-stability comparison, including both constrained and unconstrained validation\-selectededge\-popupreferences\. Here we show the underlying scalar keep\-density sweep used as part of that experiment\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_stability_edgepopup_k_sweep.png)Figure 6:Scalar keep\-density sweep foredge\-popupat requested90%90\\%sparsity\. Performance varies substantially with the density parameter, illustrating that fixed\-densityedge\-popuprequires nontrivial sparsity selection even in this controlled benchmark\. The full hyperparameter\-stability comparison, including constrained and unconstrained validation\-selected references, appears in[Figure˜2](https://arxiv.org/html/2607.20555#S6.F2)\.
## Appendix FAdditional ablation experiments

The main paper reports the ablation study at requested90%90\\%sparsity\.[Table˜8](https://arxiv.org/html/2607.20555#A6.T8)gives the full ablation table at both90%90\\%and95%95\\%requested sparsity\. These results separate the effects of magnitude\-based score ranking, auxiliary score competition, auxiliary width, frozen auxiliary coordinates, and final projection to exact original\-coordinate sparsity\.

Table 8:Full ablation results at requested90%90\\%and95%95\\%sparsity\. SingleScore\-Abs is the fixed\-densityedge\-popup\-style baseline\. The augmented variants improve strong\-ticket accuracy, while the projected\-final variant shows that the gain is tied to the effective sparsity induced by auxiliary score competition rather than simply to better original\-coordinate saliencies at a fixed density\.![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_ablation_main_scatter.png)Figure 7:Ablation accuracy versus achieved sparsity\. The augmented variants obtain high accuracy while inducing effective sparsities below the requested original\-coordinate sparsity\. The projected\-final variant enforces the requested sparsity exactly and loses much of the augmented method’s advantage\.![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_ablation_capacity_clean.png)Figure 8:Auxiliary score\-capacity ablation\. Increasing auxiliary width improves accuracy but also decreases achieved sparsity, indicating that auxiliary score capacity primarily shifts the effective sparsity/accuracy tradeoff rather than simply improving fixed\-sparsity saliency learning\.The ablations support three conclusions\. First, magnitude\-based score ranking is beneficial: removing the absolute\-value parameterization lowers performance for both single\-score and augmented variants\. Second, the auxiliary coordinates are not useful merely because they add trainable parameters\. The frozen\-auxiliary variant performs nearly identically to the fully trainable Aug\-x1\-Abs variant, consistent with the interpretation that the auxiliary coordinates act as a competitive reservoir in the top\-kkselection\. Third, the projected\-final variant shows that the augmented method’s advantage is tied to induced effective sparsity\. When the final mask is forced back to exact requested original\-coordinate sparsity, performance drops substantially\.

## Appendix GMatched\-sparsity CIFAR\-10 ConvNet sanity check

To test whether the main phenomenon is specific to FashionMNIST MLPs, we ran an additional CIFAR\-10 experiment using a VGG\-style convolutional network without BatchNorm\. Removing BatchNorm avoids the ambiguity that high initial accuracy might come from adapted normalization statistics rather than from the extracted mask\. We use the same5,000/5,0005\{,\}000/5\{,\}000train\-validation split and the full CIFAR\-10 test set\.

##### Strong ticket extraction\.

For each seed and nominal target sparsity, we first extract a DoubleScore\-Augmented mask and record its achieved sparsity\. All baselines are then run at that matched requested sparsity using the same initialization, data split, and optimizer budget\. Thus this experiment controls for the effective\-sparsity difference present in the main FashionMNIST results\.

Table 9:Matched\-sparsity CIFAR\-10 ConvNet strong\-ticket extraction\. For each seed and nominal target sparsity, DoubleScore\-Augmented is run first; all other methods are run at the achieved sparsity of the DoubleScore\-Augmented mask\. The architecture is a VGG\-style ConvNet without BatchNorm\. All masks are evaluated after rewinding weights to the shared random initialization\.The results show that DoubleScore\-Augmented remains a nontrivial strong\-ticket extractor in this harder no\-BatchNorm convolutional setting\. At nominal90%90\\%sparsity, it obtains46\.7±2\.2%46\.7\\pm 2\.2\\%test accuracy at83\.8%83\.8\\%achieved sparsity, compared with46\.2±3\.2%46\.2\\pm 3\.2\\%for matched fixededge\-popup,41\.7±2\.4%41\.7\\pm 2\.4\\%for IMP, and36\.2±4\.2%36\.2\\pm 4\.2\\%for movement pruning\. At nominal95%95\\%sparsity, DoubleScore obtains41\.8±0\.5%41\.8\\pm 0\.5\\%at90\.9%90\.9\\%achieved sparsity, comparable to matched fixededge\-popupat42\.8±1\.0%42\.8\\pm 1\.0\\%and above the other evaluated baselines\. Thus the qualitative strong\-ticket behavior is not limited to the FashionMNIST MLP setting, although in this ConvNet experiment DoubleScore is best interpreted as competitive with matchededge\-popuprather than uniformly superior to it\.

The CIFAR\-10 ConvNet result is intentionally conservative: matchededge\-popupis given the sparsity level induced by DoubleScore\-Augmented\. Thus this experiment should not be read as showing that DoubleScore uniformly dominatesedge\-popup\. Rather, it shows that DoubleScore remains competitive with a retrospectively well\-chosenedge\-popupdensity and substantially above several other baselines in a no\-BatchNorm convolutional architecture\. In practice, these sparsity values are not known in advance for a given initialization, architecture, and training protocol\. As shown,edge\-popupperformance varies substantially away from these density choices\. The CIFAR\-10 result therefore supports the same qualitative conclusion as the main experiments: DoubleScore does not necessarily dominate a well\-chosenedge\-popupdensity, but it provides a mechanism for inducing a useful effective sparsity without performing an explicit sparsity sweep\.

##### Hyperparameter stability\.

The matched CIFAR\-10 stability experiment tests whether the sparsity\-selection issue persists in the no\-BatchNorm ConvNet setting\. For each seed, DoubleScore\-Augmented is first run at nominal90%90\\%sparsity, and the resulting achieved sparsity is used as the matched reference sparsity for theedge\-popupcomparison\. This makes the comparison deliberately conservative:edge\-popupis evaluated at, or near, the sparsity level induced by DoubleScore itself\.

Table 10:Matched CIFAR\-10 ConvNet hyperparameter\-stability experiment at nominal90%90\\%DoubleScore sparsity\. DoubleScore\-Augmented is run first, achieving approximately84%84\\%sparsity\. The fixed matchededge\-popuprow uses this achieved sparsity directly\. The matched\-band oracle selects the best validationedge\-popupconfiguration whose achieved sparsity lies near the DoubleScore\-induced sparsity, while the unconstrained oracle selects over the fulledge\-popupsweep\. The unconstrained oracle attains higher accuracy by selecting much denser masks and is therefore not a same\-regime comparison\.![Refer to caption](https://arxiv.org/html/2607.20555v1/fig_conv_stability_matched_combined.png)Figure 9:Matched CIFAR\-10 ConvNet hyperparameter stability\. Left:edge\-popupexhibits substantial variation across scalar and layerwise keep\-density choices, while DoubleScore\-Augmented produces a nontrivial strong ticket without an explicit density sweep\. Right: accuracy versus achieved sparsity\. Fixed matchededge\-popupand the matched\-band oracle are evaluated near the DoubleScore\-induced sparsity, whereas the unconstrained oracle achieves higher accuracy by selecting substantially denser masks\.Under this retrospective matched comparison, fixededge\-popupreaches49\.1±1\.7%49\.1\\pm 1\.7\\%, slightly above DoubleScore\-Augmented at45\.7±1\.2%45\.7\\pm 1\.2\\%, and the matched\-band validation oracle reaches50\.1±0\.6%50\.1\\pm 0\.6\\%\. Thus, in this ConvNet setting, DoubleScore should not be interpreted as uniformly superior to a well\-chosenedge\-popupdensity\. However, the sweep also shows why the matched comparison is retrospective:edge\-popupperformance varies from near chance to57\.9%57\.9\\%across the tested configurations, and the unconstrained oracle reaches its best performance by selecting much denser masks, with only40\.0±8\.7%40\.0\\pm 8\.7\\%achieved sparsity\. These results support the main qualitative claim: DoubleScore induces a useful effective sparsity without performing an explicit sparsity sweep, whileedge\-popupcan be highly competitive when given a good density after the fact\.

##### Ablations\.

The matched CIFAR\-10 ConvNet ablations are more nuanced than the FashionMNIST MLP ablations and shows that the advantage of DoubleScore over single\-scoreedge\-popupis architecture\- and regime\-dependent\. At nominal90%90\\%, matched SingleScore\-Abs slightly outperforms Aug\-x1\-Abs, while at nominal95%95\\%Aug\-x1\-Abs is slightly higher\. We therefore do not claim uniform dominance over a retrospectively matchededge\-popupdensity\. The more robust conclusion is that DoubleScore induces useful effective sparsities without an explicit density search and remains competitive withedge\-popupeven whenedge\-popupis given the induced sparsity retrospectively\. Canonical Aug\-x1\-Abs remains competitive with matched single\-scoreedge\-popup, but does not uniformly dominate it: at nominal90%90\\%sparsity, SingleScore\-Abs slightly outperforms Aug\-x1\-Abs at the same achieved sparsity, while at nominal95%95\\%Aug\-x1\-Abs is slightly higher\. The main mechanistic conclusion comes from the projected\-final variant\. When augmented training is followed by projection back to the original\-coordinate density, performance collapses, reaching only21\.0±2\.2%21\.0\\pm 2\.2\\%at the90%90\\%nominal setting and15\.5±5\.4%15\.5\\pm 5\.4\\%at the95%95\\%nominal setting\. Thus the useful effect of augmentation is not simply improved original\-coordinate saliency at a fixed density; it is the effective sparsity and layerwise mask distribution induced by competition with auxiliary score coordinates\.

Table 11:Matched CIFAR\-10 ConvNet ablations\. For each seed and nominal target, canonical Aug\-x1\-Abs is run first; all other variants are run at the achieved sparsity induced by the canonical variant\. The projected\-final variant trains with augmented score competition but extracts its final mask by projecting back to the original score coordinates at the matched original\-coordinate sparsity\. Its large performance drop shows that the augmented method’s advantage is not merely better original\-coordinate saliency learning, but the effective sparsity pattern induced by auxiliary score competition\. Auxiliary\-capacity variants induce different achieved sparsities and should be interpreted as tracing a sparsity–accuracy tradeoff rather than as same\-sparsity comparisons\.The auxiliary\-capacity variants further support this interpretation\. Increasing auxiliary capacity changes the induced achieved sparsity and can improve accuracy, but the resulting variants are denser on the original coordinates \(this phenomenon is again noted in[Section˜M\.1](https://arxiv.org/html/2607.20555#A13.SS1)\)\. These rows should therefore be read as tracing an induced sparsity/accuracy tradeoff, not as same\-sparsity comparisons\. The frozen\-auxiliary variant remains strong, indicating that trainability of the auxiliary coordinates is not essential for them to function as a competitive reservoir\.

## Appendix HAdditional motivation for theoretical results and proofs

In the same paper in which Ramanujan et al\. posed the strong lottery ticket hypothesis, they also introduced an algorithm callededge\-popupto provide empirical evidence for it\. Pseudocode for the procedure is included below in[Algorithm˜2](https://arxiv.org/html/2607.20555#alg2)\. Recall that scores are ranked by magnitude; i\.e\.TopKMask⁡\(S;k\)\\operatorname\{TopKMask\}\(S;k\)denotes the mask retaining the largest entries of\|S\|\|S\|\.

Algorithm 2Classicaledge\-popup1:Frozen randomly initialized weights

\{Wt,bt\}t=1L\\\{W\_\{t\},b\_\{t\}\\\}\_\{t=1\}^\{L\}, layerwise densities

\{kt\}t=1L\\\{k\_\{t\}\\\}\_\{t=1\}^\{L\}, training data, loss function

ℒ\\mathcal\{L\}, learning rate

η\\eta
2:Initialize score tensors

\{St,ct\}t=1L\\\{S\_\{t\},c\_\{t\}\\\}\_\{t=1\}^\{L\}, where

StS\_\{t\}has the same shape as

WtW\_\{t\}and

ctc\_\{t\}has the same shape as

btb\_\{t\}
3:foreach training iterationdo

4:for

t=1,…,Lt=1,\\dots,Ldo

5:

Ht←TopKMask⁡\(St;kt\)H\_\{t\}\\leftarrow\\operatorname\{TopKMask\}\(S\_\{t\};k\_\{t\}\)
6:

ht←TopKMask⁡\(ct;kt\)h\_\{t\}\\leftarrow\\operatorname\{TopKMask\}\(c\_\{t\};k\_\{t\}\)
7:

W~t←Wt⊙Ht\\widetilde\{W\}\_\{t\}\\leftarrow W\_\{t\}\\odot H\_\{t\}
8:

b~t←bt⊙ht\\widetilde\{b\}\_\{t\}\\leftarrow b\_\{t\}\\odot h\_\{t\}
9:Compute the network output using

\{\(W~t,b~t\)\}t=1L\\\{\(\\widetilde\{W\}\_\{t\},\\widetilde\{b\}\_\{t\}\)\\\}\_\{t=1\}^\{L\}
10:Compute loss

ℒ\\mathcal\{L\}
11:Update the scores by gradient descent with straight\-through gradients:

St←St−η​∇Stℒ,ct←ct−η​∇ctℒS\_\{t\}\\leftarrow S\_\{t\}\-\\eta\\nabla\_\{S\_\{t\}\}\\mathcal\{L\},\\qquad c\_\{t\}\\leftarrow c\_\{t\}\-\\eta\\nabla\_\{c\_\{t\}\}\\mathcal\{L\}
12:returnFinal masks

\{Ht,ht\}t=1L\\\{H\_\{t\},h\_\{t\}\\\}\_\{t=1\}^\{L\}

Ramanujan et al\. proved a useful local monotonicity statement for this procedure: under appropriate smoothness assumptions, if one swaps in a nonzero number of edges in a single layer while keeping the remainder of the network fixed, then the minibatch loss decreases\.

###### Theorem H\.1\(Ramanujan et al\.\[[15](https://arxiv.org/html/2607.20555#bib.bib2)\]\)\.

When a nonzero number of edges are swapped in one layer by a straight\-through gradients update in[Algorithm˜2](https://arxiv.org/html/2607.20555#alg2)and the rest of the network remains fixed, the minibatch loss decreases, provided the loss is sufficiently smooth\.

This makesedge\-popupquite attractive in principle\. The difficulty, as discussed in the main paper, is choosing layerwise densitiesk1,…,kℓk\_\{1\},\\ldots,k\_\{\\ell\}\. We illustrate this instability with a toy experiment in[Figure˜10](https://arxiv.org/html/2607.20555#A8.F10)\. Performance depends sharply on the density choice, both globally and layerwise, and repeated runs do not suggest any simple universal rule for choosingkk\. This is the bottleneck that preventsedge\-popupfrom fully achieving its intended aim\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/loss_landscape_k_sensitivity.png)Figure 10:Thekk\-selection bottleneck: A network with a single width 32 hidden layer was trained usingedge\-popupto approximate a sine function across layerwise density tuples\. Left: The mean\-squared errors of the resulting masked networks are plotted above over a two\-layer grid of the possible layerwise densities\. Right: A heat map of the same figure with contour lines\. As is apparent, the landscape is jagged, highly non\-uniform, and has little to no discernible structure\.On toy networks, one can brute\-force all layerwise density choices and, when one does so,edge\-popupoften performs very well\.[Figure˜11](https://arxiv.org/html/2607.20555#A8.F11)shows the issue on small, synthetic problems where exhaustive search is actually feasible\. Thus, the problem is not thatedge\-popupis incapable of finding strong masks\. Rather, the problem is that the search overkkbecomes prohibitive extremely quickly\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_2.1.2.png)Figure 11:A network with a single width 8 hidden layer was trained usingedge\-popupto approximate a sine function across all layerwise density tuples\. The loss curves are compared to the loss of the best possible mask, found via an exhaustive search of all possible masks\. In the left chart, all loss curves are overlaid\. In the right, each is plotted sequentially\. This reveals thatedge\-popupcan match the best brute\-force mask on a genuinely tiny search space, but also reveals that the dependence onkk\-tuples is highly nonuniform\.Despite the difficulties associated with choosing a good value ofkk, there is one important regime in which the sparsity\-selection problem largely disappears: the case of already sparse weight tensors\. That is, for weights that are already equal to zero, the corresponding mask entry has no effect in terms of the resulting subnetwork\.

###### Example H\.2\.

Observe that the following two masks are distinct \(as they differ in the entry in the second row and second column\) but produce the same result when applied to the weight matrix below:

\[0110\]⊙\[w1,1w1,2w2,10\]=\[0w1,2w2,10\]=\[0111\]⊙\[w1,1w1,2w2,10\]\\begin\{bmatrix\}0&1\\\\ 1&\{\\color\[rgb\]\{1,0,0\}0\}\\\\ \\end\{bmatrix\}\\odot\\begin\{bmatrix\}w\_\{1,1\}&w\_\{1,2\}\\\\ w\_\{2,1\}&0\\\\ \\end\{bmatrix\}=\\begin\{bmatrix\}0&w\_\{1,2\}\\\\ w\_\{2,1\}&0\\\\ \\end\{bmatrix\}=\\begin\{bmatrix\}0&1\\\\ 1&\{\\color\[rgb\]\{1,0,0\}1\}\\\\ \\end\{bmatrix\}\\odot\\begin\{bmatrix\}w\_\{1,1\}&w\_\{1,2\}\\\\ w\_\{2,1\}&0\\\\ \\end\{bmatrix\}
Thus distinct masks can represent the same masked tensor whenever they differ only on locations where the weights are already zero\.

This suggests a natural modification ofedge\-popup\. This is not technically needed for the basicdouble\-scoringconstruction; but instead it records a quantitative selector\-mask version of the same principle\. The core idea is that if many entries of a weight tensor are already zero, then there is no need to assign scores to all coordinates\. Rather, one need only score the entries whose mask values actually matter\. More generally, there are situations in which one may wish to applyedge\-popuponly to a specified subset of the weights\. For instance, one might wish to preserve a distinguished part of the tensor, or enforce that masking respects some additional structural constraint such as symmetry\. This motivates the following selector\-mask version ofedge\-popup\([Algorithm˜3](https://arxiv.org/html/2607.20555#alg3)\)\.

Algorithm 3Network\-leveledge\-popupwith scoring selection1:Frozen network parameters

\{Wt,bt\}t=1L\\\{W\_\{t\},b\_\{t\}\\\}\_\{t=1\}^\{L\}, selector masks

\{Qt,qt\}t=1L\\\{Q\_\{t\},q\_\{t\}\\\}\_\{t=1\}^\{L\}, density parameters

\{kt\}t=1L\\\{k\_\{t\}\\\}\_\{t=1\}^\{L\}, training data, loss function

ℒ\\mathcal\{L\}, learning rate

η\\eta
2:for

t=1,…,Lt=1,\\dots,Ldo

3:Let

It=\{α:\(Qt\)α=1\}I\_\{t\}=\\\{\\alpha:\(Q\_\{t\}\)\_\{\\alpha\}=1\\\}and

Jt=\{β:\(qt\)β=1\}J\_\{t\}=\\\{\\beta:\(q\_\{t\}\)\_\{\\beta\}=1\\\}
4:Initialize score vectors

st∈ℝ\|It\|s\_\{t\}\\in\\mathbb\{R\}^\{\|I\_\{t\}\|\}and

ct∈ℝ\|Jt\|c\_\{t\}\\in\\mathbb\{R\}^\{\|J\_\{t\}\|\}
5:foreach training iterationdo

6:for

t=1,…,Lt=1,\\dots,Ldo

7:

ut←TopKMask⁡\(st;kt\)u\_\{t\}\\leftarrow\\operatorname\{TopKMask\}\(s\_\{t\};k\_\{t\}\)
8:

vt←TopKMask⁡\(ct;kt\)v\_\{t\}\\leftarrow\\operatorname\{TopKMask\}\(c\_\{t\};k\_\{t\}\)
9:Extend

utu\_\{t\}to a full mask

HtH\_\{t\}on

WtW\_\{t\}using selector mask

QtQ\_\{t\}
10:Extend

vtv\_\{t\}to a full mask

hth\_\{t\}on

btb\_\{t\}using selector mask

qtq\_\{t\}
11:

W~t←Wt⊙Ht\\widetilde\{W\}\_\{t\}\\leftarrow W\_\{t\}\\odot H\_\{t\}
12:

b~t←bt⊙ht\\widetilde\{b\}\_\{t\}\\leftarrow b\_\{t\}\\odot h\_\{t\}
13:Compute the network output using

\{\(W~t,b~t\)\}t=1L\\\{\(\\widetilde\{W\}\_\{t\},\\widetilde\{b\}\_\{t\}\)\\\}\_\{t=1\}^\{L\}
14:Compute loss

ℒ\\mathcal\{L\}
15:for

t=1,…,Lt=1,\\dots,Ldo

16:Update

sts\_\{t\}and

ctc\_\{t\}by gradient descent with straight\-through gradients

17:returnFinal masks

\{Ht,ht\}t=1L\\\{H\_\{t\},h\_\{t\}\\\}\_\{t=1\}^\{L\}

The point of[Algorithm˜3](https://arxiv.org/html/2607.20555#alg3)is that it decouples the set of coordinates that receive popup scores from the full support of the weight tensor\. In particular, when a layer is already sparse, one may score all nonzero entries and only some of the zero entries, thereby shrinking the effective search space without losing expressive power\.

We first record a quantitative version of this principle\.

###### Theorem H\.3\.

LetMMbe a weight tensor withnnentries and sparsity

12−1n≤s≤12−2n\.\\frac\{1\}\{2\}\-\\frac\{1\}\{n\}\\leq s\\leq\\frac\{1\}\{2\}\-\\frac\{2\}\{n\}\.Suppose one applies[Algorithm˜3](https://arxiv.org/html/2607.20555#alg3)toMMby assigning popup scores to all nonzero entries ofMMand to any additionaln​\(1−s\)−2n\(1\-s\)\-2zero entries ofMM\. Then, by fixing the target mask density atk=12k=\\frac\{1\}\{2\}, one can represent every function obtainable by a nontrivial masking ofMM\.

###### Proof\.

LetM0=\{i:Mi=0\}M\_\{0\}=\\\{i:M\_\{i\}=0\\\}andM0c=\{i:Mi≠0\}M\_\{0\}^\{c\}=\\\{i:M\_\{i\}\\neq 0\\\}, so that\|M0\|=s​n\|M\_\{0\}\|=snand\|M0c\|=n​\(1−s\)\|M\_\{0\}^\{c\}\|=n\(1\-s\)\. Suppose that popup scores are assigned to all nonzero entries ofMMand tottzero entries ofMM\. Writez:=t\|M0\|z:=\\frac\{t\}\{\|M\_\{0\}\|\}, so thats​z=t/nsz=t/nis the proportion of tracked zero entries relative to the total size ofMM\. The total number of scored entries is therefore

n​\(1−s\)\+n​s​z=n​\(1−s​\(1−z\)\)\.n\(1\-s\)\+nsz=n\\bigl\(1\-s\(1\-z\)\\bigr\)\.
Fix a density parameterkk, and suppose the resulting mask retainsmmof the scored entries\. Then

m=k​n​\(1−s​\(1−z\)\)\.m=kn\\bigl\(1\-s\(1\-z\)\\bigr\)\.Writem=m1\+m2m=m\_\{1\}\+m\_\{2\}, wherem1m\_\{1\}is the number of retained nonzero entries andm2m\_\{2\}is the number of retained scored zero entries\. If

k1:=m1n​\(1−s\)andk2:=m2n​s​z,k\_\{1\}:=\\frac\{m\_\{1\}\}\{n\(1\-s\)\}\\qquad\\text\{and\}\\qquad k\_\{2\}:=\\frac\{m\_\{2\}\}\{nsz\},then

k1​n​\(1−s\)\+k2​n​s​z=k​n​\(1−s​\(1−z\)\)\.k\_\{1\}n\(1\-s\)\+k\_\{2\}nsz=kn\\bigl\(1\-s\(1\-z\)\\bigr\)\.Since0≤k2≤10\\leq k\_\{2\}\\leq 1, it follows that

k​\(1−s​\(1−z\)\)−s​z1−s≤k1≤k​\(1−s​\(1−z\)\)1−s\.\\frac\{k\\bigl\(1\-s\(1\-z\)\\bigr\)\-sz\}\{1\-s\}\\leq k\_\{1\}\\leq\\frac\{k\\bigl\(1\-s\(1\-z\)\\bigr\)\}\{1\-s\}\.Thus, for fixedss,zz, andkk, the proportionk1k\_\{1\}of retained nonzero entries ranges over an interval of widths​z1−s\\frac\{sz\}\{1\-s\}\. Now, choosekkso that this interval is centered at12\\frac\{1\}\{2\}\. One convenient choice is

k=1−s\+s​z2​\(1−s​\(1−z\)\)=12,k=\\frac\{1\-s\+sz\}\{2\\bigl\(1\-s\(1\-z\)\\bigr\)\}=\\frac\{1\}\{2\},which is the midpoint of the admissible interval above\. With this choice,k1k\_\{1\}ranges over an interval centered at12\\frac\{1\}\{2\}of widths​z1−s\\frac\{sz\}\{1\-s\}\.

The nontrivial masking proportions on the nonzero entries are precisely

1n​\(1−s\),2n​\(1−s\),…,n​\(1−s\)−1n​\(1−s\)\.\\frac\{1\}\{n\(1\-s\)\},\\frac\{2\}\{n\(1\-s\)\},\\dots,\\frac\{n\(1\-s\)\-1\}\{n\(1\-s\)\}\.Hence, it is enough for the interval of possiblek1k\_\{1\}values to contain all of these, which will occur provided its width is1−2n​\(1−s\)1\-\\frac\{2\}\{n\(1\-s\)\}\. Thus, we impose

s​z1−s=1−2n​\(1−s\)\.\\frac\{sz\}\{1\-s\}=1\-\\frac\{2\}\{n\(1\-s\)\}\.Solving forzzgives

z=n​\(1−s\)−2n​s\.z=\\frac\{n\(1\-s\)\-2\}\{ns\}\.Equivalently, the number of zero entries that must be assigned scores is

t=\|M0\|​z=s​n⋅n​\(1−s\)−2n​s=n​\(1−s\)−2\.t=\|M\_\{0\}\|z=sn\\cdot\\frac\{n\(1\-s\)\-2\}\{ns\}=n\(1\-s\)\-2\.Therefore, by tracking scores on all nonzero entries ofMMand on any additionaln​\(1−s\)−2n\(1\-s\)\-2zero entries, the selector\-mask version ofedge\-popupat fixed densityk=12k=\\frac\{1\}\{2\}can realize every nontrivial masking pattern on the nonzero entries ofMM\. Since the zero entries ofMMdo not affect the represented function, this proves the claim\. ∎

The content of[Theorem˜H\.3](https://arxiv.org/html/2607.20555#A8.Thmtheorem3)is that, on a sufficiently sparse tensor, selector\-based scoring allows one to simulate the full family of nontrivial masking levels while keeping the target density fixed atk=12k=\\frac\{1\}\{2\}\. Thus, the apparent freedom in the density parameter is, in this regime, largely illusory: one may trade a search over densities for a suitable choice of scored coordinates\. This has a particularly clean consequence when one is interested only in representability rather than in the bookkeeping of scored zero coordinates\. This is the content of[Proposition˜5\.1](https://arxiv.org/html/2607.20555#S5.Thmtheorem1)in the main paper, the proof of which is included below\.

##### Proof of[Proposition˜5\.1](https://arxiv.org/html/2607.20555#S5.Thmtheorem1)

###### Proof\.

LetPPbe the set of indices at whichWWis nonzero, and letZZbe the set of indices at whichWWis zero\. Since at least half of the entries ofWWvanish, we have\|Z\|≥⌈d/2⌉≥⌊d/2⌋=r\|Z\|\\geq\\lceil d/2\\rceil\\geq\\lfloor d/2\\rfloor=r, and, hence,\|P\|=d−\|Z\|≤d−⌈d/2⌉=⌊d/2⌋=r\|P\|=d\-\|Z\|\\leq d\-\\lceil d/2\\rceil=\\lfloor d/2\\rfloor=r\.

LetM∈\{0,1\}m×nM\\in\\\{0,1\\\}^\{m\\times n\}and defineM∗M^\{\\ast\}as follows\. First, setMi​j∗=Mi​jM^\{\\ast\}\_\{ij\}=M\_\{ij\}for all\(i,j\)∈P\(i,j\)\\in P\. This guarantees thatW⊙MW\\odot MandW⊙M∗W\\odot M^\{\\ast\}agree on every nonzero entry ofWW\. Now add ones toM∗M^\{\\ast\}on enough indices fromZZso thatM∗M^\{\\ast\}has exactlyrrones in total, and set all remaining entries onZZequal to zero\.

This construction is always possible because the number of ones already prescribed onPPis at most\|P\|≤r\|P\|\\leq r, while there are at leastrrindices available inZZ\. Moreover, every additional one placed onZZmultiplies a zero entry ofWW, and therefore does not change the masked tensor\. Hence,W⊙M=W⊙M∗W\\odot M=W\\odot M^\{\\ast\}\. ∎

The layerwise version of this then follows in[Corollary˜5\.2](https://arxiv.org/html/2607.20555#S5.Thmtheorem2)\.

##### Proof of[Corollary˜5\.2](https://arxiv.org/html/2607.20555#S5.Thmtheorem2)

###### Proof\.

Apply[Proposition˜5\.1](https://arxiv.org/html/2607.20555#S5.Thmtheorem1)independently to each layer\. ∎

Applied layerwise,[Corollary˜5\.2](https://arxiv.org/html/2607.20555#S5.Thmtheorem2)says that once each layer is at least half sparse, the entire search space of masked subnetworks can be represented using a single fixed density near12\\frac\{1\}\{2\}in every layer, thereby removing the principal*expressive*obstruction\.

###### Example H\.5\.

Simple sparse\-network experiments line up with this picture \([Figure˜12](https://arxiv.org/html/2607.20555#A8.F12)\)\. When the base model is initialized with substantial sparsity, good performance tends to cluster near the half\-density regime rather than requiring a finely tuned layerwise search\. The comparison is particularly stark when contrasted with[Figure˜10](https://arxiv.org/html/2607.20555#A8.F10): in the sparse regime, the test accuracy landscape is substantially smoother with respect to variations in the layerwisekkvector\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_3.1.1.1.png)

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_3.1.1.2.png)

Figure 12:Left: test accuracy as a function ofkk\-tuples for a 50% sparsity two\-layer network trained on the scikit\-learn digits dataset\. The best performance begins to appear around the center of eachkkinterval\. Right: A heat map of test losses for the same network acrosskk\-tuples\. The best\-performing region concentrates near the fixed\-density regime suggested by[Proposition˜5\.1](https://arxiv.org/html/2607.20555#S5.Thmtheorem1), consistent with the claim that the sparse regime reduces the effective burden of layerwise density selection\.
These observations lead naturally to our implementation ofdouble\-scoring, which treats a network as being synthetically embedded in a larger dense network\. This is enough by[Proposition˜5\.3](https://arxiv.org/html/2607.20555#S5.Thmtheorem3), the proof of which is given below\.

##### Proof of[Proposition˜5\.3](https://arxiv.org/html/2607.20555#S5.Thmtheorem3)

###### Proof\.

Letm^\\widehat\{m\}agree withmmon the firstddcoordinates\. Ifmmcontainsqqones, then add exactlyd−qd\-qones among the finalddcoordinates ofm^\\widehat\{m\}and set all remaining auxiliary coordinates to zero\. Since the finalddcoordinates ofw^\\widehat\{w\}are zero, these additional ones do not affect the masked vector\. By construction,m^\\widehat\{m\}has exactlyddones and restricts tommon the original coordinates\.

∎

This is the key representational idea: by enlarging the*score space*, a dense layer can be treated as though it were embedded inside a half\-sparse augmented layer, without ever modifying the underlying weights\. This yieldsdouble\-scoring\. The comparison ofdouble\-scoringto the embedded network in[Proposition˜5\.3](https://arxiv.org/html/2607.20555#S5.Thmtheorem3)is not merely representationally equivalent toedge\-popupon an augmented half\-sparse system; the optimization dynamics agree exactly as well\.

###### Theorem H\.6\(double\-scoringas augmentededge\-popup\)\.

Fix a feed\-forward network with frozen weights\{Wt,bt\}t=1L\\\{W\_\{t\},b\_\{t\}\\\}\_\{t=1\}^\{L\}\. For each layertt, let

W^t:=\(Wt,0\)andb^t:=\(bt,0\),\\widehat\{W\}\_\{t\}:=\(W\_\{t\},0\)\\qquad\\text\{and\}\\qquad\\widehat\{b\}\_\{t\}:=\(b\_\{t\},0\),where the second half consists of zero entries and the concatenation is taken after flattening the tensors\. Likewise, let

S^t:=\(St,Tt\)andf^t:=\(ft,gt\)\\widehat\{S\}\_\{t\}:=\(S\_\{t\},T\_\{t\}\)\\qquad\\text\{and\}\\qquad\\widehat\{f\}\_\{t\}:=\(f\_\{t\},g\_\{t\}\)be the corresponding doubled score tensors\.

Consider the following two procedures, both run at density1/21/2:

1. 1\.classicaledge\-popupapplied to the augmented tensors\(W^t,b^t\)\(\\widehat\{W\}\_\{t\},\\widehat\{b\}\_\{t\}\)with scores\(S^t,h^t\)\(\\widehat\{S\}\_\{t\},\\widehat\{h\}\_\{t\}\);
2. 2\.double\-scoringapplied to the original tensors\(Wt,bt\)\(W\_\{t\},b\_\{t\}\)with score pairs\(St,Tt\)\(S\_\{t\},T\_\{t\}\)and\(ft,gt\)\(f\_\{t\},g\_\{t\}\), where the effective masks are obtained by restrictingTopKMask⁡\(S^t;1/2\)\\operatorname\{TopKMask\}\(\\widehat\{S\}\_\{t\};1/2\)andTopKMask⁡\(f^t;1/2\)\\operatorname\{TopKMask\}\(\\widehat\{f\}\_\{t\};1/2\)to the original coordinates\.

Assume that the backward pass forTopKMask\\operatorname\{TopKMask\}uses the usual straight\-through estimator, i\.e\. it acts as the identity on gradients\.

Then, at every training iteration:

1. 1\.the masked parameters used by the two procedures agree on the original coordinates, and hence the network outputs and losses are identical;
2. 2\.the gradients of the original score coordinates agree in the two procedures;
3. 3\.the gradients of the auxiliary score coordinates are zero in both procedures\.

Consequently, after restricting to the original coordinates,double\-scoringproduces exactly the same score iterates, masks, network outputs, and losses as classicaledge\-popupon the augmented zero\-padded network\.

###### Proof\.

We compare gradients first with respect to the magnitude scoresAt=\|S^t\|A\_\{t\}=\|\\widehat\{S\}\_\{t\}\|\. Since thedouble\-scoringloss depends onH^t\\widehat\{H\}\_\{t\}only through its restrictionHtH\_\{t\}to the original coordinates, its gradient with respect toH^t\\widehat\{H\}\_\{t\}has the form

∂ℒ∂H^t=\(∂ℒ∂Ht,0\)\.\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\widehat\{H\}\_\{t\}\}=\\left\(\\frac\{\\partial\\mathcal\{L\}\}\{\\partial H\_\{t\}\},0\\right\)\.Under the straight\-through estimator for the hard top\-kkmap, this is also the gradient with respect to the magnitude scoresAtA\_\{t\}\. Thus the auxiliary magnitude\-score coordinates receive zero gradient\.

For classicaledge\-popupon the augmented zero\-padded tensor, the loss depends onH^t\\widehat\{H\}\_\{t\}only through

W^t⊙H^t=\(Wt⊙Ht,0\)\.\\widehat\{W\}\_\{t\}\\odot\\widehat\{H\}\_\{t\}=\(W\_\{t\}\\odot H\_\{t\},0\)\.Consequently,

∂ℒ∂H^t=∂ℒ∂\(W^t⊙H^t\)⊙W^t=\(∂ℒ∂\(Wt⊙Ht\)⊙Wt,0\),\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\widehat\{H\}\_\{t\}\}=\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\(\\widehat\{W\}\_\{t\}\\odot\\widehat\{H\}\_\{t\}\)\}\\odot\\widehat\{W\}\_\{t\}=\\left\(\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\(W\_\{t\}\\odot H\_\{t\}\)\}\\odot W\_\{t\},0\\right\),which is the same gradient obtained in the restricteddouble\-scoringformulation\. Finally, gradients with respect to the raw scores are obtained by multiplying bysign⁡\(S^t\)\\operatorname\{sign\}\(\\widehat\{S\}\_\{t\}\), away from zero\. Since the raw scores are the same in the two procedures, this sign factor is identical, and the raw\-score gradients agree as well\. The argument for the bias terms is analogous\. ∎

This allows us to directly inherit the local monotonicity result \([Theorem˜H\.1](https://arxiv.org/html/2607.20555#A8.Thmtheorem1)\) obtained by Ramanujan et al\.

###### Corollary H\.8\.

Any local monotonicity result \(e\.g\.[Theorem˜H\.1](https://arxiv.org/html/2607.20555#A8.Thmtheorem1)\) for classicaledge\-popupat density1/21/2applies verbatim todouble\-scoringvia the augmented zero\-padded realization of[Theorem˜H\.6](https://arxiv.org/html/2607.20555#A8.Thmtheorem6)\.

###### Proof\.

By[Theorem˜H\.6](https://arxiv.org/html/2607.20555#A8.Thmtheorem6),double\-scoringis exactly classicaledge\-popupon the augmented zero\-padded network after restriction to the original coordinates\. Hence any one\-step loss improvement statement for classicaledge\-popuptransfers immediately\. ∎

###### Example H\.9\.

To demonstrate with a toy example, the effect ofdouble\-scoringis illustrated in[Figure˜13](https://arxiv.org/html/2607.20555#A8.F13)\. The fixed\-density procedure tracks the outcome of an exhaustive search over layerwise densities without requiring separate runs for each choice of\(k1,…,kL\)\(k\_\{1\},\\dots,k\_\{L\}\)\. In particular, it recovers performance comparable to the best density configuration while avoiding the associated combinatorial cost\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_4.1.1.png)Figure 13:Comparison between the proposed fixed\-density method and brute\-force layerwise density search, with the loss curves presented sequentially\. The loss curve fordouble\-scoringis repeated for each layerwise density tuple for ease of comparison \(albeit at the cost of some visual clarity\)\. As is apparent, thedouble\-scoringprocedure rapidly reaches the same level of loss asedge\-popupwith the bestkk\-tuple choices over an exhaustive search\.

## Appendix IScaling heuristics

Equipped now with the ability to obtain strong lottery tickets, we are able to provide empirical support for the logarithmic overparameterization scaling predicted by Pensia et al\.\[[14](https://arxiv.org/html/2607.20555#bib.bib16)\]\. As the width of the network increases, the likelihood of successfully recovering near\-optimal subnetworks improves markedly, even when the number of parameters grows only modestly\. This observation is consistent with the theoretical prediction that relatively mild overparameterization suffices to ensure the existence \(and practical recoverability\) of high\-quality lottery tickets\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.4.1.png)Figure 14:Approximation error trends are broadly consistent with the logarithmic\-overparameterization heuristics suggested by\[[14](https://arxiv.org/html/2607.20555#bib.bib16)\]\(and, in fact, often more favorable\)\. We find only very occasional deviation in the small weight regime where initialization noise dominates\.[Figure˜14](https://arxiv.org/html/2607.20555#A9.F14)illustrates the relationship between the theoretical scaling law and the empirical performance of thedouble\-scoringmodel\. The left\-most chart plots the parameter counts of the masked model \(mm\) against those of the comparison model \(nn\), overlaid with the theoretical scaling curvesm=n​log2⁡\(1/ε\)m=n\\log\_\{2\}\(1/\\varepsilon\)for several target errorsε\\varepsilon\. Each point corresponds to a trained model and is colored according to the logarithmic ratiolog10⁡\(εachεth\)\\log\_\{10\}\\\!\\left\(\\frac\{\\varepsilon\_\{\\mathrm\{ach\}\}\}\{\\varepsilon\_\{\\mathrm\{th\}\}\}\\right\), whereεth=2−m/n\\varepsilon\_\{\\mathrm\{th\}\}=2^\{\-m/n\}is the predicted error andεach\\varepsilon\_\{\\mathrm\{ach\}\}is the observed test error\. The concentration of points near or below the theoretical curves indicates that the parameter allocation is correctly bounded by the predicted scaling\. Because the model parameter countsmmandnnare integers, the realized theoretical valuesεth=2−m/n\\varepsilon\_\{\\mathrm\{th\}\}=2^\{\-m/n\}differ slightly from the nominal target values; accordingly, the guide lines in this graph correspond to the realized theoreticalε\\varepsilonvalues rather than the nominal targets\.

The center chart provides a direct comparison between achieved and predicted errors on a log–log scale\. Each point represents a single experiment, and the diagonaly=xy=xcorresponds to perfect agreement between theory and practice\. Across a wide range of configurations, the points lie close to or below this diagonal, demonstrating that the empirical error tracks the theoretical prediction\. The deviations are relatively small and do not exhibit a systematic bias, suggesting that the scaling law captures the dominant behavior\.

The right\-most chart summarizes this agreement by plotting the ratioεachεth\\frac\{\\varepsilon\_\{\\mathrm\{ach\}\}\}\{\\varepsilon\_\{\\mathrm\{th\}\}\}for each experiment\. Values near or below11indicate strong agreement with theory\. We observe that this ratio remains approximately bounded by unity across all tested configurations, with only modest variance\. Taken together, these results provide strong empirical evidence that the approximation error achieved by the masked model scales asε≈2−m/n\\varepsilon\\approx 2^\{\-m/n\}, in accordance with the theoretical prediction\.

## Appendix JBrute\-force optimal subnetwork search and limitations

To better understand the behavior ofdouble\-scoring, we conduct experiments on small networks where exhaustive search over all possible masks is computationally feasible\. This allows us to directly compare the subnetworks found bydouble\-scoringagainst the true optimal subnetworks\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.3.1.png)Figure 15:Loss curves for training a network with one hidden layer of width 8 trained to approximate a sine curve using an exhaustive layerwise density search \(dashed lines\), thedouble\-scoringalgorithm \(solid orange line\), compared to the loss values for the overall best mask found via a brute\-force search of all possible masks \(solid green line\)\. In this toy example, the subnetwork produced bydouble\-scoringmatches or exceeds the performance of the subnetworks produced by the classicaledge\-popupalgorithm at all layerwise density tuples and nearly reaches the performance of the optimal subnetwork\.In general,double\-scoringtypically recovers subnetworks whose performance approaches that of the optimal mask in the toy case\. However, it does not always recover the optimal mask: as[Figure˜15](https://arxiv.org/html/2607.20555#A10.F15)illustrates, in some instances, the procedure converges to high\-performing subnetworks that are, all the same, suboptimal relative to the global optimum\. This behavior is consistent with the optimization landscape of classical training \(as illustrated in[Figure˜16](https://arxiv.org/html/2607.20555#A10.F16)\)\. That is, while the search is performed over score variables rather than weights, the method is subject to the same types of limitations as classical neural network training, including sensitivity to initialization and the presence of local minima\. In this sense, the difficulty lies not in the expressivity of the search space \(which, as shown in previous sections, is sufficiently rich\) but in the optimization dynamics required to navigate it\. Taken together, these results suggest that the primary obstacle in lottery ticket extraction is not representational, but algorithmic: even when the correct subnetwork exists and is accessible within the search space, finding it remains an optimization problem with the same type of inherent difficulties as classical training\. This may be possible to ameliorate by taking into account the sparsity dynamics of the obtained mask \(see[Section˜M\.1](https://arxiv.org/html/2607.20555#A13.SS1)for further details\), though this is purely speculation at this point\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.3.2.png)Figure 16:Two identical networks were trained for 20 epochs on the MNIST Fashion data set\. The loss curve during training of withdouble\-scoring\(blue\) mirrors the loss curve of a network trained classically \(orange\), albeit with a somewhat slower initial rate of convergence\.
## Appendix KStrong tickets as weak tickets

A commonly held view in the lottery ticket literature is that strong and weak lottery tickets represent fundamentally different phenomena\. In particular, prior work \(e\.g\.,\[[15](https://arxiv.org/html/2607.20555#bib.bib2)\]\) suggests that subnetworks which perform well without training need not retain this advantage after training, and vice versa\.

In contrast, our experiments indicate that, in the setting considered here, the distinction between strong and weak lottery tickets appears to vanish\. Specifically, the subnetworks identified by the proposed method not only achieve high performance without training \(as strong tickets\), but also train effectively when optimized in isolation, achieving accuracy comparable to the unmasked network at essentially exactly the same rate during training\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.5.1.png)Figure 17:Observationally, it often appears to be the case that the obtained strong lottery tickets are also weak lottery tickets\. Here, the left\-most blue curve is the loss curve of a network during double\-scorededge\-popuptraining, the right green curve is the masked network trained classically, and the orange curve is an identical unmasked network trained classically\. As is readily apparent, the training dynamics of the masked network and the unmasked network are virtually identical\.Empirically, this behavior is consistent with that displayed in[Section˜6\.2](https://arxiv.org/html/2607.20555#S6.SS2)\. In particular, subnetworks extracted viadouble\-scoringexhibit both strong initial performance and robust trainability, suggesting that the properties defining strong and weak lottery tickets may, in practice, coincide under appropriate extraction procedures\. That is, the apparent gap between strong and weak lottery tickets may not be intrinsic to the networks themselves, but rather to the methods previously used to identify them\. Furthermore, as evidenced in[Section˜6\.1](https://arxiv.org/html/2607.20555#S6.SS1)movement pruning appears to produce weak tickets that performbetterafter training than the original frozen network after a similar number of iterations, whiledouble\-scoringappears to produce subnetworks that more closely match the original network after training\. While speculative, this suggests that it is possible for strong and weak lottery tickets to coincide, yet deviate from some form of as\-of\-yet unidentified ’super’ weak tickets that movement pruning is able to extract\. We leave an investigation of this possibility for future work, and claim here only that these results support the interpretation thatdouble\-scoringsubstantially narrows the gap between strong and weak tickets in this setting: the extracted subnetworks require little additional optimization to reach their trained sparse accuracy\.

These findings do not contradict existing theoretical results, which establish existence but do not characterize the typical behavior of extracted subnetworks\. However, they do suggest that the commonly assumed separation between strong and weak lottery tickets may be, at least in part, an artifact of the extraction methods used\. By removing the sparsity\-selection bottleneck, the proposed approach appears to recover subnetworks that are simultaneously strong and trainable\.

One possible explanation for this phenomenon is that enlarging the score space allows the optimization procedure to more effectively explore subnetworks that are both well\-aligned with the target function at initialization and stable under subsequent training\. In this sense, the method may be implicitly biasing the search toward subnetworks that satisfy both criteria simultaneously\. Whether this is actually the case, let alone why this may occur, is not clear\.

## Appendix LBroader impacts

This work is foundational and algorithmic: it proposes a method for extracting sparse subnetworks from randomly initialized neural networks and evaluates it on standard public benchmarks\. Potential positive impacts include improved computational efficiency, reduced training or inference costs, and increased accessibility of sparse neural network methods for researchers with limited compute\. Potential negative impacts are indirect\. As with other improvements in neural network efficiency, better sparse\-model extraction could reduce the cost of deploying models in downstream applications, including applications with fairness, privacy, security, or misuse risks\. These risks depend on the downstream system and deployment context rather than on the mask\-extraction method itself\.

### L\.1Declaration of LLM usage

LLM assistance was used during manuscript preparation and experimental\-code development, including drafting, editing, formatting, and generating or revising implementation code\. LLMs were not used as part of the proposed algorithm, as experimental data, as model components, as baselines, or as evaluation signals\. The authors reviewed, modified, debugged, and validated the final code and are responsible for the methodology, experiments, results, and claims\.

## Appendix MSpeculative extensions

This appendix reports exploratory experiments that are not used to support the main claims\. They are included to document possible directions for future work\. Many are, admittedly, largely toy configurations and proof\-of\-concept tests; their purpose is not to rigorously and empirically demonstrate an established hypothesis, but to illustrate potential avenues for further study and describe some phenomena that was observed in testing which may be of interest\.

### M\.1Extra capacity in the score space & sparsity control

A natural question is whether the performance of the proposed method can be further improved by enlarging the score space beyond the doubling used in thedouble\-scoringconstruction\. To investigate this, we consider variants in which each weight tensor is associated with more than two score tensors, thereby increasing the dimensionality of the score space while keeping the weight tensors fixed\.

Across a number of toy experiments \([Figure˜18](https://arxiv.org/html/2607.20555#A13.F18)\), we observe that this additional capacity tends to yield no similarly meaningful improvement in performance; the primary gain comes from the first doubling\. In particular, increasing the number of auxiliary score tensors or their size does not lead to substantially higher test accuracy, faster convergence, or more reliable recovery of high\-quality subnetworks\. In general, the results tend to be effectively indistinguishable from those obtained with standarddouble\-scoring\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.6.1.png)Figure 18:Two identical networks were trained for 20 epochs on the MNIST Fashion dataset with thedouble\-scoring, with one network receiving an a large number of additional trainable score parameters\. As is readily apparent, this produces virtually no change in the training dynamics\.This behavior suggests that the benefit ofdouble\-scoringis not simply due to an increase in the number of trainable parameters, but rather to the specific structural effect of embedding the original network into a half\-sparse augmented representation\. Once this embedding is achieved, further enlarging the score space appears to provide negligible, if any, returns\. From this perspective,double\-scoringmay be viewed as a minimal modification that removes the sparsity\-selection bottleneck without introducing unnecessary additional capacity\. This further supports the interpretation that the key role of the augmented score space is to remove a structural constraint, rather than to provide additional expressive capacity\.

However, itis not the casethat increasing the score dimension beyond this point yields no further expansion to the set of practically reachable subnetworks\. Certainly, it does not improve theirperformance, but instead it appears to act as a fine\-grained control knob on thesparsityof the obtained subnetwork\. This makes some sense intuitively: From a given initialization of scores and a specifiedkk\-value, the size of the score tensor controls the proportion of selected scores that will be used in a mask\. As a general rule, it appears that setting thekk\-value determines the starting point of the mask’s sparsity, while the scale of extra capacity in the score space determines the rate at which the obtained mask tends to deviate in sparsity from the initialkk\-value\. Interestingly, it appears to be the case that, often, the best performing mask seems to occur around a sparsity of12\\frac\{1\}\{2\}and all other masks, given sufficient extra capacity, seem to pull towards this value irrespective of their initialkk\-value, as illustrated in[Figure˜19](https://arxiv.org/html/2607.20555#A13.F19)\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.6.2.png)Figure 19:By changing the initialkk\-value and additional capacity in the score tensors, it is possible to fine\-tune the sparsity of the mask obtained viadouble\-scoring\. The initialkk\-value determines the sparsity of the mask at initialization, and the ratio of the number of parameters in the score tensors and weight tensors controls the speed of its deviation from this value\. Though it is unclear why, it appears that sufficiently large score tensors tend to eventually produce masks with a sparsity of 0\.5, irrespective of the initialkk\-value\.Additionally, it should be noted that it is possible to repeatedly apply[Algorithm˜1](https://arxiv.org/html/2607.20555#alg1), mask the weights of the network, then repeat this process\. Empirically, this appears to allow one to rapidly increase the sparsity of the obtained subnetwork \(on a scale of2−n2^\{\-n\}initially, wherennis the number of repetitions of this process\) with only mild degradation to performance for small increases innn\. This is depicted in[Figure˜20](https://arxiv.org/html/2607.20555#A13.F20)\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.6.3.png)Figure 20:By repeatedly performing[Algorithm˜1](https://arxiv.org/html/2607.20555#alg1)and applying the obtained mask to the weights of the model between each repetition, it is possible to achieve high sparsity masks with only a modest effect on the performance of the model\.Finally, we observe that the above suggests that it may be possible to strengthen the strong lottery ticket hypothesiseven further yet still\. That is, it may not just be the case that all sufficiently large networks contain, with high probability, a subnetwork approximating whatever continuous function one might choose to whatever degree of precision on a compact set they might desire, but they might containextraordinarily manysuch subnetworksat virtually all levels of sparsity\. More compactly, we make the following conjecture:

###### Conjecture M\.1\.

Given any feed\-forward networkgg, a compact subsetKKof the input space, anyϵ\>0\\epsilon\>0, and anyδ∈\(0,1\)\\delta\\in\(0,1\), there existsNNsuch that a randomly initialized network with equivalent input and output dimensions and widthsmin⁡\{ni:1≤i≤ℓ−1\}≥N\\min\\\{n\_\{i\}:1\\leq i\\leq\\ell\-1\\\}\\geq Ncontains, with probability at leastδ\\delta, a maskHHwith sparsitysssatisfying‖fW⊙H−g‖K,∞<ϵ\\\|f\_\{W\\odot H\}\-g\\\|\_\{K,\\infty\}<\\epsilonfor all realizable sparsitiess∈\(0\.5−η,0\.5\+η\)s\\in\(0\.5\-\\eta,0\.5\+\\eta\), whereη→0\.5\\eta\\rightarrow 0\.5asN→∞N\\rightarrow\\infty\.

Notice that this conjecture, if true, would provide useful intuition for whydouble\-scoringis effective at uncovering strong lottery tickets\. A fixed sparsity constraint may be overly rigid: even when a randomly initialized network contains a strong ticket, that ticket need not occur at the particular sparsity level specified in advance\. Instead, the initialization may contain strong subnetworks atnearbysparsity levels\. Standardedge\-popupmust search within a single prescribed mask density, whereasdouble\-scoringgives the optimization procedure enough flexibility to move to a nearby realizable sparsity level where a strong ticket is present\.

### M\.2Simultaneous training and masking

A natural extension of the proposed method is to combine mask selection and weight training within a single procedure\. To this end, we consider a hybrid approach in which both the score tensors \(viadouble\-scoring\) and the network weights are updated simultaneously during training\.

Empirically in toy examples \([Figure˜21](https://arxiv.org/html/2607.20555#A13.F21)\), this approach does not yield meaningful improvements toperformanceover the pure masking procedure, though it does yield a desirable level of sparsity \(tunable via[Section˜M\.1](https://arxiv.org/html/2607.20555#A13.SS1)\) rapidly in a single stage process\. In most experiments, the mask stabilizes very early in training, after which the method effectively reduces to standard training of a fixed subnetwork, as shown in[Figure˜21](https://arxiv.org/html/2607.20555#A13.F21)\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.7.1.png)Figure 21:Training with simultaneous masking and weight modification produces no meaningful difference when compared to classical training \(save for a fixed proportion parameter reduction\), with the mask stabilizing almost immediately, thereby producing–essentially–a classical model of approximately half the number of trainable parameters \(for a doubled score space andk=0\.5k=0\.5\)\.This behavior suggests an inherent imbalance between the two optimization processes\. The score updates, driven by the straight\-through estimator and the top\-kkselection mechanism, tend to induce rapid, discrete changes in the mask, while weight updates proceed more gradually\. As a result, the mask selection phase effectively terminates before the weights have sufficiently adapted, leading to early commitment to a subnetwork that may be suboptimal\. This suggests that separating mask selection from weight training may be an essential feature of effective lottery ticket extraction, rather than a limitation of existing methods\.

We emphasize, however, that this negative result should be interpreted cautiously\. The interaction between mask optimization and weight training is delicate, and the simple joint training scheme considered here may not adequately balance the two objectives\. It is plausible that alternative strategies \(scheduling, regularization, temperature\-based relaxations of the masking operation, etc\.\) could yield improved performance\. We leave a more systematic investigation of such approaches to future work\.

### M\.3Lottery tickets in other models

In this section, we note that the procedure indouble\-scoringdoes not appear to be intrinsically tied to standard feedforward neural networks, but rather exploits a more general structural feature: the representation of model components as linear combinations of basis elements\. In the classical setting, a weight matrix can be viewed as a linear combination of elementary matrices, which form an orthonormal basis for the space of linear operators with an appropriate inner product\. Masking, in this context, corresponds to an orthogonal projection onto a subspace spanned by a subset of these basis elements\. Thedouble\-scoringalgorithm effectively learns such projections, enabling the extraction of subnetworks that function as strong lottery tickets, and hence also as weak lottery tickets\.

This perspective suggests a natural generalization: any model composed of layers that admit a representation as linear combinations of basis functions interleaved with nonlinearities probably, in principle, admits an analogous masking procedure\. To provide preliminary evidence for this claim, we construct an alternative toy model based on compositions of polynomial expansions, where coefficients are taken with respect to standard bases \(e\.g\., monomial, Legendre, or Chebyshev\)\. To emphasize, this is not a well\-performing class of models for most purposes–rather, we use it here purely to expose the generality of lottery ticket phenomenon in a toy model\. Applying the samedouble\-scoringmechanism to mask coefficients, we observe behavior \([Figure˜22](https://arxiv.org/html/2607.20555#A13.F22)\) qualitatively similar to that seen in neural networks \(c\.f\.[Figure˜17](https://arxiv.org/html/2607.20555#A11.F17)\): the procedure successfully identifies sparse substructures that perform comparably to dense models, and these substructures can be further trained to yield weak lottery tickets\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.8.1.1.png)

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.8.1.2.png)

Figure 22:Function approximation using masked polynomial compositions\. Thedouble\-scoringprocedure identifies a sparse subset of basis coefficients \(strong lottery ticket\) whose induced model closely matches both a retrained weak ticket and a fully dense model, producing effectively the same picture as[Figure˜17](https://arxiv.org/html/2607.20555#A11.F17)\. This demonstrates that the lottery ticket phenomenon, interpreted as projection onto a learned subspace of basis functions, persists beyond standard neural network architectures\.We do not pursue a systematic study of this phenomenon here\. However, these results suggest that the effectiveness of double scoring may extend well beyond fully connected architectures, and we conjecture that it applies broadly to models admitting basis decompositions, where masking can be interpreted as projection onto learned subspaces\.

### M\.4Training parallelization

A distinguishing feature of score\-based strong lottery ticket extraction algorithms, likeedge\-popupanddouble\-scoringis that the underlying network weights remain fixed throughout the mask selection process\. All task\-specific adaptation is carried out exclusively through the score tensors, which determine the active subnetwork\. This separation between a shared, frozen parameterization and task\-dependent masking yields the possibility of parallelization in training\. Specifically, a single randomly initialized network may be viewed as a shared computational substrate capable of supporting many distinct tasks simultaneously\. Different tasks correspond to different masks, each selecting a subnetwork adapted to a particular objective\. Since the weights are never modified, multiple mask optimization procedures can be run in parallel on the same underlying model without interference\. The resulting masks can be interpreted as task\-specific specializations of a common architecture\. Moreover, because sufficiently sparse masks are low\-dimensional relative to the full parameter space, they can be efficiently stored, transmitted, and aggregated\.

An additional possibility is to combine information across parallel mask searches\. For instance, one may aggregate score tensors \(or induced masks\) obtained from different tasks or random initializations and use them to bias subsequent searches toward consistently useful subnetworks \(this is explored further in[Section˜M\.5](https://arxiv.org/html/2607.20555#A13.SS5)\)\. In this sense, the score space provides a natural medium for sharing information across distributed optimization processes\. This raises the possibility of collaborative or federated mask discovery, in which multiple agents explore the space of subnetworks and exchange information to accelerate convergence\.

### M\.5Implicit transfer learning for multi\-task models

Biological brains exhibit strong resource efficiency compared to artificial neural networks\. For instance, brains do not globally optimize synapses for every new task but instead reuse pre\-configured circuits \(e\.g\., olfactory information as a memory cue\)\. Using strong lottery tickets and masking the same frozen base model for multiple tasks, we observe some vaugely similar phenomenon\. Loosely speaking, there is animplicitanalog to a ‘mixture of experts’ model\[[2](https://arxiv.org/html/2607.20555#bib.bib71),[6](https://arxiv.org/html/2607.20555#bib.bib70),[12](https://arxiv.org/html/2607.20555#bib.bib69),[1](https://arxiv.org/html/2607.20555#bib.bib68),[24](https://arxiv.org/html/2607.20555#bib.bib67),[10](https://arxiv.org/html/2607.20555#bib.bib66)\]inside every network, wherein the subnetworks are each ‘experts’ at different tasks\. Looking at how these ‘experts’ relate to each other reveals the possibility of an interestingimplicit form of transfer learning\[[20](https://arxiv.org/html/2607.20555#bib.bib74),[25](https://arxiv.org/html/2607.20555#bib.bib75),[7](https://arxiv.org/html/2607.20555#bib.bib76)\]through mask similarity\. In short, we observe that, with a fixed base model, the trained masks for related tasks often have a high degree of overlap in their active parameters\. As a slogan, think: “similar tasks, similar masks"\. This mimics the biological case to some extent, reflecting the presence of something like conserved computational motifs within the network\.

To show preliminary evidence of this with a toy model, we use the MedMNIST datasets\[[21](https://arxiv.org/html/2607.20555#bib.bib65),[22](https://arxiv.org/html/2607.20555#bib.bib64)\]and mask a fixed model repeatedly with[Algorithm˜1](https://arxiv.org/html/2607.20555#alg1)\. Looking at the overlap of the obtained strong lottery ticket masks, they appear to have a degree of similarity directly proportional to the human\-intuitive degree of similarity between the tasks\. That is, breast cancer diagnosis from ultrasound data and skin lesion classification from dermatascope imaging have a large amount of overlap and involve the utilization of similar parts of the frozen base model, while less similar tasks, like breast cancer diagnosis from ultrasound data and retinal disease classification from retinal optical coherence tomography data, utilize different portions\. Maintaining the biological analogy, similarities across masks allow for something like developmental plasticity, wherein organisms repurpose evolutionarily ancient circuits \(e\.g\., basic motion detection\) for novel tasks \(e\.g\., recognizing complex gestures\)\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.10.1.png)Figure 23:Similar Tasks Have Similar Masks: Well\-performing masks are identified in a small feed\-forward network using[Algorithm˜1](https://arxiv.org/html/2607.20555#alg1)withk=0\.5k=0\.5to perform various binary or multi\-label classification tasks using image data in the MEDMNIST dataset\[[21](https://arxiv.org/html/2607.20555#bib.bib65),[22](https://arxiv.org/html/2607.20555#bib.bib64)\]\. The resulting masks are compared for overlap \(both in all parameters and in active parameters only\) using the mask from the BreastMNIST dataset as a reference, clearly showing that similar tasks \(i\.e\. breast cancer diagnosis from ultrasound data and skin lesion classification from dermatascope imaging\) involve the utilization of similar parts of the frozen base model, while less similar tasks \(i\.e\. breast cancer diagnosis from ultrasound data and retinal disease classification from retinal optical coherence tomography data\) utilize different portions, as evidenced by their excess overlap beyond the expected 50% overlap one would expect between two independent random masks with a keep\-density of 50%
### M\.6Dynamic subnetwork prediction and generalization

The framework ofdouble\-scoringalso suggests a natural meta\-learning problem\. Given a dataset𝒟\\mathcal\{D\}and a fixed frozen network, thedouble\-scoringprocedure produces a task\-dependent mask \(or score tensor\) adapted to𝒟\\mathcal\{D\}\. Repeating this process across many related tasks therefore yields a collection of input–output pairs of the form

𝒟⟼H𝒟,\\mathcal\{D\}\\longmapsto H\_\{\\mathcal\{D\}\},whereH𝒟H\_\{\\mathcal\{D\}\}denotes a mask extracted from the frozen base model using the data in𝒟\\mathcal\{D\}\.

This makes it possible to train an auxiliary predictor, or*router*, that takes as input a representation of the dataset and outputs a predicted mask \(or some lower dimensional representation thereof\) directly\. In this way, iterative mask optimization may be replaced by a learned one\-shot approximation\. If the new task is sufficiently similar to those seen during training, one may hope that the predicted mask is already close to a high\-performing subnetwork, thereby dramatically reducing or even eliminating the need for task\-specific mask search\.

Conceptually, this turns mask extraction into a supervised learning problem at the meta\-level: the original optimization procedure is used to generate labels, and a second model is trained to imitate its outputs\. The resulting mechanism may be viewed as an amortized version of strong lottery ticket extraction, in which experience on previous tasks is distilled into a fast predictor of subnetworks\. This perspective is especially appealing in settings where many related tasks must be solved on a shared frozen architecture\. Rather than rerunningdouble\-scoringindependently for each new dataset, one may instead learn a mapping from task statistics, support examples, or low\-dimensional summaries of the data to the corresponding mask\. In principle, this would provide a form of one\-shot or few\-shot subnetwork selection driven directly by the dataset itself\.

In line with the observations of[Section˜M\.5](https://arxiv.org/html/2607.20555#A13.SS5), for a fixed randomly initialized network, the masks obtained via thedouble\-scoringprocedure exhibit substantial structural overlap across related tasks\. This suggests that the space of effective masks is highly redundant: many masks yield comparable input\-output behavior, and the set of “good” masks occupies a low\-dimensional manifold within the ambient high\-dimensional binary space\. To exploit this structure, we show in[Figure˜24](https://arxiv.org/html/2607.20555#A13.F24)that this ’mask learning’ is possible via an experiment with two\-stages\. First, we generate a dataset of masks by applying thedouble\-scoringalgorithm to a collection of tasks drawn from a common distribution\. We then train a mask autoencoder to compress these binary masks into a low\-dimensional latent representation and reconstruct them with high fidelity\. Empirically, we observe that relatively small latent dimensions suffice to capture the salient structure of the mask distribution, indicating that the effective degrees of freedom are far smaller than the total number of parameters\. In the second stage, we train a*router*network that maps a representation of a task \(in our experiments, simple summary statistics of the dataset\) directly to the latent code of a corresponding mask\. The decoder from the autoencoder is then used to lift this latent code back into a full mask, which is applied to the frozen network\. This enables a one\-shot prediction of a high\-quality subnetwork without performing iterative score optimization\. A key feature of this approach is that it avoids directly predicting masks in the original parameter space, which is prohibitively high\-dimensional\. Instead, the router operates in the learned latent space, where the geometry of the mask distribution is significantly simpler\. This not only reduces the complexity of the prediction problem, but also implicitly captures equivalence classes of masks that induce similar functions\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.11.1.1.png)

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.11.1.2.png)

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.11.1.3.png)

Figure 24:A data set of doubled mask popup scores was generated for a very small test network usingdouble\-scoringfor a variety of scaled sine curve approximations\. Using a router and the decode head of an autoencoder trained on the data set of scores, a model whose popup scores were predicted using this router was tested on a variety of new scaled sine curve approximations, with the results compared to the performance of masks obtained viadouble\-scoring\. Top: Aligning the results to a common sine curve, we show the routed model’s approximations on the left and thedouble\-scoringapproximations on the right\. The approximations are only shown over their native domain after alignment, which occasionally produces some clipping\. Bottom Left: The average over all commonly\-aligned approximations for both the routed model and the extracteddouble\-scoringmasking model, with interpolation for out\-of\-domain values in the mean calculation\. Bottom right: A histogram of the routed model’s test loss and thedouble\-scoringmasking models’ test loss\. Throughout, it can be seen that score routers allow for generalization to unseen tasks, albeit at the cost of some degradation to performance\.In this toy example, we find that the router is able to produce masks whose performance is generally competitive with those obtained via fulledge\-popupoptimization, despite requiring no iterative training at test time\. This suggests, as mentioned above, that mask extraction admits a form of amortization: once a sufficient collection of tasks has been processed, the cost of finding a good mask for a new task can be reduced to a single forward pass through the router\. This perspective opens several directions\. First, it provides a natural mechanism for transfer learning: knowledge of previously solved tasks is distilled into the latent representation of masks and reused for new tasks\. Second, it suggests the possibility of large\-scale parallelization, where many agents independently generate masks for related tasks and contribute to a shared latent model\. Finally, it raises the question of whether the latent space admits further structure\.

### M\.7Iterated masking and implicit depth

We explore an extension of thedouble\-scoringframework motivated by the observation that, for layers with square weight matrices, one can simulate increased depth by repeatedly applying a masked linear operator interleaved with nonlinearities\. Concretely, givenW∈ℝd×dW\\in\\mathbb\{R\}^\{d\\times d\}and masksH\(1\),…,H\(T\)H^\{\(1\)\},\\dots,H^\{\(T\)\}, this yields the composition

x↦σ​\(\(W⊙H\(T\)\)​σ​\(⋯​σ​\(\(W⊙H\(1\)\)​x\)​⋯\)\),x\\mapsto\\sigma\\big\(\(W\\odot H^\{\(T\)\}\)\\,\\sigma\\big\(\\cdots\\sigma\\big\(\(W\\odot H^\{\(1\)\}\)x\\big\)\\cdots\\big\)\\big\),allowing one to realize a depth\-TTnonlinear network using a single underlying weight tensor\.

A naive implementation assigns an independent mask to each virtual layer\. While this increases expressivity in principle, it inherits the standard optimization challenges of deep compositions—most notably, the degradation of gradients across repeated nonlinear transformations\. In preliminary testing, we find that simply increasing this*implicit depth*yields only modest gains\. This suggests that the primary difficulty lies not in depth itself, but in*mask sequence selection*: determining which subnetworks should be applied, and in what order\.

This observation motivates a shift from fixed to*adaptive*compositions\. Rather than assigning masks to predetermined depths, we introduce a finite*vocabulary*of masked operators

W\(m\)=W⊙H\(m\),m=1,…,M,W^\{\(m\)\}=W\\odot H^\{\(m\)\},\\quad m=1,\\dots,M,and view each masked transformation as a reusable computational primitive\. The learning problem then becomes one of dynamically selecting and composing these primitives \(a perspective generally in line with the spirit of the Kolmogorov\-Arnold representation theorem, in which functions are represented by sums of compositions of a finite vocabulary of primitives\)\. While one could consider finer\-grained vocabularies \(e\.g\., blockwise masks or low\-dimensional mask parameterizations\), we focus here on full\-mask primitives and leave such extensions to future work\.

This viewpoint admits a natural analogy to spiking neural networks, where neuronal activity is inherently event\-driven: whether a neuron fires depends on its prior activation history and the temporal structure of incoming signals\. From this perspective, spiking dynamics can be interpreted as inducing a time\-varying mask over the network, with the active subnetwork evolving as a function of past activity\. In this sense, spiking systems implicitly perform history\-dependent mask selection, suggesting that autoregressive mask composition may provide a useful abstraction for adaptive, input\-dependent computation\.

#### M\.7\.1Autoregressive mask selection via a transformer

We formalize mask selection as an autoregressive sequence modeling problem over this vocabulary\. Fix weight and bias mask pairs\(H\(1\),h\(1\)\),…,\(H\(M\),h\(m\)\)\(H^\{\(1\)\},h^\{\(1\)\}\),\\dots,\(H^\{\(M\)\},h^\{\(m\)\}\)together with a distinguished termination token<END\>\. A computation is represented by a sequence

m1,m2,…,mT,<END\>,mt∈\{1,…,M\}\.m\_\{1\},m\_\{2\},\\dots,m\_\{T\},\\texttt\{<END\>\},\\quad m\_\{t\}\\in\\\{1,\\dots,M\\\}\.At each step, a transformer predicts the next token conditioned on the previously selected masks \(and optionally the input\)\. If<END\>is emitted, computation terminates; otherwise, selectingmim\_\{i\}updates the hidden state via

h↦σ​\(\(W⊙H\(mi\)\)​h\+b⊙h\(mi\)\)\.h\\mapsto\\sigma\\big\(\(W\\odot H^\{\(m\_\{i\}\)\}\)h\+b\\odot h^\{\(m\_\{i\}\)\}\\big\)\.
This formulation explicitly separates*what*computations are available \(the mask vocabulary\) from*how*they are composed \(the autoregressive policy\)\. Unlike naive iterated masking, the next operation depends on the entire history of prior selections, enabling richer and more flexible control over computation\. The resulting architecture can be interpreted as a sequence model over internal computational steps, where each token corresponds to the application of a masked subnetwork\.

A key consequence of this formulation is a natural mechanism for*adaptive depth*: computation proceeds until the model emits<END\>, allowing termination to depend on the input\. More broadly, the model can be viewed as generating a sequence of internal “thoughts,” where each step refines the representation via a learned computational primitive\. In essence, we create a vocabulary of computational primitives and learn a semantic and syntactic structure of this internal ’language of thought’ adapted to the problem at hand\.

We evaluate the proposed transformer\-based mask\-program model against a classical multilayer perceptron on the*scikit\-learn digits*classification task \([Figure˜25](https://arxiv.org/html/2607.20555#A13.F25)\)\. To ensure a fair comparison, we match model capacity by setting the depth of the classical network equal to the average realized program length of the learned model\. The program model applies a sequence of masked linear operators drawn from the learned vocabulary, together with a differentiable halting mechanism that determines when to terminate computation\. Training dynamics reveal that both models achieve comparable predictive performance; however, the program model exhibits substantially richer internal behavior\. In particular, the learned halting mechanism induces a nontrivial distribution over computation lengths, as evidenced by the divergence between the hard \(realized\) and soft \(expected\) program lengths and the evolution of the halting probability during training\. Additionally, our model does not seem to suffer from overfitting in the same way as a classical model when overtrained, as evidenced by a comparison of the validation loss\. We believe this may be due to the fact that the model is not only learning the classification task, but also a much more difficult high\-dimensional task of navigating in the ’thought space’ as it generates programs\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.12.1.1.png)

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.12.1.2.png)

Figure 25:We train a transformer\-based policy that generates sequences of mask tokens \(“programs”\) applied to a fixed randomly initialized network viadouble\-scoring\. For each input from the SKLearn digits dataset, the model autoregressively produces a variable\-length program which transforms a shared hidden representation before classification\. Unlike a standard model of the same capacity, this approach does not seem to suffer from overfitting \(left\)\. Over time, the length of programs increases \(right\)\.Further diagnostics \([Figure˜26](https://arxiv.org/html/2607.20555#A13.F26)\) show that the model utilizes a diverse set of computational primitives: the token usage distribution and associated entropy indicate that multiple masks are actively employed, rather than collapsing to a single dominant operation\. Finally, the distribution of programs across true class labels highlights that the model allocates computational strategies adaptively across inputs, providing empirical support for interpreting the architecture as a learned, input\-dependent computation process rather than a fixed\-depth network\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.12.2.1.png)

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.12.2.2.png)

Figure 26:For each true class, the most frequently generated programs \(top\-kkper class\) are shown with color indicating the within\-class frequency of each program\. Distinct classes exhibit concentrated mass on different subsets of programs, indicating that the model learns class\-dependent computational routines rather than relying on a single shared transformation\. At the same time, partial overlap across classes reveals reuse of subroutines, suggesting that the learned mask vocabulary supports compositional structure across tasks\.

### M\.8Learning initialization distributions and neonatal locomotion in ungulates

Organisms exhibit innate, complex behaviors \(e\.g\., neonatal locomotion in ungulates\) despite genomic limitations\. The horse \(Equus caballus\) genome \(≈2\.7\\approx 2\.7Gb diploid\) encodes far less information than required to explicitly specify trillions of neural connections\. Yet, foals achieve coordinated sensorimotor control within minutes of birth, integrating proprioceptive, cardiovascular, and environmental inputs\. This implies evolutionary optimization of developmental rules and not static parameters to generate brain architectures predisposed to critical functions\.

Such meta\-learning mechanisms, whereby genetic information biases connection probability distributions, implies anevolutionary\-developmental paradigm, wherein biological learning integrates evolutionary optimization of developmental rules with task\-specific synaptic refinement\. Unlike the training of ANNs, which globally optimize parameters, biological systems employ sparse, context\-dependent activation patterns \(e\.g\., localized neural ensembles vs\. the epileptiform “seizure\-like” global activity of ANNs at inference\)\. Evolutionary processes may select for optimized initialization rules; essentially, for genetic programs that bias neural development toward configurations pre\-disposed to critical functions \(e\.g\., breathing, locomotion\), thereby producing redundant, task\-competent neural components enabling both efficiency and robustness\. Translating this principle to machine learning, a framework centered on strong lottery tickets seems automatic: One should learn initialization distributions via evolutionary mechanisms that produce many easy\-to\-find copies of subnetworks that, at initialization, already encode the task\-specific capabilities one desires in the model\.

Having now the ability to extract strong lottery tickets reliably, this framework becomes possible to explore\. Specifically, we propose to replace the standard paradigm of fixed, hand\-designed initialization schemes with a learned family of initialization distributions, optimized to generate many easily\-discoverable subnetworks accomplishing a family of desired tasks\. To this end, we consider a simple parameterization of layerwise initialization distributions\. Rather than directly initializing network weights from a fixed distribution \(e\.g\., Gaussian or Kaiming\), we define a family of spatially\-varying Gaussian fields over each weight matrix\. Concretely, for each layer, the mean and log\-variance of the weight distribution are parameterized as low\-dimensional functions of normalized neuron indices\(i,j\)\(i,j\), yielding position\-dependent initialization statistics\. These parameters collectively define an*initialization genome*for the network\.

We then place this genome under evolutionary optimization\. Each individual in the population corresponds to a distinct initialization distribution\. Given such an individual, a network is instantiated by sampling weights from the induced distribution, after which only a small number of iterations ofdouble\-scoringare performed on the task\. The fitness of the individual is defined as test loss after this short training trajectory\. In this way, evolution does not optimize weights directly, but instead selects for initialization distributions that reliably produce subnetworksdouble\-scoringis capable of rapidly discovering for the task\.

Our numerical experiment is designed to probe this phenomenon in a controlled setting\. We consider a simple handwritten digit identification task, using the SK\-Learn digits dataset and a shallow fully\-connected network\. The evolutionary search is conducted over a population of initialization genomes, each encoding layerwise distribution parameters as described above\. At each generation, individuals are evaluated by instantiating networks from their corresponding distributions,[Algorithm˜1](https://arxiv.org/html/2607.20555#alg1)is performed for a fixed and small number of steps, and the resulting losses are recorded\. Selection, crossover, and mutation are then applied to produce the next generation\. Importantly, mutation operates directly on the parameters of the initialization distribution, thereby exploring the space of developmental rules rather than trained weights\. This provides preliminary empirical support for the evolutionary\-developmental paradigm\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.13.1.1.png)

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.13.1.4.png)

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.13.1.2.png)

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.13.1.3.png)

Figure 27:Top: The best\-achieved loss over generations and the distribution of training trajectories induced by each initialization genome\. Center: Training dynamics withdouble\-scoringfor an individual initialized from the best genome in the final generation relative to a random baseline\. Bottom: The qualitative properties of the resulting weight distributions\.This procedure can be interpreted as a form of zero\-shot architectural biasing: rather than learning weights that solve the task, we learn distributions that make solutions*easy to find*via minimal application ofdouble\-scoring\([Algorithm˜1](https://arxiv.org/html/2607.20555#alg1)\)\. This procedure effectively shifts learning from weight space to distribution space: we do not learnsolutions, but ratherlaws that make solutions abundant\.

### M\.9Zero data training with ‘dreams’

A central advantage of thedouble\-scoringframework is that it enables a form of data\-independent pretraining\. In contrast to standard learning paradigms, where performance is tightly coupled to the availability of labeled data, the present approach allows the model to improve its performance without ever observing samples from the target task distribution\.

The key mechanism underlying this phenomenon is what we refer to as dreaming\. Rather than training on external data, the model instead samples from its own hypothesis class\. Concretely, given a randomly initialized network with frozen weights, we generate behaviors by selecting random subnetworks \(via masking\) and evaluating them on randomly chosen probe inputs\. These input–output pairs form internally generated “dreams,” which are guaranteed to lie within the representational capacity of the model\. Crucially, we do not store these subnetworks directly\. Instead, we use thedouble\-scoringprocedure to relearn masks that approximate the generated behaviors \(as the randomly induced masks may not be easily reachable from a random score initialization\), thereby constructing a library of subnetworks indexed by their functional behavior on probe sets\.

This process yields a collection of input–output examples paired with masks, which can be used to train a routing mechanism\. The router learns to map small sets of probe examples to subnetworks capable of reproducing similar behavior\. Importantly, this entire pipeline is constructed without reference to any external task distribution\. The model is, in effect, learning to organize and index its own functional capabilities\.

To evaluate this framework, we consider \([Figure˜28](https://arxiv.org/html/2607.20555#A13.F28)\) a toy family of downstream tasks \(e\.g\., parameterized sine functions\) that are never observed during the dream\-generation or router\-training phases\. At test time, the model is provided with a small probe set from the target task, and must select an appropriate subnetwork via retrieval or routing\. We compare this performance against a baseline consisting of randomly initialized classical neural networks of comparable size, evaluated without task\-specific training\.

![Refer to caption](https://arxiv.org/html/2607.20555v1/figures/fig_5.14.1.png)Figure 28:Performance of dream\-based subnet selection compared to a random classical baseline\. For each task, the boxplot shows the distribution of evaluation errors across randomly initialized dense networks of matching scale, while points denote the performance of subnetworks selected via retrieval and routing from the dream library\. In a substantial fraction of tasks, the dream\-based methods lie below the bulk of the random baseline distribution, often outperforming even strong random draws\. This demonstrates that the model, despite receiving no task\-specific training data, is able to systematically select subnetworks with above\-random performance\. The figure highlights the central phenomenon: dream\-based pretraining organizes the model’s internal function space in a way that enables meaningful generalization without exposure to the target task\.The results consistently show that the dream\-based approach outperforms the random baseline across tasks\. That is, despite never having been trained on data from the target task \(or even from any related distribution\) the model achieves lower evaluation error than what one would expect from a randomly sampled network of the same capacity\. This demonstrates a clear separation between data and performance that is, to us, genuinely surprising: The model’s ability to solve new tasks is not derived from exposure to those tasks, but ratherfrom an internal organization of its own representational structure\. In essence, the model need only know about itself on some level\. We suspect that this would work particularly well in a task\-dependent way when combined with the genetic initialization distribution selection process of[Section˜M\.8](https://arxiv.org/html/2607.20555#A13.SS8), allowing for the emergence of untrained and effectively ‘instinctual’ capabilities in a model \(analogously to how a beaver in captivity will begin to build dam\-like structures entirely unprompted and with no prior examples of such behavior\), but leave investigation of this for future work\.

From a conceptual standpoint, this suggests that the primary role of data in traditional training may be possible to circumvent to a small degree\. Thedouble\-scoringframework offers this alternative: by learning from a synthetic dataset generated via self\-sampling, one can partially bypass some of the need for task\-specific data\. The manner in which the gains to performance from such a procedure scale is unclear to the authors\.

### M\.10Speculation and outlook

The results presented in[Appendix˜M](https://arxiv.org/html/2607.20555#A13)suggest a perspective that, while highly speculative, may help frame a number of persistent discrepancies between artificial and biological learning systems\.

Modern machine learning methods are, at their core, global optimization procedures: parameters are adjusted directly, typically via gradient\-based methods, and the computational and energetic cost of training scales accordingly with model size\. In contrast, biological systems appear to operate under dramatically different constraints\. The human brain, for instance, contains on the order of101410^\{14\}synaptic connections, yet develops and learns under an energy budget that is negligible compared to what would be required to train an artificial system of comparable scale using conventional techniques\. That is, to train a large language model in any familiar architecture with a biologically\-comparable number parameters in its feed\-forward layers would require an energy budget that–crudely measured–would cost something in the neighborhood of the global GDP while, from birth to high school graduation, the total energy consumption of an average human brain is around what is used by a standard residential air conditioner in a single summer\.

Moreover, biological systems exhibit a degree of robustness that is difficult to reconcile with standard artificial models: localized damage, noise, or loss of connections often results in minimal functional degradation, whereas analogous perturbations in trained neural networks typically lead to significant performance loss\. In fact, many large language models contain single parameters \(so\-called ’superweights’\) that, if deleted, render the model unable to output coherent language entirely\[[23](https://arxiv.org/html/2607.20555#bib.bib126)\]\. Biologically, this would be the equivalent of bumping your head or sneezing somewhat forcefully and permanently losing the ability to communicate\.

A further discrepancy arises in the apparent efficiency of biological initialization\. Many organisms exhibit complex behaviors with little to no postnatal training\. The canonical example is that of a newborn ungulate, which is capable of coordinated locomotion within minutes of birth\. It is difficult to attribute such capabilities to a direct encoding of a fully specified control policy in the genome, given the severe informational constraints\. Instead, it suggests that what is inherited is not a single model, but rather a structured distribution over possible models, heavily biased toward those that are functionally effective\.

Taken together, these observations motivate the following hypothesis: biological learning after birth may rely less on direct parameter optimization and more on search within a highly structured space of pre\-configured subnetworks \(which, we note, combines both of the scalable approaches detailed in Sutton’sBitter Lesson\[[17](https://arxiv.org/html/2607.20555#bib.bib125)\]: Learning is conducted on a population’s initialization distribution and search is utilized within an individual after birth\)\. In this view, traditional learning via global parameter optimization is more akin to evolution, which acts as the primary optimizer to shape distributions over connectivity and local structure so that, at birth or after the final development of the brain post\-birth, the system already contains a vast number of viable "functional fragments\." Learning, then, consists primarily of searching through this space and discovering how to select and activate appropriate subsets of this structure in response to input, rather than constructing functionality from scratch\.

From this perspective, several qualitative features of biological systems become more natural\. Robustness arises from redundancy: many distinct subnetworks can perform similar functions, so damage to any particular subset has limited effect\. Efficiency arises because the search is constrained to a highly favorable region of the space\. Rapid acquisition of behavior is possible because the relevant structures are already present, requiring only selection rather than synthesis\.

By contrast, much of contemporary machine learning can be interpreted as optimizing what is, in effect, a “seizure mode” of computation\. That is, biological brains almost never engage anything near their full parameter set simultaneously \(barring, as noted, seizures\) and prefer highly localized refinements after maturity\. While the existing paradigm has proven extraordinarily powerful, it may also represent a particularly inefficient point in the design space\. That is, speaking broadly, what we have done so far in the study of artificial intelligence is focus essentially on the evolutionary side of things by selecting the distribution of parameters for our models with training\. Training with a particular data set is effectively just a roundabout way of accessing a probability distribution on the model’s parameters; if one initialized a model according to this distribution directly without training, it would be fundamentally the same\. We propose that this is effectively what evolution does, i\.e\., evolutionary processes slowly optimize brain development\-related parameters that encode a distributions of likely models\. What we have essentially donein silicois show that, under the right conditions, an analog of this evolutionary force is enough to make ’brains’ whose seizures are capable of performing a wide range of tasks\. All the same, optimizing seizures is also probably one of thehardest possible ways we could have chosen to do things\. Nature doesn’t optimize seizures to perform well on many different tasks; nature is much more fine grained than that in its optimizations\. Biological brains use different chunks for different things\. Nature, essentially, ignores the seizure mode of operation entirely\.

The framework explored in[Appendix˜M](https://arxiv.org/html/2607.20555#A13)can be viewed as a suggestion for an alternative: rather than learning a single set of parameters, we instead study systems in which functionality emerges from selecting among many latent configurations packaged densely together\.