提取与蒸馏相遇——从深度神经网络中恢复参数的密码分析技术

# Extraction meets Distillation – cryptanalytic techniques to recover parameters from a Deep Neural Network
Source: [https://arxiv.org/html/2610.33671](https://arxiv.org/html/2610.33671)
MSC Classification\]94A60, 68T10

This paper significantly expands the conference paper accepted for CSCML 2026\. The last\-named author joined the collaboration after the conference version’s acceptance, contributing to the mathematical analysis and theoretical development of the current version\. This is presently a working draft towards a possible journal submission\.Arya Anshuman OjhaEmail:[kalingasandhababun@gmail\.com](mailto:[email protected])Affiliation:National Institute of Science Education and Research, Bhubaneswar, IndiaYubaraj DasEmail:[yyubarajddas9876@gmail\.com](mailto:[email protected])Affiliation:Jadavpur University, Kolkata, IndiaDevansh KhoslaEmail:[devanshkhosla289@gmail\.com](mailto:[email protected])Affiliation:Institute of Mathematics and Application, Bhubaneswar, IndiaKamran RaisEmail:[kamranrais41@gmail\.com](mailto:[email protected])Affiliation:Institute of Mathematics and Application, Bhubaneswar, IndiaSaankhya SamantaEmail:[saankhya\.samanta99@gmail\.com](mailto:[email protected])Affiliation:Indian Institute of Technology, Kharagpur, IndiaSubhamoy MaitraEmail:[subho@isical\.ac\.in, maitra\.subhamoy@gmail\.com](mailto:[email protected],%[email protected])Affiliation:Indian Statistical Institute, Kolkata, IndiaPantelimon StănicăEmail:[pstanica@nps\.edu, pstanica@gmail\.com](mailto:[email protected],%[email protected])Affiliation:Naval Postgraduate School, Applied Mathematics Department, Monterey, CA 93943, USA

###### Abstract

We study fully connected neural\-network reconstruction by combining cryptanalytic weight\-direction extraction with learning from predicted labels\. We adapt a ReLU extraction implementation to Leaky ReLU, compare sign\-search methods, and estimate biases and output layers\. We prove conditional bounds for direction recovery with approximate prefixes, preservation of local distance votes, prediction changes from aligned parameter errors, and sign selection with independent scoring data\. We also specify exact row restoration and distinguish its guarantee from that of gradient masking\. The experiments evaluate assisted extraction, with privileged access to gradients, hidden activations, and victim parameters, followed by label\-based reconstruction\. Across eight search seeds on the largest synthetic architecture, Leaky ReLU recovers 51–57 neurons in the deepest hidden layer, while ReLU recovers none\. This advantage does not hold uniformly across architectures and layers\. Removing victim\-weight access from the sign walk gives less accurate signs, despite high final prediction agreement\. Across 40 seeded runs on the two smallest synthetic victims, agreement on 5,000 inputs excluded from fitting and selection ranges from0\.98900\.9890to0\.99920\.9992\. On two CIFAR\-10 victims, the reconstructed models achieve56\.40%56\.40\\%and58\.29%58\.29\\%agreement and exceed the distillation baseline on three functional metrics\. We report label requests separately from privileged access and explain why prediction agreement does not identify network parameters\.

###### keywords

Cryptanalysis, Machine Learning, Deep Neural Networks \(DNNs\), Parameter Extraction, Rectified Linear Unit \(ReLU\), Leaky ReLU\.

## 1Introduction

A model extraction attack uses access to a trained model to construct another model with similar predictions or parameters\. These objectives differ\. A model can agree with a classifier on a test set without recovering its weights, and a collection of accurate weight directions need not produce a useful classifier\. We study how to assemble partially recovered directions into a complete network while retaining those directions during subsequent training\.

Cryptanalytic extraction uses the piecewise\-affine structure of networks with rectified linear activations\. Carlini et al\.\[[5](https://arxiv.org/html/2610.33671#bib.bib5)\]developed an extraction method using numerical network outputs\. Canales\-Martínez et al\.\[[3](https://arxiv.org/html/2610.33671#bib.bib3)\]improved the recovery procedures, including sign recovery\. Chen et al\.\[[9](https://arxiv.org/html/2610.33671#bib.bib9)\]and Carlini et al\.\[[6](https://arxiv.org/html/2610.33671#bib.bib6)\]studied extraction from predicted labels\. RayS and HopSkipJumpAttack\[[7](https://arxiv.org/html/2610.33671#bib.bib7),[8](https://arxiv.org/html/2610.33671#bib.bib8)\]are related decision\-based adversarial attacks\. They find inputs that change a prediction, rather than recover the model parameters\. Liu et al\.\[[17](https://arxiv.org/html/2610.33671#bib.bib17)\]studied end\-to\-end extraction with numerical outputs\. Qi et al\.\[[20](https://arxiv.org/html/2610.33671#bib.bib20)\]considered several activation functions, including Leaky ReLU\. We do not claim that extending extraction beyond ReLU is new by itself\.

We distinguish five output interfaces, namely the predicted label, the label and its probability, the top\-kkprobabilities, the full probability vector, and the raw logits\. A label fromKKclasses carries at mostlog2⁡K\\log\_\{2\}Kbits\. This is an upper bound on the information in one response, not a parameter\-recovery lower bound\. Numerical outputs provide additional constraints, but they do not make exact recovery of a deep network an ordinary linear regression problem\. Our intended oracle returns only a predicted label\.

Our implementation builds on the code of Carlini et al\.\[[6](https://arxiv.org/html/2610.33671#bib.bib6)\]\. Several extraction stages read information unavailable through that oracle, including gradients, hidden activations, and true prefix weights\. We disclose these accesses in Section[3\.4](https://arxiv.org/html/2610.33671#S3.SS4)\. We therefore report the experiments as assisted extraction and reconstruction\. The later learning stages use cached victim labels, but this does not remove the privileged information used to construct their initial parameters\.

In this paper, we combine three procedures\. First, we adapt the local slope calculations to Leaky ReLU\. We evaluate accepted directions across recorded search seeds, subject to the ground\-truth acceptance procedure described below\. The advantage depends on architecture and layer\. Second, we compare simulated annealing and parallel tempering for sign search using cached labels and the reconstructed model’s logits\. Third, we estimate biases from dual points, fit an output layer, and refine the network with recovered\-row gradient masks\. We state separately how row restoration enforces exact equality\. We also batch the dual\-point search and stream the clustering data\. Our implementation and experimental scripts are listed in\[[1](https://arxiv.org/html/2610.33671#bib.bib1)\]\.

Section[2](https://arxiv.org/html/2610.33671#S2)defines the notation and the relevant identification ambiguities\. Section[3](https://arxiv.org/html/2610.33671#S3)describes the extraction stages and their privileged accesses\. Sections[4](https://arxiv.org/html/2610.33671#S4),[5](https://arxiv.org/html/2610.33671#S5), and[6](https://arxiv.org/html/2610.33671#S6)describe the Leaky ReLU adaptation, sign search, and reconstruction\. Section[7](https://arxiv.org/html/2610.33671#S7)reports the experiments\. We compare functional agreement separately from the reported direction coverage, because agreement alone does not establish parameter recovery\.

## 2Preliminaries and Notation

We writeh0​\(x\)=xh\_\{0\}\(x\)=xand define a network withrrhidden layers by

ηi​\(x\)=A\(i\)​hi−1​\(x\)\+𝒃\(i\),\\displaystyle\\eta\_\{i\}\(x\)=A^\{\(i\)\}h\_\{i\-1\}\(x\)\+\\boldsymbol\{b\}^\{\(i\)\},hi​\(x\)=σ⁡\(ηi​\(x\)\)\(1≤i≤r\),\\displaystyle h\_\{i\}\(x\)=\\sigma\(\\eta\_\{i\}\(x\)\)\\quad\(1\\leq i\\leq r\),fθ​\(x\)=A\(r\+1\)​hr​\(x\)\+𝒃\(r\+1\)\.\\displaystyle f\_\{\\theta\}\(x\)=A^\{\(r\+1\)\}h\_\{r\}\(x\)\+\\boldsymbol\{b\}^\{\(r\+1\)\}\.HereA\(i\)∈ℝdi×di−1A^\{\(i\)\}\\in\\mathbb\{R\}^\{d\_\{i\}\\times d\_\{i\-1\}\},𝒃\(i\)∈ℝdi\\boldsymbol\{b\}^\{\(i\)\}\\in\\mathbb\{R\}^\{d\_\{i\}\}, anddr\+1=Kd\_\{r\+1\}=Kis the number of classes\. The activation acts on each coordinate\. We write‖v‖2\\\|v\\\|\_\{2\}and‖v‖∞\\\|v\\\|\_\{\\infty\}for the Euclidean and maximum norms\. For matrices,‖A‖2\\\|A\\\|\_\{2\}denotes the induced Euclidean operator norm\. We also define‖A‖2→∞=maxj⁡‖Aj‖2\\\|A\\\|\_\{2\\to\\infty\}=\\max\_\{j\}\\\|A\_\{j\}\\\|\_\{2\}, the norm from Euclidean input to maximum output\. HereAjA\_\{j\}is rowjjofAA\. We define‖A‖F=\(∑j,kAj​k2\)1/2\\\|A\\\|\_\{F\}=\(\\sum\_\{j,k\}A\_\{jk\}^\{2\}\)^\{1/2\}as the Frobenius norm\. We write⟨u,v⟩=u𝖳​v\\langle u,v\\rangle=u^\{\\mathsf\{T\}\}vfor the Euclidean inner product,V⟂V^\{\\perp\}for the orthogonal complement of a subspaceVV, and𝟏​\[B\]\\mathbf\{1\}\[B\]for the indicator of a statementBB\. A unit vector has Euclidean norm one\. The singular values of a matrix withnncolumns are thennnonnegative square roots of the eigenvalues ofA𝖳​AA^\{\\mathsf\{T\}\}A, counted with multiplicity\. This convention includes zero values when the matrix has fewer thannnrows\. The kernel and rank of a matrix are its null space and the dimension of its image, respectively\. We writeAj\(i\)A\_\{j\}^\{\(i\)\}for rowjjofA\(i\)A^\{\(i\)\}, andηi,j\\eta\_\{i,j\}for coordinatejjofηi\\eta\_\{i\}\. The parameter tuple isθ=\(\(A\(i\),𝒃\(i\)\)\)i=1r\+1\\theta=\(\(A^\{\(i\)\},\\boldsymbol\{b\}^\{\(i\)\}\)\)\_\{i=1\}^\{r\+1\}\. The number of hidden neurons isN=∑i=1rdiN=\\sum\_\{i=1\}^\{r\}d\_\{i\}\.

We define the hard\-label oracle by

𝒪⁡\(x\)=arg⁡max1≤k≤K⁡\[fθ​\(x\)\]k,\\mathcal\{O\}\(x\)=\\arg\\max\_\{1\\leq k\\leq K\}\[f\_\{\\theta\}\(x\)\]\_\{k\},with a fixed rule for ties\. Replacing logitsfkf\_\{k\}by the softmax probabilitiesexp⁡\(fk\)/∑cexp⁡\(fc\)\\exp\(f\_\{k\}\)/\\sum\_\{c\}\\exp\(f\_\{c\}\)gives the same label\. The intended attacker knows the architecture and activation and may query𝒪\\mathcal\{O\}, but does not knowθ\\theta\.

We use ReLU and Leaky ReLU\[[19](https://arxiv.org/html/2610.33671#bib.bib19)\]\. For0≤α<10\\leq\\alpha<1, we write

σα​\(t\)=\{t,t\>0,α​t,t≤0\.\\sigma\_\{\\alpha\}\(t\)=\\begin\{cases\}t,&t\>0,\\\\ \\alpha t,&t\\leq 0\.\\end\{cases\}ReLU is the caseα=0\\alpha=0\. For0<α<10<\\alpha<1, this equalsmax⁡\(α​t,t\)\\max\(\\alpha t,t\)\. The restrictionα<1\\alpha<1is essential for the slope change used by the attack\. Both activations are 1\-Lipschitz, meaning that\|σα​\(u\)−σα​\(v\)\|≤\|u−v\|\|\\sigma\_\{\\alpha\}\(u\)\-\\sigma\_\{\\alpha\}\(v\)\|\\leq\|u\-v\|\. This follows from their piecewise slopes in\[0,1\]\[0,1\], including when the segment crosses zero\. Applying the inequality coordinatewise gives the same property in Euclidean norm\. In general, a mapggisLL\-Lipschitz from one norm to another when‖g⁡\(u\)−g⁡\(v\)‖out≤L​‖u−v‖in\\\|g\(u\)\-g\(v\)\\\|\_\{\\mathrm\{out\}\}\\leq L\\\|u\-v\\\|\_\{\\mathrm\{in\}\}for all inputs under consideration\.

An activation pattern records which hidden pre\-activations are positive\. We assign zero pre\-activations to the nonpositive state\. On a region with fixed patternρ\\rho, define the diagonal matrixDi​\(ρ\)D\_\{i\}\(\\rho\)to have entry11at a positive pre\-activation andα\\alphaotherwise\. This convention givesσα​\(ηi\)=Di​\(ρ\)​ηi\\sigma\_\{\\alpha\}\(\\eta\_\{i\}\)=D\_\{i\}\(\\rho\)\\eta\_\{i\}also when a coordinate vanishes\. The network is affine on that region\. We write it asMρ​x\+cρM\_\{\\rho\}x\+c\_\{\\rho\}, where

Mρ=A\(r\+1\)Dr\(ρ\)A\(r\)⋯D1\(ρ\)A\(1\)\.M\_\{\\rho\}=A^\{\(r\+1\)\}D\_\{r\}\(\\rho\)A^\{\(r\)\}\\cdots D\_\{1\}\(\\rho\)A^\{\(1\)\}\.Empty patterns are omitted\. The closures of the nonempty regions cover the input space and may overlap on their boundaries\.

We define the critical set of neuron\(i,j\)\(i,j\)byℋi,j=\{x:ηi,j​\(x\)=0\}\\mathcal\{H\}\_\{i,j\}=\\\{x:\\eta\_\{i,j\}\(x\)=0\\\}\. Fori=1i=1, this is a hyperplane ifAj\(1\)≠0A\_\{j\}^\{\(1\)\}\\neq 0\. For deeper layers, its intersection with a fixed prefix region is the zero set of an affine function restricted to that region\. This intersection can be empty, the whole region, or a piece of a hyperplane\. In the first layer, we setA~j\(1\)=Aj\(1\)\\widetilde\{A\}\_\{j\}^\{\(1\)\}=A\_\{j\}^\{\(1\)\}\. Fori\>1i\>1, the effective normal within a fixed prefix region is

A~j\(i\)=Aj\(i\)Di−1A\(i−1\)⋯D1A\(1\)\.\\widetilde\{A\}\_\{j\}^\{\(i\)\}=A\_\{j\}^\{\(i\)\}D\_\{i\-1\}A^\{\(i\-1\)\}\\cdots D\_\{1\}A^\{\(1\)\}\.This normal may vanish\. The term*critical hyperplane*refers to a nondegenerate local piece of the critical set\.

For distinct classesm,nm,n, we define

𝒟m,n=\{x:\[fθ​\(x\)\]m=\[fθ​\(x\)\]n=maxk⁡\[fθ​\(x\)\]k\}\.\\mathcal\{D\}\_\{m,n\}=\\\{x:\[f\_\{\\theta\}\(x\)\]\_\{m\}=\[f\_\{\\theta\}\(x\)\]\_\{n\}=\\max\_\{k\}\[f\_\{\\theta\}\(x\)\]\_\{k\}\\\}\.A regular piece of this set is a decision boundary\. It is locally a hyperplane when the corresponding logit difference has a nonzero gradient\. A*dual point*for neuron\(i,j\)\(i,j\)lies inℋi,j∩𝒟m,n\\mathcal\{H\}\_\{i,j\}\\cap\\mathcal\{D\}\_\{m,n\}\. For the geometric recovery procedure, we additionally require that only this neuron changes state, that the two classes remain the relevant competitors, and that the intersection is transverse, meaning that its critical normal and decision\-boundary normal are linearly independent\. We call the local intersection the dual space\. Its tangent space consists of the infinitesimal directions parallel to both local hyperplanes, which we use in the rank calculations\. Under these assumptions, the dual tangent space has codimension two\. The Jacobian of a vector\-valued map is its matrix of first partial derivatives on the relevant affine region\. A critical crossing need not create a visible decision\-boundary bend\.

Hard labels do not identify a unique parameter tuple\. Hidden\-neuron permutations preserve the network function\. Positive homogeneity,σα​\(c​t\)=c​σα​\(t\)\\sigma\_\{\\alpha\}\(ct\)=c\\sigma\_\{\\alpha\}\(t\)forc\>0c\>0, gives another ambiguity\. Multiplying an incoming weight row and its bias byccand dividing the corresponding outgoing column byccpreserves all logits\. Moreover, replacing the output layer by

A\(r\+1\)⟼λ​A\(r\+1\)\+𝟏​v𝖳,𝒃\(r\+1\)⟼λ​𝒃\(r\+1\)\+β​𝟏,λ\>0,A^\{\(r\+1\)\}\\longmapsto\\lambda A^\{\(r\+1\)\}\+\\boldsymbol\{1\}v^\{\\mathsf\{T\}\},\\qquad\\boldsymbol\{b\}^\{\(r\+1\)\}\\longmapsto\\lambda\\boldsymbol\{b\}^\{\(r\+1\)\}\+\\beta\\boldsymbol\{1\},\\qquad\\lambda\>0,preserves every label\. Here𝟏∈ℝK\\boldsymbol\{1\}\\in\\mathbb\{R\}^\{K\},v∈ℝdrv\\in\\mathbb\{R\}^\{d\_\{r\}\}, andβ∈ℝ\\beta\\in\\mathbb\{R\}\. Thus an output\-layer fit from labels is a classifier reconstruction, not a unique recovery of the victim’s output weights\. When we discuss recovered directions, we compare them after matching neurons and allowing a nonzero scalar\.

## 3The Extraction Pipeline

We extract hidden\-layer directions sequentially\. The three stages find dual points, cluster them to estimate directions, and assign signs\. Reconstruction then estimates biases and fits the remaining parameters\. We separate the geometric procedure from the privileged accesses in its current implementation\.

### 3\.1Dual Point Discovery

Following Carlini et al\.\[[6](https://arxiv.org/html/2610.33671#bib.bib6)\], we start with two inputs having different labels and bisect their connecting segment to locate a label transition\. We then estimate a local boundary direction and walk along the boundary, reprojecting after each step\. A change in its orientation provides a candidate dual point\. We store a triplet\(xleft,xdual,xright\)\(x\_\{\\mathrm\{left\}\},x\_\{\\mathrm\{dual\}\},x\_\{\\mathrm\{right\}\}\)consisting of the candidate and a boundary point on either side of it\. The left and right points are not inherently the positive and negative sides of the neuron\. That assignment requires sign information\.

This interpretation assumes a regular two\-class boundary and an isolated critical crossing\. A change in the competing classes, simultaneous neuron crossings, or numerical error can also change the estimated orientation\. Conversely, a critical crossing with zero downstream effect need not be visible\.

We advance multiple boundary walks in batches\. The batched implementation retains the search thresholds and step rules\. This reduces repeated evaluation overhead, but retaining those constants does not prove that sequential and parallel implementations generate identical samples\. Such a comparison would also require matched random streams, stopping rules, and numerical operations\. The implementation reads exact gradients and logit gaps at the points described in Section[3\.4](https://arxiv.org/html/2610.33671#S3.SS4)\.

### 3\.2Signature Recovery

We call a nonzero scalar multiplec​Aj\(i\)cA\_\{j\}^\{\(i\)\}of a neuron’s weight row its signature\. The sign ofccis unresolved at this stage\. A positive change of scale can be absorbed into the next layer only when the bias and outgoing column are transformed consistently\. Negating the incoming row and bias generally changes the network, although degenerate networks can have additional ambiguities\.

The geometric recovery method originates in Carlini et al\.\([6](https://arxiv.org/html/2610.33671#bib.bib6), Section 5\)\. We give its local linear\-algebra formulation to specify exactly what one dual point determines\. Writed=d0d=d\_\{0\}, and letx∗x^\{\*\}be an isolated dual point\. In a neighborhood ofx∗x^\{\*\}, suppose the target critical hyperplane has equationa𝖳​\(x−x∗\)=0a^\{\\mathsf\{T\}\}\(x\-x^\{\*\}\)=0\. Suppose also that the same two classes compete on both sides and that their affine logit differences have gradientsn−n\_\{\-\}andn\+n\_\{\+\}\. The calculation in Section[4\.1](https://arxiv.org/html/2610.33671#S4.SS1)givesn\+−n−=γ​an\_\{\+\}\-n\_\{\-\}=\\gamma afor a scalarγ\\gamma\.

###### Proposition 1\.

Assumeγ≠0\\gamma\\neq 0and thata,n−a,n\_\{\-\}are linearly independent\. The tangent space of the dual intersection is

T=ker⁡\(n−𝖳\)∩ker⁡\(n\+𝖳\)=ker⁡\(a𝖳\)∩ker⁡\(n−𝖳\)\.T=\\ker\(n\_\{\-\}^\{\\mathsf\{T\}\}\)\\cap\\ker\(n\_\{\+\}^\{\\mathsf\{T\}\}\)=\\ker\(a^\{\\mathsf\{T\}\}\)\\cap\\ker\(n\_\{\-\}^\{\\mathsf\{T\}\}\)\.It has dimensiond−2d\-2\. IfT1,…,TqT\_\{1\},\\ldots,T\_\{q\}are tangent spaces from the same first\-layer neuron andU=T1\+⋯\+TqU=T\_\{1\}\+\\cdots\+T\_\{q\}has dimensiond−1d\-1, thenU⟂=span⁡\(a\)U^\{\\perp\}=\\operatorname\{span\}\(a\)\. In particular, two distinct such tangent spaces suffice whend≥3d\\geq 3\.

###### Proof\.

A vectorvvsatisfiesn−𝖳​v=n\+𝖳​v=0n\_\{\-\}^\{\\mathsf\{T\}\}v=n\_\{\+\}^\{\\mathsf\{T\}\}v=0exactly when it satisfiesn−𝖳​v=0n\_\{\-\}^\{\\mathsf\{T\}\}v=0andγ​a𝖳​v=0\\gamma a^\{\\mathsf\{T\}\}v=0\. This proves the equality of the two kernels\. The two independent linear equations give dimensiond−2d\-2\.

EachTℓT\_\{\\ell\}lies inker⁡\(a𝖳\)\\ker\(a^\{\\mathsf\{T\}\}\), soUUlies in this\(d−1\)\(d\-1\)\-dimensional space\. Equality of dimensions givesU=ker⁡\(a𝖳\)U=\\ker\(a^\{\\mathsf\{T\}\}\), and taking orthogonal complements proves the claim\. For two distinct\(d−2\)\(d\-2\)\-dimensional subspaces of a common\(d−1\)\(d\-1\)\-dimensional space, their sum has dimension greater thand−2d\-2and at mostd−1d\-1\. It therefore has dimensiond−1d\-1\. ∎

A single dual point supplies a codimension\-two tangent space\. Its orthogonal complement is two\-dimensional and contains both the critical normal and a boundary normal\. It therefore does not generally determine the weight direction\. In dimension two, all these tangent spaces are zero, so the two\-point tangent argument is unavailable\. Point locations or other constraints can still provide information\. Nor do nonparallel neuron normals alone guarantee that different\-neuron clusters have full rank\. The rank test requires diversity among the boundary normals as well\.

For layeri\>1i\>1, letJℓJ\_\{\\ell\}be the Jacobian ofhi−1h\_\{i\-1\}in the prefix region containing dual pointxℓx\_\{\\ell\}\. Transport each tangent space by definingBℓ=Jℓ​Tℓ⊆ℝdi−1B\_\{\\ell\}=J\_\{\\ell\}T\_\{\\ell\}\\subseteq\\mathbb\{R\}^\{d\_\{i\-1\}\}\. Witha=Aj\(i\)​𝖳a=A\_\{j\}^\{\(i\)\\mathsf\{T\}\}, differentiating the zero pre\-activation along the dual intersection givesa𝖳​Jℓ​v=0a^\{\\mathsf\{T\}\}J\_\{\\ell\}v=0for everyv∈Tℓv\\in T\_\{\\ell\}\. ThusBℓ⊆ker⁡\(a𝖳\)B\_\{\\ell\}\\subseteq\\ker\(a^\{\\mathsf\{T\}\}\)\. If a matrixMMhas rows that span theBℓB\_\{\\ell\}and

rank⁡M=di−1−1,\\operatorname\{rank\}M=d\_\{i\-1\}\-1,\(1\)thenker⁡M=span⁡\(a\)\\ker M=\\operatorname\{span\}\(a\)\. This uses transported tangent vectors\. In general, pushing a normal throughJℓJ\_\{\\ell\}does not give the normal in hidden coordinates\.

Point locations provide additional affine constraints\. If the true prefix is known, writezℓ=hi−1​\(xℓ\)z\_\{\\ell\}=h\_\{i\-1\}\(x\_\{\\ell\}\)and form an augmented matrix with rows\(zℓ𝖳,1\)\(z\_\{\\ell\}^\{\\mathsf\{T\}\},1\)and\(u𝖳,0\)\(u^\{\\mathsf\{T\}\},0\)for transported tangent vectorsu∈Bℓu\\in B\_\{\\ell\}\. Its kernel contains\(Aj\(i\),bj\(i\)\)𝖳\(A\_\{j\}^\{\(i\)\},b\_\{j\}^\{\(i\)\}\)^\{\\mathsf\{T\}\}\. Rankdi−1d\_\{i\-1\}determines that affine parameter vector up to scale\. Equivalently, after fixing one pointz1z\_\{1\}, we may addzℓ−z1z\_\{\\ell\}\-z\_\{1\}to the tangent constraints\. These additional constraints can identify a direction even when tangent spaces alone are insufficient\. The empirical signature files nevertheless omit bias estimates, which we compute later from the reconstructed prefix\.

Numerically, we choose a unit vector minimizing the residual of the transported matrix\. The following result makes the effects of tangent error and prefix error explicit\. We compare prefix coordinates after fixing neuron permutations and positive scaling conventions\. Otherwise a coordinate mismatch can change the meaning of a recovered row\.

###### Proposition 2\.

Leta∈ℝna\\in\\mathbb\{R\}^\{n\}be a unit direction, withn≥2n\\geq 2\. For1≤ℓ≤m1\\leq\\ell\\leq m, letvℓ∈ℝd0v\_\{\\ell\}\\in\\mathbb\{R\}^\{d\_\{0\}\}be a unit dual\-tangent vector and letJℓ∈ℝn×d0J\_\{\\ell\}\\in\\mathbb\{R\}^\{n\\times d\_\{0\}\}be the exact prefix Jacobian\. FormMMwith rows\(Jℓ​vℓ\)𝖳\(J\_\{\\ell\}v\_\{\\ell\}\)^\{\\mathsf\{T\}\}\. SupposeM​a=0Ma=0andrank⁡M=n−1\\operatorname\{rank\}M=n\-1\. Writeδ\>0\\delta\>0for its smallest positive singular value\. Assume estimates satisfy

‖v^ℓ−vℓ‖2≤tℓ,‖J^ℓ−Jℓ‖2≤pℓ,‖Jℓ‖2≤Lℓ\.\\\|\\widehat\{v\}\_\{\\ell\}\-v\_\{\\ell\}\\\|\_\{2\}\\leq t\_\{\\ell\},\\qquad\\\|\\widehat\{J\}\_\{\\ell\}\-J\_\{\\ell\}\\\|\_\{2\}\\leq p\_\{\\ell\},\\qquad\\\|J\_\{\\ell\}\\\|\_\{2\}\\leq L\_\{\\ell\}\.FormM^\\widehat\{M\}with rows\(J^ℓ​v^ℓ\)𝖳\(\\widehat\{J\}\_\{\\ell\}\\widehat\{v\}\_\{\\ell\}\)^\{\\mathsf\{T\}\}and define

ε=\[∑ℓ=1m\(pℓ\+Lℓ​tℓ\+pℓ​tℓ\)2\]1/2\.\\varepsilon=\\left\[\\sum\_\{\\ell=1\}^\{m\}\(p\_\{\\ell\}\+L\_\{\\ell\}t\_\{\\ell\}\+p\_\{\\ell\}t\_\{\\ell\}\)^\{2\}\\right\]^\{1/2\}\.Ifa^\\widehat\{a\}minimizes‖M^​z‖2\\\|\\widehat\{M\}z\\\|\_\{2\}over unit vectorszz, then

1−⟨a^,a⟩2≤min⁡\{1,2​ε/δ\}\.\\sqrt\{1\-\\langle\\widehat\{a\},a\\rangle^\{2\}\}\\leq\\min\\\{1,2\\varepsilon/\\delta\\\}\.\(2\)The left side is the sine of the acute angle betweena^\\widehat\{a\}and the line spanned byaa\. Ifδ^\\widehat\{\\delta\}is the second\-smallest of thennsingular values ofM^\\widehat\{M\}andδ^\>ε\\widehat\{\\delta\}\>\\varepsilon, the denominatorδ\\deltamay be replaced byδ^−ε\\widehat\{\\delta\}\-\\varepsilon\.

###### Proof\.

We subtract one transported vector by writing

J^ℓ​v^ℓ−Jℓ​vℓ=\\displaystyle\\widehat\{J\}\_\{\\ell\}\\widehat\{v\}\_\{\\ell\}\-J\_\{\\ell\}v\_\{\\ell\}=\{\}\(J^ℓ−Jℓ\)​vℓ\+Jℓ​\(v^ℓ−vℓ\)\\displaystyle\(\\widehat\{J\}\_\{\\ell\}\-J\_\{\\ell\}\)v\_\{\\ell\}\+J\_\{\\ell\}\(\\widehat\{v\}\_\{\\ell\}\-v\_\{\\ell\}\)\+\(J^ℓ−Jℓ\)​\(v^ℓ−vℓ\)\.\\displaystyle\+\(\\widehat\{J\}\_\{\\ell\}\-J\_\{\\ell\}\)\(\\widehat\{v\}\_\{\\ell\}\-v\_\{\\ell\}\)\.The triangle inequality bounds its norm bypℓ\+Lℓ​tℓ\+pℓ​tℓp\_\{\\ell\}\+L\_\{\\ell\}t\_\{\\ell\}\+p\_\{\\ell\}t\_\{\\ell\}\. Summing the squared row bounds gives‖M^−M‖2≤‖M^−M‖F≤ε\\\|\\widehat\{M\}\-M\\\|\_\{2\}\\leq\\\|\\widehat\{M\}\-M\\\|\_\{F\}\\leq\\varepsilon\. Sinceaais an admissible unit vector andM​a=0Ma=0, minimality gives‖M^​a^‖2≤‖M^​a‖2≤ε\\\|\\widehat\{M\}\\widehat\{a\}\\\|\_\{2\}\\leq\\\|\\widehat\{M\}a\\\|\_\{2\}\\leq\\varepsilon\. Another triangle inequality gives‖M​a^‖2≤2​ε\\\|M\\widehat\{a\}\\\|\_\{2\}\\leq 2\\varepsilon\.

We writec=⟨a^,a⟩c=\\langle\\widehat\{a\},a\\rangleandz=a^−c​az=\\widehat\{a\}\-ca, soz⟂az\\perp a\. Sinceker⁡M=span⁡\(a\)\\ker M=\\operatorname\{span\}\(a\), the singular\-value decomposition gives‖M​z‖2≥δ​‖z‖2\\\|Mz\\\|\_\{2\}\\geq\\delta\\\|z\\\|\_\{2\}\. AlsoM​a^=M​zM\\widehat\{a\}=Mzand‖z‖2=1−c2\\\|z\\\|\_\{2\}=\\sqrt\{1\-c^\{2\}\}\. These facts prove \([2](https://arxiv.org/html/2610.33671#S3.E2)\)\.

For completeness, the singular\-value comparison follows from the variational formula

sk​\(B\)=minV⊆ℝndimV=n−k\+1⁡maxz∈V‖z‖2=1​‖B​z‖2,s1​\(B\)≥⋯≥sn​\(B\)\.s\_\{k\}\(B\)=\\min\_\{\\begin\{subarray\}\{c\}V\\subseteq\\mathbb\{R\}^\{n\}\\\\ \\dim V=n\-k\+1\\end\{subarray\}\}\\max\_\{\\begin\{subarray\}\{c\}z\\in V\\\\ \\\|z\\\|\_\{2\}=1\\end\{subarray\}\}\\\|Bz\\\|\_\{2\},\\qquad s\_\{1\}\(B\)\\geq\\cdots\\geq s\_\{n\}\(B\)\.The formula follows by expanding in right singular vectors\. Every subspace of the stated dimension intersects the span of the firstkksuch vectors, and choosing the span of vectorsk,…,nk,\\ldots,nattains the bound\. The inequality‖M^​z‖2≤‖M​z‖2\+ε\\\|\\widehat\{M\}z\\\|\_\{2\}\\leq\\\|Mz\\\|\_\{2\}\+\\varepsilontherefore givessk​\(M^\)≤sk​\(M\)\+εs\_\{k\}\(\\widehat\{M\}\)\\leq s\_\{k\}\(M\)\+\\varepsilon\. Takingk=n−1k=n\-1yieldsδ≥δ^−ε\\delta\\geq\\widehat\{\\delta\}\-\\varepsilon\. Substitution proves the final claim\. ∎

The estimate requires both the exact rank condition and a justified error budget\. An observed gap alone does not establish either premise\. The prefix estimates must approximate the appropriate local Jacobians, not only the prefix values at the dual points\. The minimization also includes null directions of wide matrices, which an economy singular\-value decomposition can omit\. If the two unit directions are oriented to have nonnegative inner product, their Euclidean distance is at most2\\sqrt\{2\}times the left side of \([2](https://arxiv.org/html/2610.33671#S3.E2)\)\. Indeed, for the acute angleϑ\\vartheta, we have2​\(1−cos⁡ϑ\)≤2​sin2⁡ϑ2\(1\-\\cos\\vartheta\)\\leq 2\\sin^\{2\}\\vartheta\. This converts a line estimate to a vector estimate after orientation has been resolved\. It does not determine a row magnitude or bias\.

The reference procedures use rank tests to group compatible dual points\[[3](https://arxiv.org/html/2610.33671#bib.bib3),[6](https://arxiv.org/html/2610.33671#bib.bib6)\]\. The number of points needed depends on their transported span\. In the evaluated implementation, clustering also uses true hidden activations, and extraction of deeper layers uses the true prefix\. These accesses are separate from the linear\-algebra statement above\.

We read triplet files sequentially and retain at most two thousand points per neuron\. This bounds the stored points per cluster, though it can change which constraints are available for recovery\. We store each estimated direction separately from its magnitude\-corrected row and metadata\. The stored direction leaves its sign unresolved for the later search\. Magnitude correction and ground\-truth comparisons are privileged operations, as described below\.

### 3\.3Sign Recovery

We use the distance\-to\-toggle procedure of Carlini et al\.\([6](https://arxiv.org/html/2610.33671#bib.bib6), Section 6\)\. The procedure estimates which side of a critical hyperplane tends to reach a subsequent detectable event sooner\. We separate the exact local calculation from the statistical assumption used by the vote\.

At an isolated target crossing of neuron\(i,j\)\(i,j\), we fix the activation pattern of the preceding layers\. LetJ∈ℝdi×d0J\\in\\mathbb\{R\}^\{d\_\{i\}\\times d\_\{0\}\}be the Jacobian of the target layer’s pre\-activations with respect to the input\. We writeej∈ℝdie\_\{j\}\\in\\mathbb\{R\}^\{d\_\{i\}\}for thejjth standard coordinate vector and definea=J𝖳​ej∈ℝd0a=J^\{\\mathsf\{T\}\}e\_\{j\}\\in\\mathbb\{R\}^\{d\_\{0\}\}\. Thusaais the target neuron’s critical normal in input coordinates\. We assumea≠0a\\neq 0\. For the nonzero decision\-boundary normalsn\+n\_\{\+\}andn−n\_\{\-\}on the positive and negative sides, define the orthogonal tangent projectors

P±=Id0−n±​n±𝖳‖n±‖22,P\_\{\\pm\}=I\_\{d\_\{0\}\}\-\\frac\{n\_\{\\pm\}n\_\{\\pm\}^\{\\mathsf\{T\}\}\}\{\\\|n\_\{\\pm\}\\\|\_\{2\}^\{2\}\},whereId0I\_\{d\_\{0\}\}is the identity matrix onℝd0\\mathbb\{R\}^\{d\_\{0\}\}\. WhenP\+​aP\_\{\+\}aandP−​aP\_\{\-\}aare nonzero, we define the unit tangent directions

v\+=P\+​a‖P\+​a‖2,v−=−P−​a‖P−​a‖2\.v\_\{\+\}=\\frac\{P\_\{\+\}a\}\{\\\|P\_\{\+\}a\\\|\_\{2\}\},\\qquad v\_\{\-\}=\-\\frac\{P\_\{\-\}a\}\{\\\|P\_\{\-\}a\\\|\_\{2\}\}\.These signs ensure that the directions enter opposite sides of the target critical hyperplane\. Indeed, symmetry and idempotence of each orthogonal projector givea𝖳​P±​a=‖P±​a‖22a^\{\\mathsf\{T\}\}P\_\{\\pm\}a=\\\|P\_\{\\pm\}a\\\|\_\{2\}^\{2\}\. Hencea𝖳​v\+=‖P\+​a‖2\>0a^\{\\mathsf\{T\}\}v\_\{\+\}=\\\|P\_\{\+\}a\\\|\_\{2\}\>0anda𝖳​v−=−‖P−​a‖2<0a^\{\\mathsf\{T\}\}v\_\{\-\}=\-\\\|P\_\{\-\}a\\\|\_\{2\}<0\. Different projections and intervening activation changes can still give different effects on the other neurons\.

For a controlled comparison, supposeJ​v\+=r\+κ​ejJv\_\{\+\}=r\+\\kappa e\_\{j\}andJ​v−=−r−κ​ejJv\_\{\-\}=\-r\-\\kappa e\_\{j\}, whereeje\_\{j\}is the target coordinate vector,rj=0r\_\{j\}=0, andκ\>0\\kappa\>0\. LetD−j∈ℝdi×diD\_\{\-j\}\\in\\mathbb\{R\}^\{d\_\{i\}\\times d\_\{i\}\}be diagonal, with zero in entryjjand the fixed activation slopes in the other entries and setq=D−j​rq=D\_\{\-j\}r\. The post\-activation velocities are thenq\+κ​ejq\+\\kappa e\_\{j\}and−q−α​κ​ej\-q\-\\alpha\\kappa e\_\{j\}, respectively\. Therefore

‖h˙\+‖22−‖h˙−‖22=\(1−α2\)​κ2\.\\\|\\dot\{h\}\_\{\+\}\\\|\_\{2\}^\{2\}\-\\\|\\dot\{h\}\_\{\-\}\\\|\_\{2\}^\{2\}=\(1\-\\alpha^\{2\}\)\\kappa^\{2\}\.\(3\)This follows becauseqj=0q\_\{j\}=0, so the cross terms vanish\. The identity explains how controlling other coordinates creates a norm difference\. The matched\-control condition is an assumption and need not hold for boundary\-projected walks\.

Before any intervening activation change, consider a future pre\-activation withzk​\(0\)≠0z\_\{k\}\(0\)\\neq 0along a unit\-speed straight segment\. It has the formzk​\(t\)=zk​\(0\)\+t​νkz\_\{k\}\(t\)=z\_\{k\}\(0\)\+t\\nu\_\{k\}\. Its first positive zero occurs at the following distance, with\+∞\+\\inftydenoting the absence of a positive zero\.

τk=\{−zk\(0\)/νk,zk​\(0\)​νk<0,\+∞,otherwise\.\\tau\_\{k\}=\\begin\{cases\}\-z\_\{k\}\(0\)/\\nu\_\{k\},&z\_\{k\}\(0\)\\nu\_\{k\}<0,\\\\ \+\\infty,&\\text\{otherwise\}\.\\end\{cases\}The earliest future toggle on that segment ismink⁡τk\\min\_\{k\}\\tau\_\{k\}\. The rateνk\\nu\_\{k\}depends on a downstream linear functional of the target\-layer velocity\. A larger velocity norm alone does not order these hitting times\. After an earlier\-layer crossing, the local maps must be recomputed and distances accumulated across segments\.

We record the distancesd\+d\_\{\+\}andd−d\_\{\-\}on the two hypothesized sides and vote for the side with smaller distance as the positive side\. Censored walks and ties require an explicit rule\. The following bound states what a majority vote establishes when the required probabilistic assumption holds\.

###### Lemma 3\.

LetL≥1L\\geq 1be an integer\. SupposeV1,…,VLV\_\{1\},\\ldots,V\_\{L\}are independent Bernoulli variables indicating correct votes, and𝔼​Vℓ≥1/2\+γ\\mathbb\{E\}V\_\{\\ell\}\\geq 1/2\+\\gammafor everyℓ\\ell, withγ\>0\\gamma\>0\. Then

Pr⁡\(∑ℓ=1LVℓ≤L/2\)≤exp⁡\(−2​L​γ2\)\.\\Pr\\left\(\\sum\_\{\\ell=1\}^\{L\}V\_\{\\ell\}\\leq L/2\\right\)\\leq\\exp\(\-2L\\gamma^\{2\}\)\.For0<ϵ<10<\\epsilon<1, takingL≥log⁡\(1/ϵ\)/\(2​γ2\)L\\geq\\log\(1/\\epsilon\)/\(2\\gamma^\{2\}\)suffices for error probability at mostϵ\\epsilon\.

###### Proof\.

We writeS=∑ℓ=1LVℓS=\\sum\_\{\\ell=1\}^\{L\}V\_\{\\ell\}andm=𝔼​Sm=\\mathbb\{E\}S\. The expectation assumption givesm≥L⁡\(1/2\+γ\)m\\geq L\(1/2\+\\gamma\)\. On the eventS≤L/2S\\leq L/2, we therefore haveS−m≤−L​γS\-m\\leq\-L\\gamma\. Independence and the bounds0≤Vℓ≤10\\leq V\_\{\\ell\}\\leq 1allow the one\-sided Hoeffding inequality\[[13](https://arxiv.org/html/2610.33671#bib.bib13)\]\. Hence

Pr⁡\(S≤L/2\)≤Pr⁡\(S−m≤−L​γ\)≤exp⁡\(−2​\(L​γ\)2∑ℓ=1L\(1−0\)2\)=exp⁡\(−2​L​γ2\)\.\\Pr\(S\\leq L/2\)\\leq\\Pr\(S\-m\\leq\-L\\gamma\)\\leq\\exp\\\!\\left\(\-\\frac\{2\(L\\gamma\)^\{2\}\}\{\\sum\_\{\\ell=1\}^\{L\}\(1\-0\)^\{2\}\}\\right\)=\\exp\(\-2L\\gamma^\{2\}\)\.This bound includes a tied vote count in the failure event\. The requirementexp⁡\(−2​L​γ2\)≤ϵ\\exp\(\-2L\\gamma^\{2\}\)\\leq\\epsilonis equivalent toL≥log⁡\(1/ϵ\)/\(2​γ2\)L\\geq\\log\(1/\\epsilon\)/\(2\\gamma^\{2\}\), which proves the sample\-size assertion\. ∎

The lemma does not prove a positiveγ\\gammafor a given neuron\. A consistent implementation artifact can generate unanimous incorrect votes\. Reused or adaptively selected walks also require justification before this independence\-based bound is interpreted as an error guarantee\. The exact concentration result is classical\[[13](https://arxiv.org/html/2610.33671#bib.bib13)\], while the usefulness of the distance asymmetry is an empirical property of the extraction method\.

We next give a deterministic condition for preserving a local distance vote under perturbation\. This condition concerns prescribed affine continuations\. We compare paths with the same arclength parameter and matched downstream coordinates\. Applying it to paths with intervening activation changes requires a separate analysis of the later segments\.

###### Proposition 4\.

Letℐ\+\\mathcal\{I\}\_\{\+\}andℐ−\\mathcal\{I\}\_\{\-\}be finite nonempty sets of downstream coordinates\. On sides∈\{\+,−\}s\\in\\\{\+,\-\\\}, write the exact affine continuations aszs,k​\(t\)=as,k\+t​bs,kz\_\{s,k\}\(t\)=a\_\{s,k\}\+tb\_\{s,k\}and their approximations asz^s,k​\(t\)=a^s,k\+t​b^s,k\\widehat\{z\}\_\{s,k\}\(t\)=\\widehat\{a\}\_\{s,k\}\+t\\widehat\{b\}\_\{s,k\}fort≥0t\\geq 0\. Suppose, for every coordinate on both sides,

as,k​bs,k<0,\|as,k\|≥m\>0,\|bs,k\|≥λ\>0,\|as,k\|≤A,a\_\{s,k\}b\_\{s,k\}<0,\\quad\|a\_\{s,k\}\|\\geq m\>0,\\quad\|b\_\{s,k\}\|\\geq\\lambda\>0,\\quad\|a\_\{s,k\}\|\\leq A,and suppose\|a^s,k−as,k\|≤εa<m\|\\widehat\{a\}\_\{s,k\}\-a\_\{s,k\}\|\\leq\\varepsilon\_\{a\}<mand\|b^s,k−bs,k\|≤εb<λ\|\\widehat\{b\}\_\{s,k\}\-b\_\{s,k\}\|\\leq\\varepsilon\_\{b\}<\\lambda\. Define the first affine hitting distancesTs=mink∈ℐs\(−as,k/bs,k\)T\_\{s\}=\\min\_\{k\\in\\mathcal\{I\}\_\{s\}\}\(\-a\_\{s,k\}/b\_\{s,k\}\)andT^s=mink∈ℐs\(−a^s,k/b^s,k\)\\widehat\{T\}\_\{s\}=\\min\_\{k\\in\\mathcal\{I\}\_\{s\}\}\(\-\\widehat\{a\}\_\{s,k\}/\\widehat\{b\}\_\{s,k\}\)\. With

E=εaλ−εb\+A​εbλ⁡\(λ−εb\),E=\\frac\{\\varepsilon\_\{a\}\}\{\\lambda\-\\varepsilon\_\{b\}\}\+\\frac\{A\\varepsilon\_\{b\}\}\{\\lambda\(\\lambda\-\\varepsilon\_\{b\}\)\},we have\|T^s−Ts\|≤E\|\\widehat\{T\}\_\{s\}\-T\_\{s\}\|\\leq E\. IfT−−T\+\>2​ET\_\{\-\}\-T\_\{\+\}\>2E, thenT^\+<T^−\\widehat\{T\}\_\{\+\}<\\widehat\{T\}\_\{\-\}\. The reverse statement holds after exchanging the sides\.

###### Proof\.

Fix a sidessand a coordinatek∈ℐsk\\in\\mathcal\{I\}\_\{s\}\. Since\|as,k\|≥m\>εa\|a\_\{s,k\}\|\\geq m\>\\varepsilon\_\{a\}, the perturbation cannot change the sign of the intercept\. Likewise,\|bs,k\|≥λ\>εb\|b\_\{s,k\}\|\\geq\\lambda\>\\varepsilon\_\{b\}preserves the sign of the slope\. The exact intercept and slope have opposite signs, so the approximate ones do as well\. Both affine functions consequently have a unique positive zero\. We denote these zeros byτs,k=−as,k/bs,k\\tau\_\{s,k\}=\-a\_\{s,k\}/b\_\{s,k\}andτ^s,k=−a^s,k/b^s,k\\widehat\{\\tau\}\_\{s,k\}=\-\\widehat\{a\}\_\{s,k\}/\\widehat\{b\}\_\{s,k\}\.

The reverse triangle inequality gives\|b^s,k\|≥λ−εb\>0\|\\widehat\{b\}\_\{s,k\}\|\\geq\\lambda\-\\varepsilon\_\{b\}\>0\. Subtracting the two ratios yields

\|τ^s,k−τs,k\|\\displaystyle\|\\widehat\{\\tau\}\_\{s,k\}\-\\tau\_\{s,k\}\|≤\|a^s,k−as,k\|\|b^s,k\|\+\|as,k\|​\|bs,k−b^s,k\|\|bs,k\|​\|b^s,k\|\\displaystyle\\leq\\frac\{\|\\widehat\{a\}\_\{s,k\}\-a\_\{s,k\}\|\}\{\|\\widehat\{b\}\_\{s,k\}\|\}\+\\frac\{\|a\_\{s,k\}\|\\,\|b\_\{s,k\}\-\\widehat\{b\}\_\{s,k\}\|\}\{\|b\_\{s,k\}\|\\,\|\\widehat\{b\}\_\{s,k\}\|\}≤εaλ−εb\+A​εbλ⁡\(λ−εb\)=E\.\\displaystyle\\leq\\frac\{\\varepsilon\_\{a\}\}\{\\lambda\-\\varepsilon\_\{b\}\}\+\\frac\{A\\varepsilon\_\{b\}\}\{\\lambda\(\\lambda\-\\varepsilon\_\{b\}\)\}=E\.Becauseℐs\\mathcal\{I\}\_\{s\}is finite and nonempty, both minima are attained\. Chooseksk\_\{s\}minimizing the exact distances\. ThenT^s≤τ^s,ks≤τs,ks\+E=Ts\+E\\widehat\{T\}\_\{s\}\\leq\\widehat\{\\tau\}\_\{s,k\_\{s\}\}\\leq\\tau\_\{s,k\_\{s\}\}\+E=T\_\{s\}\+E\. Choosek^s\\widehat\{k\}\_\{s\}minimizing the approximate distances\. ThenTs≤τs,k^s≤τ^s,k^s\+E=T^s\+ET\_\{s\}\\leq\\tau\_\{s,\\widehat\{k\}\_\{s\}\}\\leq\\widehat\{\\tau\}\_\{s,\\widehat\{k\}\_\{s\}\}\+E=\\widehat\{T\}\_\{s\}\+E\. These two inequalities prove\|T^s−Ts\|≤E\|\\widehat\{T\}\_\{s\}\-T\_\{s\}\|\\leq Eon each side\.

IfT−−T\+\>2​ET\_\{\-\}\-T\_\{\+\}\>2E, we haveT^−−T^\+≥\(T−−E\)−\(T\+\+E\)=T−−T\+−2​E\>0\\widehat\{T\}\_\{\-\}\-\\widehat\{T\}\_\{\+\}\\geq\(T\_\{\-\}\-E\)\-\(T\_\{\+\}\+E\)=T\_\{\-\}\-T\_\{\+\}\-2E\>0\. Thus the approximate distances have the same strict ordering\. Exchanging the two sides proves the reverse assertion\. ∎

For a fixed local affine mapz⁡\(t\)=c\+B⁡\(h\+t​v\)z\(t\)=c\+B\(h\+tv\), the intercept isc\+B​hc\+Bhand the slope isB​vBv\. If the approximations satisfy‖B^−B‖2≤eB\\\|\\widehat\{B\}\-B\\\|\_\{2\}\\leq e\_\{B\},‖c^−c‖2≤ec\\\|\\widehat\{c\}\-c\\\|\_\{2\}\\leq e\_\{c\},‖h^−h‖2≤eh\\\|\\widehat\{h\}\-h\\\|\_\{2\}\\leq e\_\{h\}and‖v^−v‖2≤ev\\\|\\widehat\{v\}\-v\\\|\_\{2\}\\leq e\_\{v\}, then valid coordinatewise bounds areεa=ec\+eB​‖h‖2\+\(‖B‖2\+eB\)​eh\\varepsilon\_\{a\}=e\_\{c\}\+e\_\{B\}\\\|h\\\|\_\{2\}\+\(\\\|B\\\|\_\{2\}\+e\_\{B\}\)e\_\{h\}andεb=eB​‖v‖2\+\(‖B‖2\+eB\)​ev\\varepsilon\_\{b\}=e\_\{B\}\\\|v\\\|\_\{2\}\+\(\\\|B\\\|\_\{2\}\+e\_\{B\}\)e\_\{v\}\. Indeed, subtract the two intercepts and write the difference as\(c^−c\)\+\(B^−B\)​h\+B^​\(h^−h\)\(\\widehat\{c\}\-c\)\+\(\\widehat\{B\}\-B\)h\+\\widehat\{B\}\(\\widehat\{h\}\-h\)\. The slope difference is\(B^−B\)​v\+B^​\(v^−v\)\(\\widehat\{B\}\-B\)v\+\\widehat\{B\}\(\\widehat\{v\}\-v\)\. The triangle inequality and‖B^‖2≤‖B‖2\+eB\\\|\\widehat\{B\}\\\|\_\{2\}\\leq\\\|B\\\|\_\{2\}\+e\_\{B\}give the bounds\.

The proposition requires control of the downstream affine maps, the path directions, and a positive distance gap\. Matching the target row alone is insufficient\. If these continuations describe actual walks only until distanceHH, the hitting distances must be belowHHand no unmodelled activation crossing may occur before them\. Coordinates omitted fromℐs\\mathcal\{I\}\_\{s\}must be certified not to toggle before the proposed first event\. A certificate using the computed gap is also possible\. IfT^−−T^\+\>2​E\\widehat\{T\}\_\{\-\}\-\\widehat\{T\}\_\{\+\}\>2E, thenT\+<T−T\_\{\+\}<T\_\{\-\}by the same bound\. These certificates preserve a distance vote\. Identifying that vote with the neuron’s true sign still requires the separate sign\-signal assumption in Lemma[3](https://arxiv.org/html/2610.33671#Thmproposition3)\.

We must also specify the frame in which a vote is written\. Letw∗w\_\{\*\}be a nonzero victim row and supposew^=c​w∗\\widehat\{w\}=cw\_\{\*\}withc≠0c\\neq 0, with the bias scaled by the same factor\. The victim frame labels the side where the victim pre\-activation is positive\. The recovered frame labels the side where the recovered pre\-activation is positive\. If a votes∗∈\{−1,1\}s\_\{\*\}\\in\\\{\-1,1\\\}specifies an orientation in the victim frame, its representation in the recovered frame issign⁡\(c\)​s∗\\operatorname\{sign\}\(c\)s\_\{\*\}\. This follows because positiveccpreserves the two sides and negativeccexchanges them\. The conversion is an algebraic identity\. Computingsign⁡\(c\)\\operatorname\{sign\}\(c\)from⟨w^,w∗⟩\\langle\\widehat\{w\},w\_\{\*\}\\rangleuses victim parameters\. The orientation conversion makes this computation before the walk and uses its result in selecting directions\. We therefore include that access in the assisted procedure, rather than classify it as evaluation only\.

For distances, we can certify an ordering without assigning a value to a censored event\. Letτ\+,τ−∈\[0,∞\]\\tau\_\{\+\},\\tau\_\{\-\}\\in\[0,\\infty\]denote distances to the same specified stopping event on two prescribed walks\. Suppose recorded intervals\[ℓ\+,u\+\]\[\\ell\_\{\+\},u\_\{\+\}\]and\[ℓ−,u−\]\[\\ell\_\{\-\},u\_\{\-\}\]contain the corresponding distances\. Ifu\+<ℓ−u\_\{\+\}<\\ell\_\{\-\}, thenτ\+≤u\+<ℓ−≤τ−\\tau\_\{\+\}\\leq u\_\{\+\}<\\ell\_\{\-\}\\leq\\tau\_\{\-\}\. The reverse ordering follows whenu−<ℓ\+u\_\{\-\}<\\ell\_\{\+\}\. Otherwise these intervals alone do not certify an ordering\. A walk with no event through a verified distanceHHgives the interval\[H,∞\]\[H,\\infty\]\. A numerical failure gives no such lower bound unless the traversal toHHwas verified\. This certificate concerns the event ordering\. The relation between that ordering and the neuron’s sign remains a separate assumption\.

We can obtain a positive event\-probability difference from the controlled velocities under an explicit local model\. This gives one sufficient mechanism for a sign signal\. It does not assume that trained downstream weights follow this model\. We writeΦ\(t\)=\(2π\)−1/2∫−∞texp\(−u2/2\)du\\Phi\(t\)=\(2\\pi\)^\{\-1/2\}\\int\_\{\-\\infty\}^\{t\}\\exp\(\-u^\{2\}/2\)\\,dufor the standard normal distribution function\.

###### Proposition 5\.

Letq∈ℝdiq\\in\\mathbb\{R\}^\{d\_\{i\}\}satisfyqj=0q\_\{j\}=0, letκ\>0\\kappa\>0, and let0≤α<10\\leq\\alpha<1\. Definev\+=q\+κ​ejv\_\{\+\}=q\+\\kappa e\_\{j\}andv−=−q−α​κ​ejv\_\{\-\}=\-q\-\\alpha\\kappa e\_\{j\}\. Letgghave independent standard normal coordinates, and fixz0≠0z\_\{0\}\\neq 0andH\>0H\>0independently ofgg\. In the affine continuationsz±​\(t\)=z0\+t⁡⟨g,v±⟩z\_\{\\pm\}\(t\)=z\_\{0\}\+t\\langle g,v\_\{\\pm\}\\rangle, letB±B\_\{\\pm\}indicate a zero at somet∈\(0,H\]t\\in\(0,H\]\. Then

p\+=𝔼​B\+=Φ⁡\(−\|z0\|H​‖q‖22\+κ2\),p−=𝔼​B−=Φ⁡\(−\|z0\|H​‖q‖22\+α2​κ2\),p\_\{\+\}=\\mathbb\{E\}B\_\{\+\}=\\Phi\\\!\\left\(\-\\frac\{\|z\_\{0\}\|\}\{H\\sqrt\{\\\|q\\\|\_\{2\}^\{2\}\+\\kappa^\{2\}\}\}\\right\),\\qquad p\_\{\-\}=\\mathbb\{E\}B\_\{\-\}=\\Phi\\\!\\left\(\-\\frac\{\|z\_\{0\}\|\}\{H\\sqrt\{\\\|q\\\|\_\{2\}^\{2\}\+\\alpha^\{2\}\\kappa^\{2\}\}\}\\right\),wherep−=0p\_\{\-\}=0if its denominator is zero\. We haveμ=p\+−p−\>0\\mu=p\_\{\+\}\-p\_\{\-\}\>0\. For an integerL≥1L\\geq 1andLLindependent draws ofgg, with paired indicatorsBℓ,\+,Bℓ,−B\_\{\\ell,\+\},B\_\{\\ell,\-\},

Pr\(∑ℓ=1L\(Bℓ,\+−Bℓ,−\)≤0\)≤exp\(−Lμ2/2\)\.\\Pr\\\!\\left\(\\sum\_\{\\ell=1\}^\{L\}\(B\_\{\\ell,\+\}\-B\_\{\\ell,\-\}\)\\leq 0\\right\)\\leq\\exp\(\-L\\mu^\{2\}/2\)\.

###### Proof\.

For a fixed vectorvv, the rate⟨g,v⟩\\langle g,v\\rangleis normal with mean zero and variance‖v‖22\\\|v\\\|\_\{2\}^\{2\}\. Ifz0\>0z\_\{0\}\>0, a zero byHHoccurs exactly when this rate is at most−z0/H\-z\_\{0\}/H\. Ifz0<0z\_\{0\}<0, it occurs when the rate is at least\|z0\|/H\|z\_\{0\}\|/H\. Normal symmetry gives the displayed probability in either case\. A zero rate cannot reach zero fromz0≠0z\_\{0\}\\neq 0, which proves the convention for zero variance\.

Sinceqj=0q\_\{j\}=0, the two variances are‖q‖22\+κ2\\\|q\\\|\_\{2\}^\{2\}\+\\kappa^\{2\}and‖q‖22\+α2​κ2\\\|q\\\|\_\{2\}^\{2\}\+\\alpha^\{2\}\\kappa^\{2\}\. The first is strictly larger\. For the positive threshold\|z0\|/H\|z\_\{0\}\|/H, the displayed normal tail is strictly increasing in the positive standard deviation\. Alsop\+\>0p\_\{\+\}\>0when the negative\-side variance vanishes\. Thusμ\>0\\mu\>0in every case\.

For each independent draw, defineDℓ=Bℓ,\+−Bℓ,−D\_\{\\ell\}=B\_\{\\ell,\+\}\-B\_\{\\ell,\-\}\. The two indicators within a pair need not be independent\. Their difference satisfies−1≤Dℓ≤1\-1\\leq D\_\{\\ell\}\\leq 1and𝔼​Dℓ=p\+−p−=μ\\mathbb\{E\}D\_\{\\ell\}=p\_\{\+\}\-p\_\{\-\}=\\mu\. The draws ofggmake the variablesD1,…,DLD\_\{1\},\\ldots,D\_\{L\}independent\. The event∑ℓDℓ≤0\\sum\_\{\\ell\}D\_\{\\ell\}\\leq 0implies∑ℓ\(Dℓ−μ\)≤−L​μ\\sum\_\{\\ell\}\(D\_\{\\ell\}\-\\mu\)\\leq\-L\\mu\. Hoeffding’s lower\-tail inequality\[[13](https://arxiv.org/html/2610.33671#bib.bib13)\]therefore bounds its probability by

exp\(−2​\(L​μ\)2∑ℓ=1L\(1−\(−1\)\)2\)=exp\(−Lμ2/2\),\\exp\\\!\\left\(\-\\frac\{2\(L\\mu\)^\{2\}\}\{\\sum\_\{\\ell=1\}^\{L\}\(1\-\(\-1\)\)^\{2\}\}\\right\)=\\exp\(\-L\\mu^\{2\}/2\),which proves the stated bound\. ∎

This proposition concerns the excess of fixed\-horizon events under affine continuation\. It does not order the two first\-event distances for every draw or prove correctness of the implemented distance vote\. The assumptions exclude changes in the local map beforeHHand require an independent Gaussian downstream response with the stated controlled velocities\. Actual walks can violate each condition\. We can test those conditions using logged local rates and intervening events, but should not identify the model’s positiveμ\\muwith a measured sign advantage without those checks\.

The last hidden layer has no later hidden activation to toggle\. The reference method can instead use intersections with other class decision boundaries\([6](https://arxiv.org/html/2610.33671#bib.bib6), Section 6\.2\)\. Thus the absence of future hidden layers does not make last\-layer sign recovery impossible\. The first layer also need not lack a sign signal\. The weak endpoint results in our implementation concern its chosen walks and measured events\.

The modified runner usesalong\_decision\_boundaryin every layer, replacing theperfect\_control\_along\_decision\_boundaryconfiguration at the endpoints\. The latter caps a recorded negative\-side distance at three times the positive\-side distance\. A capmin⁡\{d−,3​d\+\}\\min\\\{d\_\{\-\},3d\_\{\+\}\\\}withd\+\>0d\_\{\+\}\>0preserves the strict comparisond−\>d\+d\_\{\-\}\>d\_\{\+\}when that comparison already holds\. The cap alone therefore does not prove a universal positive\-vote bias\. Censoring, unsuccessful walks, and default sign decisions must be analyzed together\. The runner also bounds the inner bisection iterations and saves per\-neuron signs and vote summaries\. We treat low\-information sign estimates as initialization for the label\-based search, rather than proof that their confidence values are calibrated\.

### 3\.4Privileged Information in the Implementation

The reference code\[[6](https://arxiv.org/html/2610.33671#bib.bib6)\]contains operations that depart from the intended hard\-label oracle\. We retain its D1–D6 identifiers so that the manuscript can be compared with the code\. Table[1](https://arxiv.org/html/2610.33671#S3.T1)separates runtime information unavailable through labels \(type A\) from direct parameter reads \(type W\)\. These operations assist the attack itself\. They are distinct from ground\-truth comparisons used only to evaluate a completed attack\.

Table 1:Privileged accesses in the evaluated extraction implementation\.OperationReadsTypeRole in the reported implementationD1 – Boundary detectionLogit gapAIn the D1 ablation, disabling it reduces accepted directions on the tiny ReLU victim from 142/256 to 24/256\. This is an implementation ablation, not an impossibility result\.D2 – Boundary refinementExact logit gradientsAA label\-based fallback exists in the reference code\. Its use must be specified in a strict\-oracle experiment\.D3 – Neuron clusteringHidden activationsAThe reference code also provides an SVD\-based alternative\. Its reported higher cost does not make privileged access necessary in principle\.D4 – Prefix initializationTrue weight matricesWSupplies the prefix used to extract deeper layers\. The present experiments do not replace it with a fully reconstructed prefix\.D5 – Signature scalingTrue weight vectorWCorrects scale and checks candidates\. In the Leaky ReLU path, ground\-truth agreement also determines candidate acceptance\.D6 – Sign\-walk parametersTrue weights and biasesWSupplies the prefix used by the sign\-recovery walk\. The recovered\-frame conversion also compares recovered rows with true rows before choosing directions\.

D4 and D6 avoid the error propagation that a reconstructed prefix would introduce\. D5 also affects the interpretation of direction coverage, since accepting a candidate by comparison with its true row uses information unavailable to an attacker\. Removing these accesses requires a separate end\-to\-end evaluation\. Later fitting from labels does not retrospectively make the extraction stages label\-only\.

## 4Leaky ReLU Adaptation

We replace the zero negative\-side slope by a fixedα∈\(0,1\)\\alpha\\in\(0,1\)\. Qi et al\.\[[20](https://arxiv.org/html/2610.33671#bib.bib20)\]also study extraction with several activation functions\. Our experiments concern the adaptation of this reference implementation and its use in the subsequent reconstruction\.

### 4\.1Local Geometry

For a fixed scalar pre\-activation, ReLU and Leaky ReLU change slope at the same zero\. For fixed first\-layer weights and biases, their first\-layer critical hyperplanes therefore agree\. This does not imply that all deeper critical sets or decision boundaries agree\. A deeper pre\-activation depends on the outputs of earlier activations, which change whenα\\alphachanges\. For example, a second\-layer pre\-activationσα​\(x\)\+1\\sigma\_\{\\alpha\}\(x\)\+1has no zero whenα=0\\alpha=0, whereas forα\>0\\alpha\>0it vanishes atx=−1/αx=\-1/\\alpha\.

The local bend calculation nevertheless has the same form\. Consider an isolated crossing of neuron\(i,j\)\(i,j\), with the prefix and all other local activation states fixed\. On either side, write its pre\-activation asηi,j​\(x\)=a𝖳​x\+b\\eta\_\{i,j\}\(x\)=a^\{\\mathsf\{T\}\}x\+b\. Letu∈ℝKu\\in\\mathbb\{R\}^\{K\}be the downstream linear response to its activation\. The difference between the two affine logit maps, extended to the same neighborhood, is

f\+​\(x\)−f−​\(x\)=\(1−α\)​u​\(a𝖳​x\+b\)\.f\_\{\+\}\(x\)\-f\_\{\-\}\(x\)=\(1\-\\alpha\)u\(a^\{\\mathsf\{T\}\}x\+b\)\.Thus the gradients of themm–nnlogit difference satisfy

∇\(fm−fn\)\+−∇\(fm−fn\)−=\(1−α\)​\(um−un\)​a\.\\nabla\(f\_\{m\}\-f\_\{n\}\)\_\{\+\}\-\\nabla\(f\_\{m\}\-f\_\{n\}\)\_\{\-\}=\(1\-\\alpha\)\(u\_\{m\}\-u\_\{n\}\)a\.This follows by subtracting the slopes11andα\\alphaof the target activation and composing with the fixed downstream affine map\. Ifum=unu\_\{m\}=u\_\{n\}, the crossing has no effect on this logit difference\. Even when the right\-hand side is nonzero, a visible bend requires the two boundary normals to be nonparallel and the same classes to remain maximal\. We therefore transfer the local method, not an assertion of identical geometry or guaranteed detectability\.

### 4\.2Slope Calculations and Candidate Acceptance

Prefix linearization uses the matricesDi​\(ρ\)D\_\{i\}\(\\rho\)defined in Section[2](https://arxiv.org/html/2610.33671#S2)\. Negative coordinates contribute a slopeα\\alphainstead of zero\. We apply this substitution consistently in the PyTorch and NumPy activation functions, the prefix linearization, and the sign\-walk forward simulations\. The coordinate\-hit buffer is allocated with the prefix output dimensiondi−1d\_\{i\-1\}, rather than the target output dimensiondid\_\{i\}\. This corrects a shape error on nonuniform architectures under either activation\. The same substitution changes the downstream response used by the distance\-to\-toggle procedure\. It can alter the strength of the sign statistic, and no universal accuracy bound follows fromα\\alphabeing small\.

The ReLU implementation checks for zero\-hit prefix coordinates and requires singular valuesslast<10−4s\_\{\\mathrm\{last\}\}<10^\{\-4\}andspenultimate\>10−2s\_\{\\mathrm\{penultimate\}\}\>10^\{\-2\}\. These thresholds approximate the one\-dimensional\-kernel condition in Eq\. \([1](https://arxiv.org/html/2610.33671#S3.E1)\)\. A hit uses the finite threshold10−410^\{\-4\}, so zero hits need not mean mathematically zero activation\. The ReLU implementation accepts a direction when its dual\-space matrix has a numerically isolated smallest singular value\. In the Leaky ReLU path we relax this gate and retain candidates that agree sufficiently well with the true neuron\. This comparison uses privileged information\. Accordingly, the higher Leaky ReLU coverage measures candidates accepted by the assisted implementation\. It does not establish the same coverage for an attacker who must select candidates from labels alone\.

### 4\.3Rank and Inactive Coordinates

For a fixed matrixBBmapping an input perturbation to pre\-activation perturbations, the local activation map givesD​BDB\. When0<α<10<\\alpha<1, all diagonal entries ofDDlie in\[α,1\]\[\\alpha,1\]\. Hence

###### Proposition 6\.

LetB∈ℝk×nB\\in\\mathbb\{R\}^\{k\\times n\}and letD∈ℝk×kD\\in\\mathbb\{R\}^\{k\\times k\}be diagonal with entries in\[α,1\]\[\\alpha,1\], where0<α<10<\\alpha<1\. For everyv∈ℝnv\\in\\mathbb\{R\}^\{n\},α​‖B​v‖2≤‖D​B​v‖2≤‖B​v‖2\.\\alpha\\\|Bv\\\|\_\{2\}\\leq\\\|DBv\\\|\_\{2\}\\leq\\\|Bv\\\|\_\{2\}\.Consequentlyker⁡\(D​B\)=ker⁡B\\ker\(DB\)=\\ker Bandrank⁡\(D​B\)=rank⁡B\\operatorname\{rank\}\(DB\)=\\operatorname\{rank\}B\. If insteadDDhas diagonal entries in\{0,1\}\\\{0,1\\\}, as for ReLU, thenrank⁡\(D​B\)≤rank⁡B\\operatorname\{rank\}\(DB\)\\leq\\operatorname\{rank\}B, with strict inequality possible\.

###### Proof\.

WritingD=diag⁡\(d1,…,dk\)D=\\operatorname\{diag\}\(d\_\{1\},\\ldots,d\_\{k\}\)gives‖D​B​v‖22=∑ℓdℓ2​\(B​v\)ℓ2\\\|DBv\\\|\_\{2\}^\{2\}=\\sum\_\{\\ell\}d\_\{\\ell\}^\{2\}\(Bv\)\_\{\\ell\}^\{2\}\. Bounding eachdℓ2d\_\{\\ell\}^\{2\}betweenα2\\alpha^\{2\}and 1 proves the inequalities\. Sinceα\>0\\alpha\>0,D​B​v=0DBv=0if and only ifB​v=0Bv=0\. Rank–nullity then gives equality of ranks\. For ReLU, the image ofD​BDBis the image underDDof the image ofBB, so its dimension cannot exceedrank⁡B\\operatorname\{rank\}B\. Strict inequality occurs, for example, withB=I2B=I\_\{2\}andD=diag⁡\(1,0\)D=\\operatorname\{diag\}\(1,0\), whose ranks are two and one, respectively\. ∎

This proposition concerns one fixed pre\-activation matrix\. It does not assert equality of ReLU and Leaky ReLU network Jacobians, since changing the activation also changes deeper patterns and maps\. Products with later weight matrices can remain rank deficient\. Repeated multiplication by slopes as small asα\\alphacan worsen conditioning\. Thus Leaky ReLU avoids rank loss caused solely by this local diagonal activation, but does not guarantee a unique or well\-conditioned SVD solution\. Both this mechanism and the relaxed candidate gate can affect the reported coverage\.

## 5Metaheuristic Sign Search

We search over the signs of recovered rows while using cached victim labels to score candidate networks\. Within one scoring call, candidate evaluations reuse the cached labels\. The driver can invoke this procedure repeatedly and request the same labels again, so we count label acquisition at every invocation\. It does not guarantee recovery of the victim’s signs, because different parameter assignments can agree on a finite set\.

### 5\.1Sign Assignments

Forkkrecovered rows in a layer, we writes∈\{−1,1\}ks\\in\\\{\-1,1\\\}^\{k\}for a sign assignment andf^s\\widehat\{f\}\_\{s\}for the corresponding reconstructed logits\. The recovered row for neuronjjissj​A^js\_\{j\}\\widehat\{A\}\_\{j\}\. When we negate a row, we also negate its bias estimate, or recompute that estimate from stored dual points\. This preserves the zero set of its affine pre\-activation in the current prefix coordinates\.

LetE=\{\(xℓ,yℓ\)\}ℓ=1LE=\\\{\(x\_\{\\ell\},y\_\{\\ell\}\)\\\}\_\{\\ell=1\}^\{L\}be a cached scoring set withyℓ=𝒪⁡\(xℓ\)y\_\{\\ell\}=\\mathcal\{O\}\(x\_\{\\ell\}\)\. We define agreement by

AE\(s\)=1L∑ℓ=1L𝟏\[argmaxc\[f^s\(xℓ\)\]c=yℓ\]\.A\_\{E\}\(s\)=\\frac\{1\}\{L\}\\sum\_\{\\ell=1\}^\{L\}\\mathbf\{1\}\[\\arg\\max\_\{c\}\[\\widehat\{f\}\_\{s\}\(x\_\{\\ell\}\)\]\_\{c\}=y\_\{\\ell\}\]\.Exhaustive search evaluates all2k2^\{k\}assignments\. It is practical only for modestkk, and identifies an optimum of this finite\-set objective rather than necessarily the true signs\.

### 5\.2Single\-Flip Search

A greedy search changes one sign at a time and retains changes that improve agreement\. The discrete objective can have plateaus and local optima\. Several simultaneous flips can improve agreement even when no individual flip does\. This motivates methods that sometimes accept a worse score\. The controlled benchmark starts from the true 256\-hidden\-neuron ReLU model with half its signs corrupted\. The reported mean sign accuracies are0\.5570\.557for greedy search,0\.5690\.569for simulated annealing with agreement,0\.6390\.639for simulated annealing with the margin, and0\.6660\.666for parallel tempering with the margin\. The first three means use eight seeds, whereas parallel tempering uses seeds 0–5\. These numbers evaluate correction from a privileged initialization and are separate from the end\-to\-end results in Table[6](https://arxiv.org/html/2610.33671#S7.T6)\. The unequal seed sets also preclude treating their mean differences as paired estimates\. Per\-seed benchmark outputs are needed for a statistical comparison\.

### 5\.3Agreement and Margin Objectives

We define the reconstructed model’s mean label margin by

mE​\(s\)=1L​∑ℓ=1L\(\[f^s​\(xℓ\)\]yℓ−maxc≠yℓ⁡\[f^s​\(xℓ\)\]c\)\.m\_\{E\}\(s\)=\\frac\{1\}\{L\}\\sum\_\{\\ell=1\}^\{L\}\\left\(\[\\widehat\{f\}\_\{s\}\(x\_\{\\ell\}\)\]\_\{y\_\{\\ell\}\}\-\\max\_\{c\\neq y\_\{\\ell\}\}\[\\widehat\{f\}\_\{s\}\(x\_\{\\ell\}\)\]\_\{c\}\\right\)\.The labelyℓy\_\{\\ell\}comes from the victim, but all logits in this expression come from the reconstructed model\. Thus computing the margin does not require victim logits\. The margin can distinguish assignments with equal agreement\. It is a surrogate objective, not a smooth function of discrete signs, and increasing it need not increase agreement\. It also depends on logit scale\.

We use agreement to select the final saved candidate and a margin\-based score to guide exploration\. The margin score ismEm\_\{E\}without a stated additional normalization\. The shared driver first runs greedy search, stores the best state byAEA\_\{E\}, and restores that state after exploration\. Including the initial state among the saved candidates ensures that the returned training\-set agreement is no smaller than at entry\. This guarantees neither improved held\-out agreement nor more correct signs\. Search stops when agreement reaches1−10−121\-10^\{\-12\}\. We do not equate final selection by agreement with accepting every search move by agreement\.

### 5\.4Simulated Annealing and Parallel Tempering

For a chosen exploration scoreSS, simulated annealing\[[15](https://arxiv.org/html/2610.33671#bib.bib15)\]proposes a sign change fromsstos′s^\{\\prime\}and, for symmetric proposals, accepts it with probability

min⁡\{1,exp⁡\(\(S⁡\(s′\)−S⁡\(s\)\)/T\)\},T\>0\.\\min\\\{1,\\exp\(\(S\(s^\{\\prime\}\)\-S\(s\)\)/T\)\\\},\\qquad T\>0\.The default schedule decreases geometrically from0\.020\.02to10−410^\{\-4\}over four sweeps\. A decreasing temperature makes worsening moves less likely\. At low temperature this approaches greedy search forSS, not gradient ascent\.

Parallel tempering\[[12](https://arxiv.org/html/2610.33671#bib.bib12)\]runs chains at several positive temperatures\. For chains atTa,TbT\_\{a\},T\_\{b\}with statessa,sbs\_\{a\},s\_\{b\}, the standard swap acceptance probability for a score being maximized is

min⁡\{1,exp⁡\(\(Ta−1−Tb−1\)​\(S⁡\(sb\)−S⁡\(sa\)\)\)\}\.\\min\\\{1,\\exp\(\(T\_\{a\}^\{\-1\}\-T\_\{b\}^\{\-1\}\)\(S\(s\_\{b\}\)\-S\(s\_\{a\}\)\)\)\\\}\.The parallel\-tempering defaults use six replicas on a geometric temperature ladder from10−310^\{\-3\}to0\.050\.05\. Both methods can move between local optima, but neither guarantees the best assignment within a fixed computation budget\. We use simulated annealing as the default and compare parallel tempering on selected victims\. The tables report the combined reconstruction, not an isolated proof of correct sign recovery\.

### 5\.5Selection with Independent Scoring Data

We separate the statistical selection problem from the optimizer’s search\. Letℱ=\{gs:s∈𝒮\}\\mathcal\{F\}=\\\{g\_\{s\}:s\\in\\mathcal\{S\}\\\}be a nonempty finite family of candidate classifiers, where𝒮\\mathcal\{S\}is a finite index set andM=\|𝒮\|M=\|\\mathcal\{S\}\|\. We construct the family using fitting data and fix it before observing the scoring sample\. An index can encode a sign assignment and an output layer fitted using only the fitting labels\. We allow distinct indices to define the same classifier\.

LetX1,…,XnX\_\{1\},\\ldots,X\_\{n\}be independent scoring inputs, withn≥1n\\geq 1\. They need not have identical distributions\. We define the mean target agreement and its empirical estimate by

ps=1n∑ℓ=1nPr\(gs\(Xℓ\)=𝒪\(Xℓ\)\),p^s=1n∑ℓ=1n𝟏\[gs\(Xℓ\)=𝒪\(Xℓ\)\]\.p\_\{s\}=\\frac\{1\}\{n\}\\sum\_\{\\ell=1\}^\{n\}\\Pr\(g\_\{s\}\(X\_\{\\ell\}\)=\\mathcal\{O\}\(X\_\{\\ell\}\)\),\\qquad\\widehat\{p\}\_\{s\}=\\frac\{1\}\{n\}\\sum\_\{\\ell=1\}^\{n\}\\mathbf\{1\}\[g\_\{s\}\(X\_\{\\ell\}\)=\\mathcal\{O\}\(X\_\{\\ell\}\)\]\.For identically distributed inputs,psp\_\{s\}is ordinary population agreement under their common distribution\. For fixed\-size independent strata, it is the agreement under the mixture weighted by the stratum proportions\. If the family is random, we condition on the fitting data and every random choice used to construct it\. The scoring inputs must remain independent with the specified distributions after this conditioning\.

###### Proposition 7\.

Under these assumptions, forε\>0\\varepsilon\>0,

Pr⁡\(maxs∈𝒮⁡\|p^s−ps\|\>ε\)≤2​M​exp⁡\(−2​n​ε2\)\.\\Pr\\\!\\left\(\\max\_\{s\\in\\mathcal\{S\}\}\|\\widehat\{p\}\_\{s\}\-p\_\{s\}\|\>\\varepsilon\\right\)\\leq 2M\\exp\(\-2n\\varepsilon^\{2\}\)\.Suppose the returned indexs^\\widehat\{s\}satisfiesp^s^≥maxs⁡p^s−ρ\\widehat\{p\}\_\{\\widehat\{s\}\}\\geq\\max\_\{s\}\\widehat\{p\}\_\{s\}\-\\rhofor a specifiedρ≥0\\rho\\geq 0\. For0<β<10<\\beta<1, with probability at least1−β1\-\\beta,

maxs⁡ps−ps^≤2​log⁡\(2​M/β\)2​n\+ρ\.\\max\_\{s\}p\_\{s\}\-p\_\{\\widehat\{s\}\}\\leq 2\\sqrt\{\\frac\{\\log\(2M/\\beta\)\}\{2n\}\}\+\\rho\.\(4\)IfM\>1M\>1, there is a unique maximizing indexs∗s\_\{\*\}, andps∗−ps≥Δ\>0p\_\{s\_\{\*\}\}\-p\_\{s\}\\geq\\Delta\>0for everys≠s∗s\\neq s\_\{\*\}, then an exact empirical maximizer satisfies

Pr\(s^≠s∗\)≤\(M−1\)exp\(−nΔ2/2\)\.\\Pr\(\\widehat\{s\}\\neq s\_\{\*\}\)\\leq\(M\-1\)\\exp\(\-n\\Delta^\{2\}/2\)\.\(5\)

###### Proof\.

For each fixedss, the agreement indicators are independent and lie in\[0,1\]\[0,1\]\. The two one\-sided Hoeffding bounds\[[13](https://arxiv.org/html/2610.33671#bib.bib13)\]givePr⁡\(\|p^s−ps\|\>ε\)≤2​exp⁡\(−2​n​ε2\)\\Pr\(\|\\widehat\{p\}\_\{s\}\-p\_\{s\}\|\>\\varepsilon\)\\leq 2\\exp\(\-2n\\varepsilon^\{2\}\)\. Summing over theMMindices proves the first assertion\. No independence between candidates is needed\.

On the event that all deviations are at mostε\\varepsilon, choose an indexs∗s\_\{\*\}maximizingpsp\_\{s\}\. We have

ps∗−ps^=\\displaystyle p\_\{s\_\{\*\}\}\-p\_\{\\widehat\{s\}\}=\{\}\(ps∗−p^s∗\)\+\(p^s∗−p^s^\)\+\(p^s^−ps^\)\\displaystyle\(p\_\{s\_\{\*\}\}\-\\widehat\{p\}\_\{s\_\{\*\}\}\)\+\(\\widehat\{p\}\_\{s\_\{\*\}\}\-\\widehat\{p\}\_\{\\widehat\{s\}\}\)\+\(\\widehat\{p\}\_\{\\widehat\{s\}\}\-p\_\{\\widehat\{s\}\}\)≤\\displaystyle\\leq\{\}2​ε\+ρ\.\\displaystyle 2\\varepsilon\+\\rho\.Settingε=log⁡\(2​M/β\)/\(2​n\)\\varepsilon=\\sqrt\{\\log\(2M/\\beta\)/\(2n\)\}proves \([4](https://arxiv.org/html/2610.33671#S5.E4)\)\.

For the gap assertion, define the paired differences

Zℓ\(s\)=𝟏\[gs∗\(Xℓ\)=𝒪\(Xℓ\)\]−𝟏\[gs\(Xℓ\)=𝒪\(Xℓ\)\]\.Z\_\{\\ell\}\(s\)=\\mathbf\{1\}\[g\_\{s\_\{\*\}\}\(X\_\{\\ell\}\)=\\mathcal\{O\}\(X\_\{\\ell\}\)\]\-\\mathbf\{1\}\[g\_\{s\}\(X\_\{\\ell\}\)=\\mathcal\{O\}\(X\_\{\\ell\}\)\]\.They are independent acrossℓ\\ell, lie in\[−1,1\]\[\-1,1\], and their average expectation is at leastΔ\\Delta\. Hoeffding’s lower\-tail inequality therefore givesPr\(∑ℓZℓ\(s\)≤0\)≤exp\(−nΔ2/2\)\\Pr\(\\sum\_\{\\ell\}Z\_\{\\ell\}\(s\)\\leq 0\)\\leq\\exp\(\-n\\Delta^\{2\}/2\)\. An incorrect empirical selection requires some other index to tie or exceeds∗s\_\{\*\}in empirical agreement\. Summing the bound over theM−1M\-1competitors proves \([5](https://arxiv.org/html/2610.33671#S5.E5)\), counting ties conservatively as failures\. ∎

Exhaustive search overkkbinary signs hasM≤2kM\\leq 2^\{k\}\. A sufficient sample count for selection error at mostβ\\betain the gap setting isn≥2​log⁡\(\(M−1\)/β\)/Δ2n\\geq 2\\log\(\(M\-1\)/\\beta\)/\\Delta^\{2\}\. Fork=8k=8,Δ=0\.05\\Delta=0\.05, andβ=0\.05\\beta=0\.05, this gives 6,830 inputs after rounding upward\. The value ofΔ\\Deltahere is an assumption, not an estimate for the experimental victims\. Proposition[7](https://arxiv.org/html/2610.33671#Thmproposition7)proves sign identification only when the intended assignment belongs to the family and uniquely maximizes the target agreement\. Prediction\-equivalent assignments have no positive gap\.

For simulated annealing or parallel tempering, \([4](https://arxiv.org/html/2610.33671#S5.E4)\) requires a justified optimization errorρ\\rho\. The algorithms do not supply that guarantee merely by saving their best visited state\. Nor can we apply a fixed\-family bound using only the number of visited candidates when those candidates are trained or refined on the scoring data\. A fixed family of all possible candidates may still be considered if it is specified independently, but its size must include that entire family\. The exhaustive block experiments in Section[7\.7](https://arxiv.org/html/2610.33671#S7.SS7)address this distinction\.

## 6Learning\-Assisted Reconstruction

We construct a complete model from the accepted directions\. We estimate biases, fit an output layer, search signs, refit the output layer, and train the remaining parameters while masking recovered\-row gradients\. Exact row restoration is a separate update rule\. These stages use stored dual points and cached labels\. Their initialization retains the privileged accesses described in Section[3\.4](https://arxiv.org/html/2610.33671#S3.SS4)\.

### 6\.1Incomplete Parameters and Sign Errors

Directions alone leave signs, biases, unrecovered neurons, and the output layer unresolved\. A sign error has an exact effect at the target coordinate because

σα​\(−z\)−σα​\(z\)=−\(1\+α\)​z\(z∈ℝ\)\.\\sigma\_\{\\alpha\}\(\-z\)\-\\sigma\_\{\\alpha\}\(z\)=\-\(1\+\\alpha\)z\\qquad\(z\\in\\mathbb\{R\}\)\.\(6\)Forz≥0z\\geq 0, the left side is−α​z−z\-\\alpha z\-z\. Forz<0z<0, it is−z−α​z\-z\-\\alpha z, giving the same expression\. The identity assumes that both the incoming row and its bias are negated\. Negating only the row is a different change\.

###### Proposition 8\.

Consider two networks with identical parameters except that one incoming row and its bias in hidden layeriiare negated\. We denote the original and modified logit maps byffandf^\\widehat\{f\}, respectively\. Letz⁡\(x\)z\(x\)be that neuron’s original pre\-activation\. Ifgig\_\{i\}is the common map fromhih\_\{i\}to the logits and has Lipschitz constantLiL\_\{i\}from Euclidean norm to maximum norm, then

‖f^​\(x\)−f⁡\(x\)‖∞≤\(1\+α\)​Li​\|z⁡\(x\)\|\.\\\|\\widehat\{f\}\(x\)\-f\(x\)\\\|\_\{\\infty\}\\leq\(1\+\\alpha\)L\_\{i\}\|z\(x\)\|\.Ifffhas a unique predicted classyyand marginμ⁡\(x\)=fy​\(x\)−maxc≠y⁡fc​\(x\)\\mu\(x\)=f\_\{y\}\(x\)\-\\max\_\{c\\neq y\}f\_\{c\}\(x\), its label is unchanged whenever

μ⁡\(x\)\>2​\(1\+α\)​Li​\|z⁡\(x\)\|\.\\mu\(x\)\>2\(1\+\\alpha\)L\_\{i\}\|z\(x\)\|\.

###### Proof\.

The networks have the same prefix through layeri−1i\-1, so their pre\-activations at the modified neuron arez⁡\(x\)z\(x\)and−z⁡\(x\)\-z\(x\)\. All other pre\-activations in layeriiagree\. By Eq\. \([6](https://arxiv.org/html/2610.33671#S6.E6)\),

h^i​\(x\)−hi​\(x\)=−\(1\+α\)​z​\(x\)​ej,\\widehat\{h\}\_\{i\}\(x\)\-h\_\{i\}\(x\)=\-\(1\+\\alpha\)z\(x\)e\_\{j\},whereeje\_\{j\}is the coordinate vector of the modified neuron\. Its Euclidean norm is\(1\+α\)​\|z⁡\(x\)\|\(1\+\\alpha\)\|z\(x\)\|\. The parameters after layeriiagree, so the same downstream mapgig\_\{i\}acts on both hidden vectors\. Its Lipschitz property gives‖f^​\(x\)−f⁡\(x\)‖∞≤Li​‖h^i​\(x\)−hi​\(x\)‖2=\(1\+α\)​Li​\|z⁡\(x\)\|\\\|\\widehat\{f\}\(x\)\-f\(x\)\\\|\_\{\\infty\}\\leq L\_\{i\}\\\|\\widehat\{h\}\_\{i\}\(x\)\-h\_\{i\}\(x\)\\\|\_\{2\}=\(1\+\\alpha\)L\_\{i\}\|z\(x\)\|\.

Setϵ=\(1\+α\)​Li​\|z⁡\(x\)\|\\epsilon=\(1\+\\alpha\)L\_\{i\}\|z\(x\)\|\. For every classc≠yc\\neq y, the logit bound givesf^y​\(x\)≥fy​\(x\)−ϵ\\widehat\{f\}\_\{y\}\(x\)\\geq f\_\{y\}\(x\)\-\\epsilonandf^c​\(x\)≤fc​\(x\)\+ϵ\\widehat\{f\}\_\{c\}\(x\)\\leq f\_\{c\}\(x\)\+\\epsilon\. Hencef^y​\(x\)−f^c​\(x\)≥fy​\(x\)−fc​\(x\)−2​ϵ≥μ⁡\(x\)−2​ϵ\\widehat\{f\}\_\{y\}\(x\)\-\\widehat\{f\}\_\{c\}\(x\)\\geq f\_\{y\}\(x\)\-f\_\{c\}\(x\)\-2\\epsilon\\geq\\mu\(x\)\-2\\epsilon\. Under the stated margin condition, this quantity is positive for every competitor\. Thusyyremains the unique predicted class\. ∎

Since each hidden activation is 1\-Lipschitz, we may take

Li=‖A\(r\+1\)‖2→∞​∏k=i\+1r‖A\(k\)‖2,L\_\{i\}=\\\|A^\{\(r\+1\)\}\\\|\_\{2\\to\\infty\}\\prod\_\{k=i\+1\}^\{r\}\\\|A^\{\(k\)\}\\\|\_\{2\},where an empty product is 1\. To see this, an affine hidden layer changes the Euclidean distance by at most its matrix norm, its bias cancels from the difference, and the activation does not increase that distance\. Composing these bounds through the hidden layers and applying the output norm gives the displayed constant\. This proves a sufficient condition for label preservation, not an error\-rate estimate\. A sign error can be harmless when its outgoing response vanishes or the classification margin is large\. No assumption about weight\-direction recovery forcesz⁡\(x\)z\(x\)to be positive on half the inputs\.

The raw reconstruction also has zero biases and a randomly initialized output layer, as well as missing hidden rows\. Its low agreement therefore cannot be attributed to signs alone\. The stage\-by\-stage ablation in Section[7\.6](https://arxiv.org/html/2610.33671#S7.SS6)separates these changes empirically\.

We can also bound errors arising at several layers without assuming that a single sign is the only change\. Consider two networks with the same architecture and activation, with neurons and positive scale conventions aligned\. We writeA^\(i\),𝒃^\(i\)\\widehat\{A\}^\{\(i\)\},\\widehat\{\\boldsymbol\{b\}\}^\{\(i\)\}for the reconstructed parameters,h^i\\widehat\{h\}\_\{i\}for their hidden representations, andf^\\widehat\{f\}for their logit map\. For hidden layers defineei=‖A^\(i\)−A\(i\)‖2e\_\{i\}=\\\|\\widehat\{A\}^\{\(i\)\}\-A^\{\(i\)\}\\\|\_\{2\}andqi=‖𝒃^\(i\)−𝒃\(i\)‖2q\_\{i\}=\\\|\\widehat\{\\boldsymbol\{b\}\}^\{\(i\)\}\-\\boldsymbol\{b\}^\{\(i\)\}\\\|\_\{2\}\.

###### Proposition 9\.

FixR≥0R\\geq 0and consider inputs with‖x‖2≤R\\\|x\\\|\_\{2\}\\leq R\. DefineB0=RB\_\{0\}=R,U0=0U\_\{0\}=0, and, for1≤i≤r1\\leq i\\leq r,

Bi=‖A\(i\)‖2​Bi−1\+‖𝒃\(i\)‖2,Ui=‖A^\(i\)‖2​Ui−1\+ei​Bi−1\+qi\.B\_\{i\}=\\\|A^\{\(i\)\}\\\|\_\{2\}B\_\{i\-1\}\+\\\|\\boldsymbol\{b\}^\{\(i\)\}\\\|\_\{2\},\\qquad U\_\{i\}=\\\|\\widehat\{A\}^\{\(i\)\}\\\|\_\{2\}U\_\{i\-1\}\+e\_\{i\}B\_\{i\-1\}\+q\_\{i\}\.Then‖hi​\(x\)‖2≤Bi\\\|h\_\{i\}\(x\)\\\|\_\{2\}\\leq B\_\{i\}and‖h^i​\(x\)−hi​\(x\)‖2≤Ui\\\|\\widehat\{h\}\_\{i\}\(x\)\-h\_\{i\}\(x\)\\\|\_\{2\}\\leq U\_\{i\}\. Define

V=‖A^\(r\+1\)‖2→∞​Ur\+‖A^\(r\+1\)−A\(r\+1\)‖2→∞​Br\+‖𝒃^\(r\+1\)−𝒃\(r\+1\)‖∞\.V=\\\|\\widehat\{A\}^\{\(r\+1\)\}\\\|\_\{2\\to\\infty\}U\_\{r\}\+\\\|\\widehat\{A\}^\{\(r\+1\)\}\-A^\{\(r\+1\)\}\\\|\_\{2\\to\\infty\}B\_\{r\}\+\\\|\\widehat\{\\boldsymbol\{b\}\}^\{\(r\+1\)\}\-\\boldsymbol\{b\}^\{\(r\+1\)\}\\\|\_\{\\infty\}\.Then‖f^​\(x\)−fθ​\(x\)‖∞≤V\\\|\\widehat\{f\}\(x\)\-f\_\{\\theta\}\(x\)\\\|\_\{\\infty\}\\leq V\. If the victim has a unique winning class and marginμ⁡\(x\)\>2​V\\mu\(x\)\>2V, the two classifiers give the same label\.

###### Proof\.

Fix an input with‖x‖2≤R\\\|x\\\|\_\{2\}\\leq R\. At layer zero,h0​\(x\)=h^0​\(x\)=xh\_\{0\}\(x\)=\\widehat\{h\}\_\{0\}\(x\)=x, so‖h0​\(x\)‖2≤B0\\\|h\_\{0\}\(x\)\\\|\_\{2\}\\leq B\_\{0\}and‖h^0​\(x\)−h0​\(x\)‖2=U0=0\\\|\\widehat\{h\}\_\{0\}\(x\)\-h\_\{0\}\(x\)\\\|\_\{2\}=U\_\{0\}=0\.

Suppose the stated bounds hold at layeri−1i\-1\. The activation vanishes at zero and is 1\-Lipschitz\. Therefore

‖hi​\(x\)‖2\\displaystyle\\\|h\_\{i\}\(x\)\\\|\_\{2\}≤‖A\(i\)​hi−1​\(x\)\+𝒃\(i\)‖2\\displaystyle\\leq\\\|A^\{\(i\)\}h\_\{i\-1\}\(x\)\+\\boldsymbol\{b\}^\{\(i\)\}\\\|\_\{2\}≤‖A\(i\)‖2​Bi−1\+‖𝒃\(i\)‖2=Bi\.\\displaystyle\\leq\\\|A^\{\(i\)\}\\\|\_\{2\}B\_\{i\-1\}\+\\\|\\boldsymbol\{b\}^\{\(i\)\}\\\|\_\{2\}=B\_\{i\}\.For the reconstructed pre\-activation, writeη^i​\(x\)=A^\(i\)​h^i−1​\(x\)\+𝒃^\(i\)\\widehat\{\\eta\}\_\{i\}\(x\)=\\widehat\{A\}^\{\(i\)\}\\widehat\{h\}\_\{i\-1\}\(x\)\+\\widehat\{\\boldsymbol\{b\}\}^\{\(i\)\}\. Subtract the two pre\-activations and insertA^\(i\)​hi−1​\(x\)\\widehat\{A\}^\{\(i\)\}h\_\{i\-1\}\(x\)\. This gives

η^i​\(x\)−ηi​\(x\)=\\displaystyle\\widehat\{\\eta\}\_\{i\}\(x\)\-\\eta\_\{i\}\(x\)=\{\}A^\(i\)​\(h^i−1​\(x\)−hi−1​\(x\)\)\\displaystyle\\widehat\{A\}^\{\(i\)\}\(\\widehat\{h\}\_\{i\-1\}\(x\)\-h\_\{i\-1\}\(x\)\)\+\(A^\(i\)−A\(i\)\)​hi−1​\(x\)\+\(𝒃^\(i\)−𝒃\(i\)\)\.\\displaystyle\+\(\\widehat\{A\}^\{\(i\)\}\-A^\{\(i\)\}\)h\_\{i\-1\}\(x\)\+\(\\widehat\{\\boldsymbol\{b\}\}^\{\(i\)\}\-\\boldsymbol\{b\}^\{\(i\)\}\)\.The induction hypotheses and the triangle inequality imply

‖h^i​\(x\)−hi​\(x\)‖2\\displaystyle\\\|\\widehat\{h\}\_\{i\}\(x\)\-h\_\{i\}\(x\)\\\|\_\{2\}≤‖η^i​\(x\)−ηi​\(x\)‖2\\displaystyle\\leq\\\|\\widehat\{\\eta\}\_\{i\}\(x\)\-\\eta\_\{i\}\(x\)\\\|\_\{2\}≤‖A^\(i\)‖2​Ui−1\+ei​Bi−1\+qi=Ui\.\\displaystyle\\leq\\\|\\widehat\{A\}^\{\(i\)\}\\\|\_\{2\}U\_\{i\-1\}\+e\_\{i\}B\_\{i\-1\}\+q\_\{i\}=U\_\{i\}\.This proves both hidden\-layer bounds by induction\.

At the output layer, write

f^​\(x\)−fθ​\(x\)=\\displaystyle\\widehat\{f\}\(x\)\-f\_\{\\theta\}\(x\)=\{\}A^\(r\+1\)​\(h^r​\(x\)−hr​\(x\)\)\\displaystyle\\widehat\{A\}^\{\(r\+1\)\}\(\\widehat\{h\}\_\{r\}\(x\)\-h\_\{r\}\(x\)\)\+\(A^\(r\+1\)−A\(r\+1\)\)​hr​\(x\)\+\(𝒃^\(r\+1\)−𝒃\(r\+1\)\)\.\\displaystyle\+\(\\widehat\{A\}^\{\(r\+1\)\}\-A^\{\(r\+1\)\}\)h\_\{r\}\(x\)\+\(\\widehat\{\\boldsymbol\{b\}\}^\{\(r\+1\)\}\-\\boldsymbol\{b\}^\{\(r\+1\)\}\)\.For each term involving a matrix, use‖B​z‖∞≤‖B‖2→∞​‖z‖2\\\|Bz\\\|\_\{\\infty\}\\leq\\\|B\\\|\_\{2\\to\\infty\}\\\|z\\\|\_\{2\}\. The bounds just proved give‖f^​\(x\)−fθ​\(x\)‖∞≤V\\\|\\widehat\{f\}\(x\)\-f\_\{\\theta\}\(x\)\\\|\_\{\\infty\}\\leq V\.

Letyybe the victim’s unique winning class\. For everyc≠yc\\neq y, we havef^y​\(x\)−f^c​\(x\)≥\[fθ​\(x\)\]y−\[fθ​\(x\)\]c−2​V≥μ⁡\(x\)−2​V\\widehat\{f\}\_\{y\}\(x\)\-\\widehat\{f\}\_\{c\}\(x\)\\geq\[f\_\{\\theta\}\(x\)\]\_\{y\}\-\[f\_\{\\theta\}\(x\)\]\_\{c\}\-2V\\geq\\mu\(x\)\-2V\. Ifμ⁡\(x\)\>2​V\\mu\(x\)\>2V, every such difference is positive\. The reconstructed classifier therefore also predictsyy\. ∎

This result connects aligned parameter errors with sufficient conditions for label preservation\. It does not supply small errors for missing rows\. Direction estimates alone also do not boundeie\_\{i\}until magnitudes, signs, and coordinate matching have been resolved\. Products of weight norms can make the bound uninformative even when measured agreement is high\. Those limitations must be checked before using the estimate to certify a reconstructed prefix or a classifier\.

### 6\.2Bias Estimates from Dual Points

For an exact dual pointx∗x^\{\*\}of neuron\(i,j\)\(i,j\),

Aj\(i\)​hi−1​\(x∗\)\+bj\(i\)=0\.A\_\{j\}^\{\(i\)\}h\_\{i\-1\}\(x^\{\*\}\)\+b\_\{j\}^\{\(i\)\}=0\.Consequently, ifA^j\(i\)=c​Aj\(i\)\\widehat\{A\}\_\{j\}^\{\(i\)\}=cA\_\{j\}^\{\(i\)\}and the true prefix features are known in matching coordinates, the compatible bias isb^j\(i\)=c​bj\(i\)\\widehat\{b\}\_\{j\}^\{\(i\)\}=cb\_\{j\}^\{\(i\)\}\. In the reconstructed network we instead define

b^j\(i\)​\(x∗\)=−A^j\(i\)​h^i−1​\(x∗\)\.\\widehat\{b\}\_\{j\}^\{\(i\)\}\(x^\{\*\}\)=\-\\widehat\{A\}\_\{j\}^\{\(i\)\}\\widehat\{h\}\_\{i\-1\}\(x^\{\*\}\)\.\(7\)For each neuron with at least one assigned point, we take the median over at most 30 such estimates, proceeding from the first layer to the last\. For a deeper layer, this is an estimate relative to the reconstructed prefix, not an unconditional recovery of the true bias\.

To include approximate dual points, letxℓx\_\{\\ell\}be an assigned point and define its true critical residual byrℓ=Aj\(i\)​hi−1​\(xℓ\)\+bj\(i\)r\_\{\\ell\}=A\_\{j\}^\{\(i\)\}h\_\{i\-1\}\(x\_\{\\ell\}\)\+b\_\{j\}^\{\(i\)\}\. WriteA^j\(i\)=c​Aj\(i\)\+e𝖳\\widehat\{A\}\_\{j\}^\{\(i\)\}=cA\_\{j\}^\{\(i\)\}\+e^\{\\mathsf\{T\}\}andh^i−1​\(xℓ\)=hi−1​\(xℓ\)\+δℓ\\widehat\{h\}\_\{i\-1\}\(x\_\{\\ell\}\)=h\_\{i\-1\}\(x\_\{\\ell\}\)\+\\delta\_\{\\ell\}\. Substituting into \([7](https://arxiv.org/html/2610.33671#S6.E7)\) gives

b^j\(i\)​\(xℓ\)−c​bj\(i\)=−c​rℓ−c​Aj\(i\)​δℓ−e𝖳​hi−1​\(xℓ\)−e𝖳​δℓ\.\\widehat\{b\}\_\{j\}^\{\(i\)\}\(x\_\{\\ell\}\)\-cb\_\{j\}^\{\(i\)\}=\-cr\_\{\\ell\}\-cA\_\{j\}^\{\(i\)\}\\delta\_\{\\ell\}\-e^\{\\mathsf\{T\}\}h\_\{i\-1\}\(x\_\{\\ell\}\)\-e^\{\\mathsf\{T\}\}\\delta\_\{\\ell\}\.Thus its absolute value is at most

\|c\|​\(\|rℓ\|\+‖Aj\(i\)‖2​‖δℓ‖2\)\+‖e‖2​\(‖hi−1​\(xℓ\)‖2\+‖δℓ‖2\)\.\|c\|\\bigl\(\|r\_\{\\ell\}\|\+\\\|A\_\{j\}^\{\(i\)\}\\\|\_\{2\}\\\|\\delta\_\{\\ell\}\\\|\_\{2\}\\bigr\)\+\\\|e\\\|\_\{2\}\\bigl\(\\\|h\_\{i\-1\}\(x\_\{\\ell\}\)\\\|\_\{2\}\+\\\|\\delta\_\{\\ell\}\\\|\_\{2\}\\bigr\)\.The critical residual vanishes at an exact point\. If more than half the estimates have this bound at mostϵ\\epsilon, their median lies in\[c​bj\(i\)−ϵ,c​bj\(i\)\+ϵ\]\[cb\_\{j\}^\{\(i\)\}\-\\epsilon,cb\_\{j\}^\{\(i\)\}\+\\epsilon\]\. To prove this, order the estimates\. Fewer than half lie below the interval and fewer than half lie above it\. The middle estimate for an odd sample size therefore belongs to the interval\. For an even sample size, both middle estimates belong to it, so their arithmetic mean also belongs to it\. This is the median used by the implementation\. The argument permits a minority of arbitrary outliers, but does not remove a common prefix error or a common critical\-point residual\. Later training may reduce prediction loss without correcting the true bias\.

### 6\.3Output\-Layer Fitting

The output layer introduces no activation kink of its own, so hidden\-neuron signature recovery does not directly estimate its rows\. Its class differences do affect decision boundaries and can be studied by other methods\[[4](https://arxiv.org/html/2610.33671#bib.bib4)\]\. Here we fit a multinomial logistic regression model usingzℓ=h^r​\(xℓ\)z\_\{\\ell\}=\\widehat\{h\}\_\{r\}\(x\_\{\\ell\}\)and cached labelsyℓy\_\{\\ell\}\. For weightsWWand biasesbb, the data loss is

ℒ\(W,b\)=1L∑ℓ=1L\(log∑c=1Kexp\(\(Wzℓ\+b\)c\)−\(Wzℓ\+b\)yℓ\)\.\\mathcal\{L\}\(W,b\)=\\frac\{1\}\{L\}\\sum\_\{\\ell=1\}^\{L\}\\left\(\\log\\sum\_\{c=1\}^\{K\}\\exp\(\(Wz\_\{\\ell\}\+b\)\_\{c\}\)\-\(Wz\_\{\\ell\}\+b\)\_\{y\_\{\\ell\}\}\\right\)\.A finite fit requires an appropriate stopping rule or regularization, especially for separable data\. We use scikit\-learn’s multinomialLogisticRegressionwith thelbfgssolver, an intercept,C=106C=10^\{6\}, and a maximum of 2000 iterations\. This is weakly regularized fitting, rather than an unregularized identification theorem\. If some classes are absent, copying zeros into their output rows does not prevent those classes from winning the reconstructed argmax\. A coverage check or explicit restriction to fitted classes is needed\. The fitted coefficients define a decoder for the reconstructed features\. They are not identified as the victim’s unique coefficients from labels\.

When sign–refinement cycles are enabled, we fit a provisional output layer before the first sign pass, alternate search with 20\-epoch refinement bursts, and fit the output layer again afterward\. With zero cycles, sign search instead precedes the output fit\. Tables[3](https://arxiv.org/html/2610.33671#S7.T3)and[4](https://arxiv.org/html/2610.33671#S7.T4)specify the reported cyclic runs\. Both ordering choices use labels rather than victim logits, but repeated function calls can recache those labels and must be counted separately\.

### 6\.4Gradient Masking and Exact Row Preservation

LetRiR\_\{i\}be the indices of accepted rows in layerii, and letA^j\(i\)\\widehat\{A\}\_\{j\}^\{\(i\)\}denote each selected row after sign assignment and scale selection\. We writeW\(i\)W^\{\(i\)\}for the current trainable matrix of the reconstructed network, to distinguish it from the victim matrixA\(i\)A^\{\(i\)\}\. The intended exact constraint for refinement isWj\(i\)=A^j\(i\)​\(j∈Ri\)W\_\{j\}^\{\(i\)\}=\\widehat\{A\}\_\{j\}^\{\(i\)\}\\ \(j\\in R\_\{i\}\)\. The biases, output layer, and unaccepted rows remain trainable\. An update satisfying this constraint retains the selected directions by construction\. It does not prove that they were recovered without privileged information, that every retained sign is correct, or that the final hidden features equal the victim’s features\.

The implementation masks recovered\-row gradients before optimization\. We analyze AdamW with decoupled weight decay\[[18](https://arxiv.org/html/2610.33671#bib.bib18)\]\. Adam without weight decay is included by setting the decay coefficient to zero\. Coupled weight decay inside the Adam gradient is a different update and is not covered by the following argument\.

###### Proposition 10\.

Suppose a nonzero recovered roww0w\_\{0\}is masked at every optimizer step, its Adam first and second moments initially vanish, and no operation adds an unmasked update to it\. Letηt≥0\\eta\_\{t\}\\geq 0be the step size andλt≥0\\lambda\_\{t\}\\geq 0its decoupled weight\-decay coefficient\. If0≤ηt​λt<10\\leq\\eta\_\{t\}\\lambda\_\{t\}<1, then, without row restoration and in exact arithmetic, afterTTsteps

wT=\(∏t=0T−1\(1−ηt​λt\)\)​w0\.w\_\{T\}=\\left\(\\prod\_\{t=0\}^\{T\-1\}\(1\-\\eta\_\{t\}\\lambda\_\{t\}\)\\right\)w\_\{0\}\.Its signed direction is preserved\. Its exact value is preserved whenλt=0\\lambda\_\{t\}=0for all steps, or when the saved row is restored after each update\.

###### Proof\.

Letgtg\_\{t\}be the masked gradient of the recovered row at steptt\. The masking assumption givesgt=0g\_\{t\}=0\. Writemtm\_\{t\}andvtv\_\{t\}for its first and second Adam moments at the start of that step, withm0=v0=0m\_\{0\}=v\_\{0\}=0\. For Adam coefficientsβ1,β2∈\[0,1\)\\beta\_\{1\},\\beta\_\{2\}\\in\[0,1\), the moment updates are

mt\+1=β1​mt\+\(1−β1\)​gt,vt\+1=β2​vt\+\(1−β2\)​\(gt⊙gt\),m\_\{t\+1\}=\\beta\_\{1\}m\_\{t\}\+\(1\-\\beta\_\{1\}\)g\_\{t\},\\qquad v\_\{t\+1\}=\\beta\_\{2\}v\_\{t\}\+\(1\-\\beta\_\{2\}\)\(g\_\{t\}\\odot g\_\{t\}\),where⊙\\odotdenotes coordinatewise multiplication\. Ifmt=vt=0m\_\{t\}=v\_\{t\}=0, both updated moments are zero\. Induction therefore givesmt=vt=0m\_\{t\}=v\_\{t\}=0at every step\. LetϵAdam\>0\\epsilon\_\{\\mathrm\{Adam\}\}\>0be the denominator offset\. The bias\-corrected moments are zero because their denominators1−β1t\+11\-\\beta\_\{1\}^\{t\+1\}and1−β2t\+11\-\\beta\_\{2\}^\{t\+1\}are positive\. Each adaptive\-update coordinate is therefore0/\(0\+ϵAdam\)=00/\(\\sqrt\{0\}\+\\epsilon\_\{\\mathrm\{Adam\}\}\)=0\.

Only the decoupled weight\-decay term remains\. Thuswt\+1=wt−ηt​λt​wt=\(1−ηt​λt\)​wtw\_\{t\+1\}=w\_\{t\}\-\\eta\_\{t\}\\lambda\_\{t\}w\_\{t\}=\(1\-\\eta\_\{t\}\\lambda\_\{t\}\)w\_\{t\}\. Iterating fromt=0t=0toT−1T\-1gives the displayed product\. Every factor is strictly positive by hypothesis, so the product is positive andwT/‖wT‖2=w0/‖w0‖2w\_\{T\}/\\\|w\_\{T\}\\\|\_\{2\}=w\_\{0\}/\\\|w\_\{0\}\\\|\_\{2\}\. This proves signed\-direction preservation\. If everyλt\\lambda\_\{t\}is zero, the product equals one andwT=w0w\_\{T\}=w\_\{0\}\. If the row is copied back from its saved value after each update, every completed step instead ends withwt\+1=w0w\_\{t\+1\}=w\_\{0\}, independently of the tentative decayed value\. This proves the restoration assertion\. ∎

This proposition distinguishes fixed directions from fixed row values\. Nonzero inherited moments, unmasked mini\-refinement, or later sign changes require separate handling\. Moreover, training a bias while rescaling its row generally changes its critical hyperplane\. Direction preservation does not imply preservation of the original hidden feature map\.

We can enforce the constraint without assumptions on the optimizer moments\. LetPiP\_\{i\}be the diagonal matrix whose entry is one onRiR\_\{i\}and zero elsewhere, and letWlock\(i\)W\_\{\\mathrm\{lock\}\}^\{\(i\)\}store the selected rows at entry to the refinement call\. After any proposed update producingW~\(i\)\\widetilde\{W\}^\{\(i\)\}, define

W\(i\)=\(I−Pi\)​W~\(i\)\+Pi​Wlock\(i\)\.W^\{\(i\)\}=\(I\-P\_\{i\}\)\\widetilde\{W\}^\{\(i\)\}\+P\_\{i\}W\_\{\\mathrm\{lock\}\}^\{\(i\)\}\.\(8\)Multiplying byPiP\_\{i\}and usingPi2=PiP\_\{i\}^\{2\}=P\_\{i\}givesPi​W\(i\)=Pi​Wlock\(i\)P\_\{i\}W^\{\(i\)\}=P\_\{i\}W\_\{\\mathrm\{lock\}\}^\{\(i\)\}\. Thus every projected step satisfies the constraint regardless of weight decay or inherited moments\. In floating\-point code, copying the saved coordinates back after each update preserves their bit patterns\. A checkpoint restored by early stopping must satisfy the same constraint\. Intentional sign changes begin a new refinement call with newly selected locked rows\. This projection does not prove convergence to a global minimum of the label loss\.

The complete ablation update masks gradients and applies AdamW without this restoration\. In 906 nonempty layer comparisons immediately before and after refinement, every recovered\-row group changed\. The largest absolute coordinate change was approximately1\.23241×10−41\.23241\\times 10^\{\-4\}, while the smallest computed row cosine was0\.99999999999999960\.9999999999999996\. These measurements are consistent with direction retention under fresh moments, not exact row equality\. The separate 288\-configuration refinement study reports exact preservation in its 144 restoration configurations\. Its numeric trajectories contain at least the reported number of invariant rows in each corresponding layer\. A direct check of the intended row indices additionally requires exporting the recovered masks independently of the model\-object checkpoints\. The restoration study does not change the update used by the 48\-run ablation sweep\.

We also make the checkpoint\-selection guarantee explicit\. LetG0G\_\{0\}be the model entering a refinement call, letG1,…,GTG\_\{1\},\\ldots,G\_\{T\}be its checked checkpoints, and letSSbe a fixed nonempty stopping sample with cached victim labelsy⁡\(x\)y\(x\)\. Definep^S\(G\)=\|S\|−1∑x∈S𝟏\[G\(x\)=y\(x\)\]\\widehat\{p\}\_\{S\}\(G\)=\|S\|^\{\-1\}\\sum\_\{x\\in S\}\\mathbf\{1\}\[G\(x\)=y\(x\)\], whereG⁡\(x\)G\(x\)denotes the model’s predicted class\. If we retain a maximizerG∗G\_\{\*\}overG0,…,GTG\_\{0\},\\ldots,G\_\{T\}, thenp^S​\(G∗\)≥p^S​\(G0\)\\widehat\{p\}\_\{S\}\(G\_\{\*\}\)\\geq\\widehat\{p\}\_\{S\}\(G\_\{0\}\)becauseG0G\_\{0\}belongs to the candidate set\. We implement this rule by evaluating and savingG0G\_\{0\}before the first optimizer step and restoring the best checked state\. If the initial state is excluded, this inequality need not hold\. The guarantee concerns the stopping sample only\. The selected model may agree less often on an independent test sample\. The fixed\-family guarantee in Proposition[7](https://arxiv.org/html/2610.33671#Thmproposition7)can provide an additional population bound only when its independence and family assumptions hold\. Reusing stopping labels to construct the checkpoint path does not meet those assumptions automatically\.

## 7Experimental Results

We compare assisted reconstruction with distillation and examine the individual shortcuts in Section[7\.7](https://arxiv.org/html/2610.33671#S7.SS7)\. We call fitting onXtest∪Xtest2X\_\{\\mathrm\{test\}\}\\cup X\_\{\\mathrm\{test2\}\}the joint\-fit protocol\. The split\-fit protocol fits onXtestX\_\{\\mathrm\{test\}\}, scores sign candidates onXtest2X\_\{\\mathrm\{test2\}\}, and reservesXtest3X\_\{\\mathrm\{test3\}\}for final reporting\. Its refinement also uses the scoring slice for stopping unless a separate stopping pool is specified\. Direction coverage uses the extraction procedures and privileged checks of Section[3\.4](https://arxiv.org/html/2610.33671#S3.SS4)\. The comparison therefore does not measure the benefit of a strict hard\-label cryptanalytic attack at equal information access\.

Table[2](https://arxiv.org/html/2610.33671#S7.T2)describes three data slices\. The first two provide cached labels for fitting and search\. The third is also used by the early\-stopping watchdog in Table[4](https://arxiv.org/html/2610.33671#S7.T4)\. It is disjoint from the training inputs but participates in model selection\. We consequently call it the evaluation and selection slice, rather than an untouched test set\. An independent final test set would be needed for evaluation without this reuse\.

Table 2:Dataset contract for joint\-fit experiments\. The split\-fit comparison uses the separate roles described in Section[7\.7](https://arxiv.org/html/2610.33671#S7.SS7)\.SliceRoleSource \(make\_blobs/ CIFAR\-10\)XtestX\_\{\\mathrm\{test\}\}Phase\-3 training poolseed 42 / test batchXtest2X\_\{\\mathrm\{test2\}\}Phase\-3 training poolseed 99 / train idx 40000–49999Xtest3X\_\{\\mathrm\{test3\}\}Evaluation and early stoppingseed 123 / train idx 10000–19999Tables[3](https://arxiv.org/html/2610.33671#S7.T3)and[4](https://arxiv.org/html/2610.33671#S7.T4)give the settings recorded in the manuscript\. The synthetic victims use data generated bymake\_blobs\. The CIFAR\-10 runs use a separate driver and a V100 GPU with float64 arithmetic\. The settings were chosen within the available computation budget\. They do not establish optimal query or computation complexity\.

The synthetic\-data contract fixes the cluster centers at seed 42, uses the same scaling transform for all slices, and changes only the sampling seeds 42, 99, and 123\. The inputs are clipped and scaled byx↦clip⁡\(x,−3,3\)/3x\\mapsto\\operatorname\{clip\}\(x,\-3,3\)/3\. Thus different slice seeds do not create different randomly chosen cluster centers\. The six\-victim batch log records 18:11:05–18:25:24 UTC on 21 June 2026, an elapsed time of 14 minutes 19 seconds including both search arms and reporting\. This is a measured run duration, not a controlled speedup ratio\. We describe the shortcut\-comparison experiment separately in Section[7\.7](https://arxiv.org/html/2610.33671#S7.SS7)\.

Table 3:Per\-architecture tuned parameters\. The CIFAR runs use simulated annealing \(SA\) with the margin score\. PT denotes parallel tempering\.Parametertiniesttiniertinyfull\(CIFAR\)dual\-search rounds6820150dual workers / batch7 / 2567 / 2567 / 2561–3 / 256target triplets / round300020001000010000sign restarts1120–1sign pair\-lookahead8884sign refine cycles3331refine epochs300500500500

Table 4:Pipeline parameters held fixed across all victims\.PhaseParameterValueDual searchimplementationtorch, CUDA float64Sign recoveryrunnerbatched\_sign\_recovery\.py, float64Reconstructionsign\-search methodSA\+margin \(default\), PT\+margin \(comparison\)mini\-refine20 epochs, lr5×10−35\{\\times\}10^\{\-3\}, wd10−410^\{\-4\}refine optimiserAdamW, cosine lr, wd10−410^\{\-4\}early stoppingpatience 5, evaluation every 10 epochsdata useselect on X\_test3, train on X\_test∪\\cupX\_test2ActivationLEAKY\_ALPHA0\.0 \(ReLU\) / 0\.01 \(Leaky ReLU\)Label accessreconstruction batches15 measured calls in the seeded synthetic sweep

We count one hard\-label request for each input whose victim label is requested\. If stagejjcalls the victim on a batchBjB\_\{j\}, the request count is∑j\|Bj\|\\sum\_\{j\}\|B\_\{j\}\|\. Repeated requests count again\. We distinguish this count from batched calls and from the number of distinct inputs labelled\. The seeded synthetic sweep records 15 batched calls per phase\-3 execution, including the stopping\-set labels\. Under the joint\-fit protocol, each fitting or scoring input is labelled 14 times\. Across the six victims, phase 3 requests 57,000–211,024 labels under that protocol and 29,024–96,024 under the split\-fit protocols\. These are phase\-3 costs only\. Section[7\.8](https://arxiv.org/html/2610.33671#S7.SS8)reports the additional phase\-1 requests and the privileged accesses separately\. Caching could reduce repeated label requests, but the measured implementation does not use one cache across all these calls\.

### 7\.1Evaluation Metric

We write𝒪^​\(x\)=arg⁡maxc⁡\[f^​\(x\)\]c\\widehat\{\\mathcal\{O\}\}\(x\)=\\arg\\max\_\{c\}\[\\widehat\{f\}\(x\)\]\_\{c\}for a reconstructed classifier\. For a finite nonempty setXX, define

C\(X\)=1\|X\|∑x∈X𝟏\[𝒪^\(x\)=𝒪\(x\)\]\.C\(X\)=\\frac\{1\}\{\|X\|\}\\sum\_\{x\\in X\}\\mathbf\{1\}\[\\widehat\{\\mathcal\{O\}\}\(x\)=\\mathcal\{O\}\(x\)\]\.We report three functional metrics\.C1=C⁡\(Xtest3\)C\_\{1\}=C\(X\_\{\\mathrm\{test3\}\}\)measures agreement on the evaluation and selection slice in the joint\-fit protocol\. In the split\-fit comparison, this slice is reserved for final reporting\. ForC2C\_\{2\}, we generate two pools of 5000 probes\. The uniform pool uses the coordinatewise bounding box of the reference sliceXtestX\_\{\\mathrm\{test\}\}\. The Gaussian pool has the reference mean and coordinatewise standard deviations multiplied by 3, with a10−1210^\{\-12\}guard on the standard deviations\. We define

C2=12​\(C⁡\(Xuniform\)\+C⁡\(XGaussian\)\)\.C\_\{2\}=\\tfrac\{1\}\{2\}\\bigl\(C\(X\_\{\\mathrm\{uniform\}\}\)\+C\(X\_\{\\mathrm\{Gaussian\}\}\)\\bigr\)\.Interpolation\-path agreement is reported separately and is not included in this average\.

ForC3C\_\{3\}, we select at most 1500 evaluation inputs and sample eight unit directions at each point\. For each direction, we query the label at radius 2\. If that label differs from the starting label, we bisect the segment 12 times while maintaining endpoints with different labels and record the upper endpoint radius\. Otherwise the direction is assigned infinite radius\. The minimum directional result is clipped at 2, giving a proxyd^​\(x\)\\widehat\{d\}\(x\)\. This is a label\-based search radius, not the exact distance to the nearest decision boundary\. Without monotonicity of labels along a ray, bisection need not locate its first crossing, and matching endpoint labels can conceal intermediate crossings\.

We bin the proxy by its empirical tertiles, deduplicating repeated quantile edges, and defineC3C\_\{3\}as agreement in the resulting far bin\. If that bin is absent, the implementation usesC1C\_\{1\}\. Ties and clipping mean that the far bin need not contain one third of the subsample\. For instance, the CIFAR ReLU and Leaky ReLU far bins contain 1349 and 1327 of 1500 inputs\. This input\-space proxy differs from the reconstructed logit margin used by sign search\.

Table 5:Functional agreement metrics\. None identifies the victim’s parameters\.MetricInterpretationC1C\_\{1\}– Evaluation\-slice agreementPrediction agreement on the slice also used for early stopping\.C2C\_\{2\}– Off\-distribution agreementPrediction agreement under the specified alternative input distributions\.C3C\_\{3\}– Margin\-conditioned agreementPrediction agreement in the far bin of the directional search\-radius proxy\.

Off\-distribution agreement can reveal differences missed by a single evaluation distribution, but highC2C\_\{2\}does not prove structural recovery\. A distilled model can also agree away from its training samples\. Likewise, conditioning on margin does not remove all boundary\-estimation or sampling effects\. Structural evaluation instead requires direct parameter comparisons after accounting for neuron permutations and scaling ambiguities\.

### 7\.2Per\-Victim Extraction Results

Table[6](https://arxiv.org/html/2610.33671#S7.T6)gives accepted directions per layer, the reported sign accuracy, task accuracy, and agreement with the victim\. The four hidden layers are denoted fc1–fc4\. Direction counts incorporate D5 candidate checking\. We omit a direction\-cosine column because ground\-truth acceptance would make that statistic conditional on the check itself\.

For a layer withri\>0r\_\{i\}\>0accepted rows, definesis\_\{i\}as the fraction whose dot product with the matched true row is positive\. The sign column reports the unweighted mean over nonempty layers,

s¯=1\|I\|​∑i∈Isi,I=\{i:ri\>0\}\.\\overline\{s\}=\\frac\{1\}\{\|I\|\}\\sum\_\{i\\in I\}s\_\{i\},\\qquad I=\\\{i:r\_\{i\}\>0\\\}\.It is not the pooled neuron fraction∑iri​si/∑iri\\sum\_\{i\}r\_\{i\}s\_\{i\}/\\sum\_\{i\}r\_\{i\}\. For example, the first row averages3/83/8,3/73/7, and5/55/5, giving0\.601190​…0\.601190\\ldots, whereas the pooled fraction is11/20=0\.5511/20=0\.55\. This resolves the nonintegral\-looking sign entry\. Neuron matching follows the assisted extraction metadata\.

Table 6:Assisted reconstruction across the victim suite \(SA with a margin score\)\. Sign accuracy averages over nonempty hidden layers\.ModelRecoveredfc1/fc2/fc3/fc4SignaccVictimaccExtaccAgreementtiniest\_relu20/32 \(62\.5%\)8/8⋅\\cdot7/8⋅\\cdot0/8⋅\\cdot5/80\.60199\.90%99\.40%99\.30%tiniest\_leakyrelu25/32 \(78\.1%\)8/8⋅\\cdot8/8⋅\\cdot3/8⋅\\cdot6/80\.771100\.00%100\.00%100\.00%tinier\_relu29/56 \(51\.8%\)16/16⋅\\cdot12/16⋅\\cdot1/16⋅\\cdot0/80\.347100\.00%100\.00%100\.00%tinier\_leakyrelu35/56 \(62\.5%\)16/16⋅\\cdot13/16⋅\\cdot4/16⋅\\cdot2/80\.463100\.00%100\.00%100\.00%tiny\_relu139/256 \(54\.3%\)64/64⋅\\cdot61/64⋅\\cdot14/64⋅\\cdot0/640\.503100\.00%100\.00%100\.00%tiny\_leakyrelu226/256 \(88\.3%\)63/64⋅\\cdot60/64⋅\\cdot49/64⋅\\cdot54/640\.500100\.00%100\.00%100\.00%cifar\_relu499/832 \(60\.0%\)252/256⋅\\cdot247/256⋅\\cdot0/256⋅\\cdot0/640\.53151\.79%46\.60%56\.40%cifar\_leakyrelu807/832 \(97\.0%\)254/256⋅\\cdot252/256⋅\\cdot243/256⋅\\cdot58/640\.53751\.18%46\.60%58\.29%

The six synthetic\-data victims have agreement of at least99\.30%99\.30\\%in Table[6](https://arxiv.org/html/2610.33671#S7.T6)\. In particular,tiny\_relureaches100%100\\%agreement with 139 of 256 directions accepted\. Thus complete direction coverage is not necessary for agreement on these inputs\. Conversely,cifar\_leakyreluhas 807 of 832 directions accepted but only58\.29%58\.29\\%agreement\. High direction coverage is not sufficient for high functional agreement\.

Task accuracy compares predictions with dataset labels, whereas agreement compares them with victim predictions\. They need not coincide because the victim itself makes errors\. The distillation comparison below evaluates prediction agreement and does not identify which retained parameters cause a difference\.

### 7\.3ReLU versus Leaky ReLU Recovery

Table[7](https://arxiv.org/html/2610.33671#S7.T7)compares accepted directions\. Leaky ReLU has 5, 6, 87, and 308 more accepted directions on the four architectures, respectively\. The activation, trained parameters, sampled geometry, and acceptance rule all affect this comparison\. The counts therefore do not isolate the effect of the negative\-side slope alone\. The differences also occur across several layers, rather than only at fc4\.

Table 7:Phase 1–2 recovery, ReLU vs Leaky ReLU\.ArchitectureReLUrecoveredLeakyrecoveredDeltafc4 ReLUfc4 Leakytiniest20/3225/32\+5\+55/86/8tinier29/5635/56\+6\+60/82/8tiny139/256226/256\+87\+870/6454/64cifar499/832807/832\+308\+3080/6458/64

### 7\.4Depth and the Deepest Hidden Layer

In thetiniest\_relurun in Table[6](https://arxiv.org/html/2610.33671#S7.T6), fc3 has no accepted directions and fc4 has five\. Table[8](https://arxiv.org/html/2610.33671#S7.T8)gives an aggregate summary of 40 repeated extractions\. In that summary, fc4 has at least as many accepted directions as fc3 in all 40 runs and strictly more in 35\. These counts describe the sampled runs and do not imply monotonicity of coverage with depth\. The table gives aggregate counts rather than run\-level variation\. We examine variation across recorded seeds separately in Section[7\.8](https://arxiv.org/html/2610.33671#S7.SS8)\. An unseeded search does not by itself establish independence between runs\.

Table 8:Per\-layer recovered neurons over4040reported repeated extractions oftiniest\_relu\.LayerMeanSDRangefc18\.000\.008fc26\.970\.166–7fc32\.251\.370–5fc44\.600\.543–5

To examine inactive coordinates, the reported probe set𝒫\\mathcal\{P\}containsXtest3X\_\{\\mathrm\{test3\}\}and200,000200\{,\}000uniform samples from a threefold\-inflated data bounding box\. We define

Zi​\(𝒫\)=\{j:ηi,j​\(x\)≤0​for every​x∈𝒫\}\.Z\_\{i\}\(\\mathcal\{P\}\)=\\\{j:\\eta\_\{i,j\}\(x\)\\leq 0\\text\{ for every \}x\\in\\mathcal\{P\}\\\}\.This records inactivity on a finite probe set, not permanent inactivity over the input space or over all extraction points\.

For a particular cluster at layerii, letMiM\_\{i\}have rows equal to its transported tangent directions inhi−1h\_\{i\-1\}coordinates\. LetZZcontain coordinates ofhi−1h\_\{i\-1\}that are identically zero in the neighborhoods used by all those tangents\. Under ReLU, columnjjofMiM\_\{i\}is then zero for everyj∈Zj\\in Z\. Writingeje\_\{j\}for the standard coordinate vector gives

span⁡\{ej:j∈Z\}⊆ker⁡Mi\.\\operatorname\{span\}\\\{e\_\{j\}:j\\in Z\\\}\\subseteq\\ker M\_\{i\}\.If the cluster is exact, its true weight rowaaalso satisfiesMi​a𝖳=0M\_\{i\}a^\{\\mathsf\{T\}\}=0\. Therefore

dimkerMi≥\|Z\|\+𝟏\[a𝖳∉span\{ej:j∈Z\}\]\.\\dim\\ker M\_\{i\}\\geq\|Z\|\+\\mathbf\{1\}\\left\[a^\{\\mathsf\{T\}\}\\notin\\operatorname\{span\}\\\{e\_\{j\}:j\\in Z\\\}\\right\]\.\(9\)Indeed, the coordinate vectors are independent, anda𝖳a^\{\\mathsf\{T\}\}contributes one further independent kernel vector precisely when it is outside their span\. The stronger bound1\+\|Z\|1\+\|Z\|needs this additional condition\. If\|Z\|≥1\|Z\|\\geq 1and that condition holds, a test requiring a one\-dimensional kernel rejects the cluster\. Probe\-set inactivity alone does not establish the premises of this argument\.

The reported probe counts\|Z3​\(𝒫\)\|\|Z\_\{3\}\(\\mathcal\{P\}\)\|are 0, 1, and 2 for the three ReLU synthetic architectures\. Their fc4 coverage in Table[7](https://arxiv.org/html/2610.33671#S7.T7)is 5/8, 0/8, and 0/64\. This is consistent with inactive coordinates reducing available constraints, but does not prove that the probe counts cause the observed rejection\. Under Leaky ReLU, a strictly negative coordinate has nonzero slopeα\\alpha, so this particular zero\-column argument no longer applies\. Other rank deficiencies remain possible\.

The three reportedtiny\_leakyreluruns have fc4 counts 54, 56, and 57 and corresponding fc3 counts 49, 50, and 47\. These observations show that accepted direction counts need not decrease with depth\.

### 7\.5Two\-Arm Metric Comparison

Table[9](https://arxiv.org/html/2610.33671#S7.T9)compares assisted reconstruction \(ext\) with a learning\-only distillation baseline \(dis\)\. The baseline is described as using the same label\-query budget\. A complete budget comparison must also count extraction and evaluation queries per input and disclose the assisted arm’s privileged accesses\. Matching cached labels alone does not match the information used by the two procedures\.

Table 9:Per\-metric agreement for the extraction arm and a matched distillation baseline\.ModelC1C\_\{1\}C2C\_\{2\}C3C\_\{3\}extdisextdisextdistiniest\_relu0\.99300\.96100\.21660\.33040\.99100\.9731tiniest\_leakyrelu1\.00000\.98600\.58800\.37561\.00001\.0000tinier\_relu1\.00001\.00000\.52970\.52951\.00001\.0000tinier\_leakyrelu1\.00001\.00000\.61230\.51321\.00001\.0000tiny\_relu \(SA\)1\.00001\.00000\.43350\.33381\.00001\.0000tiny\_relu \(PT\)1\.00001\.00000\.49000\.33381\.00001\.0000tiny\_leakyrelu1\.00001\.00000\.28350\.37411\.00001\.0000cifar\_relu0\.56400\.50330\.33650\.21730\.60120\.5271cifar\_leakyrelu0\.58290\.52860\.33830\.22460\.62920\.5486

On five of the nine rows, bothC1C\_\{1\}andC3C\_\{3\}equal1\.00001\.0000in both arms, so onlyC2C\_\{2\}separates their reported scores\. Distillation exceeds assisted reconstruction onC2C\_\{2\}by0\.11380\.1138fortiniest\_reluand0\.09060\.0906fortiny\_leakyrelu\. The synthetic experiments therefore do not show a uniform reconstruction advantage\.

All three metrics favor assisted reconstruction on both CIFAR victims\. The differences are0\.06070\.0607,0\.11920\.1192, and0\.07410\.0741for ReLU and0\.05430\.0543,0\.11370\.1137, and0\.08060\.0806for Leaky ReLU\. These are differences in the reported scores\. The CIFAR scorecards also record paired discordant counts against the victim reference\. On ReLU, the counts of inputs matched only by reconstruction and only by distillation areb=1851b=1851andc=1244c=1244\. For Leaky ReLU they areb=1852b=1852andc=1309c=1309\. The continuity\-corrected McNemar statistic is\(\|b−c\|−1\)2/\(b\+c\)\(\|b\-c\|\-1\)^\{2\}/\(b\+c\), giving118\.655118\.655and92\.93492\.934\. The corresponding one\-degree\-of\-freedom chi\-square tail probabilities are approximately1\.25×10−271\.25\\times 10^\{\-27\}and5\.41×10−225\.41\\times 10^\{\-22\}\. These summarize paired disagreement in the reported single runs\. They do not measure variation across training seeds, and selection using the evaluation slice limits their interpretation as independent\-test significance\. The reported bootstrap gap intervals are\[4\.99,7\.18\]\[4\.99,7\.18\]and\[4\.36,6\.55\]\[4\.36,6\.55\]percentage points\. We interpret these intervals as within\-run summaries, rather than uncertainty estimates across independently trained victims\.

The simulated\-annealing and parallel\-tempering runs reuse the same extracted artifacts on each synthetic victim\. Five victims have the same reported final sign mean in both arms\. Ontiny\_relu, the mean changes from0\.502756​…0\.502756\\ldotsto0\.518381​…0\.518381\\ldots, whileC1C\_\{1\}andC3C\_\{3\}remain 1\. The off\-distribution score changes from0\.43350\.4335to0\.49000\.4900\. These are single\-run comparisons\. The search can terminate immediately when its scoring agreement is saturated, which limits how much such a comparison tests the two optimizers\.

The evaluation scripts also form a composite score from functional agreement and an extraction\-specific structural term\. We retain the component comparisons in Table[9](https://arxiv.org/html/2610.33671#S7.T9)rather than use that composite as evidence that distillation cannot recover structure\. Its structural term averages layerwise absolute cosine, the sign mean, and accepted\-row coverage, and assigns the distillation arm zero because it has no extraction records\. That zero is an accounting convention, not a measured absence of aligned parameters\. Assigning it creates a built\-in advantage for the extraction arm\. A fair structural comparison would independently match and assess the distillation weights, allowing the same permutation and scale ambiguities\.

### 7\.6Additive Reconstruction Ablation

Table[10](https://arxiv.org/html/2610.33671#S7.T10)builds the Phase\-3 reconstruction one component at a time on eachmake\_blobsvictim, recording agreement on the evaluation and selection slice at every stage\. This ablation study is a separate execution of the pipeline, thus its final row need not coincide exactly with Table[6](https://arxiv.org/html/2610.33671#S7.T6)\.

Table 10:Additive Phase\-3 ablation on the sixmake\_blobsvictims, evaluated on the selection slice X\_test3\.ArchStageReLU AgrLeaky ReLU AgrtiniestRAW \(Phase 1\+2\)12\.40%12\.50%\+ bias recovery22\.80%2\.80%\+ output\-layer fit92\.40%98\.00%\+ sign search99\.40%97\.90%\+ masked refinement99\.70%98\.90%distillation baseline96\.00%98\.60%tinierRAW \(Phase 1\+2\)23\.30%25\.00%\+ bias recovery0\.00%6\.58%\+ output\-layer fit99\.94%99\.92%\+ sign search100\.00%100\.00%\+ masked refinement100\.00%100\.00%distillation baseline100\.00%100\.00%tinyRAW \(Phase 1\+2\)3\.00%10\.68%\+ bias recovery14\.34%12\.60%\+ output\-layer fit100\.00%100\.00%\+ sign search100\.00%100\.00%\+ masked refinement100\.00%100\.00%distillation baseline100\.00%100\.00%

Output\-layer fitting produces the largest agreement increase in this ablation\. Fortiny\_relu, the score rises from14\.34%14\.34\\%after bias estimation to100\.00%100\.00\\%after the fit\. The3\.00%3\.00\\%raw score precedes both steps\. A successful decoder fit shows that the reconstructed features separate these labeled samples, not that they equal the victim’s features\.

Fortinierandtiny, agreement after fitting is at least99\.92%99\.92\\%, leaving at most0\.080\.08percentage points before saturation\. This score cannot usefully distinguish later improvements on those samples\. Ontiniest, sign search changes agreement by\+7\.00\+7\.00percentage points under ReLU and−0\.10\-0\.10under Leaky ReLU\. Refinement adds\+0\.30\+0\.30and\+1\.00\+1\.00points, respectively\. The decrease on the Leaky ReLU row is possible because search selects by a separate cached scoring set, rather than guaranteeing improvement on the evaluation slice\.

Bias estimation sometimes lowers agreement before the output layer is fitted\. At that stage the decoder remains randomly initialized, so the score alone is not a reliable measure of the bias estimates\. Finally, reconstruction and distillation tie ontinierandtiny\. Reconstruction exceeds distillation by3\.703\.70and0\.300\.30percentage points on the twotiniestrows\.

### 7\.7Shortcut and Reconstruction Comparisons

We examine a separate batch of 48 synthetic\-victim runs, comprising four shortcut comparisons, six victims, and two arms\. We vary D1, D2, D3, and D5 individually, retaining the true\-prefix and true\-walk assistance\. The ON configurations retain the corresponding assistance, and the OFF configurations disable the named shortcut\. The batch evaluates two phase\-3 protocols on each shared phase\-1 and phase\-2 output\. The joint\-fit protocol fits on the union of fitting and scoring inputs and uses the reporting slice for early stopping\. The split\-fit protocol fits onXtestX\_\{\\mathrm\{test\}\}, scores onXtest2X\_\{\\mathrm\{test2\}\}, uses those two slices for refinement, and reservesXtest3X\_\{\\mathrm\{test3\}\}for final reporting\. This separation concerns data use in phase 3, not the privileged accesses in extraction\.

The tiniest fitting, scoring, and reporting sets have 2,000, 2,000, and 1,000 inputs\. The corresponding sizes for tinier and tiny are 10,000, 5,000, and 5,000\. The recorded split\-fit phase\-3 access logs contain no requests outside their phase\-specific input contract\. No identical input rows occur across these slices\. This overlap check does not by itself establish statistical independence\. The input guard also returns logits even though the inspected consumers reduce them to predicted labels\. A strict label\-only interface must restrict the return value itself\.

We check the predictions directly from the numeric checkpoint parameters\. Across 1,344 models, all 4,032 prediction vectors on the three input slices agree with the checkpoint evaluations, and all 576 stage\-agreement values agree with the recomputed values\. Table[11](https://arxiv.org/html/2610.33671#S7.T11)gives one D1 ON trace per victim under the split\-fit protocol\. This comparison uses gradient masking without row restoration\.

Table 11:D1 ON split\-fit reconstruction\. Accepted rows are counted across all hidden layers\. Agreement is measured on the reserved reporting slice\.VictimAccepted rowsOutput fitSign searchRefinementtiniest\_relu20/3299\.60%99\.90%99\.90%tiniest\_leakyrelu20/3297\.90%99\.50%99\.20%tinier\_relu28/56100\.00%100\.00%100\.00%tinier\_leakyrelu31/56100\.00%100\.00%100\.00%tiny\_relu142/256100\.00%100\.00%100\.00%tiny\_leakyrelu231/256100\.00%100\.00%100\.00%

The D1 comparison on tiny ReLU accepts 142/256 rows with assistance and 24/256 without it\. For tiny Leaky ReLU, the corresponding counts are 231/256 and 88/256\. These differences describe the assisted implementation under distinct search realizations\. The split\-fit D1 tiniest Leaky ReLU ON run has final agreement0\.9920\.992, while its joint\-fit counterpart has agreement0\.9950\.995\. Thus a claim that every D1 run reaches0\.9950\.995is false across the two protocols\.

The four ON configurations yield four canonical repetitions per victim\. For tiniest ReLU, their fc1–fc4 means are88,77,11, and4\.754\.75, and their sample standard deviations are00,00,1\.15471\.1547, and0\.50\.5\. These observations show that accepted counts need not decrease at every layer\. These four repetitions are distinct from the 40\-run experiment in Table[8](https://arxiv.org/html/2610.33671#S7.T8)\. The unseeded dual\-point searches also prevent a paired interpretation of ON–OFF differences based on a common recorded random stream\. Across the batch, 2,063 of 3,002 decided rows have correct recovered\-frame votes\. The operational orientation conversion uses victim rows as described in Section[3\.3](https://arxiv.org/html/2610.33671#S3.SS3)\.

The exhaustive sign\-block experiment enumerates 256 assignments in each of 27 blocks\. It considers a fixed decoder fitted on the union pool, a fixed decoder fitted on the fitting slice alone, and a decoder refitted on that slice for every assignment\. The initial assignment maximizes scoring agreement, allowing ties, in 24/27 blocks with either fixed decoder and in 3/27 with refitting\. All 81 block/mode agreement records recompute exactly\. In six selected blocks, replacing scoring and reporting labels with random labels leaves the stored decoder coefficients and candidate predictions unchanged in all 18 checked cases\. This supports separation of candidate construction from scoring for those block procedures\. Proposition[7](https://arxiv.org/html/2610.33671#Thmproposition7)additionally requires the stated sampling assumptions\. The union\-fitted decoder depends on scoring labels and does not meet that proposition’s fixed\-family condition\. These block tests also do not establish global parameter identification or the optimization accuracy of the full adaptive search\.

The split\-fit phase\-3 logs record 720 calls and 3,537,152 input evaluations across the 48 runs, ranging from 29,024 to 96,024 evaluations per run\. Each repeated input is counted again\. The harness passes the constant 20,000 as both its budget and its used\-query count\. That constant does not equal these recorded totals\. We must distinguish pool size, distinct queried inputs, repeated evaluations, and privileged reads before making a comparison at equal attack cost\. A phase\-3 ledger also does not supply the entire cost of phases 1 and 2\.

### 7\.8Row restoration, victim\-free walks, and seeded repetitions

We evaluate row restoration, replacement sign walks, refinement protocols, and recovery depth\. The restoration and walk comparisons use shared extraction outputs to compare downstream procedures\. The refinement and depth experiments vary recorded dual\-search seeds\. We specify the protocol, sample size, and information access for each comparison\.

##### Exact row restoration\.

We compare gradient masking with the restoration update in Eq\. \([8](https://arxiv.org/html/2610.33671#S6.E8)\), under the joint\-fit and split\-fit protocols on 48 extraction runs\. All 48 restoration runs per protocol have zero recorded change in the accepted row coordinates after every recorded optimizer step, with equality at all 672 checked checkpoints\. The restoration reference is reset at each refinement call, so intentional sign changes between calls remain possible\. For masking alone, the largest recorded coordinate change is approximately1\.54×10−41\.54\\times 10^\{\-4\}under the joint\-fit protocol and1\.23×10−41\.23\\times 10^\{\-4\}under the split\-fit protocol\. Restoration changes final agreement by at most0\.0040\.004and0\.0030\.003, respectively, in these paired comparisons\. These small observed differences do not establish statistical equivalence\. Proposition[10](https://arxiv.org/html/2610.33671#Thmproposition10)proves the preservation property for the specified update independently of its effect on agreement\.

##### Replacing victim\-weight access in the walk\.

We compare the canonical walk with two replacements on the extraction outputs from 24 runs with the corresponding shortcuts enabled\. One replacement uses a reconstructed hidden stack with the victim output layer\. The other uses reconstructed hidden layers and a label\-fitted output layer\. The latter is the victim\-free walk variant\. The replacement uses a guard that blocks victim\-model loading and parameter access\. In the guard tests, replacing victim\-model files leaves the walk outputs unchanged\. Its input artifacts still originate from assisted extraction, so it is not an end\-to\-end hard\-label experiment\. Fitting its provisional heads reuses a label pool of 2,000 or 10,000 inputs\. One batch call to obtain that pool counts as 2,000 or 10,000 label requests\.

Across decided rows, the canonical walk has 1,255 correct votes out of 1,819, or68\.99%68\.99\\%\. The victim\-free replacement has 836 out of 1,810, or46\.19%46\.19\\%\. The replacement retaining the victim output layer has 876 out of 1,820, or48\.13%48\.13\\%\. These denominators differ, so the pooled percentages are not a paired comparison on identical decided rows\. Every replacement walk terminates by exhausting its available dual points\. The resulting comparisons include incomplete walks\. Exhausting a point pool is not a certified hitting\-distance comparison\. After split\-fit reconstruction with row restoration, the victim\-free variant nevertheless has mean final agreement0\.996580\.99658across the 24 runs\. Thus high functional agreement coexists with poor sign accuracy in this experiment\. Proposition[4](https://arxiv.org/html/2610.33671#Thmproposition4)specifies a sufficient local stability condition, but this experiment does not establish the required slope, perturbation, or gap bounds\.

##### Refinement and independent reporting\.

We repeat the two tiniest victims with D1 enabled or disabled and dual\-search seeds 1–10, giving 40 extraction runs\. Each is evaluated under four phase\-3 protocols\. The joint\-fit protocol fits onXtest∪Xtest2X\_\{\\mathrm\{test\}\}\\cup X\_\{\\mathrm\{test2\}\}and stops on the 1,000\-input stopping set\. The split\-fit protocol fits onXtestX\_\{\\mathrm\{test\}\}, selects signs onXtest2X\_\{\\mathrm\{test2\}\}, and also stops on that scoring set\. Thus it separates reporting from selection, but does not give refinement a separate stopping set\. A third variant adds exact row restoration and includes the entering model among the checked checkpoints\. A fourth also uses a separate stopping poolXvalX\_\{\\mathrm\{val\}\}of 2,000 inputs, of which 1,024 are labelled for the stopping rule\. All four variants are scored on the same held\-out pool of 5,000 inputs, disjoint from the fitting, scoring, and stopping pools and unused during fitting or selection\.

Table 12:Final agreement on 5,000 held\-out inputs across 40 seeded tiniest runs\. The extraction and canonical sign walk retain privileged access\.Phase\-3 protocolMinimumMaximumRuns below0\.9950\.995Joint\-fit0\.99180\.99984/40Split\-fit0\.98900\.999215/40Split\-fit, restoration and initial checkpoint0\.99100\.999616/40Also separate stopping pool0\.99100\.999813/40In the D1 ON tiniest Leaky ReLU split\-fit run ending at0\.9920\.992on 1,000 reporting inputs, refinement increases the errors from five to eight\. On its stopping set, errors rise from ten to eighteen because the entering model was not eligible for selection\. Including that model prevents this deterioration on the stopping sample\. Across the 80 refinement executions with initial\-checkpoint selection, the returned stopping agreement is never below its entering value\. It is higher in 62 executions and equal in 18\. The held\-out benefit is smaller and not uniform\. The variant with restoration and a separate stopping pool improves agreement over split\-fit by0\.0004250\.000425on average, with 22 runs improving, 14 worsening, and four unchanged\. This comparison changes restoration and checkpoint selection together, so it does not isolate their separate effects\.

The0\.98900\.9890minimum in Table[12](https://arxiv.org/html/2610.33671#S7.T12)applies to the 40 seeded runs in that experiment\. The D1 ON tiniest Leaky ReLU model with agreement0\.9920\.992on 1,000 inputs has agreement0\.98820\.9882on the 5,000\-input pool\. Thus the minimum in the table is specific to its experimental configuration and seed set\. Likewise, the joint\-fit protocol’s higher average score on its own stopping sample does not by itself quantify the causal effect of selection\. Its score on that sample is not an independent test estimate\. Both protocols’ scores on the held\-out pool avoid this direct reuse\.

##### Recovery depth across recorded seeds\.

We run the canonical configuration on six victims with seeds 1–8, giving 48 extraction runs and three phase\-3 protocols per run\. The 16 tiniest extractions repeat point\-for\-point the corresponding seed cases in the 40\-run study and are not additional independent repetitions\. For these repeated seed cases, the dual points, recovered rows, and signs have matching hashes, and the phase\-3 results agree\. Search rounds use recorded seeds derived from the experiment seed and round index\. These repetitions use the same machine, library versions, and thread setting\. Reproducibility across other configurations is not established\. The runs use a CPU, float64 arithmetic, three dual workers, and two threads per process\. All shortcuts remain enabled\. This seeded experiment is restricted to the six synthetic victims\.

Table 13:Accepted rows across eight recorded dual\-search seeds per synthetic victim\. Parentheses give means\.VictimTotal accepted rowsDeepest hidden layerTiniest ReLU20–23 \(22\.125\)4–5 \(4\.875\)Tiniest Leaky ReLU21–25 \(22\.875\)2–5 \(3\.250\)Tinier ReLU29–31 \(30\.000\)0 \(0\.000\)Tinier Leaky ReLU31–34 \(32\.875\)0–1 \(0\.250\)Tiny ReLU131–141 \(136\.000\)0 \(0\.000\)Tiny Leaky ReLU221–229 \(225\.875\)51–57 \(54\.375\)The largest architecture consistently shows more accepted directions under Leaky ReLU, including substantial coverage in its deepest layer\. On tiniest, the deepest\-layer ordering is reversed on average\. The single\-run Leaky ReLU advantage in Table[6](https://arxiv.org/html/2610.33671#S7.T6)therefore does not imply a uniform advantage across layers or seeds\. These are comparisons of fixed trained victims\. They do not isolate activation choice from differences in training or learned parameters\. The eight seeded runs per victim are distinct from the 40\-run experiment in Table[8](https://arxiv.org/html/2610.33671#S7.T8)\. All 32 tinier and tiny runs have final agreement 1 on their 5,000 reporting inputs under each protocol\. This is agreement on those finite samples, not equality of the classifiers on every input\.

##### Measured access to the victim\.

We classify each victim access by its returned information\. Label\-only responses count as hard\-label requests\. Margins, gradients, hidden activations, and parameters count as privileged access\. Unclassified accesses are assigned to the privileged category\. Across the 48 runs, the phase\-1 counter matches the separate dual\-search ledger for labels, detection margins, Newton margins, and gradients\. For these seeded runs, the phase\-3 guard encodes only the predicted label as a one\-hot vector, with a single entry equal to one and all other entries zero\. The counter checks find no untyped accesses or unassigned search rows\. These consistency checks concern the implemented access paths\. They do not establish that an arbitrary implementation has no other access path\.

Phase 1 requests between 704,964 and 12,027,750 hard labels per run\. Phase 3 makes 15 batched label calls per protocol execution\. Nine calls score sign\-search stages, two fit output layers, two support short refinements, one supports final refinement, and one obtains stopping labels\. Under the joint\-fit protocol, this totals 57,000 requests on tiniest and 211,024 on tinier or tiny\. Its distinct phase\-3 labelled\-input counts are 5,000 and 16,024\. Under the split\-fit protocols, the corresponding repeated\-request totals are 29,024 and 96,024\. The canonical sign walk makes no label queries, but reads victim weights and uses them in its simulated evaluations\. Zero label queries at that stage is therefore not evidence of zero privileged access\.

We do not combine heterogeneous accesses into a hard\-label query budget\. Detection\-margin and gradient channels record input rows, while parameter channels count scalar entries read, often repeatedly\. Further channels include clustering activations, true prefix parameters and prefix evaluations, acceptance and scaling parameters, and walk orientation rows\. A subtotal containing label requests and the D1 and D2 channels excludes those additional accesses and cannot represent the total victim\-access cost\. The distillation baseline is also assisted in this sweep, starting from victim biases and the victim output layer\. A comparison with independent label\-only distillation requires a baseline without those initial parameters\. A cost comparison must specify whether it counts batched calls, input\-level requests, distinct inputs, or parameter entries\.

## 8Conclusion

We combined assisted weight\-direction extraction with bias estimation, sign search, output\-layer fitting, and gradient\-masked refinement\. We proved conditional bounds for direction estimation with approximate prefixes, preservation of local hitting\-distance orderings, prediction changes from aligned parameter errors, and sign selection with independent scoring data\. We also specified a projected update that enforces exact row preservation\. The seeded experiments show substantially more accepted directions under Leaky ReLU on the largest synthetic architecture, but no uniform deepest\-layer advantage on the smallest\. The reconstruction procedure produces a complete classifier, but the privileged accesses used by extraction prevent these experiments from establishing an end\-to\-end attack using labels alone\.

The synthetic\-data comparisons do not show a uniform advantage over distillation\. The two CIFAR comparisons favor assisted reconstruction on all three reported functional metrics, although agreement remains below60%60\\%\. These results support studying reconstruction from partial directions\. They do not establish exact parameter recovery or uniformly reliable sign recovery\.

A strict\-oracle evaluation must replace the true prefixes and ground\-truth acceptance checks, account for queries per input, and reserve a final test slice that is not used for early stopping\. The gradient\-masked update changes row magnitudes\. The restoration variant preserves the accepted rows at every recorded update and checked checkpoint, consistent with Eq\. \([8](https://arxiv.org/html/2610.33671#S6.E8)\)\. Including the entering checkpoint prevents deterioration on the stopping sample, while the seeded results show that held\-out agreement may still decrease\. The victim\-free walk replacement removes the operational victim\-weight access in the guard tests, but gives less accurate signs and still receives assisted extraction artifacts\. High agreement after reconstruction therefore does not resolve the sign\-recovery problem\. The local hitting\-distance stability bound identifies the perturbation and gap conditions needed to preserve a prescribed distance vote, without assuming that the vote is correct\. Further work can compare the label\-trained output layer with recovery from decision\-boundary constraints\[[4](https://arxiv.org/html/2610.33671#bib.bib4)\]and determine when the local rank conditions suffice for reliable direction estimation\.

#### Data availability

The experimental data will be made available at\[[2](https://arxiv.org/html/2610.33671#bib.bib2)\]\. The implementation and experimental scripts are available at\[[1](https://arxiv.org/html/2610.33671#bib.bib1)\]\.

## References

- \(1\)B\. Biswas, Implementation of the extraction and reconstruction procedures\.[https://github\.com/biprakode/Hard\_Label\_Work/blob/main/reviewer\_readme\.md](https://github.com/biprakode/Hard_Label_Work/blob/main/reviewer_readme.md)
- \(2\)B\. Biswas, Experimental data for Extraction meets Distillation\. Data will be made available at[https://mega\.nz/folder/xjYg1IrK\#ExzAmC\_uNjGt\-yl3qWNIAA](https://mega.nz/folder/xjYg1IrK#ExzAmC_uNjGt-yl3qWNIAA)\.
- \(3\)Canales\-Martínez, I\. A\., Chávez\-Saab, J\., Hambitzer, A\., Rodríguez\-Henríquez, F\., Satpute, N\., Shamir, A\.: Polynomial time cryptanalytic extraction of neural network models\. In EUROCRYPT 2024, Part III\. LNCS, vol\. 14653, pp\. 3–33\. Springer, 2024\.[https://doi\.org/10\.1007/978\-3\-031\-58734\-4\_1](https://doi.org/10.1007/978-3-031-58734-4_1)
- \(4\)Canales\-Martínez, I\. A\., Santos, D\.: Extracting some layers of deep neural networks in the hard\-label setting\. In LATINCRYPT 2025\. LNCS, vol\. 16129, pp\. 399–421\. Springer \(2026\)\.[https://doi\.org/10\.1007/978\-3\-032\-06754\-8\_15](https://doi.org/10.1007/978-3-032-06754-8_15)
- \(5\)Carlini, N\., Jagielski, M\., Mironov, I\.: Cryptanalytic extraction of neural network models\. In CRYPTO 2020, Part III\. LNCS, vol\. 12172, pp\. 189–218\. Springer \(2020\)\.[https://doi\.org/10\.1007/978\-3\-030\-56877\-1\_7](https://doi.org/10.1007/978-3-030-56877-1_7)
- \(6\)Carlini, N\., Chávez\-Saab, J\., Hambitzer, A\., Rodríguez\-Henríquez, F\., Shamir, A\.: Polynomial time cryptanalytic extraction of deep neural networks in the hard\-label setting\. In EUROCRYPT 2025, Part I\. LNCS, vol\. 15601, pp\. 364–396\. Springer \(2025\)\.[https://doi\.org/10\.1007/978\-3\-031\-91107\-1\_13](https://doi.org/10.1007/978-3-031-91107-1_13)\. Code\-base:[https://github\.com/Jchavezsaab/hard\-label\-dnn\-extraction](https://github.com/Jchavezsaab/hard-label-dnn-extraction)\.
- \(7\)Chen, J\., Gu, Q\.: RayS: a ray searching method for hard\-label adversarial attack\. In 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining \(KDD ’20\), pp\. 1739–1747\. ACM \(2020\)\.[https://doi\.org/10\.1145/3394486\.3403225](https://doi.org/10.1145/3394486.3403225)
- \(8\)Chen, J\., Jordan, M\.I\., Wainwright, M\.J\.: HopSkipJumpAttack: a query\-efficient decision\-based attack\. In: 2020 IEEE Symposium on Security and Privacy \(SP\), pp\. 1277–1294\. IEEE \(2020\)\.[https://doi\.org/10\.1109/SP40000\.2020\.00045](https://doi.org/10.1109/SP40000.2020.00045)
- \(9\)Chen, Y\., Dong, X\., Guo, J\., Shen, Y\., Wang, A\., Wang, X\.: Hard\-label cryptanalytic extraction of neural network models\. In ASIACRYPT 2024, Part VIII\. LNCS, vol\. 15491, pp\. 207–236\. Springer, Singapore \(2025\)\.[https://doi\.org/10\.1007/978\-981\-96\-0944\-4\_7](https://doi.org/10.1007/978-981-96-0944-4_7)
- \(10\)Chen, Y\., Dong, X\., Ma, R\., Shen, Y\., Wang, A\., Wang, X\., Yu, H\.: Delving into cryptanalytic extraction of PReLU neural networks\. In ASIACRYPT 2025, Part II, LNCS, vol\. 16246, pp\. 576–607, Springer\.[https://doi\.org/10\.1007/978\-981\-95\-5096\-8\_18](https://doi.org/10.1007/978-981-95-5096-8_18)
- \(11\)Foerster, H\., Mullins, R\.D\., Shumailov, I\., Hayes, J\.: Beyond slow signs in high\-fidelity model extraction\. In: Advances in Neural Information Processing Systems 37 \(NeurIPS 2024\), pp\. 19496–19522\. Curran Associates \(2024\)
- \(12\)Geyer, C\. J\.: Markov chain Monte Carlo maximum likelihood\. In: Computing Science and Statistics, Proceedings of the 23rd Symposium on the Interface \(1991\)\.[https://www\.stat\.umn\.edu/geyer/f05/8931/c\.pdf](https://www.stat.umn.edu/geyer/f05/8931/c.pdf)
- \(13\)Hoeffding, W\.: Probability inequalities for sums of bounded random variables\. Journal of the American Statistical Association 58\(301\), 13–30 \(1963\)\.[https://doi\.org/10\.1080/01621459\.1963\.10500830](https://doi.org/10.1080/01621459.1963.10500830)
- \(14\)Jagielski, M\., Carlini, N\., Berthelot, D\., Kurakin, A\., Papernot, N\.: High accuracy and high fidelity extraction of neural networks\. In: 29th USENIX Security Symposium \(USENIX Security 20\), pp\. 1345–1362\. USENIX Association \(2020\)
- \(15\)Kirkpatrick, S\., Gelatt, C\. D\., Jr\., Vecchi, M\. P\.: Optimization by simulated annealing\. Science 220\(4598\), 671–680 \(1983\)\.[https://doi\.org/10\.1126/science\.220\.4598\.671](https://doi.org/10.1126/science.220.4598.671)
- \(16\)Krizhevsky, A\.: Learning multiple layers of features from tiny images\. Technical Report, University of Toronto \(2009\)\.[https://www\.cs\.toronto\.edu/~kriz/cifar\.html](https://www.cs.toronto.edu/~kriz/cifar.html)
- \(17\)Liu, H\., Siproudhis, A\., Experton, S\., Lorenz, P\., Boura, C\., Peyrin, T\.: Navigating the deep: end\-to\-end extraction on deep neural networks\. In: Advances in Cryptology – EUROCRYPT 2026\. Pages 482–512, LNCS 16546, Springer \(2026\)\.[https://arxiv\.org/abs/2506\.17047](https://arxiv.org/abs/2506.17047)
- \(18\)Loshchilov, I\., Hutter, F\.: Decoupled weight decay regularization\. In: International Conference on Learning Representations \(ICLR\), 2019\.[https://arxiv\.org/abs/1711\.05101](https://arxiv.org/abs/1711.05101)
- \(19\)Maas, A\. L\., Hannun, A\.Y\., Ng, A\.Y\.: Rectifier nonlinearities improve neural network acoustic models\. In: ICML Workshop on Deep Learning for Audio, Speech and Language Processing \(2013\)
- \(20\)Qi, X\., Lei, H\., Wei, L\., Sun, X\., Wang, M\.: Cryptanalytic Extraction of Neural Networks with Various Activation Functions\. IACR Transactions on Symmetric Cryptology, 2026\(1\), pp\. 468–505, 2026\.[https://tosc\.iacr\.org/index\.php/ToSC/article/view/12793](https://tosc.iacr.org/index.php/ToSC/article/view/12793)

## Appendix AMathematical Results and Implementation

We give the correspondence between the mathematical results and the implementation\[[1](https://arxiv.org/html/2610.33671#bib.bib1)\]\. The propositions state sufficient conditions for their conclusions\. The numerical experiments evaluate the implemented procedures under the configurations in Section[7](https://arxiv.org/html/2610.33671#S7)\.

The signature\-recovery routinesintersect,is\_consistent, andextract\_weightsinsignature\_recovery/recover\_weights\.pyimplement the tangent\-space calculations of Proposition[1](https://arxiv.org/html/2610.33671#Thmproposition1)\. The routinesforward\_aroundandrelu\_aroundtransport those constraints through the prefix\. Proposition[2](https://arxiv.org/html/2610.33671#Thmproposition2)bounds the effect of errors in that transport\. Section[4\.2](https://arxiv.org/html/2610.33671#S4.SS2)specifies the Leaky ReLU slopes and the dimensions of the local maps\.

The sign walk insign\_recovery\.pycompares distances on the two sides of a critical hyperplane\. Equation \([3](https://arxiv.org/html/2610.33671#S3.E3)\) describes the local velocity asymmetry under matched controls\. Proposition[4](https://arxiv.org/html/2610.33671#Thmproposition4)gives conditions under which an approximate affine calculation preserves a distance ordering\. Lemma[3](https://arxiv.org/html/2610.33671#Thmproposition3)bounds majority\-vote error when the individual votes are independent and have a positive correctness advantage\. These are separate conditions\. Stability of a distance ordering does not imply correctness of its sign interpretation\.

The routinerecover\_biases\_from\_dualsinbias\_recovery\.pyuses Eq\. \([7](https://arxiv.org/html/2610.33671#S6.E7)\)\. The routines\_score\_objectiveand\_flip\_neuroninsign\_search\.pyevaluate candidates and jointly change a row and its bias\. Proposition[7](https://arxiv.org/html/2610.33671#Thmproposition7)applies when the candidate family is fixed independently of the scoring sample\. The routinerecover\_output\_layerinoutput\_layer\_recovery\.pyfits the decoder to labels\. Proposition[9](https://arxiv.org/html/2610.33671#Thmproposition9)bounds prediction changes from aligned parameter errors\.

Proposition[10](https://arxiv.org/html/2610.33671#Thmproposition10)describes the masked AdamW update inrefinement\.py\. Equation \([8](https://arxiv.org/html/2610.33671#S6.E8)\) specifies exact row restoration\. The checkpoint\-selection rule in Section[6\.4](https://arxiv.org/html/2610.33671#S6.SS4)retains the entering model as a candidate and compares checkpoints on a fixed stopping sample\. Its deterministic guarantee concerns that sample, while held\-out agreement is evaluated separately\.