Adversarial Training Without Input Gradients via Low-Rank Householder Expansions

# Adversarial Training Without Input Gradients via Low-Rank Householder Expansions
Source: [https://arxiv.org/html/2608.26963](https://arxiv.org/html/2608.26963)
Tiana C\. JohnsonAddress:Department of Mathematics, Washington University in St\. Louis, St\. Louis, MO 63130Email address:[j\.tiana@wustl\.edu](mailto:[email protected])Donsub RimAddress:Department of Mathematics, Washington University in St\. Louis, St\. Louis, MO 63130Email address:[rim@wustl\.edu](mailto:[email protected])

Date: August 27, 2026

###### Abstract\.

This work concerns adversarial training against the small\-norm adversarial examples that arise from the inherent input instability of a trained deep neural network\. Examples in this class are small as measured in the relativeℓ2\\ell^\{2\}\-norm, and therefore lie in the neighborhood of the input on which the model acts approximately linearly, the regime in which the perturbation remains imperceptible\. We first show that such examples can be computed directly from the trained network parameters, without input gradient iterations, by means of a linearization called the low\-rank Householder expansion \(LRHE\)\. The expansion describes the composed affine map rather than any individual layer, and the directions it identifies are read from the activation pattern already available in the forward pass\. We then propose a simple adversarial training scheme built on this construction\. No differentiation with respect to the input is performed at any point: training requires only additional forward evaluations, with weight parameters updated by the standard backward pass, and the inner maximization of the usual min\-max formulation is eliminated entirely\. That such a regularizer exists is our main finding: the methods that dispense with the inner search all obtain their local geometry by differentiating with respect to the input, and we show this is not necessary\. The regularizer costs the equivalent of2\.82\.8PGD steps per epoch, an8\.7×8\.7\\timesreduction relative to 40\-step adversarial training on MNIST and below the cost of 3\-step training\. The resulting models match three\-step PGD adversarial training for relativeℓ2\\ell^\{2\}budgetsε≤0\.02\\varepsilon\\leq 0\.02and 40\-step training forε≤0\.012\\varepsilon\\leq 0\.012, falling away beyond, consistent with the locality of the expansion\.

## 1\.Introduction

The observation that deep neural networks can be made to misclassify inputs altered by perturbations too small to matter to a human observer is now more than a decade old\[[27](https://arxiv.org/html/2608.26963#bib.bib27),[9](https://arxiv.org/html/2608.26963#bib.bib9)\], and it has proved remarkably resistant to resolution\. The intervening years have produced a large number of proposed defenses and an almost equally large number of demonstrations that they do not work\. A survey of defenses presented at a single conference\[[3](https://arxiv.org/html/2608.26963#bib.bib3)\]found that the majority relied on obfuscated gradients and could be circumvented once that reliance was recognized, and similar conclusions were reached for later defenses\[[28](https://arxiv.org/html/2608.26963#bib.bib28)\]\. What has resisted this attrition is adversarial training in the min\-max formulation of\[[15](https://arxiv.org/html/2608.26963#bib.bib15)\], which augments each parameter update with an inner search for the worst\-case perturbation\. It has consistently withstood adaptive evaluation, and is the standard against which subsequent proposals are measured\[[4](https://arxiv.org/html/2608.26963#bib.bib4),[7](https://arxiv.org/html/2608.26963#bib.bib7)\]\. The persistence of the problem is therefore not a matter of an effective defense being unknown, but of its cost\.

That cost has a specific and unvarying structure\. The inner maximization admits no closed form and is approximated byKKsteps of projected gradient ascent, each requiring a gradient of the loss with respect to the input, so that adversarial training is roughlyKKtimes more expensive than ordinary training\. Efforts to reduce this expense fall into three groups\. The first reuses computation across updates: perturbations may be carried between minibatches so that the adversarial and parameter updates share a backward pass\[[23](https://arxiv.org/html/2608.26963#bib.bib23)\], or the inner loop confined to the first layer through an optimal\-control formulation\[[35](https://arxiv.org/html/2608.26963#bib.bib35)\]\. The second reducesKKdirectly, usually to one, and then repairs the resulting instability; FGSM with random initialization and a cyclic learning rate has been shown to suffice\[[34](https://arxiv.org/html/2608.26963#bib.bib34)\], and the catastrophic overfitting that single\-step methods exhibit can be prevented by penalizing gradient misalignment\[[2](https://arxiv.org/html/2608.26963#bib.bib2)\]\. The third dispenses with the inner maximization altogether, replacing it with a penalty on the local geometry of the loss: input\-gradient regularization\[[22](https://arxiv.org/html/2608.26963#bib.bib22)\], Jacobian regularization\[[11](https://arxiv.org/html/2608.26963#bib.bib11)\], curvature regularization\[[17](https://arxiv.org/html/2608.26963#bib.bib17)\], and local linearity regularization\[[18](https://arxiv.org/html/2608.26963#bib.bib18)\], the last of which explicitly trades a geometric penalty for a reduction in the number of projected gradient descent \(PGD\) steps\. What unites these otherwise dissimilar approaches is that every one of them obtains its information about the network’s local behavior by differentiating with respect to the input\. They differ in how many such derivatives they take, not in whether they take them\.

We show that for perturbations of small relativeℓ2\\ell^\{2\}norm this is unnecessary\. Our starting point is the low\-rank Householder expansion \(LRHE\) introduced in\[[21](https://arxiv.org/html/2608.26963#bib.bib21)\]\. A ReLU network is piecewise affine\. On the region of input space where the activation pattern is constant, the network acts as a single affine map\. Concretely, one may write a feedforward ReLU neural network, with the bias terms omitted, as multiplication by the matrix

\(1\.1\)F=WLΣL−1WL−1⋯Σ1W1,F=W\_\{L\}\\Sigma\_\{L\-1\}W\_\{L\-1\}\\cdots\\Sigma\_\{1\}W\_\{1\},where eachΣℓ\\Sigma\_\{\\ell\}is the diagonal00–11matrix recording which units at layerℓ\\ellare active, andWℓW\_\{\\ell\}are weight matrices\. A change of activation pattern perturbs the network’s local action by a term of rank one\. To see this, fix a layerℓ\\elland split the product \([1\.1](https://arxiv.org/html/2608.26963#S1.E1)\) on either side ofΣℓ\\Sigma\_\{\\ell\},

\(1\.2\)F=CℓΣℓBℓ,Cℓ=WLΣL−1⋯Wℓ\+1,Bℓ=WℓΣℓ−1⋯Σ1W1,F=C\_\{\\ell\}\\Sigma\_\{\\ell\}B\_\{\\ell\},\\quad C\_\{\\ell\}=W\_\{L\}\\Sigma\_\{L\-1\}\\cdots W\_\{\\ell\+1\},\\quad B\_\{\\ell\}=W\_\{\\ell\}\\Sigma\_\{\\ell\-1\}\\cdots\\Sigma\_\{1\}W\_\{1\},so thatBℓB\_\{\\ell\}collects the factors before layerℓ\\ellandCℓC\_\{\\ell\}those after\. Suppose the input is displaced across a single activation boundary, so that unitiiof layerℓ\\ellchanges from active to inactive while every other unit retains its state\. ThenΣℓ\\Sigma\_\{\\ell\}is replaced byΣℓ−ei​ei⊤\\Sigma\_\{\\ell\}\-e\_\{i\}e\_\{i\}^\{\\top\}whereeie\_\{i\}is theii\-th standard basis vector, and the matrix representation of the network on inputs in the new region is

\(1\.3\)F1=Cℓ​\(Σℓ−ei​ei⊤\)​Bℓ=F−\(Cℓ​ei\)​\(Bℓ⊤​ei\)⊤\.F\_\{1\}=C\_\{\\ell\}\(\\Sigma\_\{\\ell\}\-e\_\{i\}e\_\{i\}^\{\\top\}\)B\_\{\\ell\}=F\-\(C\_\{\\ell\}e\_\{i\}\)\(B\_\{\\ell\}^\{\\top\}e\_\{i\}\)^\{\\\!\\top\}\.The perturbation is thus rank one, with left factorCℓ​eiC\_\{\\ell\}e\_\{i\}and right factorBℓ⊤​eiB\_\{\\ell\}^\{\\top\}e\_\{i\}, i\.e\. theii\-th column ofCℓC\_\{\\ell\}and theii\-th row ofBℓB\_\{\\ell\}, respectively\.

The right factorsBℓ⊤​eiB^\{\\top\}\_\{\\ell\}e\_\{i\}appearing in \([1\.3](https://arxiv.org/html/2608.26963#S1.E3)\) are vectors in input space, and it is these that determine the directions in which the composed map is sensitive\. Note that they are available from the forward pass:BℓB\_\{\\ell\}is the partial composition below layerℓ\\ell, andeie\_\{i\}selects the unit in question\. Through \([1\.3](https://arxiv.org/html/2608.26963#S1.E3)\) the effect of activating one unit on the output becomes clear, and no differentiation with respect to the input is involved at any stage\.

The LRHE is derived by exploiting this property further\. SubstituteΣℓ\\Sigma\_\{\\ell\}as a rank\-one perturbation of the identity in the formI−vℓ​vℓ⊤I\-v\_\{\\ell\}v\_\{\\ell\}^\{\\top\}, in whichvℓv\_\{\\ell\}is a unit vector\. Key to this expansion is howvℓv\_\{\\ell\}is found by rewriting the ReLU activation as a Householder reflection\. The reflection vectors are not learned and they are computed during the forward pass, and require no differentiation\. The expansion describes the effect of perturbing the hidden states on the composed mapsBℓB\_\{\\ell\}orCℓC\_\{\\ell\}rather than any individual layer, so the directions it identifies are those in which the network as a whole amplifies, not those in which an individual weight matrixWℓW\_\{\\ell\}does\. It was shown in\[[21](https://arxiv.org/html/2608.26963#bib.bib21)\]that adversarial examples found for a tsunami waveform prediction task\[[20](https://arxiv.org/html/2608.26963#bib.bib20)\]have significant components along these basis directions\.

The present work takes that construction as given and asks a different question: whether the subspace it identifies is the one in which adversarial perturbations reside for a classification task, and whether restricting attention to that subspace suffices to train robust models\. Two consequences follow\. First, adversarial examples in this small\-norm class can be constructed directly from the trained parameters, with no input gradient iterations at all\. Second, and the subject of the remainder of this work, adversarial training against them reduces to a minimization: the inner search disappears, the network is penalized for its response along a set of directions recomputed at each forward pass, and no derivative with respect to the input is taken at any point during training\.

Since LRHE enables the identification of a linear subspace of adversarial perturbation, this can be exploited to reduce the cost of adversarial training by restricting the inner maximization\. Our method thus belongs to the third group discussed above, but differs in where its geometric information originates\. Curvature, Jacobian, and local\-linearity penalties are all measured, each requiring derivative evaluations beyond those of standard training, so that the saving relative to multi\-step PGD is real but partial\. The expansion instead supplies the relevant subspace algebraically\. That such a restriction should be possible at all is suggested by an early observation\[[29](https://arxiv.org/html/2608.26963#bib.bib29)\]that adversarial perturbations occupy a contiguous subspace whose dimension is far below that of the input, on the order of tens at MNIST scale\. If the adversarially relevant directions are so few, searching the full input space at every step is wasteful\. Subspace adversarial training\[[14](https://arxiv.org/html/2608.26963#bib.bib14)\]reaches a related conclusion, but extracts its subspace from the optimization trajectory and works in parameter space; ours is determined by the structure of the network at the input in question and lies in input space\.

A second distinction separates our approach from the certified robustness literature, where the geometry of a network is controlled by constraining each layer individually\. Parseval networks\[[5](https://arxiv.org/html/2608.26963#bib.bib5)\], spectral normalization\[[16](https://arxiv.org/html/2608.26963#bib.bib16)\], and the orthogonal convolution constructions of\[[13](https://arxiv.org/html/2608.26963#bib.bib13),[30](https://arxiv.org/html/2608.26963#bib.bib30),[24](https://arxiv.org/html/2608.26963#bib.bib24)\]all enforce a norm or orthogonality condition on every weight matrix, so that the Lipschitz constant of the composition is bounded by the product of the layerwise bounds; the programme is extended to the activations in\[[25](https://arxiv.org/html/2608.26963#bib.bib25),[26](https://arxiv.org/html/2608.26963#bib.bib26)\]\. A global guarantee then follows from purely local conditions, and a certificate is obtained as a by\-product\. The cost is that the product bound is loose\. For the map \([1\.1](https://arxiv.org/html/2608.26963#S1.E1)\), the estimate‖F‖≤∏ℓ‖Wℓ‖\\\|F\\\|\\leq\\prod\_\{\\ell\}\\\|W\_\{\\ell\}\\\|holds with equality only when the dominant singular directions of successive factors align, which they generically do not, and how far the naive bound sits above the true constant is documented in\[[33](https://arxiv.org/html/2608.26963#bib.bib33),[8](https://arxiv.org/html/2608.26963#bib.bib8)\]\. A network constrained layerwise is thus constrained more tightly than the robustness objective requires, and the excess is paid in expressivity\.

Our approach acts on the product rather than on its factors\. The expansion is an expansion ofFFitself, so no layer is required to be orthogonal or norm\-bounded, no activation is replaced, and the network is trained in the usual way\. We obtain no certificate, since we impose no condition from which one could be derived; what we obtain instead is a description of where the composed map is sensitive that is not conservative by construction\. This also underpins the relation to the prior works using Householder reflections\[[25](https://arxiv.org/html/2608.26963#bib.bib25)\]\. There the Householder structure is imposed: it is a design constraint on the activation, satisfied exactly, and the reflection vectors are trained parameters\. Here it is descriptive, read in from a network trained without any direct restrictions\. The theorem of\[[25](https://arxiv.org/html/2608.26963#bib.bib25)\]is nonetheless a useful independent justification for our choice of basis, since it establishes that Householder reflections are the canonical form for Jacobian transitions across activation boundaries in piecewise linear networks\. Our expansion is thus not an arbitrary change of coordinates but one aligned with the intrinsic structure of the map being approximated\.

Our experiments on MNIST support the following claims\. First, the subspace identified by the expansion is the one in which adversarial perturbations lie: projecting perturbations onto its orthogonal complement reduces attack success from28\.01%28\.01\\%to9\.20%9\.20\\%, whereas removing a random subspace of the same dimension reduces it only to25\.42%25\.42\\%\(Section[5](https://arxiv.org/html/2608.26963#S5)\)\. Second, training against these directions costs the equivalent of2\.82\.8PGD steps per epoch, an8\.7×8\.7\\timesreduction relative to4040\-step adversarial training and less than the cost of 3\-step training\. Third, over the budget range in which the expansion is descriptive the resulting models match multi\-step adversarial training: they match the robustness of 3\-step PGD training for relativeℓ2\\ell^\{2\}budgetsε≤0\.02\\varepsilon\\leq 0\.02and of4040\-step training forε≤0\.012\\varepsilon\\leq 0\.012, with multi\-step training pulling ahead beyond \(Section[6](https://arxiv.org/html/2608.26963#S6)\)\. The falling away is consistent with the locality of the expansion, which ceases to describe the network once the perturbation is large enough to leave the affine cell\.

We regard this parity as the substantive result\. Every method in the third group above obtains its information about the network’s local behavior by differentiating with respect to the input, and it would be natural to assume that such differentiation is critical to obtaining that information\. The comparison shows otherwise: the geometry is already present in the activation pattern, and a regularizer that never forms an input derivative reaches similar performance as one that forms forty\.

We focus our attention on small budgets\. The motivation for studying adversarial examples at all rests on the premise that the perturbation is imperceptible, although the budgets now conventional in the literature have evolved well beyond the point where that premise holds\. The class we treat is the one the original problem concerned\. We state plainly what is not claimed\. We do not report state\-of\-the\-art robust accuracy at all budgets, and we provide no certificates\. Our experiments are confined to MNIST, and we make no claim that the approximation accuracy required to capture the adversarially relevant subspace, or the budget range over which the expansion remains descriptive, transfers to higher\-resolution data\.

## 2\.Preliminaries

### 2\.1\.Adversarial examples

The term*adversarial examples*refers to a broad class of transformationsT⁡\(x\)T\(x\)of the input dataxxthat cause an incommensurate change in the deep learning model predictionf⁡\(T⁡\(x\)\)f\(T\(x\)\)when compared tof⁡\(x\)f\(x\)\. For classification tasks, this change refers to the change in the output classification label from a correct one to an incorrect one\. The literature\[[28](https://arxiv.org/html/2608.26963#bib.bib28)\]categorizes adversarial examples into the case whenT⁡\(x\)T\(x\)is small, sometimes referred to as sensitivity adversarial examples, and the case whenT⁡\(x\)T\(x\)is an admissible family of group actions, called invariance adversarial examples; e\.g\. geometric distortions like rotation or translation\. On the other hand,*natural adversarial examples*\[[10](https://arxiv.org/html/2608.26963#bib.bib10)\]are out\-of\-distribution samples that arise without any well\-defined transformationTTand occur in datasets collected without synthetic generation\. These examples are not necessarily minuscule perturbations of the original image, nor described straightforwardly by geometric invariance\.

This work focuses on the specific subclass defined in terms of theℓp\\ell\_\{p\}\-norm ofT⁡\(x\)T\(x\), i\.e\. the sensitivity adversarial examples in the terminology above\. The precise definition follows, with the slight difference that we measure the distance in the relative sense\.

###### Definition 2\.1\(Adversarial examples\)\.

Given a modelf:ℝnin→ℝnoutf:\\mathbb\{R\}^\{n\_\{\\text\{in\}\}\}\\to\\mathbb\{R\}^\{n\_\{\\text\{out\}\}\}, and an inputx∈ℝninx\\in\\mathbb\{R\}^\{n\_\{\\text\{in\}\}\}, the perturbed inputx\+δ​xx\+\\delta xis an*adversarial example*if

\(2\.1\)∥δ​x∥p≤ε​∥x∥pandsoftargmax⁡f⁡\(x\)≠softargmax⁡f⁡\(x\+δ​x\)\.\\lVert\{\\delta x\}\\rVert\_\{p\}\\leq\\varepsilon\\lVert\{x\}\\rVert\_\{p\}\\quad\\text\{ and \}\\quad\\softargmax f\(x\)\\neq\\softargmax f\(x\+\\delta x\)\.

The notation∥⋅∥p\\lVert\{\\cdot\}\\rVert\_\{p\}denotes the standardℓp\\ell\_\{p\}norm, and we will focus onp=2p=2throughout this paper, and show some results forp=∞p=\\infty\. Note that in many settings whenp=∞p=\\inftythis definition ofε\\varepsilonrelative to the norm of the input reduces to the absoluteε\\varepsilon\. For example, for grayscale images in MNIST,∥x∥∞=1\\lVert\{x\}\\rVert\_\{\\infty\}=1or approximately so, reducing the constraint to an absolute one\. Note that we separate out thesoftargmax\\softargmaxfunction from the modelff, which outputs logits\.

More generally, adversarial examples for regression tasks are defined by replacing the misclassification requirement \([2\.1](https://arxiv.org/html/2608.26963#S2.E1)\) with

\(2\.2\)∥f⁡\(x\+δ​x\)∥p≥τ​∥f⁡\(x\)∥p,\\lVert\{f\(x\+\\delta x\)\}\\rVert\_\{p\}\\geq\\tau\\lVert\{f\(x\)\}\\rVert\_\{p\},for some appropriate percentage thresholdτ\\tau, which is specially selected for each domain\-specific dataset and learning task\. For example, for the geophysical data set for tsunami waveheight prediction\[[21](https://arxiv.org/html/2608.26963#bib.bib21)\], a budget ofε=0\.005\\varepsilon=0\.005relative change in geodetic measurement input causingτ=0\.3\\tau=0\.3change in surface elevation was considered adversarial, a reasonable assumption due to the stability of the underlying physical processes\.

We focus on the case where the modelffis a feedforward ReLU neural network\. For givenx∈ℝninx\\in\\mathbb\{R\}^\{n\_\{\\text\{in\}\}\}the feedforward neural network model is a parametrized family of functions

\(2\.3\)fθ\(x\)=ALσAL−1σ⋯σA1x,Aℓz=Wℓz\+bℓ,f\_\{\\theta\}\(x\)=A\_\{L\}\\,\\sigma\\,A\_\{L\-1\}\\,\\sigma\\,\\cdots\\,\\sigma\\,A\_\{1\}\\,x,\\quad A\_\{\\ell\}z=W\_\{\\ell\}z\+b\_\{\\ell\},where the parameters of the network areθ=\(W1,…,WL,b1,…,bL\)\\theta=\(W\_\{1\},\.\.\.\\,,W\_\{L\},b\_\{1\},\.\.\.\\,,b\_\{L\}\), andσ\\sigmahere is the Rectified Linear Unit \(ReLU\) activationσ⁡\(z\)=max⁡\{0,z\}\\sigma\(z\)=\\max\\\{0,z\\\}forz∈ℝz\\in\\mathbb\{R\}\. The left\-multiplications byAℓA\_\{\\ell\}andσ\\sigmadenote application of an affine map and the entry\-wise application of a ReLU to the hidden state vector, respectively\.

Adversarial examples for a trained neural networkfθf\_\{\\theta\}are found by solving an optimization problem, typically by increasing the misfit lossℒ:ℝnout×\{1,…,C\}→ℝ\+\\mathcal\{L\}:\\mathbb\{R\}^\{n\_\{\\text\{out\}\}\}\\times\\\{1,\.\.\.\\,,C\\\}\\to\\mathbb\{R\}\_\{\+\}\(e\.g\. the cross\-entropy function\), while keeping the clean inputxxand trained weightsθ\\thetafixed, that is,

\(2\.4\)max∥δ​x∥p≤ε​∥x∥p⁡ℒ⁡\(fθ​\(x\+δ​x\),y\)\.\\max\_\{\\lVert\{\\delta x\}\\rVert\_\{p\}\\leq\\varepsilon\\lVert\{x\}\\rVert\_\{p\}\}\\mathcal\{L\}\(f\_\{\\theta\}\(x\+\\delta x\),y\)\.The designated optimization problem and the particular optimization approach is referred to as a*threat model*\. TheAutoAttackpackage\[[7](https://arxiv.org/html/2608.26963#bib.bib7)\]implements an accepted benchmark with standard threat models, such as variants of PGD, the Fast Adaptive Boundary attack\[[6](https://arxiv.org/html/2608.26963#bib.bib6)\], and the Square attack\[[1](https://arxiv.org/html/2608.26963#bib.bib1)\]\. To supplementAutoAttack, we have implemented a strong PGD attack with multiple configurations and thousands of iterations\[[9](https://arxiv.org/html/2608.26963#bib.bib9),[3](https://arxiv.org/html/2608.26963#bib.bib3),[15](https://arxiv.org/html/2608.26963#bib.bib15)\]\. We found thatAutoAttackreliably outperformed vanilla PGD attacks, so we report only those results\.

### 2\.2\.Low\-rank Householder expansion \(LRHE\)

In a prior work\[[21](https://arxiv.org/html/2608.26963#bib.bib21)\], the feedforward neural networkfθf\_\{\\theta\}was shown to naturally yield an expansion which writes it as a low\-rank correction of an affine mapping\. Denote the hidden states as

\(2\.5\)\{z1:=W1​x\+b1,zℓ\+1:=Wℓ\+1​σ​zℓ\+bℓ\+1,ℓ=1,2,…,L−1,\\left\\\{\\begin\{aligned\} z\_\{1\}&:=W\_\{1\}x\+b\_\{1\},\\\\ z\_\{\\ell\+1\}&:=W\_\{\\ell\+1\}\\sigma z\_\{\\ell\}\+b\_\{\\ell\+1\},&&\\ell=1,2,\.\.\.\\,,L\-1,\\end\{aligned\}\\right\.then the evaluation of a ReLUσ\\sigmafor a given vectorzℓ∈ℝnℓz\_\{\\ell\}\\in\\mathbb\{R\}^\{n\_\{\\ell\}\}admits the alternative form

\(2\.6\)σ​zℓ=12​\(zℓ\+\|zℓ\|\)\.\\sigma z\_\{\\ell\}=\\frac\{1\}\{2\}\(z\_\{\\ell\}\+\\left\\lvert z\_\{\\ell\}\\right\\rvert\)\.The absolute value operation\|⋅\|\\left\\lvert\\cdot\\right\\rvertis an orthogonal transformation onℝnℓ\\mathbb\{R\}^\{\{n\_\{\\ell\}\}\}which is a natural candidate to represent as a Householder reflection, a rank\-one perturbation of the identity of the form

\(2\.7\)\|zℓ\|=\(I−2​vℓ​vℓ⊤\)​zℓ,\\left\\lvert z\_\{\\ell\}\\right\\rvert=\(I\-2v\_\{\\ell\}v\_\{\\ell\}^\{\\top\}\)z\_\{\\ell\},where the vectorvℓv\_\{\\ell\}is determined modulo sign by the two conditions

\(2\.8\)vℓ\|zℓ−\|zℓ\|,∥vℓ∥2=1\.v\_\{\\ell\}\\parallel z\_\{\\ell\}\-\\left\\lvert z\_\{\\ell\}\\right\\rvert,\\qquad\\lVert\{v\_\{\\ell\}\}\\rVert\_\{2\}=1\.Inserting this formulation into \([2\.6](https://arxiv.org/html/2608.26963#S2.E6)\) and into the neural network \([2\.3](https://arxiv.org/html/2608.26963#S2.E3)\), we obtain the product

\(2\.9\)fθ\(x\)=AL\(I−vL−1vL−1⊤\)AL−1⋯A2\(I−v1v1⊤\)A1x\.f\_\{\\theta\}\(x\)=A\_\{L\}\(I\-v\_\{L\-1\}v\_\{L\-1\}^\{\\top\}\)A\_\{L\-1\}\\cdots A\_\{2\}\(I\-v\_\{1\}v\_\{1\}^\{\\top\}\)A\_\{1\}x\.Expanding the product on the RHS, one obtains2L−12^\{L\-1\}terms\. All but the input\-independent termF0F\_\{0\}below are input\-dependent and rank one\. That is, writing

\(2\.10\)fθ​\(x\)=F0​x\+Fσ​x,f\_\{\\theta\}\(x\)=F\_\{0\}x\+F\_\{\\sigma\}x,the affine mapsF0F\_\{0\}andFσF\_\{\\sigma\}are given by

\(2\.11\)F0=ALAL−1⋯A2A1,andFσ=∑β=12L−1−1Fβ,F\_\{0\}=A\_\{L\}A\_\{L\-1\}\\cdots A\_\{2\}A\_\{1\},\\quad\\text\{ and \}\\quad F\_\{\\sigma\}=\\sum\_\{\\beta=1\}^\{2^\{L\-1\}\-1\}F\_\{\\beta\},and every term in the sumFσF\_\{\\sigma\}over the indexβ\\betais of the form

\(2\.12\)AL⋯Aℓ\+1vℓvℓ⊤Aℓ⋯A1,A\_\{L\}\\cdots A\_\{\\ell\+1\}v\_\{\\ell\}v\_\{\\ell\}^\{\\top\}A\_\{\\ell\}\\cdots A\_\{1\},where there is at least one rank\-one termvℓ​vℓ⊤v\_\{\\ell\}v\_\{\\ell\}^\{\\top\}present as a factor, so the product as a whole is rank\-one by the rank inequality for products\. Furthermore, the column and row spaces ofFσF\_\{\\sigma\}are each spanned byL−1L\-1vectors,

\(2\.13\)WL⋯Wℓ\+1vℓ,andW1⊤⋯Wℓ⊤vℓ,forℓ=1,⋯,L−1\.W\_\{L\}\\cdots W\_\{\\ell\+1\}v\_\{\\ell\},\\quad\\text\{ and \}\\quad W\_\{1\}^\{\\top\}\\cdots W\_\{\\ell\}^\{\\top\}v\_\{\\ell\},\\quad\\text\{ for \}\\ell=1,\\cdots,L\-1\.Here the terms*column*and*row space*of the affine mapFσF\_\{\\sigma\}refer to the column and the row space of the linear part ofFσF\_\{\\sigma\}, respectively; similarly the rank ofFσF\_\{\\sigma\}refers to the dimension of the column or row space ofFσF\_\{\\sigma\}\.

Hence the crucial property of this expansion is that the affine transformationFσF\_\{\\sigma\}has rank at mostL−1L\-1, and is therefore low\-rank wheneverL−1L\-1is small relative to the layer widths\. This expansion is called the low\-rank Householder expansion \(see Theorem 2\.4 in\[[21](https://arxiv.org/html/2608.26963#bib.bib21)\]\)\.

###### Definition 2\.2\.

The*Low\-Rank Householder Expansion \(LRHE\)*of the neural networkfθf\_\{\\theta\}is the expansion given by the substitution of the activationσ\\sigmaafter theℓ\\ell\-th layer by the rank\-one perturbation of the identityI−vℓ​vℓ⊤I\-v\_\{\\ell\}v\_\{\\ell\}^\{\\top\}which reads

\(2\.14\)fθ​\(x\)=F0​\(x\)\+Fσ​\(x\),f\_\{\\theta\}\(x\)=F\_\{0\}\(x\)\+F\_\{\\sigma\}\(x\),whereF0F\_\{0\}is an input\-independent affine transformation andFσF\_\{\\sigma\}is an input\-dependent affine transformation and satisfiesrank​\(Fσ\)≤L−1\\text\{rank\}\(F\_\{\\sigma\}\)\\leq L\-1\. The column and row spaces of low\-rank affine transformationFσF\_\{\\sigma\}are

\(2\.15\)col⁡\(Fσ\)\\displaystyle\\col\(F\_\{\\sigma\}\)=span\{WLWL−1⋯Wℓ\+2Wℓ\+1vℓ∣ℓ=1,2,…,L−1\},\\displaystyle=\\Span\\\{W\_\{L\}W\_\{L\-1\}\\cdots W\_\{\\ell\+2\}W\_\{\\ell\+1\}v\_\{\\ell\}\\mid\\ell=1,2,\.\.\.\\,,L\-1\\\},row⁡\(Fσ\)\\displaystyle\\row\(F\_\{\\sigma\}\)=span\{W1⊤W2⊤⋯Wℓ−1⊤Wℓ⊤vℓ∣ℓ=1,2,…,L−1\}\.\\displaystyle=\\Span\\\{W\_\{1\}^\{\\top\}W\_\{2\}^\{\\top\}\\cdots W\_\{\\ell\-1\}^\{\\top\}W\_\{\\ell\}^\{\\top\}v\_\{\\ell\}\\mid\\ell=1,2,\.\.\.\\,,L\-1\\\}\.

For a fixed inputxx, the representation of a ReLU using a Householder reflection in LRHE reveals a new low\-rank structure inFσF\_\{\\sigma\}that pertains purely to the nonlinear behavior of the neural network, independently from the linear analysis ofF0F\_\{0\}\. Note that this representation of the ReLU is distinct from so\-called*Householder activations*\[[25](https://arxiv.org/html/2608.26963#bib.bib25)\], whose definition involves Householder reflections\.

The core idea motivating this work is the hypothesis that LRHE and adversarial examples are closely related; LRHE provides a way to explain how the adversarial examples can arise\[[21](https://arxiv.org/html/2608.26963#bib.bib21)\]\. In LRHE the linear part of the input\-dependent low\-rank termFσF\_\{\\sigma\}can be factorized into a singular value decomposition of the linear part,

\(2\.16\)Fσ​z=bσ\+∑i=1L−1di​ζi​ξi⊤​z,bσ=Fσ​0,di∈ℝ\+,ζi∈ℝnout,ξi∈ℝnin\.F\_\{\\sigma\}z=b\_\{\\sigma\}\+\\sum\_\{i=1\}^\{L\-1\}d\_\{i\}\\zeta\_\{i\}\\xi\_\{i\}^\{\\top\}z,\\quad b\_\{\\sigma\}=F\_\{\\sigma\}0,\\,\\,d\_\{i\}\\in\\mathbb\{R\}\_\{\+\},\\,\\,\\zeta\_\{i\}\\in\\mathbb\{R\}^\{n\_\{\\text\{out\}\}\},\\,\\,\\xi\_\{i\}\\in\\mathbb\{R\}^\{n\_\{\\text\{in\}\}\}\.Instability can arise whend1≫1d\_\{1\}\\gg 1even ifF0F\_\{0\}is well\-conditioned, so that∥F0​δ​x∥p∼∥δ​x∥p\\lVert\{F\_\{0\}\\,\\delta x\}\\rVert\_\{p\}\\sim\\lVert\{\\delta x\}\\rVert\_\{p\}\. When the perturbationδ​x\\delta xis in the direction ofξ1\\xi\_\{1\}, it is possible to have

\(2\.17\)∥δ​x∥p≪1and∥Fσ​δ​x∥p≫1\.\\lVert\{\\delta x\}\\rVert\_\{p\}\\ll 1\\quad\\text\{ and \}\\quad\\lVert\{F\_\{\\sigma\}\\;\\delta x\}\\rVert\_\{p\}\\gg 1\.This decouples the linear conditioning ofF0F\_\{0\}from the nonlinear conditioning ofFσF\_\{\\sigma\}and of the neural networkfθf\_\{\\theta\}\.

There are previous approaches aimed at makingF0F\_\{0\}well\-conditioned via the submultiplicative inequality∥F0∥p≤∏ℓ∥Wℓ∥p≲1\\lVert\{F\_\{0\}\}\\rVert\_\{p\}\\leq\\prod\_\{\\ell\}\\lVert\{W\_\{\\ell\}\}\\rVert\_\{p\}\\lesssim 1considered already in\[[27](https://arxiv.org/html/2608.26963#bib.bib27)\], for example by imposing weight decay during training\. However, this reduces to penalizing∥Wℓ∥\\lVert\{W\_\{\\ell\}\}\\rVert, i\.e\. weight decay, and standard regularization does not generally lead to adversarial robustness; more sophisticated considerations are necessary, e\.g\.\[[25](https://arxiv.org/html/2608.26963#bib.bib25),[26](https://arxiv.org/html/2608.26963#bib.bib26)\]as discussed in the introduction\.

## 3\.Adversarial training based on LRHE

### 3\.1\.The min\-max formulation

*Adversarial training*\[[9](https://arxiv.org/html/2608.26963#bib.bib9),[15](https://arxiv.org/html/2608.26963#bib.bib15)\]refers to training strategies that broaden the scope of the usual empirical risk over the data distribution𝒟\\mathcal\{D\}

\(3\.1\)minθ⁡𝔼\(x,y\)∼𝒟​\[ℒ⁡\(fθ​\(x\),y\)\]\\min\_\{\\theta\}\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\}\\left\[\\mathcal\{L\}\(f\_\{\\theta\}\(x\),y\)\\right\]to cover worst\-case or near\-worst\-case adversarial examples\. For example, one takes the minimization problem to be

\(3\.2\)minθ⁡𝔼\(x,y\)∼𝒟​\[max∥δ​x∥p≤ε​∥x∥p⁡ℒ⁡\(fθ​\(x\+δ​x\),y\)\]\.\\min\_\{\\theta\}\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\}\\left\[\\max\_\{\\lVert\{\\delta x\}\\rVert\_\{p\}\\leq\\varepsilon\\lVert\{x\}\\rVert\_\{p\}\}\\mathcal\{L\}\(f\_\{\\theta\}\(x\+\\delta x\),y\)\\right\]\.Inner maximization has no explicit solution and is commonly approximated byKKsteps of projected gradient ascent\. Each step requires a gradient of the loss with respect to the input, and the entire cost of adversarial training in excess of ordinary training is the cost of this search\. What the search returns, however, is a direction, and in the regime where the network is affine that direction is not arbitrary: the maximizer of the linearized inner objective over∥δ​x∥p≤ε​∥x∥p\\lVert\{\\delta x\}\\rVert\_\{p\}\\leq\\varepsilon\\lVert\{x\}\\rVert\_\{p\}isε\\varepsilontimes the unit vector most amplified by the model in the metric induced by the output\-space loss gradient\. PGD is thus an iterative procedure for locating a direction that is already determined by the local action of the network\.

The LRHE supplies such directions directly\. Its right factorsξ1,…,ξL−1\\xi\_\{1\},\\dots,\\xi\_\{L\-1\}in the SVD ofFσF\_\{\\sigma\}\([2\.16](https://arxiv.org/html/2608.26963#S2.E16)\) are, by construction, the directions in which the composed map amplifies most strongly, and they are obtained from the activation pattern without any differentiation with respect toxx\.

We first motivate our training objective conceptually\. The key ingredient is the restriction of the inner maximization of \([3\.2](https://arxiv.org/html/2608.26963#S3.E2)\) by a penalty evaluated on this fixed set of directions by the orthonormal vectors\(ξℓ\)ℓ\(\\xi\_\{\\ell\}\)\_\{\\ell\},

\(3\.3\)minθ⁡𝔼\(x,y\)​\[ℒ⁡\(fθ​\(x\),y\)\+λ​∑ℓ=1L−1ℒ⁡\(fθ​\(x\+ε​ξℓ\),softargmax⁡fθ​\(x\)\)\],\\min\_\{\\theta\}\\;\\mathbb\{E\}\_\{\(x,y\)\}\\Bigl\[\\mathcal\{L\}\\bigl\(f\_\{\\theta\}\(x\),y\\bigr\)\+\\lambda\\sum\_\{\\ell=1\}^\{L\-1\}\\mathcal\{L\}\(f\_\{\\theta\}\(x\+\\varepsilon\\xi\_\{\\ell\}\),\\softargmax f\_\{\\theta\}\(x\)\)\\Bigr\],so that the min\-max problem collapses to a minimization\. The structure of adversarial training is retained, that is, the model is penalized for its response to perturbations of sizeε\\varepsilonin the directions that most affect it\. In contrast, the search is eliminated, and with it theKKinput\-gradient computations that dominate its cost\.

Following these discussions, we propose a new training procedure based on the LRHE introduced above\. A naïve approach would penalize the largest singular valued1d\_\{1\}ofFσF\_\{\\sigma\}\([2\.16](https://arxiv.org/html/2608.26963#S2.E16)\), however, the direct computation of the SVD ofFσF\_\{\\sigma\}would involve accumulating2L−12^\{L\-1\}terms in the expansion \([2\.11](https://arxiv.org/html/2608.26963#S2.E11)\), which is prohibitive\. Instead, we penalize a basis for the column space ofFσF\_\{\\sigma\}\([2\.15](https://arxiv.org/html/2608.26963#S2.E15)\)\.

We keep the bias terms in the affine maps, assuming that the large singular values in the linear part ofFσF\_\{\\sigma\}\([2\.16](https://arxiv.org/html/2608.26963#S2.E16)\) make the linear part dominant\. The*output basis vectors*are computed during a forward pass

\(3\.4\)ϕℓ\(x\):=ALAL−1⋯Aℓ\+1vℓvℓ⊤Aℓ−1σ⋯σA1x\.ℓ=1,…,L−1\.\\phi\_\{\\ell\}\(x\):=A\_\{L\}A\_\{L\-1\}\\cdots A\_\{\\ell\+1\}v\_\{\\ell\}v\_\{\\ell\}^\{\\top\}A\_\{\\ell\-1\}\\sigma\\cdots\\sigma A\_\{1\}x\.\\qquad\\ell=1,\.\.\.\\,,L\-1\.The vectorϕℓ\\phi\_\{\\ell\}has the alternative expression,

\(3\.5\)ϕℓ\(x\)=ALAL−1⋯Aℓ\+1\(σzℓ−zℓ\),ℓ=1,…,L−1\.\\phi\_\{\\ell\}\(x\)=A\_\{L\}A\_\{L\-1\}\\cdots A\_\{\\ell\+1\}\(\\sigma z\_\{\\ell\}\-z\_\{\\ell\}\),\\quad\\ell=1,\.\.\.\\,,L\-1\.The input basis, i\.e\. the basis for the row space in \([2\.15](https://arxiv.org/html/2608.26963#S2.E15)\), is computed by reversing the computation for the output basisϕℓ\\phi\_\{\\ell\}\([3\.5](https://arxiv.org/html/2608.26963#S3.E5)\),

\(3\.6\)ψℓ\(x\)=W1⊤W2⊤⋯Wℓ−1⊤Wℓ⊤\(σzℓ−zℓ\),ℓ=1,…,L−1\.\\psi\_\{\\ell\}\(x\)=W\_\{1\}^\{\\top\}W\_\{2\}^\{\\top\}\\cdots W\_\{\\ell\-1\}^\{\\top\}W\_\{\\ell\}^\{\\top\}\(\\sigma z\_\{\\ell\}\-z\_\{\\ell\}\),\\quad\\ell=1,\.\.\.\\,,L\-1\.This computation gives us the row and column basis ofFσF\_\{\\sigma\}modulo the bias terms; the output basis is computed with the affine maps and therefore lies incol⁡\(Fσ\)\\col\(F\_\{\\sigma\}\)only modulo the bias offset, whereas the input basis is exact\. Neither basis coincides with the singular vectors ofFσF\_\{\\sigma\}, however\. Note the discussion on their relation in\[[21](https://arxiv.org/html/2608.26963#bib.bib21)\]which shows the singular vectors can be estimated empirically by a sparse sampling of the terms in the sum \([2\.11](https://arxiv.org/html/2608.26963#S2.E11)\)\.

Finally, we approximate the response of perturbing the input byε​ξi\\varepsilon\\xi\_\{i\}in \([3\.3](https://arxiv.org/html/2608.26963#S3.E3)\) via the approximation

\(3\.7\)fθ​\(x\+ε​ξℓ\)−fθ​\(x\)≈ε⁡\(F0\+Fσ\)​ξℓ,f\_\{\\theta\}\(x\+\\varepsilon\\xi\_\{\\ell\}\)\-f\_\{\\theta\}\(x\)\\approx\\varepsilon\(F\_\{0\}\+F\_\{\\sigma\}\)\\xi\_\{\\ell\},via the nonlinear partFσF\_\{\\sigma\}of LRHE, and based on our discussion regarding how the adversarial examples can arise from the viewpoint regarding \([2\.16](https://arxiv.org/html/2608.26963#S2.E16)\), our priority is to suppress the dominant term

\(3\.8\)dℓ​ζℓ\+bσ=Fσ​ξℓ≫F0​ξℓ,d\_\{\\ell\}\\zeta\_\{\\ell\}\+b\_\{\\sigma\}=F\_\{\\sigma\}\\xi\_\{\\ell\}\\gg F\_\{0\}\\xi\_\{\\ell\},that is, the singular valuesdℓd\_\{\\ell\}and the corresponding left singular vectorsζℓ\\zeta\_\{\\ell\}\. As already mentioned, these are hard to compute directly, hence we compute\(ϕℓ\)ℓ\(\\phi\_\{\\ell\}\)\_\{\\ell\}instead\. From this alternative, we want to penalize the change

\(3\.9\)∥fθ​\(x\+ε​ξℓ\)−fθ​\(x\)∥p≲ε​dℓ​∥ζℓ∥p\\lVert\{f\_\{\\theta\}\(x\+\\varepsilon\\xi\_\{\\ell\}\)\-f\_\{\\theta\}\(x\)\}\\rVert\_\{p\}\\lesssim\\varepsilon d\_\{\\ell\}\\lVert\{\\zeta\_\{\\ell\}\}\\rVert\_\{p\}indirectly by penalizing the size ofϕℓ\\phi\_\{\\ell\}\. For classification tasks, it is natural to penalize the deviationfθ​\(x\)\+γ​ϕℓf\_\{\\theta\}\(x\)\+\\gamma\\phi\_\{\\ell\}so long as the label does not change, leading to the penalization

\(3\.10\)ℒ⁡\(fθ​\(x\)\+γ​ϕℓ,softargmax⁡fθ​\(x\)\),\\mathcal\{L\}\(f\_\{\\theta\}\(x\)\+\\gamma\\phi\_\{\\ell\},\\softargmax f\_\{\\theta\}\(x\)\),with some scalingγ\\gamma\.

Putting it all together, we propose our adversarial training as the regularized problem

\(3\.11\)minθ⁡𝔼\(x,y\)∼𝒟​\[ℒ⁡\(fθ​\(x\),y\)\+λ​ℛ​\(θ,x,γ\)\],\\min\_\{\\theta\}\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\}\\left\[\\mathcal\{L\}\(f\_\{\\theta\}\(x\),y\)\+\\lambda\\mathcal\{R\}\(\\theta,x;\\gamma\)\\right\],where the regularization term penalizes the change in the final prediction,

\(3\.12\)ℛ⁡\(θ,x,γ\)=∑ℓ=1L−1ℒ⁡\(fθ​\(x\)\+γ​ϕℓ,softargmax⁡fθ​\(x\)\),γ∈\(−1,1\),\\mathcal\{R\}\(\\theta,x;\\gamma\)=\\sum\_\{\\ell=1\}^\{L\-1\}\\mathcal\{L\}\(f\_\{\\theta\}\(x\)\+\\gamma\\phi\_\{\\ell\},\\softargmax f\_\{\\theta\}\(x\)\),\\quad\\gamma\\in\(\-1,1\),whereλ\\lambdaandγ\\gammaare hyper\-parameters\.

The computation of these bases\(ϕℓ\)ℓ\(\\phi\_\{\\ell\}\)\_\{\\ell\}and\(ψℓ\)ℓ\(\\psi\_\{\\ell\}\)\_\{\\ell\}is straightforward; one computes the difference of the pre\- and post\-activation\-states, then applies all the subsequent affine layers to obtain the output basis, or all the preceding transposed weights for the input basis\.

Figure 1\.Diagram of a forward pass that computes the output basis\(ϕℓ\)ℓ\(\\phi\_\{\\ell\}\)\_\{\\ell\}\([3\.5](https://arxiv.org/html/2608.26963#S3.E5)\)\. The computation of the output basis involves additional partial forward passes that omit the nonlinearities\. The output basis is then used in the regularization termℛ\\mathcal\{R\}\([3\.12](https://arxiv.org/html/2608.26963#S3.E12)\) to penalize large changes in the output logits in the direction of the output basis\.During training of the regularized problem \([3\.11](https://arxiv.org/html/2608.26963#S3.E11)\),fθf\_\{\\theta\}is evaluated as usual until theℓ\\ell\-th layer, then only the affine mappings are applied in the subsequent layers omitting the nonlinearities\. This propagates the nonlinear correctionσ​zℓ\+1−zℓ\+1\\sigma z\_\{\\ell\+1\}\-z\_\{\\ell\+1\}through the subsequent layers\. At each gradient computation, only one forward and one backward pass are required\. The cost of forward evaluation of this regularizer is𝒪⁡\(L\)\\mathcal\{O\}\(L\)times the standard evaluation of the forward modelfθf\_\{\\theta\}, and one backward pass is required to compute the gradient of the loss function with respect toθ\\thetaduring training\. For models withLLnot too large, however, minimizing the objective \([3\.11](https://arxiv.org/html/2608.26963#S3.E11)\) is more efficient than the standard adversarial training \([3\.2](https://arxiv.org/html/2608.26963#S3.E2)\) which would require multiple forward\-backward passes with respect to both parametersθ\\thetaand inputxxwithin the inner\-max\-loop \(PGD iterations\)\.

Figure[1](https://arxiv.org/html/2608.26963#S3.F1)depicts a diagram showing the flow of computation for the output basis\(ϕℓ\)ℓ\(\\phi\_\{\\ell\}\)\_\{\\ell\}and the regularization term\.

### 3\.2\.Remarks on the training procedure

Let us discuss the design of this regularizer\. In the LRHE in Definition[2\.2](https://arxiv.org/html/2608.26963#S2.Thmgenericthm2), the input and output bases are coupled via the input\-dependent affine mapFσF\_\{\\sigma\}\. If these bases are indeed closely related to adversarial examples as surmised in Section[2\.2](https://arxiv.org/html/2608.26963#S2.SS2), then the adversarial examples occur because a perturbationψ∈span⁡\(ψℓ\)ℓ\\psi\\in\\Span\(\\psi\_\{\\ell\}\)\_\{\\ell\}causes the outputfθ​\(x\+ψ\)f\_\{\\theta\}\(x\+\\psi\)to have a significant component inspan⁡\{ϕℓ\}ℓ\\Span\\\{\\phi\_\{\\ell\}\\\}\_\{\\ell\}\. In other words,

\(3\.13\)ϕ∗=argminϕ∈span⁡\(ϕℓ\)ℓ⁡∥fθ​\(x\+ψ\)−fθ​\(x\)−ϕ∥p\\phi\_\{\*\}=\\argmin\_\{\\phi\\in\\Span\(\\phi\_\{\\ell\}\)\_\{\\ell\}\}\\lVert\{f\_\{\\theta\}\(x\+\\psi\)\-f\_\{\\theta\}\(x\)\-\\phi\}\\rVert\_\{p\}would form the main component of the output perturbation that causes misclassification\. Figure[2](https://arxiv.org/html/2608.26963#S3.F2)shows an illustration of the input and output basis and their roles in increasing the logit of an incorrect class\.

![Refer to caption](https://arxiv.org/html/2608.26963v1/manifold_epsilon_disk.png)Figure 2\.An illustration relating LRHE input basis to adversarial examples\. The surface denotes the model logit output for the incorrect class, which increases quickly in one of the input\-basis directions\. In the linear regime, when the input is perturbed in the direction ofψ1\\psi\_\{1\}the logit changes in the direction ofϕ1\\phi\_\{1\}resulting in a misclassification\.Note that the computation ofϕℓ\\phi\_\{\\ell\}involves not just the weights of a single layer, but their compositions\. The regularization controls the stability of the composed mappings

\(3\.14\)ALAL−1⋯Aℓ\+1forℓ=1,2,…,L−1,A\_\{L\}A\_\{L\-1\}\\cdots A\_\{\\ell\+1\}\\quad\\quad\\text\{ for \}\\ell=1,2,\\dots,L\-1,rather than the behavior of the individualAℓA\_\{\\ell\}, or the usual per\-layer weight norm∥Wℓ∥\\lVert\{W\_\{\\ell\}\}\\rVert\.

Additional differences of the proposed approach \([3\.11](https://arxiv.org/html/2608.26963#S3.E11)\) from the standard adversarial training \([3\.2](https://arxiv.org/html/2608.26963#S3.E2)\) should be acknowledged\. The LRHE directions depend on the input and the network but not on the label: they describe where the map is sensitive, not where the decision boundary is nearest\. The perturbations we penalize are therefore not adversarial in the strict sense of maximizing the loss, and the correspondence between the two objectives is exact only to the extent that loss\-increasing directions coincide with amplifying ones\. Moreover, projected gradient ascent is free to follow the loss surface across activation boundaries, whereas the expansion describes a single affine cell close to the input\.

The mechanics of the training proposed above are very general and flexible; the formulation \([3\.11](https://arxiv.org/html/2608.26963#S3.E11)\) is merely the most straightforward of numerous possibilities\. As explained already, the computation of the output basis used in the regularization term \([3\.12](https://arxiv.org/html/2608.26963#S3.E12)\) is made up of two simple steps: \(1\) compute the difference between the states before and after the activation, then \(2\) apply all the subsequent layers while omitting the activation\. To compute the input basis, simply apply the transposes of all the preceding layers while ignoring the activations\. These operations generalize to other architectures that are not feedforward, for example, to architectures with attention layers\[[32](https://arxiv.org/html/2608.26963#bib.bib32)\]\. While this manuscript focuses on adversarial training that exploits the output basis only, similar training strategies can be derived using the input basis\. Another important consideration is that, if small\-norm adversarial examples are not present, the input basis can represent the directions along which the output varies most sensitively\. Due to the ease with which the input basis can be computed, the basis can be useful for sensitivity analysis of deep learning models\.

## 4\.Convolutional Neural Network for MNIST

We perform experiments using the MNIST data set\. This section describes the model and the attack used throughout, and then treats a preliminary that the expansion requires: max\-pooling layers must be rewritten before the LRHE can be computed for a convolutional network\.

Figure 3\.A diagram of the CNN architecture used for the MNIST experiments \(left\)\. The feedforward form of the same architecture, where the max\-pooling layers are converted into a ReLU network, is shown alongside \(right\)\. Four new ReLUs are added to the feedforward form, which implies additional output basis vectorsϕℓ\\phi\_\{\\ell\}will be computed\.### 4\.1\.Clean training and adversarial examples

To prepare our experiments, we train a convolutional neural network \(CNN\) using standard training\. The architecture is two convolutional layers, each followed by ReLU and two\-dimensional max\-pooling, and then two linear layers\. The hyperparameters for each layer are

- •Convolutional layer 1: 1 input channel, 64 output channels
- •2D max\-pooling layer 1: stride 2, kernel size 2
- •Convolutional layer 2: 64 input channels, 128 output channels
- •2D max\-pooling layer 2: stride 2, kernel size 2
- •Linear layer 1: input dim\.128⋅7⋅7=6272128\\cdot 7\\cdot 7=6272, output dim\. 1024
- •Linear layer 2: input dim\. 1024, output dim\. 10\.

A diagram depicting this architecture is shown in Figure[3](https://arxiv.org/html/2608.26963#S4.F3)\. We use the cross entropy loss and train with the Adam optimizer\[[12](https://arxiv.org/html/2608.26963#bib.bib12)\]at learning rate1e\-3for 20 epochs\. The trained model achieves99\.48%99\.48\\%test accuracy\.

We conduct attacks on this model, at the relative norm sizes of Definition[2\.1](https://arxiv.org/html/2608.26963#S2.E1), withAutoAttack\[[7](https://arxiv.org/html/2608.26963#bib.bib7)\]\. All hyperparameters of the ensemble are fixed, so no tuning is performed on our behalf, and the model is deterministic at inference, so the standard version of the ensemble is the appropriate one\. The ensemble attack raised none of its automatic warnings regarding the reliability of the evaluation\. For example, at a relativeℓ2\\ell^\{2\}radius ofε=0\.075\\varepsilon=0\.075, the ensemble reduced the accuracy of the clean model to88\.40%88\.40\\%\. We report the attack result for the clean model below in Section[6](https://arxiv.org/html/2608.26963#S6), in Table[2](https://arxiv.org/html/2608.26963#S6.T2)\.

To confirm that the attack above is not merely a weak one, we additionally attack the clean model with2020runs of1,0001\{,\}000PGD iterations\. These parameters differ somewhat from those used commonly in theℓ∞\\ell^\{\\infty\}case; our hyperparameter exploration found that large PGD step sizes yielded more examples\. The attack had a success rate of9\.79%9\.79\\%at a relativeℓ2\\ell^\{2\}radius ofε=0\.1\\varepsilon=0\.1\. This is significantly weaker than the28\.01%28\.01\\%success rate ofAutoAttack, but comparable to the11\.08%11\.08\\%thatAutoAttackachieves at the smaller budgetε=0\.075\\varepsilon=0\.075\.

Some of the adversarial examples found are shown in Figure[4](https://arxiv.org/html/2608.26963#S4.F4)\. They are considerably smoother than theℓ∞\\ell^\{\\infty\}\-norm adversarial examples obtained at largerε\\varepsilon\[[9](https://arxiv.org/html/2608.26963#bib.bib9)\], and appear as though the original hand\-written digits had been written in pencil and then smudged, a plausible physical scenario that can occur in real data\.

![Refer to caption](https://arxiv.org/html/2608.26963v1/pgd_attack_orig_000005.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/pgd_attack_orig_000007.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/pgd_attack_orig_000017.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/pgd_attack_orig_000021.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/pgd_attack_orig_000028.png)

Figure 4\.Adversarial examples found for the standard CNN model using PGD with relativeℓ2\\ell^\{2\}normε=0\.1\\varepsilon=0\.1\.Computing the LRHE for this model is not immediate, because the expansion presumes a particular form of the network\. We require the model in feedforward form, by which we mean a strict alternation of affine maps and a single pointwise nonlinearity \([2\.3](https://arxiv.org/html/2608.26963#S2.E3)\), where eachAℓA\_\{\\ell\}is affine andσ\\sigmaacts entrywise\. The expansion of Section[2\.2](https://arxiv.org/html/2608.26963#S2.SS2)is built directly on this structure: the correctionσ⁡\(zℓ\)−zℓ\\sigma\(z\_\{\\ell\}\)\-z\_\{\\ell\}is defined relative to the pre\-activationzℓz\_\{\\ell\}of each nonlinear layer, and the basis functionsψℓ\\psi\_\{\\ell\}are obtained by propagating that correction through the remaining affine maps\. A layer that is neither affine nor a pointwise application ofσ\\sigmatherefore has no place in the expansion until it is rewritten\.

For most of the architecture above this requirement is already met\. Fully\-connected and convolutional layers are affine in the input and are absorbed into a singleAℓA\_\{\\ell\}; strides, padding, and channel mixing change the matrix but not its affineness\. Average pooling is likewise affine and would need no special treatment\. Max\-pooling is the only component of the standard CNN that is not affine, and it is the subject of the remainder of this section\.

### 4\.2\.Max\-pooling layers in feedforward form

There are several admissible ways of rewriting a max\-pooling layer in feedforward form\. We adopt the one that expresses the pairwise maximum as a single nonlinear layer, and then build the pooling operator by composition\. Both of the representations below rest on elementary properties ofσ\\sigma\. The first recovers a linear passthrough from a nonlinearity,

\(4\.1\)z=σ⁡\(z\)−σ⁡\(−z\)for all​z∈ℝ,z\\;=\\;\\sigma\(z\)\-\\sigma\(\-z\)\\qquad\\text\{for all \}z\\in\\mathbb\{R\},and is what allows a coordinate to survive a nonlinear layer unchanged; it costs two rows rather than one, since the network form \([2\.3](https://arxiv.org/html/2608.26963#S2.E3)\) admits no skip connections\. The second expresses the maximum as a shifted rectifier,

\(4\.2\)max⁡\(a,b\)=a\+σ⁡\(b−a\)\.\\max\(a,b\)=a\+\\sigma\(b\-a\)\.Combining \([4\.1](https://arxiv.org/html/2608.26963#S4.E1)\) and \([4\.2](https://arxiv.org/html/2608.26963#S4.E2)\) witha=x1a=x\_\{1\}andb=x2b=x\_\{2\}gives the representation

\(4\.3\)max⁡\(x1,x2\)=\[1−11\]​σ​\(\[10−10−11\]​\[x1x2\]\),\\max\(x\_\{1\},x\_\{2\}\)=\\begin\{bmatrix\}1&\-1&1\\end\{bmatrix\}\\sigma\\left\(\\begin\{bmatrix\}1&0\\\\ \-1&0\\\\ \-1&1\\end\{bmatrix\}\\begin\{bmatrix\}x\_\{1\}\\\\ x\_\{2\}\\end\{bmatrix\}\\right\),which is exact for allx1,x2∈ℝx\_\{1\},x\_\{2\}\\in\\mathbb\{R\}\. Reading the three rows of the inner matrix in turn, the pre\-activations arez=\(x1,−x1,x2−x1\)z=\(x\_\{1\},\\,\-x\_\{1\},\\,x\_\{2\}\-x\_\{1\}\), so the outer row vector forms

\(4\.4\)σ⁡\(x1\)−σ⁡\(−x1\)\+σ⁡\(x2−x1\)=x1\+σ⁡\(x2−x1\)=max⁡\(x1,x2\),\\sigma\(x\_\{1\}\)\-\\sigma\(\-x\_\{1\}\)\+\\sigma\(x\_\{2\}\-x\_\{1\}\)\\;=\\;x\_\{1\}\+\\sigma\(x\_\{2\}\-x\_\{1\}\)\\;=\\;\\max\(x\_\{1\},x\_\{2\}\),where the first two terms collapse by \([4\.1](https://arxiv.org/html/2608.26963#S4.E1)\) and the remainder is \([4\.2](https://arxiv.org/html/2608.26963#S4.E2)\)\. The first two rows carryx1x\_\{1\}past the nonlinearity; only the third performs the comparison\.

This representation is not unique\. For example, note that in a CNN the input to a pooling layer is the output of a ReLU, hence nonnegative, and the passthrough may then be had for a single row rather than two\. Forx1,x2≥0x\_\{1\},x\_\{2\}\\geq 0we havex1=σ⁡\(x1\)x\_\{1\}=\\sigma\(x\_\{1\}\), and \([4\.2](https://arxiv.org/html/2608.26963#S4.E2)\) gives the two\-row representation

\(4\.5\)max⁡\(x1,x2\)=\[11\]​σ​\(\[10−11\]​\[x1x2\]\),\\max\(x\_\{1\},x\_\{2\}\)=\\begin\{bmatrix\}1&1\\end\{bmatrix\}\\sigma\\left\(\\begin\{bmatrix\}1&0\\\\ \-1&1\\end\{bmatrix\}\\begin\{bmatrix\}x\_\{1\}\\\\ x\_\{2\}\\end\{bmatrix\}\\right\),with pre\-activationsz=\(x1,x2−x1\)z=\(x\_\{1\},\\,x\_\{2\}\-x\_\{1\}\)\. This is a more economical form, but we use \([4\.3](https://arxiv.org/html/2608.26963#S4.E3)\) in practice without assuming the pooling layers’ input is known to be nonnegative\. One may also choose a symmetric version of \([4\.2](https://arxiv.org/html/2608.26963#S4.E2)\); we do not explore these options here\.

The pooling operator itself is assembled from these pairwise maxima\. LetPPdenote the patch\-extraction operator that gathers, for each output position, thessinputs lying in its pooling window\. For non\-overlapping windowsPPis a permutation followed by a reshape; for stride smaller than the window it duplicates entries\. In either casePPis linear, and is absorbed into the adjacent affine map at no cost\. The pooling layer is then thess\-way maximum applied independently to each extracted window\. Such a maximum is obtained by composing pairwise maxima in a balanced binary tree, requirings−1s\-1applications of \([4\.3](https://arxiv.org/html/2608.26963#S4.E3)\) arranged in⌈log2⁡s⌉\\lceil\\log\_\{2\}s\\rceilsuccessive nonlinear layers\. All windows of a given feature map are processed in parallel, so the maxima at a common depth of the tree occupy a single layer of \([2\.3](https://arxiv.org/html/2608.26963#S2.E3)\) whose width is the sum over windows\. For the usuals=2s=2the tree has depth one, and pooling along a single axis costs exactly one nonlinear layer\.

Two\-dimensional pooling reduces to this case, since the maximum over ans×ts\\times twindow is separable, i\.e\.max1≤i≤s,1≤j≤t⁡xi​j=max1≤i≤s⁡\(max1≤j≤t⁡xi​j\),\\max\_\{1\\leq i\\leq s,\\,1\\leq j\\leq t\}x\_\{ij\}=\\max\_\{1\\leq i\\leq s\}\(\\max\_\{1\\leq j\\leq t\}x\_\{ij\}\),so the layer is realised by applying the one\-dimensional construction along each axis in turn, with a permutation between the two stages that is again absorbed into the intervening affine map\. For the standard2×22\\times 2window this gives two nonlinear layers, one per axis, and hence three affine mapsAℓA\_\{\\ell\}in the alternating form \([2\.3](https://arxiv.org/html/2608.26963#S2.E3)\)\. In the assembled network the outer two of these merge with the affine maps of the preceding and following layers, so the net effect of a2×22\\times 2max\-pool is to insert two applications ofσ\\sigma\. Equivalently, one may treat thes​tstwindow entries as a single flat tree of depth⌈log2⁡s​t⌉\\lceil\\log\_\{2\}st\\rceil, which for the window sizes in common use coincides with the separable count and is never larger\.

Figure[3](https://arxiv.org/html/2608.26963#S4.F3)shows a diagram of the feedforward form\.

The width of the resulting layers can be counted\. Let the input to the pooling layer haveCCchannels and spatial extentH×WH\\times W, with a non\-overlapping2×22\\times 2window\. Using \([4\.3](https://arxiv.org/html/2608.26963#S4.E3)\), the first stage producesC​H​\(W/2\)CH\(W/2\)maxima at three rows each, for a pre\-activation width of32​C​H​W\\frac\{3\}\{2\}CHW; the second stage producesC​\(H/2\)​\(W/2\)C\(H/2\)\(W/2\)maxima, for a width of34​C​H​W\\frac\{3\}\{4\}CHW\. The pooling layer in the general form \([4\.3](https://arxiv.org/html/2608.26963#S4.E3)\) would cost94​C​H​W\\tfrac\{9\}\{4\}CHWpre\-activations in total, againstC​H​WCHWentering it\. Using \([4\.5](https://arxiv.org/html/2608.26963#S4.E5)\) would reduce the total to32​C​H​W\\tfrac\{3\}\{2\}CHW, a third less\.

Both \([4\.3](https://arxiv.org/html/2608.26963#S4.E3)\) and \([4\.5](https://arxiv.org/html/2608.26963#S4.E5)\) are identities, not approximations\. The rewritten network computes precisely the same function as the original CNN, and no retraining is required\. Only the representation changes, and with it the number of nonlinear layers used in the expansion\. Each nonlinear layer introduced by a pooling operator carries its own correction and therefore contributes an additional basis functionψℓ\\psi\_\{\\ell\}; a network with two2×22\\times 2pooling layers gains four basis functions, which is counted towardsLLof Section[2\.2](https://arxiv.org/html/2608.26963#S2.SS2)\.

## 5\.LRHE basis results

### 5\.1\.LRHE basis attack

We explore the relation between the LRHE input basis\(ψℓ\)ℓ\(\\psi\_\{\\ell\}\)\_\{\\ell\}and adversarial examples\.

First, we test if the input perturbations using the basis functions themselves directly lead to adversarial examples, by perturbing the clean model

\(5\.1\)fθ​\(x±α⋅ε​∥x∥p⋅ψℓ∥ψℓ∥p\),ℓ=1,2,…,L−1,f\_\{\\theta\}\\left\(x\\pm\\alpha\\cdot\\varepsilon\\lVert\{x\}\\rVert\_\{p\}\\cdot\\frac\{\\psi\_\{\\ell\}\}\{\\lVert\{\\psi\_\{\\ell\}\}\\rVert\_\{p\}\}\\right\),\\quad\\ell=1,2,\.\.\.\\,,L\-1,for step\-sizesα=0\.5,1,1\.5,…,3\.0\\alpha=0\.5,1,1\.5,\.\.\.\\,,3\.0, withε=0\.1\\varepsilon=0\.1and checking whether the predicted label switches to an incorrect one\. We do not clip the resulting perturbed input to the range\[0,1\]\[0,1\]since that would alter the linear scaling with respect toα\\alpha, so the final values veer outside of the unit interval by a small amount\.

Table 1\.Adversarial accuracy of the clean model after LRHE basis attack for each step\-sizeα\\alpha\([5\.1](https://arxiv.org/html/2608.26963#S5.E1)\)\. Difference from clean accuracy of 99\.48% is in the parentheses\.![Refer to caption](https://arxiv.org/html/2608.26963v1/lrhe_attack_009966_3-0000.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/lrhe_attack_009468_2-5000.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/lrhe_attack_009980_2-0000.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/lrhe_attack_008992_1-5000.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/lrhe_attack_007974_1-0000.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/lrhe_attack_009951_0-5000.png)

Figure 5\.LRHE basis attack results\. Six adversarial examples found for varying step\-sizeα\\alpha\. The original image and correctly predicted label is shown, alongside the perturbed input and the perturbation\. The corresponding logit scores and the changes are also shown below the examples\.This LRHE basis attack had a success rate of 7\.79% as shown in Table[1](https://arxiv.org/html/2608.26963#S5.T1), showing that these basis functions are indeed related to directions in the input space that the model is sensitive with respect to\. Largerα\\alpha’s tend to yield more examples, which is to be expected\. To ascertain that this is a significant success rate, we conduct the same experiment withα≤3\.0\\alpha\\leq 3\.0and random vectors in place ofψℓ\\psi\_\{\\ell\}, and find a median success rate of 1\.88% \(max 1\.96%, min 1\.81%, five different trials\)\.

This shows that LRHE input basis functions span a linear space of perturbations that are significantly adversarial\. On the other hand, this attack result is not as strong asAutoAttackwhich had a success rate of 28\.01% for the rangeα≤1\.0\\alpha\\leq 1\.0\(see Table[2](https://arxiv.org/html/2608.26963#S6.T2)below\)\.

### 5\.2\.Projection to the orthogonal complement of LRHE basis

Next, we check if the LRHE input basis\(ψℓ\)ℓ\(\\psi\_\{\\ell\}\)\_\{\\ell\}can be used to remove the adversarial effects from examples discovered throughAutoAttack\. We form an orthogonal matrix by forming the matrixΨ\\Psiwhose columns are made up of the LRHE input basis, then taking the QR factorization to obtain the matrixΨ^\\widehat\{\\Psi\}with orthonormal columns,

\(5\.2\)Ψ:=\[ψ1​∣⋯∣​ψL−1\],Ψ^​R=Ψ,Ψ^⊤​Ψ^=I\.\\Psi:=\[\\psi\_\{1\}\\mid\\cdots\\mid\\psi\_\{L\-1\}\],\\quad\\widehat\{\\Psi\}R=\\Psi,\\quad\\widehat\{\\Psi\}^\{\\top\}\\widehat\{\\Psi\}=I\.For each adversarial exampleδ​x\\delta x, we compute the projected versionδ​xproj\\delta x\_\{\\text\{proj\}\}

\(5\.3\)δ​xproj=clip\[0,1\]​\(\(I−Ψ^​Ψ^⊤\)⋅δ​x\),\\delta x\_\{\\text\{proj\}\}=\\text\{clip\}\_\{\[0,1\]\}\(\(I\-\\widehat\{\\Psi\}\\widehat\{\\Psi\}^\{\\top\}\)\\cdot\\delta x\),then check if these examples are still adversarial\. After projections, a majority of the adversarial examples no longer switched the predicted labels, that is, the projected perturbations were no longer adversarial\. For the projected examples, the adversarial success rate fell to 9\.20% from 28\.01%, about a third\. To check that this correction is significant, we conduct the same experiment with a random orthogonal basis, and find median success rate 25\.42% \(max\. 25\.45%, min\. 25\.34%, five different trials\)\. A sample of these projection test results is shown in Figure[6](https://arxiv.org/html/2608.26963#S5.F6)\. Note that a similar test was performed for the regression task in\[[21](https://arxiv.org/html/2608.26963#bib.bib21)\]\.

![Refer to caption](https://arxiv.org/html/2608.26963v1/adv_proj_000019.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/adv_proj_000025.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/adv_proj_000089.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/adv_proj_000175.png)

Figure 6\.Orthogonal complement projection of adversarial examples leads to corrections\. Original test image, the adversarial example, and the projected example are shown, along with the differences\. Output logit scores and corresponding changes are also shown\.

## 6\.Adversarial training results

### 6\.1\.Training heuristics

The regularized training objective \([3\.11](https://arxiv.org/html/2608.26963#S3.E11)\) requires more fine\-tuning compared to the standard objective involving only the misfit loss\. Straightforward Adam optimizer with standard fixed choices forλ\\lambdaandγ\\gammaresulted in improved adversarial robustness but test accuracy was 1–2% lower than that of the original model\. As the regularized problem has new hyper\-parametersλ\\lambdaandγ\\gamma, we tune them to achieve better results\. As for the hyper\-parameterγ\\gamma, it is sampled randomly from a uniform distribution at each iteration,γk∼𝒰⁡\[−1,1\]\.\\gamma\_\{k\}\\sim\\mathcal\{U\}\[\-1,1\]\.Note thatγ\\gammais not to be confused with the budgetε\\varepsilon; it is a dimensionless scale onϕℓ\\phi\_\{\\ell\}and not a perturbation radius\. The regularization parameterλ\\lambdais updated at each Adam iteration to be adjusted from its base valueλbase=1e\-2\\lambda\_\{\\textrm\{base\}\}=\\texttt\{1e\-2\}, that is, at iterationkk,

\(6\.1\)λk:=ℒ​\(fθk​\(x\),y\)ℛ⁡\(θk,x,γk\)⋅λbase\\lambda\_\{k\}:=\\frac\{\\mathcal\{L\}\(f\_\{\\theta\_\{k\}\}\(x\),y\)\}\{\\mathcal\{R\}\(\\theta\_\{k\},x;\\gamma\_\{k\}\)\}\\cdot\\lambda\_\{\\textrm\{base\}\}so that the two terms are roughly at the same scale during training\. This adjustment is done before the backward pass computation\.

We also deploy a fixed decay rate schedulerMultiplicativeLRimplemented in PyTorch, which reduces the learning rate byfactor=0\.9every two epochs\. We run up to 70 epochs of Adam optimizer, and select the model with the best validation accuracy amoung 25 different random initializations\. The trained model achieves test accuracy of 99\.53%, on par with the original model\.

To provide a comparison with adversarial examples with boundedℓ∞\\ell^\{\\infty\}\-norms we setε=0\.05\\varepsilon=0\.05, the level at which the adversarial accuracy of the original model is similar to our setting\. The threat model is againAutoAttack, which yielded stronger attack results than our own PGD attack which had the configuration 25 runs of PGD attack with 500 iterations with step\-size 0\.01 plus 25 runs of PGD attack with 500 iterations and step\-size 0\.005\.

![Refer to caption](https://arxiv.org/html/2608.26963v1/inout_basis_orig_000000.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/inout_basis_orig_000001.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/inout_basis_reg_009989.png)

![Refer to caption](https://arxiv.org/html/2608.26963v1/inout_basis_reg_009998.png)

Figure 7\.LRHE input and output bases for original and adversarially trained models\. Top two examples are from the LRHE of the original model, and bottom two examples from the adversarially trained model\.
### 6\.2\.Adversarial robustness results

To illustrate the effect of the adversarial training, we compare the input and output basis of the clean model and adversarially trained model\. The norm of the output basis is significantly smaller in the case of the regularized model, by two orders of magnitude; see Figure[7](https://arxiv.org/html/2608.26963#S6.F7)for a sample of LRHE input and output basis\.

Table 2\.Adversarial accuracy \(percentage\) under relativeℓ2\\ell^\{2\}perturbations \(∥δ​x∥2≤ε​‖x‖2\\lVert\{\\delta x\}\\rVert\_\{2\}\\leq\\varepsilon\\\|x\\\|\_\{2\}; with‖x‖2≈9\.2\\\|x\\\|\_\{2\}\\approx 9\.2on MNIST,ε=0\.150\\varepsilon=0\.150corresponds to an absolute radius‖δ​x‖2\\\|\\delta x\\\|\_\{2\}of≈1\.4\\approx 1\.4\) and underℓ∞\\ell^\{\\infty\}perturbations\. All defended models were trained atε=0\.075\\varepsilon=0\.075in the relativeℓ2\\ell^\{2\}norm; theℓ∞\\ell^\{\\infty\}block therefore measures cross\-norm transfer\. Parenthesized values give accuracy lost relative to each model’s own clean accuracy\.The adversarial training results are shown in Table[2](https://arxiv.org/html/2608.26963#S6.T2)\.

#### 6\.2\.1\.Clean accuracy

Table[2](https://arxiv.org/html/2608.26963#S6.T2)reports clean test accuracy alongside adversarial accuracy under both norms\. The four models reach99\.48%99\.48\\%for the unregularized baseline,99\.53%99\.53\\%for the LRHE\-regularized model,99\.46%99\.46\\%for4040\-step and99\.43%99\.43\\%for 3\-step adversarial training\. These are separated by at most ten test examples, so the ordering is inconclusive\. The loss of clean accuracy that adversarial training is sometimes reported to incur\[[31](https://arxiv.org/html/2608.26963#bib.bib31)\]is not visible at this scale, though this is potentially because MNIST clean accuracy is saturated\. All four models attain clean accuracy above99\.4%99\.4\\%, and we therefore confine the comparison to adversarial accuracy\.

#### 6\.2\.2\.The small\-budget regime

At small perturbation budgets the regularized model matches both PGD configurations\. Under relativeℓ2\\ell^\{2\}perturbations atε=0\.010\\varepsilon=0\.010it retains99\.27%99\.27\\%against99\.20%99\.20\\%for4040\-step and99\.19%99\.19\\%for 3\-step training; atε=0\.020\\varepsilon=0\.020the three stand at98\.97%98\.97\\%,99\.02%99\.02\\%and98\.96%98\.96\\%\. Across the rangeε≤0\.020\\varepsilon\\leq 0\.020the three defended models are on par, differences among them between one and fourteen test examples\. This parity is of particular interest: a regularizer that takes no derivative with respect to the input is indistinguishable here from adversarial training that takes 40 per update\.

#### 6\.2\.3\.Larger budgets

This parity does not extend to larger budgets\. Under relativeℓ2\\ell^\{2\}perturbations the PGD\-40\-AT model moves ahead betweenε=0\.012\\varepsilon=0\.012andε=0\.014\\varepsilon=0\.014, and the PGD\-3\-AT model betweenε=0\.020\\varepsilon=0\.020andε=0\.025\\varepsilon=0\.025; past these points PGD\-AT is increasingly robust by comparison, reaching a margin of15\.7015\.70percentage points over the regularized model atε=0\.150\\varepsilon=0\.150\. Underℓ∞\\ell^\{\\infty\}perturbations multi\-step training is ahead at every budget we tabulate, and the margin atε=0\.150\\varepsilon=0\.150is35\.1935\.19percentage points\. To summarize, the regularizer matches PGD\-AT forε≤0\.02\\varepsilon\\leq 0\.02in the relativeℓ2\\ell^\{2\}norm, and is weaker beyond that range, as well as underℓ∞\\ell^\{\\infty\}at budgetsε≥0\.01\\varepsilon\\geq 0\.01\.

This behavior is consistent with the approximation on which the method relies\. The expansion truncates the ReLU representation at rank one, that is, once the vectorvℓv\_\{\\ell\}is fixed it is first\-order accurate in a cone about the input whose extent is determined by the neglected terms; as the perturbation budget grows, the adversary is increasingly free to leave the region in which the retained directions describe the network’s local action\. Degradation at largeε\\varepsilonis thus predicted by the construction, and suggests that the usable range should extend with improvement of the approximation \([2\.11](https://arxiv.org/html/2608.26963#S2.E11)\)\.

#### 6\.2\.4\.Asymmetry between the two norms

The steeper decline underℓ∞\\ell^\{\\infty\}merits comment\. The subspace identified by the expansion is defined by orthogonal projections and is intrinsically anℓ2\\ell^\{2\}object\. A fixedℓ∞\\ell^\{\\infty\}budget on MNIST admits perturbations ofℓ2\\ell^\{2\}norm up tonin​ε=28​ε\\sqrt\{n\_\{\\text\{in\}\}\}\\varepsilon=28\\varepsilon, so theℓ∞\\ell^\{\\infty\}rows of Table[2](https://arxiv.org/html/2608.26963#S6.T2)probe substantially largerℓ2\\ell^\{2\}displacements than the corresponding relative\-ℓ2\\ell^\{2\}rows\. The asymmetry is therefore expected, and the two blocks of the table should not be read as directly comparable\.

Table 3\.Black\-box attack for the adversarially trained model\. Adversarial accuracy \(percentage\) computed forℓ2\\ell^\{2\}\-relativeε=0\.075\\varepsilon=0\.075\.
#### 6\.2\.5\.Black\-box attacks

We adversarially train a second model with a different random initialization, apply the same white\-boxAutoAttackto the new model to find adversarial examples, then use these examples to attack our adversarially trained model, in a black\-box attack\. The adversarial accuracy of our adversarially trained model with respect to white\-box attack was94\.94%94\.94\\%, whereas black\-box attacks using adversarial examples from the second model yield an adversarial accuracy of97\.28%97\.28\\%, and black\-box attacks using examples from the original \(clean\) model yield98\.47%98\.47\\%\. The results are shown in Table[3](https://arxiv.org/html/2608.26963#S6.T3)\.

Table 4\.Wall time per epoch during PGD adversarial training\. Batch size is 30\. The wall time was measured using standard PyTorch implementation on NVIDIA GeForce RTX 3090 with 24GB of memory\.
#### 6\.2\.6\.Computational cost

Wall\-clock time per epoch for PGD adversarial training is linear in the number of inner steps,

\(6\.2\)T⁡\(K\)≈5\.4\+2\.65​Kseconds,T\(K\)\\approx 5\.4\+2\.65K\\quad\\text\{seconds\},at batch size3030, the fixed term reflecting the cost of a standard training epoch and the marginal term the cost of one input\-gradient computation over the training set\. LRHE\-regularized training requires12\.712\.7s/epoch under identical conditions, equivalent toK≈2\.8K\\approx 2\.8PGD steps and an8\.7×8\.7\\timesreduction relative to the4040\-step configuration of Table[2](https://arxiv.org/html/2608.26963#S6.T2)\. It is also below the13\.413\.4s that \([6\.2](https://arxiv.org/html/2608.26963#S6.E2)\) predicts for 3\-step training, so the cost\-matched comparison in Table[2](https://arxiv.org/html/2608.26963#S6.T2)is made against a baseline that is if anything slightly more expensive than our method\. Single\-step methods such as those of\[[34](https://arxiv.org/html/2608.26963#bib.bib34)\]and\[[23](https://arxiv.org/html/2608.26963#bib.bib23)\]remain cheaper in absolute terms; our claim is not to be the cheapest available defense but to obtain robustness at a fraction of the cost of multi\-step training without differentiating with respect to the input at any point\. This is a constant\-factor trade rather than an asymptotic improvement\. LRHE does not eliminate the extra computation adversarial training requires: Where PGD performsKKsequential forward\-backward passes differentiating with respect to the input, LRHE performs a single parameter\-gradient computation over a forward map that isLLtimes more expensive\. The comparison therefore has a critical point: LRHE is cheaper thanKK\-step PGD training forK≥3K\\geq 3and more expensive below it\. Since robust accuracy on MNIST is conventionally obtained withK=40K=40, the regime of interest lies well above this point\.

The two costs also differ in structure, and not only in size\. TheL−1L\-1partial products in \([3\.5](https://arxiv.org/html/2608.26963#S3.E5)\) are mutually independent, and they share suffixes:ϕℓ\\phi\_\{\\ell\}andϕℓ′\\phi\_\{\\ell^\{\\prime\}\}pass through the same affine maps abovemax⁡\(ℓ,ℓ′\)\\max\(\\ell,\\ell^\{\\prime\}\)\. They may therefore be accumulated in a single sweep down the layers, adding the correctionσ​zℓ−zℓ\\sigma z\_\{\\ell\}\-z\_\{\\ell\}at each stage before applying the next affine map, rather than being formed one at a time\. TheKKsteps of projected gradient ascent admit no such reorganization, since each step requires the perturbation produced by the one before it\. The timings above were obtained without exploiting this, and are in that respect conservative with respect to the regularizer\.

#### 6\.2\.7\.Limitations

Two limitations of this evaluation should be stated plainly\. First, both defended models were trained atε=0\.075\\varepsilon=0\.075with the relativeℓ2\\ell^\{2\}\-norm, and we evaluate over budgets up toε=0\.150\\varepsilon=0\.150; we do not report results at theε=0\.3\\varepsilon=0\.3ℓ∞\\ell^\{\\infty\}budget conventional for MNIST, at which neither model retains useful accuracy\. Second, all results are on MNIST, and we refrain from claiming that the accuracy of the approximation required to capture the adversarially relevant subspace, or the budget range over which the approximation remains accurate, transfers to higher\-resolution data\.

## 7\.Conclusion

We have argued that for adversarial examples of small relativeℓ2\\ell^\{2\}norm, the directions along which a network is most sensitive need not be searched for\. They are already determined by the activation pattern, and the expansion of\[[21](https://arxiv.org/html/2608.26963#bib.bib21)\]makes them available as a byproduct of the forward pass\. The consequence for adversarial training is that the inner maximization of the min\-max formulation can be replaced by a penalty on a fixed set of directions recomputed at each step, so that no derivative with respect to the input is taken at any point during training\. The main contribution of this work is to establish that input\-gradient\-free adversarial training is possible\. The other methods that dispense with the inner maximization invariably measure the local geometry by differentiating with respect to the input; the construction here shows that this is not required\.

The following observations are made regarding the proposed construction for the MNIST setting\. The subspace spanned by the input basis is where much of the adversarial perturbations lie: it has dimension seven against an input dimension of784784, and projecting perturbations onto its orthogonal complement removes roughly two thirds of the adversarial effect, against almost none for a random subspace of the same dimension\. The resulting training procedure costs the equivalent of2\.82\.8PGD steps per epoch, less than 3\-step adversarial training and a factor of8\.78\.7below 40\-step\. Within the small budget range for which the expansion is descriptive, the models it produces match both, the 3\-step robust accuracy up toε≤0\.02\\varepsilon\\leq 0\.02and the 40\-step robust accuracy up toε≤0\.012\\varepsilon\\leq 0\.012; beyond that range the multi\-step training pulls ahead, for the reasons discussed below\. Still, we regard the small\-budget regime as the setting in which the original motivation for studying adversarial examples, that the perturbation be imperceptible, is still intact\. Finally, the robustness so obtained does not appear to rest on gradient masking: adversarial examples transferred from an independently trained model, and from the undefended model, are considerably less effective against the regularized network than examples constructed against it directly, which is the ordering a genuine defense is expected to show\[[3](https://arxiv.org/html/2608.26963#bib.bib3)\]\.

The limits of the method follow from the same source as its efficiency\. The reflection vectors are fixed by the activation pattern at the clean input, so the description they provide is local, and it ceases to hold once a perturbation is large enough to cross activation boundaries in quantity\. Multi\-step adversarial training is under no such restriction, and at large budgets the difference is substantial\. Our directions are moreover independent of the label: they record where the map amplifies, not where the decision boundary lies, and the two coincide only approximately\.

Several future directions follow naturally\. The most immediate is a wider empirical studies: results on datasets where clean accuracy is not saturated, and where the robustness\-accuracy tradeoff is pronounced enough that the absence of a drop would carry weight\. Comprehensive replication across random seeds is needed before the margins reported here at individual budgets can be regarded as settled\.

A second direction concerns architecture\. The two operations underlying the construction, taking the difference between pre\- and post\-activation states and propagating it through the remaining affine maps, do not depend on the network being feedforward\. They require only that the nonlinearity be separable from the affine structure at a given input, which holds for attention layers as well, where the attention matrix computed in the forward pass plays the role that the activation pattern plays here\. Whether the resulting subspace carries the same relation to adversarial perturbations in that setting is open\.

Finally, the input basis has uses beyond the training procedure studied here\. It is cheap to compute, it requires no differentiation, and in the absence of adversarial examples it identifies the input directions to which the model responds most strongly\. That makes it a candidate tool for sensitivity analysis of trained models generally, independently of any robustness objective\.

### Code availability

The code used to generate the results above are publicly accessible in the code repository\[[19](https://arxiv.org/html/2608.26963#bib.bib19)\]\.

## References

- \[1\]M\. Andriushchenko, F\. Croce, N\. Flammarion, and M\. Hein,Square attack: A query\-efficient black\-box adversarial attack via random search, in European Conference on Computer Vision \(ECCV\), 2020\.
- \[2\]M\. Andriushchenko and N\. Flammarion,Understanding and improving fast adversarial training, in Advances in Neural Information Processing Systems \(NeurIPS\), 2020\.
- \[3\]A\. Athalye, N\. Carlini, and D\. Wagner,Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples, in Proceedings of the 35th International Conference on Machine Learning \(ICML\), vol\. 80 of Proceedings of Machine Learning Research, 2018, pp\. 274–283\.
- \[4\]N\. Carlini, A\. Athalye, N\. Papernot, W\. Brendel, J\. Rauber, D\. Tsipras, I\. Goodfellow, A\. Madry, and A\. Kurakin,On evaluating adversarial robustness, arXiv preprint arXiv:1902\.06705, \(2019\)\.
- \[5\]M\. Cisse, P\. Bojanowski, E\. Grave, Y\. Dauphin, and N\. Usunier,Parseval networks: Improving robustness to adversarial examples, in Proceedings of the 34th International Conference on Machine Learning \(ICML\), vol\. 70 of Proceedings of Machine Learning Research, 2017\.
- \[6\]F\. Croce and M\. Hein,Minimally distorted adversarial examples with a fast adaptive boundary attack, in Proceedings of the 37th International Conference on Machine Learning \(ICML\), vol\. 119 of Proceedings of Machine Learning Research, 2020\.
- \[7\]F\. Croce and M\. Hein,Reliable evaluation of adversarial robustness with an ensemble of diverse parameter\-free attacks, in Proceedings of the 37th International Conference on Machine Learning \(ICML\), vol\. 119 of Proceedings of Machine Learning Research, 2020, pp\. 2206–2216\.
- \[8\]M\. Fazlyab, A\. Robey, H\. Hassani, M\. Morari, and G\. J\. Pappas,Efficient and accurate estimation of Lipschitz constants for deep neural networks, in Advances in Neural Information Processing Systems \(NeurIPS\), 2019\.
- \[9\]I\. J\. Goodfellow, J\. Shlens, and C\. Szegedy,Explaining and harnessing adversarial examples, in International Conference on Learning Representations \(ICLR\), 2015\.
- \[10\]D\. Hendrycks, K\. Zhao, S\. Basart, J\. Steinhardt, and D\. Song,Natural adversarial examples, in IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\), 2021, pp\. 15262–15271\.
- \[11\]J\. Hoffman, D\. A\. Roberts, and S\. Yaida,Robust learning with jacobian regularization, arXiv preprint arXiv:1908\.02729, \(2019\)\.
- \[12\]D\. P\. Kingma and J\. Ba,Adam: A method for stochastic optimization, in International Conference on Learning Representations \(ICLR\), 2015\.
- \[13\]Q\. Li, S\. Haque, C\. Anil, J\. Lucas, R\. Grosse, and J\.\-H\. Jacobsen,Preventing gradient attenuation in Lipschitz constrained convolutional networks, in Advances in Neural Information Processing Systems \(NeurIPS\), 2019\.
- \[14\]T\. Li, Y\. Wu, S\. Chen, K\. Fang, and X\. Huang,Subspace adversarial training, in IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\), 2022\.
- \[15\]A\. Madry, A\. Makelov, L\. Schmidt, D\. Tsipras, and A\. Vladu,Towards deep learning models resistant to adversarial attacks, in International Conference on Learning Representations \(ICLR\), 2018\.
- \[16\]T\. Miyato, T\. Kataoka, M\. Koyama, and Y\. Yoshida,Spectral normalization for generative adversarial networks, in International Conference on Learning Representations \(ICLR\), 2018\.
- \[17\]S\.\-M\. Moosavi\-Dezfooli, A\. Fawzi, J\. Uesato, and P\. Frossard,Robustness via curvature regularization, and vice versa, in IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\), 2019\.
- \[18\]C\. Qin, J\. Martens, S\. Gowal, D\. Krishnan, K\. Dvijotham, A\. Fawzi, S\. De, R\. Stanforth, and P\. Kohli,Adversarial robustness through local linearization, in Advances in Neural Information Processing Systems \(NeurIPS\), 2019\.
- \[19\]T\.C\. Johnson, D\. Rim,dsrim/lrhe\-at: Code repository, Aug\. 2026,[doi:10\.5281/zenodo\.22119131](http://dx.doi.org/10.5281/zenodo.22119131)\.
- \[20\]D\. Rim, R\. Baraldi, C\. M\. Liu, R\. J\. LeVeque, and K\. Terada,Tsunami early warning from global navigation satellite system data using convolutional neural networks, Geophysical Research Letters, 49 \(2022\), p\. e2022GL099511,[doi:10\.1029/2022GL099511](http://dx.doi.org/10.1029/2022GL099511)\.
- \[21\]D\. Rim, S\. Suri, S\. Hong, K\. Lee, and R\. J\. LeVeque,A stability analysis of neural networks and its application to tsunami early warning, Journal of Geophysical Research: Machine Learning and Computation, 1 \(2024\), p\. e2024JH000223,[doi:10\.1029/2024JH000223](http://dx.doi.org/10.1029/2024JH000223)\.
- \[22\]A\. S\. Ross and F\. Doshi\-Velez,Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients, in AAAI Conference on Artificial Intelligence, 2018\.
- \[23\]A\. Shafahi, M\. Najibi, A\. Ghiasi, Z\. Xu, J\. Dickerson, C\. Studer, L\. S\. Davis, G\. Taylor, and T\. Goldstein,Adversarial training for free\!, in Advances in Neural Information Processing Systems \(NeurIPS\), 2019\.
- \[24\]S\. Singla and S\. Feizi,Skew orthogonal convolutions, in Proceedings of the 38th International Conference on Machine Learning \(ICML\), vol\. 139 of Proceedings of Machine Learning Research, 2021\.
- \[25\]S\. Singla, S\. Singla, and S\. Feizi,Householder activations for provable robustness against adversarial attacks, arXiv preprint arXiv:2108\.04062, \(2021\)\.
- \[26\]S\. Singla, S\. Singla, and S\. Feizi,Improved deterministicℓ2\\ell\_\{2\}robustness on CIFAR\-10 and CIFAR\-100, in International Conference on Learning Representations \(ICLR\), 2022\.
- \[27\]C\. Szegedy, W\. Zaremba, I\. Sutskever, J\. Bruna, D\. Erhan, I\. Goodfellow, and R\. Fergus,Intriguing properties of neural networks, in International Conference on Learning Representations \(ICLR\), 2014\.
- \[28\]F\. Tramèr, N\. Carlini, W\. Brendel, and A\. Madry,On adaptive attacks to adversarial example defenses, in Advances in Neural Information Processing Systems \(NeurIPS\), 2020\.
- \[29\]F\. Tramèr, N\. Papernot, I\. Goodfellow, D\. Boneh, and P\. McDaniel,The space of transferable adversarial examples, arXiv preprint arXiv:1704\.03453, \(2017\)\.
- \[30\]A\. Trockman and J\. Z\. Kolter,Orthogonalizing convolutional layers with the Cayley transform, in International Conference on Learning Representations \(ICLR\), 2021\.
- \[31\]D\. Tsipras, S\. Santurkar, L\. Engstrom, A\. Turner, and A\. Madry,Robustness may be at odds with accuracy, in International Conference on Learning Representations \(ICLR\), 2019\.
- \[32\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin,Attention is all you need, in Advances in Neural Information Processing Systems \(NeurIPS\), vol\. 30, 2017, pp\. 5998–6008\.
- \[33\]A\. Virmaux and K\. Scaman,Lipschitz regularity of deep neural networks: Analysis and efficient estimation, in Advances in Neural Information Processing Systems \(NeurIPS\), 2018\.
- \[34\]E\. Wong, L\. Rice, and J\. Z\. Kolter,Fast is better than free: Revisiting adversarial training, in International Conference on Learning Representations \(ICLR\), 2020\.
- \[35\]D\. Zhang, T\. Zhang, Y\. Lu, Z\. Zhu, and B\. Dong,You only propagate once: Accelerating adversarial training via maximal principle, in Advances in Neural Information Processing Systems \(NeurIPS\), 2019\.