MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers

# MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
Source: [https://arxiv.org/html/2607.10034](https://arxiv.org/html/2607.10034)
Roberto Garcia1†Jerry Liu1\*†Ronny Junkins2\* Sabri Eyuboglu2Atri Rudra3Chris Ré2 1Institute for Computational & Mathematical Engineering, Stanford University 2Department of Computer Science, Stanford University 3Department of Computer Science and Engineering, University at Buffalo

###### Abstract

Large language models \(LLMs\) store factual knowledge in their parameters\. While recent work has shown that this knowledge resides in MLP layers, existing constructive and mechanistic interpretability models of fact\-storage in LLMs fail to explain the surprising empirical phenomenon that they store facts at an information\-theoretically optimal rate\. In this work, we develop a theoretical account of this phenomenon\. We develop the first Transformer\-compatible fact\-storing MLP closed\-form construction that satisfies the following three properties empirically observed in LLMs: it \(i\) attains optimal fact storage scaling, \(ii\) handles arbitrary input/output geometries, and \(iii\) works inside Transformers\. Key to our work is to analyze the*decoding margin*of MLPs, whereas prior work only studies MLP fact storage\. Under isotropic embeddings, our construction achieves information\-theoretically optimal storage capacity scaling and requires1010\-104×104\\timesfewer parameters at matched fact count than prior constructions\. For arbitrary key and value embeddings, we show that our construction attains the same storage capacity scaling, up to penalization factors depending on the embedding geometries\. Moreover, we demonstrate that our constructed MLPs can be used within Transformer blocks for factual recall tasks at optimal capacity scaling, requiring1515\-63×63\\timesfewer parameters at matched fact count than prior constructions\. Finally, as a proof\-of\-concept, we show that fact\-storing MLPs enable*modular fact editing*by swapping a Transformer’s MLP with a new one\.

22footnotetext:Corresponding authors:[robgarct@stanford\.edu](https://arxiv.org/html/2607.10034v1/mailto:[email protected]),[jerrywliu@stanford\.edu](https://arxiv.org/html/2607.10034v1/mailto:[email protected])\.33footnotetext:Code is released at[https://github\.com/HazyResearch/hebbian\-mlps](https://github.com/HazyResearch/hebbian-mlps)\.## 1Introduction

Large language models \(LLMs\) achieve remarkable performance across domains such as mathematics, science, and law\(Google DeepMind,[2024](https://arxiv.org/html/2607.10034#bib.bib53); Guhaet al\.,[2023](https://arxiv.org/html/2607.10034#bib.bib18); Saabet al\.,[2024](https://arxiv.org/html/2607.10034#bib.bib15)\), in part because they can store vast amounts of knowledge in their parameters\(Petroniet al\.,[2019](https://arxiv.org/html/2607.10034#bib.bib16); Menget al\.,[2023a](https://arxiv.org/html/2607.10034#bib.bib38)\)\. Prior work suggests that knowledge in Transformers is stored in Multi\-Layer Perceptrons \(MLPs\) as key\-value mappings, or*facts*\(Gevaet al\.,[2021](https://arxiv.org/html/2607.10034#bib.bib33); Daiet al\.,[2022](https://arxiv.org/html/2607.10034#bib.bib39)\)\. However, despite these findings, fact\-storing MLPs remain poorly understood\.

While prior work has made important progress toward understanding and modeling fact storage in MLPs, existing models fail to capture three empirically observed properties of LLM fact storage: MLPs must \(i\) attain optimal fact storage scaling, \(ii\) handle arbitrary input/output geometries, and \(iii\) work inside Transformers\. Mechanistic interpretability work\(Gevaet al\.,[2021](https://arxiv.org/html/2607.10034#bib.bib33); Daiet al\.,[2022](https://arxiv.org/html/2607.10034#bib.bib39)\)assumes MLPs store facts in individual neurons, but these models lead to suboptimal storage\-capacity scaling\. More recently,Nichaniet al\.\([2024](https://arxiv.org/html/2607.10034#bib.bib40)\)study LLM fact storage by introducing an MLP*weight construction*\(NTK MLP\) with theoretical capacity guarantees\. However, existing constructions \(i\) theoretically and empirically do not attain the empirically observed information\-theoretically optimal capacity scaling of LLMs; \(ii\) are restricted to isotropic \(e\.g\., uniformly spherical\) embedding distributions, whereas LLM embeddings are anisotropic\(Ethayarajh,[2019](https://arxiv.org/html/2607.10034#bib.bib81); Razzhigaevet al\.,[2024](https://arxiv.org/html/2607.10034#bib.bib82)\); and \(iii\) cannot be used by Transformer blocks for factual recall tasks, such as answering “What is the capital of France?”\.

Our core insight is to study MLP decoding margin scaling \([Section2\.1](https://arxiv.org/html/2607.10034#S2.SS1.SSS0.Px1)\)\. Where prior works focus on fact\-storage under noiseless key queries, our decoding margin study allows us to develop the first closed\-form MLP construction that is usable by Transformers for factual recall\. Consequently, we theoretically show that*Transformer blocks are capable of information\-theoretically optimal fact storage*– providing the first theoretical explanation that aligns with the empirical capacity scaling observed in pretrained LLMs\.

Our construction is the first to capture all three empirically observed properties of LLM fact storage:

![Refer to caption](https://arxiv.org/html/2607.10034v1/x1.png)Figure 1:\(A\) Decoding margin illustration:Top plot shows an MLP with large decoding margin, bottom plot with low decoding margin\. The MLP with larger decoding margin has a larger “margin of error”, making it more robust when queried with perturbed versions of𝐤1\\mathbf\{k\}\_\{1\}\.\(B\) MLPs are information\-theoretically optimal:we show that MLPs are Hebbian memories in kernel space and that the storage capacity of MLPs scales at the information\-theoretically optimal rate\.\(C\) Transformer blocks are information\-theoretically optimal:we show that the storage capacity of Transformer blocks scales at the information\-theoretically optimal rate, provided the attention noise remains bounded\.- •Optimal fact\-storage capacity and margin \(Sections[3](https://arxiv.org/html/2607.10034#S3),[4](https://arxiv.org/html/2607.10034#S4)\)\.Our first step toward matching the empirically optimal fact\-storage scaling of LLMs is to demonstrate that 1\) MLPs are Hebbian kernel memories and 2\) that Hebbian kernel memories achieve asymptotically optimal fact\-storage scaling\. We show this by developing a closed\-form MLP construction, equivalent to a Hebbian memory with sketched quadratic kernel, which provably attains a fact storage capacity ofF=Θ​\(W/log⁡W\)F=\\Theta\(W/\\log W\)forFFfacts usingWWparameters \([Section4\.3](https://arxiv.org/html/2607.10034#S4.SS3)\)\. Theoretically, our construction closes the optimality gap over prior constructions by a factor oflog11⁡F\\log^\{11\}Funder isotropic embeddings\. Moreover, we show that, at matched fact count, the NTK baseline requires1010\-104×104\\timesmore parameters than our best data\-dependent kernel construction \([Figure2](https://arxiv.org/html/2607.10034#S4.F2)c\)\.
- •Handling arbitrary embedding geometries \([Section4](https://arxiv.org/html/2607.10034#S4)\)\.We next study how arbitrary embedding geometries affect MLP margin and storage capacity scaling\. We show that generalizing the MLP margin and capacity scaling to arbitrary embedding geometries introduces four embedding\-geometric statistics multiplicatively into the information\-theoretically optimal scaling derived for isotropic embeddings \([Theorem4\.3](https://arxiv.org/html/2607.10034#S4.Thmtheorem3)\)\. Intuitively, these statistics penalize the margin and capacity scaling by how clustered the key and value embeddings are\. Empirically, we show that our generalized bounds characterize the empirical decoding margin scaling precisely \(R2≥0\.95R^\{2\}\\geq 0\.95;[Figure5](https://arxiv.org/html/2607.10034#A1.F5)c\)\. Furthermore, we find that the capacity gap between our construction and trained MLPs is preserved even under anisotropic embeddings\.
- •MLPs usable within Transformers for factual recall \([Section5](https://arxiv.org/html/2607.10034#S5)\)\.Towards understanding MLP usage within LLMs, we find that non\-trivial decoding margin is needed by MLPs to be used for factual recall within Transformer blocks\. Attention layers produce imperfect, noisy queries, so the fact\-storing MLPs they query should be robust to noise\. Building on this insight, we demonstrate theoretically and empirically, for the first time, that Transformer blocks can retrieve facts from MLPs with optimal fact\-storage capacity to solve factual recall tasks \([Theorem5\.2](https://arxiv.org/html/2607.10034#S5.Thmtheorem2),[Figure3](https://arxiv.org/html/2607.10034#S5.F3)c\)\. Moreover, in these Transformer experiments, at matched fact count, the NTK baseline requires roughly1515\-63×63\\timesmore parameters than Transformer blocks using our data\-dependent construction\.

Finally, as a proof\-of\-concept, we show that fact\-storing MLPs enable*modular fact editing*\([Section5\.2](https://arxiv.org/html/2607.10034#S5.SS2)\) by replacing a Transformer’s MLP with one storing new facts\. Our method,MLP Swapping, achieves near\-perfect*fact\-editing score*—correctly editing target facts while avoiding off\-target effects—whereas prior state\-of\-the\-art methods degrade to as low as∼30\\sim 30% score when editing 10% of the fact\-set\.

In summary, our work takes a constructive step toward understanding MLPs in Transformers\. We present a fact\-storing MLP construction that achieves optimal margin and fact\-storage capacity, provides provable decoding\-margin guarantees under arbitrary embeddings, and is usable within Transformer blocks for factual recall at optimal capacity\. We also demonstrate an application to modular fact editing, illustrating a path toward robust and modular knowledge manipulation in LLMs\.

## 2Preliminaries

### 2\.1Formalizing Factual Knowledge

##### Fact sets and storage\.

Given key embeddings𝐊∈ℝ\|K\|×d\\mathbf\{K\}\\in\\mathbb\{R\}^\{\|K\|\\times d\}and value embeddings𝐕∈ℝ\|V\|×d\\mathbf\{V\}\\in\\mathbb\{R\}^\{\|V\|\\times d\}, afact setis a mapf:\[\|𝐊\|\]→\[\|𝐕\|\]f:\[\|\\mathbf\{K\}\|\]\\to\[\|\\mathbf\{V\}\|\]\. We write𝐤i\\mathbf\{k\}\_\{i\}and𝐯i\\mathbf\{v\}\_\{i\}for theiith key and value embedding, respectively\.

###### Definition 2\.1\(Fact storage\)\.

A model𝐠θ:ℝd→ℝd\\mathbf\{g\}\_\{\\theta\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\}stores a fact setf:\[\|𝐊\|\]→\[\|𝐕\|\]f:\[\|\\mathbf\{K\}\|\]\\to\[\|\\mathbf\{V\}\|\]given embeddings𝐊\\mathbf\{K\}and𝐕\\mathbf\{V\}if, for alli∈\[\|𝐊\|\]i\\in\[\|\\mathbf\{K\}\|\]and allj≠f​\(i\)∈\[\|𝐕\|\]j\\neq f\(i\)\\in\[\|\\mathbf\{V\}\|\],

⟨𝐠θ​\(𝐤i\),𝐯f​\(i\)⟩\>⟨𝐠θ​\(𝐤i\),𝐯j⟩\.\\langle\\mathbf\{g\}\_\{\\theta\}\(\\mathbf\{k\}\_\{i\}\),\\mathbf\{v\}\_\{f\(i\)\}\\rangle\>\\langle\\mathbf\{g\}\_\{\\theta\}\(\\mathbf\{k\}\_\{i\}\),\\mathbf\{v\}\_\{j\}\\rangle\.\(1\)

Notably, this definition is equivalent to correct softmax decoding in language modeling\.

###### Definition 2\.2\(Margin\)\.

The margin of𝐠θ\\mathbf\{g\}\_\{\\theta\}on factiiagainst competitorj≠f​\(i\)j\\neq f\(i\)is

γi,j:=⟨𝐠θ​\(𝐤i\),𝐯f​\(i\)⟩−⟨𝐠θ​\(𝐤i\),𝐯j⟩,\\gamma\_\{i,j\}:=\\langle\\mathbf\{g\}\_\{\\theta\}\(\\mathbf\{k\}\_\{i\}\),\\mathbf\{v\}\_\{f\(i\)\}\\rangle\-\\langle\\mathbf\{g\}\_\{\\theta\}\(\\mathbf\{k\}\_\{i\}\),\\mathbf\{v\}\_\{j\}\\rangle,\(2\)and the minimum margin isγmin:=mini,j≠f​\(i\)⁡γi,j\\gamma\_\{\\min\}:=\\min\_\{i,j\\neq f\(i\)\}\\gamma\_\{i,j\}\. Note that storing a fact set \(in the sense of[Section2\.1](https://arxiv.org/html/2607.10034#S2.SS1.SSS0.Px1)\) is equivalent toγmin\>0\\gamma\_\{\\min\}\>0\.

##### Fact\-storage cost and capacity\.

To measure parameter efficiency, we define the smallest parameter budget needed for a model class to store*every*fact set on fixed embeddings\.

###### Definition 2\.3\(Fact\-storage cost and capacity\)\.

The*fact\-storage cost*of a model class𝐠\\mathbf\{g\}on embeddings𝐊\\mathbf\{K\}and𝐕\\mathbf\{V\}is the minimum parameter count needed to represent*all*possible fact sets:

W​\(𝐠;𝐊,𝐕\)=min⁡\{\#​\(θ\)\|∀f:\[\|𝐊\|\]→\[\|𝐕\|\],∃θ​s\.t\.​𝐠θ​stores​f\}\.W\(\\mathbf\{g\};\\mathbf\{K\},\\mathbf\{V\}\)=\\min\\left\\\{\\\#\(\\theta\)\\Bigg\|\\;\\begin\{aligned\} &\\forall f:\[\|\\mathbf\{K\}\|\]\\to\[\|\\mathbf\{V\}\|\],\\\\ &\\exists\\,\\theta\\;\\text\{s\.t\.\}\\;\\mathbf\{g\}\_\{\\theta\}\\text\{ stores \}f\\end\{aligned\}\\right\\\}\.\(3\)The corresponding*fact\-storage capacity*is the maximum number of facts storable with a fixed parameter budget\.

###### Theorem 2\.4\(Information\-theoretic lower bound\)\.

Assuming a constant number of bits per parameter, the fact\-storage cost of embeddings𝐊\\mathbf\{K\}and𝐕\\mathbf\{V\}for*any*model class𝐠\\mathbf\{g\}satisfies

W​\(𝐠;𝐊,𝐕\)=Ω​\(\|𝐊\|​log⁡\[\|𝐕\|\]\)\.W\(\\mathbf\{g\};\\mathbf\{K\},\\mathbf\{V\}\)=\\Omega\(\|\\mathbf\{K\}\|\\log\[\|\\mathbf\{V\}\|\]\)\.

See Appendix[B\.1](https://arxiv.org/html/2607.10034#A2.SS1)for proof\.

### 2\.2Model Classes

In this work, we study two model classes: gated one\-hidden\-layer MLPs and Hebbian memories\(Kohonen,[1972](https://arxiv.org/html/2607.10034#bib.bib74); Hopfield,[1982](https://arxiv.org/html/2607.10034#bib.bib75); Bubecket al\.,[2020](https://arxiv.org/html/2607.10034#bib.bib43); Cabanneset al\.,[2024](https://arxiv.org/html/2607.10034#bib.bib76); Nichaniet al\.,[2024](https://arxiv.org/html/2607.10034#bib.bib40)\)\.

MLPs\.We consider models𝐠θ:ℝd→ℝdv\\mathbf\{g\}\_\{\\theta\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\_\{v\}\}of the form

MLP​\(𝐱\)=𝐠θ​\(𝐱\)=𝐁​\(\(𝐀𝐱\)⊙σ​\(𝐆𝐱\)\),\\text\{MLP\}\(\\mathbf\{x\}\)=\\mathbf\{g\}\_\{\\theta\}\(\\mathbf\{x\}\)=\\mathbf\{B\}\\\!\\left\(\(\\mathbf\{A\}\\mathbf\{x\}\)\\odot\\sigma\(\\mathbf\{G\}\\mathbf\{x\}\)\\right\),\(4\)where𝐱∈ℝd\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\},𝐀,𝐆∈ℝm×d\\mathbf\{A\},\\mathbf\{G\}\\in\\mathbb\{R\}^\{m\\times d\}, and𝐁∈ℝdv×m\\mathbf\{B\}\\in\\mathbb\{R\}^\{d\_\{v\}\\times m\}\. This family includes SwiGLU\-style MLPs\(Shazeer,[2020](https://arxiv.org/html/2607.10034#bib.bib77)\)used in modern language models\(Yanget al\.,[2025](https://arxiv.org/html/2607.10034#bib.bib51); DeepSeek\-AIet al\.,[2025](https://arxiv.org/html/2607.10034#bib.bib78); Dubeyet al\.,[2024](https://arxiv.org/html/2607.10034#bib.bib1)\)\. Our explicit construction \([Section4\.1](https://arxiv.org/html/2607.10034#S4.SS1)\) usesσ=id\\sigma=\\mathrm\{id\}\.

Hebbian memories\.These linear models are maps𝐠𝐖:ℝd→ℝdv\\mathbf\{g\}\_\{\\mathbf\{W\}\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\_\{v\}\}of the form𝐠𝐖​\(𝐱\)=𝐖𝐱\\mathbf\{g\}\_\{\\mathbf\{W\}\}\(\\mathbf\{x\}\)=\\mathbf\{W\}\\mathbf\{x\}, where𝐱∈ℝd\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\}and𝐖∈ℝdv×d\\mathbf\{W\}\\in\\mathbb\{R\}^\{d\_\{v\}\\times d\}\. A fact set is stored by taking

𝐖=∑j=1\|𝐊\|𝐯f​\(j\)​𝐤j⊤\.\\mathbf\{W\}=\\sum\_\{j=1\}^\{\|\\mathbf\{K\}\|\}\\mathbf\{v\}\_\{f\(j\)\}\\mathbf\{k\}\_\{j\}^\{\\top\}\.\(5\)Section[3](https://arxiv.org/html/2607.10034#S3)shows that MLPs can be recast as Hebbian memories in a kernel feature space\.

### 2\.3Related Work

The two closest works to ours areNichaniet al\.\([2024](https://arxiv.org/html/2607.10034#bib.bib40)\)andZhonget al\.\([2025](https://arxiv.org/html/2607.10034#bib.bib79)\)\.Nichaniet al\.\([2024](https://arxiv.org/html/2607.10034#bib.bib40)\)gave the first explicit construction of fact\-storing MLPs and showed near\-optimal fact\-storage capacity, up to a polylogarithmic factor, but their analysis is restricted to isotropic embeddings and studies only the separability conditionγmin\>0\\gamma\_\{\\min\}\>0\.Zhonget al\.\([2025](https://arxiv.org/html/2607.10034#bib.bib79)\)developed a unified associative\-memory view of attention and MLPs , but focused on average retrieval fidelity rather than worst\-case margins\. In contrast, we derive explicit margin bounds beyond isotropy, identify margin bounded away from zero as the condition for Transformer usability, and show that constructed MLPs can be integrated into Transformer blocks for factual recall\.

Additional discussion of probing, editing, and scaling studies of factual knowledge in language models appears in Appendix[C](https://arxiv.org/html/2607.10034#A3)\.

## 3MLPs, Hebbians, and Margins

We first establish two observations that let us analyze MLP fact storage within Transformers\. First, MLPs are equivalent to*Hebbian kernel memories*after whitening the empirical feature covariance\. Second, MLPs in Transformers need*margins bounded away from zero*\(not just positive separability\) because attention layers pass noisy queries to MLPs\.

### 3\.1MLPs Are Hebbian Kernel Memories

Our first observation is that any MLP can be rewritten as a*Hebbian kernel memory*on its stored examples\. The only gap between the plain Hebbian predictor and the original MLP is the empirical feature covariance𝚺^\\hat\{\{\\bm\{\\Sigma\}\}\}, so whitening converts any MLP into a \(kernel\) Hebbian\.

Key Result: MLPs are Hebbian Memories###### Theorem 3\.1\(MLPs as kernel Hebbians, informal\)\. For stored examples\(𝐱i,𝐲i\)\(\\mathbf\{x\}\_\{i\},\\mathbf\{y\}\_\{i\}\)with𝐲i=MLP​\(𝐱i\)\\mathbf\{y\}\_\{i\}=\\mathrm\{MLP\}\(\\mathbf\{x\}\_\{i\}\)andMLP​\(𝐱\)=𝐁​ϕ​\(𝐱\)\\mathrm\{MLP\}\(\\mathbf\{x\}\)=\\mathbf\{B\}\\phi\(\\mathbf\{x\}\)\(where for gated MLPsϕ​\(𝐱\)=\(𝐀𝐱\)⊙σ​\(𝐆𝐱\)\\phi\(\\mathbf\{x\}\)=\(\\mathbf\{A\}\\mathbf\{x\}\)\\odot\\sigma\(\\mathbf\{G\}\\mathbf\{x\}\)\), define the empirical feature covariance𝚺^:=1F​∑i=1Fϕ​\(𝐱i\)​ϕ​\(𝐱i\)⊤\.\\hat\{\{\\bm\{\\Sigma\}\}\}:=\\frac\{1\}\{F\}\\sum\_\{i=1\}^\{F\}\\phi\(\\mathbf\{x\}\_\{i\}\)\\phi\(\\mathbf\{x\}\_\{i\}\)^\{\\top\}\.Assuming𝚺^\\hat\{\{\\bm\{\\Sigma\}\}\}is invertible, define the whitened Hebbian memoryHwhite​\(𝐳\):=1F​∑i=1F𝐲i​K​\(𝐱i,𝐳\)H\_\{\\mathrm\{white\}\}\(\\mathbf\{z\}\):=\\frac\{1\}\{F\}\\sum\_\{i=1\}^\{F\}\\mathbf\{y\}\_\{i\}\\,K\(\\mathbf\{x\}\_\{i\},\\mathbf\{z\}\)induced by the kernelK​\(𝐱,𝐳\):=ϕ​\(𝐱\)⊤​𝚺^−1​ϕ​\(𝐳\)\.K\(\\mathbf\{x\},\\mathbf\{z\}\):=\\phi\(\\mathbf\{x\}\)^\{\\top\}\\hat\{\{\\bm\{\\Sigma\}\}\}^\{\-1\}\\phi\(\\mathbf\{z\}\)\.ThenHwhite​\(𝐳\)=MLP​\(𝐳\)for all​𝐳\.H\_\{\\mathrm\{white\}\}\(\\mathbf\{z\}\)=\\mathrm\{MLP\}\(\\mathbf\{z\}\)\\qquad\\text\{for all \}\\mathbf\{z\}\.Thus, after feature whitening, the MLP is exactly a Hebbian kernel memory\.See Appendix[B\.2\.1](https://arxiv.org/html/2607.10034#A2.SS2.SSS1)for formal statement and proof\.

This reduction motivates the rest of the paper: constructing an MLP with desirable margin and storage capacity scaling amounts to designing an effective kernel for Hebbian memories\.

### 3\.2Margins Govern Transformer Usability

Our second observation is that positive margin at stored keys is not enough for a Transformer block to use a fact\-storing MLP reliably\. Intuitively, because the attention mechanism perturbs the query before it reaches the MLP, end\-to\-end usability requires the decoding margin to be bounded away from zero\.

###### Definition 3\.2\(Synthetic Sequential Factual Recall \(SSFR\)\)\.

Fix a junk\-token vocabulary of sizeVJV\_\{J\}, a model dimensiondd, and a set ofFFkey\-value pairs\{\(𝐤i,𝐯f​\(i\)\)\}i=1F\\\{\(\\mathbf\{k\}\_\{i\},\\mathbf\{v\}\_\{f\(i\)\}\)\\\}\_\{i=1\}^\{F\}wheref:\[F\]→\[F\]f:\[F\]\\to\[F\]is a fact set\. Then*SSFR*inputs have the form:

j1,…,jJ/2⏟junk prefix,ki⏟key,jJ/2\+1,…,jJ⏟junk suffix,q⏟query→vf​\(i\)⏟value,\\underbrace\{j\_\{1\},\\ldots,j\_\{J/2\}\}\_\{\\text\{junk prefix\}\},\\;\\underbrace\{k\_\{i\}\}\_\{\\text\{key\}\},\\;\\underbrace\{j\_\{J/2\+1\},\\ldots,j\_\{J\}\}\_\{\\text\{junk suffix\}\},\\;\\underbrace\{q\}\_\{\\text\{query\}\}\\rightarrow\\underbrace\{v\_\{f\(i\)\}\}\_\{\\text\{value\}\},wherejt​∼i\.i\.d\.​Unif​\(\[VJ\]\)j\_\{t\}\\overset\{\\mathrm\{i\.i\.d\.\}\}\{\\sim\}\\mathrm\{Unif\}\(\[V\_\{J\}\]\)andqqis a fixed query token\. The junk lengthJJcontrols how difficult it is for attention to isolate the relevant key\.

##### Empirical verification\.

We pretrain an attention\-only Transformer block, freeze it, insert a frozen GD\-trained fact\-storing MLP, and sweep hidden width \(see Appendix[A\.2\.1](https://arxiv.org/html/2607.10034#A1.SS2.SSS1)for details\)\. Crucially, although the MLP stores the fact\-set as soon asγmin\\gamma\_\{\\min\}becomes positive, end\-to\-end SSFR accuracy lags until the margin is bounded away from zero \(Figure[2](https://arxiv.org/html/2607.10034#S4.F2)a\)\. This observation motivates our theoretical study of Hebbian margin bounds in[Section4](https://arxiv.org/html/2607.10034#S4)\.

## 4Margin and Storage Capacity Analysis of Hebbian MLPs

In[Section3](https://arxiv.org/html/2607.10034#S3), we identified decoding margin as a property of interest in MLPs\. In this section we turn to study the margin and fact storage capacity scaling of MLPs\. We begin by proposing a simple bilinear MLP construction, which we term the*Hebbian MLP*, allowing us to characterize how decoding margin and fact storage capacity scales with the number of facts and MLP parameters\. Equipped with our simple construction, we demonstrate that MLPs realize optimal margin scaling \([Theorem4\.3](https://arxiv.org/html/2607.10034#S4.Thmtheorem3)\) and that their fact storage capacity scales at the information\-theoretically optimal rate \([Section4\.3](https://arxiv.org/html/2607.10034#S4.SS3)\)\. Finally, we develop kernel\-whitened and data\-dependent variants of our MLP construction that realize optimal capacity scaling empirically; at matched fact count, the NTK baseline requires roughly1010\-104×104\\timesmore parameters than our data\-dependent construction\.

##### Decoding margin\.

We study the MLP decoding margin when viewed as a Hebbian kernel memory with kernelKK\([Theorem3\.1](https://arxiv.org/html/2607.10034#S3.Thmtheorem1)\)\. Intuitively, the decoding margin is the “slack” a model is allowed in its outputs so that it still decodes to the right values \(see[Figure1](https://arxiv.org/html/2607.10034#S1.F1)a\)\. Our margin analysis follows from decomposing the margin into*signal*and*cross\-talk*terms \(Appendix[B\.8\.1](https://arxiv.org/html/2607.10034#A2.SS8.SSS1)\):

γi,j=⟨𝐯f​\(i\)−𝐯j,𝐯f​\(i\)⟩​K​\(𝐤i,𝐤i\)⏟signal\+∑t≠i⟨𝐯f​\(i\)−𝐯j,𝐯f​\(t\)⟩​K​\(𝐤t,𝐤i\)⏟cross\-talk\.\\gamma\_\{i,j\}=\\underbrace\{\\langle\\mathbf\{v\}\_\{f\(i\)\}\-\\mathbf\{v\}\_\{j\},\\;\\mathbf\{v\}\_\{f\(i\)\}\\rangle\\,K\(\\mathbf\{k\}\_\{i\},\\mathbf\{k\}\_\{i\}\)\}\_\{\\text\{signal\}\}\+\\underbrace\{\\sum\_\{t\\neq i\}\\langle\\mathbf\{v\}\_\{f\(i\)\}\-\\mathbf\{v\}\_\{j\},\\;\\mathbf\{v\}\_\{f\(t\)\}\\rangle\\,K\(\\mathbf\{k\}\_\{t\},\\mathbf\{k\}\_\{i\}\)\}\_\{\\text\{cross\-talk\}\}\.\(6\)Intuitively, we wish to develop a kernelKKthat 1\) maximizes the signal\-to\-cross talk ratio while 2\) remaining implementable as a gated MLP\.

### 4\.1A Bilinear Hebbian MLP Construction

Our first step toward understanding the decoding margin and storage\-capacity scaling of MLPs is to develop a simple closed\-form gated MLP construction capable of storing a fact set\. Our*Hebbian MLP*construction is defined as

MLP​\(𝐱\)=𝐁​\(𝐀𝐱⊙𝐆𝐱\)\\mathrm\{MLP\}\(\\mathbf\{x\}\)=\\mathbf\{B\}\\bigl\(\\mathbf\{A\}\\mathbf\{x\}\\odot\\mathbf\{G\}\\mathbf\{x\}\\bigr\)\(7\)with𝐀i,j∼N​\(0,1m\)\\mathbf\{A\}\_\{i,j\}\\sim N\(0,\\frac\{1\}\{m\}\),𝐆i,j∼N​\(0,1m\)\\mathbf\{G\}\_\{i,j\}\\sim N\(0,\\frac\{1\}\{m\}\), and𝐁=𝐕T​𝚽\\mathbf\{B\}=\\mathbf\{V\}^\{T\}\{\\bm\{\\Phi\}\}, where𝚽i=𝐀𝐤i⊙𝐆𝐤i\{\\bm\{\\Phi\}\}\_\{i\}=\\mathbf\{A\}\\mathbf\{k\}\_\{i\}\\odot\\mathbf\{G\}\\mathbf\{k\}\_\{i\}\. The key component of this construction is that it is equivalent to a Hebbian kernel memory \([SectionB\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2)\), using anmm\-dimensional feature map to sketch the exact quadratic kernelK2​\(𝐱,𝐳\)=⟨𝐱,𝐳⟩2K\_\{2\}\(\\mathbf\{x\},\\mathbf\{z\}\)=\\langle\\mathbf\{x\},\\mathbf\{z\}\\rangle^\{2\}, which we can analyze:

H​\(𝐳\)=∑i=1F𝐯i​K^2​\(𝐤i,𝐳\),K^2​\(𝐤,𝐳\)=∑r=1m\(𝐀r⊤​𝐤\)​\(𝐀r⊤​𝐳\)​\(𝐆r⊤​𝐤\)​\(𝐆r⊤​𝐳\)\.H\(\\mathbf\{z\}\)=\\sum\_\{i=1\}^\{F\}\\mathbf\{v\}\_\{i\}\\hat\{K\}\_\{2\}\(\\mathbf\{k\}\_\{i\},\\mathbf\{z\}\),\\quad\\hat\{K\}\_\{2\}\(\\mathbf\{k\},\\mathbf\{z\}\)=\\sum\_\{r=1\}^\{m\}\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{k\}\)\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{z\}\)\\,\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{k\}\)\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{z\}\)\.\(8\)We provide a pseudocode implementation of our construction in[Algorithm1](https://arxiv.org/html/2607.10034#alg1)\.

### 4\.2Margin Scaling

#### 4\.2\.1Isotropic Embeddings

We first show that in the isotropic keys and values setting, our bilinear MLP construction’s margin scales at an asymptotically optimal rate:

Key Result: MLP margin scales at optimal rate###### Theorem 4\.1\(MLP Margin Scaling \(Isotropic Embeddings Setting\) \- Informal\)\. Under isotropic key and value embeddings, the decoding margin of our bilinear MLP construction \([Equation7](https://arxiv.org/html/2607.10034#S4.E7)\) scales as:γmin≥1⏟signal−C​F​log⁡\(F\)m​d⏟cross\-talk\.\\gamma\_\{\\min\}\\ \\geq\\ \\underbrace\{1\}\_\{\\text\{signal\}\}\\;\-\\;\\underbrace\{C\\sqrt\{\\frac\{F\\log\(F\)\}\{md\}\}\}\_\{\\text\{cross\-talk\}\}\.\(9\)See[SectionB\.8\.4](https://arxiv.org/html/2607.10034#A2.SS8.SSS4.Px3)for formal statement and proof\.

##### Empirical verification\.

We evaluate the empirical margin scaling against the theoretical scaling as we sweep factsFFand MLP hidden\-dimensionmmunder isotropic embeddings\. Figure[2](https://arxiv.org/html/2607.10034#S4.F2)b validates that our margin bounds closely match the empirical minimum margins \(R2≥0\.97R^\{2\}\\geq 0\.97\)\.

![Refer to caption](https://arxiv.org/html/2607.10034v1/x2.png)Figure 2:Decoding margins govern Transformer usability and yield provable capacity bounds\.\(A\)In a Transformer, inserted fact\-storing MLPs become usable only once their decoding margin is bounded away from zero\.\(B\)Under isotropic keys and values, the empirical margin follows our predicted scaling as the MLP hidden dimensionmmincreases\.\(C\)Our construction achieves asymptotically optimal fact\-storage capacity scaling under isotropic keys and values\.

#### 4\.2\.2Beyond isotropic embeddings

Our margin decomposition \([Equation6](https://arxiv.org/html/2607.10034#S4.E6)\) analysis extends beyond isotropic key/value embeddings to arbitrary embedding geometries, like those found in LLMs\(Ethayarajh,[2019](https://arxiv.org/html/2607.10034#bib.bib81); Razzhigaevet al\.,[2024](https://arxiv.org/html/2607.10034#bib.bib82)\)\. To this end, we present the most general margin bound for arbitrary embedding geometries, illustrating how embedding\-geometric statistics enter the margin scaling\. Furthermore, we provide a full ladder of bounds across the key/value geometry regimes \([Table2](https://arxiv.org/html/2607.10034#A2.T2)\)\.

###### Theorem 4\.3\(MLP Margin Scaling \(Arbitrary Embeddings Setting\) – Informal\)\.

Under*arbitrary*key and value embeddings, and in the regimed≳log⁡Fd\\gtrsim\\log F, the decoding margin of our bilinear MLP construction \([Equation7](https://arxiv.org/html/2607.10034#S4.E7)\) scales as:

γmin≥C​Ssig⏟signal−F​log⁡\(F\)m​d​Pkey​Pval​Palign⏟cross\-talk\.\\gamma\_\{\\min\}\\ \\geq\\ \\underbrace\{C\\ S\_\{\\mathrm\{sig\}\}\}\_\{\\text\{signal\}\}\\;\-\\;\\underbrace\{\\sqrt\{\\frac\{F\\log\(F\)\}\{md\}\}\\ P\_\{\\mathrm\{key\}\}\\ P\_\{\\mathrm\{val\}\}\\ P\_\{\\mathrm\{align\}\}\}\_\{\\text\{cross\-talk\}\}\.\(10\)

See[SectionB\.8\.1](https://arxiv.org/html/2607.10034#A2.SS8.SSS1)for formal statement and proof, including the formal definitions of the four embedding\-geometric statisticsPkeyP\_\{\\mathrm\{key\}\},PvalP\_\{\\mathrm\{val\}\},PalignP\_\{\\mathrm\{align\}\}, andSsigS\_\{\\mathrm\{sig\}\}\.

For a fact set mapping𝐤i→𝐯i\{\\mathbf\{k\}\}\_\{i\}\\to\{\\mathbf\{v\}\}\_\{i\}, define the kernel vectors𝑲i∈ℝF−1\\bm\{K\}\_\{i\}\\in\\mathbb\{R\}^\{F\-1\}and the value interference vectors𝑽i,j∈ℝF−1\\bm\{V\}\_\{i,j\}\\in\\mathbb\{R\}^\{F\-1\}as:

\(𝑲i\)t:=K^​\(𝐤i,𝐤t\),\(𝑽i,j\)t:=⟨𝐯i−𝐯j,𝐯t⟩,t≠i\.\(\\bm\{K\}\_\{i\}\)\_\{t\}\\vcentcolon=\\hat\{K\}\(\{\\mathbf\{k\}\}\_\{i\},\{\\mathbf\{k\}\}\_\{t\}\),\\qquad\(\\bm\{V\}\_\{i,j\}\)\_\{t\}\\vcentcolon=\\langle\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\},\{\\mathbf\{v\}\}\_\{t\}\\rangle,\\qquad t\\neq i\.From our signal and cross talk decomposition \([Equation6](https://arxiv.org/html/2607.10034#S4.E6)\),𝑲i\\bm\{K\}\_\{i\}captures which other facts are activated when retrieving factii, while𝑽i,j\\bm\{V\}\_\{i,j\}measures which values interfere with distinguishing factiifrom competitorjj\. Using these vectors as building blocks, four embedding\-geometric statistics enter the isotropic margin bound multiplicatively when generalizing it to arbitrary embedding geometries, with each statistic isolating a distinct source of signal or cross\-talk:

- •Key crowding penalty\. Pkey:=EKF/m\(EK:=maxi∈\[F\]⁡‖𝑲i‖22\)\.P\_\{\\mathrm\{key\}\}\\vcentcolon=\\frac\{\\sqrt\{E\_\{K\}\}\}\{\\sqrt\{F/m\}\}\\qquad\\left\(E\_\{K\}\\vcentcolon=\\max\_\{i\\in\[F\]\}\\\|\\bm\{K\}\_\{i\}\\\|\_\{2\}^\{2\}\\right\)\.This measures how much the stored keys overlap under the bilinear featurization, relative to the random/isotropic baseline scaleF/m\\sqrt\{F/m\}\. SmallerPkeyP\_\{\\mathrm\{key\}\}\(and thus smaller cross\-talk\) means the key features are more separated, so fewer irrelevant facts are activated by a query\.
- •Value crowding penalty\. Pval:=EvF/d\(Ev:=maxj≠i∈\[F\]⁡‖𝑽i,j‖22\)\.P\_\{\\mathrm\{val\}\}\\vcentcolon=\\frac\{\\sqrt\{E\_\{v\}\}\}\{\\sqrt\{F/d\}\}\\qquad\\left\(E\_\{v\}\\vcentcolon=\\max\_\{j\\neq i\\in\[F\]\}\\\|\\bm\{V\}\_\{i,j\}\\\|\_\{2\}^\{2\}\\right\)\.This measures how much the stored value directions overlap, relative to the random/isotropic baseline scaleF/d\\sqrt\{F/d\}\. SmallerPvalP\_\{\\mathrm\{val\}\}\(and thus smaller cross\-talk\) means the incorrect values are less aligned with the correct value margin direction\.
- •Key–value alignment penalty\. Palign:=κlog⁡\(F\)/F\(κ:=maxj≠i∈\[F\]⁡\|cos⁡∠​\(𝑲i,𝑽i,j\)\|\)\.P\_\{\\mathrm\{align\}\}\\vcentcolon=\\frac\{\\kappa\}\{\\sqrt\{\\log\(F\)/F\}\}\\qquad\\left\(\\kappa\\vcentcolon=\\max\_\{j\\neq i\\in\[F\]\}\\left\|\\cos\\angle\\\!\\left\(\\bm\{K\}\_\{i\},\\bm\{V\}\_\{i,j\}\\right\)\\right\|\\right\)\.This measures alignment between which facts a query activates \(kernel column\) and which values are most confusable \(value interference\), relative to the isotropic baselinelog⁡\(F\)/F\\sqrt\{\\log\(F\)/F\}\. SmallerPalignP\_\{\\mathrm\{align\}\}\(and thus smaller cross\-talk\) means key and value errors are orthogonal rather than compounding each other\.
- •Signal strength\. Ssig:=Kmindiag​Vmin\(1−log⁡\(F\)/d\)\(Kmindiag:=mini∈\[F\]⁡K^​\(𝐤i,𝐤i\),Vmin:=mini≠j⁡⟨𝐯i−𝐯j,𝐯i⟩\)\.S\_\{\\mathrm\{sig\}\}\\vcentcolon=\\frac\{K\_\{\\min\}^\{\\mathrm\{diag\}\}V\_\{\\min\}\}\{\(1\-\\sqrt\{\\log\(F\)/d\}\)\}\\qquad\\left\(K\_\{\\min\}^\{\\mathrm\{diag\}\}\\vcentcolon=\\min\_\{i\\in\[F\]\}\\hat\{K\}\(\{\\mathbf\{k\}\}\_\{i\},\{\\mathbf\{k\}\}\_\{i\}\),\\quad V\_\{\\min\}\\vcentcolon=\\min\_\{i\\neq j\}\\langle\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\},\{\\mathbf\{v\}\}\_\{i\}\\rangle\\right\)\.This measures the strength of the signal relative to the1−log⁡\(F\)/d1\-\\log\(F\)/dbaseline\. LargerSsigS\_\{\\mathrm\{sig\}\}\(and thus stronger decoding signal\) means featurized keys have larger norm while value embeddings remain nearly\-orthogonal\.

##### Empirical verification\.

We evaluate our general margin bound \(and those from[Table2](https://arxiv.org/html/2607.10034#A2.T2)\) in[Figure5](https://arxiv.org/html/2607.10034#A1.F5), finding they closely track the empirically observed margin as we vary the key and value crowding \(R2≥0\.95R^\{2\}\\geq 0\.95\)\.

### 4\.3Fact\-Storage Capacity

Equipped with our margin scaling bounds \([Equations9](https://arxiv.org/html/2607.10034#S4.E9)and[10](https://arxiv.org/html/2607.10034#S4.E10)\), we develop storage capacity scaling laws for our bilinear gated MLPs by simply solving for the parameters necessary to make their margin positive\. We find that the storage capacity of our simple MLP construction scales at the information\-theoretically optimal rate, under the isotropic embeddings setting, while for non\-isotropic embeddings it does so up to penalization factors\.

Key Result: MLPs store facts at an info\-theoretically optimal rate\.###### Corollary 4\.4\(MLP Fact\-storage Capacity \(Isotropic Embeddings Setting\) \- Informal\)\. For isotropic key and value embeddings, our bilinear MLP construction storesFFfacts usingW=Θ​\(m​d\)=Θ​\(F​log⁡\(F\)\)W=\\Theta\(md\)=\\Theta\\\!\\left\(F\\log\(F\)\\right\)parameters\. Thus MLPs achieveinformation\-theoretically optimalfact\-storage capacity\.See[SectionB\.8\.4](https://arxiv.org/html/2607.10034#A2.SS8.SSS4.Px3)for formal statement and proof\.

###### Corollary 4\.5\(Optimal Fact\-storage Capacity \(Arbitrary Embeddings Setting\) \- Informal\)\.

For arbitrary key and value embeddings, our bilinear MLP construction storesFFfacts using

W=Θ​\(m​d\)=Θ​\(F​log⁡\(F\)​\(Pkey​Pval​PalignSsig\)2\)W=\\Theta\(md\)=\\Theta\\\!\\left\(F\\log\(F\)\\left\(\\frac\{P\_\{\\mathrm\{key\}\}P\_\{\\mathrm\{val\}\}P\_\{\\mathrm\{align\}\}\}\{S\_\{\\mathrm\{sig\}\}\}\\right\)^\{2\}\\right\)parameters\. Thus MLPs achieve*information\-theoretically optimal*fact\-storage capacity up to penalization factors\.

See[SectionB\.8\.1](https://arxiv.org/html/2607.10034#A2.SS8.SSS1)for formal statement and proof\.

[Section4\.3](https://arxiv.org/html/2607.10034#S4.SS3)counts real\-valued parameters; Appendix[B\.2\.6](https://arxiv.org/html/2607.10034#A2.SS2.SSS6)shows that, under bounded precision, our construction incurs only an extra logarithmic factor in total bit complexity\.

##### Empirical verification\.

Figure[2](https://arxiv.org/html/2607.10034#S4.F2)c compares the storage scaling of our construction to gradient descent\-trained \(GD\) MLPs and the NTK construction fromNichaniet al\.\([2024](https://arxiv.org/html/2607.10034#bib.bib40)\)\. Notably, we include two more closed\-form variants of our construction in our empirical analysis:

- •Kernel\-whitened construction\.Our kernel\-whitened MLP construction whitens the sketched quadratic kernelK^\\hat\{K\}in our vanilla MLP construction with its empirical covariance\. This construction is motivated by the observations that 1\) the key crowding penaltyPkeyP\_\{\\mathrm\{key\}\}governs the cross\-talk magnitude scale in our signal and cross talk decomposition \([Equation10](https://arxiv.org/html/2607.10034#S4.E10)\) and 2\) whitening the sketched quadratic kernelK^\\hat\{K\}with its empirical covariance \(Appendix[B\.2\.4](https://arxiv.org/html/2607.10034#A2.SS2.SSS4.Px1)\) reducesPkeyP\_\{\\mathrm\{key\}\}\([SectionB\.2\.4](https://arxiv.org/html/2607.10034#A2.SS2.SSS4.Px1)\)\. To our knowledge, our*kernel\-whitened*construction is the first closed\-form MLP construction to empirically achieve the optimal fact\-storage capacity scalingW=Θ​\(F​log⁡\(Fδ\)\)W\\,=\\Theta\\Big\(F\\log\\\!\\Big\(\\frac\{F\}\{\\delta\}\\Big\)\\Big\)\.
- •Data\-dependent construction\.Our data\-dependent MLP closed\-form construction solves for each weight matrix in our vanilla MLP construction via a least\-squares objective, as opposed to initializing them randomly\. Intuitively, the least squares objective we solve for each matrix leverages the key and value geometry to maximize the construction’s margin \(Appendix[B\.2\.5](https://arxiv.org/html/2607.10034#A2.SS2.SSS5)\)\. This approach improves storage capacity without gradient descent: at matched fact count, the NTK baseline requires roughly1010\-104×104\\timesmore parameters than our data\-dependent construction, while our data\-dependent construction only requires about66\-10×10\\timesmore parameters than GD\.

## 5Integrating Fact\-Storing MLPs into Transformers

In[Section4](https://arxiv.org/html/2607.10034#S4), we characterized the margin scaling of*Hebbian MLPs*\. In this section we leverage our margin scaling bounds to study the storage capacity scaling of Transformer blocks, where MLP inputs are no longer exact keys but instead are noisy attention outputs\. Crucially, we demonstrate that the fact storage capacity of Transformer blocks scales at the information\-theoretically optimal rate, provided the attention noise remains bounded\. Further, we empirically show that our construction remains usable within Transformer blocks for factual recall tasks; at matched fact count, the NTK baseline requires roughly1515\-63×63\\timesmore parameters than Transformer blocks using our data\-dependent construction\. Finally, we show that fact\-storing MLPs unlock a new capability: modular, zero\-shot fact editing within a Transformer by*swapping out its MLP*\.

![Refer to caption](https://arxiv.org/html/2607.10034v1/x3.png)Figure 3:Transformer blocks achieve information\-theoretic optimal fact\-storage capacity and enable modular fact editing\.\(A\)Attention noise ceilingεattn\\varepsilon\_\{\\mathrm\{attn\}\}scales with junk context lengthJJin SSFR\.\(B\)Under bounded attention noise, Transformer blocks with our MLP construction achieve information\-theoretic optimal fact\-storage capacity scaling\.\(C\)MLP Swappingachieves near\-perfect fact\-editing score \(\>0\.99\>\\\!0\.99\) at up to10%10\\%edited facts, more than 40% better than existing fact\-editing baselines\.### 5\.1MLPs in Transformers Achieve Optimal Fact\-Storage Capacity

We start by investigating why MLPs need positive margin in Transformers\. Unlike in the standalone setting, attention does not query the MLP with the exact stored key\. We quantify the worst\-case deviation formally with an*attention noise ceiling*, which we define as the maximum noise an attention layer can produce when querying an MLP for a fact:

###### Definition 5\.1\(Attention noise ceiling – informal\)\.

LetQi⊂ℝdQ\_\{i\}\\subset\\mathbb\{R\}^\{d\}be the set of all possible queries an attention layer can produce when querying the MLP for the fact corresponding to the key𝐤i\\mathbf\{k\}\_\{i\}\. The*attention noise ceiling*is defined as

εattn:=maxi∈\[F\]⁡max𝐪∈Qi⁡‖𝐪−𝐤i‖2\.\\varepsilon\_\{\\mathrm\{attn\}\}:=\\max\_\{i\\in\[F\]\}\\;\\max\_\{\\mathbf\{q\}\\in Q\_\{i\}\}\\\|\\mathbf\{q\}\-\\mathbf\{k\}\_\{i\}\\\|\_\{2\}\.

Intuitively, we find that the attention noise ceiling increases with the number of distractor tokens in the sequence being processed by a Transformer block \(Figure[3](https://arxiv.org/html/2607.10034#S5.F3)\)\. We next show that Transformer usability reduces to whether the MLP margin can handle perturbations at this scale\.

##### MLPs achieve optimal fact\-storage capacity in Transformers\.

We now present our main result\. Equipped with our margin scaling analysis in MLPs, we demonstrate that Transformer blocks can store facts at an information\-theoretic optimal rate, provided bounded attention noise ceiling:

Key Result: Transformer blocks can store facts at an info\-theory optimal rate\.###### Theorem 5\.2\(Transformer Block Fact\-storage Capacity \(Isotropic Embeddings\) \- Informal\)\. A Transformer block equipped with a fact\-storing bilinear MLP, with non\-trivial marginγmin\>c0\>0\\gamma\_\{\\min\}\>c\_\{0\}\>0, for constantc0c\_\{0\}, can storeFFfacts usingW=Θ​\(m​d\)=Θ​\(F​log⁡\(F\)\)W=\\Theta\(md\)=\\Theta\\\!\\left\(F\\log\(F\)\\right\)MLP parameters, provided the attention layer in the block satisfies the attention noise ceilingεattn≲c0Lbil​\(F​log⁡\(F\)d\)\.\\varepsilon\_\{\\mathrm\{attn\}\}\\lesssim\\frac\{c\_\{0\}\}\{L\_\{\\mathrm\{bil\}\}\\\!\\left\(\\sqrt\{\\frac\{F\\log\(F\)\}\{d\}\}\\right\)\}\.\(11\)whereLbilL\_\{\\mathrm\{bil\}\}is the Lipschitz constant of the MLP\.See Appendix[B\.10](https://arxiv.org/html/2607.10034#A2.SS10)for formal statement and proof\.

[SectionB\.10\.1](https://arxiv.org/html/2607.10034#A2.SS10.SSS1.Px8)presents the formal statement and proof\. We note that this result can be easily extended to arbitrary embedding geometries, incurring the same penalization factors from[Equation10](https://arxiv.org/html/2607.10034#S4.E10)\.

##### Empirical verification\.

We produce GD, NTK, and our constructed MLPs, freeze their parameters, then insert them into a 1\-layer Transformer and train on the SSFR task\. Figure[3](https://arxiv.org/html/2607.10034#S5.F3)b validates the predicted asymptotically optimal capacity scaling for Transformer blocks using our construction\. Appendix[B\.10](https://arxiv.org/html/2607.10034#A2.SS10)reports a complementary per\-key margin diagnostic; in particular,[Figure7](https://arxiv.org/html/2607.10034#A1.F7)shows that the usable\-key fraction inferred from per\-key margins closely tracks end\-to\-end Transformer accuracy\. Among the constructions we evaluate, our data\-dependent construction only requires at most3×3\\timesmore parameters than GD MLPs at matched fact count\. Relative to the NTK baseline, our data\-dependent construction requires1515\-63×63\\timesfewer parameters at matched fact count\. See Appendix[B\.10](https://arxiv.org/html/2607.10034#A2.SS10)for further diagnostics\.

### 5\.2Fact Editing viaMLP Swapping

Having demonstrated that fact\-storing MLPs are usable inside Transformers, we now use GD\-trained fact\-storing MLPs to show a simple proof\-of\-concept method for*zero\-shot fact editing*\. We call this procedureMLP Swapping: to edit the model’s facts, we construct a new MLP storing the revised fact set and swap it into the Transformer, with*no further tuning of the Transformer’s parameters*\.

We evaluate on a synthetic author\-book language\-modeling task \(Appendix[A\.4\.5](https://arxiv.org/html/2607.10034#A1.SS4.SSS5)\) using a one\-layer Transformer trained to store book–author facts through a frozen fact\-storing MLP\. After training, we edit a subset of stored facts and evaluate two metrics\. The first is the standard fact\-editing score\(Menget al\.,[2023c](https://arxiv.org/html/2607.10034#bib.bib24)\), which jointly captures edit efficacy \(edited facts predict the new values\), specificity \(unedited facts stay correct\), and paraphrase generalization \(edits transfer to paraphrased prompts\)\. The second is the non\-fact PPL ratio, measuring post\-edit versus pre\-edit perplexity on non\-fact tokens\.MLP Swappingachieves near\-perfect fact\-editing score across the edit fractions we test\. At10%10\\%edited facts,MLP Swappingachieves score0\.9990\.999—a44\.944\.9\-percentage\-point gain over the strongest baseline, AlphaEdit\(Fanget al\.,[2025](https://arxiv.org/html/2607.10034#bib.bib55)\)\(score0\.5500\.550\)—while achieving a non\-fact PPL ratio of only1\.021\.02\(Figure[3](https://arxiv.org/html/2607.10034#S5.F3)c\)\.

Appendix[A\.4\.5](https://arxiv.org/html/2607.10034#A1.SS4.SSS5)further shows thatMLP Swappingalso works for a data\-dependent constructed Hebbian MLP\.MLP Swappingwith Hebbian MLPs keeps the edit score above0\.980\.98through up to10%10\\%edited facts, while the strongest baseline reaches only0\.8470\.847—a 13\.3\-percentage\-point gain \(Figure[9](https://arxiv.org/html/2607.10034#A1.F9)\)\.

## 6Discussion

Our work presents a stepping stone toward understanding MLPs within Transformers from a constructive lens\. We present an MLP construction that achieves optimal margin and fact\-storage capacity, provides provable margin guarantees for arbitrary embeddings, and remains usable within Transformer blocks for factual recall\. We also show an application of modular fact\-storing MLPs in fact editing, illustrating a path toward robust, modular knowledge manipulation in LLMs\.

Our analysis currently applies to constructed MLPs in a single\-layer Transformer setting\. Extending to MLPs in pretrained LLMs—for example, by understanding how the geometries of real LLM embeddings affect the kernels MLPs learn—would provide a principled lens for investigating how trained MLPs store knowledge\. Furthermore, moving beyond the single\-layer setting would let us study multi\-hop recall and more realistic editing scenarios\.

## Impact Statement

This paper presents work whose goal is to advance the field of machine learning\. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here\.

## Acknowledgements

The authors thank Yasa Baig, Kelly Buchanan, Mayee Chen, Vivien Cheng, Catherine Deng, Owen Dugan, Rajat Vadiraj Dwaraknath, Neel Guha, Junmiao Hu, Ishan Khare, Hermann Kumbong, Eshaan Nichani, Jon Saad\-Falcon, Thanawat Sornwanee, Stuart Sul, Alex Waitz, John Winnicki, Morris Yau, Michael Zhang, and Dylan Zinsley for their helpful feedback and discussion\.

The authors gratefully acknowledge the support of NIH under No\. U54EB020405 \(Mobilize\), NSF under Nos\. CCF2247015 \(Hardware\-Aware\), CCF1763315 \(Beyond Sparsity\), CCF1563078 \(Volume to Velocity\), and 1937301 \(RTML\); US DEVCOM ARL under Nos\. W911NF\-23\-2\-0184 \(Long\-context\) and W911NF\-21\-2\-0251 \(Interactive Human\-AI Teaming\); ONR under Nos\. N000142312633 \(Deep Signal Processing\); Stanford HAI under No\. 247183; NXP, Xilinx, LETI\-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI\-GCP Cloud Credits for Research program, the Stanford Data Science Initiative \(SDSI\), and members of the Stanford DAWN project: Meta, Google, and VMWare\. The U\.S\. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon\. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U\.S\. Government\. JL is supported by the Department of Energy Computational Science Graduate Fellowship under Award Number DE\-SC0023112\. AR’s research is supported by NSF grant CCF\#2247014\.

## References

- Z\. Allen\-Zhu and Y\. Li \(2024\)Physics of language models: part 3\.3, knowledge capacity scaling laws\.External Links:2404\.05405,[Link](https://arxiv.org/abs/2404.05405)Cited by:[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px2.p1.1)\.
- H\. Bong and A\. K\. Kuchibhotla \(2023\)Tight concentration inequality for sub\-Weibull random variables with generalized Bernstein Orlicz norms\.arXiv preprint arXiv:2302\.03850\.External Links:[Link](https://arxiv.org/abs/2302.03850)Cited by:[§B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5.3.p1.5)\.
- S\. Bubeck, R\. Eldan, Y\. T\. Lee, and D\. Mikulincer \(2020\)Network size and weights size for memorization with two\-layers neural networks\.External Links:2006\.02855,[Link](https://arxiv.org/abs/2006.02855)Cited by:[§2\.2](https://arxiv.org/html/2607.10034#S2.SS2.p1.1)\.
- V\. Cabannes, E\. Dohmatob, and A\. Bietti \(2024\)Scaling laws for associative memories\.External Links:2310\.02984,[Link](https://arxiv.org/abs/2310.02984)Cited by:[§2\.2](https://arxiv.org/html/2607.10034#S2.SS2.p1.1)\.
- D\. Dai, L\. Dong, Y\. Hao, Z\. Sui, B\. Chang, and F\. Wei \(2022\)Knowledge neurons in pretrained transformers\.External Links:2104\.08696,[Link](https://arxiv.org/abs/2104.08696)Cited by:[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.10034#S1.p1.1),[§1](https://arxiv.org/html/2607.10034#S1.p2.1)\.
- DeepSeek\-AI, A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan, D\. Dai, D\. Guo, D\. Yang, D\. Chen, D\. Ji, E\. Li, F\. Lin, F\. Dai, F\. Luo, G\. Hao, G\. Chen, G\. Li, H\. Zhang, H\. Bao, H\. Xu, H\. Wang, H\. Zhang, H\. Ding, H\. Xin, H\. Gao, H\. Li, H\. Qu, J\. L\. Cai, J\. Liang, J\. Guo, J\. Ni, J\. Li, J\. Wang, J\. Chen, J\. Chen, J\. Yuan, J\. Qiu, J\. Li, J\. Song, K\. Dong, K\. Hu, K\. Gao, K\. Guan, K\. Huang, K\. Yu, L\. Wang, L\. Zhang, L\. Xu, L\. Xia, L\. Zhao, L\. Wang, L\. Zhang, M\. Li, M\. Wang, M\. Zhang, M\. Zhang, M\. Tang, M\. Li, N\. Tian, P\. Huang, P\. Wang, P\. Zhang, Q\. Wang, Q\. Zhu, Q\. Chen, Q\. Du, R\. J\. Chen, R\. L\. Jin, R\. Ge, R\. Zhang, R\. Pan, R\. Wang, R\. Xu, R\. Zhang, R\. Chen, S\. S\. Li, S\. Lu, S\. Zhou, S\. Chen, S\. Wu, S\. Ye, S\. Ye, S\. Ma, S\. Wang, S\. Zhou, S\. Yu, S\. Zhou, S\. Pan, T\. Wang, T\. Yun, T\. Pei, T\. Sun, W\. L\. Xiao, W\. Zeng, W\. Zhao, W\. An, W\. Liu, W\. Liang, W\. Gao, W\. Yu, W\. Zhang, X\. Q\. Li, X\. Jin, X\. Wang, X\. Bi, X\. Liu, X\. Wang, X\. Shen, X\. Chen, X\. Zhang, X\. Chen, X\. Nie, X\. Sun, X\. Wang, X\. Cheng, X\. Liu, X\. Xie, X\. Liu, X\. Yu, X\. Song, X\. Shan, X\. Zhou, X\. Yang, X\. Li, X\. Su, X\. Lin, Y\. K\. Li, Y\. Q\. Wang, Y\. X\. Wei, Y\. X\. Zhu, Y\. Zhang, Y\. Xu, Y\. Xu, Y\. Huang, Y\. Li, Y\. Zhao, Y\. Sun, Y\. Li, Y\. Wang, Y\. Yu, Y\. Zheng, Y\. Zhang, Y\. Shi, Y\. Xiong, Y\. He, Y\. Tang, Y\. Piao, Y\. Wang, Y\. Tan, Y\. Ma, Y\. Liu, Y\. Guo, Y\. Wu, Y\. Ou, Y\. Zhu, Y\. Wang, Y\. Gong, Y\. Zou, Y\. He, Y\. Zha, Y\. Xiong, Y\. Ma, Y\. Yan, Y\. Luo, Y\. You, Y\. Liu, Y\. Zhou, Z\. F\. Wu, Z\. Z\. Ren, Z\. Ren, Z\. Sha, Z\. Fu, Z\. Xu, Z\. Huang, Z\. Zhang, Z\. Xie, Z\. Zhang, Z\. Hao, Z\. Gou, Z\. Ma, Z\. Yan, Z\. Shao, Z\. Xu, Z\. Wu, Z\. Zhang, Z\. Li, Z\. Gu, Z\. Zhu, Z\. Liu, Z\. Li, Z\. Xie, Z\. Song, Z\. Gao, and Z\. Pan \(2025\)DeepSeek\-v3 technical report\.External Links:2412\.19437,[Link](https://arxiv.org/abs/2412.19437)Cited by:[§2\.2](https://arxiv.org/html/2607.10034#S2.SS2.p2.5)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv e\-prints,pp\. arXiv–2407\.Cited by:[§2\.2](https://arxiv.org/html/2607.10034#S2.SS2.p2.5)\.
- N\. Elhage, T\. Hume, C\. Olsson, N\. Schiefer, T\. Henighan, S\. Kravec, Z\. Hatfield\-Dodds, R\. Lasenby, D\. Drain, C\. Chen, R\. Grosse, S\. McCandlish, J\. Kaplan, D\. Amodei, M\. Wattenberg, and C\. Olah \(2022\)Toy models of superposition\.External Links:2209\.10652,[Link](https://arxiv.org/abs/2209.10652)Cited by:[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px2.p1.1)\.
- K\. Ethayarajh \(2019\)How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt\-2 embeddings\.External Links:1909\.00512,[Link](https://arxiv.org/abs/1909.00512)Cited by:[§1](https://arxiv.org/html/2607.10034#S1.p2.1),[§4\.2\.2](https://arxiv.org/html/2607.10034#S4.SS2.SSS2.p1.1)\.
- J\. Fang, H\. Jiang, K\. Wang, Y\. Ma, S\. Jie, X\. Wang, X\. He, and T\. Chua \(2025\)AlphaEdit: null\-space constrained knowledge editing for language models\.External Links:2410\.02355,[Link](https://arxiv.org/abs/2410.02355)Cited by:[4th item](https://arxiv.org/html/2607.10034#A1.I4.i4.p1.1),[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2607.10034#S5.SS2.p2.5)\.
- M\. Geva, A\. Caciularu, K\. R\. Wang, and Y\. Goldberg \(2022\)Transformer feed\-forward layers build predictions by promoting concepts in the vocabulary space\.External Links:2203\.14680,[Link](https://arxiv.org/abs/2203.14680)Cited by:[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px1.p1.1)\.
- M\. Geva, R\. Schuster, J\. Berant, and O\. Levy \(2021\)Transformer feed\-forward layers are key\-value memories\.External Links:2012\.14913,[Link](https://arxiv.org/abs/2012.14913)Cited by:[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.10034#S1.p1.1),[§1](https://arxiv.org/html/2607.10034#S1.p2.1)\.
- Google DeepMind \(2024\)AI achieves silver\-medal standard solving international mathematical olympiad problems with alphaproof\.Note:Blog post“AlphaProof is a system that trains itself to prove mathematical statements in the formal language Lean …”, solves 3 IMO problems etc\.External Links:[Link](https://deepmind.google/discover/blog/ai-solves-imo-problems-at-silver-medal-level/)Cited by:[§1](https://arxiv.org/html/2607.10034#S1.p1.1)\.
- J\. Gu, H\. Xu, J\. Ma, P\. Lu, Z\. Ling, K\. Chang, and N\. Peng \(2024\)Model editing harms general abilities of large language models: regularization to the rescue\.External Links:2401\.04700,[Link](https://arxiv.org/abs/2401.04700)Cited by:[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px1.p1.1)\.
- N\. Guha, J\. Nyarko, D\. Ho, C\. Ré, A\. Chilton, A\. Chohlas\-Wood, A\. Peters, B\. Waldon, D\. Rockmore, D\. Zambrano,et al\.\(2023\)Legalbench: a collaboratively built benchmark for measuring legal reasoning in large language models\.Advances in Neural Information Processing Systems36,pp\. 44123–44279\.Cited by:[§1](https://arxiv.org/html/2607.10034#S1.p1.1)\.
- A\. Gupta, A\. Rao, and G\. Anumanchipalli \(2024\)Model editing at scale leads to gradual and catastrophic forgetting\.External Links:2401\.07453,[Link](https://arxiv.org/abs/2401.07453)Cited by:[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px1.p1.1)\.
- J\. J\. Hopfield \(1982\)Neural networks and physical systems with emergent collective computational abilities\.\.Proceedings of the National Academy of Sciences79\(8\),pp\. 2554–2558\.External Links:[Document](https://dx.doi.org/10.1073/pnas.79.8.2554),[Link](https://www.pnas.org/doi/abs/10.1073/pnas.79.8.2554),https://www\.pnas\.org/doi/pdf/10\.1073/pnas\.79\.8\.2554Cited by:[§2\.2](https://arxiv.org/html/2607.10034#S2.SS2.p1.1)\.
- T\. Kohonen \(1972\)Correlation matrix memories\.IEEE Trans\. Comput\.21\(4\),pp\. 353–359\.External Links:ISSN 0018\-9340,[Link](https://doi-org.stanford.idm.oclc.org/10.1109/TC.1972.5008975),[Document](https://dx.doi.org/10.1109/TC.1972.5008975)Cited by:[§2\.2](https://arxiv.org/html/2607.10034#S2.SS2.p1.1)\.
- A\. K\. Kuchibhotla and A\. Chakrabortty \(2022\)Moving beyond sub\-gaussianity in high\-dimensional statistics: applications in covariance estimation and linear regression\.Information and Inference: A Journal of the IMA11\(4\),pp\. 1389–1456\.Cited by:[§B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5.5.p2.6)\.
- B\. Laurent and P\. Massart \(2000\)Adaptive estimation of a quadratic functional by model selection\.Annals of Statistics28\(5\),pp\. 1302–1338\.Cited by:[§B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5.p3.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2023a\)Locating and editing factual associations in gpt\.External Links:2202\.05262,[Link](https://arxiv.org/abs/2202.05262)Cited by:[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.10034#S1.p1.1)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2023b\)Locating and editing factual associations in gpt\.External Links:2202\.05262,[Link](https://arxiv.org/abs/2202.05262)Cited by:[3rd item](https://arxiv.org/html/2607.10034#A1.I4.i3.p1.1)\.
- K\. Meng, A\. S\. Sharma, A\. Andonian, Y\. Belinkov, and D\. Bau \(2023c\)Mass\-editing memory in a transformer\.External Links:2210\.07229,[Link](https://arxiv.org/abs/2210.07229)Cited by:[2nd item](https://arxiv.org/html/2607.10034#A1.I4.i2.p1.1),[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px1.p1.1),[§5\.2](https://arxiv.org/html/2607.10034#S5.SS2.p2.5)\.
- J\. X\. Morris, C\. Sitawarin, C\. Guo, N\. Kokhlikyan, G\. E\. Suh, A\. M\. Rush, K\. Chaudhuri, and S\. Mahloujifar \(2025\)How much do language models memorize?\.External Links:2505\.24832,[Link](https://arxiv.org/abs/2505.24832)Cited by:[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px2.p1.1)\.
- N\. Nanda, S\. Rajamanoharan, J\. Kramár, and R\. Shah \(2023\)Fact finding: attempting to reverse\-engineer factual recall on the neuron level\.External Links:[Link](https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall)Cited by:[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px1.p1.1)\.
- E\. Nichani, J\. D\. Lee, and A\. Bietti \(2024\)Understanding factual recall in transformers via associative memories\.External Links:2412\.06538,[Link](https://arxiv.org/abs/2412.06538)Cited by:[itemNTK\.](https://arxiv.org/html/2607.10034#A1.I1.ix2.p1.1),[§A\.3\.1](https://arxiv.org/html/2607.10034#A1.SS3.SSS1.p1.7),[§B\.3](https://arxiv.org/html/2607.10034#A2.SS3.p1.1),[§B\.3](https://arxiv.org/html/2607.10034#A2.SS3.p4.8),[§1](https://arxiv.org/html/2607.10034#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.10034#S2.SS2.p1.1),[§2\.3](https://arxiv.org/html/2607.10034#S2.SS3.p1.1),[§4\.3](https://arxiv.org/html/2607.10034#S4.SS3.SSS0.Px1.p1.1)\.
- R\. O’Donnell \(2014\)Analysis of boolean functions\.Cambridge University Press\.Cited by:[§B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5.p1.1)\.
- F\. Petroni, T\. Rocktäschel, P\. Lewis, A\. Bakhtin, Y\. Wu, A\. H\. Miller, and S\. Riedel \(2019\)Language models as knowledge bases?\.arXiv preprint arXiv:1909\.01066\.Cited by:[§1](https://arxiv.org/html/2607.10034#S1.p1.1)\.
- A\. Razzhigaev, M\. Mikhalchuk, E\. Goncharova, I\. Oseledets, D\. Dimitrov, and A\. Kuznetsov \(2024\)The shape of learning: anisotropy and intrinsic dimensions in transformer\-based models\.External Links:2311\.05928,[Link](https://arxiv.org/abs/2311.05928)Cited by:[§1](https://arxiv.org/html/2607.10034#S1.p2.1),[§4\.2\.2](https://arxiv.org/html/2607.10034#S4.SS2.SSS2.p1.1)\.
- K\. Saab, T\. Tu, W\. Weng, R\. Tanno, D\. Stutz, E\. Wulczyn, F\. Zhang, T\. Strother, C\. Park, E\. Vedadi,et al\.\(2024\)Capabilities of gemini models in medicine\.arXiv preprint arXiv:2404\.18416\.Cited by:[§1](https://arxiv.org/html/2607.10034#S1.p1.1)\.
- N\. Shazeer \(2020\)GLU variants improve transformer\.External Links:2002\.05202,[Link](https://arxiv.org/abs/2002.05202)Cited by:[§2\.2](https://arxiv.org/html/2607.10034#S2.SS2.p2.5)\.
- W\. Sun, T\. Qu, M\. Li, J\. Davis, and M\. Moens \(2025\)Mitigating negative interference in multilingual sequential knowledge editing through null\-space constraints\.External Links:2506\.10800,[Link](https://arxiv.org/abs/2506.10800)Cited by:[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px1.p1.1)\.
- R\. Vershynin \(2018\)High\-dimensional probability: an introduction with applications in data science\.Vol\.47,pp\. 67–68\.Cited by:[§B\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2.p2.7),[§B\.2\.6](https://arxiv.org/html/2607.10034#A2.SS2.SSS6.Px1.10.p1.2)\.
- R\. Vershynin \(2026\)High\-dimensional probability: an introduction with applications in data science\.2nd edition,Cambridge Series in Statistical and Probabilistic Mathematics,Cambridge University Press\.External Links:[Document](https://dx.doi.org/10.1017/9781009490672),ISBN 9781009490672Cited by:[§B\.9\.1](https://arxiv.org/html/2607.10034#A2.SS9.SSS1.2.p1.1),[§B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5.p2.1)\.
- M\. Wan and J\. McAuley \(2018\)Item recommendation on monotonic behavior chains\.InProceedings of the 12th ACM Conference on Recommender Systems,RecSys ’18,New York, NY, USA,pp\. 86–94\.External Links:ISBN 9781450359016,[Link](https://doi.org/10.1145/3240323.3240369),[Document](https://dx.doi.org/10.1145/3240323.3240369)Cited by:[§A\.4\.5](https://arxiv.org/html/2607.10034#A1.SS4.SSS5.Px1.p1.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§2\.2](https://arxiv.org/html/2607.10034#S2.SS2.p2.5)\.
- S\. Zhong, M\. Xu, T\. Ao, and G\. Shi \(2025\)Understanding transformer from the perspective of associative memory\.External Links:2505\.19488,[Link](https://arxiv.org/abs/2505.19488)Cited by:[§2\.3](https://arxiv.org/html/2607.10034#S2.SS3.p1.1)\.
- N\. Zucchet, J\. Bornschein, S\. Chan, A\. Lampinen, R\. Pascanu, and S\. De \(2025\)How do language models learn facts? dynamics, curricula and hallucinations\.External Links:2503\.21676,[Link](https://arxiv.org/abs/2503.21676)Cited by:[Appendix C](https://arxiv.org/html/2607.10034#A3.SS0.SSS0.Px2.p1.1)\.

## Appendix

## Appendix AExperiments

### A\.1Model Definitions

#### A\.1\.1MLP Variants

Across the standalone fact\-storage\-capacity experiments \([Sections4\.3](https://arxiv.org/html/2607.10034#S4.SS3)and[A\.3\.1](https://arxiv.org/html/2607.10034#A1.SS3.SSS1)\) and the Transformer\-block capacity experiments \([SectionA\.4\.3](https://arxiv.org/html/2607.10034#A1.SS4.SSS3)\), we compare the following MLP variants at matched hidden widthmmon the same synthetic fact sets:

GD \(gradient\-descent trained\)\.Our default GD baseline is a bilinear gated\-identity MLP, i\.e\. the bilinear specialization of the gated MLP family in[Equation4](https://arxiv.org/html/2607.10034#S2.E4)\. Unless stated otherwise, we train these models for10,00010\{,\}000epochs using Adam with initial learning rate10−310^\{\-3\}and a cosine\-annealing schedule down to10−610^\{\-6\}\.

NTK\.Our NTK baseline is the degree\-11Hermite weight construction ofNichaniet al\.\([2024](https://arxiv.org/html/2607.10034#bib.bib40)\), which uses a gated ReLU MLP architecture\. A self\-contained description is given in[SectionB\.3](https://arxiv.org/html/2607.10034#A2.SS3)\.

Our construction\.Our closed\-form construction is a Hebbian MLP with sketched\-K2K\_\{2\}kernel, i\.e\. the bilinear random\-feature construction described in[Sections4\.1](https://arxiv.org/html/2607.10034#S4.SS1)and[1](https://arxiv.org/html/2607.10034#alg1)\. We study three variants:

- •Unwhitened\.This is the raw sketched\-K2K\_\{2\}Hebbian construction, with the bilinear random\-feature map used directly in the final Hebbian readout\.
- •Whitened\.This uses the same bilinear feature map, but applies the full readout whitening procedure from[SectionB\.2\.4](https://arxiv.org/html/2607.10034#A2.SS2.SSS4)with ridge parameter10−610^\{\-6\}\.
- •Data\-dependent\.This keeps the same bilinear architecture, but performs least squares solves to obtain the random feature vectors within the Hebbian kernel, as described in[SectionB\.2\.5](https://arxiv.org/html/2607.10034#A2.SS2.SSS5), then applies whitening with ridge parameter10−610^\{\-6\}\.

#### A\.1\.2Transformer Setups

We describe details about the modified 1\-layer, 1\-head GPT\-style Transformer architecture used across all of our SSFR experiments unless otherwise specified\.

We freeze the input and output embeddings, and when an experiment includes a fact\-storing MLP, that MLP uses the same embeddings as the Transformer\. Positional encodings are disabled throughout\. We use RMSNorm before the attention layer, and use unit\-RMSNorm \(which projects to the unit sphere\) before the MLP and before the final Transformer output\. We train with AdamW using learning rate2×10−42\\times 10^\{\-4\}, batch size1,2801\{,\}280, and4,0004\{,\}000iterations\.

Our experiments include two different training protocols:

- •In the*pretrained attention*setting, we first train attention alone on a dummy SSFR task using the identity map, so that it learns to query the MLP when faced with distractor junk tokens in its context\. After training, we freeze the attention layer, then train a frozen fact\-storing MLP on a fresh random\-permutation fact set using the same embeddings, insert it into the Transformer, and evaluate the combined model on SSFR with that fact set\.
- •In the*inserted MLP*setting, we first construct or train a frozen fact\-storing MLP on a random\-permutation fact set, then insert it into the Transformer block\. We then train the attention layer around that frozen MLP on SSFR using the same fact set\.

### A\.2Experimental details for[Section3](https://arxiv.org/html/2607.10034#S3)

#### A\.2\.1Synthetic Sequential Factual Recall \(SSFR\)

We instantiate SSFR using the contextual single\-token recall task

𝒮SSFR​\[f\]:=\{concat⁡\(jpre,k,jsuf,q,f​\(k\)\):k∈𝒮k,jpre∈𝒥pre,jsuf∈𝒥suf\},\\mathcal\{S\}\_\{\\mathrm\{SSFR\}\}\[f\]:=\\left\\\{\\operatorname\{concat\}\(j^\{\\mathrm\{pre\}\},\\,k,\\,j^\{\\mathrm\{suf\}\},\\,q,\\,f\(k\)\)\\;:\\;k\\in\\mathcal\{S\}\_\{k\},\\;j^\{\\mathrm\{pre\}\}\\in\\mathcal\{J\}\_\{\\mathrm\{pre\}\},\\;j^\{\\mathrm\{suf\}\}\\in\\mathcal\{J\}\_\{\\mathrm\{suf\}\}\\right\\\},wheref:𝒮k→𝒮vf:\\mathcal\{S\}\_\{k\}\\to\\mathcal\{S\}\_\{v\}is a bijection defining the fact set,qqis a dedicated query token, andjpre,jsufj^\{\\mathrm\{pre\}\},j^\{\\mathrm\{suf\}\}are junk\-token strings that do not belong to the fact vocabulary\. Keys and values are treated as single tokens\.

Unless otherwise specified, the default SSFR configuration used in our experiments uses a junk vocabulary of size99and junk\-prefixes and junk\-suffixes of length99\. Transformer architecture and training details are summarized in Appendix[A\.1\.2](https://arxiv.org/html/2607.10034#A1.SS1.SSS2)\.

#### A\.2\.2Margins Govern Transformer Usability Sweep

The hidden\-width sweep in Figure[2](https://arxiv.org/html/2607.10034#S4.F2)a uses the*pretrained attention*setting from Appendix[A\.1\.2](https://arxiv.org/html/2607.10034#A1.SS1.SSS2)and uses GD\-trained fact\-storing MLPs\. We usedmodel=128d\_\{\\mathrm\{model\}\}=128,F=2048F=2048,J=VJ=9J=V\_\{J\}=9, and sweep across1616logarithmically spaced hidden widths in\[16,256\]\[16,256\], with four seeds per width\. For each width we report the standalone MLP margin and accuracy together with the accuracy of the combined Transformer block on the random fact set\.

### A\.3Experimental details for[Section4](https://arxiv.org/html/2607.10034#S4)

#### A\.3\.1MLP Fact\-Storage Capacity

We estimate the MLP fact\-storage capacity in Figure[2](https://arxiv.org/html/2607.10034#S4.F2)c by binary\-searching over hidden widthmmat fixed embedding dimensionddand fact countFF\. We sweep over

d∈\{64,90,128\},α:=F/d2∈\{1/32,1/16,1/8,1/4,1/2,3/4,1\}\.d\\in\\\{64,90,128\\\},\\qquad\\alpha:=F/d^\{2\}\\in\\\{1/32,1/16,1/8,1/4,1/2,3/4,1\\\}\.For each\(method,d,α\)\(\\text\{method\},d,\\alpha\)tuple, we generate a random\-permutation fact set and sample key and value embeddings from the unit sphere\. We then binary\-search for the smallest hidden widthmmthat attains 100% fact storage on the sampled fact set and report the corresponding parameter countWW\. We compare GD\-trained bilinear MLPs, our sketched\-K2K\_\{2\}Hebbian construction \(with and without whitening, and our data\-dependent kernel variant\), and the NTK baseline fromNichaniet al\.\([2024](https://arxiv.org/html/2607.10034#bib.bib40)\), as described in Appendix[A\.1](https://arxiv.org/html/2607.10034#A1.SS1)\.

#### A\.3\.2Margin Sweeps

We use the same bilinear sketched\-K2K\_\{2\}construction for the isotropic margin sweeps in Figure[2](https://arxiv.org/html/2607.10034#S4.F2)b, as well as for the arbitrary\-geometry comparisons summarized in Table[2](https://arxiv.org/html/2607.10034#A2.T2)\.

##### Isotropic keys and values\.

For the isotropic keys and isotropic values sweeps supporting Appendix[B\.8\.4](https://arxiv.org/html/2607.10034#A2.SS8.SSS4), we sample unit\-norm spherical keys and values from a random\-permutation fact set, vary eitherFFormm, and construct the MLP at each point\. We then compare the measured minimum margin with the theoretical prediction \([Equation56](https://arxiv.org/html/2607.10034#A2.E56)\)\. TheFF\-sweep \(exactK2K\_\{2\}\) usesd=32d=32,m=512m=512,F∈\[32,2048\]F\\in\[32,2048\]\(30 log\-spaced points, 5 seeds\)\. TheFF\-sweep \(bilinear sketched\-K2K\_\{2\}\) usesd=64d=64,m=512m=512,F∈\[32,512\]F\\in\[32,512\]\(30 log\-spaced points, 5 seeds\)\. Themm\-sweep \(bilinear bilinear sketched\-K2K\_\{2\}\) usesd=64d=64,F=256F=256,m∈\[64,1024\]m\\in\[64,1024\]\(30 log\-spaced points, 5 seeds\)\.

![Refer to caption](https://arxiv.org/html/2607.10034v1/x4.png)Figure 4:Margin scaling with number of facts \(isotropic setting\)\.Minimum margin of bilinear sketched\-K2K\_\{2\}MLPs decreases with the number of stored factsFF, following the predicted scaling from[Equation56](https://arxiv.org/html/2607.10034#A2.E56)\. The empirical margin closely tracks the theoretical bound across theFF\-sweep withd=64d=64,m=512m=512\.
##### Arbitrary keys, isotropic values\.

For the arbitrary\-key sweep supporting Appendix[B\.8\.2](https://arxiv.org/html/2607.10034#A2.SS8.SSS2), we start from spherical keys and apply a rank\-1 spike transform

𝐤i′=𝐤i\+β​⟨𝐤i,𝐮⟩​𝐮‖𝐤i\+β​⟨𝐤i,𝐮⟩​𝐮‖2,\\mathbf\{k\}\_\{i\}^\{\\prime\}=\\frac\{\\mathbf\{k\}\_\{i\}\+\\beta\\langle\\mathbf\{k\}\_\{i\},\\mathbf\{u\}\\rangle\\mathbf\{u\}\}\{\\\|\\mathbf\{k\}\_\{i\}\+\\beta\\langle\\mathbf\{k\}\_\{i\},\\mathbf\{u\}\\rangle\\mathbf\{u\}\\\|\_\{2\}\},where𝐮∈𝕊d−1\\mathbf\{u\}\\in\\mathbb\{S\}^\{d\-1\}is fixed within a sweep andβ≥0\\beta\\geq 0controls the anisotropy strength\. Asβ\\betaincreases, the keys crowd along𝐮\\mathbf\{u\}, increasing the key\-side quantities entering the bound, especiallyEKE\_\{K\}\. Values remain isotropic, sampled uniformly from the unit sphere\. At eachβ\\beta, we construct the MLP, measureγmin\\gamma\_\{\\min\}, and compare it against the theorem, plug\-in, and heuristic predictions for the resulting key geometry \([Equation46](https://arxiv.org/html/2607.10034#A2.E46)\)\. We used=64d=64,F=128F=128,m=512m=512,β∈\[0,5\]\\beta\\in\[0,5\]\.

##### Isotropic keys, arbitrary values\.

For the arbitrary\-value sweep supporting Appendix[B\.8\.3](https://arxiv.org/html/2607.10034#A2.SS8.SSS3), we keep the keys isotropic and apply the same rank\-1 spike transform to the values:

𝐯i′=𝐯i\+β​⟨𝐯i,𝐮⟩​𝐮‖𝐯i\+β​⟨𝐯i,𝐮⟩​𝐮‖2\.\\mathbf\{v\}\_\{i\}^\{\\prime\}=\\frac\{\\mathbf\{v\}\_\{i\}\+\\beta\\langle\\mathbf\{v\}\_\{i\},\\mathbf\{u\}\\rangle\\mathbf\{u\}\}\{\\\|\\mathbf\{v\}\_\{i\}\+\\beta\\langle\\mathbf\{v\}\_\{i\},\\mathbf\{u\}\\rangle\\mathbf\{u\}\\\|\_\{2\}\}\.Again𝐮\\mathbf\{u\}is fixed within a sweep andβ\\betais the control parameter\. This changes the value\-side quantitiesVminV\_\{\\min\},BYB\_\{Y\}, andEvE\_\{v\}while preserving unit norm\. Keys remain isotropic, sampled uniformly from the unit sphere\. At eachβ\\beta, we rebuild the MLP, measure the empirical decoding margin, and compare it against the corresponding theorem, plug\-in, and heuristic predictions \([Equation51](https://arxiv.org/html/2607.10034#A2.E51)\)\. We used=64d=64,F=128F=128,m=512m=512,β∈\[0,10\]\\beta\\in\[0,10\]\.

##### Arbitrary keys and values\.

For the fully structured sweep supporting Appendix[B\.8\.1](https://arxiv.org/html/2607.10034#A2.SS8.SSS1), we apply the same rank\-1 spike model to both keys and values, using the same spike strengthβ\\betaand fixed direction𝐮\\mathbf\{u\}:

𝐞i′=𝐞i\+β​⟨𝐞i,𝐮⟩​𝐮‖𝐞i\+β​⟨𝐞i,𝐮⟩​𝐮‖2,𝐞i∈\{𝐤i,𝐯i\}\.\\mathbf\{e\}\_\{i\}^\{\\prime\}=\\frac\{\\mathbf\{e\}\_\{i\}\+\\beta\\langle\\mathbf\{e\}\_\{i\},\\mathbf\{u\}\\rangle\\mathbf\{u\}\}\{\\\|\\mathbf\{e\}\_\{i\}\+\\beta\\langle\\mathbf\{e\}\_\{i\},\\mathbf\{u\}\\rangle\\mathbf\{u\}\\\|\_\{2\}\},\\qquad\\mathbf\{e\}\_\{i\}\\in\\\{\\mathbf\{k\}\_\{i\},\\mathbf\{v\}\_\{i\}\\\}\.This setting makes key crowding and value interference vary coherently, which is the regime where the coupling factorκ\\kappaemerges\. For eachβ\\beta, we recompute the geometric summary statistics entering the deterministic bound and compare the measured margin against the theorem, plug\-in, and heuristic predictions, focusing on the composite cross\-talk scaleEK​Ev​κ\\sqrt\{E\_\{K\}\}\\sqrt\{E\_\{v\}\}\\,\\kappa\([Equation44](https://arxiv.org/html/2607.10034#A2.E44)\)\. We used=64d=64,F=128F=128,m=512m=512,β∈\[0,3\.35\]\\beta\\in\[0,3\.35\]\.

![Refer to caption](https://arxiv.org/html/2607.10034v1/x5.png)\(a\)Arbitrary keys, isotropic values \(β\\beta\-sweep\)\. Asβ\\betaincreases, key crowding grows and the measured margin tracks the theoretical bound via key\-geometry termsKmindiagK\_\{\\min\}^\{\\mathrm\{diag\}\},KmaxoffK\_\{\\max\}^\{\\mathrm\{off\}\}, andEKE\_\{K\}\.
![Refer to caption](https://arxiv.org/html/2607.10034v1/x6.png)\(b\)Isotropic keys, arbitrary values \(β\\beta\-sweep\)\. Increasingβ\\betaconcentrates values and the margin closely follows the bound through value\-geometry termsVminV\_\{\\min\},BYB\_\{Y\}, andEvE\_\{v\}\.
![Refer to caption](https://arxiv.org/html/2607.10034v1/x7.png)\(c\)Arbitrary keys and values \(β\\beta\-sweep\)\. Both geometries are spiked simultaneously; the margin degradation is governed by the composite cross\-talk scaleEK​Ev​κ\\sqrt\{E\_\{K\}\}\\sqrt\{E\_\{v\}\}\\,\\kappa, matching the deterministic bound\.

Figure 5:Marginβ\\beta\-sweeps across arbitrary key/value geometry regimes\.Each panel applies a rank\-1 spike transform with strengthβ\\betato the keys \(left\), values \(middle\), or both \(right\), usingF=128F=128facts\. In all three cases the theoretical bound tracks the empirically measured minimum margin \(R2≥0\.95R^\{2\}\\geq 0\.95\), validating the margin decomposition of[Equation6](https://arxiv.org/html/2607.10034#S4.E6)and the geometric summary statistics of[Table2](https://arxiv.org/html/2607.10034#A2.T2)\. The arbitrary keys and values plot uses the raw summary\-statistic form of[TheoremB\.25](https://arxiv.org/html/2607.10034#A2.Thmtheorem25), which is equivalent to the penalty\-statistic general margin/capacity bound in[SectionB\.8\.1](https://arxiv.org/html/2607.10034#A2.SS8.SSS1)of the main text\.

### A\.4Experimental details for[Section5](https://arxiv.org/html/2607.10034#S5)

#### A\.4\.1Attention\-Noise Sweep

The attention\-noise experiment in Figure[3](https://arxiv.org/html/2607.10034#S5.F3)a uses the*pretrained attention*Transformer training setup \(Appendix[A\.1\.2](https://arxiv.org/html/2607.10034#A1.SS1.SSS2)\) and isolates the attention module during the attention\-only pretraining phase\. At evaluation, for each stored key and junk context sampled, we measure theℓ2\\ell\_\{2\}deviation between the attention output at the query position and the corresponding stored key embedding\. We then aggregate these deviations to estimate the attention noise floor\.

We vary junk lengthJ∈\{2,4,8,16\}J\\in\\\{2,4,8,16\\\}and couple the junk vocabulary size to the length \(VJ=JV\_\{J\}=J\)\. We run our sweeps using model sizes

\(dmodel,F\)∈\{\(64,512\),\(96,1152\),\(128,2048\)\}\.\(d\_\{\\mathrm\{model\}\},F\)\\in\\\{\(64,512\),\(96,1152\),\(128,2048\)\\\}\.and average over four seeds\.

#### A\.4\.2Noisy\-Margin Diagnostic

The noisy\-margin curve in Figure[3](https://arxiv.org/html/2607.10034#S5.F3)b measures how the minimum margin of our bilinear construction changes under perturbed key queries\. We use isotropic keys and values withd=64d=64,F=256F=256, and hidden widthm=512m=512\. For each stored key, we perturb it via

𝐤~i=𝐤i\+ϵ​𝐮i,𝐮i∼Unif​\(Sd−1\),\\widetilde\{\\mathbf\{k\}\}\_\{i\}=\\mathbf\{k\}\_\{i\}\+\\epsilon\\mathbf\{u\}\_\{i\},\\qquad\\mathbf\{u\}\_\{i\}\\sim\\mathrm\{Unif\}\(S^\{d\-1\}\),and sweepϵ\\epsilonlogarithmically from0\.010\.01to2\.02\.0over1515values\. For each sweep point, we report the empirical noisy minimum margin and the predicted linear degradation term, averaged over three seeds\.

![Refer to caption](https://arxiv.org/html/2607.10034v1/x8.png)Figure 6:Margin degradation under noisy queries\.Bilinear MLP minimum margin decreases linearly with the magnitude of key perturbationsϵ\\epsilon, validating the noisy\-margin degradation bound\. The empirical margin \(solid line\) closely follows the predicted linear degradation term \(dashed line\) withd=64d=64,F=256F=256,m=512m=512\.
#### A\.4\.3Transformer Capacity Sweep

Our Transformer capacity plots use the SSFR task from Appendix[A\.2\.1](https://arxiv.org/html/2607.10034#A1.SS2.SSS1)and the*inserted MLP*training setup from Appendix[A\.1\.2](https://arxiv.org/html/2607.10034#A1.SS1.SSS2)\. In our experiments, we compare GD, our unwhitened construction, our whitened construction, our data\-dependent construction, and the NTK baseline\. Unless otherwise stated, we use embedding dimensiond=128d=128, junk lengthJ=9J=9, junk vocabulary sizeVJ=9V\_\{J\}=9,4,0004\{,\}000training epochs, and one seed\.

We evaluate Transformer capacity using two complementary success criteria:

- •Training accuracy\.In this variant, we evaluate a Transformer’s accuracy on SSFR using the MLP and the fact set it is trained with\. This criterion assesses whether the inserted MLP is well\-behaved enough that it can be used inside a Transformer at all\. Crucially, this criterion is unable to distinguish between when a Transformer is learning to query its fact\-storing MLP to perform factual recall and when it is simply using its attention parameters to memorize the fact set\.
- •Evaluation \(fact\-adaptive\) accuracy\.In this variant, we train a Transformer using an MLPM​L​PAMLP\_\{A\}storing a given fact setAA, but during evaluation, we generate a new fact setBBand fact\-storing MLP,M​L​PBMLP\_\{B\}, that stores it\. We then insertM​L​PBMLP\_\{B\}into the pretrained Transformer, with no additional training, and evaluate the Transformer’s end\-to\-end accuracy on SSFR*using fact setBB*\. This criterion assesses whether the Transformer is performing factual recall by learning to query its fact\-storing MLP, while being prevented from introducing additional fact\-set\-dependent computation\.

In practice, we find that we need to make two changes to the standard GPT\-style architecture to ensure that Transformers trained on a given fact set will learn a fact\-adaptive solution: \(i\) we disable the residual connection after the attention layer to prevent extra signal propagation, and \(ii\) we freeze the value and output projections to identity matrices to hinder memorization\.

For the main\-text Transformer\-capacity plot in Figure[1](https://arxiv.org/html/2607.10034#S1.F1)c, we define Transformer capacity as the smallest model that achieves 100% fact\-adaptive accuracy\. We sweep

F∈\{26,27,…,213\}=\{64,128,256,512,1024,2048,4096,8192\}F\\in\\\{2^\{6\},2^\{7\},\\dots,2^\{13\}\\\}=\\\{64,128,256,512,1024,2048,4096,8192\\\}and binary\-search over hidden widthm∈\[1,65536\]m\\in\[1,65536\]with precision1616for each method\.

#### A\.4\.4Per\-Key Margin Diagnostic

The violin plot in[Figure7](https://arxiv.org/html/2607.10034#A1.F7)visualizes the full distribution of per\-key margins rather than only the minimum margin, as in the other margin sweeps\. For this experiment, we use the*pretrained attention*setting from Appendix[A\.1\.2](https://arxiv.org/html/2607.10034#A1.SS1.SSS2)\. We used=128d=128,F=2048F=2048,J=VJ=9J=V\_\{J\}=9,1616logarithmically spaced hidden widths in\[16,256\]\[16,256\], and four seeds\. For each width, we pool the per\-key margins across seeds and compare their distribution against the corresponding combined Transformer accuracy and standalone MLP accuracy\.

![Refer to caption](https://arxiv.org/html/2607.10034v1/sections/section_5_transformer_integration/figs/margin_violin.png)Figure 7:Per\-key margin distributions is predictable of end\-to\-end Transformer accuracy\.The usable\-key fraction inferred from per\-key margins is predictable of the combined attention\+MLP Transformer accuracy across the tested models\.
#### A\.4\.5Fact Editing

##### Language Modeling Task\.

We introduce a simple language modeling \(LM\) task to evaluate a Transformer’s ability to perform next\-token prediction while recalling factual information\. In this task, the model is presented with a natural\-language sentence expressing a\(book,author\)\(\\textit\{book\},\\textit\{author\}\)relation and is required to predict each subsequent token in the sequence\. We curate this dataset using author\-book relations from the Goodreads Book Graph Dataset\(Wan and McAuley,[2018](https://arxiv.org/html/2607.10034#bib.bib68)\)\.

Formally, letf:Sk→Svf:S\_\{k\}\\to S\_\{v\}be the authorsfact set, whereSk=\{“It”,“1984”,“And Then There Were None”,…\}S\_\{k\}=\\\{\\text\{\`\`It''\},\\ \\text\{\`\`1984''\},\\ \\text\{\`\`And Then There Were None''\},\\ \\ldots\\\}is the set of book titles \(keys\) andSv=\{“Stephen King”,“George Orwell”,“Agatha Christie”,…\}S\_\{v\}=\\\{\\text\{\`\`Stephen King''\},\\ \\text\{\`\`George Orwell''\},\\ \\text\{\`\`Agatha Christie''\},\\ \\ldots\\\}is the set of corresponding authors \(values\)\. To simplify analysis, we select exactly one book per author\. LetJ=\{\(“The author of”,“is”\),\(“Who is the author of”,“? It is”\),…\}J=\\\{\(\\text\{\`\`The author of''\},\\ \\text\{\`\`is''\}\),\\ \(\\text\{\`\`Who is the author of''\},\\ \\text\{\`\`? It is''\}\),\\ \\ldots\\\}denote the set of natural\-language template prefix–suffix pairs\. The LM task givenffcan then be defined as:

𝒮L​M​\[f\]=\{concat​\(tprefix,k,tsuffix,f​\(k\)\)\|\(tprefix,tsuffix\)∈J,k∈Sk\}\.\\mathcal\{S\}\_\{LM\}\[f\]=\\\{\\text\{concat\}\(t\_\{\\text\{prefix\}\},\\ k,\\ t\_\{\\text\{suffix\}\},f\(k\)\)\\ \|\\ \(t\_\{\\text\{prefix\}\},t\_\{\\text\{suffix\}\}\)\\in J,\\ k\\in S\_\{k\}\\\}\.
For example, given the sequence:

The author of⏟template prefix​1984⏟key​is⏟template suffix​George Orwell⏟value\\underbrace\{\\text\{The author of\}\}\_\{\\text\{template prefix\}\}\\ \\underbrace\{1984\}\_\{\\text\{key\}\}\\ \\underbrace\{\\text\{is\}\}\_\{\\text\{template suffix\}\}\\ \\underbrace\{\\text\{George Orwell\}\}\_\{\\text\{value\}\}from𝒮L​M​\[f\]\\mathcal\{S\}\_\{LM\}\[f\], the model’s task is to perform next\-token predictionat every positionin the sentence\. This LM task allows us to study factual recall in a more natural language modeling setting, complementing the SSFR setup\.

##### Model Parameterization\.

For this experiment, we use a one\-layer Transformer architecture with one head and the following non\-standard design choices\. We use this parameterization because it was the most realistic and simplest Transformer variant in our sweep that was able to achieve near\-perfect accuracy on the book–author task when trained with an inserted fact\-storing MLP\.

The model uses frozen tied input/output embeddings, no attention or MLP residual connection, RoPE positional encoding, freezes the RMSNorm before the MLP, uses an RMSNorm before attention and the language\-modeling head, and sets value and output projections of the attention layer frozen to identity matrices\. The model hidden dimension isdmodel=256d\_\{\\mathrm\{model\}\}=256\.

The attention layer keeps standard causal softmax attention, but uses learned nonlinear query and key projections\. Ifztz\_\{t\}is the attention\-normalized residual stream, then, omitting biases,

qt=WQ,2​GELU​\(WQ,1​LN​\(zt\)\),kt=WK,2​GELU​\(WK,1​LN​\(zt\)\),vt=zt,q\_\{t\}=W\_\{Q,2\}\\,\\mathrm\{GELU\}\(W\_\{Q,1\}\\,\\mathrm\{LN\}\(z\_\{t\}\)\),\\qquad k\_\{t\}=W\_\{K,2\}\\,\\mathrm\{GELU\}\(W\_\{K,1\}\\,\\mathrm\{LN\}\(z\_\{t\}\)\),\\qquad v\_\{t\}=z\_\{t\},and the attention output is

at=∑s≤tsoftmaxs​\(qt⊤​ksdmodel\)​vs,a\_\{t\}=\\sum\_\{s\\leq t\}\\mathrm\{softmax\}\_\{s\}\\\!\\left\(\\frac\{q\_\{t\}^\{\\top\}k\_\{s\}\}\{\\sqrt\{d\_\{\\mathrm\{model\}\}\}\}\\right\)v\_\{s\},with the output projection fixed to the identity\.

The feedforward layer is a two\-expert module\. Letxtx\_\{t\}denote the residual stream entering the feedforward block at token positiontt, and letx~t\\tilde\{x\}\_\{t\}be the normalized MLP input after the block’s pre\-MLP RMSNorm\. The feedforward output is

FFN​\(xt,x~t\)=αt​ffact​\(x~t\)\+\(1−αt\)​faux​\(xt\),αt=σ​\(g2​\(GELU​\(g1​\(xt\)\)\)\)\.\\mathrm\{FFN\}\(x\_\{t\},\\tilde\{x\}\_\{t\}\)=\\alpha\_\{t\}f\_\{\\mathrm\{fact\}\}\(\\tilde\{x\}\_\{t\}\)\+\(1\-\\alpha\_\{t\}\)f\_\{\\mathrm\{aux\}\}\(x\_\{t\}\),\\qquad\\alpha\_\{t\}=\\sigma\(g\_\{2\}\(\\mathrm\{GELU\}\(g\_\{1\}\(x\_\{t\}\)\)\)\)\.Hereffactf\_\{\\mathrm\{fact\}\}is the inserted fact\-storing MLP, held fixed during Transformer training, andfaux​\(x\)=x​U​Vf\_\{\\mathrm\{aux\}\}\(x\)=xUVis a trainable rank\-88low\-rank linear expert\. The router is a two\-layer scalar sigmoid MLP that forms a linear combination of the fact expert and auxiliary expert outputs\. Intuitively, our goal is for the model to learn to use the fact\-storing MLP to store the book–author relations and the auxiliary expert to learn the natural\-language sentence formats\.

Book titles and author names are added to the model tokenizer as atomic tokens before training\. For the fact MLP, the key embedding representing each book is the normalized version of the corresponding atomic book token\.

##### GD MLP Setup\.

In our fact\-editing experiment, we use GD\-trained fact\-storing MLPs \(see Appendix[A\.1\.1](https://arxiv.org/html/2607.10034#A1.SS1.SSS1)for the bilinear gated architecture and optimizer settings\), but we train the MLP with a MSE objective under arg\-max decoding:

LM​L​P​\(𝐊,𝐕,f\)∝∑i=1\|𝐊\|‖M​L​P​\(𝐤i\)−𝐯f​\(i\)‖22\.L\_\{MLP\}\(\\mathbf\{K\},\\mathbf\{V\},f\)\\propto\\sum\_\{i=1\}^\{\|\\mathbf\{K\}\|\}\\left\\lVert MLP\(\\mathbf\{k\}\_\{i\}\)\-\\mathbf\{v\}\_\{f\(i\)\}\\right\\rVert\_\{2\}^\{2\}\.The inserted fact expert is a gated MLP with hidden widthh=512h=512, trained by GD for up to10,00010\{,\}000epochs with learning rate10−310^\{\-3\}, minimum learning rate10−610^\{\-6\}, and early stopping once the objective falls below10−710^\{\-7\}\. Crucially, we train this MLP with MSE rather than cross entropy because MSE matches the full author\-value embedding vectors, not only the nearest\-token classifier\. We find that a parameter\-matched cross\-entropy GD MLP reaches100%100\\%fact classification accuracy when evaluated standalone, but when inserted into the Transformer the model reaches only about90%90\\%accuracy\.

##### Training Setup\.

We train on16,38416\{,\}384book\-author facts with1616rephrases per fact\. We initialize embeddings with Kaiming\-uniform initialization and train the Transformer with AdamW using learning rate2×10−42\\times 10^\{\-4\}, weight decay0\.10\.1, batch size3232,1818epochs, and81928192optimizer steps per epoch\. The trained base reaches near\-perfect standalone MLP accuracy0\.99980\.9998and Transformer value\-token accuracy0\.99840\.9984\.

##### Evaluation\.

We divide the16,38416\{,\}384stored facts into a*preserved*set whose answers should remain unchanged and an*altered*set whose answers are replaced by a fresh permutation of author values\. We edit328328,819819, or16381638facts, corresponding to2%2\\%,5%5\\%, and10%10\\%of the fact set\. For altered facts, we evaluate the new answer on both the edited training template and held\-out rephrases; for preserved facts, we evaluate the original answer\.

We report four fact\-editing quantities\.*Efficacy*is author\-token accuracy on altered facts under the edited labels\.*Paraphrase*is accuracy on held\-out rephrasings of the altered facts under the edited labels\.*Specificity*is accuracy on preserved facts under the original labels\.*Score*is the harmonic mean of efficacy, paraphrase, and specificity\. We additionally report the*non\-fact PPL ratio*:

rNF=PPLpost,NFPPLpre,NF=exp⁡\(CEpost,NF−CEpre,NF\),r\_\{\\mathrm\{NF\}\}=\\frac\{\\mathrm\{PPL\}\_\{\\mathrm\{post,NF\}\}\}\{\\mathrm\{PPL\}\_\{\\mathrm\{pre,NF\}\}\}=\\exp\\\!\\left\(\\mathrm\{CE\}\_\{\\mathrm\{post,NF\}\}\-\\mathrm\{CE\}\_\{\\mathrm\{pre,NF\}\}\\right\),where the CE is averaged over all next\-token positions except the author\-value tokens, across preserved and altered prompts before and after editing\. For example,rNF=1r\_\{\\mathrm\{NF\}\}=1means the edit leaves non\-fact\-token language\-modeling loss unchanged\.

##### Baselines\.

We compare four editing methods:

- •Our method,MLP Swapping, constructs a replacement GD fact expert for the complete post\-edit fact set and swaps that expert into the frozen Transformer, with no update to the surrounding Transformer\.
- •MEMIT\(Menget al\.,[2023c](https://arxiv.org/html/2607.10034#bib.bib24)\)applies a multi\-edit linear residual update to the fact expert’s output projection, using altered\-fact keys and a regularized solve\.
- •ROME\(Menget al\.,[2023b](https://arxiv.org/html/2607.10034#bib.bib23)\)applies a rank\-one residual update for each edit, adapted to our one\-layer setting and applied at the final prompt token\.
- •AlphaEdit\(Fanget al\.,[2025](https://arxiv.org/html/2607.10034#bib.bib55)\)first estimates a preserve\-key subspace and then projects the residual edit into the approximate nullspace of that preserve subspace before updating the fact expert\.

For the weight\-update editors, residuals are computed from a single templated prompt per fact and applied to the inserted MLP output immediately upstream of the logits\. We omit random prefix contexts and the ROME KL term because this synthetic dataset has a single relation and a unique author per book\.

For each weight\-update baseline and edit fraction, we sweep method\-specific hyperparameters and report the score\-best configuration\. AlphaEdit uses the best targeted grid settingtrain\_steps=100\\texttt\{train\\\_steps\}=100,lr=0\.05\\texttt\{lr\}=0\.05,clip\_norm=None\\texttt\{clip\\\_norm\}=\\text\{None\}, andsingular\_value\_tolerance=10\\texttt\{singular\\\_value\\\_tolerance\}=10at all three edit fractions\. MEMIT useslr=0\.05\\texttt\{lr\}=0\.05,λ=150\\lambda=150, andclip\_norm=0\.75\\texttt\{clip\\\_norm\}=0\.75, with100100steps at2%2\\%and5%5\\%edits and2525steps at10%10\\%edits\. ROME useslr=0\.05\\texttt\{lr\}=0\.05,wd=1\.5×10−3\\texttt\{wd\}=1\.5\\times 10^\{\-3\}, andearly\_stopping\_loss=5×10−2\\texttt\{early\\\_stopping\\\_loss\}=5\\times 10^\{\-2\}, with1010steps at2%2\\%edits and100100steps at5%5\\%and10%10\\%edits\.

MethodEfficacyParaphraseSpecificityScorerNFr\_\{\\mathrm\{NF\}\}2\.00%2\.00\\%edited factsMLP Swapping1\.0001\.0000\.9991\.000*1\.016*AlphaEdit*0\.896**0\.899*0\.981*0\.924*1\.000MEMIT0\.6430\.645*0\.990*0\.7291\.000ROME1\.0001\.0000\.0090\.0261\.2385\.00%5\.00\\%edited factsMLP Swapping0\.9980\.9980\.9980\.998*1\.057*AlphaEdit0\.7470\.7450\.909*0\.794*1\.001MEMIT0\.1280\.126*0\.986*0\.1791\.001ROME*0\.991**0\.991*0\.0020\.0061\.64210\.00%10\.00\\%edited factsMLP Swapping0\.9980\.9990\.9990\.9991\.021AlphaEdit0\.4820\.4820\.766*0\.550**1\.010*MEMIT0\.0040\.003*0\.995*0\.0051\.002ROME*0\.834**0\.832*0\.0010\.0032\.405Table 1:Fact\-editing metrics for baselines andMLP Swappingon the GD MLP base model\.rNFr\_\{\\mathrm\{NF\}\}is the non\-fact\-token perplexity ratio\. Bold marks the best value and italics mark the second\-best value within each edit fraction and metric; lower is better forrNFr\_\{\\mathrm\{NF\}\}, and higher is better for all other metrics\. Ties at the displayed precision share the same marking\.![Refer to caption](https://arxiv.org/html/2607.10034v1/x9.png)Figure 8:Non\-fact\-token perplexity ratio for the fact\-editing setup of Figure[3](https://arxiv.org/html/2607.10034#S5.F3)c\. AlphaEdit and MEMIT have nearly unchanged non\-fact loss but weaker edit scores at larger edit fractions; ROME edits target prompts while severely damaging non\-fact loss and specificity\.MLP Swappingkeeps near\-perfect edit scores with a small non\-fact PPL increase, at most1\.06×1\.06\\times\.
##### Fact editing with constructed MLPs\.

We also repeat the fact\-editing experiment from Figure[3](https://arxiv.org/html/2607.10034#S5.F3)c using the*data\-dependent Hebbian MLP construction*\. Because the data\-dependent construction has slightly worse storage capacity scaling than GD MLPs, we find we need to increase the hidden dimension toh=1024h=1024to achieve near\-perfect MLP and Transformer accuracy\. Other than this change, our experimental setup is the same as described above for GD MLPs\. We retune the method\-specific hyperparameters for each of the fact\-editing baselines\. We show thatMLP Swappingremains effective even with constructed Hebbian MLPs in Figure[9](https://arxiv.org/html/2607.10034#A1.F9)\. Swapping out ah=1024h=1024data\-dependent constructed MLP keeps the edit score above0\.980\.98through up to10%10\\%edited facts, while the strongest tuned local\-editing baseline reaches only0\.8470\.847at10%10\\%\. TheMLP Swappingnon\-fact PPL ratio remains below1\.11×1\.11\\times\.

![Refer to caption](https://arxiv.org/html/2607.10034v1/x10.png)

![Refer to caption](https://arxiv.org/html/2607.10034v1/x11.png)

Figure 9:Fact\-editing score \(top\) and non\-fact\-token perplexity ratio \(bottom\) for theh=1024h=1024data\-dependent constructed\-MLP setting\.MLP Swappingachieves an edit score above0\.980\.98through up to10%10\\%edited facts—13 percentage points higher than the next highest baseline—all while non\-fact PPL ratio remains below1\.11×1\.11\\times\.

### A\.5Additional Empirical Results

#### A\.5\.1Training\-Accuracy Transformer Capacity

In[Figure10](https://arxiv.org/html/2607.10034#A1.F10), we rerun the main\-text Transformer capacity experiment from Figure[1](https://arxiv.org/html/2607.10034#S1.F1)c, but with a 99%*training accuracy*criterion instead of fact\-adaptive accuracy \(as defined in[SectionA\.4\.3](https://arxiv.org/html/2607.10034#A1.SS4.SSS3)\)\. Unlike the evaluation accuracy plot, we fix the hidden dimensionm∈\{44,88,176,352,704,1408\}m\\in\\\{44,88,176,352,704,1408\\\}and binary\-search for the maximum number of factsF∈\[1,65536\]F\\in\[1,65536\]that the Transformer is able to store\.

![Refer to caption](https://arxiv.org/html/2607.10034v1/sections/section_5_transformer_integration/figs/transformer_capacity_scaling_train99.png)Figure 10:Transformer fact\-storage capacity under a 99% training accuracy criterion\.Transformer train\-accuracy capacity is22\-8×8\\timeshigher than fact\-adaptive capacity for our constructions and fixes the asymptotics for NTK – this suggests that attention parameters are learning to participate in storing the fact set\.Relative to the fact\-adaptive frontier in Figure[1](https://arxiv.org/html/2607.10034#S1.F1)c, the train\-side plot stores roughly22–8×8\\timesmore facts at a fixed parameter budget for GD MLPs and for all of our construction variants\. The gaps close the most for the weakest constructions, especially NTK and our unwhitened method; in particular, the suboptimal fixed\-ddasymptotics of the NTK construction are avoided under the train\-accuracy criterion\. This suggests that without the stricter fact\-adaptive criterion, the surrounding attention layer uses its parameters to help store the fact set, rather than solely relying upon the inserted fact\-storing MLP\.

#### A\.5\.2Anisotropic MLP Capacity

[Figure11](https://arxiv.org/html/2607.10034#A1.F11)uses the same MLP capacity protocol as Appendix[A\.3\.1](https://arxiv.org/html/2607.10034#A1.SS3.SSS1), but evaluates the different MLP families on*anisotropic*keys and values\. We apply the rank\-1 spike model from Appendix[A\.3\.2](https://arxiv.org/html/2607.10034#A1.SS3.SSS2.Px4)withβ=1\.5\\beta=1\.5\. We then estimate the resulting fact\-storage frontier ford∈\{64,90,128\}d\\in\\\{64,90,128\\\}\.

![Refer to caption](https://arxiv.org/html/2607.10034v1/sections/section_4_hebbian_kernel_mlps/figs/mlp_capacity_anisotropic_beta1p5_both_final.png)Figure 11:MLP fact\-storage capacity under anisotropic keys and values\.Our data\-dependent construction achieves asymptotically optimal fact\-storage capacity \(only44\-8×8\\timesworse than GD\), while NTK fails to achieve the desiredW=Θ​\(F​log⁡F\)W=\\Theta\(F\\log F\)scaling under anisotropy\.Although anisotropic embeddings degrade the attainable fact\-storage capacity for all methods relative to the isotropic setting \(Figure[2](https://arxiv.org/html/2607.10034#S4.F2)c\), our data\-dependent construction remains only44–8×8\\timesworse than GD, as in the isotropic case\. On the other hand, the NTK and unwhitened constructions struggle to store facts at theW=Θ​\(F​log⁡F\)W=\\Theta\(F\\log F\)scaling once both keys and values are anisotropic\. We find that whitening improves fact\-storage capacity substantially more in the anisotropic case\.

#### A\.5\.3Margin and Capacity Scaling Under LLM Embeddings

In this section, we explore the margin scaling and MLP and Transformer Block capacity scaling when using our construction on embeddings sampled from an intermediate LM layer\. Concretely, we replace the synthetic spherical key/value embeddings used in Sections[4](https://arxiv.org/html/2607.10034#S4)and[5](https://arxiv.org/html/2607.10034#S5)with paired\(𝐱,𝐲\)\(\\mathbf\{x\},\\mathbf\{y\}\)embeddings captured from a real language model\. The intent is to check that the margin bounds and the predicted storage\-capacity scaling remain meaningful when the keys and values come from those that an intermediate layer of an LM presents to its MLP blocks\.

##### LM embeddings\.

We stream the train split of WikiText through Qwen3\-0\.6B\-Base and, at the middle decoder block \(layer 14 of 28, the “mid layer”\), record per\-token MLP inputs𝐱\\mathbf\{x\}\(post\-RMSNorm hidden state\) and outputs𝐲\\mathbf\{y\}\(pre\-residual\), keepingN=500,000N=500\{,\}000pairs\. A factset of sizeFFis then built by uniformly samplingFFof these\(𝐱i,𝐲i\)\(\\mathbf\{x\}\_\{i\},\\mathbf\{y\}\_\{i\}\)pairs, taking𝐱i\\mathbf\{x\}\_\{i\}as the input and𝐲i\\mathbf\{y\}\_\{i\}as the output embedding under the identity mapping\.

##### Margin scaling\.

[Figure12](https://arxiv.org/html/2607.10034#A1.F12)shows the Section[4](https://arxiv.org/html/2607.10034#S4)margin scaling sweep on the captured Qwen3 mid\-layer embeddings\. We construct our bilinear random\-feature Hebbian variant withd=M=1024d=M=1024, sweepingFFfrom44to128128over five seeds per point\. In both the arbitrary\-keys/values and the isotropic regimes, the fitted bound tracks the empirical minimum marginγmin\\gamma\_\{\\min\}closely \(R2≥0\.95R^\{2\}\\geq 0\.95\), suggesting that our margin scaling laws continue to hold on real, anisotropic LM embeddings\.

![Refer to caption](https://arxiv.org/html/2607.10034v1/x12.png)

![Refer to caption](https://arxiv.org/html/2607.10034v1/x13.png)

Figure 12:Bilinear\-RF margin scaling on Qwen3\-0\.6B mid\-layer embeddings\.Left: arbitrary keys and values, where the measured margin tracks the deterministic bound through the key/value\-geometry termsKmindiagK\_\{\\min\}^\{\\mathrm\{diag\}\},EKE\_\{K\},EvE\_\{v\}and the composite cross\-talk scaleEK​Ev​κ\\sqrt\{E\_\{K\}\}\\sqrt\{E\_\{v\}\}\\,\\kappa\. Right: the isotropic\-key / isotropic\-value bound, where the margin follows the single cross\-talk termF​L/\(m​d\)\\sqrt\{FL/\(md\)\}\. In both regimes the bound tracks the empirically measured minimum marginγmin\\gamma\_\{\\min\}\(R2≥0\.95R^\{2\}\\geq 0\.95\), suggesting that our margin scaling laws hold for LM embeddings\.
##### MLP storage capacity\.

[Figure13](https://arxiv.org/html/2607.10034#A1.F13)reproduces the standalone\-MLP fact\-storage capacity scaling of Section[4](https://arxiv.org/html/2607.10034#S4), but with the random key/value embeddings replaced by the captured Qwen3 mid\-layer embeddings\. Following the protocol in Appendix[A\.3\.1](https://arxiv.org/html/2607.10034#A1.SS3.SSS1), we binary\-search over MLP hidden widthWWfor the minimum width that achieves a98%98\\%per\-fact recall threshold using our whitened bilinear\-RF Hebbian construction forF∈\{29,210,…,214\}F\\in\\\{2^\{9\},2^\{10\},\\dots,2^\{14\}\\\}\. Notably, the factsFFvs\. parametersWWscaling follows the predictedW≈Θ​\(F​log⁡F\)W\\approx\\Theta\(F\\log F\)capacity scaling on LM embeddings\.

![Refer to caption](https://arxiv.org/html/2607.10034v1/sections/section_4_hebbian_kernel_mlps/figs/mlp_capacity_llm_qwen3_06b_layer14_whitened.png)Figure 13:Standalone\-MLP fact\-storage capacity on Qwen3\-0\.6B mid\-layer embeddings\.Our whitened bilinear\-RF construction achieves the expectedW≈Θ​\(F​log⁡F\)W\\approx\\Theta\(F\\log F\)capacity scaling on LM embeddings\.
##### Transformer block storage capacity\.

For the Transformer block \([Figure14](https://arxiv.org/html/2607.10034#A1.F14)\), we use the same insert\-then\-train\-attention hidden\-width capacity sweep as the main\-text Figure[1](https://arxiv.org/html/2607.10034#S1.F1)c: a frozen whitened\-construction MLP is inserted into a single Transformer block whose input and output embeddings are the captured Qwen3𝐱\\mathbf\{x\}and𝐲\\mathbf\{y\}, and we binary\-search for the smallest hidden width reaching the success threshold\. Here, we require100%100\\%SSFR*training*accuracy on the trained fact set rather than fact\-adaptive evaluation accuracy\. Moreover, we keep the post\-attention residual \(rather than disabling it\), leave the value/output projections trainable \(rather than freezing them to identity\), and evaluate on the same inserted MLP \(rather than swapping in an eval\-MLP for a held\-out fact set\)\. Notably, as in the MLP case, the factsFFvs\. parametersWWscaling follows the predictedW≈Θ​\(F​log⁡F\)W\\approx\\Theta\(F\\log F\)capacity scaling on LM embeddings\.

![Refer to caption](https://arxiv.org/html/2607.10034v1/sections/section_5_transformer_integration/figs/transformer_capacity_llm_qwen3_06b_layer14.png)Figure 14:Transformer\-block fact\-storage capacity on Qwen3\-0\.6B mid\-layer embeddings \(training\-accuracy criterion\)\.Our whitened bilinear\-RF construction, inserted into a full Transformer block, retains the expectedW≈Θ​\(F​log⁡F\)W\\approx\\Theta\(F\\log F\)scaling on LM embeddings\.

## Appendix BTheory

### B\.1Information\-Theoretic Lower Bound on Fact Storage

We prove the counting lower bound used in[Theorem2\.4](https://arxiv.org/html/2607.10034#S2.Thmtheorem4)\.

###### Proof of[Theorem2\.4](https://arxiv.org/html/2607.10034#S2.Thmtheorem4)\.

Letbbbe the constant number of bits used to store each trainable parameter\. A model withWWtrainable parameters has at most2b​W2^\{bW\}distinct parameter settings, and therefore can realize at most2b​W2^\{bW\}distinct input\-output behaviors on the fixed key set𝐊\\mathbf\{K\}\.

On the other hand, the number of possible fact sets on𝐊\\mathbf\{K\}and𝐕\\mathbf\{V\}is

\|\{f:\[\|𝐊\|\]→\[\|𝐕\|\]\}\|=\|𝐕\|\|𝐊\|,\\left\|\\\{f:\[\|\\mathbf\{K\}\|\]\\to\[\|\\mathbf\{V\}\|\]\\\}\\right\|=\|\\mathbf\{V\}\|^\{\|\\mathbf\{K\}\|\},since each key can be assigned any value independently\.

If a model class𝐠\\mathbf\{g\}stores every fact set usingWWparameters, then the number of realizable behaviors must be at least the number of fact sets:

2b​W≥\|𝐕\|\|𝐊\|\.2^\{bW\}\\geq\|\\mathbf\{V\}\|^\{\|\\mathbf\{K\}\|\}\.Taking logarithms gives

b​W≥\|𝐊\|​log2⁡\|𝐕\|\.bW\\geq\|\\mathbf\{K\}\|\\log\_\{2\}\|\\mathbf\{V\}\|\.Becausebbis constant,

W≥\|𝐊\|​log2⁡\|𝐕\|b=Ω​\(\|𝐊\|​log⁡\|𝐕\|\)\.W\\geq\\frac\{\|\\mathbf\{K\}\|\\log\_\{2\}\|\\mathbf\{V\}\|\}\{b\}=\\Omega\\\!\\left\(\|\\mathbf\{K\}\|\\log\|\\mathbf\{V\}\|\\right\)\.∎

### B\.2Hebbian MLP Construction

#### B\.2\.1MLPs Are Hebbians

We formally restate and prove[Theorem3\.1](https://arxiv.org/html/2607.10034#S3.Thmtheorem1)\.

###### Theorem B\.1\(MLPs are Hebbians\)\.

Fix a feature mapϕ:ℝd→ℝm\\phi:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{m\}and𝐁∈ℝdv×m\\mathbf\{B\}\\in\\mathbb\{R\}^\{d\_\{v\}\\times m\}, and define the MLP

MLP​\(𝐱\):=𝐁​ϕ​\(𝐱\)\.\\mathrm\{MLP\}\(\\mathbf\{x\}\):=\\mathbf\{B\}\\phi\(\\mathbf\{x\}\)\.For the gated architecture used in this paper, the feature map is

ϕ​\(𝐱\)=\(𝐀𝐱\)⊙σ​\(𝐆𝐱\)\.\\phi\(\\mathbf\{x\}\)=\(\\mathbf\{A\}\\mathbf\{x\}\)\\odot\\sigma\(\\mathbf\{G\}\\mathbf\{x\}\)\.Given stored inputs𝐱1,…,𝐱F\\mathbf\{x\}\_\{1\},\\dots,\\mathbf\{x\}\_\{F\}, let𝐲i:=MLP​\(𝐱i\)\\mathbf\{y\}\_\{i\}:=\\mathrm\{MLP\}\(\\mathbf\{x\}\_\{i\}\)and define the empirical feature covariance

Σ^:=1F​∑i=1Fϕ​\(𝐱i\)​ϕ​\(𝐱i\)⊤\.\\hat\{\\Sigma\}:=\\frac\{1\}\{F\}\\sum\_\{i=1\}^\{F\}\\phi\(\\mathbf\{x\}\_\{i\}\)\\phi\(\\mathbf\{x\}\_\{i\}\)^\{\\top\}\.AssumeΣ^\\hat\{\\Sigma\}is invertible, and define the whitened kernel

K​\(𝐱,𝐳\):=ϕ​\(𝐱\)⊤​Σ^−1​ϕ​\(𝐳\)K\(\\mathbf\{x\},\\mathbf\{z\}\):=\\phi\(\\mathbf\{x\}\)^\{\\top\}\\hat\{\\Sigma\}^\{\-1\}\\phi\(\\mathbf\{z\}\)and the corresponding whitened kernel Hebbian memory

Hwhite​\(𝐳\):=1F​∑i=1F𝐲i​K​\(𝐱i,𝐳\)\.H\_\{\\mathrm\{white\}\}\(\\mathbf\{z\}\):=\\frac\{1\}\{F\}\\sum\_\{i=1\}^\{F\}\\mathbf\{y\}\_\{i\}\\,K\(\\mathbf\{x\}\_\{i\},\\mathbf\{z\}\)\.Then

Hwhite​\(𝐳\)=MLP​\(𝐳\)for all​𝐳∈ℝd\.H\_\{\\mathrm\{white\}\}\(\\mathbf\{z\}\)=\\mathrm\{MLP\}\(\\mathbf\{z\}\)\\qquad\\text\{for all \}\\mathbf\{z\}\\in\\mathbb\{R\}^\{d\}\.In other words, the MLP is exactly a kernel Hebbian memory with whitened kernelKK\.

###### Proof\.

Let

𝐖^:=1F​∑i=1F𝐲i​ϕ​\(𝐱i\)⊤\\hat\{\\mathbf\{W\}\}:=\\frac\{1\}\{F\}\\sum\_\{i=1\}^\{F\}\\mathbf\{y\}\_\{i\}\\phi\(\\mathbf\{x\}\_\{i\}\)^\{\\top\}denote the Hebbian weight matrix on the stored examples\. Since𝐲i=MLP​\(𝐱i\)=𝐁​ϕ​\(𝐱i\)\\mathbf\{y\}\_\{i\}=\\mathrm\{MLP\}\(\\mathbf\{x\}\_\{i\}\)=\\mathbf\{B\}\\phi\(\\mathbf\{x\}\_\{i\}\), we have

𝐖^=1F​∑i=1F𝐁​ϕ​\(𝐱i\)​ϕ​\(𝐱i\)⊤=𝐁​\(1F​∑i=1Fϕ​\(𝐱i\)​ϕ​\(𝐱i\)⊤\)=𝐁​Σ^\.\\hat\{\\mathbf\{W\}\}=\\frac\{1\}\{F\}\\sum\_\{i=1\}^\{F\}\\mathbf\{B\}\\phi\(\\mathbf\{x\}\_\{i\}\)\\phi\(\\mathbf\{x\}\_\{i\}\)^\{\\top\}=\\mathbf\{B\}\\left\(\\frac\{1\}\{F\}\\sum\_\{i=1\}^\{F\}\\phi\(\\mathbf\{x\}\_\{i\}\)\\phi\(\\mathbf\{x\}\_\{i\}\)^\{\\top\}\\right\)=\\mathbf\{B\}\\hat\{\\Sigma\}\.
Now expand the whitened kernel Hebbian memory:

Hwhite​\(𝐳\)\\displaystyle H\_\{\\mathrm\{white\}\}\(\\mathbf\{z\}\)=1F​∑i=1F𝐲i​K​\(𝐱i,𝐳\)\\displaystyle=\\frac\{1\}\{F\}\\sum\_\{i=1\}^\{F\}\\mathbf\{y\}\_\{i\}\\,K\(\\mathbf\{x\}\_\{i\},\\mathbf\{z\}\)=1F​∑i=1F𝐲i​ϕ​\(𝐱i\)⊤​Σ^−1​ϕ​\(𝐳\)\\displaystyle=\\frac\{1\}\{F\}\\sum\_\{i=1\}^\{F\}\\mathbf\{y\}\_\{i\}\\,\\phi\(\\mathbf\{x\}\_\{i\}\)^\{\\top\}\\hat\{\\Sigma\}^\{\-1\}\\phi\(\\mathbf\{z\}\)=\(1F​∑i=1F𝐲i​ϕ​\(𝐱i\)⊤\)​Σ^−1​ϕ​\(𝐳\)\\displaystyle=\\left\(\\frac\{1\}\{F\}\\sum\_\{i=1\}^\{F\}\\mathbf\{y\}\_\{i\}\\phi\(\\mathbf\{x\}\_\{i\}\)^\{\\top\}\\right\)\\hat\{\\Sigma\}^\{\-1\}\\phi\(\\mathbf\{z\}\)=𝐖^​Σ^−1​ϕ​\(𝐳\)\\displaystyle=\\hat\{\\mathbf\{W\}\}\\hat\{\\Sigma\}^\{\-1\}\\phi\(\\mathbf\{z\}\)=𝐁​Σ^​Σ^−1​ϕ​\(𝐳\)\\displaystyle=\\mathbf\{B\}\\hat\{\\Sigma\}\\hat\{\\Sigma\}^\{\-1\}\\phi\(\\mathbf\{z\}\)=𝐁​ϕ​\(𝐳\)\\displaystyle=\\mathbf\{B\}\\phi\(\\mathbf\{z\}\)=MLP​\(𝐳\)\.\\displaystyle=\\mathrm\{MLP\}\(\\mathbf\{z\}\)\.ThereforeHwhite​\(𝐳\)=MLP​\(𝐳\)H\_\{\\mathrm\{white\}\}\(\\mathbf\{z\}\)=\\mathrm\{MLP\}\(\\mathbf\{z\}\)for all𝐳∈ℝd\\mathbf\{z\}\\in\\mathbb\{R\}^\{d\}, so the MLP is exactly a kernel Hebbian memory with whitened kernelKK\. ∎

#### B\.2\.2Bilinear MLP Featurization Induces theK2K\_\{2\}Kernel

We first prove that our bilinear MLP construction converges to a Hebbian MLP with quadratic \(K2K\_\{2\}\) kernel when the number of random featuresmmscales asΘ​\(d2\)\\Theta\(d^\{2\}\):

###### Lemma B\.2\(Bilinear MLP featurization induces sketchedK2K\_\{2\}kernel\.\)\.

Let rows\(𝐀r,𝐆r\)r=1m\(\\mathbf\{A\}\_\{r\},\\mathbf\{G\}\_\{r\}\)\_\{r=1\}^\{m\}of matrices𝐀,𝐆\\mathbf\{A\},\\mathbf\{G\}be i\.i\.d\. standard Gaussian vectors inℝd\\mathbb\{R\}^\{d\}, and define the bilinear feature map

g​\(𝐱\):=1m​\(\(𝐀r⊤​𝐱\)​\(𝐆r⊤​𝐱\)\)r=1m∈ℝm\.g\(\\mathbf\{x\}\)\\vcentcolon=\\frac\{1\}\{\\sqrt\{m\}\}\\big\(\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\\big\)\_\{r=1\}^\{m\}\\in\\mathbb\{R\}^\{m\}\.Then the kernel induced by this feature map is

K^​\(𝐱,𝐳\):=⟨g​\(𝐱\),g​\(𝐳\)⟩=1m​∑r=1m\(𝐀r⊤​𝐱\)​\(𝐀r⊤​𝐳\)​\(𝐆r⊤​𝐱\)​\(𝐆r⊤​𝐳\)\.\\hat\{K\}\(\\mathbf\{x\},\\mathbf\{z\}\)\\vcentcolon=\\langle\{g\(\\mathbf\{x\}\)\},\{g\(\\mathbf\{z\}\)\}\\rangle=\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{z\}\)\\,\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{z\}\)\.and satisfies

𝔼​\[K^​\(𝐱,𝐳\)\]=⟨𝐱,𝐳⟩2=K2​\(𝐱,𝐳\),\\mathbb\{E\}\[\\hat\{K\}\(\\mathbf\{x\},\\mathbf\{z\}\)\]=\\langle\{\\mathbf\{x\}\},\{\\mathbf\{z\}\}\\rangle^\{2\}=K\_\{2\}\(\\mathbf\{x\},\\mathbf\{z\}\),soK^\\hat\{K\}is an unbiased random\-feature sketch of the exact quadratic kernelK2K\_\{2\}\.

###### Proof\.

The kernel identity follows by expanding the inner product of the feature vectors:

⟨g​\(𝐱\),g​\(𝐳\)⟩=1m​∑r=1m\(𝐀r⊤​𝐱\)​\(𝐆r⊤​𝐱\)​\(𝐀r⊤​𝐳\)​\(𝐆r⊤​𝐳\)=1m​∑r=1m\(𝐀r⊤​𝐱\)​\(𝐀r⊤​𝐳\)​\(𝐆r⊤​𝐱\)​\(𝐆r⊤​𝐳\)\.\\langle\{g\(\\mathbf\{x\}\)\},\{g\(\\mathbf\{z\}\)\}\\rangle=\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{z\}\)\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{z\}\)=\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{z\}\)\\,\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{z\}\)\.
For the expectation, the pairs\(𝐀r,𝐆r\)\(\\mathbf\{A\}\_\{r\},\\mathbf\{G\}\_\{r\}\)are i\.i\.d\., so it suffices to compute one summand\. Using independence of𝐀r\\mathbf\{A\}\_\{r\}and𝐆r\\mathbf\{G\}\_\{r\}and the Gaussian covariance identity,

𝔼​\[\(𝐀r⊤​𝐱\)​\(𝐀r⊤​𝐳\)\]=⟨𝐱,𝐳⟩,𝔼​\[\(𝐆r⊤​𝐱\)​\(𝐆r⊤​𝐳\)\]=⟨𝐱,𝐳⟩\.\\mathbb\{E\}\[\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{z\}\)\]=\\langle\{\\mathbf\{x\}\},\{\\mathbf\{z\}\}\\rangle,\\qquad\\mathbb\{E\}\[\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{z\}\)\]=\\langle\{\\mathbf\{x\}\},\{\\mathbf\{z\}\}\\rangle\.Therefore

𝔼​\[K^​\(𝐱,𝐳\)\]=𝔼​\[\(𝐚1⊤​𝐱\)​\(𝐚1⊤​𝐳\)\]​𝔼​\[\(𝐛1⊤​𝐱\)​\(𝐛1⊤​𝐳\)\]=⟨𝐱,𝐳⟩2\.\\mathbb\{E\}\[\\hat\{K\}\(\\mathbf\{x\},\\mathbf\{z\}\)\]=\\mathbb\{E\}\[\(\{\\mathbf\{a\}\}\_\{1\}^\{\\top\}\\mathbf\{x\}\)\(\{\\mathbf\{a\}\}\_\{1\}^\{\\top\}\\mathbf\{z\}\)\]\\,\\mathbb\{E\}\[\(\{\\mathbf\{b\}\}\_\{1\}^\{\\top\}\\mathbf\{x\}\)\(\{\\mathbf\{b\}\}\_\{1\}^\{\\top\}\\mathbf\{z\}\)\]=\\langle\{\\mathbf\{x\}\},\{\\mathbf\{z\}\}\\rangle^\{2\}\.∎

For fixed𝐱,𝐳\\mathbf\{x\},\\mathbf\{z\}, standard concentration bounds for Gaussian chaos terms\(Vershynin,[2018](https://arxiv.org/html/2607.10034#bib.bib48)\)imply that the sketching error scales as\|K^​\(𝐱,𝐳\)−K2​\(𝐱,𝐳\)\|=O​\(m−1/2\)\\bigl\|\\hat\{K\}\(\\mathbf\{x\},\\mathbf\{z\}\)\-K\_\{2\}\(\\mathbf\{x\},\\mathbf\{z\}\)\\bigr\|=O\(m^\{\-1/2\}\)with high probability\. In Appendix[B\.8\.4](https://arxiv.org/html/2607.10034#A2.SS8.SSS4), we show that exact off\-diagonalK2K\_\{2\}entries have sizeΘ​\(1/d\)\\Theta\(1/d\)with high probability in the isotropic keys and values setting; these off\-diagonal terms are precisely what drive the cross\-talk contribution in the margin decomposition\. Consequently, takingm≳d2m\\gtrsim d^\{2\}suffices to make the sketching noise smaller than the natural off\-diagonalK2K\_\{2\}scale, so that the sketching makes the margin no worse than the margin with the exact\-K2K\_\{2\}kernel\.

#### B\.2\.3Sketched\-K2K\_\{2\}Construction

[Algorithm1](https://arxiv.org/html/2607.10034#alg1)provides a pseudocode implementation of our the full bilinear MLP construction used in our experiments\.

Algorithm 1Sketched\-K2K\_\{2\}Hebbian Construction with Optional Whitening0:Keys

\{𝐤i\}i=1F⊂ℝd\\\{\\mathbf\{k\}\_\{i\}\\\}\_\{i=1\}^\{F\}\\subset\\mathbb\{R\}^\{d\}, value embeddings

\{𝐯j\}j=1F⊂ℝd\\\{\\mathbf\{v\}\_\{j\}\\\}\_\{j=1\}^\{F\}\\subset\\mathbb\{R\}^\{d\}, mapping

f:\[F\]→\[F\]f:\[F\]\\to\[F\]
0:Codes

\{𝐜j\}j=1F⊂ℝdc\\\{\\mathbf\{c\}\_\{j\}\\\}\_\{j=1\}^\{F\}\\subset\\mathbb\{R\}^\{d\_\{c\}\}\(default

𝐜j=𝐯j\\mathbf\{c\}\_\{j\}=\\mathbf\{v\}\_\{j\}\), feature width

mm
0:Optional whitening: mode

∈\{none,diag,full\}\\in\\\{\\texttt\{none\},\\texttt\{diag\},\\texttt\{full\}\\\}\(defaultnone\), ridge

λ\>0\\lambda\>0
0:Feature map

gg, readout

𝐁∈ℝdc×m\\mathbf\{B\}\\in\\mathbb\{R\}^\{d\_\{c\}\\times m\}, predictor

𝐜^​\(𝐱\)=𝐁​g​\(𝐱\)\\hat\{\\mathbf\{c\}\}\(\\mathbf\{x\}\)=\\mathbf\{B\}\\,g\(\\mathbf\{x\}\)
1:Sample rows

𝐀r,𝐆r∼i\.i\.d\.𝒩​\(0,Id\)\\mathbf\{A\}\_\{r\},\\mathbf\{G\}\_\{r\}\\stackrel\{\{\\scriptstyle i\.i\.d\.\}\}\{\{\\sim\}\}\\mathcal\{N\}\(0,I\_\{d\}\)for

r=1,…,mr=1,\\dots,m
2:Define

g​\(𝐱\)=1m​\(\(𝐀r⊤​𝐱\)​\(𝐆r⊤​𝐱\)\)r=1mg\(\\mathbf\{x\}\)=\\frac\{1\}\{\\sqrt\{m\}\}\\big\(\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\\big\)\_\{r=1\}^\{m\}
3:Form

𝐂f∈ℝF×dc\\mathbf\{C\}\_\{f\}\\in\\mathbb\{R\}^\{F\\times d\_\{c\}\}with row

iiequal to

𝐜f​\(i\)⊤\\mathbf\{c\}\_\{f\(i\)\}^\{\\top\}
4:Form

𝚽∈ℝF×m\{\\bm\{\\Phi\}\}\\in\\mathbb\{R\}^\{F\\times m\}with row

iiequal to

g​\(𝐤i\)⊤g\(\\mathbf\{k\}\_\{i\}\)^\{\\top\}
5:Compute raw Hebbian readout

𝐁0←1F​𝐂f⊤​𝚽\\mathbf\{B\}\_\{0\}\\leftarrow\\frac\{1\}\{F\}\\mathbf\{C\}\_\{f\}^\{\\top\}\{\\bm\{\\Phi\}\}
6:ifmode isnonethen

7:

𝐁←𝐁0\\mathbf\{B\}\\leftarrow\\mathbf\{B\}\_\{0\}
8:elseifmode isdiagthen

9:

Σ←1F​𝚽⊤​𝚽\\Sigma\\leftarrow\\frac\{1\}\{F\}\{\\bm\{\\Phi\}\}^\{\\top\}\{\\bm\{\\Phi\}\}
10:for

r=1,…,mr=1,\\dots,mdo

11:

𝐁:,r←𝐁0,:,r/\(Σr​r\+λ\)\\mathbf\{B\}\_\{:,r\}\\leftarrow\\mathbf\{B\}\_\{0,:,r\}/\(\\Sigma\_\{rr\}\+\\lambda\)
12:endfor

13:elseifmode isfullthen

14:if

m≤Fm\\leq Fthen

15:

Σ←1F​𝚽⊤​𝚽\\Sigma\\leftarrow\\frac\{1\}\{F\}\{\\bm\{\\Phi\}\}^\{\\top\}\{\\bm\{\\Phi\}\}
16:

𝐁←𝐁0​\(Σ\+λ​Im\)−1\\mathbf\{B\}\\leftarrow\\mathbf\{B\}\_\{0\}\(\\Sigma\+\\lambda I\_\{m\}\)^\{\-1\}
17:else

18:

𝐁←𝐂f⊤​\(𝚽​𝚽⊤\+λ​IF\)−1​𝚽\\mathbf\{B\}\\leftarrow\\mathbf\{C\}\_\{f\}^\{\\top\}\(\{\\bm\{\\Phi\}\}\{\\bm\{\\Phi\}\}^\{\\top\}\+\\lambda I\_\{F\}\)^\{\-1\}\{\\bm\{\\Phi\}\}\{dual branch\}

19:endif

20:endif

21:Return

𝐜^​\(𝐱\)=𝐁​g​\(𝐱\)\\hat\{\\mathbf\{c\}\}\(\\mathbf\{x\}\)=\\mathbf\{B\}\\,g\(\\mathbf\{x\}\)and retrieval scores

sj​\(𝐱\)=⟨𝐯j,𝐜^​\(𝐱\)⟩s\_\{j\}\(\\mathbf\{x\}\)=\\langle\\mathbf\{v\}\_\{j\},\\hat\{\\mathbf\{c\}\}\(\\mathbf\{x\}\)\\rangle

#### B\.2\.4Kernel Whitening

Here, we describe the kernel whitening procedure used in[Section4](https://arxiv.org/html/2607.10034#S4)to improve our Hebbian MLP’s fact storage capacity\. For the sketched\-K2K\_\{2\}feature map, let

𝚽=\[g​\(𝐤1\)⊤⋮g​\(𝐤F\)⊤\]∈ℝF×m,𝐂f=\[𝐜f​\(1\)⊤⋮𝐜f​\(F\)⊤\]∈ℝF×dc,\{\\bm\{\\Phi\}\}=\\begin\{bmatrix\}g\(\{\\mathbf\{k\}\}\_\{1\}\)^\{\\top\}\\\\ \\vdots\\\\ g\(\{\\mathbf\{k\}\}\_\{F\}\)^\{\\top\}\\end\{bmatrix\}\\in\\mathbb\{R\}^\{F\\times m\},\\qquad\\mathbf\{C\}\_\{f\}=\\begin\{bmatrix\}\{\\mathbf\{c\}\}\_\{f\(1\)\}^\{\\top\}\\\\ \\vdots\\\\ \{\\mathbf\{c\}\}\_\{f\(F\)\}^\{\\top\}\\end\{bmatrix\}\\in\\mathbb\{R\}^\{F\\times d\_\{c\}\},withg​\(𝐱\)=1m​\(\(𝐀r⊤​𝐱\)​\(𝐆r⊤​𝐱\)\)r=1mg\(\\mathbf\{x\}\)=\\frac\{1\}\{\\sqrt\{m\}\}\\big\(\(\\mathbf\{A\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\\mathbf\{G\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\\big\)\_\{r=1\}^\{m\}as in[Theorem4\.3](https://arxiv.org/html/2607.10034#S4.Thmtheorem3)\. The raw Hebbian readout is

𝐁0=1F​𝐂f⊤​𝚽,𝚺^=1F​𝚽⊤​𝚽\.\\mathbf\{B\}\_\{0\}=\\frac\{1\}\{F\}\\mathbf\{C\}\_\{f\}^\{\\top\}\{\\bm\{\\Phi\}\},\\qquad\\hat\{\{\\bm\{\\Sigma\}\}\}=\\frac\{1\}\{F\}\{\\bm\{\\Phi\}\}^\{\\top\}\{\\bm\{\\Phi\}\}\.\(12\)
The whitened construction replaces the raw Hebbian readout by the full ridge\-whitened readout indexed byλ≥0\\lambda\\geq 0,

𝐁λ=𝐁0​\(𝚺^\+λ​Im\)−1\.\\mathbf\{B\}\_\{\\lambda\}\\;=\\;\\mathbf\{B\}\_\{0\}\(\\hat\{\{\\bm\{\\Sigma\}\}\}\+\\lambda I\_\{m\}\)^\{\-1\}\.\(13\)This rescales the bilinear features according to their empirical covariance, mitigating the feature imbalance that appears at finite width\. Unless stated otherwise, our construction usesλ=10−6\\lambda=10^\{\-6\}by default\.

Whenm\>nm\>n, we instead perform the corresponding dual solve for numerical stability:

𝐁λ=𝐂f⊤​\(𝚽​𝚽⊤\+λ​In\)−1​𝚽\.\\mathbf\{B\}\_\{\\lambda\}=\\mathbf\{C\}\_\{f\}^\{\\top\}\(\{\\bm\{\\Phi\}\}\{\\bm\{\\Phi\}\}^\{\\top\}\+\\lambda I\_\{n\}\)^\{\-1\}\{\\bm\{\\Phi\}\}\.\(14\)The primal and dual forms are equivalent up to scaling\.

##### Whitening to reduce key crowdingEKE\_\{K\}\.

Letg:ℝd→ℝmg:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{m\},𝚽∈ℝn×m\{\\bm\{\\Phi\}\}\\in\\mathbb\{R\}^\{n\\times m\}and𝚺^\\hat\{\{\\bm\{\\Sigma\}\}\}be as defined above\. The unwhitened Gram matrix on the stored keys is𝐊raw:=𝚽​𝚽⊤\\mathbf\{K\}\_\{\\mathrm\{raw\}\}:=\{\\bm\{\\Phi\}\}\{\\bm\{\\Phi\}\}^\{\\top\}\. Full whitening replaces this by the*preconditioned*Gram matrix

𝐊white:=𝚽​𝚺^−1​𝚽⊤=\(𝚽​𝚺^−1/2\)​\(𝚽​𝚺^−1/2\)⊤,\\mathbf\{K\}\_\{\\mathrm\{white\}\}\\;:=\\;\{\\bm\{\\Phi\}\}\\hat\{\{\\bm\{\\Sigma\}\}\}^\{\-1\}\{\\bm\{\\Phi\}\}^\{\\top\}\\;=\\;\\big\(\{\\bm\{\\Phi\}\}\\hat\{\{\\bm\{\\Sigma\}\}\}^\{\-1/2\}\\big\)\\big\(\{\\bm\{\\Phi\}\}\\hat\{\{\\bm\{\\Sigma\}\}\}^\{\-1/2\}\\big\)^\{\\top\},equivalently using whitened featuresg~​\(⋅\)=𝚺^−1/2​g​\(⋅\)\\tilde\{g\}\(\\cdot\)=\\hat\{\{\\bm\{\\Sigma\}\}\}^\{\-1/2\}g\(\\cdot\)\.

Recall the key\-crowding statistic:

EK​\(i\):=∑t≠iKt​i2,EK:=maxi⁡EK​\(i\)\.E\_\{K\}\(i\)\\;:=\\;\\sum\_\{t\\neq i\}K\_\{ti\}^\{2\},\\qquad E\_\{K\}\\;:=\\;\\max\_\{i\}E\_\{K\}\(i\)\.SinceEK​\(i\)≤∑tKt​i2E\_\{K\}\(i\)\\leq\\sum\_\{t\}K\_\{ti\}^\{2\}andmaxi⁡\(⋅\)≤∑i\(⋅\)\\max\_\{i\}\(\\cdot\)\\leq\\sum\_\{i\}\(\\cdot\), we always have

EK≤‖𝐊‖F2\.E\_\{K\}\\;\\leq\\;\\\|\\mathbf\{K\}\\\|\_\{F\}^\{2\}\.\(15\)
###### Lemma B\.3\(Whitening minimizes upper bound onEKE\_\{K\}\)\.

Assume𝚺^≻0\\hat\{\{\\bm\{\\Sigma\}\}\}\\succ 0\. For any PSD preconditioner𝐌⪰0\\mathbf\{M\}\\succeq 0, define𝐊𝐌:=𝚽​𝐌​𝚽⊤\\mathbf\{K\}\_\{\\mathbf\{M\}\}:=\{\\bm\{\\Phi\}\}\\mathbf\{M\}\{\\bm\{\\Phi\}\}^\{\\top\}\(i\.e\. using features𝐌1/2​g​\(⋅\)\\mathbf\{M\}^\{1/2\}g\(\\cdot\)\)\. Among all𝐌⪰0\\mathbf\{M\}\\succeq 0with fixed average self\-kernel

1F​tr​\(𝐊𝐌\)=tr​\(𝐌​𝚺^\)=m,\\frac\{1\}\{F\}\\mathrm\{tr\}\(\\mathbf\{K\}\_\{\\mathbf\{M\}\}\)\\;=\\;\\mathrm\{tr\}\(\\mathbf\{M\}\\hat\{\{\\bm\{\\Sigma\}\}\}\)\\;=\\;m,\(16\)the choice𝐌⋆=𝚺^−1\\mathbf\{M\}^\{\\star\}=\\hat\{\{\\bm\{\\Sigma\}\}\}^\{\-1\}\(full whitening\) minimizes‖𝐊𝐌‖F2\\\|\\mathbf\{K\}\_\{\\mathbf\{M\}\}\\\|\_\{F\}^\{2\}\. Consequently, by equation[15](https://arxiv.org/html/2607.10034#A2.E15), whitening minimizes an explicit upper bound onEKE\_\{K\}:

EK​\(𝐊white\)≤‖𝐊white‖F2=min𝐌⪰0:tr​\(𝐌​𝚺^\)=m⁡‖𝐊𝐌‖F2\.E\_\{K\}\(\\mathbf\{K\}\_\{\\mathrm\{white\}\}\)\\;\\leq\\;\\\|\\mathbf\{K\}\_\{\\mathrm\{white\}\}\\\|\_\{F\}^\{2\}\\;=\\;\\min\_\{\\begin\{subarray\}\{c\}\\mathbf\{M\}\\succeq 0:\\\\ \\mathrm\{tr\}\(\\mathbf\{M\}\\hat\{\{\\bm\{\\Sigma\}\}\}\)=m\\end\{subarray\}\}\\;\\\|\\mathbf\{K\}\_\{\\mathbf\{M\}\}\\\|\_\{F\}^\{2\}\.

###### Proof\.

Write𝚽¯:=𝚽/F\\bar\{\{\\bm\{\\Phi\}\}\}:=\{\\bm\{\\Phi\}\}/\\sqrt\{F\}so that𝚺^=𝚽¯⊤​𝚽¯\\hat\{\{\\bm\{\\Sigma\}\}\}=\\bar\{\{\\bm\{\\Phi\}\}\}^\{\\top\}\\bar\{\{\\bm\{\\Phi\}\}\}and𝐊𝐌=F​𝚽¯​𝐌​𝚽¯⊤\\mathbf\{K\}\_\{\\mathbf\{M\}\}=F\\,\\bar\{\{\\bm\{\\Phi\}\}\}\\mathbf\{M\}\\bar\{\{\\bm\{\\Phi\}\}\}^\{\\top\}\. Using‖𝐀𝐀⊤‖F=‖𝐀⊤​𝐀‖F\\\|\\mathbf\{A\}\\mathbf\{A\}^\{\\top\}\\\|\_\{F\}=\\\|\\mathbf\{A\}^\{\\top\}\\mathbf\{A\}\\\|\_\{F\}with𝐀=𝚽¯​𝐌1/2\\mathbf\{A\}=\\bar\{\{\\bm\{\\Phi\}\}\}\\mathbf\{M\}^\{1/2\},

‖𝐊𝐌‖F2=F2​‖𝚽¯​𝐌​𝚽¯⊤‖F2=F2​‖𝐌1/2​𝚺^​𝐌1/2‖F2\.\\\|\\mathbf\{K\}\_\{\\mathbf\{M\}\}\\\|\_\{F\}^\{2\}=F^\{2\}\\\|\\bar\{\{\\bm\{\\Phi\}\}\}\\mathbf\{M\}\\bar\{\{\\bm\{\\Phi\}\}\}^\{\\top\}\\\|\_\{F\}^\{2\}=F^\{2\}\\\|\\mathbf\{M\}^\{1/2\}\\hat\{\{\\bm\{\\Sigma\}\}\}\\mathbf\{M\}^\{1/2\}\\\|\_\{F\}^\{2\}\.Let𝐀:=𝚺^1/2​𝐌​𝚺^1/2⪰0\\mathbf\{A\}:=\\hat\{\{\\bm\{\\Sigma\}\}\}^\{1/2\}\\mathbf\{M\}\\hat\{\{\\bm\{\\Sigma\}\}\}^\{1/2\}\\succeq 0\. Thentr​\(𝐀\)=tr​\(𝐌​𝚺^\)=m\\mathrm\{tr\}\(\\mathbf\{A\}\)=\\mathrm\{tr\}\(\\mathbf\{M\}\\hat\{\{\\bm\{\\Sigma\}\}\}\)=mand‖𝐌1/2​𝚺^​𝐌1/2‖F=‖𝐀‖F\\\|\\mathbf\{M\}^\{1/2\}\\hat\{\{\\bm\{\\Sigma\}\}\}\\mathbf\{M\}^\{1/2\}\\\|\_\{F\}=\\\|\\mathbf\{A\}\\\|\_\{F\}, so minimizing‖𝐊𝐌‖F2\\\|\\mathbf\{K\}\_\{\\mathbf\{M\}\}\\\|\_\{F\}^\{2\}under equation[16](https://arxiv.org/html/2607.10034#A2.E16)is equivalent to minimizing‖𝐀‖F2\\\|\\mathbf\{A\}\\\|\_\{F\}^\{2\}subject to𝐀⪰0\\mathbf\{A\}\\succeq 0andtr​\(𝐀\)=m\\mathrm\{tr\}\(\\mathbf\{A\}\)=m\. If\{λr\}r=1m\\\{\\lambda\_\{r\}\\\}\_\{r=1\}^\{m\}are the eigenvalues of𝐀\\mathbf\{A\}, then‖𝐀‖F2=∑rλr2\\\|\\mathbf\{A\}\\\|\_\{F\}^\{2\}=\\sum\_\{r\}\\lambda\_\{r\}^\{2\}and∑rλr=m\\sum\_\{r\}\\lambda\_\{r\}=m\. By Cauchy–Schwarz,∑rλr2≥\(∑rλr\)2/m=m\\sum\_\{r\}\\lambda\_\{r\}^\{2\}\\geq\(\\sum\_\{r\}\\lambda\_\{r\}\)^\{2\}/m=m, with equality iff allλr=1\\lambda\_\{r\}=1, i\.e\.𝐀=Im\\mathbf\{A\}=I\_\{m\}\. Thus𝚺^1/2​𝐌​𝚺^1/2=Im\\hat\{\{\\bm\{\\Sigma\}\}\}^\{1/2\}\\mathbf\{M\}\\hat\{\{\\bm\{\\Sigma\}\}\}^\{1/2\}=I\_\{m\}, so𝐌=𝚺^−1\\mathbf\{M\}=\\hat\{\{\\bm\{\\Sigma\}\}\}^\{\-1\}\. ∎

#### B\.2\.5Data\-Dependent Construction

Our data\-dependent construction used in[Section4](https://arxiv.org/html/2607.10034#S4)keeps the same bilinear architecture as the sketched\-K2K\_\{2\}MLP, but refines the bilinear feature factors using the fact set and key/value embeddings\. We describe our procedure here\.

Let

𝐊∈ℝn×d,𝐂f∈ℝn×dc,\\mathbf\{K\}\\in\\mathbb\{R\}^\{n\\times d\},\\qquad\\mathbf\{C\}\_\{f\}\\in\\mathbb\{R\}^\{n\\times d\_\{c\}\},where the rows of𝐊\\mathbf\{K\}are the stored keys and the rows of𝐂f\\mathbf\{C\}\_\{f\}are the corresponding target codes\. For random feature matrices𝐀,𝐆∈ℝm×d\\mathbf\{A\},\\mathbf\{G\}\\in\\mathbb\{R\}^\{m\\times d\}, define

𝚽​\(𝐀,𝐆\):=\(𝐊𝐀⊤\)⊙\(𝐊𝐆⊤\)∈ℝn×m\.\{\\bm\{\\Phi\}\}\(\\mathbf\{A\},\\mathbf\{G\}\)\\;:=\\;\(\\mathbf\{K\}\\mathbf\{A\}^\{\\top\}\)\\odot\(\\mathbf\{K\}\\mathbf\{G\}^\{\\top\}\)\\in\\mathbb\{R\}^\{n\\times m\}\.We initialize𝐀0\\mathbf\{A\}\_\{0\}and𝐆0\\mathbf\{G\}\_\{0\}with the same random bilinear sketch used in the sketched\-K2K\_\{2\}construction, and form the corresponding Hebbian

𝐁0:=1n​𝐂f⊤​𝚽​\(𝐀0,𝐆0\)∈ℝdc×m\.\\mathbf\{B\}\_\{0\}\\;:=\\;\\frac\{1\}\{n\}\\mathbf\{C\}\_\{f\}^\{\\top\}\{\\bm\{\\Phi\}\}\(\\mathbf\{A\}\_\{0\},\\mathbf\{G\}\_\{0\}\)\\in\\mathbb\{R\}^\{d\_\{c\}\\times m\}\.
The data\-dependent kernel is obtained by performing two least\-squares solves:

𝐆1\\displaystyle\\mathbf\{G\}\_\{1\}∈argmin𝐆∈ℝm×d‖𝐂f−𝚽​\(𝐀0,𝐆\)​𝐁0⊤‖F2,\\displaystyle\\in\\operatorname\*\{argmin\}\_\{\\mathbf\{G\}\\in\\mathbb\{R\}^\{m\\times d\}\}\\bigl\\\|\\mathbf\{C\}\_\{f\}\-\{\\bm\{\\Phi\}\}\(\\mathbf\{A\}\_\{0\},\\mathbf\{G\}\)\\mathbf\{B\}\_\{0\}^\{\\top\}\\bigr\\\|\_\{F\}^\{2\},\(17\)𝐀1\\displaystyle\\mathbf\{A\}\_\{1\}∈argmin𝐀∈ℝm×d‖𝐂f−𝚽​\(𝐀,𝐆1\)​𝐁0⊤‖F2\.\\displaystyle\\in\\operatorname\*\{argmin\}\_\{\\mathbf\{A\}\\in\\mathbb\{R\}^\{m\\times d\}\}\\bigl\\\|\\mathbf\{C\}\_\{f\}\-\{\\bm\{\\Phi\}\}\(\\mathbf\{A\},\\mathbf\{G\}\_\{1\}\)\\mathbf\{B\}\_\{0\}^\{\\top\}\\bigr\\\|\_\{F\}^\{2\}\.\(18\)Note that each subproblem is linear because𝚽​\(𝐀,𝐆\)\{\\bm\{\\Phi\}\}\(\\mathbf\{A\},\\mathbf\{G\}\)is linear in either factor once the other is held fixed\.

After these two updates, we discard the intermediate readout𝐁0\\mathbf\{B\}\_\{0\}and form the learned feature matrix

𝚽1:=𝚽​\(𝐀1,𝐆1\)\.\{\\bm\{\\Phi\}\}\_\{1\}\\;:=\\;\{\\bm\{\\Phi\}\}\(\\mathbf\{A\}\_\{1\},\\mathbf\{G\}\_\{1\}\)\.We then replace𝐁0\\mathbf\{B\}\_\{0\}with the full ridge\-whitened readout from[Equation13](https://arxiv.org/html/2607.10034#A2.E13):

𝐁λ=1n​𝐂f⊤​𝚽1​\(1n​𝚽1⊤​𝚽1\+λ​Im\)−1\.\\mathbf\{B\}\_\{\\lambda\}\\;=\\;\\frac\{1\}\{n\}\\mathbf\{C\}\_\{f\}^\{\\top\}\{\\bm\{\\Phi\}\}\_\{1\}\\Bigl\(\\frac\{1\}\{n\}\{\\bm\{\\Phi\}\}\_\{1\}^\{\\top\}\{\\bm\{\\Phi\}\}\_\{1\}\+\\lambda I\_\{m\}\\Bigr\)^\{\-1\}\.

#### B\.2\.6Bit Complexity

We now extend the real\-valued parameter\-count statement of[Section4\.3](https://arxiv.org/html/2607.10034#S4.SS3)to a bounded\-precision*bit complexity*theorem\. The proof has three steps:\(i\)positive margin implies robustness to output perturbations,\(ii\)the bilinear MLP is Lipschitz in its parameters on the stored keys, and\(iii\)sufficiently fine parameter quantization therefore preserves all margins\.

##### Setup\.

Recall from[Equation7](https://arxiv.org/html/2607.10034#S4.E7)that our bilinear MLP has the form

g𝜽​\(𝐱\)=𝐁​\(\(𝐀𝐱\)⊙\(𝐆𝐱\)\),𝜽=\(𝐀,𝐆,𝐁\),g\_\{\\bm\{\\theta\}\}\(\\mathbf\{x\}\)=\\mathbf\{B\}\\bigl\(\(\\mathbf\{A\}\\mathbf\{x\}\)\\odot\(\\mathbf\{G\}\\mathbf\{x\}\)\\bigr\),\\qquad\\bm\{\\theta\}=\(\\mathbf\{A\},\\mathbf\{G\},\\mathbf\{B\}\),\(19\)with𝐀,𝐆∈ℝm×d\\mathbf\{A\},\\mathbf\{G\}\\in\\mathbb\{R\}^\{m\\times d\}and𝐁∈ℝd×m\\mathbf\{B\}\\in\\mathbb\{R\}^\{d\\times m\}\. LetP:=dim\(𝜽\)P:=\\dim\(\\bm\{\\theta\}\)denote the total number of scalar parameters, soP≍m​dP\\asymp mdup to an absolute constant factor\.

For a stored fact setf:\[F\]→\[\|V\|\]f:\[F\]\\to\[\|V\|\], define the minimum margin

γmin​\(𝜽\):=mini∈\[F\]⁡minj≠f​\(i\)⁡⟨g𝜽​\(𝐤i\),𝐯f​\(i\)−𝐯j⟩\.\\gamma\_\{\\min\}\(\\bm\{\\theta\}\):=\\min\_\{i\\in\[F\]\}\\min\_\{j\\neq f\(i\)\}\\bigl\\langle g\_\{\\bm\{\\theta\}\}\(\\mathbf\{k\}\_\{i\}\),\\,\\mathbf\{v\}\_\{f\(i\)\}\-\\mathbf\{v\}\_\{j\}\\bigr\\rangle\.We also write

RK:=maxi∈\[F\]⁡‖𝐤i‖2,RV:=maxa∈\[\|V\|\]⁡‖𝐯a‖2\.R\_\{K\}:=\\max\_\{i\\in\[F\]\}\\\|\\mathbf\{k\}\_\{i\}\\\|\_\{2\},\\qquad R\_\{V\}:=\\max\_\{a\\in\[\|V\|\]\}\\\|\\mathbf\{v\}\_\{a\}\\\|\_\{2\}\.Note that in the isotropic and unit\-norm key/value regime of[Section4\.1](https://arxiv.org/html/2607.10034#S4.SS1)and Appendix[B\.8\.4](https://arxiv.org/html/2607.10034#A2.SS8.SSS4), one hasRK≤1R\_\{K\}\\leq 1andRV≤1R\_\{V\}\\leq 1\.

###### Lemma B\.4\(Margin robustness under output perturbations\)\.

Let𝛉⋆\\bm\{\\theta\}^\{\\star\}be any parameter vector such thatγmin​\(𝛉⋆\)≥γ0\>0\\gamma\_\{\\min\}\(\\bm\{\\theta\}^\{\\star\}\)\\geq\\gamma\_\{0\}\>0\. Assume that for some𝛉~\\widetilde\{\\bm\{\\theta\}\},

maxi∈\[F\]⁡‖g𝜽~​\(𝐤i\)−g𝜽⋆​\(𝐤i\)‖2≤γ04​RV\.\\max\_\{i\\in\[F\]\}\\bigl\\\|g\_\{\\widetilde\{\\bm\{\\theta\}\}\}\(\\mathbf\{k\}\_\{i\}\)\-g\_\{\\bm\{\\theta\}^\{\\star\}\}\(\\mathbf\{k\}\_\{i\}\)\\bigr\\\|\_\{2\}\\leq\\frac\{\\gamma\_\{0\}\}\{4R\_\{V\}\}\.Then𝛉~\\widetilde\{\\bm\{\\theta\}\}stores the same fact set in the sense of[Section2\.1](https://arxiv.org/html/2607.10034#S2.SS1.SSS0.Px1), and in fact

γmin​\(𝜽~\)≥γ02\.\\gamma\_\{\\min\}\(\\widetilde\{\\bm\{\\theta\}\}\)\\geq\\frac\{\\gamma\_\{0\}\}\{2\}\.

###### Proof\.

Fix any stored key𝐤i\\mathbf\{k\}\_\{i\}and any competitorj≠f​\(i\)j\\neq f\(i\)\. Then

⟨g𝜽~​\(𝐤i\),𝐯f​\(i\)−𝐯j⟩\\displaystyle\\Bigl\\langle g\_\{\\widetilde\{\\bm\{\\theta\}\}\}\(\\mathbf\{k\}\_\{i\}\),\\,\\mathbf\{v\}\_\{f\(i\)\}\-\\mathbf\{v\}\_\{j\}\\Bigr\\rangle=⟨g𝜽⋆​\(𝐤i\),𝐯f​\(i\)−𝐯j⟩\+⟨g𝜽~​\(𝐤i\)−g𝜽⋆​\(𝐤i\),𝐯f​\(i\)−𝐯j⟩\\displaystyle=\\Bigl\\langle g\_\{\\bm\{\\theta\}^\{\\star\}\}\(\\mathbf\{k\}\_\{i\}\),\\,\\mathbf\{v\}\_\{f\(i\)\}\-\\mathbf\{v\}\_\{j\}\\Bigr\\rangle\+\\Bigl\\langle g\_\{\\widetilde\{\\bm\{\\theta\}\}\}\(\\mathbf\{k\}\_\{i\}\)\-g\_\{\\bm\{\\theta\}^\{\\star\}\}\(\\mathbf\{k\}\_\{i\}\),\\,\\mathbf\{v\}\_\{f\(i\)\}\-\\mathbf\{v\}\_\{j\}\\Bigr\\rangle≥γ0−‖g𝜽~​\(𝐤i\)−g𝜽⋆​\(𝐤i\)‖2​‖𝐯f​\(i\)−𝐯j‖2\\displaystyle\\geq\\gamma\_\{0\}\-\\bigl\\\|g\_\{\\widetilde\{\\bm\{\\theta\}\}\}\(\\mathbf\{k\}\_\{i\}\)\-g\_\{\\bm\{\\theta\}^\{\\star\}\}\(\\mathbf\{k\}\_\{i\}\)\\bigr\\\|\_\{2\}\\,\\\|\\mathbf\{v\}\_\{f\(i\)\}\-\\mathbf\{v\}\_\{j\}\\\|\_\{2\}≥γ0−γ04​RV​\(‖𝐯f​\(i\)‖2\+‖𝐯j‖2\)\\displaystyle\\geq\\gamma\_\{0\}\-\\frac\{\\gamma\_\{0\}\}\{4R\_\{V\}\}\\bigl\(\\\|\\mathbf\{v\}\_\{f\(i\)\}\\\|\_\{2\}\+\\\|\\mathbf\{v\}\_\{j\}\\\|\_\{2\}\\bigr\)≥γ0−γ04​RV​\(2​RV\)=γ02\.\\displaystyle\\geq\\gamma\_\{0\}\-\\frac\{\\gamma\_\{0\}\}\{4R\_\{V\}\}\(2R\_\{V\}\)=\\frac\{\\gamma\_\{0\}\}\{2\}\.Since this holds for everyiiand everyj≠f​\(i\)j\\neq f\(i\), we obtainγmin​\(𝜽~\)≥γ0/2\>0\\gamma\_\{\\min\}\(\\widetilde\{\\bm\{\\theta\}\}\)\\geq\\gamma\_\{0\}/2\>0, which implies that𝜽~\\widetilde\{\\bm\{\\theta\}\}stores the same fact set\. ∎

###### Lemma B\.5\(Bilinear MLPs are Lipschitz in their parameters on stored keys\)\.

Fix any𝐱∈ℝd\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\}\. Let𝛉=\(𝐀,𝐆,𝐁\)\\bm\{\\theta\}=\(\\mathbf\{A\},\\mathbf\{G\},\\mathbf\{B\}\)and𝛉′=\(𝐀′,𝐆′,𝐁′\)\\bm\{\\theta\}^\{\\prime\}=\(\\mathbf\{A\}^\{\\prime\},\\mathbf\{G\}^\{\\prime\},\\mathbf\{B\}^\{\\prime\}\), and define

M:=max⁡\{‖𝐀‖op,‖𝐆‖op,‖𝐁‖op,‖𝐀′‖op,‖𝐆′‖op,‖𝐁′‖op\}\.M:=\\max\\Bigl\\\{\\\|\\mathbf\{A\}\\\|\_\{\\mathrm\{op\}\},\\\|\\mathbf\{G\}\\\|\_\{\\mathrm\{op\}\},\\\|\\mathbf\{B\}\\\|\_\{\\mathrm\{op\}\},\\\|\\mathbf\{A\}^\{\\prime\}\\\|\_\{\\mathrm\{op\}\},\\\|\\mathbf\{G\}^\{\\prime\}\\\|\_\{\\mathrm\{op\}\},\\\|\\mathbf\{B\}^\{\\prime\}\\\|\_\{\\mathrm\{op\}\}\\Bigr\\\}\.Then

‖g𝜽​\(𝐱\)−g𝜽′​\(𝐱\)‖2≤3​M2​‖𝐱‖22​‖𝜽−𝜽′‖2,\\bigl\\\|g\_\{\\bm\{\\theta\}\}\(\\mathbf\{x\}\)\-g\_\{\\bm\{\\theta\}^\{\\prime\}\}\(\\mathbf\{x\}\)\\bigr\\\|\_\{2\}\\leq 3M^\{2\}\\\|\\mathbf\{x\}\\\|\_\{2\}^\{2\}\\\|\\bm\{\\theta\}\-\\bm\{\\theta\}^\{\\prime\}\\\|\_\{2\},where

‖𝜽−𝜽′‖22:=‖𝐀−𝐀′‖F2\+‖𝐆−𝐆′‖F2\+‖𝐁−𝐁′‖F2\.\\\|\\bm\{\\theta\}\-\\bm\{\\theta\}^\{\\prime\}\\\|\_\{2\}^\{2\}:=\\\|\\mathbf\{A\}\-\\mathbf\{A\}^\{\\prime\}\\\|\_\{F\}^\{2\}\+\\\|\\mathbf\{G\}\-\\mathbf\{G\}^\{\\prime\}\\\|\_\{F\}^\{2\}\+\\\|\\mathbf\{B\}\-\\mathbf\{B\}^\{\\prime\}\\\|\_\{F\}^\{2\}\.Consequently, on any parameter region on which the operator norms of𝐀,𝐆,𝐁\\mathbf\{A\},\\mathbf\{G\},\\mathbf\{B\}are uniformly bounded byMM, the map𝛉↦g𝛉​\(𝐤i\)\\bm\{\\theta\}\\mapsto g\_\{\\bm\{\\theta\}\}\(\\mathbf\{k\}\_\{i\}\)is3​M2​RK23M^\{2\}R\_\{K\}^\{2\}\-Lipschitz for every stored key𝐤i\\mathbf\{k\}\_\{i\}satisfying‖𝐤i‖2≤RK\\\|\\mathbf\{k\}\_\{i\}\\\|\_\{2\}\\leq R\_\{K\}\.

###### Proof\.

Write

g𝜽​\(𝐱\)−g𝜽′​\(𝐱\)\\displaystyle g\_\{\\bm\{\\theta\}\}\(\\mathbf\{x\}\)\-g\_\{\\bm\{\\theta\}^\{\\prime\}\}\(\\mathbf\{x\}\)=𝐁​\(𝐀𝐱⊙𝐆𝐱\)−𝐁′​\(𝐀′​𝐱⊙𝐆′​𝐱\)\\displaystyle=\\mathbf\{B\}\(\\mathbf\{A\}\\mathbf\{x\}\\odot\\mathbf\{G\}\\mathbf\{x\}\)\-\\mathbf\{B\}^\{\\prime\}\(\\mathbf\{A\}^\{\\prime\}\\mathbf\{x\}\\odot\\mathbf\{G\}^\{\\prime\}\\mathbf\{x\}\)=\(𝐁−𝐁′\)​\(𝐀𝐱⊙𝐆𝐱\)\+𝐁′​\(\(𝐀𝐱⊙𝐆𝐱\)−\(𝐀′​𝐱⊙𝐆′​𝐱\)\)\\displaystyle=\(\\mathbf\{B\}\-\\mathbf\{B\}^\{\\prime\}\)\(\\mathbf\{A\}\\mathbf\{x\}\\odot\\mathbf\{G\}\\mathbf\{x\}\)\+\\mathbf\{B\}^\{\\prime\}\\Bigl\(\(\\mathbf\{A\}\\mathbf\{x\}\\odot\\mathbf\{G\}\\mathbf\{x\}\)\-\(\\mathbf\{A\}^\{\\prime\}\\mathbf\{x\}\\odot\\mathbf\{G\}^\{\\prime\}\\mathbf\{x\}\)\\Bigr\)=\(𝐁−𝐁′\)​\(𝐀𝐱⊙𝐆𝐱\)\+𝐁′​\(\(\(𝐀−𝐀′\)​𝐱\)⊙𝐆𝐱\)\+𝐁′​\(𝐀′​𝐱⊙\(\(𝐆−𝐆′\)​𝐱\)\)\.\\displaystyle=\(\\mathbf\{B\}\-\\mathbf\{B\}^\{\\prime\}\)\(\\mathbf\{A\}\\mathbf\{x\}\\odot\\mathbf\{G\}\\mathbf\{x\}\)\+\\mathbf\{B\}^\{\\prime\}\\Bigl\(\(\(\\mathbf\{A\}\-\\mathbf\{A\}^\{\\prime\}\)\\mathbf\{x\}\)\\odot\\mathbf\{G\}\\mathbf\{x\}\\Bigr\)\+\\mathbf\{B\}^\{\\prime\}\\Bigl\(\\mathbf\{A\}^\{\\prime\}\\mathbf\{x\}\\odot\(\(\\mathbf\{G\}\-\\mathbf\{G\}^\{\\prime\}\)\\mathbf\{x\}\)\\Bigr\)\.Using‖𝐮⊙𝐯‖2≤‖𝐮‖2​‖𝐯‖∞≤‖𝐮‖2​‖𝐯‖2\\\|\\mathbf\{u\}\\odot\\mathbf\{v\}\\\|\_\{2\}\\leq\\\|\\mathbf\{u\}\\\|\_\{2\}\\\|\\mathbf\{v\}\\\|\_\{\\infty\}\\leq\\\|\\mathbf\{u\}\\\|\_\{2\}\\\|\\mathbf\{v\}\\\|\_\{2\}together with‖𝐓𝐱‖2≤‖𝐓‖op​‖𝐱‖2\\\|\\mathbf\{T\}\\mathbf\{x\}\\\|\_\{2\}\\leq\\\|\\mathbf\{T\}\\\|\_\{\\mathrm\{op\}\}\\\|\\mathbf\{x\}\\\|\_\{2\}gives

‖\(𝐁−𝐁′\)​\(𝐀𝐱⊙𝐆𝐱\)‖2\\displaystyle\\\|\(\\mathbf\{B\}\-\\mathbf\{B\}^\{\\prime\}\)\(\\mathbf\{A\}\\mathbf\{x\}\\odot\\mathbf\{G\}\\mathbf\{x\}\)\\\|\_\{2\}≤‖𝐁−𝐁′‖op​‖𝐀𝐱‖2​‖𝐆𝐱‖2≤M2​‖𝐱‖22​‖𝐁−𝐁′‖F,\\displaystyle\\leq\\\|\\mathbf\{B\}\-\\mathbf\{B\}^\{\\prime\}\\\|\_\{\\mathrm\{op\}\}\\\|\\mathbf\{A\}\\mathbf\{x\}\\\|\_\{2\}\\\|\\mathbf\{G\}\\mathbf\{x\}\\\|\_\{2\}\\leq M^\{2\}\\\|\\mathbf\{x\}\\\|\_\{2\}^\{2\}\\\|\\mathbf\{B\}\-\\mathbf\{B\}^\{\\prime\}\\\|\_\{F\},‖𝐁′​\(\(\(𝐀−𝐀′\)​𝐱\)⊙𝐆𝐱\)‖2\\displaystyle\\\|\\mathbf\{B\}^\{\\prime\}\(\(\(\\mathbf\{A\}\-\\mathbf\{A\}^\{\\prime\}\)\\mathbf\{x\}\)\\odot\\mathbf\{G\}\\mathbf\{x\}\)\\\|\_\{2\}≤‖𝐁′‖op​‖\(𝐀−𝐀′\)​𝐱‖2​‖𝐆𝐱‖2≤M2​‖𝐱‖22​‖𝐀−𝐀′‖F,\\displaystyle\\leq\\\|\\mathbf\{B\}^\{\\prime\}\\\|\_\{\\mathrm\{op\}\}\\\|\(\\mathbf\{A\}\-\\mathbf\{A\}^\{\\prime\}\)\\mathbf\{x\}\\\|\_\{2\}\\\|\\mathbf\{G\}\\mathbf\{x\}\\\|\_\{2\}\\leq M^\{2\}\\\|\\mathbf\{x\}\\\|\_\{2\}^\{2\}\\\|\\mathbf\{A\}\-\\mathbf\{A\}^\{\\prime\}\\\|\_\{F\},‖𝐁′​\(𝐀′​𝐱⊙\(\(𝐆−𝐆′\)​𝐱\)\)‖2\\displaystyle\\\|\\mathbf\{B\}^\{\\prime\}\(\\mathbf\{A\}^\{\\prime\}\\mathbf\{x\}\\odot\(\(\\mathbf\{G\}\-\\mathbf\{G\}^\{\\prime\}\)\\mathbf\{x\}\)\)\\\|\_\{2\}≤‖𝐁′‖op​‖𝐀′​𝐱‖2​‖\(𝐆−𝐆′\)​𝐱‖2≤M2​‖𝐱‖22​‖𝐆−𝐆′‖F\.\\displaystyle\\leq\\\|\\mathbf\{B\}^\{\\prime\}\\\|\_\{\\mathrm\{op\}\}\\\|\\mathbf\{A\}^\{\\prime\}\\mathbf\{x\}\\\|\_\{2\}\\\|\(\\mathbf\{G\}\-\\mathbf\{G\}^\{\\prime\}\)\\mathbf\{x\}\\\|\_\{2\}\\leq M^\{2\}\\\|\\mathbf\{x\}\\\|\_\{2\}^\{2\}\\\|\\mathbf\{G\}\-\\mathbf\{G\}^\{\\prime\}\\\|\_\{F\}\.Summing the three bounds and usinga\+b\+c≤3​a2\+b2\+c2a\+b\+c\\leq 3\\sqrt\{a^\{2\}\+b^\{2\}\+c^\{2\}\}yields

‖g𝜽​\(𝐱\)−g𝜽′​\(𝐱\)‖2≤3​M2​‖𝐱‖22​‖𝐀−𝐀′‖F2\+‖𝐆−𝐆′‖F2\+‖𝐁−𝐁′‖F2=3​M2​‖𝐱‖22​‖𝜽−𝜽′‖2\.\\\|g\_\{\\bm\{\\theta\}\}\(\\mathbf\{x\}\)\-g\_\{\\bm\{\\theta\}^\{\\prime\}\}\(\\mathbf\{x\}\)\\\|\_\{2\}\\leq 3M^\{2\}\\\|\\mathbf\{x\}\\\|\_\{2\}^\{2\}\\sqrt\{\\\|\\mathbf\{A\}\-\\mathbf\{A\}^\{\\prime\}\\\|\_\{F\}^\{2\}\+\\\|\\mathbf\{G\}\-\\mathbf\{G\}^\{\\prime\}\\\|\_\{F\}^\{2\}\+\\\|\\mathbf\{B\}\-\\mathbf\{B\}^\{\\prime\}\\\|\_\{F\}^\{2\}\}=3M^\{2\}\\\|\\mathbf\{x\}\\\|\_\{2\}^\{2\}\\\|\\bm\{\\theta\}\-\\bm\{\\theta\}^\{\\prime\}\\\|\_\{2\}\.∎

###### Theorem B\.6\(Bounded\-bit implementation\)\.

Fix a fact set and a real\-valued parameter vector𝛉⋆\\bm\{\\theta\}^\{\\star\}for a bilinear MLP of the form in[Equation19](https://arxiv.org/html/2607.10034#A2.E19)\. Assume:

1. \(i\)Positive margin: γmin​\(𝜽⋆\)≥γ0\>0\.\\gamma\_\{\\min\}\(\\bm\{\\theta\}^\{\\star\}\)\\geq\\gamma\_\{0\}\>0\.
2. \(ii\)Bounded dynamic range:every coordinate of𝜽⋆\\bm\{\\theta\}^\{\\star\}lies in\[−Crng,Crng\]\[\-C\_\{\\mathrm\{rng\}\},C\_\{\\mathrm\{rng\}\}\]\.
3. \(iii\)Parameter Lipschitzness on stored keys:there existsLθ\>0L\_\{\\theta\}\>0such that for all𝜽,𝜽′∈\[−Crng,Crng\]P\\bm\{\\theta\},\\bm\{\\theta\}^\{\\prime\}\\in\[\-C\_\{\\mathrm\{rng\}\},C\_\{\\mathrm\{rng\}\}\]^\{P\}, maxi∈\[F\]⁡‖g𝜽​\(𝐤i\)−g𝜽′​\(𝐤i\)‖2≤Lθ​‖𝜽−𝜽′‖2\.\\max\_\{i\\in\[F\]\}\\\|g\_\{\\bm\{\\theta\}\}\(\\mathbf\{k\}\_\{i\}\)\-g\_\{\\bm\{\\theta\}^\{\\prime\}\}\(\\mathbf\{k\}\_\{i\}\)\\\|\_\{2\}\\leq L\_\{\\theta\}\\\|\\bm\{\\theta\}\-\\bm\{\\theta\}^\{\\prime\}\\\|\_\{2\}\.\(20\)

Then there exists a floating\-point parameter vector𝛉~\\widetilde\{\\bm\{\\theta\}\}that stores the same fact set and satisfies

γmin​\(𝜽~\)≥γ02\.\\gamma\_\{\\min\}\(\\widetilde\{\\bm\{\\theta\}\}\)\\geq\\frac\{\\gamma\_\{0\}\}\{2\}\.Moreover, one may choose𝛉~\\widetilde\{\\bm\{\\theta\}\}by rounding each coordinate of𝛉⋆\\bm\{\\theta\}^\{\\star\}to a binary floating\-point representation withttmantissa bits, where

t=O​\(log⁡\(4​Crng​Lθ​RV​Pγ0\)\)\.t=O\\\!\\left\(\\log\\\!\\left\(\\frac\{4C\_\{\\mathrm\{rng\}\}L\_\{\\theta\}R\_\{V\}\\sqrt\{P\}\}\{\\gamma\_\{0\}\}\\right\)\\right\)\.In this case, each coordinate can be encoded using at most

O​\(log⁡log⁡Crng\+log⁡\(4​Crng​Lθ​RV​Pγ0\)\)O\\\!\\left\(\\log\\log C\_\{\\mathrm\{rng\}\}\+\\log\\\!\\left\(\\frac\{4C\_\{\\mathrm\{rng\}\}L\_\{\\theta\}R\_\{V\}\\sqrt\{P\}\}\{\\gamma\_\{0\}\}\\right\)\\right\)bits\. Hence, the total number of bits needed to encode the quantized parameter vector𝛉~\\widetilde\{\\bm\{\\theta\}\}is at most

Bits​\(𝜽~\)≤P⋅O​\(log⁡log⁡Crng\+log⁡\(4​Crng​Lθ​RV​Pγ0\)\)\.\\mathrm\{Bits\}\(\\widetilde\{\\bm\{\\theta\}\}\)\\leq P\\cdot O\\\!\\left\(\\log\\log C\_\{\\mathrm\{rng\}\}\+\\log\\\!\\left\(\\frac\{4C\_\{\\mathrm\{rng\}\}L\_\{\\theta\}R\_\{V\}\\sqrt\{P\}\}\{\\gamma\_\{0\}\}\\right\)\\right\)\.\(21\)

###### Proof\.

Let𝜽~\\widetilde\{\\bm\{\\theta\}\}be obtained by rounding each coordinate of𝜽⋆\\bm\{\\theta\}^\{\\star\}to a binary floating\-point representation withttmantissa bits\. For standard floating\-point rounding, each coordinate incurs relative error at mostO​\(2−t\)O\(2^\{\-t\}\), and hence absolute error at most

O​\(2−t\)​Crng\.O\(2^\{\-t\}\)\\,C\_\{\\mathrm\{rng\}\}\.Therefore

‖𝜽~−𝜽⋆‖2≤O​\(2−t\)​Crng​P\.\\\|\\widetilde\{\\bm\{\\theta\}\}\-\\bm\{\\theta\}^\{\\star\}\\\|\_\{2\}\\leq O\(2^\{\-t\}\)\\,C\_\{\\mathrm\{rng\}\}\\sqrt\{P\}\.Choosing

t=O​\(log⁡\(4​Crng​Lθ​RV​Pγ0\)\)t=O\\\!\\left\(\\log\\\!\\left\(\\frac\{4C\_\{\\mathrm\{rng\}\}L\_\{\\theta\}R\_\{V\}\\sqrt\{P\}\}\{\\gamma\_\{0\}\}\\right\)\\right\)ensures that

‖𝜽~−𝜽⋆‖2≤γ04​Lθ​RV\.\\\|\\widetilde\{\\bm\{\\theta\}\}\-\\bm\{\\theta\}^\{\\star\}\\\|\_\{2\}\\leq\\frac\{\\gamma\_\{0\}\}\{4L\_\{\\theta\}R\_\{V\}\}\.Applying the Lipschitz assumption \([Equation20](https://arxiv.org/html/2607.10034#A2.E20)\),

maxi∈\[F\]⁡‖g𝜽~​\(𝐤i\)−g𝜽⋆​\(𝐤i\)‖2≤Lθ​‖𝜽~−𝜽⋆‖2≤γ04​RV\.\\max\_\{i\\in\[F\]\}\\\|g\_\{\\widetilde\{\\bm\{\\theta\}\}\}\(\\mathbf\{k\}\_\{i\}\)\-g\_\{\\bm\{\\theta\}^\{\\star\}\}\(\\mathbf\{k\}\_\{i\}\)\\\|\_\{2\}\\leq L\_\{\\theta\}\\\|\\widetilde\{\\bm\{\\theta\}\}\-\\bm\{\\theta\}^\{\\star\}\\\|\_\{2\}\\leq\\frac\{\\gamma\_\{0\}\}\{4R\_\{V\}\}\.Lemma[B\.2\.6](https://arxiv.org/html/2607.10034#A2.SS2.SSS6.Px1)therefore implies that𝜽~\\widetilde\{\\bm\{\\theta\}\}stores the same fact set and thatγmin​\(𝜽~\)≥γ0/2\\gamma\_\{\\min\}\(\\widetilde\{\\bm\{\\theta\}\}\)\\geq\\gamma\_\{0\}/2\.

It remains to count bits\. Each nonzero floating\-point coordinate can be written in the form

where the mantissammis represented tottbits of precision\. Since every coordinate lies in\[−Crng,Crng\]\[\-C\_\{\\mathrm\{rng\}\},C\_\{\\mathrm\{rng\}\}\], the exponent satisfies\|e\|=O​\(log⁡Crng\)\|e\|=O\(\\log C\_\{\\mathrm\{rng\}\}\), and hence the exponent can be encoded usingO​\(log⁡log⁡Crng\)O\(\\log\\log C\_\{\\mathrm\{rng\}\}\)bits\. Thus each coordinate can be encoded using at most

O​\(log⁡log⁡Crng\+t\)=O​\(log⁡log⁡Crng\+log⁡\(4​Crng​Lθ​RV​Pγ0\)\)O\\\!\\left\(\\log\\log C\_\{\\mathrm\{rng\}\}\+t\\right\)=O\\\!\\left\(\\log\\log C\_\{\\mathrm\{rng\}\}\+\\log\\\!\\left\(\\frac\{4C\_\{\\mathrm\{rng\}\}L\_\{\\theta\}R\_\{V\}\\sqrt\{P\}\}\{\\gamma\_\{0\}\}\\right\)\\right\)bits\. Multiplying by thePPcoordinates yields[Equation21](https://arxiv.org/html/2607.10034#A2.E21)\. ∎

Next, we specialize the previous bit complexity theorem to our sketched bilinear MLP construction \([Algorithm1](https://arxiv.org/html/2607.10034#alg1)\) by bounding its dynamic range and Lipschitz constant\. In the isotropic unit\-norm regime of[Section4\.1](https://arxiv.org/html/2607.10034#S4.SS1)and Appendix[B\.8\.4](https://arxiv.org/html/2607.10034#A2.SS8.SSS4), they reduce toCrng,Lθ≤dO​\(1\)C\_\{\\mathrm\{rng\}\},L\_\{\\theta\}\\leq d^\{O\(1\)\}, yielding an explicit bit\-complexity bound\.

###### Proposition B\.7\(Dynamic range and stored\-key Lipschitzness for the unwhitened sketched\-K2K\_\{2\}construction\)\.

Let𝐚1,…,𝐚m,𝐛1,…,𝐛m∈ℝd\\mathbf\{a\}\_\{1\},\\dots,\\mathbf\{a\}\_\{m\},\\mathbf\{b\}\_\{1\},\\dots,\\mathbf\{b\}\_\{m\}\\in\\mathbb\{R\}^\{d\}be sampled i\.i\.d\. fromN​\(0,𝐈d\)N\(0,\\mathbf\{I\}\_\{d\}\), and consider the sketched\-K2K\_\{2\}feature map

g​\(𝐱\)=1m​\(\(𝐚r⊤​𝐱\)​\(𝐛r⊤​𝐱\)\)r=1m\.g\(\\mathbf\{x\}\)=\\frac\{1\}\{\\sqrt\{m\}\}\\bigl\(\(\\mathbf\{a\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\\mathbf\{b\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\\bigr\)\_\{r=1\}^\{m\}\.Let𝐀¯,𝐆¯∈ℝm×d\\bar\{\\mathbf\{A\}\},\\bar\{\\mathbf\{G\}\}\\in\\mathbb\{R\}^\{m\\times d\}be the matrices whose rows are𝐚r⊤\\mathbf\{a\}\_\{r\}^\{\\top\}and𝐛r⊤\\mathbf\{b\}\_\{r\}^\{\\top\}, respectively, and realizeggin bilinear MLP form by setting

𝐀=m−1/4​𝐀¯,𝐆=m−1/4​𝐆¯\.\\mathbf\{A\}=m^\{\-1/4\}\\bar\{\\mathbf\{A\}\},\\qquad\\mathbf\{G\}=m^\{\-1/4\}\\bar\{\\mathbf\{G\}\}\.Let the raw Hebbian readout be

𝐁0=1F​𝐂f⊤​𝚽,\\mathbf\{B\}\_\{0\}=\\frac\{1\}\{F\}\\mathbf\{C\}\_\{f\}^\{\\top\}\\bm\{\\Phi\},where the rows of𝚽∈ℝF×m\\bm\{\\Phi\}\\in\\mathbb\{R\}^\{F\\times m\}areg​\(𝐤i\)⊤g\(\\mathbf\{k\}\_\{i\}\)^\{\\top\}and the rows of𝐂f∈ℝF×d\\mathbf\{C\}\_\{f\}\\in\\mathbb\{R\}^\{F\\times d\}are the value embeddings𝐯f​\(i\)⊤\\mathbf\{v\}\_\{f\(i\)\}^\{\\top\}\.

Assume

m,F,δ−1≤dO​\(1\)\.m,\\;F,\\;\\delta^\{\-1\}\\leq d^\{O\(1\)\}\.Then, with probability at least1−δ1\-\\deltaover the draw of\{𝐚r,𝐛r\}r=1m\\\{\\mathbf\{a\}\_\{r\},\\mathbf\{b\}\_\{r\}\\\}\_\{r=1\}^\{m\}, the resulting bilinear MLP

𝐱↦𝐁0​\(\(𝐀𝐱\)⊙\(𝐆𝐱\)\)\\mathbf\{x\}\\mapsto\\mathbf\{B\}\_\{0\}\\bigl\(\(\\mathbf\{A\}\\mathbf\{x\}\)\\odot\(\\mathbf\{G\}\\mathbf\{x\}\)\\bigr\)has parameter dynamic range and stored\-key parameter Lipschitz constant bounded by

Crng,Lθ≤dO​\(1\)​poly⁡\(RK,RV\)\.C\_\{\\mathrm\{rng\}\},\\;L\_\{\\theta\}\\leq d^\{O\(1\)\}\\,\\operatorname\{poly\}\(R\_\{K\},R\_\{V\}\)\.where one may take

Crng≤dO​\(1\)​max⁡\{1,RV​RK2\},Lθ≤dO​\(1\)​RK2​max⁡\{1,RV2​RK4\}\.C\_\{\\mathrm\{rng\}\}\\leq d^\{O\(1\)\}\\max\\\{1,R\_\{V\}R\_\{K\}^\{2\}\\\},\\qquad L\_\{\\theta\}\\leq d^\{O\(1\)\}\\,R\_\{K\}^\{2\}\\max\\\{1,R\_\{V\}^\{2\}R\_\{K\}^\{4\}\\\}\.

###### Proof\.

Standard Gaussian random matrix bounds\(Vershynin,[2018](https://arxiv.org/html/2607.10034#bib.bib48)\)imply that with probability at least1−δ1\-\\deltaover the draw of\{𝐚r,𝐛r\}r=1m\\\{\\mathbf\{a\}\_\{r\},\\mathbf\{b\}\_\{r\}\\\}\_\{r=1\}^\{m\},

‖𝐀‖op,‖𝐆‖op≲m−1/4​\(m\+d\+log⁡\(1/δ\)\)\.\\\|\\mathbf\{A\}\\\|\_\{\\mathrm\{op\}\},\\\|\\mathbf\{G\}\\\|\_\{\\mathrm\{op\}\}\\lesssim m^\{\-1/4\}\\Bigl\(\\sqrt\{m\}\+\\sqrt\{d\}\+\\sqrt\{\\log\(1/\\delta\)\}\\Bigr\)\.Under the assumptionm,δ−1≤dO​\(1\)m,\\delta^\{\-1\}\\leq d^\{O\(1\)\}, it follows that

‖𝐀‖op,‖𝐆‖op≤dO​\(1\)\.\\\|\\mathbf\{A\}\\\|\_\{\\mathrm\{op\}\},\\\|\\mathbf\{G\}\\\|\_\{\\mathrm\{op\}\}\\leq d^\{O\(1\)\}\.
Next, since each value embedding has norm at mostRVR\_\{V\},

‖𝐂f‖op≤‖𝐂f‖F≤F​RV\.\\\|\\mathbf\{C\}\_\{f\}\\\|\_\{\\mathrm\{op\}\}\\leq\\\|\\mathbf\{C\}\_\{f\}\\\|\_\{F\}\\leq\\sqrt\{F\}\\,R\_\{V\}\.Also,

‖𝚽‖op≤‖𝚽‖F≤F​maxi∈\[F\]⁡‖g​\(𝐤i\)‖2\.\\\|\\bm\{\\Phi\}\\\|\_\{\\mathrm\{op\}\}\\leq\\\|\\bm\{\\Phi\}\\\|\_\{F\}\\leq\\sqrt\{F\}\\,\\max\_\{i\\in\[F\]\}\\\|g\(\\mathbf\{k\}\_\{i\}\)\\\|\_\{2\}\.For every stored key𝐤i\\mathbf\{k\}\_\{i\},

‖g​\(𝐤i\)‖2=‖\(𝐀𝐤i\)⊙\(𝐆𝐤i\)‖2≤‖𝐀𝐤i‖2​‖𝐆𝐤i‖2≤‖𝐀‖op​‖𝐆‖op​RK2\.\\\|g\(\\mathbf\{k\}\_\{i\}\)\\\|\_\{2\}=\\bigl\\\|\(\\mathbf\{A\}\\mathbf\{k\}\_\{i\}\)\\odot\(\\mathbf\{G\}\\mathbf\{k\}\_\{i\}\)\\bigr\\\|\_\{2\}\\leq\\\|\\mathbf\{A\}\\mathbf\{k\}\_\{i\}\\\|\_\{2\}\\,\\\|\\mathbf\{G\}\\mathbf\{k\}\_\{i\}\\\|\_\{2\}\\leq\\\|\\mathbf\{A\}\\\|\_\{\\mathrm\{op\}\}\\\|\\mathbf\{G\}\\\|\_\{\\mathrm\{op\}\}R\_\{K\}^\{2\}\.Therefore

‖𝚽‖op≤F​‖𝐀‖op​‖𝐆‖op​RK2,\\\|\\bm\{\\Phi\}\\\|\_\{\\mathrm\{op\}\}\\leq\\sqrt\{F\}\\,\\\|\\mathbf\{A\}\\\|\_\{\\mathrm\{op\}\}\\\|\\mathbf\{G\}\\\|\_\{\\mathrm\{op\}\}R\_\{K\}^\{2\},and hence

‖𝐁0‖op≤1F​‖𝐂f‖op​‖𝚽‖op≤RV​‖𝐀‖op​‖𝐆‖op​RK2≤dO​\(1\)​RV​RK2\\\|\\mathbf\{B\}\_\{0\}\\\|\_\{\\mathrm\{op\}\}\\leq\\frac\{1\}\{F\}\\\|\\mathbf\{C\}\_\{f\}\\\|\_\{\\mathrm\{op\}\}\\\|\\bm\{\\Phi\}\\\|\_\{\\mathrm\{op\}\}\\leq R\_\{V\}\\\|\\mathbf\{A\}\\\|\_\{\\mathrm\{op\}\}\\\|\\mathbf\{G\}\\\|\_\{\\mathrm\{op\}\}R\_\{K\}^\{2\}\\leq d^\{O\(1\)\}R\_\{V\}R\_\{K\}^\{2\}with probability at least1−δ1\-\\delta\.

Now define

M:=max⁡\{‖𝐀‖op,‖𝐆‖op,‖𝐁0‖op\}\.M:=\\max\\\{\\\|\\mathbf\{A\}\\\|\_\{\\mathrm\{op\}\},\\\|\\mathbf\{G\}\\\|\_\{\\mathrm\{op\}\},\\\|\\mathbf\{B\}\_\{0\}\\\|\_\{\\mathrm\{op\}\}\\\}\.From the previous bounds,

M≤dO​\(1\)​max⁡\{1,RV​RK2\}M\\leq d^\{O\(1\)\}\\max\\\{1,R\_\{V\}R\_\{K\}^\{2\}\\\}with probability at least1−δ1\-\\delta\.

To pass from operator norms to coordinate bounds, note that for any matrix𝐓\\mathbf\{T\},

\|Ta​b\|=\|𝐞a⊤​𝐓𝐞b\|≤‖𝐓‖op\.\|T\_\{ab\}\|=\|\\mathbf\{e\}\_\{a\}^\{\\top\}\\mathbf\{T\}\\mathbf\{e\}\_\{b\}\|\\leq\\\|\\mathbf\{T\}\\\|\_\{\\mathrm\{op\}\}\.Hence every scalar entry of𝐀,𝐆,𝐁0\\mathbf\{A\},\\mathbf\{G\},\\mathbf\{B\}\_\{0\}is bounded byMM, and so one may take

Crng≤M≤dO​\(1\)​max⁡\{1,RV​RK2\}\.C\_\{\\mathrm\{rng\}\}\\leq M\\leq d^\{O\(1\)\}\\max\\\{1,R\_\{V\}R\_\{K\}^\{2\}\\\}\.
Finally,[SectionB\.2\.6](https://arxiv.org/html/2607.10034#A2.SS2.SSS6.Px1)yields

Lθ≤3​M2​RK2≤dO​\(1\)​RK2​max⁡\{1,RV2​RK4\},L\_\{\\theta\}\\leq 3M^\{2\}R\_\{K\}^\{2\}\\leq d^\{O\(1\)\}\\,R\_\{K\}^\{2\}\\max\\\{1,R\_\{V\}^\{2\}R\_\{K\}^\{4\}\\\},which proves the claim\. ∎

###### Corollary B\.8\(Bit complexity in the isotropic regime for the unwhitened construction\)\.

Assume the isotropic key/value regime of[Section4\.3](https://arxiv.org/html/2607.10034#S4.SS3), and consider the sketched\-K2K\_\{2\}construction of[Algorithm1](https://arxiv.org/html/2607.10034#alg1)\. Choose the widthmmso that the corresponding real\-valued construction satisfies

P≍m​d≍F​log⁡\(F/δ\)P\\asymp md\\asymp F\\log\(F/\\delta\)and has constant slack margin

γmin​\(𝜽⋆\)≥c0\>0\\gamma\_\{\\min\}\(\\bm\{\\theta\}^\{\\star\}\)\\geq c\_\{0\}\>0with high probability\. Assume further thatF,δ−1≤dCF,\\delta^\{\-1\}\\leq d^\{C\}for an absolute constantCC\. Then with high probability the same fact set can be stored using

Bits​\(𝜽~\)=O​\(F​log⁡\(F/δ\)​log⁡d\)\.\\mathrm\{Bits\}\(\\widetilde\{\\bm\{\\theta\}\}\)=O\\\!\\Bigl\(F\\log\(F/\\delta\)\\log d\\Bigr\)\.\(22\)

###### Proof\.

By the isotropic real\-valued capacity result, the unwhitened construction storesFFfacts using

P≍m​d≍F​log⁡\(F/δ\)P\\asymp md\\asymp F\\log\(F/\\delta\)parameters, up to an absolute constant factor\. In the isotropic unit\-norm regime,RK,RV≤1R\_\{K\},R\_\{V\}\\leq 1\. SinceF≤dO​\(1\)F\\leq d^\{O\(1\)\},[SectionB\.2\.6](https://arxiv.org/html/2607.10034#A2.SS2.SSS6.Px1)yields

Crng,Lθ≤dO​\(1\)C\_\{\\mathrm\{rng\}\},L\_\{\\theta\}\\leq d^\{O\(1\)\}with high probability\. Applying[TheoremB\.6](https://arxiv.org/html/2607.10034#A2.Thmtheorem6),

Bits​\(𝜽~\)≤P​O​\(log⁡log⁡Crng\+log⁡\(4​Crng​Lθ​RV​Pc0\)\)\.\\mathrm\{Bits\}\(\\widetilde\{\\bm\{\\theta\}\}\)\\leq P\\,O\\\!\\left\(\\log\\log C\_\{\\mathrm\{rng\}\}\+\\log\\\!\\left\(\\frac\{4C\_\{\\mathrm\{rng\}\}L\_\{\\theta\}R\_\{V\}\\sqrt\{P\}\}\{c\_\{0\}\}\\right\)\\right\)\.SinceCrng,Lθ,P≤dO​\(1\)C\_\{\\mathrm\{rng\}\},L\_\{\\theta\},P\\leq d^\{O\(1\)\}under the standing assumptionF,δ−1≤dCF,\\delta^\{\-1\}\\leq d^\{C\}, the quantity inside the outerO​\(⋅\)O\(\\cdot\)isO​\(log⁡d\)O\(\\log d\)\. Combining this withP≍F​log⁡\(F/δ\)P\\asymp F\\log\(F/\\delta\)yields[Equation22](https://arxiv.org/html/2607.10034#A2.E22)\. ∎

### B\.3NTK Baseline

For completeness, we describe the NTK baseline we implement fromNichaniet al\.\([2024](https://arxiv.org/html/2607.10034#bib.bib40)\)\.

Let𝐊∈ℝF×d\\mathbf\{K\}\\in\\mathbb\{R\}^\{F\\times d\}be the matrix whoseiith row is𝐤i⊤\\mathbf\{k\}\_\{i\}^\{\\top\}, and let𝐂f∈ℝF×dc\\mathbf\{C\}\_\{f\}\\in\\mathbb\{R\}^\{F\\times d\_\{c\}\}be the matrix whoseiith row is𝐜f​\(i\)⊤\\mathbf\{c\}\_\{f\(i\)\}^\{\\top\}, where typically𝐜j=𝐯j\\mathbf\{c\}\_\{j\}=\\mathbf\{v\}\_\{j\}\. Given hidden widthmm, sample random gate directions𝐰r∼i\.i\.d\.𝒩​\(0,Id\)\\mathbf\{w\}\_\{r\}\\stackrel\{\{\\scriptstyle i\.i\.d\.\}\}\{\{\\sim\}\}\\mathcal\{N\}\(0,I\_\{d\}\)and random output directions𝐩r∈ℝdc\\mathbf\{p\}\_\{r\}\\in\\mathbb\{R\}^\{d\_\{c\}\}with‖𝐩r‖2=1\\\|\\mathbf\{p\}\_\{r\}\\\|\_\{2\}=1forr=1,…,mr=1,\\dots,m\. Writing𝐖gate∈ℝm×d\\mathbf\{W\}\_\{\\mathrm\{gate\}\}\\in\\mathbb\{R\}^\{m\\times d\}for the matrix with rows𝐰r⊤\\mathbf\{w\}\_\{r\}^\{\\top\}and𝐏=\[𝐩1,…,𝐩m\]∈ℝdc×m\\mathbf\{P\}=\[\\mathbf\{p\}\_\{1\},\\dots,\\mathbf\{p\}\_\{m\}\]\\in\\mathbb\{R\}^\{d\_\{c\}\\times m\}, define the degree\-11Hermite feature matrix

𝐇:=He1​\(𝐊𝐖gate⊤\)=𝐊𝐖gate⊤∈ℝF×m,\\mathbf\{H\}\\;:=\\;\\mathrm\{He\}\_\{1\}\(\\mathbf\{K\}\\mathbf\{W\}\_\{\\mathrm\{gate\}\}^\{\\top\}\)\\;=\\;\\mathbf\{K\}\\mathbf\{W\}\_\{\\mathrm\{gate\}\}^\{\\top\}\\in\\mathbb\{R\}^\{F\\times m\},whereHe1​\(t\)=t\\mathrm\{He\}\_\{1\}\(t\)=tis applied entrywise\. The up\-projection is then chosen as

𝐖up:=1m​\(𝐇⊙\(𝐂f​𝐏\)\)⊤​𝐊∈ℝm×d\.\\mathbf\{W\}\_\{\\mathrm\{up\}\}\\;:=\\;\\frac\{1\}\{m\}\\big\(\\mathbf\{H\}\\odot\(\\mathbf\{C\}\_\{f\}\\mathbf\{P\}\)\\big\)^\{\\top\}\\mathbf\{K\}\\in\\mathbb\{R\}^\{m\\times d\}\.
The NTK MLP construction is then

𝐜^NTK​\(𝐱\)=𝐏​\(σ​\(𝐖gate​𝐱\)⊙\(𝐖up​𝐱\)\)\.\\hat\{\\mathbf\{c\}\}\_\{\\mathrm\{NTK\}\}\(\\mathbf\{x\}\)\\;=\\;\\mathbf\{P\}\\Big\(\\sigma\(\\mathbf\{W\}\_\{\\mathrm\{gate\}\}\\mathbf\{x\}\)\\odot\(\\mathbf\{W\}\_\{\\mathrm\{up\}\}\\mathbf\{x\}\)\\Big\)\.Throughout this work, we takeσ=ReLU\\sigma=\\mathrm\{ReLU\}\.

Note thatNichaniet al\.\([2024](https://arxiv.org/html/2607.10034#bib.bib40)\)’s construction requires choosing a Hermite degreekk\. This choice plays a role analogous to our kernel choice: it determines which degree polynomial interactions are emphasized by the construction\. In the most favorable casek=1k=1, the fit uses the linear featuresHe1​\(𝐊𝐖gate⊤\)=𝐊𝐖gate⊤\\mathrm\{He\}\_\{1\}\(\\mathbf\{K\}\\mathbf\{W\}\_\{\\mathrm\{gate\}\}^\{\\top\}\)=\\mathbf\{K\}\\mathbf\{W\}\_\{\\mathrm\{gate\}\}^\{\\top\}, and the realized finite\-width model remains a gated bilinear MLP\. Thus, in expectation over the random features, thek=1k=1baseline captures quadratic interactions, like how our sketched bilinear construction approximates a quadratic kernel\. This result, and the fact that the capacity bounds inNichaniet al\.\([2024](https://arxiv.org/html/2607.10034#bib.bib40)\)degrade exponentially withkk, makes usingk=1k=1the fairest comparison to our sketched\-K2K\_\{2\}construction\. The main difference is that the NTK construction folds the target codes into the hidden coefficients through𝐂f​𝐏\\mathbf\{C\}\_\{f\}\\mathbf\{P\}, whereas our construction uses an explicit bilinear random\-feature map followed by a Hebbian readout\.

### B\.4Margin Bounds Setup

We lay out the setup we’ll use throughout the appendix to prove our margin bounds\.

### Stored items and kernel

We storeFFkey–value items\{\(𝐤t,𝐯t\)\}t=1F\\\{\(\{\\mathbf\{k\}\}\_\{t\},\{\\mathbf\{v\}\}\_\{t\}\)\\\}\_\{t=1\}^\{F\}with keys𝐤t∈ℝd\{\\mathbf\{k\}\}\_\{t\}\\in\\mathbb\{R\}^\{d\}and values𝐯t∈ℝd\{\\mathbf\{v\}\}\_\{t\}\\in\\mathbb\{R\}^\{d\}\. A feature mapϕ:ℝd→ℝp\\phi:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{p\}induces a kernelK:ℝd×ℝd→ℝK:\\mathbb\{R\}^\{d\}\\times\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}

𝐊^t​i:=K​\(𝐤t,𝐤i\)=⟨ϕ​\(𝐤t\),ϕ​\(𝐤i\)⟩\(t,i∈\[F\]\)\.\\hat\{\\mathbf\{K\}\}\_\{ti\}\\vcentcolon=K\(\{\\mathbf\{k\}\}\_\{t\},\{\\mathbf\{k\}\}\_\{i\}\)=\\langle\{\\phi\(\{\\mathbf\{k\}\}\_\{t\}\)\},\{\\phi\(\{\\mathbf\{k\}\}\_\{i\}\)\}\\rangle\\qquad\(t,i\\in\[F\]\)\.We write𝐊^∈ℝF×F\\hat\{\\mathbf\{K\}\}\\in\\mathbb\{R\}^\{F\\times F\}for the Gram matrix\.

### Codes and retrieval output

Each indext∈\[F\]t\\in\[F\]has an associated code vector𝐜t∈ℝd\{\\mathbf\{c\}\}\_\{t\}\\in\\mathbb\{R\}^\{d\}\. Given a stored query at indexii\(i\.e\. query key𝐤i\{\\mathbf\{k\}\}\_\{i\}\), the retrieval output is

𝐲i:=∑t=1F𝐜t​𝐊^t​i∈ℝd\.\{\\mathbf\{y\}\}\_\{i\}\\vcentcolon=\\sum\_\{t=1\}^\{F\}\{\\mathbf\{c\}\}\_\{t\}\\,\\hat\{\\mathbf\{K\}\}\_\{ti\}\\in\\mathbb\{R\}^\{d\}\.\(23\)Unless otherwise noted, for simplicity, we assume𝐜t=𝐯t\{\\mathbf\{c\}\}\_\{t\}=\{\\mathbf\{v\}\}\_\{t\}for all the upcoming theorems and their proofs\.

### Pairwise margin

For a stored indexiiand a competitorj≠ij\\neq i, we define the pairwise margin

γi​j:=⟨𝐯i−𝐯j,𝐲i⟩\.\\gamma\_\{ij\}\\vcentcolon=\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{y\}\}\_\{i\}\}\\rangle\.\(24\)We writeγmin:=mini≠j⁡γi​j\\gamma\_\{\\min\}\\vcentcolon=\\min\_\{i\\neq j\}\\gamma\_\{ij\}for the worst\-case \(minimum\) pairwise margin\. Expanding, we see that the pairwise margin decomposes into signal and cross\-talk components\.

γi​j=𝐊^i​i​⟨𝐯i−𝐯j,𝐜i⟩⏟signal\+∑t≠i𝐊^t​i​⟨𝐯i−𝐯j,𝐜t⟩⏟cross\-talk\.\\gamma\_\{ij\}=\\underbrace\{\\hat\{\\mathbf\{K\}\}\_\{ii\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{c\}\}\_\{i\}\}\\rangle\}\_\{\\text\{signal \}\}\+\\underbrace\{\\sum\_\{t\\neq i\}\\hat\{\\mathbf\{K\}\}\_\{ti\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{c\}\}\_\{t\}\}\\rangle\}\_\{\\text\{cross\-talk \}\}\.\(25\)

### Margin convention \(absorbing the competitor\)

For the pairwise marginγi​j\\gamma\_\{ij\}, the single termt=jt=jinside the cross\-talk sum in equation[25](https://arxiv.org/html/2607.10034#A2.E25)corresponds to the specific*competitor*item\. For simplicity of our proofs, in the rest of the appendix, we re\-define the signal and cross\-talk by absorbing the competitor term into the signal:

γi​j=s~i​j⏟signal\+z~i​j⏟cross\-talk\\gamma\_\{ij\}=\\underbrace\{\\widetilde\{s\}\_\{ij\}\}\_\{\\text\{signal\}\}\+\\underbrace\{\\widetilde\{z\}\_\{ij\}\}\_\{\\text\{cross\-talk\}\}\(26\)with

s~i​j:=𝐊^i​i​⟨𝐯i−𝐯j,𝐜i⟩\+𝐊^j​i​⟨𝐯i−𝐯j,𝐜j⟩,z~i​j:=∑t∉\{i,j\}𝐊^t​i​⟨𝐯i−𝐯j,𝐜t⟩\\widetilde\{s\}\_\{ij\}\\vcentcolon=\\hat\{\\mathbf\{K\}\}\_\{ii\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{c\}\}\_\{i\}\}\\rangle\+\\hat\{\\mathbf\{K\}\}\_\{ji\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{c\}\}\_\{j\}\}\\rangle,\\qquad\\widetilde\{z\}\_\{ij\}\\vcentcolon=\\sum\_\{t\\notin\\\{i,j\\\}\}\\hat\{\\mathbf\{K\}\}\_\{ti\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{c\}\}\_\{t\}\}\\rangle\(27\)

### B\.5Embedding Geometric Summary Statistics Definitions

We collect here the formal definitions of all geometric summary statistics that enter our margin bounds\. Throughout,𝐊^\\hat\{\\mathbf\{K\}\}denotes the kernel Gram matrix with entries𝐊^t​i=K​\(𝐤t,𝐤i\)\\hat\{\\mathbf\{K\}\}\_\{ti\}=K\(\{\\mathbf\{k\}\}\_\{t\},\{\\mathbf\{k\}\}\_\{i\}\), and𝐯1,…,𝐯F∈ℝd\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}\\in\\mathbb\{R\}^\{d\}are the stored value vectors with associated Hebbian codes𝐜1,…,𝐜F∈ℝd\{\\mathbf\{c\}\}\_\{1\},\\dots,\{\\mathbf\{c\}\}\_\{F\}\\in\\mathbb\{R\}^\{d\}\.

##### Key\-geometry statistics\.

###### Definition B\.9\(Diagonal and off\-diagonal kernel energies\)\.

Kmindiag:=mini∈\[F\]⁡𝐊^i​i,Kmaxoff:=maxi≠j⁡\|𝐊^i​j\|\.K\_\{\\min\}^\{\\mathrm\{diag\}\}\\;\\vcentcolon=\\;\\min\_\{i\\in\[F\]\}\\,\\hat\{\\mathbf\{K\}\}\_\{ii\},\\qquad K\_\{\\max\}^\{\\mathrm\{off\}\}\\;\\vcentcolon=\\;\\max\_\{i\\neq j\}\\,\|\\hat\{\\mathbf\{K\}\}\_\{ij\}\|\.KmindiagK\_\{\\min\}^\{\\mathrm\{diag\}\}is the minimum kernel self\-similarity;KmaxoffK\_\{\\max\}^\{\\mathrm\{off\}\}is the maximum off\-diagonal kernel entry\.

###### Definition B\.10\(Kernel column energy\)\.

EK:=maxi∈\[F\]​∑t≠i𝐊^t​i2\.E\_\{K\}\\;\\vcentcolon=\\;\\max\_\{i\\in\[F\]\}\\,\\sum\_\{t\\neq i\}\\hat\{\\mathbf\{K\}\}\_\{ti\}^\{2\}\.EKE\_\{K\}measures the worst\-case squaredℓ2\\ell\_\{2\}energy of an off\-diagonal kernel column, capturing key\-embedding crowding: it grows when keys cluster in embedding space and kernel overlaps are large\.

##### Value\-geometry statistics\.

###### Definition B\.11\(Value separability\)\.

Vmin:=mini≠j⁡⟨𝐯i−𝐯j,𝐯i⟩,Vmax:=maxi≠j⁡\|⟨𝐯i−𝐯j,𝐯j⟩\|\.V\_\{\\min\}\\;\\vcentcolon=\\;\\min\_\{i\\neq j\}\\,\\langle\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\},\\,\{\\mathbf\{v\}\}\_\{i\}\\rangle,\\qquad V\_\{\\max\}\\;\\vcentcolon=\\;\\max\_\{i\\neq j\}\\,\|\\langle\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\},\\,\{\\mathbf\{v\}\}\_\{j\}\\rangle\|\.VminV\_\{\\min\}is the signal\-side value separability floor;VmaxV\_\{\\max\}is the cross\-talk\-side value interaction ceiling\.

For each pairi≠ji\\neq j, let𝐘\(i​j\):=\(⟨𝐯i−𝐯j,𝐜t⟩\)t∉\{i,j\}∈ℝF−2\\mathbf\{Y\}^\{\(ij\)\}\\vcentcolon=\\bigl\(\\langle\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\},\\,\{\\mathbf\{c\}\}\_\{t\}\\rangle\\bigr\)\_\{t\\notin\\\{i,j\\\}\}\\in\\mathbb\{R\}^\{F\-2\}denote the vector of value\-difference inner products with all non\-target, non\-competitor codes, and let𝟏∈ℝF−2\{\\mathbf\{1\}\}\\in\\mathbb\{R\}^\{F\-2\}denote the all\-ones vector\.

###### Definition B\.12\(Mean competitor alignment\)\.

BY:=maxi≠j⁡\|∑t∉\{i,j\}⟨𝐯i−𝐯j,𝐜t⟩\|=maxi≠j⁡\|⟨𝟏,𝐘\(i​j\)⟩\|\.B\_\{Y\}\\vcentcolon=\\max\_\{i\\neq j\}\\left\|\\sum\_\{t\\notin\\\{i,j\\\}\}\\left\\langle\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\},\{\\mathbf\{c\}\}\_\{t\}\\right\\rangle\\right\|=\\max\_\{i\\neq j\}\\left\|\\left\\langle\{\\mathbf\{1\}\},\\mathbf\{Y\}^\{\(ij\)\}\\right\\rangle\\right\|\.BYB\_\{Y\}controls the bias contribution to cross\-talk arising from the mean of the kernel off\-diagonal entries \(equal to1/d1/dunder isotropic keys\)\.

###### Definition B\.13\(Value\-difference energy\)\.

Ev:=maxi∈\[F\]⁡maxj≠i​∑t≠i⟨𝐯i−𝐯j,𝐜t⟩2\.E\_\{v\}\\;\\vcentcolon=\\;\\max\_\{i\\in\[F\]\}\\max\_\{j\\neq i\}\\,\\sum\_\{t\\neq i\}\\langle\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\},\\,\{\\mathbf\{c\}\}\_\{t\}\\rangle^\{2\}\.EvE\_\{v\}measures the worst\-case squaredℓ2\\ell\_\{2\}energy of the value\-difference inner\-product vector, capturing value\-embedding interference: it grows when value embeddings cluster and non\-target codes align with the target direction\.

###### Definition B\.14\(Value sparsity\)\.

Recall𝐘\(i​j\)\\mathbf\{Y\}^\{\(ij\)\}from[SectionB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px2)\. Define

Lv:=maxi≠j⁡‖𝐘\(i​j\)‖12‖𝐘\(i​j\)‖22L\_\{v\}\\;\\vcentcolon=\\;\\max\_\{i\\neq j\}\\,\\frac\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{1\}^\{2\}\}\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}^\{2\}\}Lv∈\[1,F−2\]L\_\{v\}\\in\[1,F\-2\]is an effective sparsity parameter: it equals11when the energy of𝐘\(i​j\)\\mathbf\{Y\}^\{\(ij\)\}is concentrated on a single coordinate and equalsF−2F\-2when it is spread uniformly\. It amplifies cross\-talk when value\-difference inner products have heavy\-tailed distributions across competitors\.

##### Key–value coupling\.

###### Definition B\.15\(Coupling factor\)\.

For stored indexiiand competitorj≠ij\\neq i, letEK​\(i\):=∑t≠i𝐊^t​i2E\_\{K\}\(i\)\\vcentcolon=\\sum\_\{t\\neq i\}\\hat\{\\mathbf\{K\}\}\_\{ti\}^\{2\}andEv​\(i,j\):=∑t≠i⟨𝐯i−𝐯j,𝐜t⟩2E\_\{v\}\(i,j\)\\vcentcolon=\\sum\_\{t\\neq i\}\\langle\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\},\{\\mathbf\{c\}\}\_\{t\}\\rangle^\{2\}be the pairwise energies\. Define the pairwise coupling

κi​j:=\|∑t≠i𝐊^t​i​⟨𝐯i−𝐯j,𝐜t⟩\|EK​\(i\)​Ev​\(i,j\)∈\[0,1\],\\kappa^\{ij\}\\;\\vcentcolon=\\;\\frac\{\\displaystyle\\left\|\\sum\_\{t\\neq i\}\\hat\{\\mathbf\{K\}\}\_\{ti\}\\,\\langle\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\},\\,\{\\mathbf\{c\}\}\_\{t\}\\rangle\\right\|\}\{\\sqrt\{E\_\{K\}\(i\)\}\\,\\sqrt\{E\_\{v\}\(i,j\)\}\}\\;\\in\\;\[0,1\],κ:=maxi∈\[F\]⁡maxj≠i⁡κi​j\.\\kappa\\;\\vcentcolon=\\;\\max\_\{i\\in\[F\]\}\\,\\max\_\{j\\neq i\}\\;\\kappa^\{ij\}\.κ\\kappaquantifies the worst\-case alignment between the kernel column pattern and the value\-interference pattern:κ=0\\kappa=0when the two are orthogonal andκ=1\\kappa=1when they are perfectly aligned\.

### B\.6Cross\-Talk Bounds

We start by providing upper bounds for the cross\-talk termz~i​j\\widetilde\{z\}\_\{ij\}\.

##### Cross\-talk as inner product\.

First, we define cross\-talk as an inner product\. Concretely, fix\(i,j\)\(i,j\)withj≠ij\\neq iand define the off\-diagonal kernel column

𝐗\(i​j\):=\(𝐊^t​i\)t∉\{i,j\}∈ℝF−2\.\\mathbf\{X\}^\{\(ij\)\}\\vcentcolon=\\big\(\\hat\{\\mathbf\{K\}\}\_\{ti\}\\big\)\_\{t\\notin\\\{i,j\\\}\}\\in\\mathbb\{R\}^\{F\-2\}\.\(28\)And for the value/code side:

𝐘\(i​j\):=\(⟨𝐯i−𝐯j,𝐜t⟩\)t∉\{i,j\}∈ℝF−2,\\mathbf\{Y\}^\{\(ij\)\}\\vcentcolon=\\big\(\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{c\}\}\_\{t\}\}\\rangle\)\_\{t\\notin\\\{i,j\\\}\}\\in\\mathbb\{R\}^\{F\-2\},\(29\)Then the cross\-talk term is the inner product

z~i​j:=∑t∉\{i,j\}𝐊^t​i​⟨𝐯i−𝐯j,𝐜t⟩=⟨𝐗\(i​j\),𝐘\(i​j\)⟩\.\\widetilde\{z\}\_\{ij\}\\vcentcolon=\\sum\_\{t\\notin\\\{i,j\\\}\}\\hat\{\\mathbf\{K\}\}\_\{ti\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{c\}\}\_\{t\}\}\\rangle=\\langle\{\\mathbf\{X\}^\{\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\.\(30\)We consider the largest \(worst case\) possible cross\-talk:

z~max:=maxi≠j⁡\|z~i​j\|\.\\widetilde\{z\}\_\{\\max\}\\ \\vcentcolon=\\ \\max\_\{i\\neq j\}\|\\widetilde\{z\}\_\{ij\}\|\.\(31\)Note thatz~i​j≤z~max\\widetilde\{z\}\_\{ij\}\\leq\\widetilde\{z\}\_\{\\max\}for alli≠ji\\neq j\.

##### Cross\-talk summary statistics\.

For each\(i,j\)\(i,j\)define the \(squared\) energies

E~v​\(i,j\):=‖𝐘\(i​j\)‖22=∑t∉\{i,j\}⟨𝐯i−𝐯j,𝐜t⟩2,E~K​\(i,j\):=‖𝐗\(i​j\)‖22=∑t∉\{i,j\}𝐊^t​i2\.\\widetilde\{E\}\_\{v\}\(i,j\)\\vcentcolon=\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}^\{2\}=\\sum\_\{t\\notin\\\{i,j\\\}\}\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{c\}\}\_\{t\}\}\\rangle^\{2\},\\qquad\\widetilde\{E\}\_\{K\}\(i,j\)\\vcentcolon=\\\|\\mathbf\{X\}^\{\(ij\)\}\\\|\_\{2\}^\{2\}=\\sum\_\{t\\notin\\\{i,j\\\}\}\\hat\{\\mathbf\{K\}\}\_\{ti\}^\{2\}\.\(32\)Define the coupling factor

κ~i​j:=\|⟨𝐗\(i​j\),𝐘\(i​j\)⟩\|‖𝐗\(i​j\)‖2​‖𝐘\(i​j\)‖2∈\[0,1\],\\widetilde\{\\kappa\}^\{ij\}\\vcentcolon=\\frac\{\|\\langle\{\\mathbf\{X\}^\{\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\|\}\{\\\|\\mathbf\{X\}^\{\(ij\)\}\\\|\_\{2\}\\,\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\}\\in\[0,1\],\(33\)\(settingκ~i​j=0\\widetilde\{\\kappa\}^\{ij\}=0if a norm is zero\)\. Now define the worst\-case summary statistics

E~v:=maxi≠j⁡E~v​\(i,j\),E~K:=maxi≠j⁡E~K​\(i,j\),κ~:=maxi≠j⁡κ~i​j\.\\widetilde\{E\}\_\{v\}\\vcentcolon=\\max\_\{i\\neq j\}\\widetilde\{E\}\_\{v\}\(i,j\),\\qquad\\widetilde\{E\}\_\{K\}\\vcentcolon=\\max\_\{i\\neq j\}\\widetilde\{E\}\_\{K\}\(i,j\),\\qquad\\widetilde\{\\kappa\}\\vcentcolon=\\max\_\{i\\neq j\}\\widetilde\{\\kappa\}^\{ij\}\.\(34\)

#### B\.6\.1Arbitrary Keys, Arbitrary Values

###### Theorem B\.16\(Cross\-talk bound — arbitrary keys, arbitrary values\)\.

With the summary statistics \(equation[34](https://arxiv.org/html/2607.10034#A2.E34)\),

z~max≤E~K​E~v​κ~\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ \\sqrt\{\\widetilde\{E\}\_\{K\}\}\\,\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\widetilde\{\\kappa\}\.\(35\)

###### Proof\.

Fixi∈\[F\]i\\in\[F\]andj≠ij\\neq i\. If‖𝐗\(i​j\)‖2=0\\\|\\mathbf\{X\}^\{\(ij\)\}\\\|\_\{2\}=0or‖𝐘\(i​j\)‖2=0\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}=0, thenz~i​j=0\\widetilde\{z\}\_\{ij\}=0andκ~i​j=0\\widetilde\{\\kappa\}^\{ij\}=0, so the claim is trivial\. Otherwise, by definition ofκ~i​j\\widetilde\{\\kappa\}^\{ij\}in equation[33](https://arxiv.org/html/2607.10034#A2.E33),

\|z~i​j\|=\|⟨𝐗\(i​j\),𝐘\(i​j\)⟩\|=‖𝐗\(i​j\)‖2​‖𝐘\(i​j\)‖2​κ~i​j=E~K​\(i,j\)​E~v​\(i,j\)​κ~i​j\.\|\\widetilde\{z\}\_\{ij\}\|=\|\\langle\{\\mathbf\{X\}^\{\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\|=\\\|\\mathbf\{X\}^\{\(ij\)\}\\\|\_\{2\}\\,\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\\,\\widetilde\{\\kappa\}^\{ij\}=\\sqrt\{\\widetilde\{E\}\_\{K\}\(i,j\)\}\\,\\sqrt\{\\widetilde\{E\}\_\{v\}\(i,j\)\}\\,\\widetilde\{\\kappa\}^\{ij\}\.UsingE~K​\(i,j\)≤E~K\\widetilde\{E\}\_\{K\}\(i,j\)\\leq\\widetilde\{E\}\_\{K\},E~v​\(i,j\)≤E~v\\widetilde\{E\}\_\{v\}\(i,j\)\\leq\\widetilde\{E\}\_\{v\}, andκ~i​j≤κ~\\widetilde\{\\kappa\}^\{ij\}\\leq\\widetilde\{\\kappa\}and takingmaxi≠j\\max\_\{i\\neq j\}yields equation[35](https://arxiv.org/html/2607.10034#A2.E35)\. ∎

#### B\.6\.2Arbitrary Keys, Isotropic Values

###### Theorem B\.17\(Cross\-talk bound — arbitrary keys, isotropic values\)\.

Assume𝐯1,…,𝐯F\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1⊂ℝd\\mathbb\{S\}^\{d\-1\}\\subset\\mathbb\{R\}^\{d\}\. Treat the kernel matrix𝐊^\\hat\{\\mathbf\{K\}\}as arbitrary\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and let

L:=log⁡\(C0​F2δ\)L\\ \\vcentcolon=\\ \\log\\Big\(\\frac\{C\_\{0\}F^\{2\}\}\{\\delta\}\\Big\)for a sufficiently large absolute constantC0C\_\{0\}\. Then with probability at least1−δ1\-\\delta\(over the values\), the following hold:

E~v\\displaystyle\\widetilde\{E\}\_\{v\}≤C1​\(F−2\)\+Ld−1,\\displaystyle\\ \\leq\\ C\_\{1\}\\,\\frac\{\(F\-2\)\+L\}\{d\-1\},κ~\\displaystyle\\widetilde\{\\kappa\}≤C2​LF−2\.\\displaystyle\\ \\leq\\ C\_\{2\}\\,\\sqrt\{\\frac\{L\}\{F\-2\}\}\.Consequently, combining these summary\-statistic bounds with Theorem[B\.16](https://arxiv.org/html/2607.10034#A2.Thmtheorem16)yields

z~max≤C3​E~K​Ld−1​1\+LF−2\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ C\_\{3\}\\,\\sqrt\{\\widetilde\{E\}\_\{K\}\}\\,\\sqrt\{\\frac\{L\}\{d\-1\}\}\\,\\sqrt\{1\+\\frac\{L\}\{F\-2\}\}\.In particular, ifF−2≥LF\-2\\geq Landd≥2d\\geq 2, then

z~max≤C4​E~K​Ld\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ C\_\{4\}\\,\\sqrt\{\\widetilde\{E\}\_\{K\}\}\\,\\sqrt\{\\frac\{L\}\{d\}\}\.

###### Proof\.

The proof combines two auxiliary concentration bounds via a union bound\.

##### Concentration ofE~v\\widetilde\{E\}\_\{v\}\(invoke Lemma[B\.9\.3](https://arxiv.org/html/2607.10034#A2.SS9.SSS3)\)\.

Apply Lemma[B\.9\.3](https://arxiv.org/html/2607.10034#A2.SS9.SSS3)with failure probabilityδ/2\\delta/2\. This yields an eventℰ1\\mathcal\{E\}\_\{1\}such thatℙ​\(ℰ1c\)≤δ/2\\mathbb\{P\}\(\\mathcal\{E\}\_\{1\}^\{c\}\)\\leq\\delta/2and onℰ1\\mathcal\{E\}\_\{1\},E~v≤C​\(F−2\)\+L~d−1\\widetilde\{E\}\_\{v\}\\leq C\\frac\{\(F\-2\)\+\\widetilde\{L\}\}\{d\-1\}withL~=log⁡\(C0​F2δ/2\)\\widetilde\{L\}=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta/2\}\)\. SinceL~=log⁡\(2​C0​F2δ\)≤log⁡\(C0′​F2δ\)\\widetilde\{L\}=\\log\(\\tfrac\{2C\_\{0\}F^\{2\}\}\{\\delta\}\)\\leq\\log\(\\tfrac\{C\_\{0\}^\{\\prime\}F^\{2\}\}\{\\delta\}\)forC0′=2​C0C\_\{0\}^\{\\prime\}=2C\_\{0\}, we may rewrite the bound usingL=log⁡\(C0′​F2δ\)L=\\log\(\\tfrac\{C\_\{0\}^\{\\prime\}F^\{2\}\}\{\\delta\}\)after adjusting the leading constant\.

##### Concentration ofκ~\\widetilde\{\\kappa\}\(invoke Lemma[B\.9\.3](https://arxiv.org/html/2607.10034#A2.SS9.SSS3)\)\.

Apply Lemma[B\.9\.3](https://arxiv.org/html/2607.10034#A2.SS9.SSS3)with failure probabilityδ/2\\delta/2\. This yields an eventℰ2\\mathcal\{E\}\_\{2\}such thatℙ​\(ℰ2c\)≤δ/2\\mathbb\{P\}\(\\mathcal\{E\}\_\{2\}^\{c\}\)\\leq\\delta/2and onℰ2\\mathcal\{E\}\_\{2\},κ~≤C​L~F−2,\\widetilde\{\\kappa\}\\leq C\\sqrt\{\\tfrac\{\\widetilde\{L\}\}\{F\-2\}\},with the same logarithmic factorL~\\widetilde\{L\}as above\. Again we rewrite in terms ofL=log⁡\(C0​F2δ\)L=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)by enlarging the absolute constant\.

##### Union bound and plug\-in\.

By a union bound,

ℙ​\(ℰ1∩ℰ2\)≥1−ℙ​\(ℰ1c\)−ℙ​\(ℰ2c\)≥1−δ\.\\mathbb\{P\}\(\\mathcal\{E\}\_\{1\}\\cap\\mathcal\{E\}\_\{2\}\)\\ \\geq\\ 1\-\\mathbb\{P\}\(\\mathcal\{E\}\_\{1\}^\{c\}\)\-\\mathbb\{P\}\(\\mathcal\{E\}\_\{2\}^\{c\}\)\\ \\geq\\ 1\-\\delta\.Onℰ1∩ℰ2\\mathcal\{E\}\_\{1\}\\cap\\mathcal\{E\}\_\{2\}, plug the bounds onE~v\\widetilde\{E\}\_\{v\}andκ~\\widetilde\{\\kappa\}into Theorem[B\.16](https://arxiv.org/html/2607.10034#A2.Thmtheorem16)to obtain the stated cross\-talk bound\. If additionallyF−2≥LF\-2\\geq L, then1\+L/\(F−2\)≤2\\sqrt\{1\+L/\(F\-2\)\}\\leq\\sqrt\{2\}and hence

z~max≤C​E~K​Ld−1≤C​E~K​Ld\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ C\\,\\sqrt\{\\widetilde\{E\}\_\{K\}\}\\,\\sqrt\{\\frac\{L\}\{d\-1\}\}\\ \\leq\\ C\\,\\sqrt\{\\widetilde\{E\}\_\{K\}\}\\,\\sqrt\{\\frac\{L\}\{d\}\}which is exactly the final bound stated in the theorem\. ∎

#### B\.6\.3Isotropic Keys, Arbitrary Values \(Bilinear Kernel\)

###### Lemma B\.18\(Mean\+\+residual deterministic cross\-talk decomposition\)\.

Fix a pair\(i,j\)\(i,j\)withj≠ij\\neq i\. For any scalarμ∈ℝ\\mu\\in\\mathbb\{R\}, define the*centered*truncated column

𝐗∘\(i​j\):=𝐗\(i​j\)−μ​1\.\\mathbf\{X\}^\{\\circ\(ij\)\}\\vcentcolon=\\mathbf\{X\}^\{\(ij\)\}\-\\mu\\,\{\\mathbf\{1\}\}\.Then

z~i​j=⟨𝐗\(i​j\),𝐘\(i​j\)⟩=μ​⟨𝟏,𝐘\(i​j\)⟩\+⟨𝐗∘\(i​j\),𝐘\(i​j\)⟩\.\\widetilde\{z\}\_\{ij\}=\\langle\{\\mathbf\{X\}^\{\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle=\\mu\\,\\langle\{\{\\mathbf\{1\}\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\+\\langle\{\\mathbf\{X\}^\{\\circ\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\.Consequently, define

BY:=maxi≠j⁡\|⟨𝟏,𝐘\(i​j\)⟩\|,EK∘​\(i,j\):=‖𝐗∘\(i​j\)‖22,κ∘\(i​j\):=\|⟨𝐗∘\(i​j\),𝐘\(i​j\)⟩\|‖𝐗∘\(i​j\)‖2​‖𝐘\(i​j\)‖2,B\_\{Y\}\\vcentcolon=\\max\_\{i\\neq j\}\\big\|\\langle\{\{\\mathbf\{1\}\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\\big\|,\\qquad E\_\{K\}^\{\\circ\}\(i,j\)\\vcentcolon=\\\|\\mathbf\{X\}^\{\\circ\(ij\)\}\\\|\_\{2\}^\{2\},\\qquad\\kappa^\{\\circ\(ij\)\}\\vcentcolon=\\frac\{\|\\langle\{\\mathbf\{X\}^\{\\circ\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\|\}\{\\\|\\mathbf\{X\}^\{\\circ\(ij\)\}\\\|\_\{2\}\\,\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\},with the conventionκ∘\(i​j\)=0\\kappa^\{\\circ\(ij\)\}=0if‖𝐗∘\(i​j\)‖2​‖𝐘\(i​j\)‖2=0\\\|\\mathbf\{X\}^\{\\circ\(ij\)\}\\\|\_\{2\}\\,\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}=0\. Also define the worst\-case centered energy and centered coupling

EK∘:=maxi≠j⁡EK∘​\(i,j\),κ∘:=maxi≠j⁡κ∘\(i​j\)\.E\_\{K\}^\{\\circ\}\\vcentcolon=\\max\_\{i\\neq j\}E\_\{K\}^\{\\circ\}\(i,j\),\\qquad\\kappa^\{\\circ\}\\vcentcolon=\\max\_\{i\\neq j\}\\kappa^\{\\circ\(ij\)\}\.RecallingE~v\\widetilde\{E\}\_\{v\}from equation[34](https://arxiv.org/html/2607.10034#A2.E34), we have the deterministic bound

z~max≤\|μ\|​BY\+EK∘​E~v​κ∘\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ \|\\mu\|\\,B\_\{Y\}\\ \+\\ \\sqrt\{E\_\{K\}^\{\\circ\}\}\\,\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\kappa^\{\\circ\}\.Moreover, an “effective coupling” form follows by taking the maximum*before*separating the kernel\-side summary statistics:

z~max≤\|μ\|​BY\+E~v​maxi≠j⁡\(EK∘​\(i,j\)​κ∘\(i​j\)\)\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ \|\\mu\|\\,B\_\{Y\}\\ \+\\ \\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\max\_\{i\\neq j\}\\Big\(\\sqrt\{E\_\{K\}^\{\\circ\}\(i,j\)\}\\,\\kappa^\{\\circ\(ij\)\}\\Big\)\.Defining

κeff∘:=maxi≠j⁡\(EK∘​\(i,j\)​κ∘\(i​j\)\)=maxi≠j⁡\|⟨𝐗∘\(i​j\),𝐘\(i​j\)⟩\|‖𝐘\(i​j\)‖2\\kappa\_\{\\mathrm\{eff\}\}^\{\\circ\}\\ \\vcentcolon=\\ \\max\_\{i\\neq j\}\\Big\(\\sqrt\{E\_\{K\}^\{\\circ\}\(i,j\)\}\\,\\kappa^\{\\circ\(ij\)\}\\Big\)\\ =\\ \\max\_\{i\\neq j\}\\frac\{\|\\langle\{\\mathbf\{X\}^\{\\circ\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\|\}\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\}we have the deterministic bound

z~max≤\|μ\|​BY\+E~v​κeff∘\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ \|\\mu\|\\,B\_\{Y\}\\ \+\\ \\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\kappa\_\{\\mathrm\{eff\}\}^\{\\circ\}\.

###### Proof\.

The decomposition follows by substituting𝐗\(i​j\)=μ​1\+𝐗∘\(i​j\)\\mathbf\{X\}^\{\(ij\)\}=\\mu\\,\{\\mathbf\{1\}\}\+\\mathbf\{X\}^\{\\circ\(ij\)\}intoz~i​j=⟨𝐗\(i​j\),𝐘\(i​j\)⟩\\widetilde\{z\}\_\{ij\}=\\langle\{\\mathbf\{X\}^\{\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\.

For the bound, fix\(i,j\)\(i,j\)\. By the triangle inequality,

\|z~i​j\|≤\|μ\|​\|⟨𝟏,𝐘\(i​j\)⟩\|\+\|⟨𝐗∘\(i​j\),𝐘\(i​j\)⟩\|\.\|\\widetilde\{z\}\_\{ij\}\|\\ \\leq\\ \|\\mu\|\\,\\big\|\\langle\{\{\\mathbf\{1\}\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\\big\|\\ \+\\ \\big\|\\langle\{\\mathbf\{X\}^\{\\circ\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\\big\|\.By definition of the pairwise couplingκ∘\(i​j\)\\kappa^\{\\circ\(ij\)\},\|⟨𝐗∘\(i​j\),𝐘\(i​j\)⟩\|≤‖𝐗∘\(i​j\)‖2​‖𝐘\(i​j\)‖2​κ∘\(i​j\)\.\\big\|\\langle\{\\mathbf\{X\}^\{\\circ\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\\big\|\\leq\\\|\\mathbf\{X\}^\{\\circ\(ij\)\}\\\|\_\{2\}\\,\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\\,\\kappa^\{\\circ\(ij\)\}\.Since‖𝐘\(i​j\)‖2≤E~v\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\\leq\\sqrt\{\\widetilde\{E\}\_\{v\}\}and‖𝐗∘\(i​j\)‖2=EK∘​\(i,j\)\\\|\\mathbf\{X\}^\{\\circ\(ij\)\}\\\|\_\{2\}=\\sqrt\{E\_\{K\}^\{\\circ\}\(i,j\)\}, we obtain

\|z~i​j\|≤\|μ\|​BY\+E~v​EK∘​\(i,j\)​κ∘\(i​j\)\.\|\\widetilde\{z\}\_\{ij\}\|\\ \\leq\\ \|\\mu\|\\,B\_\{Y\}\\ \+\\ \\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\sqrt\{E\_\{K\}^\{\\circ\}\(i,j\)\}\\,\\kappa^\{\\circ\(ij\)\}\.Takingmaxi≠j\\max\_\{i\\neq j\}gives

z~max≤\|μ\|​BY\+E~v​maxi≠j⁡\(EK∘​\(i,j\)​κ∘\(i​j\)\)=\|μ\|​BY\+E~v​κeff∘\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ \|\\mu\|\\,B\_\{Y\}\\ \+\\ \\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\max\_\{i\\neq j\}\\Big\(\\sqrt\{E\_\{K\}^\{\\circ\}\(i,j\)\}\\,\\kappa^\{\\circ\(ij\)\}\\Big\)\\ =\\ \|\\mu\|\\,B\_\{Y\}\\ \+\\ \\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\kappa\_\{\\mathrm\{eff\}\}^\{\\circ\}\.Finally, sinceEK∘​\(i,j\)≤EK∘\\sqrt\{E\_\{K\}^\{\\circ\}\(i,j\)\}\\leq\\sqrt\{E\_\{K\}^\{\\circ\}\}andκ∘\(i​j\)≤κ∘\\kappa^\{\\circ\(ij\)\}\\leq\\kappa^\{\\circ\}for all\(i,j\)\(i,j\), we also haveκeff∘≤EK∘​κ∘,\\kappa\_\{\\mathrm\{eff\}\}^\{\\circ\}\\leq\\sqrt\{E\_\{K\}^\{\\circ\}\}\\,\\kappa^\{\\circ\},which yields the stated boundz~max≤\|μ\|​BY\+EK∘​E~v​κ∘\.\\widetilde\{z\}\_\{\\max\}\\leq\|\\mu\|\\,B\_\{Y\}\+\\sqrt\{E\_\{K\}^\{\\circ\}\}\\,\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\kappa^\{\\circ\}\.∎

###### Theorem B\.19\(Cross\-talk bound — isotropic keys, arbitrary values \(bilinear kernel\)\)\.

Assume keys𝐤1,…,𝐤F\{\\mathbf\{k\}\}\_\{1\},\\dots,\{\\mathbf\{k\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}and independent of the feature weights\(𝐚r,𝐛r\)r=1m\(\{\\mathbf\{a\}\}\_\{r\},\{\\mathbf\{b\}\}\_\{r\}\)\_\{r=1\}^\{m\}, as defined in the bilinear random features setup \([SectionB\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2)\)\. Fix*deterministic*values\{𝐯t\}t=1F⊂ℝd\\\{\{\\mathbf\{v\}\}\_\{t\}\\\}\_\{t=1\}^\{F\}\\subset\\mathbb\{R\}^\{d\}and codes\{𝐜t\}t=1F⊂ℝd\\\{\{\\mathbf\{c\}\}\_\{t\}\\\}\_\{t=1\}^\{F\}\\subset\\mathbb\{R\}^\{d\}\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and setL:=log⁡\(C0​F2δ\)L\\vcentcolon=\\log\\big\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\\big\)\. Assumem≤d2/Lm\\leq d^\{2\}/Land letμ=1/d\\mu=1/d\. Letσ2:=𝔼​\[\(𝐊^t​i−μ\)2\]=Θ​\(1d2\+1m\)\\sigma^\{2\}\\vcentcolon=\\mathbb\{E\}\\big\[\(\\hat\{\\mathbf\{K\}\}\_\{ti\}\-\\mu\)^\{2\}\\big\]=\\Theta\(\\tfrac\{1\}\{d^\{2\}\}\+\\tfrac\{1\}\{m\}\)\. LetLvL\_\{v\}be as in[SectionB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px2)\. Then with probability at least1−δ1\-\\delta\(over keys and features\),

z~max≤1d​BY\+C​E~v​\(σ​L\+L2m\)​Lv\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ \\frac\{1\}\{d\}\\,B\_\{Y\}\\ \+\\ C\\,\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\left\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\right\)\\sqrt\{L\_\{v\}\}\.\(36\)In particular, underm≤d2/Lm\\leq d^\{2\}/Lone hasσ≍1/m\\sigma\\asymp 1/\\sqrt\{m\}, so

z~max≤1d​BY\+C​E~v​\(Lm\+L2m\)​Lv\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ \\frac\{1\}\{d\}\\,B\_\{Y\}\\ \+\\ C\\,\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\left\(\\sqrt\{\\frac\{L\}\{m\}\}\+\\frac\{L^\{2\}\}\{m\}\\right\)\\sqrt\{L\_\{v\}\}\.

###### Proof\.

We use the effective\-coupling reduction from Lemma[B\.6\.3](https://arxiv.org/html/2607.10034#A2.SS6.SSS3)and then invoke Lemma[B\.9\.4](https://arxiv.org/html/2607.10034#A2.SS9.SSS4.Px3)\.

##### Deterministic reduction toκeff∘\\kappa\_\{\\mathrm\{eff\}\}^\{\\circ\}\.

Lemma[B\.6\.3](https://arxiv.org/html/2607.10034#A2.SS6.SSS3)\(withμ=1/d\\mu=1/d\) gives

z~max≤1d​BY\+E~v​κeff∘\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ \\frac\{1\}\{d\}\\,B\_\{Y\}\\ \+\\ \\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\kappa\_\{\\mathrm\{eff\}\}^\{\\circ\}\.

##### Concentration ofκeff∘\\kappa\_\{\\mathrm\{eff\}\}^\{\\circ\}\(invoke Lemma[B\.9\.4](https://arxiv.org/html/2607.10034#A2.SS9.SSS4.Px3)\)\.

Apply Lemma[B\.9\.4](https://arxiv.org/html/2607.10034#A2.SS9.SSS4.Px3)with failure probabilityδ\\deltato obtain, with probability at least1−δ1\-\\delta,

κeff∘≤C​\(σ​L\+L2m\)​Lv\.\\kappa\_\{\\mathrm\{eff\}\}^\{\\circ\}\\ \\leq\\ C\\left\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\right\)\\sqrt\{L\_\{v\}\}\.Substitute this bound into the deterministic reduction to obtain equation[36](https://arxiv.org/html/2607.10034#A2.E36)\. ∎

#### B\.6\.4Isotropic Keys, Isotropic Values \(Bilinear Kernel\)

###### Theorem B\.20\(Cross\-talk bound — isotropic keys, isotropic values \(bilinear kernel\)\)\.

AssumeF≥3F\\geq 3\. Keys𝐤1,…,𝐤F\{\\mathbf\{k\}\}\_\{1\},\\dots,\{\\mathbf\{k\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}\. Values𝐯1,…,𝐯F\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}and independent of the keys and features\. The kernel is the bilinear random\-feature kernel from[SectionB\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2)withmmfeatures\.

Fixδ∈\(0,1\)\\delta\\in\(0,1\), and define

L:=log⁡\(C0​F2δ\)\.L\\vcentcolon=\\log\\left\(\\frac\{C\_\{0\}F^\{2\}\}\{\\delta\}\\right\)\.Let

σ2:=𝔼​\[\(𝐊^t​i−1/d\)2\]=Θ​\(1d2\+1m\),t≠i\.\\sigma^\{2\}\\vcentcolon=\\mathbb\{E\}\[\(\\hat\{\\mathbf\{K\}\}\_\{ti\}\-1/d\)^\{2\}\]=\\Theta\\left\(\\frac\{1\}\{d^\{2\}\}\+\\frac\{1\}\{m\}\\right\),\\qquad t\\neq i\.Assume

m≤d2L,L3≤c0​σ2​m2\.m\\leq\\frac\{d^\{2\}\}\{L\},\\qquad L^\{3\}\\leq c\_\{0\}\\sigma^\{2\}m^\{2\}\.Then, with probability at least1−δ1\-\\delta, the following hold simultaneously:

BY\\displaystyle B\_\{Y\}≤C1​F​Ld,\\displaystyle\\ \\leq\\ C\_\{1\}\\sqrt\{\\frac\{FL\}\{d\}\},E~v\\displaystyle\\widetilde\{E\}\_\{v\}≤C2​\(F−2\)\+Ld−1,\\displaystyle\\ \\leq\\ C\_\{2\}\\,\\frac\{\(F\-2\)\+L\}\{d\-1\},EK∘\\displaystyle\\sqrt\{E\_\{K\}^\{\\circ\}\}≤C3​σ​\(F−2\)\+L,\\displaystyle\\ \\leq\\ C\_\{3\}\\,\\sigma\\sqrt\{\(F\-2\)\+L\},κ∘\\displaystyle\\kappa^\{\\circ\}≤C4​LF−2\.\\displaystyle\\ \\leq\\ C\_\{4\}\\sqrt\{\\frac\{L\}\{F\-2\}\}\.Consequently,

z~max≤1d​BY\+EK∘​E~v​κ∘\.\\widetilde\{z\}\_\{\\max\}\\leq\\frac\{1\}\{d\}B\_\{Y\}\+\\sqrt\{E\_\{K\}^\{\\circ\}\}\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\kappa^\{\\circ\}\.In particular, ifF−2≥LF\-2\\geq L, then

z~max≤C6​L​Fd3\+C7​L​Fm​d\.\\widetilde\{z\}\_\{\\max\}\\leq C\_\{6\}\\sqrt\{L\}\\sqrt\{\\frac\{F\}\{d^\{3\}\}\}\+C\_\{7\}\\sqrt\{L\}\\sqrt\{\\frac\{F\}\{md\}\}\.

###### Proof\.

##### Concentration and union bound\.

Apply Lemma[B\.9\.3](https://arxiv.org/html/2607.10034#A2.SS9.SSS3), Lemma[B\.9\.3](https://arxiv.org/html/2607.10034#A2.SS9.SSS3), Lemma[B\.9\.4](https://arxiv.org/html/2607.10034#A2.SS9.SSS4.Px3), and Lemma[B\.9\.3](https://arxiv.org/html/2607.10034#A2.SS9.SSS3.Px4), each with failure probabilityδ/4\\delta/4\. IncreasingC0C\_\{0\}if necessary lets all four events be written with the same

L=log⁡\(C0​F2δ\)\.L=\\log\\left\(\\frac\{C\_\{0\}F^\{2\}\}\{\\delta\}\\right\)\.A union bound gives simultaneous validity with probability at least1−δ1\-\\delta\.

##### Mean–residual reduction\.

On this event, Lemma[B\.6\.3](https://arxiv.org/html/2607.10034#A2.SS6.SSS3)withμ=1/d\\mu=1/dgives

z~max≤1d​BY\+EK∘​E~v​κ∘\.\\widetilde\{z\}\_\{\\max\}\\leq\\frac\{1\}\{d\}B\_\{Y\}\+\\sqrt\{E\_\{K\}^\{\\circ\}\}\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\kappa^\{\\circ\}\.

##### Mean term\.

The mean term satisfies

1d​BY≤C​F​Ld3=C​L​Fd3\.\\frac\{1\}\{d\}B\_\{Y\}\\leq C\\sqrt\{\\frac\{FL\}\{d^\{3\}\}\}=C\\sqrt\{L\}\\sqrt\{\\frac\{F\}\{d^\{3\}\}\}\.

##### Residual term\.

Assume nowF−2≥LF\-2\\geq L\. By Lemma[B\.9\.4](https://arxiv.org/html/2607.10034#A2.SS9.SSS4.Px3),

EK∘≤C​σ​F−2\.\\sqrt\{E\_\{K\}^\{\\circ\}\}\\leq C\\sigma\\sqrt\{F\-2\}\.Therefore

EK∘​E~v​κ∘≤C​σ​F−2​\(F−2\)\+Ld−1​LF−2\.\\sqrt\{E\_\{K\}^\{\\circ\}\}\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\kappa^\{\\circ\}\\leq C\\sigma\\sqrt\{F\-2\}\\sqrt\{\\frac\{\(F\-2\)\+L\}\{d\-1\}\}\\sqrt\{\\frac\{L\}\{F\-2\}\}\.CancelingF−2\\sqrt\{F\-2\},

EK∘​E~v​κ∘≤C​σ​L​\(F−2\)\+Ld−1\.\\sqrt\{E\_\{K\}^\{\\circ\}\}\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\kappa^\{\\circ\}\\leq C\\sigma\\sqrt\{L\}\\sqrt\{\\frac\{\(F\-2\)\+L\}\{d\-1\}\}\.SinceF−2≥LF\-2\\geq L,

\(F−2\)\+L≤2​\(F−2\)≤2​F\.\(F\-2\)\+L\\leq 2\(F\-2\)\\leq 2F\.Alsod−1≍dd\-1\\asymp dafter adjusting constants\. Hence

EK∘​E~v​κ∘≤C​σ​F​Ld\.\\sqrt\{E\_\{K\}^\{\\circ\}\}\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\kappa^\{\\circ\}\\leq C\\sigma\\sqrt\{\\frac\{FL\}\{d\}\}\.Finally,m≤d2/Lm\\leq d^\{2\}/Limplies

1d2≤1m​L≤1m,\\frac\{1\}\{d^\{2\}\}\\leq\\frac\{1\}\{mL\}\\leq\\frac\{1\}\{m\},so

σ2=Θ​\(1d2\+1m\)≤Cm\.\\sigma^\{2\}=\\Theta\\left\(\\frac\{1\}\{d^\{2\}\}\+\\frac\{1\}\{m\}\\right\)\\leq\\frac\{C\}\{m\}\.Thus

σ​F​Ld≤C​F​Lm​d=C​L​Fm​d\.\\sigma\\sqrt\{\\frac\{FL\}\{d\}\}\\leq C\\sqrt\{\\frac\{FL\}\{md\}\}=C\\sqrt\{L\}\\sqrt\{\\frac\{F\}\{md\}\}\.

##### Combine\.

Combining the mean and residual estimates gives the final bound\. ∎

### B\.7Signal Bounds

We now provide lower bounds for the signal terms~i​j\\widetilde\{s\}\_\{ij\}\.

##### Signal as difference of weighted kernel values\.

Recall that under the “absorbing the competitor” convention \(equation[26](https://arxiv.org/html/2607.10034#A2.E26)\), the two\-term signal is

s~i​j=𝐊^i​i​⟨𝐯i−𝐯j,𝐯i⟩\+𝐊^j​i​⟨𝐯i−𝐯j,𝐯j⟩=𝐊^i​i​⟨𝐯i−𝐯j,𝐯i⟩−𝐊^j​i​⟨𝐯j−𝐯i,𝐯j⟩\.\\widetilde\{s\}\_\{ij\}=\\hat\{\\mathbf\{K\}\}\_\{ii\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{i\}\}\\rangle\+\\hat\{\\mathbf\{K\}\}\_\{ji\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle=\\hat\{\\mathbf\{K\}\}\_\{ii\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{i\}\}\\rangle\-\\hat\{\\mathbf\{K\}\}\_\{ji\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{j\}\-\{\\mathbf\{v\}\}\_\{i\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\.\(37\)Since𝐊^\\hat\{\\mathbf\{K\}\}is a Gram matrix, we always have𝐊^i​i=‖ϕ​\(𝐤i\)‖22≥0\.\\hat\{\\mathbf\{K\}\}\_\{ii\}=\\\|\\phi\(\{\\mathbf\{k\}\}\_\{i\}\)\\\|\_\{2\}^\{2\}\\geq 0\.Further, define the smallest \(worst\-case\) signal by

s~min:=mini≠j⁡s~i​j\.\\widetilde\{s\}\_\{\\min\}\\ \\vcentcolon=\\ \\min\_\{i\\neq j\}\\widetilde\{s\}\_\{ij\}\.\(38\)

##### Signal summary statistics\.

Define the value\-side inner products

Vi​j:=⟨𝐯i−𝐯j,𝐯i⟩,Bi​j:=⟨𝐯i−𝐯j,𝐯j⟩,\(i≠j\)\.V\_\{ij\}\\ \\vcentcolon=\\ \\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{i\}\}\\rangle,\\qquad B\_\{ij\}\\ \\vcentcolon=\\ \\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle,\\qquad\(i\\neq j\)\.\(39\)Across our bounds below we will assume, unless stated otherwise,

Vi​j≥0for all​i≠j\.V\_\{ij\}\\ \\geq\\ 0\\qquad\\text\{for all \}i\\neq j\.\(40\)This holds automatically when all values share a common norm: if‖𝐯i‖2=‖𝐯j‖2=r\\\|\{\\mathbf\{v\}\}\_\{i\}\\\|\_\{2\}=\\\|\{\\mathbf\{v\}\}\_\{j\}\\\|\_\{2\}=rthen by Cauchy–Schwarz⟨𝐯i,𝐯j⟩≤‖𝐯i‖2​‖𝐯j‖2=r2,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\\leq\\\|\{\\mathbf\{v\}\}\_\{i\}\\\|\_\{2\}\\\|\{\\mathbf\{v\}\}\_\{j\}\\\|\_\{2\}=r^\{2\},henceVi​j=r2−⟨𝐯i,𝐯j⟩≥0\.V\_\{ij\}=r^\{2\}\-\\langle\{\{\\mathbf\{v\}\}\_\{i\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\\geq 0\.

Recall from[SectionsB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px1)and[B\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px2)the kernel and value\-side extrema:

Kmindiag=mini∈\[F\]⁡𝐊^i​i,Kmaxoff=maxi≠j⁡\|𝐊^i​j\|,K\_\{\\min\}^\{\\mathrm\{diag\}\}\\ =\\ \\min\_\{i\\in\[F\]\}\\hat\{\\mathbf\{K\}\}\_\{ii\},\\qquad K\_\{\\max\}^\{\\mathrm\{off\}\}\\ =\\ \\max\_\{i\\neq j\}\|\\hat\{\\mathbf\{K\}\}\_\{ij\}\|,\(41\)Vmin=mini≠j⁡⟨𝐯i−𝐯j,𝐯i⟩,Vmax=maxi≠j⁡\|⟨𝐯i−𝐯j,𝐯j⟩\|\.V\_\{\\min\}\\ =\\ \\min\_\{i\\neq j\}\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{i\}\}\\rangle,\\qquad V\_\{\\max\}\\ =\\ \\max\_\{i\\neq j\}\\big\|\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\\big\|\.\(42\)

#### B\.7\.1Arbitrary Keys, Arbitrary Values

###### Theorem B\.21\(Deterministic signal lower bound — arbitrary keys, arbitrary values\)\.

Under the positivity condition \(equation[40](https://arxiv.org/html/2607.10034#A2.E40)\), and with the summary statistics \([SectionsB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px1)and[B\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px2)\),

s~min≥Kmindiag​Vmin−Kmaxoff​Vmax\.\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}\\,V\_\{\\min\}\\ \-\\ K\_\{\\max\}^\{\\mathrm\{off\}\}\\,V\_\{\\max\}\.

###### Proof\.

Fix a pair\(i,j\)\(i,j\)withi≠ji\\neq j\. From equation[37](https://arxiv.org/html/2607.10034#A2.E37)and equation[39](https://arxiv.org/html/2607.10034#A2.E39),

s~i​j=𝐊^i​i​Vi​j\+𝐊^j​i​Bi​j\.\\widetilde\{s\}\_\{ij\}=\\hat\{\\mathbf\{K\}\}\_\{ii\}\\,V\_\{ij\}\+\\hat\{\\mathbf\{K\}\}\_\{ji\}\\,B\_\{ij\}\.
By definition,𝐊^i​i≥Kmindiag\\hat\{\\mathbf\{K\}\}\_\{ii\}\\geq K\_\{\\min\}^\{\\mathrm\{diag\}\}\. AlsoVi​j≥VminV\_\{ij\}\\geq V\_\{\\min\}by definition ofVminV\_\{\\min\}\. Under the positivity assumption \(equation[40](https://arxiv.org/html/2607.10034#A2.E40)\), we haveVi​j≥0V\_\{ij\}\\geq 0for alli≠ji\\neq j, and therefore multiplying the inequalities𝐊^i​i≥Kmindiag\\hat\{\\mathbf\{K\}\}\_\{ii\}\\geq K\_\{\\min\}^\{\\mathrm\{diag\}\}andVi​j≥VminV\_\{ij\}\\geq V\_\{\\min\}preserves order:

𝐊^i​i​Vi​j≥Kmindiag​Vmin\.\\hat\{\\mathbf\{K\}\}\_\{ii\}\\,V\_\{ij\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}\\,V\_\{\\min\}\.
For the second term, we usex​y≥−\|x\|​\|y\|xy\\geq\-\|x\|\\,\|y\|:

𝐊^j​i​Bi​j≥−\|𝐊^j​i\|​\|Bi​j\|\.\\hat\{\\mathbf\{K\}\}\_\{ji\}\\,B\_\{ij\}\\ \\geq\\ \-\|\\hat\{\\mathbf\{K\}\}\_\{ji\}\|\\,\|B\_\{ij\}\|\.By definition\|𝐊^j​i\|≤Kmaxoff\|\\hat\{\\mathbf\{K\}\}\_\{ji\}\|\\leq K\_\{\\max\}^\{\\mathrm\{off\}\}and\|Bi​j\|≤Vmax\|B\_\{ij\}\|\\leq V\_\{\\max\}, hence

𝐊^j​i​Bi​j≥−Kmaxoff​Vmax\.\\hat\{\\mathbf\{K\}\}\_\{ji\}\\,B\_\{ij\}\\ \\geq\\ \-K\_\{\\max\}^\{\\mathrm\{off\}\}\\,V\_\{\\max\}\.
Adding the two bounds yieldss~i​j≥Kmindiag​Vmin−Kmaxoff​Vmax,\\widetilde\{s\}\_\{ij\}\\geq K\_\{\\min\}^\{\\mathrm\{diag\}\}V\_\{\\min\}\-K\_\{\\max\}^\{\\mathrm\{off\}\}V\_\{\\max\},resulting in the claimed bound fors~min\\widetilde\{s\}\_\{\\min\}\. ∎

#### B\.7\.2Arbitrary Keys, Isotropic Values

###### Theorem B\.22\(Signal lower bound — arbitrary keys, isotropic values\)\.

Assume𝐯1,…,𝐯F\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1⊂ℝd\\mathbb\{S\}^\{d\-1\}\\subset\\mathbb\{R\}^\{d\}, and treat the kernel matrix𝐊^\\hat\{\\mathbf\{K\}\}as arbitrary \(or condition on it\)\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and let

L:=log⁡\(C0​F2δ\),εv:=C1​Ld−1\.L\\ \\vcentcolon=\\ \\log\\Big\(\\frac\{C\_\{0\}F^\{2\}\}\{\\delta\}\\Big\),\\qquad\\varepsilon\_\{v\}\\ \\vcentcolon=\\ C\_\{1\}\\sqrt\{\\frac\{L\}\{d\-1\}\}\.Then with probability at least1−δ1\-\\delta\(over the values\),

s~min≥Kmindiag​\(1−εv\)−Kmaxoff​\(1\+εv\),\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}\\,\(1\-\\varepsilon\_\{v\}\)\\ \-\\ K\_\{\\max\}^\{\\mathrm\{off\}\}\\,\(1\+\\varepsilon\_\{v\}\),whereKmindiagK\_\{\\min\}^\{\\mathrm\{diag\}\}andKmaxoffK\_\{\\max\}^\{\\mathrm\{off\}\}are the kernel extrema \([SectionB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px1)\)\.

###### Proof\.

The proof relies on a single auxiliary concentration bound\.

##### Concentration of the value\-side extrema\.

Apply Lemma[B\.9\.1](https://arxiv.org/html/2607.10034#A2.SS9.SSS1)with failure probabilityδ\\delta\. This yields an eventℰv\\mathcal\{E\}\_\{v\}such thatℙ​\(ℰvc\)≤δ\\mathbb\{P\}\(\\mathcal\{E\}\_\{v\}^\{c\}\)\\leq\\deltaand onℰv\\mathcal\{E\}\_\{v\},

maxi≠j⁡\|⟨𝐯i,𝐯j⟩\|≤εv\.\\max\_\{i\\neq j\}\\big\|\\langle\{\{\\mathbf\{v\}\}\_\{i\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\\big\|\\ \\leq\\ \\varepsilon\_\{v\}\.Hence, onℰv\\mathcal\{E\}\_\{v\},

Vmin≥1−εv,Vmax≤1\+εv\.V\_\{\\min\}\\ \\geq\\ 1\-\\varepsilon\_\{v\},\\qquad V\_\{\\max\}\\ \\leq\\ 1\+\\varepsilon\_\{v\}\.

##### Plug\-in\.

Onℰv\\mathcal\{E\}\_\{v\}, the deterministic signal lower bound gives

s~min≥Kmindiag​Vmin−Kmaxoff​Vmax≥Kmindiag​\(1−εv\)−Kmaxoff​\(1\+εv\),\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}V\_\{\\min\}\-K\_\{\\max\}^\{\\mathrm\{off\}\}V\_\{\\max\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}\(1\-\\varepsilon\_\{v\}\)\-K\_\{\\max\}^\{\\mathrm\{off\}\}\(1\+\\varepsilon\_\{v\}\),which is exactly the claimed bound\. ∎

#### B\.7\.3Isotropic Keys, Arbitrary Values \(Bilinear Kernel\)

###### Theorem B\.23\(Signal lower bound — isotropic keys, arbitrary values \(bilinear kernel\)\)\.

Assume keys𝐤1,…,𝐤F\{\\mathbf\{k\}\}\_\{1\},\\dots,\{\\mathbf\{k\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}, the kernel is the bilinear random\-feature kernel from[SectionB\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2)withmmfeatures, and the values\{𝐯t\}t=1F\\\{\{\\mathbf\{v\}\}\_\{t\}\\\}\_\{t=1\}^\{F\}are deterministic and satisfy the positivity condition \(equation[40](https://arxiv.org/html/2607.10034#A2.E40)\)\. LetVminV\_\{\\min\}andVmaxV\_\{\\max\}be as in[SectionB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px2)\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and setL:=log⁡\(C0​F2δ\)L\\vcentcolon=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)\. Letμ:=1/d\\mu\\vcentcolon=1/dand letσ2:=𝔼​\[\(𝐊^i​j−μ\)2\]=Θ​\(1d2\+1m\)\\sigma^\{2\}\\vcentcolon=\\mathbb\{E\}\[\(\\hat\{\\mathbf\{K\}\}\_\{ij\}\-\\mu\)^\{2\}\]=\\Theta\(\\tfrac\{1\}\{d^\{2\}\}\+\\tfrac\{1\}\{m\}\)\(see Lemma[B\.9\.4](https://arxiv.org/html/2607.10034#A2.SS9.SSS4)\)\. Then with probability at least1−δ1\-\\delta\(over keys and features\),

s~min≥\(1−C1​Lm\)​Vmin−\(μ\+C2​\(σ​L\+L2/m\)\)​Vmax\.\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ \\Big\(1\-C\_\{1\}\\sqrt\{\\tfrac\{L\}\{m\}\}\\Big\)\\,V\_\{\\min\}\\ \-\\ \\Big\(\\mu\+C\_\{2\}\(\\sigma\\sqrt\{L\}\+L^\{2\}/m\)\\Big\)\\,V\_\{\\max\}\.Moreover, ifL≥1L\\geq 1andm≥L3m\\geq L^\{3\}, then the off\-diagonal coefficient simplifies and

s~min≥\(1−C1​Lm\)​Vmin−C3​σ​L​Vmax\.\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ \\Big\(1\-C\_\{1\}\\sqrt\{\\tfrac\{L\}\{m\}\}\\Big\)\\,V\_\{\\min\}\\ \-\\ C\_\{3\}\\,\\sigma\\sqrt\{L\}\\,V\_\{\\max\}\.

###### Proof\.

The proof combines two auxiliary concentration bounds via a union bound\.

##### Concentration of the diagonal term\.

Apply Lemma[B\.9\.2](https://arxiv.org/html/2607.10034#A2.SS9.SSS2)with failure probabilityδ/2\\delta/2\. This yields an eventℰ1\\mathcal\{E\}\_\{1\}such thatℙ​\(ℰ1c\)≤δ/2\\mathbb\{P\}\(\\mathcal\{E\}\_\{1\}^\{c\}\)\\leq\\delta/2and onℰ1\\mathcal\{E\}\_\{1\},

Kmindiag≥1−C​Lm,K\_\{\\min\}^\{\\mathrm\{diag\}\}\\ \\geq\\ 1\-C\\sqrt\{\\frac\{L\}\{m\}\},after adjusting the absolute constant hidden inL=log⁡\(C0​F2δ\)L=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)\.

##### Concentration of the off\-diagonal term\.

Apply Lemma[B\.9\.2](https://arxiv.org/html/2607.10034#A2.SS9.SSS2)with failure probabilityδ/2\\delta/2\. This yields an eventℰ2\\mathcal\{E\}\_\{2\}such thatℙ​\(ℰ2c\)≤δ/2\\mathbb\{P\}\(\\mathcal\{E\}\_\{2\}^\{c\}\)\\leq\\delta/2and onℰ2\\mathcal\{E\}\_\{2\},

Kmaxoff≤μ\+C​\(σ​L\+L2m\),K\_\{\\max\}^\{\\mathrm\{off\}\}\\ \\leq\\ \\mu\+C\\Big\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\Big\),again after adjusting the absolute constant hidden inLL\.

##### Union bound and plug\-in\.

By a union bound,

ℙ​\(ℰ1∩ℰ2\)≥1−ℙ​\(ℰ1c\)−ℙ​\(ℰ2c\)≥1−δ\.\\mathbb\{P\}\(\\mathcal\{E\}\_\{1\}\\cap\\mathcal\{E\}\_\{2\}\)\\ \\geq\\ 1\-\\mathbb\{P\}\(\\mathcal\{E\}\_\{1\}^\{c\}\)\-\\mathbb\{P\}\(\\mathcal\{E\}\_\{2\}^\{c\}\)\\ \\geq\\ 1\-\\delta\.Onℰ1∩ℰ2\\mathcal\{E\}\_\{1\}\\cap\\mathcal\{E\}\_\{2\}, the deterministic signal lower bound gives

s~min≥Kmindiag​Vmin−Kmaxoff​Vmax\.\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}V\_\{\\min\}\-K\_\{\\max\}^\{\\mathrm\{off\}\}V\_\{\\max\}\.Substituting the bounds forKmindiagK\_\{\\min\}^\{\\mathrm\{diag\}\}andKmaxoffK\_\{\\max\}^\{\\mathrm\{off\}\}gives, after renaming absolute constants,

s~min≥\(1−C1​Lm\)​Vmin−\(μ\+C2​\(σ​L\+L2/m\)\)​Vmax,\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ \\Big\(1\-C\_\{1\}\\sqrt\{\\tfrac\{L\}\{m\}\}\\Big\)V\_\{\\min\}\\ \-\\ \\Big\(\\mu\+C\_\{2\}\(\\sigma\\sqrt\{L\}\+L^\{2\}/m\)\\Big\)V\_\{\\max\},which is exactly the stated lower bound\.

##### Simplification of the off\-diagonal coefficient\.

IfL≥1L\\geq 1andm≥L3m\\geq L^\{3\}, thenσ2=Θ​\(1d2\+1m\)\\sigma^\{2\}=\\Theta\(\\tfrac\{1\}\{d^\{2\}\}\+\\tfrac\{1\}\{m\}\)impliesσ≳1/d=μ,\\sigma\\gtrsim 1/d=\\mu,soμ≤C​σ≤C​σ​L\.\\mu\\leq C\\sigma\\leq C\\sigma\\sqrt\{L\}\.Hence

μ\+C​\(σ​L\+L2m\)≤C​σ​L,\\mu\+C\\Big\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\Big\)\\ \\leq\\ C\\sigma\\sqrt\{L\},and substituting this into the previous bound yields the refined inequality\. ∎

#### B\.7\.4Isotropic Keys, Isotropic Values \(Bilinear Kernel\)

###### Theorem B\.24\(Signal lower bound — isotropic keys, isotropic values \(bilinear kernel\)\)\.

Assume keys𝐤1,…,𝐤F\{\\mathbf\{k\}\}\_\{1\},\\dots,\{\\mathbf\{k\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}, values𝐯1,…,𝐯F\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}and independent of the keys and features, and the kernel is the bilinear random\-feature kernel from[SectionB\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2)withmmfeatures\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and setL:=log⁡\(C0​F2δ\)L\\vcentcolon=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)\. Assumem≤d2/Lm\\leq d^\{2\}/L\. Letμ:=1/d\\mu\\vcentcolon=1/dandσ2:=𝔼​\[\(𝐊^i​j−μ\)2\]=Θ​\(1d2\+1m\)\\sigma^\{2\}\\vcentcolon=\\mathbb\{E\}\[\(\\hat\{\\mathbf\{K\}\}\_\{ij\}\-\\mu\)^\{2\}\]=\\Theta\(\\tfrac\{1\}\{d^\{2\}\}\+\\tfrac\{1\}\{m\}\)\. Then with probability at least1−δ1\-\\delta\(over keys, features, and values\),

s~min≥\(1−C1​Lm\)​\(1−εv\)−\(μ\+C2​\(σ​L\+L2/m\)\)​\(1\+εv\),εv=C3​Ld−1\.\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ \\Big\(1\-C\_\{1\}\\sqrt\{\\tfrac\{L\}\{m\}\}\\Big\)\\,\(1\-\\varepsilon\_\{v\}\)\\ \-\\ \\Big\(\\mu\+C\_\{2\}\(\\sigma\\sqrt\{L\}\+L^\{2\}/m\)\\Big\)\\,\(1\+\\varepsilon\_\{v\}\),\\qquad\\varepsilon\_\{v\}=C\_\{3\}\\sqrt\{\\tfrac\{L\}\{d\-1\}\}\.Moreover, ifL≥1L\\geq 1andm≥L3m\\geq L^\{3\}, then

s~min≥\(1−C1​Lm\)​\(1−εv\)−C4​σ​L​\(1\+εv\),εv=C3​Ld−1\.\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ \\Big\(1\-C\_\{1\}\\sqrt\{\\tfrac\{L\}\{m\}\}\\Big\)\\,\(1\-\\varepsilon\_\{v\}\)\\ \-\\ C\_\{4\}\\,\\sigma\\sqrt\{L\}\\,\(1\+\\varepsilon\_\{v\}\),\\qquad\\varepsilon\_\{v\}=C\_\{3\}\\sqrt\{\\tfrac\{L\}\{d\-1\}\}\.

###### Proof\.

The proof combines three auxiliary concentration bounds via a union bound\.

##### Concentration of the value\-side extrema\.

Apply Lemma[B\.9\.1](https://arxiv.org/html/2607.10034#A2.SS9.SSS1)with failure probabilityδ/3\\delta/3\. This yields an eventℰv\\mathcal\{E\}\_\{v\}such thatℙ​\(ℰvc\)≤δ/3\\mathbb\{P\}\(\\mathcal\{E\}\_\{v\}^\{c\}\)\\leq\\delta/3and onℰv\\mathcal\{E\}\_\{v\},

maxi≠j⁡\|⟨𝐯i,𝐯j⟩\|≤εv\.\\max\_\{i\\neq j\}\\big\|\\langle\{\{\\mathbf\{v\}\}\_\{i\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\\big\|\\ \\leq\\ \\varepsilon\_\{v\}\.Hence, onℰv\\mathcal\{E\}\_\{v\},

Vmin≥1−εv,Vmax≤1\+εv\.V\_\{\\min\}\\ \\geq\\ 1\-\\varepsilon\_\{v\},\\qquad V\_\{\\max\}\\ \\leq\\ 1\+\\varepsilon\_\{v\}\.

##### Concentration of the diagonal term\.

Apply Lemma[B\.9\.2](https://arxiv.org/html/2607.10034#A2.SS9.SSS2)with failure probabilityδ/3\\delta/3\. This yields an eventℰ1\\mathcal\{E\}\_\{1\}such thatℙ​\(ℰ1c\)≤δ/3\\mathbb\{P\}\(\\mathcal\{E\}\_\{1\}^\{c\}\)\\leq\\delta/3and onℰ1\\mathcal\{E\}\_\{1\},

Kmindiag≥1−C​Lm,K\_\{\\min\}^\{\\mathrm\{diag\}\}\\ \\geq\\ 1\-C\\sqrt\{\\frac\{L\}\{m\}\},after adjusting the absolute constant hidden inLL\.

##### Concentration of the off\-diagonal term\.

Apply Lemma[B\.9\.2](https://arxiv.org/html/2607.10034#A2.SS9.SSS2)with failure probabilityδ/3\\delta/3\. This yields an eventℰ2\\mathcal\{E\}\_\{2\}such thatℙ​\(ℰ2c\)≤δ/3\\mathbb\{P\}\(\\mathcal\{E\}\_\{2\}^\{c\}\)\\leq\\delta/3and onℰ2\\mathcal\{E\}\_\{2\},

Kmaxoff≤μ\+C​\(σ​L\+L2m\),K\_\{\\max\}^\{\\mathrm\{off\}\}\\ \\leq\\ \\mu\+C\\Big\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\Big\),again after adjusting the absolute constant hidden inLL\.

##### Union bound and plug\-in\.

By a union bound,

ℙ​\(ℰv∩ℰ1∩ℰ2\)≥1−ℙ​\(ℰvc\)−ℙ​\(ℰ1c\)−ℙ​\(ℰ2c\)≥1−δ\.\\mathbb\{P\}\(\\mathcal\{E\}\_\{v\}\\cap\\mathcal\{E\}\_\{1\}\\cap\\mathcal\{E\}\_\{2\}\)\\ \\geq\\ 1\-\\mathbb\{P\}\(\\mathcal\{E\}\_\{v\}^\{c\}\)\-\\mathbb\{P\}\(\\mathcal\{E\}\_\{1\}^\{c\}\)\-\\mathbb\{P\}\(\\mathcal\{E\}\_\{2\}^\{c\}\)\\ \\geq\\ 1\-\\delta\.On this intersection, the deterministic signal lower bound gives

s~min≥Kmindiag​Vmin−Kmaxoff​Vmax\.\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}V\_\{\\min\}\-K\_\{\\max\}^\{\\mathrm\{off\}\}V\_\{\\max\}\.Substituting the bounds forKmindiagK\_\{\\min\}^\{\\mathrm\{diag\}\},KmaxoffK\_\{\\max\}^\{\\mathrm\{off\}\},VminV\_\{\\min\}, andVmaxV\_\{\\max\}yields the stated lower bound\.

##### Simplification of the off\-diagonal coefficient\.

IfL≥1L\\geq 1andm≥L3m\\geq L^\{3\}, then the same reasoning as above gives

μ\+C​\(σ​L\+L2m\)≤C​σ​L,\\mu\+C\\Big\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\Big\)\\ \\leq\\ C\\sigma\\sqrt\{L\},and substituting this estimate into the previous bound yields the refined inequality\. ∎

### B\.8Margin Bounds

We now combine our signal and cross\-talk bounds to bound the Hebbian memory margin\.

#### B\.8\.1Arbitrary Keys, Arbitrary Values

###### Theorem B\.25\(Margin bound — arbitrary keys, arbitrary values\)\.

Assume the positivity condition equation[40](https://arxiv.org/html/2607.10034#A2.E40)\(i\.e\.,Vi​j≥0V\_\{ij\}\\geq 0for alli≠ji\\neq j\)\. LetKmindiagK\_\{\\min\}^\{\\mathrm\{diag\}\},KmaxoffK\_\{\\max\}^\{\\mathrm\{off\}\}\([SectionB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px1)\),VminV\_\{\\min\}, andVmaxV\_\{\\max\}\([SectionB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px2)\) be the signal\-side summary statistics, and letE~K\\widetilde\{E\}\_\{K\},E~v\\widetilde\{E\}\_\{v\}, andκ~\\widetilde\{\\kappa\}be the cross\-talk summary statistics from equation[34](https://arxiv.org/html/2607.10034#A2.E34)\. Then

γmin≥Kmindiag​Vmin−Kmaxoff​Vmax−E~K​E~v​κ~\.\\gamma\_\{\\min\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}\\,V\_\{\\min\}\\ \-\\ K\_\{\\max\}^\{\\mathrm\{off\}\}\\,V\_\{\\max\}\\ \-\\ \\sqrt\{\\widetilde\{E\}\_\{K\}\}\\,\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\widetilde\{\\kappa\}\.\(43\)Moreover, if we instead keep the original signal/cross\-talk decomposition \(equation[25](https://arxiv.org/html/2607.10034#A2.E25)\) \(i\.e\., without absorbing the competitor term\), then applying Cauchy–Schwarz with the coupling factorκ\\kappafrom[SectionB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px3)yields the alternative bound

γmin≥Kmindiag​Vmin−EK​Ev​κ,\\gamma\_\{\\min\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}\\,V\_\{\\min\}\\ \-\\ \\sqrt\{E\_\{K\}\}\\,\\sqrt\{E\_\{v\}\}\\,\\kappa,\(44\)whereEKE\_\{K\}andEvE\_\{v\}are as defined in[SectionsB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px1)and[B\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px2)\.

###### Proof\.

Under the positivity condition \(equation[40](https://arxiv.org/html/2607.10034#A2.E40)\), the deterministic signal lower bound gives

s~min≥Kmindiag​Vmin−Kmaxoff​Vmax\.\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}\\,V\_\{\\min\}\\ \-\\ K\_\{\\max\}^\{\\mathrm\{off\}\}\\,V\_\{\\max\}\.
Further, Theorem[B\.16](https://arxiv.org/html/2607.10034#A2.Thmtheorem16)yields

z~max≤E~K​E~v​κ~\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ \\sqrt\{\\widetilde\{E\}\_\{K\}\}\\,\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\widetilde\{\\kappa\}\.
Recalling thatγmin≥s~min−z~max\\gamma\_\{\\min\}\\geq\\widetilde\{s\}\_\{\\min\}\-\\widetilde\{z\}\_\{\\max\}, we therefore have

γmin≥s~min−z~max≥Kmindiag​Vmin−Kmaxoff​Vmax−E~K​E~v​κ~,\\gamma\_\{\\min\}\\ \\geq\\ \\widetilde\{s\}\_\{\\min\}\-\\widetilde\{z\}\_\{\\max\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}\\,V\_\{\\min\}\-K\_\{\\max\}^\{\\mathrm\{off\}\}\\,V\_\{\\max\}\-\\sqrt\{\\widetilde\{E\}\_\{K\}\}\\,\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\widetilde\{\\kappa\},
Finally, to obtain equation[44](https://arxiv.org/html/2607.10034#A2.E44), start from the original decomposition equation[25](https://arxiv.org/html/2607.10034#A2.E25):

γi​j=𝐊^i​i​Vi​j\+zi​j,zi​j:=∑t≠i𝐊^t​i​⟨𝐯i−𝐯j,𝐜t⟩\.\\gamma\_\{ij\}\\,=\\,\\hat\{\\mathbf\{K\}\}\_\{ii\}\\,V\_\{ij\}\\,\+\\,z\_\{ij\},\\qquad z\_\{ij\}\\ \\vcentcolon=\\ \\sum\_\{t\\neq i\}\\hat\{\\mathbf\{K\}\}\_\{ti\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{c\}\}\_\{t\}\}\\rangle\.By definition of the coupling factorκi​j\\kappa^\{ij\}\([SectionB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px3)\),\|zi​j\|≤EK​\(i\)​Ev​\(i,j\)​κi​j≤EK​Ev​κ\.\|z\_\{ij\}\|\\ \\leq\\ \\sqrt\{E\_\{K\}\(i\)\}\\,\\sqrt\{E\_\{v\}\(i,j\)\}\\,\\kappa^\{ij\}\\ \\leq\\ \\sqrt\{E\_\{K\}\}\\,\\sqrt\{E\_\{v\}\}\\,\\kappa\.Since𝐊^i​i≥Kmindiag\\hat\{\\mathbf\{K\}\}\_\{ii\}\\geq K\_\{\\min\}^\{\\mathrm\{diag\}\}andVi​j≥VminV\_\{ij\}\\geq V\_\{\\min\}, takingmini≠j\\min\_\{i\\neq j\}yields equation[44](https://arxiv.org/html/2607.10034#A2.E44)\.

∎

###### Corollary B\.26\(Arbitrary\-embedding margin scaling — simplified\)\.

Under the assumptions of[TheoremB\.25](https://arxiv.org/html/2607.10034#A2.Thmtheorem25), further reasonably assumingd≥4​log⁡Fd\\geq 4\\log Fand using the penalty statisticsSsigS\_\{\\mathrm\{sig\}\},PkeyP\_\{\\mathrm\{key\}\},PvalP\_\{\\mathrm\{val\}\},PalignP\_\{\\mathrm\{align\}\}from[Section4\.2\.2](https://arxiv.org/html/2607.10034#S4.SS2.SSS2), for someC∈\(0,1\)C\\in\(0,1\)we have

γmin≥C​Ssig−Pkey​Pval​Palign​F​log⁡Fm​d\.\\gamma\_\{\\min\}\\;\\geq\\;C\\,S\_\{\\mathrm\{sig\}\}\\;\-\\;P\_\{\\mathrm\{key\}\}\\,P\_\{\\mathrm\{val\}\}\\,P\_\{\\mathrm\{align\}\}\\sqrt\{\\frac\{F\\log F\}\{md\}\}\.

###### Proof\.

The statisticsEK,Ev,κE\_\{K\},E\_\{v\},\\kappa\([Section4\.2\.2](https://arxiv.org/html/2607.10034#S4.SS2.SSS2)\) sum overt≠it\\neq i\(the competitort=jt=jincluded\), so they coincide with the untruncated statistics of[SectionsB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px1),[B\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px2)and[B\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px3)and[TheoremB\.25](https://arxiv.org/html/2607.10034#A2.Thmtheorem25)\([Equation44](https://arxiv.org/html/2607.10034#A2.E44)\) applies:γmin≥Kmindiag​Vmin−EK​Ev​κ\.\\gamma\_\{\\min\}\\geq K\_\{\\min\}^\{\\mathrm\{diag\}\}\\,V\_\{\\min\}\-\\sqrt\{E\_\{K\}\}\\,\\sqrt\{E\_\{v\}\}\\,\\kappa\.SubstitutingEK=Pkey​F/m\\sqrt\{E\_\{K\}\}=P\_\{\\mathrm\{key\}\}\\sqrt\{F/m\},Ev=Pval​F/d\\sqrt\{E\_\{v\}\}=P\_\{\\mathrm\{val\}\}\\sqrt\{F/d\}, andκ=Palign​log⁡\(F\)/F\\kappa=P\_\{\\mathrm\{align\}\}\\sqrt\{\\log\(F\)/F\}gives the exact identity

EK​Ev​κ=Pkey​Pval​Palign​Fm⋅Fd⋅log⁡FF=Pkey​Pval​Palign​F​log⁡Fm​d\.\\sqrt\{E\_\{K\}\}\\,\\sqrt\{E\_\{v\}\}\\,\\kappa=P\_\{\\mathrm\{key\}\}\\,P\_\{\\mathrm\{val\}\}\\,P\_\{\\mathrm\{align\}\}\\sqrt\{\\tfrac\{F\}\{m\}\\cdot\\tfrac\{F\}\{d\}\\cdot\\tfrac\{\\log F\}\{F\}\}=P\_\{\\mathrm\{key\}\}\\,P\_\{\\mathrm\{val\}\}\\,P\_\{\\mathrm\{align\}\}\\sqrt\{\\tfrac\{F\\log F\}\{md\}\}\.For the signal, the definition ofSsigS\_\{\\mathrm\{sig\}\}givesKmindiag​Vmin=\(1−log⁡\(F\)/d\)​SsigK\_\{\\min\}^\{\\mathrm\{diag\}\}\\,V\_\{\\min\}=\\big\(1\-\\sqrt\{\\log\(F\)/d\}\\big\)\\,S\_\{\\mathrm\{sig\}\}\. Sinced≥4​log⁡Fd\\geq 4\\log Fforceslog⁡\(F\)/d≤12\\sqrt\{\\log\(F\)/d\}\\leq\\tfrac\{1\}\{2\}, andSsig≥0S\_\{\\mathrm\{sig\}\}\\geq 0\(asKmindiag≥0K\_\{\\min\}^\{\\mathrm\{diag\}\}\\geq 0andVmin≥0V\_\{\\min\}\\geq 0by[Equation40](https://arxiv.org/html/2607.10034#A2.E40)\), we getKmindiag​Vmin≥C​SsigK\_\{\\min\}^\{\\mathrm\{diag\}\}\\,V\_\{\\min\}\\geq C\\,S\_\{\\mathrm\{sig\}\}withC:=12C\\vcentcolon=\\tfrac\{1\}\{2\}\. Combining the two equations proves the claim\. ∎

###### Corollary B\.27\(Arbitrary\-embedding fact\-storage capacity\)\.

Under the assumptions of[SectionB\.8\.1](https://arxiv.org/html/2607.10034#A2.SS8.SSS1), the margin lower bound is positive precisely when

Pkey​Pval​Palign​F​log⁡Fm​d<C​Ssig,i\.e\.m​d\>1C2​F​log⁡F​\(Pkey​Pval​PalignSsig\)2\.P\_\{\\mathrm\{key\}\}\\,P\_\{\\mathrm\{val\}\}\\,P\_\{\\mathrm\{align\}\}\\sqrt\{\\frac\{F\\log F\}\{md\}\}<C\\,S\_\{\\mathrm\{sig\}\},\\qquad\\text\{i\.e\.\}\\qquad md\>\\frac\{1\}\{C^\{2\}\}\\,F\\log F\\left\(\\frac\{P\_\{\\mathrm\{key\}\}P\_\{\\mathrm\{val\}\}P\_\{\\mathrm\{align\}\}\}\{S\_\{\\mathrm\{sig\}\}\}\\right\)^\{\\\!2\}\.Hence our construction stores allFFfacts with positive margin using a parameter budget

W=m​d=Θ​\(F​log⁡F​\(Pkey​Pval​PalignSsig\)2\),W=md=\\Theta\\\!\\left\(F\\log F\\left\(\\frac\{P\_\{\\mathrm\{key\}\}P\_\{\\mathrm\{val\}\}P\_\{\\mathrm\{align\}\}\}\{S\_\{\\mathrm\{sig\}\}\}\\right\)^\{\\\!2\}\\right\),the same rate fact\-storage rate as in the isotropic setting up to the penalization factors\.

#### B\.8\.2Arbitrary Keys, Isotropic Values

###### Theorem B\.28\(Margin bound — arbitrary keys, isotropic values\)\.

Assume𝐯1,…,𝐯F\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1⊂ℝd\\mathbb\{S\}^\{d\-1\}\\subset\\mathbb\{R\}^\{d\}, and treat the kernel matrix𝐊^\\hat\{\\mathbf\{K\}\}as arbitrary\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and let

L:=log⁡\(C0​F2δ\),εv:=C1​Ld−1\.L\\ \\vcentcolon=\\ \\log\\\!\\Big\(\\frac\{C\_\{0\}F^\{2\}\}\{\\delta\}\\Big\),\\qquad\\varepsilon\_\{v\}\\ \\vcentcolon=\\ C\_\{1\}\\sqrt\{\\frac\{L\}\{d\-1\}\}\.Define the \(untruncated\) kernel\-column energy

EK:=maxi∈\[F\]​∑t≠i𝐊^t​i2\.E\_\{K\}\\ \\vcentcolon=\\ \\max\_\{i\\in\[F\]\}\\ \\sum\_\{t\\neq i\}\\hat\{\\mathbf\{K\}\}\_\{ti\}^\{2\}\.Then with probability at least1−δ1\-\\delta\(over the values\),

γmin≥Kmindiag​\(1−εv\)−Kmaxoff−C2​EK​Ld−1​1\+LF−2\.\\gamma\_\{\\min\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}\(1\-\\varepsilon\_\{v\}\)\-K\_\{\\max\}^\{\\mathrm\{off\}\}\-C\_\{2\}\\sqrt\{E\_\{K\}\}\\,\\sqrt\{\\frac\{L\}\{d\-1\}\}\\,\\sqrt\{1\+\\frac\{L\}\{F\-2\}\}\.\(45\)In particular, ifF−2≥LF\-2\\geq L, then

γmin≥Kmindiag​\(1−εv\)−Kmaxoff−C3​EK​Ld\.\\gamma\_\{\\min\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}\(1\-\\varepsilon\_\{v\}\)\-K\_\{\\max\}^\{\\mathrm\{off\}\}\\ \-\\ C\_\{3\}\\sqrt\{E\_\{K\}\}\\,\\sqrt\{\\frac\{L\}\{d\}\}\.\(46\)

###### Proof\.

The proof combines the signal and cross\-talk bounds via a union bound\.

##### Signal lower bound\.

Apply the signal bound from Theorem[B\.22](https://arxiv.org/html/2607.10034#A2.Thmtheorem22)with failure probabilityδ/2\\delta/2\. This yields an eventℰsig\\mathcal\{E\}\_\{\\mathrm\{sig\}\}such thatℙ​\(ℰsigc\)≤δ/2\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\mathrm\{sig\}\}^\{c\}\)\\leq\\delta/2and onℰsig\\mathcal\{E\}\_\{\\mathrm\{sig\}\},

s~min≥Kmindiag​\(1−εv\)−Kmaxoff​\(1\+εv\),\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ K\_\{\\min\}^\{\\mathrm\{diag\}\}\(1\-\\varepsilon\_\{v\}\)\\ \-\\ K\_\{\\max\}^\{\\mathrm\{off\}\}\(1\+\\varepsilon\_\{v\}\),\(47\)after the standard adjustment of the absolute constant hidden inL=log⁡\(C0​F2δ\)L=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)so that replacingδ\\deltabyδ/2\\delta/2only changes the constant\.

##### Cross\-talk bound\.

Apply the cross\-talk bound from Theorem[B\.17](https://arxiv.org/html/2607.10034#A2.Thmtheorem17)with failure probabilityδ/2\\delta/2\. This yields an eventℰcross\\mathcal\{E\}\_\{\\mathrm\{cross\}\}such thatℙ​\(ℰcrossc\)≤δ/2\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\mathrm\{cross\}\}^\{c\}\)\\leq\\delta/2and onℰcross\\mathcal\{E\}\_\{\\mathrm\{cross\}\},

z~max≤C​EK​Ld−1​1\+LF−2\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ C\\sqrt\{E\_\{K\}\}\\,\\sqrt\{\\frac\{L\}\{d\-1\}\}\\,\\sqrt\{1\+\\frac\{L\}\{F\-2\}\}\.\(48\)Here we used the trivial boundE~K≤EK\\widetilde\{E\}\_\{K\}\\leq E\_\{K\}, since dropping coordinates can only decrease theℓ2\\ell\_\{2\}norm\. Again we have absorbed the harmlessδ↦δ/2\\delta\\mapsto\\delta/2change into the absolute constant insideLL\.

##### Union bound and combine\.

By a union bound,

ℙ​\(ℰsig∩ℰcross\)≥1−ℙ​\(ℰsigc\)−ℙ​\(ℰcrossc\)≥1−δ\.\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\mathrm\{sig\}\}\\cap\\mathcal\{E\}\_\{\\mathrm\{cross\}\}\)\\ \\geq\\ 1\-\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\mathrm\{sig\}\}^\{c\}\)\-\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\mathrm\{cross\}\}^\{c\}\)\\ \\geq\\ 1\-\\delta\.Onℰsig∩ℰcross\\mathcal\{E\}\_\{\\mathrm\{sig\}\}\\cap\\mathcal\{E\}\_\{\\mathrm\{cross\}\}, useγmin≥s~min−z~max\\gamma\_\{\\min\}\\geq\\widetilde\{s\}\_\{\\min\}\-\\widetilde\{z\}\_\{\\max\}\. Substituting equation[47](https://arxiv.org/html/2607.10034#A2.E47)and equation[48](https://arxiv.org/html/2607.10034#A2.E48)yields

γmin\\displaystyle\\gamma\_\{\\min\}≥Kmindiag​\(1−εv\)−Kmaxoff​\(1\+εv\)−C1​EK​Ld−1​1\+LF−2\\displaystyle\\geq K\_\{\\min\}^\{\\mathrm\{diag\}\}\(1\-\\varepsilon\_\{v\}\)\-K\_\{\\max\}^\{\\mathrm\{off\}\}\(1\+\\varepsilon\_\{v\}\)\-C\_\{1\}\\sqrt\{E\_\{K\}\}\\,\\sqrt\{\\frac\{L\}\{d\-1\}\}\\,\\sqrt\{1\+\\frac\{L\}\{F\-2\}\}=Kmindiag​\(1−εv\)−Kmaxoff−\[Kmaxoff​εv\+C1​EK​Ld−1​1\+LF−2\]\.\\displaystyle=K\_\{\\min\}^\{\\mathrm\{diag\}\}\(1\-\\varepsilon\_\{v\}\)\-K\_\{\\max\}^\{\\mathrm\{off\}\}\-\\Bigg\[K\_\{\\max\}^\{\\mathrm\{off\}\}\\,\\varepsilon\_\{v\}\+C\_\{1\}\\sqrt\{E\_\{K\}\}\\,\\sqrt\{\\frac\{L\}\{d\-1\}\}\\,\\sqrt\{1\+\\frac\{L\}\{F\-2\}\}\\Bigg\]\.\(49\)

##### Absorb the competitor term\.

We claim that

Kmaxoff≤EK\.K\_\{\\max\}^\{\\mathrm\{off\}\}\\ \\leq\\ \\sqrt\{E\_\{K\}\}\.\(50\)Choose indices\(a,b\)\(a,b\)witha≠ba\\neq bsuch that\|𝐊^b​a\|=Kmaxoff\.\|\\hat\{\\mathbf\{K\}\}\_\{ba\}\|=K\_\{\\max\}^\{\\mathrm\{off\}\}\.Then by definition ofEKE\_\{K\},

EK≥∑t≠a𝐊^t​a2≥𝐊^b​a2=\(Kmaxoff\)2,E\_\{K\}\\ \\geq\\ \\sum\_\{t\\neq a\}\\hat\{\\mathbf\{K\}\}\_\{ta\}^\{2\}\\ \\geq\\ \\hat\{\\mathbf\{K\}\}\_\{ba\}^\{2\}\\ =\\ \\big\(K\_\{\\max\}^\{\\mathrm\{off\}\}\\big\)^\{2\},which proves equation[50](https://arxiv.org/html/2607.10034#A2.E50)\. Therefore,

Kmaxoff​εv≤EK​εv=C2​EK​Ld−1≤C3​EK​Ld−1​1\+LF−2,K\_\{\\max\}^\{\\mathrm\{off\}\}\\,\\varepsilon\_\{v\}\\ \\leq\\ \\sqrt\{E\_\{K\}\}\\,\\varepsilon\_\{v\}\\ =\\ C\_\{2\}\\sqrt\{E\_\{K\}\}\\,\\sqrt\{\\frac\{L\}\{d\-1\}\}\\ \\leq\\ C\_\{3\}\\sqrt\{E\_\{K\}\}\\,\\sqrt\{\\frac\{L\}\{d\-1\}\}\\,\\sqrt\{1\+\\frac\{L\}\{F\-2\}\},since1\+L/\(F−2\)≥1\\sqrt\{1\+L/\(F\-2\)\}\\geq 1\. Absorbing this into the bracket in equation[49](https://arxiv.org/html/2607.10034#A2.E49)yields the claimed bound\.

##### Simplified form whenF−2≥LF\-2\\geq L\.

IfF−2≥LF\-2\\geq L, then1\+L/\(F−2\)≤2\\sqrt\{1\+L/\(F\-2\)\}\\leq\\sqrt\{2\}\. Alsod−1≍dd\-1\\asymp dford≥2d\\geq 2, so after adjusting the absolute constant we obtain the simplified bound stated in the theorem\. ∎

#### B\.8\.3Isotropic Keys, Arbitrary Values \(Bilinear Kernel\)

###### Theorem B\.29\(Margin bound — isotropic keys, arbitrary values \(bilinear kernel\)\)\.

Assume keys𝐤1,…,𝐤F\{\\mathbf\{k\}\}\_\{1\},\\dots,\{\\mathbf\{k\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}, the kernel is the bilinear random\-feature kernel from[SectionB\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2)withmmfeatures, the values𝐯1,…,𝐯F∈ℝd\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}\\in\\mathbb\{R\}^\{d\}are deterministic, and the positivity condition⟨𝐯i−𝐯j,𝐯i⟩≥0\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{i\}\}\\rangle\\geq 0holds for alli≠ji\\neq j\. Let

μ:=1d,σ2:=𝔼​\[\(𝐊^t​i−μ\)2\]=Θ​\(1d2\+1m\),\\mu\\ \\vcentcolon=\\ \\frac\{1\}\{d\},\\qquad\\sigma^\{2\}\\ \\vcentcolon=\\ \\mathbb\{E\}\[\(\\hat\{\\mathbf\{K\}\}\_\{ti\}\-\\mu\)^\{2\}\]=\\Theta\\\!\\Big\(\\frac\{1\}\{d^\{2\}\}\+\\frac\{1\}\{m\}\\Big\),and fixδ∈\(0,1\)\\delta\\in\(0,1\)\. Set

L:=log⁡\(C0​F2δ\),εk:=LmL\\ \\vcentcolon=\\ \\log\\\!\\Big\(\\frac\{C\_\{0\}F^\{2\}\}\{\\delta\}\\Big\),\\qquad\\varepsilon\_\{k\}\\vcentcolon=\\sqrt\{\\frac\{L\}\{m\}\}and assume in addition thatF−2≥LF\-2\\geq L,m≥L3m\\geq L^\{3\}, andm≤d2/Lm\\leq d^\{2\}/L\.

Recall the centered\-sum budgetBYB\_\{Y\}\([SectionB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px2)\) and letLvL\_\{v\}be as in[SectionB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px2)\. Then with probability at least1−δ1\-\\delta\(over the keys and features\),

γmin≥\(1−C1​εk\)​Vmin−C2​\(BY\+Ev​Lv\)​\(σ​L\)\.\\gamma\_\{\\min\}\\ \\geq\\ \\Big\(1\-C\_\{1\}\\varepsilon\_\{k\}\\Big\)V\_\{\\min\}\-C\_\{2\}\\Big\(B\_\{Y\}\+\\sqrt\{E\_\{v\}\}\\,\\sqrt\{L\_\{v\}\}\\Big\)\\Big\(\\sigma\\sqrt\{L\}\\Big\)\.\(51\)

###### Proof\.

The proof combines the signal and cross\-talk bounds via a union bound\.

##### Signal lower bound\.

Apply the signal bound from Theorem[B\.23](https://arxiv.org/html/2607.10034#A2.Thmtheorem23)with failure probabilityδ/2\\delta/2\. This yields an eventℰsig\\mathcal\{E\}\_\{\\mathrm\{sig\}\}such thatℙ​\(ℰsigc\)≤δ/2\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\mathrm\{sig\}\}^\{c\}\)\\leq\\delta/2and onℰsig\\mathcal\{E\}\_\{\\mathrm\{sig\}\},

s~min≥\(1−C1​Lm\)​Vmin−\(1d\+C2​\(σ​L\+L2m\)\)​Vmax\.\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ \\Big\(1\-C\_\{1\}\\sqrt\{\\tfrac\{L\}\{m\}\}\\Big\)V\_\{\\min\}\-\\Big\(\\frac\{1\}\{d\}\+C\_\{2\}\\big\(\\sigma\\sqrt\{L\}\+\\tfrac\{L^\{2\}\}\{m\}\\big\)\\Big\)V\_\{\\max\}\.\(52\)As before, replacingδ\\deltabyδ/2\\delta/2only changes the absolute constant hidden inLL\.

##### Cross\-talk bound\.

Apply the cross\-talk bound from Theorem[B\.19](https://arxiv.org/html/2607.10034#A2.Thmtheorem19)with failure probabilityδ/2\\delta/2\. This yields an eventℰcross\\mathcal\{E\}\_\{\\mathrm\{cross\}\}such thatℙ​\(ℰcrossc\)≤δ/2\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\mathrm\{cross\}\}^\{c\}\)\\leq\\delta/2and onℰcross\\mathcal\{E\}\_\{\\mathrm\{cross\}\},

z~max≤1d​BY\+C1​E~v​\(σ​L\+L2m\)​Lv\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ \\frac\{1\}\{d\}B\_\{Y\}\\ \+\\ C\_\{1\}\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\Big\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\Big\)\\sqrt\{L\_\{v\}\}\.\(53\)After adjusting the absolute constant,

z~max≤1d​BY\+C2​E~v​\(σ​L\+L2m\)​Lv\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ \\frac\{1\}\{d\}B\_\{Y\}\\ \+\\ C\_\{2\}\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\Big\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\Big\)\\sqrt\{L\_\{v\}\}\.\(54\)

##### Union bound and combine\.

By a union bound,

ℙ​\(ℰsig∩ℰcross\)≥1−δ\.\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\mathrm\{sig\}\}\\cap\\mathcal\{E\}\_\{\\mathrm\{cross\}\}\)\\ \\geq\\ 1\-\\delta\.On this intersection, useγmin≥s~min−z~max\\gamma\_\{\\min\}\\geq\\widetilde\{s\}\_\{\\min\}\-\\widetilde\{z\}\_\{\\max\}\. Substituting equation[52](https://arxiv.org/html/2607.10034#A2.E52)and equation[54](https://arxiv.org/html/2607.10034#A2.E54)gives

γmin\\displaystyle\\gamma\_\{\\min\}≥\(1−C1​Lm\)​Vmin−1d​\(BY\+Vmax\)\\displaystyle\\geq\\Big\(1\-C\_\{1\}\\sqrt\{\\tfrac\{L\}\{m\}\}\\Big\)V\_\{\\min\}\-\\frac\{1\}\{d\}\(B\_\{Y\}\+V\_\{\\max\}\)−C2​\(Vmax\+E~v​Lv\)​\(σ​L\+L2m\)\.\\displaystyle\\hskip 16\.00008pt\-C\_\{2\}\\Big\(V\_\{\\max\}\+\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\sqrt\{L\_\{v\}\}\\Big\)\\Big\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\Big\)\.\(55\)

##### Absorb the value\-side terms\.

SinceVmax≤EvV\_\{\\max\}\\leq\\sqrt\{E\_\{v\}\}andE~v≤Ev\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\leq\\sqrt\{E\_\{v\}\}, andLv≥1L\_\{v\}\\geq 1by[SectionB\.5](https://arxiv.org/html/2607.10034#A2.SS5.SSS0.Px2)\(which holds even in the degenerate caseE~v=0\\widetilde\{E\}\_\{v\}=0, where the cross\-talk term vanishes but the signal\-sideVmaxV\_\{\\max\}need not, by the conventionLv=1L\_\{v\}=1\), we haveVmax\+E~v​Lv≤Ev​Lv\+Ev​Lv=2​Ev​LvV\_\{\\max\}\+\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\sqrt\{L\_\{v\}\}\\leq\\sqrt\{E\_\{v\}\}\\sqrt\{L\_\{v\}\}\+\\sqrt\{E\_\{v\}\}\\sqrt\{L\_\{v\}\}=2\\sqrt\{E\_\{v\}\}\\sqrt\{L\_\{v\}\}\. Therefore, after adjusting the absolute constant,

C2​\(Vmax\+E~v​Lv\)​\(σ​L\+L2m\)≤C3​Ev​Lv​\(σ​L\+L2m\)\.C\_\{2\}\\Big\(V\_\{\\max\}\+\\sqrt\{\\widetilde\{E\}\_\{v\}\}\\,\\sqrt\{L\_\{v\}\}\\Big\)\\Big\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\Big\)\\ \\leq\\ C\_\{3\}\\sqrt\{E\_\{v\}\}\\,\\sqrt\{L\_\{v\}\}\\Big\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\Big\)\.

##### Absorb thed−1d^\{\-1\}factor\.

SinceL=log⁡\(C0​F2δ\)≥1L=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)\\geq 1andσ2=Θ​\(1d2\+1m\)\\sigma^\{2\}=\\Theta\\\!\\big\(\\tfrac\{1\}\{d^\{2\}\}\+\\tfrac\{1\}\{m\}\\big\)impliesσ≳1/d\\sigma\\gtrsim 1/d, we have1d≤C​σ​L\\frac\{1\}\{d\}\\leq C\\,\\sigma\\sqrt\{L\}\. Therefore, using alsoVmax≤Ev≤Ev​LvV\_\{\\max\}\\leq\\sqrt\{E\_\{v\}\}\\leq\\sqrt\{E\_\{v\}\}\\,\\sqrt\{L\_\{v\}\},

1d​\(BY\+Vmax\)≤C1​\(BY\+Ev​Lv\)​σ​L≤C2​\(BY\+Ev​Lv\)​\(σ​L\+L2m\)\.\\frac\{1\}\{d\}\(B\_\{Y\}\+V\_\{\\max\}\)\\ \\leq\\ C\_\{1\}\\Big\(B\_\{Y\}\+\\sqrt\{E\_\{v\}\}\\,\\sqrt\{L\_\{v\}\}\\Big\)\\sigma\\sqrt\{L\}\\ \\leq\\ C\_\{2\}\\Big\(B\_\{Y\}\+\\sqrt\{E\_\{v\}\}\\,\\sqrt\{L\_\{v\}\}\\Big\)\\Big\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\Big\)\.Plugging the last two estimates into equation[55](https://arxiv.org/html/2607.10034#A2.E55)and usingm≥L3m\\geq L^\{3\}, we haveγmin≥\(1−C4​Lm\)​Vmin−C5​\(BY\+Ev​Lv\)​σ​L\\gamma\_\{\\min\}\\geq\\Big\(1\-C\_\{4\}\\sqrt\{\\tfrac\{L\}\{m\}\}\\Big\)V\_\{\\min\}\-C\_\{5\}\\Big\(B\_\{Y\}\+\\sqrt\{E\_\{v\}\}\\,\\sqrt\{L\_\{v\}\}\\Big\)\\sigma\\sqrt\{L\}, which is exactly the stated bound\. ∎

#### B\.8\.4Isotropic Keys, Isotropic Values \(Bilinear Kernel\)

###### Theorem B\.30\(Margin bound — isotropic keys, isotropic values \(bilinear kernel\)\)\.

Assume keys𝐤1,…,𝐤F\{\\mathbf\{k\}\}\_\{1\},\\dots,\{\\mathbf\{k\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}, values𝐯1,…,𝐯F\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}and independent of the keys, and the kernel is the bilinear random\-feature kernel from[SectionB\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2)withmmfeatures\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and set

L:=log⁡\(C0​F2δ\)\.L\\ \\vcentcolon=\\ \\log\\\!\\Big\(\\frac\{C\_\{0\}F^\{2\}\}\{\\delta\}\\Big\)\.AssumeF≥2​LF\\geq 2L,m≥L3m\\geq L^\{3\}, andm≤d2/Lm\\leq d^\{2\}/L\. Then, with probability at least1−δ1\-\\delta\(over keys, features, and values\), the minimum “absorbed\-competitor” margin satisfies

γmin≥1−C1​Lm−εv−C3​L​Fd3−C4​L​Fm​d,εv:=C2​Ld−1\.\\gamma\_\{\\min\}\\ \\geq\\ 1\\;\-\\;C\_\{1\}\\sqrt\{\\frac\{L\}\{m\}\}\\;\-\\;\\varepsilon\_\{v\}\\;\-\\;C\_\{3\}\\,\\sqrt\{L\}\\sqrt\{\\frac\{F\}\{d^\{3\}\}\}\\;\-\\;C\_\{4\}\\,\\sqrt\{L\}\\sqrt\{\\frac\{F\}\{md\}\},\\qquad\\varepsilon\_\{v\}\\vcentcolon=C\_\{2\}\\sqrt\{\\frac\{L\}\{d\-1\}\}\.\(56\)

###### Proof\.

##### Signal lower bound\.

Apply the signal bound from Theorem[B\.24](https://arxiv.org/html/2607.10034#A2.Thmtheorem24)with failure probabilityδ/2\\delta/2\. Sincem≥L3m\\geq L^\{3\}, this yields an eventℰsig\\mathcal\{E\}\_\{\\mathrm\{sig\}\}withℙ​\(ℰsigc\)≤δ/2\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\mathrm\{sig\}\}^\{c\}\)\\leq\\delta/2on which

s~min≥\(1−C1​Lm\)​\(1−εv\)−C4​σ​L​\(1\+εv\),\\widetilde\{s\}\_\{\\min\}\\ \\geq\\ \\Big\(1\-C\_\{1\}\\sqrt\{\\frac\{L\}\{m\}\}\\Big\)\(1\-\\varepsilon\_\{v\}\)\\ \-\\ C\_\{4\}\\,\\sigma\\sqrt\{L\}\\,\(1\+\\varepsilon\_\{v\}\),withεv\\varepsilon\_\{v\}as in the theorem statement, after the standard adjustment of the absolute constant hidden inLL\.

##### Cross\-talk bound\.

Apply the cross\-talk bound from Theorem[B\.20](https://arxiv.org/html/2607.10034#A2.Thmtheorem20)with failure probabilityδ/2\\delta/2\. SinceF≥2​LF\\geq 2LgivesF−2≥LF\-2\\geq L, this yields an eventℰcross\\mathcal\{E\}\_\{\\mathrm\{cross\}\}withℙ​\(ℰcrossc\)≤δ/2\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\mathrm\{cross\}\}^\{c\}\)\\leq\\delta/2on which

z~max≤C6​L​Fd3\+C7​L​Fm​d\.\\widetilde\{z\}\_\{\\max\}\\ \\leq\\ C\_\{6\}\\,\\sqrt\{L\}\\sqrt\{\\frac\{F\}\{d^\{3\}\}\}\\ \+\\ C\_\{7\}\\,\\sqrt\{L\}\\sqrt\{\\frac\{F\}\{md\}\}\.

##### Union bound and combine\.

By a union bound,ℙ​\(ℰsig∩ℰcross\)≥1−δ\\mathbb\{P\}\(\\mathcal\{E\}\_\{\\mathrm\{sig\}\}\\cap\\mathcal\{E\}\_\{\\mathrm\{cross\}\}\)\\geq 1\-\\delta\. On this intersection,γmin≥s~min−z~max\\gamma\_\{\\min\}\\geq\\widetilde\{s\}\_\{\\min\}\-\\widetilde\{z\}\_\{\\max\}gives

γmin≥\(1−C1​Lm\)​\(1−εv\)−C4​σ​L​\(1\+εv\)−C6​L​Fd3−C7​L​Fm​d\.\\gamma\_\{\\min\}\\ \\geq\\ \\Big\(1\-C\_\{1\}\\sqrt\{\\frac\{L\}\{m\}\}\\Big\)\(1\-\\varepsilon\_\{v\}\)\\ \-\\ C\_\{4\}\\,\\sigma\\sqrt\{L\}\\,\(1\+\\varepsilon\_\{v\}\)\\ \-\\ C\_\{6\}\\,\\sqrt\{L\}\\sqrt\{\\frac\{F\}\{d^\{3\}\}\}\\ \-\\ C\_\{7\}\\,\\sqrt\{L\}\\sqrt\{\\frac\{F\}\{md\}\}\.Sincem≤d2/Lm\\leq d^\{2\}/Lforcesσ≍1/m\\sigma\\asymp 1/\\sqrt\{m\}, andεv=O​\(1\)\\varepsilon\_\{v\}=O\(1\), the signal off\-diagonal penalty obeysC4​σ​L​\(1\+εv\)≤C​L/mC\_\{4\}\\,\\sigma\\sqrt\{L\}\\,\(1\+\\varepsilon\_\{v\}\)\\leq C\\sqrt\{L/m\}and is absorbed into the first term\. Expanding\(1−C​L/m\)​\(1−εv\)≥1−C​L/m−εv\\big\(1\-C\\sqrt\{L/m\}\\big\)\(1\-\\varepsilon\_\{v\}\)\\geq 1\-C\\sqrt\{L/m\}\-\\varepsilon\_\{v\}and renaming the absolute constants yields[Equation56](https://arxiv.org/html/2607.10034#A2.E56)\. ∎

###### Corollary B\.31\(Isotropic margin scaling — simplified\)\.

Under the assumptions of[TheoremB\.30](https://arxiv.org/html/2607.10034#A2.Thmtheorem30)andm,d≤Fm,d\\leq F, retaining only the dominant cross\-talk term gives, with probability at least1−δ1\-\\delta,

γmin≥1−C​F​log⁡\(F/δ\)m​d\.\\gamma\_\{\\min\}\\;\\geq\\;1\\;\-\\;C\\sqrt\{\\frac\{F\\log\(F/\\delta\)\}\{md\}\}\.

###### Proof\.

The dominant cross\-talk term in[Equation56](https://arxiv.org/html/2607.10034#A2.E56)isC4​L​F/\(m​d\)C\_\{4\}\\sqrt\{L\}\\sqrt\{F/\(md\)\}; under the standing assumptions, the remaining terms are lower order and are absorbed into the constantCC\. SinceL=log⁡\(C0​F2/δ\)=Θ​\(log⁡\(F/δ\)\)L=\\log\(C\_\{0\}F^\{2\}/\\delta\)=\\Theta\(\\log\(F/\\delta\)\), we haveL​F/\(m​d\)=Θ​\(F​log⁡\(F/δ\)/\(m​d\)\)\\sqrt\{L\}\\sqrt\{F/\(md\)\}=\\Theta\\\!\\big\(\\sqrt\{F\\log\(F/\\delta\)/\(md\)\}\\big\), which yields the stated form\. ∎

###### Corollary B\.32\(Isotropic fact\-storage capacity\)\.

Under the assumptions of[SectionB\.8\.4](https://arxiv.org/html/2607.10034#A2.SS8.SSS4.Px3), the margin lower bound is positive precisely when

C​F​log⁡\(F/δ\)m​d<1,i\.e\.m​d\>C2​F​log⁡\(F/δ\)\.C\\sqrt\{\\frac\{F\\log\(F/\\delta\)\}\{md\}\}<1,\\qquad\\text\{i\.e\.\}\\qquad md\>C^\{2\}\\,F\\log\(F/\\delta\)\.Hence our construction stores allFFfacts with positive margin using a parameter budget

W=m​d=Θ​\(F​log⁡F\),W=md=\\Theta\\\!\\left\(F\\log F\\right\),the information\-theoretically optimal fact\-storage rate\.

#### B\.8\.5Summary Table: Margin Bounds Across Regimes

RandomkeysArbitrarykeysRandomvaluesγmin≥1−C​F​Lm​d\\begin\{aligned\} \\gamma\_\{\\min\}&\\geq\\;1\\\\ &\-\\,C\\sqrt\{\\frac\{FL\}\{md\}\}\\end\{aligned\}γmin≥Kmindiag​\(1−εv\)−Kmaxoff−C​EK​Ld\\begin\{aligned\} \\gamma\_\{\\min\}&\\geq\\;K\_\{\\min\}^\{\\mathrm\{diag\}\}\(1\-\\varepsilon\_\{v\}\)\-K\_\{\\max\}^\{\\mathrm\{off\}\}\\\\ &\-\\,C\\sqrt\{E\_\{K\}\}\\,\\sqrt\{\\frac\{L\}\{d\}\}\\end\{aligned\}Arbitraryvaluesγmin≥\(1−εk\)​Vmin−C​\(BY\+Ev​Lv\)​Lm\\begin\{aligned\} \\gamma\_\{\\min\}&\\geq\\;\\Big\(1\-\\varepsilon\_\{k\}\\Big\)V\_\{\\min\}\\\\ &\-\\,C\\\!\\left\(B\_\{Y\}\+\\sqrt\{E\_\{v\}\}\\sqrt\{L\_\{v\}\}\\right\)\\\!\\sqrt\{\\frac\{L\}\{m\}\}\\end\{aligned\}γmin≥Kmindiag​Vmin−EK​Ev​κ\\begin\{aligned\} \\gamma\_\{\\min\}&\\geq\\;K\_\{\\min\}^\{\\mathrm\{diag\}\}V\_\{\\min\}\\\\ &\-\\,\\sqrt\{E\_\{K\}\}\\,\\sqrt\{E\_\{v\}\}\\,\\kappa\\end\{aligned\}Table 2:Margin bounds across key/value geometry regimes\.Each cell shows the decoding margin scaling for a given key/value geometry\. The top\-left \(isotropic\) cell is the baseline\. Relaxing the key geometry introduces key\-geometry statistics \(KmindiagK\_\{\\min\}^\{\\mathrm\{diag\}\},KmaxoffK\_\{\\max\}^\{\\mathrm\{off\}\},EKE\_\{K\}\); relaxing value geometry introduces value\-geometry statistics \(VminV\_\{\\min\},BYB\_\{Y\},VmaxV\_\{\\max\},EvE\_\{v\},LvL\_\{v\}\); relaxing both geometries introduces a coupling factorκ\\kappa\. The arbitrary keys and values cell uses the raw summary\-statistic form of[TheoremB\.25](https://arxiv.org/html/2607.10034#A2.Thmtheorem25), which is equivalent to the penalty\-statistic general margin/capacity bound in[SectionB\.8\.1](https://arxiv.org/html/2607.10034#A2.SS8.SSS1)of the main text\. See Theorems[B\.25](https://arxiv.org/html/2607.10034#A2.Thmtheorem25),[B\.28](https://arxiv.org/html/2607.10034#A2.Thmtheorem28),[B\.29](https://arxiv.org/html/2607.10034#A2.Thmtheorem29)and[B\.30](https://arxiv.org/html/2607.10034#A2.Thmtheorem30)for formal statements and Appendix[A\.3\.2](https://arxiv.org/html/2607.10034#A1.SS3.SSS2)for empirical scaling results\.
#### B\.8\.6A Welch/Frobenius Upper Bound in the Isotropic Key/Value Regime

In the main text,[Theorem4\.3](https://arxiv.org/html/2607.10034#S4.Thmtheorem3)provides a lower bound on the minimum margin achieved by our construction in the isotropic key/value regime\. We now show that this lower bound is asymptotically tight, up to constants and logarithmic factors, by proving a matching*upper bound*on the best possible margin of any admissible rank\-limited kernel memory in this regime\. The key idea is that the arbitrary\-keys / isotropic\-values analysis in[SectionB\.8\.2](https://arxiv.org/html/2607.10034#A2.SS8.SSS2)is controlled by the column energy

Ecol​\(i\):=∑t≠i𝐊^t​i2\.E\_\{\\mathrm\{col\}\}\(i\):=\\sum\_\{t\\neq i\}\\hat\{\\mathbf\{K\}\}\_\{ti\}^\{2\}\.Thus, to upper bound the achievable margin, it suffices to show that for any rank\-mmkernel, some column must have nontrivial off\-diagonal squared mass\. We achieve this using a Welch / Frobenius argument: intuitively, low rank forces a non\-negligible amount of kernel mass off the diagonal, which in turn yields an unavoidable cross\-talk floor\.

We begin by formalizing the class of kernels under consideration\.

###### Definition B\.33\(Rank\-mmkernel\)\.

Let𝐤1,…,𝐤F∈ℝd\{\\mathbf\{k\}\}\_\{1\},\\dots,\{\\mathbf\{k\}\}\_\{F\}\\in\\mathbb\{R\}^\{d\}be the stored keys\. We say that𝐊^∈ℝF×F\\hat\{\\mathbf\{K\}\}\\in\\mathbb\{R\}^\{F\\times F\}is a*rank\-mmkernel*if there exists a feature mapϕ​\(⋅\)\\phi\(\\cdot\)withϕ​\(𝐤i\)∈ℝm\\phi\(\{\\mathbf\{k\}\}\_\{i\}\)\\in\\mathbb\{R\}^\{m\}such that

𝐊^i​j=⟨ϕ​\(𝐤i\),ϕ​\(𝐤j\)⟩for all​i,j∈\[F\]\.\\hat\{\\mathbf\{K\}\}\_\{ij\}=\\langle\\phi\(\{\\mathbf\{k\}\}\_\{i\}\),\\phi\(\{\\mathbf\{k\}\}\_\{j\}\)\\rangle\\qquad\\text\{for all \}i,j\\in\[F\]\.Equivalently,𝐊^⪰0\\hat\{\\mathbf\{K\}\}\\succeq 0andrank⁡\(𝐊^\)≤m\\operatorname\{rank\}\(\\hat\{\\mathbf\{K\}\}\)\\leq m\.

The following lemma is the kernel\-side ingredient\. It lower bounds the maximum column energy of any PSD rank\-mmkernel in terms of its diagonal\.

###### Lemma B\.34\(Welch/Frobenius lower bound for rank\-mmkernels\)\.

Let𝐊^∈ℝF×F\\hat\{\\mathbf\{K\}\}\\in\\mathbb\{R\}^\{F\\times F\}be a rank\-mmkernel in the sense of Definition[B\.8\.6](https://arxiv.org/html/2607.10034#A2.SS8.SSS6)\. Define

Ecol​\(i\):=∑t≠i𝐊^t​i2,Ecol:=maxi∈\[F\]⁡Ecol​\(i\)\.E\_\{\\mathrm\{col\}\}\(i\):=\\sum\_\{t\\neq i\}\\hat\{\\mathbf\{K\}\}\_\{ti\}^\{2\},\\qquad E\_\{\\mathrm\{col\}\}:=\\max\_\{i\\in\[F\]\}E\_\{\\mathrm\{col\}\}\(i\)\.Then

‖𝐊^‖F2≥tr\(𝐊^\)2m,\\\|\\hat\{\\mathbf\{K\}\}\\\|\_\{F\}^\{2\}\\geq\\frac\{\\operatorname\{tr\}\(\\hat\{\\mathbf\{K\}\}\)^\{2\}\}\{m\},and consequently

Ecol≥1F​\(tr\(𝐊^\)2m−∑i=1F𝐊^i​i2\)\.E\_\{\\mathrm\{col\}\}\\;\\geq\\;\\frac\{1\}\{F\}\\left\(\\frac\{\\operatorname\{tr\}\(\\hat\{\\mathbf\{K\}\}\)^\{2\}\}\{m\}\-\\sum\_\{i=1\}^\{F\}\\hat\{\\mathbf\{K\}\}\_\{ii\}^\{2\}\\right\)\.In particular, if

𝐊^i​i∈\[1−εdiag,1\+εdiag\]for all​i∈\[F\],\\hat\{\\mathbf\{K\}\}\_\{ii\}\\in\[1\-\\varepsilon\_\{\\mathrm\{diag\}\},\\,1\+\\varepsilon\_\{\\mathrm\{diag\}\}\]\\qquad\\text\{for all \}i\\in\[F\],then

Ecol≥F​\(1−εdiag\)2m−\(1\+εdiag\)2\.E\_\{\\mathrm\{col\}\}\\;\\geq\\;\\frac\{F\(1\-\\varepsilon\_\{\\mathrm\{diag\}\}\)^\{2\}\}\{m\}\-\(1\+\\varepsilon\_\{\\mathrm\{diag\}\}\)^\{2\}\.

###### Proof\.

Since𝐊^⪰0\\hat\{\\mathbf\{K\}\}\\succeq 0andrank⁡\(𝐊^\)≤m\\operatorname\{rank\}\(\\hat\{\\mathbf\{K\}\}\)\\leq m, letλ1,…,λr\\lambda\_\{1\},\\dots,\\lambda\_\{r\}be the nonzero eigenvalues of𝐊^\\hat\{\\mathbf\{K\}\}, wherer≤mr\\leq m\. Then

tr⁡\(𝐊^\)=∑a=1rλa,‖𝐊^‖F2=∑a=1rλa2\.\\operatorname\{tr\}\(\\hat\{\\mathbf\{K\}\}\)=\\sum\_\{a=1\}^\{r\}\\lambda\_\{a\},\\qquad\\\|\\hat\{\\mathbf\{K\}\}\\\|\_\{F\}^\{2\}=\\sum\_\{a=1\}^\{r\}\\lambda\_\{a\}^\{2\}\.Applying Cauchy–Schwarz to\(λ1,…,λr\)\(\\lambda\_\{1\},\\dots,\\lambda\_\{r\}\)gives

\(∑a=1rλa\)2≤r​∑a=1rλa2≤m​∑a=1rλa2,\\Big\(\\sum\_\{a=1\}^\{r\}\\lambda\_\{a\}\\Big\)^\{2\}\\leq r\\sum\_\{a=1\}^\{r\}\\lambda\_\{a\}^\{2\}\\leq m\\sum\_\{a=1\}^\{r\}\\lambda\_\{a\}^\{2\},hence

‖𝐊^‖F2≥tr\(𝐊^\)2m\.\\\|\\hat\{\\mathbf\{K\}\}\\\|\_\{F\}^\{2\}\\geq\\frac\{\\operatorname\{tr\}\(\\hat\{\\mathbf\{K\}\}\)^\{2\}\}\{m\}\.
Next expand the Frobenius norm entrywise:

‖𝐊^‖F2=∑i=1F𝐊^i​i2\+∑i=1F∑t≠i𝐊^t​i2=∑i=1F𝐊^i​i2\+∑i=1FEcol​\(i\)\.\\\|\\hat\{\\mathbf\{K\}\}\\\|\_\{F\}^\{2\}=\\sum\_\{i=1\}^\{F\}\\hat\{\\mathbf\{K\}\}\_\{ii\}^\{2\}\+\\sum\_\{i=1\}^\{F\}\\sum\_\{t\\neq i\}\\hat\{\\mathbf\{K\}\}\_\{ti\}^\{2\}=\\sum\_\{i=1\}^\{F\}\\hat\{\\mathbf\{K\}\}\_\{ii\}^\{2\}\+\\sum\_\{i=1\}^\{F\}E\_\{\\mathrm\{col\}\}\(i\)\.Therefore

∑i=1FEcol​\(i\)≥tr\(𝐊^\)2m−∑i=1F𝐊^i​i2\.\\sum\_\{i=1\}^\{F\}E\_\{\\mathrm\{col\}\}\(i\)\\geq\\frac\{\\operatorname\{tr\}\(\\hat\{\\mathbf\{K\}\}\)^\{2\}\}\{m\}\-\\sum\_\{i=1\}^\{F\}\\hat\{\\mathbf\{K\}\}\_\{ii\}^\{2\}\.Since the maximum is at least the average,

Ecol=maxi⁡Ecol​\(i\)≥1F​∑i=1FEcol​\(i\)≥1F​\(tr\(𝐊^\)2m−∑i=1F𝐊^i​i2\)\.E\_\{\\mathrm\{col\}\}=\\max\_\{i\}E\_\{\\mathrm\{col\}\}\(i\)\\geq\\frac\{1\}\{F\}\\sum\_\{i=1\}^\{F\}E\_\{\\mathrm\{col\}\}\(i\)\\geq\\frac\{1\}\{F\}\\left\(\\frac\{\\operatorname\{tr\}\(\\hat\{\\mathbf\{K\}\}\)^\{2\}\}\{m\}\-\\sum\_\{i=1\}^\{F\}\\hat\{\\mathbf\{K\}\}\_\{ii\}^\{2\}\\right\)\.
If moreover𝐊^i​i∈\[1−εdiag,1\+εdiag\]\\hat\{\\mathbf\{K\}\}\_\{ii\}\\in\[1\-\\varepsilon\_\{\\mathrm\{diag\}\},1\+\\varepsilon\_\{\\mathrm\{diag\}\}\]for allii, then

tr⁡\(𝐊^\)=∑i=1F𝐊^i​i≥F​\(1−εdiag\)\\operatorname\{tr\}\(\\hat\{\\mathbf\{K\}\}\)=\\sum\_\{i=1\}^\{F\}\\hat\{\\mathbf\{K\}\}\_\{ii\}\\geq F\(1\-\\varepsilon\_\{\\mathrm\{diag\}\}\)and

∑i=1F𝐊^i​i2≤F​\(1\+εdiag\)2\.\\sum\_\{i=1\}^\{F\}\\hat\{\\mathbf\{K\}\}\_\{ii\}^\{2\}\\leq F\(1\+\\varepsilon\_\{\\mathrm\{diag\}\}\)^\{2\}\.Substituting these into the previous display yields

Ecol≥1F​\(F2​\(1−εdiag\)2m−F​\(1\+εdiag\)2\)=F​\(1−εdiag\)2m−\(1\+εdiag\)2\.E\_\{\\mathrm\{col\}\}\\geq\\frac\{1\}\{F\}\\left\(\\frac\{F^\{2\}\(1\-\\varepsilon\_\{\\mathrm\{diag\}\}\)^\{2\}\}\{m\}\-F\(1\+\\varepsilon\_\{\\mathrm\{diag\}\}\)^\{2\}\\right\)=\\frac\{F\(1\-\\varepsilon\_\{\\mathrm\{diag\}\}\)^\{2\}\}\{m\}\-\(1\+\\varepsilon\_\{\\mathrm\{diag\}\}\)^\{2\}\.∎

Regarding the assumption

𝐊^i​i∈\[1−εdiag,1\+εdiag\],\\hat\{\\mathbf\{K\}\}\_\{ii\}\\in\[1\-\\varepsilon\_\{\\mathrm\{diag\}\},1\+\\varepsilon\_\{\\mathrm\{diag\}\}\],note that for the exact quadratic kernel on unit\-norm isotropic keys, one has

K2​\(𝐤i,𝐤i\)=⟨𝐤i,𝐤i⟩2=1K\_\{2\}\(\{\\mathbf\{k\}\}\_\{i\},\{\\mathbf\{k\}\}\_\{i\}\)=\\langle\{\\mathbf\{k\}\}\_\{i\},\{\\mathbf\{k\}\}\_\{i\}\\rangle^\{2\}=1exactly\. For the sketched / bilinear random\-feature kernel used in our construction, the diagonal is generally not exactly11, but it instead concentrates near11with high probability by[SectionB\.9\.2](https://arxiv.org/html/2607.10034#A2.SS9.SSS2)\.

Combining the kernel\-side lower bound with the isotropic\-value competitor argument used in the proof of[Section4\.2\.1](https://arxiv.org/html/2607.10034#S4.SS2.SSS1)yields the desired asymptotic optimality statement\.

###### Corollary B\.35\(Rank\-mmkernels are asymptotically unbeatable in the isotropic key/value regime\)\.

Assume the isotropic key/value setting, and let𝐊^\\hat\{\\mathbf\{K\}\}be any rank\-mmkernel independent of the values, with

𝐊^i​i∈\[1−εdiag,1\+εdiag\]for all​i∈\[F\]\.\\hat\{\\mathbf\{K\}\}\_\{ii\}\\in\[1\-\\varepsilon\_\{\\mathrm\{diag\}\},\\,1\+\\varepsilon\_\{\\mathrm\{diag\}\}\]\\qquad\\text\{for all \}i\\in\[F\]\.Then the isotropic\-value competitor argument used in the proof of[Section4\.2\.1](https://arxiv.org/html/2607.10034#S4.SS2.SSS1)yields

γmin≤\(1\+εdiag\)−Ω​\(Ecol​log⁡Fd\)\+𝒪​\(log⁡Fd\)\+𝒪​\(Ecol​log⁡\(1/δ\)d\)\+𝒪​\(Ecold\)\.\\gamma\_\{\\min\}\\;\\leq\\;\(1\+\\varepsilon\_\{\\mathrm\{diag\}\}\)\-\\Omega\\\!\\left\(\\sqrt\{\\frac\{E\_\{\\mathrm\{col\}\}\\log F\}\{d\}\}\\right\)\+\\mathcal\{O\}\\\!\\left\(\\sqrt\{\\frac\{\\log F\}\{d\}\}\\right\)\+\\mathcal\{O\}\\\!\\left\(\\sqrt\{\\frac\{E\_\{\\mathrm\{col\}\}\\log\(1/\\delta\)\}\{d\}\}\\right\)\+\\mathcal\{O\}\\\!\\left\(\\sqrt\{\\frac\{E\_\{\\mathrm\{col\}\}\}\{d\}\}\\right\)\.Combining with Lemma[B\.8\.6](https://arxiv.org/html/2607.10034#A2.SS8.SSS6)yields

γmin≤\(1\+εdiag\)−Ω​\(\(F​\(1−εdiag\)2m−\(1\+εdiag\)2\)​log⁡Fd\)\+lower\-order terms\.\\gamma\_\{\\min\}\\;\\leq\\;\(1\+\\varepsilon\_\{\\mathrm\{diag\}\}\)\-\\Omega\\\!\\left\(\\sqrt\{\\frac\{\\big\(\\frac\{F\(1\-\\varepsilon\_\{\\mathrm\{diag\}\}\)^\{2\}\}\{m\}\-\(1\+\\varepsilon\_\{\\mathrm\{diag\}\}\)^\{2\}\\big\)\\log F\}\{d\}\}\\right\)\+\\text\{lower\-order terms\}\.In particular, ifεdiag=o​\(1\)\\varepsilon\_\{\\mathrm\{diag\}\}=o\(1\)andF/m→∞F/m\\to\\infty, then up to constants and logarithmic factors,

γmin≤1−Ω​\(\(F/m\)​log⁡Fd\)\.\\gamma\_\{\\min\}\\;\\leq\\;1\-\\Omega\\\!\\left\(\\sqrt\{\\frac\{\(F/m\)\\log F\}\{d\}\}\\right\)\.Equivalently, no admissible rank\-mmkernel can asymptotically beat the\(F/m\)​log⁡F/d\\sqrt\{\(F/m\)\\log F/d\}margin floor in the isotropic key/value regime, up to constants and logs\.

###### Proof\.

Apply the isotropic\-value competitor argument used in the proof of[Section4\.2\.1](https://arxiv.org/html/2607.10034#S4.SS2.SSS1)to the columni⋆i^\{\\star\}withEcol​\(i⋆\)=EcolE\_\{\\mathrm\{col\}\}\(i^\{\\star\}\)=E\_\{\\mathrm\{col\}\}\. This gives

γmin≤𝐊^i⋆​i⋆−Ω​\(Ecol​log⁡Fd\)\+𝒪​\(\|𝐊^i⋆​i⋆\|​log⁡Fd\)\+𝒪​\(Ecol​log⁡\(1/δ\)d\)\+𝒪​\(Ecold\),\\gamma\_\{\\min\}\\leq\\hat\{\\mathbf\{K\}\}\_\{i^\{\\star\}i^\{\\star\}\}\-\\Omega\\\!\\left\(\\sqrt\{\\frac\{E\_\{\\mathrm\{col\}\}\\log F\}\{d\}\}\\right\)\+\\mathcal\{O\}\\\!\\left\(\|\\hat\{\\mathbf\{K\}\}\_\{i^\{\\star\}i^\{\\star\}\}\|\\sqrt\{\\frac\{\\log F\}\{d\}\}\\right\)\+\\mathcal\{O\}\\\!\\left\(\\sqrt\{\\frac\{E\_\{\\mathrm\{col\}\}\\log\(1/\\delta\)\}\{d\}\}\\right\)\+\\mathcal\{O\}\\\!\\left\(\\sqrt\{\\frac\{E\_\{\\mathrm\{col\}\}\}\{d\}\}\\right\),after absorbing harmless constant\-factor differences into the big\-𝒪\\mathcal\{O\}/big\-Ω\\Omeganotation\.

Using𝐊^i⋆​i⋆≤1\+εdiag\\hat\{\\mathbf\{K\}\}\_\{i^\{\\star\}i^\{\\star\}\}\\leq 1\+\\varepsilon\_\{\\mathrm\{diag\}\}gives the first bound\. The second bound results from substituting the lower bound onEcolE\_\{\\mathrm\{col\}\}from[SectionB\.8\.6](https://arxiv.org/html/2607.10034#A2.SS8.SSS6)\.

Finally, ifεdiag=o​\(1\)\\varepsilon\_\{\\mathrm\{diag\}\}=o\(1\), then

F​\(1−εdiag\)2m−\(1\+εdiag\)2=Θ​\(Fm\)\\frac\{F\(1\-\\varepsilon\_\{\\mathrm\{diag\}\}\)^\{2\}\}\{m\}\-\(1\+\\varepsilon\_\{\\mathrm\{diag\}\}\)^\{2\}=\\Theta\\\!\\left\(\\frac\{F\}\{m\}\\right\)wheneverF/m→∞F/m\\to\\infty, so the dominant negative term scales as

\(F/m\)​log⁡Fd,\\sqrt\{\\frac\{\(F/m\)\\log F\}\{d\}\},which proves the last claim\. ∎

### B\.9Auxiliary Results

#### B\.9\.1Signal Bounds: Isotropic Values

###### Lemma B\.36\(Isotropic inner\-product concentration\)\.

Let𝐯∼Unif​\(𝕊d−1\)\{\\mathbf\{v\}\}\\sim\\mathrm\{Unif\}\(\\mathbb\{S\}^\{d\-1\}\)withd≥2d\\geq 2\. Then for any fixed𝐰∈ℝd\{\\mathbf\{w\}\}\\in\\mathbb\{R\}^\{d\}and allt≥0t\\geq 0,‖⟨𝐰,𝐯⟩‖ψ2≲‖𝐰‖2/d−1\\\|\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangle\\\|\_\{\\psi\_\{2\}\}\\lesssim\\\|\{\\mathbf\{w\}\}\\\|\_\{2\}/\\sqrt\{d\-1\}and‖⟨𝐰,𝐯⟩2−𝔼​\[⟨𝐰,𝐯⟩2\]‖ψ1≲‖𝐰‖22d−1\\big\\\|\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangle^\{2\}\-\\mathbb\{E\}\[\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangle^\{2\}\]\\big\\\|\_\{\\psi\_\{1\}\}\\ \\lesssim\\ \\frac\{\\\|\{\\mathbf\{w\}\}\\\|\_\{2\}^\{2\}\}\{d\-1\}\. I\.e\.⟨𝐰,𝐯⟩\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangleis sub\-Gaussian and⟨𝐰,𝐯⟩2−𝔼​\[⟨𝐰,𝐯⟩2\]\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangle^\{2\}\-\\mathbb\{E\}\[\\langle\{\{\\mathbf\{w\}\}\},\{\{\\mathbf\{v\}\}\}\\rangle^\{2\}\]is sub\-exponential, with the corresponding scale parameters\.

###### Proof\.

This is a standard consequence of concentration of measure on the sphere, seeVershynin \([2026](https://arxiv.org/html/2607.10034#bib.bib89)\)\. ∎

###### Lemma B\.37\(Max pairwise inner product for isotropic values\)\.

Assume𝐯1,…,𝐯F\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}withd≥2d\\geq 2\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and set

L:=log⁡\(C0​F2δ\)L\\ \\vcentcolon=\\ \\log\\Big\(\\frac\{C\_\{0\}F^\{2\}\}\{\\delta\}\\Big\)\(57\)for a sufficiently large absolute constantC0C\_\{0\}\. Then with probability at least1−δ1\-\\delta,

maxi≠j⁡\|⟨𝐯i,𝐯j⟩\|≤C1​Ld−1\.\\max\_\{i\\neq j\}\\big\|\\langle\{\{\\mathbf\{v\}\}\_\{i\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\\big\|\\ \\leq\\ C\_\{1\}\\,\\sqrt\{\\frac\{L\}\{d\-1\}\}\.\(58\)

###### Proof\.

Fix a pair\(i,j\)\(i,j\)withi≠ji\\neq j\. Condition on𝐯i\{\\mathbf\{v\}\}\_\{i\}and defineℱi:=σ​\(𝐯i\)\\mathcal\{F\}\_\{i\}\\vcentcolon=\\sigma\(\{\\mathbf\{v\}\}\_\{i\}\)\. Then𝐯j\{\\mathbf\{v\}\}\_\{j\}is still uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}and independent ofℱi\\mathcal\{F\}\_\{i\}\. Applying Lemma[B\.9\.1](https://arxiv.org/html/2607.10034#A2.SS9.SSS1)with𝐰=𝐯i\{\\mathbf\{w\}\}=\{\\mathbf\{v\}\}\_\{i\}\(note‖𝐯i‖2=1\\\|\{\\mathbf\{v\}\}\_\{i\}\\\|\_\{2\}=1\) yields that conditional onℱi\\mathcal\{F\}\_\{i\}, the scalar⟨𝐯i,𝐯j⟩\\langle\{\{\\mathbf\{v\}\}\_\{i\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangleis centered sub\-Gaussian with scale≲1/d−1\\lesssim 1/\\sqrt\{d\-1\}\. Equivalently, there exist absolute constantsc,C0\>0c,C\_\{0\}\>0such that for allt≥0t\\geq 0,

ℙ​\(\|⟨𝐯i,𝐯j⟩\|≥t\|ℱi\)≤2​e−c​\(d−1\)​t2\.\\mathbb\{P\}\\Big\(\\big\|\\langle\{\{\\mathbf\{v\}\}\_\{i\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\\big\|\\geq t\\ \\Big\|\\ \\mathcal\{F\}\_\{i\}\\Big\)\\ \\leq\\ 2e^\{\-c\(d\-1\)t^\{2\}\}\.\(59\)Since the bound from equation[59](https://arxiv.org/html/2607.10034#A2.E59)does not depend on the realized𝐯i\{\\mathbf\{v\}\}\_\{i\}, it also holds unconditionally:

ℙ​\(\|⟨𝐯i,𝐯j⟩\|≥t\)≤2​e−c​\(d−1\)​t2\.\\mathbb\{P\}\\Big\(\\big\|\\langle\{\{\\mathbf\{v\}\}\_\{i\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\\big\|\\geq t\\Big\)\\ \\leq\\ 2e^\{\-c\(d\-1\)t^\{2\}\}\.
Now take a union bound over all pairs\(i,j\)\(i,j\)withi≠ji\\neq j\(at mostF2F^\{2\}pairs\):

ℙ​\(maxi≠j⁡\|⟨𝐯i,𝐯j⟩\|≥t\)≤∑i≠jℙ​\(\|⟨𝐯i,𝐯j⟩\|≥t\)≤2​F2​e−c​\(d−1\)​t2\.\\mathbb\{P\}\\Big\(\\max\_\{i\\neq j\}\\big\|\\langle\{\{\\mathbf\{v\}\}\_\{i\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\\big\|\\geq t\\Big\)\\ \\leq\\ \\sum\_\{i\\neq j\}\\mathbb\{P\}\\Big\(\\big\|\\langle\{\{\\mathbf\{v\}\}\_\{i\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\\big\|\\geq t\\Big\)\\ \\leq\\ 2F^\{2\}e^\{\-c\(d\-1\)t^\{2\}\}\.Chooset:=C​L/\(d−1\)t\\vcentcolon=C\\sqrt\{L/\(d\-1\)\}withCClarge enough so that2​F2​e\(−c​\(d−1\)​t2\)≤δ2F^\{2\}e^\{\(\-c\(d\-1\)t^\{2\}\)\}\\leq\\delta\. This yields equation[58](https://arxiv.org/html/2607.10034#A2.E58)\. ∎

#### B\.9\.2Signal Bounds: Isotropic Keys and Bilinear Random Features

###### Lemma B\.38\(Uniform off\-diagonal bound \(isotropic keys, bilinear kernel\)\)\.

Assume keys are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}and the kernel is the bilinear random\-feature kernel from[SectionB\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2)withmmfeatures\. Letμ:=𝔼​\[𝐊^i​j\]=1/d\\mu\\vcentcolon=\\mathbb\{E\}\[\\hat\{\\mathbf\{K\}\}\_\{ij\}\]=1/dfori≠ji\\neq jand letσ2:=𝔼​\[\(𝐊^i​j−μ\)2\]\\sigma^\{2\}\\vcentcolon=\\mathbb\{E\}\[\(\\hat\{\\mathbf\{K\}\}\_\{ij\}\-\\mu\)^\{2\}\]as in Lemma[B\.9\.4](https://arxiv.org/html/2607.10034#A2.SS9.SSS4)\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and defineL:=log⁡\(C0​F2δ\)L\\vcentcolon=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)\. Assume thatm≤d2/Lm\\leq d^\{2\}/L\. Then with probability at least1−δ1\-\\delta\(over keys and features\),

maxi≠j⁡\|𝐊^i​j−μ\|≤C1​\(σ​L\+L2m\),\\max\_\{i\\neq j\}\\big\|\\hat\{\\mathbf\{K\}\}\_\{ij\}\-\\mu\\big\|\\ \\leq\\ C\_\{1\}\\Big\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\Big\),\(60\)and hence

Kmaxoff=maxi≠j⁡\|𝐊^i​j\|≤μ\+C2​\(σ​L\+L2m\)\.K\_\{\\max\}^\{\\mathrm\{off\}\}=\\max\_\{i\\neq j\}\|\\hat\{\\mathbf\{K\}\}\_\{ij\}\|\\ \\leq\\ \\mu\\ \+\\ C\_\{2\}\\Big\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\Big\)\.\(61\)

###### Proof\.

For any fixed pair\(i,j\)\(i,j\)withi≠ji\\neq j, apply Lemma[B\.9\.4](https://arxiv.org/html/2607.10034#A2.SS9.SSS4)with failure probabilityδ′≔δ/F2\\delta^\{\\prime\}\\coloneqq\\delta/F^\{2\}\. WithL′≔log⁡\(6/δ′\)=log⁡\(6​F2/δ\)≤LL^\{\\prime\}\\coloneqq\\log\(6/\\delta^\{\\prime\}\)=\\log\(6F^\{2\}/\\delta\)\\leq L\(forC0C\_\{0\}large enough\), the assumptionm≤d2/Lm\\leq d^\{2\}/Limpliesm≤d2/L′m\\leq d^\{2\}/L^\{\\prime\}, so Lemma[B\.9\.4](https://arxiv.org/html/2607.10034#A2.SS9.SSS4)yields

\|𝐊^i​j−μ\|≤C​\(L′m\+\(L′\)2m\+L′d\)\\big\|\\hat\{\\mathbf\{K\}\}\_\{ij\}\-\\mu\\big\|\\leq C\\Big\(\\sqrt\{\\tfrac\{L^\{\\prime\}\}\{m\}\}\+\\tfrac\{\(L^\{\\prime\}\)^\{2\}\}\{m\}\+\\tfrac\{\\sqrt\{L^\{\\prime\}\}\}\{d\}\\Big\)with probability at least1−δ′1\-\\delta^\{\\prime\}\. A union bound over the at mostF2F^\{2\}off\-diagonal pairs gives that, with probability at least1−δ1\-\\delta,

maxi≠j⁡\|𝐊^i​j−μ\|≤C​\(Lm\+L2m\+Ld\)\.\\max\_\{i\\neq j\}\\big\|\\hat\{\\mathbf\{K\}\}\_\{ij\}\-\\mu\\big\|\\leq C\\Big\(\\sqrt\{\\tfrac\{L\}\{m\}\}\+\\tfrac\{L^\{2\}\}\{m\}\+\\tfrac\{\\sqrt\{L\}\}\{d\}\\Big\)\.Finally, sinceσ2=Θ​\(1/d2\+1/m\)\\sigma^\{2\}=\\Theta\(1/d^\{2\}\+1/m\)by Lemma[B\.9\.4](https://arxiv.org/html/2607.10034#A2.SS9.SSS4), we haveL/m\+L/d≲σ​L\\sqrt\{L/m\}\+\\sqrt\{L\}/d\\lesssim\\sigma\\sqrt\{L\}, proving equation[60](https://arxiv.org/html/2607.10034#A2.E60)\. Equation equation[61](https://arxiv.org/html/2607.10034#A2.E61)follows by the triangle inequality\. ∎

###### Lemma B\.39\(Uniform diagonal lower bound \(bilinear kernel\)\)\.

Assume the bilinear random\-feature kernel from[SectionB\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2)withmmfeatures, and assume‖𝐤i‖2=1\\\|\{\\mathbf\{k\}\}\_\{i\}\\\|\_\{2\}=1\. Then there exist absolute constantsc,C0\>0c,C\_\{0\}\>0such that for everyt∈\(0,1\)t\\in\(0,1\)and everyi∈\[F\]i\\in\[F\],

ℙ​\(𝐊^i​i≤1−t\)≤e−c​m​t2\.\\mathbb\{P\}\\big\(\\hat\{\\mathbf\{K\}\}\_\{ii\}\\leq 1\-t\\big\)\\ \\leq\\ e^\{\-cmt^\{2\}\}\.\(62\)Consequently, for anyδ∈\(0,1\)\\delta\\in\(0,1\)andL:=log⁡\(C0​F2δ\)L\\vcentcolon=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\), with probability at least1−δ1\-\\delta,

Kmindiag=mini∈\[F\]⁡𝐊^i​i≥1−C1​Lm\.K\_\{\\min\}^\{\\mathrm\{diag\}\}=\\min\_\{i\\in\[F\]\}\\hat\{\\mathbf\{K\}\}\_\{ii\}\\ \\geq\\ 1\\ \-\\ C\_\{1\}\\sqrt\{\\frac\{L\}\{m\}\}\.\(63\)

###### Proof\.

Fixiiand write

𝐊^i​i=1m​∑r=1mZr,Zr:=\(\(𝐚r⊤​𝐤i\)​\(𝐛r⊤​𝐤i\)\)2=\(Gr​Hr\)2,\\hat\{\\mathbf\{K\}\}\_\{ii\}=\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}Z\_\{r\},\\qquad Z\_\{r\}:=\\big\(\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\{\\mathbf\{k\}\}\_\{i\}\)\(\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}\{\\mathbf\{k\}\}\_\{i\}\)\\big\)^\{2\}=\(G\_\{r\}H\_\{r\}\)^\{2\},whereGr,Hr∼i\.i\.d\.𝒩​\(0,1\)G\_\{r\},H\_\{r\}\\stackrel\{\{\\scriptstyle\\text\{i\.i\.d\.\}\}\}\{\{\\sim\}\}\\mathcal\{N\}\(0,1\)\. Then𝔼​\[Zr\]=1\\mathbb\{E\}\[Z\_\{r\}\]=1andVar​\(Zr\)=𝔼​\[Zr2\]−\(𝔼​\[Zr\]\)2=9−1=8\\mathrm\{Var\}\(Z\_\{r\}\)=\\mathbb\{E\}\[Z\_\{r\}^\{2\}\]\-\(\\mathbb\{E\}\[Z\_\{r\}\]\)^\{2\}=9\-1=8\. LetYr:=1−ZrY\_\{r\}:=1\-Z\_\{r\}\. Then𝔼​\[Yr\]=0\\mathbb\{E\}\[Y\_\{r\}\]=0,Var​\(Yr\)=8\\mathrm\{Var\}\(Y\_\{r\}\)=8, and sinceZr≥0Z\_\{r\}\\geq 0we haveYr≤1Y\_\{r\}\\leq 1\. Therefore, fort∈\(0,1\)t\\in\(0,1\),

ℙ​\(𝐊^i​i≤1−t\)=ℙ​\(∑r=1mYr≥m​t\)\.\\mathbb\{P\}\(\\hat\{\\mathbf\{K\}\}\_\{ii\}\\leq 1\-t\)=\\mathbb\{P\}\\\!\\left\(\\sum\_\{r=1\}^\{m\}Y\_\{r\}\\geq mt\\right\)\.By the one\-sided Bernstein inequality for independent mean\-zero variables bounded above by11,

ℙ​\(∑r=1mYr≥m​t\)≤exp⁡\(−\(m​t\)22​\(8​m\+m​t3\)\)=exp⁡\(−m​t22​\(8\+t/3\)\)≤e−c​m​t2\\mathbb\{P\}\\\!\\left\(\\sum\_\{r=1\}^\{m\}Y\_\{r\}\\geq mt\\right\)\\leq\\exp\\\!\\left\(\-\\frac\{\(mt\)^\{2\}\}\{2\(8m\+\\frac\{mt\}\{3\}\)\}\\right\)=\\exp\\\!\\left\(\-\\frac\{mt^\{2\}\}\{2\(8\+t/3\)\}\\right\)\\leq e^\{\-cmt^\{2\}\}for an absolutec\>0c\>0, proving equation[62](https://arxiv.org/html/2607.10034#A2.E62)\. The uniform bound \(equation[63](https://arxiv.org/html/2607.10034#A2.E63)\) follows by a union bound overi∈\[F\]i\\in\[F\]\. ∎

#### B\.9\.3Cross\-Talk Bounds: Isotropic Values

###### Lemma B\.40\(Mean\-term concentrationBYB\_\{Y\}\(isotropic values\)\)\.

Assume𝐯1,…,𝐯F\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and letL:=log⁡\(C0​F2δ\)L\\vcentcolon=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)\. Then with probability at least1−δ1\-\\delta\(over the values\),

BY:=maxi∈\[F\]⁡maxj≠i⁡\|⟨𝟏,𝐘\(i​j\)⟩\|≤C1​F​Ld\.B\_\{Y\}\\ \\vcentcolon=\\ \\max\_\{i\\in\[F\]\}\\max\_\{j\\neq i\}\\big\|\\langle\{\{\\mathbf\{1\}\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\\big\|\\ \\leq\\ C\_\{1\}\\sqrt\{\\frac\{FL\}\{d\}\}\.\(64\)

###### Proof\.

Fix a pair\(i,j\)\(i,j\)withj≠ij\\neq iand set𝐰i​j:=𝐯i−𝐯j\{\\mathbf\{w\}\}\_\{ij\}\\vcentcolon=\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\. Let

ℱi​j:=σ​\(𝐯i,𝐯j\)\.\\mathcal\{F\}\_\{ij\}\\ \\vcentcolon=\\ \\sigma\(\{\\mathbf\{v\}\}\_\{i\},\{\\mathbf\{v\}\}\_\{j\}\)\.Conditional onℱi​j\\mathcal\{F\}\_\{ij\}, the random variablesθt:=⟨𝐰i​j,𝐯t⟩\\theta\_\{t\}\\vcentcolon=\\langle\{\{\\mathbf\{w\}\}\_\{ij\}\},\{\{\\mathbf\{v\}\}\_\{t\}\}\\ranglefort∉\{i,j\}t\\notin\\\{i,j\\\}are independent, centered, and sub\-Gaussian with‖θt‖ψ2≲‖𝐰i​j‖2/d−1\\\|\\theta\_\{t\}\\\|\_\{\\psi\_\{2\}\}\\lesssim\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}/\\sqrt\{d\-1\}by Lemma[B\.9\.1](https://arxiv.org/html/2607.10034#A2.SS9.SSS1)\. Therefore, conditional onℱi​j\\mathcal\{F\}\_\{ij\}, the sum

⟨𝟏,𝐘\(i​j\)⟩=∑t∉\{i,j\}θt\\langle\{\{\\mathbf\{1\}\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle=\\sum\_\{t\\notin\\\{i,j\\\}\}\\theta\_\{t\}is centered sub\-Gaussian with‖⟨𝟏,𝐘\(i​j\)⟩‖ψ2≲‖𝐰i​j‖2​F−2d−1≲Fd\.\\big\\\|\\langle\{\{\\mathbf\{1\}\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\\big\\\|\_\{\\psi\_\{2\}\}\\lesssim\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}\\sqrt\{\\tfrac\{F\-2\}\{d\-1\}\}\\lesssim\\sqrt\{\\tfrac\{F\}\{d\}\}\.Hence for alls≥0s\\geq 0,

ℙ​\(\|⟨𝟏,𝐘\(i​j\)⟩\|≥C​F​sd\|ℱi​j\)≤2​e−s\.\\mathbb\{P\}\\Big\(\\big\|\\langle\{\{\\mathbf\{1\}\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\\big\|\\ \\geq\\ C\\sqrt\{\\tfrac\{Fs\}\{d\}\}\\ \\Big\|\\ \\mathcal\{F\}\_\{ij\}\\Big\)\\ \\leq\\ 2e^\{\-s\}\.This conditional tail bound is uniform inℱi​j\\mathcal\{F\}\_\{ij\}, so taking expectations overℱi​j\\mathcal\{F\}\_\{ij\}yields the same inequality unconditionally\.

Now chooses=L=log⁡\(C0​F2δ\)s=L=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)so that2​e−s≤δ/F22e^\{\-s\}\\leq\\delta/F^\{2\}\. Union bound over all pairs\(i,j\)\(i,j\)\(at mostF​\(F−1\)≤F2F\(F\-1\)\\leq F^\{2\}choices\) gives equation[64](https://arxiv.org/html/2607.10034#A2.E64)\. ∎

###### Lemma B\.41\(Value\-difference energy concentrationE~v\\widetilde\{E\}\_\{v\}\(isotropic values\)\)\.

Assume𝐯1,…,𝐯F\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and letL:=log⁡\(C0​F2δ\)L\\vcentcolon=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)\. Then with probability at least1−δ1\-\\delta\(over the values\),

E~v:=maxi∈\[F\]⁡maxj≠i​∑t∉\{i,j\}⟨𝐯i−𝐯j,𝐯t⟩2≤C1​\(F−2\)\+Ld−1\.\\widetilde\{E\}\_\{v\}\\ \\vcentcolon=\\ \\max\_\{i\\in\[F\]\}\\max\_\{j\\neq i\}\\sum\_\{t\\notin\\\{i,j\\\}\}\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{t\}\}\\rangle^\{2\}\\ \\leq\\ C\_\{1\}\\,\\frac\{\(F\-2\)\+L\}\{d\-1\}\.\(65\)

###### Proof\.

Fix a pair\(i,j\)\(i,j\)withj≠ij\\neq iand write𝐰i​j:=𝐯i−𝐯j\{\\mathbf\{w\}\}\_\{ij\}\\vcentcolon=\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\. Letℱi​j:=σ​\(𝐯i,𝐯j\)\\mathcal\{F\}\_\{ij\}\\vcentcolon=\\sigma\(\{\\mathbf\{v\}\}\_\{i\},\{\\mathbf\{v\}\}\_\{j\}\)and, fort∉\{i,j\}t\\notin\\\{i,j\\\}, defineθt:=⟨𝐰i​j,𝐯t⟩\\theta\_\{t\}\\vcentcolon=\\langle\{\{\\mathbf\{w\}\}\_\{ij\}\},\{\{\\mathbf\{v\}\}\_\{t\}\}\\rangle\. Conditional onℱi​j\\mathcal\{F\}\_\{ij\}, theθt\\theta\_\{t\}are independent, centered, sub\-Gaussian, therefore the centered squares

Zt:=θt2−𝔼​\[θt2∣ℱi​j\]Z\_\{t\}\\vcentcolon=\\theta\_\{t\}^\{2\}\-\\mathbb\{E\}\[\\theta\_\{t\}^\{2\}\\mid\\mathcal\{F\}\_\{ij\}\]are independent and sub\-exponential with‖Zt‖ψ1≲‖𝐰i​j‖22/\(d−1\)\\\|Z\_\{t\}\\\|\_\{\\psi\_\{1\}\}\\lesssim\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}^\{2\}/\(d\-1\)by Lemma[B\.9\.1](https://arxiv.org/html/2607.10034#A2.SS9.SSS1)\.

Write

E~v​\(i,j\):=∑t∉\{i,j\}θt2=∑t∉\{i,j\}𝔼​\[θt2∣ℱi​j\]\+∑t∉\{i,j\}Zt\.\\widetilde\{E\}\_\{v\}\(i,j\)\\ \\vcentcolon=\\ \\sum\_\{t\\notin\\\{i,j\\\}\}\\theta\_\{t\}^\{2\}\\ =\\ \\sum\_\{t\\notin\\\{i,j\\\}\}\\mathbb\{E\}\[\\theta\_\{t\}^\{2\}\\mid\\mathcal\{F\}\_\{ij\}\]\\ \+\\ \\sum\_\{t\\notin\\\{i,j\\\}\}Z\_\{t\}\.Since𝐯t\{\\mathbf\{v\}\}\_\{t\}is isotropic,𝔼​\[θt2∣ℱi​j\]=‖𝐰i​j‖22/d\\mathbb\{E\}\[\\theta\_\{t\}^\{2\}\\mid\\mathcal\{F\}\_\{ij\}\]=\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}^\{2\}/dand hence∑t∉\{i,j\}𝔼​\[θt2∣ℱi​j\]=\(F−2\)​‖𝐰i​j‖22/d\\sum\_\{t\\notin\\\{i,j\\\}\}\\mathbb\{E\}\[\\theta\_\{t\}^\{2\}\\mid\\mathcal\{F\}\_\{ij\}\]=\(F\-2\)\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}^\{2\}/d\. Also‖𝐰i​j‖2≤2\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}\\leq 2\. A standard Bernstein inequality for sums of independent sub\-exponential variables implies that for alls≥0s\\geq 0,

ℙ​\(∑t∉\{i,j\}Zt≥C​‖𝐰i​j‖22d−1​\(\(F−2\)​s\+s\)\|ℱi​j\)≤2​e−s\.\\mathbb\{P\}\\Big\(\\sum\_\{t\\notin\\\{i,j\\\}\}Z\_\{t\}\\ \\geq\\ C\\,\\frac\{\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}^\{2\}\}\{d\-1\}\\big\(\\sqrt\{\(F\-2\)s\}\+s\\big\)\\ \\Big\|\\ \\mathcal\{F\}\_\{ij\}\\Big\)\\ \\leq\\ 2e^\{\-s\}\.As in Lemma[B\.9\.3](https://arxiv.org/html/2607.10034#A2.SS9.SSS3), the bound is uniform inℱi​j\\mathcal\{F\}\_\{ij\}, so the same tail holds unconditionally\.

Using\(F−2\)​s≤\(F−2\)\+s2\\sqrt\{\(F\-2\)s\}\\leq\\tfrac\{\(F\-2\)\+s\}\{2\}and absorbing constants yields

ℙ​\(E~v​\(i,j\)≥C​‖𝐰i​j‖22d−1​\(\(F−2\)\+s\)\)≤2​e−c​s\.\\mathbb\{P\}\\Big\(\\widetilde\{E\}\_\{v\}\(i,j\)\\ \\geq\\ C\\,\\frac\{\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}^\{2\}\}\{d\-1\}\\big\(\(F\-2\)\+s\\big\)\\Big\)\\ \\leq\\ 2e^\{\-cs\}\.Since‖𝐰i​j‖22≤4\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}^\{2\}\\leq 4, choosings=L=log⁡\(C0​F2δ\)s=L=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)makes the right\-hand side≤δ/F2\\leq\\delta/F^\{2\}\. Union bounding over all pairs\(i,j\)\(i,j\)yields equation[65](https://arxiv.org/html/2607.10034#A2.E65)\. ∎

###### Lemma B\.42\(Coupling concentrationκ~\\widetilde\{\\kappa\}\(isotropic values\)\)\.

Assume𝐯1,…,𝐯F\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}\. Treat the kernel matrix𝐊^\\hat\{\\mathbf\{K\}\}as arbitrary/deterministic\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and letL:=log⁡\(C0​F2δ\)L\\vcentcolon=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)\. Then with probability at least1−δ1\-\\delta\(over the values\), the coupling statisticκ~\\widetilde\{\\kappa\}from equation[34](https://arxiv.org/html/2607.10034#A2.E34)satisfies

κ~≤C1​LF−2\.\\widetilde\{\\kappa\}\\ \\leq\\ C\_\{1\}\\,\\sqrt\{\\frac\{L\}\{F\-2\}\}\.\(66\)

###### Proof\.

Fix a pair\(i,j\)\(i,j\)withj≠ij\\neq i\. If‖𝐗\(i​j\)‖2=0\\\|\\mathbf\{X\}^\{\(ij\)\}\\\|\_\{2\}=0or‖𝐘\(i​j\)‖2=0\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}=0thenκ~i​j=0\\widetilde\{\\kappa\}^\{ij\}=0, so assume both norms are positive and set

𝐮\(i​j\):=𝐗\(i​j\)/‖𝐗\(i​j\)‖2\.\{\\mathbf\{u\}\}^\{\(ij\)\}\\vcentcolon=\\mathbf\{X\}^\{\(ij\)\}/\\\|\\mathbf\{X\}^\{\(ij\)\}\\\|\_\{2\}\.Write𝐰i​j:=𝐯i−𝐯j\{\\mathbf\{w\}\}\_\{ij\}\\vcentcolon=\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}and letℱi​j:=σ​\(𝐯i,𝐯j\)\\mathcal\{F\}\_\{ij\}\\vcentcolon=\\sigma\(\{\\mathbf\{v\}\}\_\{i\},\{\\mathbf\{v\}\}\_\{j\}\)\. Fort∉\{i,j\}t\\notin\\\{i,j\\\}defineθt:=⟨𝐰i​j,𝐯t⟩\\theta\_\{t\}\\vcentcolon=\\langle\{\{\\mathbf\{w\}\}\_\{ij\}\},\{\{\\mathbf\{v\}\}\_\{t\}\}\\rangle\. Let

Si​j:=⟨𝐮\(i​j\),𝐘\(i​j\)⟩=∑t∉\{i,j\}𝐮t\(i​j\)​θt\.S\_\{ij\}\\ \\vcentcolon=\\ \\langle\{\{\\mathbf\{u\}\}^\{\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\\ =\\ \\sum\_\{t\\notin\\\{i,j\\\}\}\{\\mathbf\{u\}\}^\{\(ij\)\}\_\{t\}\\,\\theta\_\{t\}\.Then

κ~i​j=\|⟨𝐗\(i​j\),𝐘\(i​j\)⟩\|‖𝐗\(i​j\)‖2​‖𝐘\(i​j\)‖2=\|⟨𝐮\(i​j\),𝐘\(i​j\)⟩\|‖𝐘\(i​j\)‖2=\|Si​j\|‖𝐘\(i​j\)‖2\.\\widetilde\{\\kappa\}^\{ij\}=\\frac\{\|\\langle\{\\mathbf\{X\}^\{\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\|\}\{\\\|\\mathbf\{X\}^\{\(ij\)\}\\\|\_\{2\}\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\}=\\frac\{\|\\langle\{\{\\mathbf\{u\}\}^\{\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\|\}\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\}=\\frac\{\|S\_\{ij\}\|\}\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\}\.

##### Numerator tail\.

Conditional onℱi​j\\mathcal\{F\}\_\{ij\}, the vectors\(𝐯t\)t∉\{i,j\}\(\{\\mathbf\{v\}\}\_\{t\}\)\_\{t\\notin\\\{i,j\\\}\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}, so eachθt=⟨𝐰i​j,𝐯t⟩\\theta\_\{t\}=\\langle\{\{\\mathbf\{w\}\}\_\{ij\}\},\{\{\\mathbf\{v\}\}\_\{t\}\}\\rangleis centered sub\-Gaussian with‖θt‖ψ2≲‖𝐰i​j‖2/d−1\\\|\\theta\_\{t\}\\\|\_\{\\psi\_\{2\}\}\\lesssim\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}/\\sqrt\{d\-1\}by Lemma[B\.9\.1](https://arxiv.org/html/2607.10034#A2.SS9.SSS1)\. Since∑t∉\{i,j\}\(𝐮t\(i​j\)\)2≤1\\sum\_\{t\\notin\\\{i,j\\\}\}\(\{\\mathbf\{u\}\}^\{\(ij\)\}\_\{t\}\)^\{2\}\\leq 1,Si​jS\_\{ij\}is centered sub\-Gaussian with‖Si​j‖ψ2≲‖𝐰i​j‖2/d−1\\\|S\_\{ij\}\\\|\_\{\\psi\_\{2\}\}\\lesssim\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}/\\sqrt\{d\-1\}\. Hence for alls≥0s\\geq 0,

ℙ​\(\|Si​j\|≥C​‖𝐰i​j‖2​sd−1\|ℱi​j\)≤2​e−s\.\\mathbb\{P\}\\Big\(\|S\_\{ij\}\|\\ \\geq\\ C\\,\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}\\sqrt\{\\tfrac\{s\}\{d\-1\}\}\\ \\Big\|\\ \\mathcal\{F\}\_\{ij\}\\Big\)\\ \\leq\\ 2e^\{\-s\}\.The right\-hand side does not depend onℱi​j\\mathcal\{F\}\_\{ij\}, so taking expectation overℱi​j\\mathcal\{F\}\_\{ij\}yields the same bound unconditionally\.

##### Denominator lower tail\.

Conditional onℱi​j\\mathcal\{F\}\_\{ij\}, the centered squaresZt:=θt2−𝔼​\[θt2∣ℱi​j\]Z\_\{t\}\\vcentcolon=\\theta\_\{t\}^\{2\}\-\\mathbb\{E\}\[\\theta\_\{t\}^\{2\}\\mid\\mathcal\{F\}\_\{ij\}\]are independent, mean\-zero, and sub\-exponential with‖Zt‖ψ1≲‖𝐰i​j‖22/\(d−1\)\\\|Z\_\{t\}\\\|\_\{\\psi\_\{1\}\}\\lesssim\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}^\{2\}/\(d\-1\)\(Lemma[B\.9\.1](https://arxiv.org/html/2607.10034#A2.SS9.SSS1)\)\. A Bernstein inequality for sums of independent sub\-exponential variables \(applied to−Zt\-Z\_\{t\}\) yields that for alls≥0s\\geq 0,

ℙ​\(‖𝐘\(i​j\)‖22≤∑t∉\{i,j\}𝔼​\[θt2∣ℱi​j\]−C​‖𝐰i​j‖22d−1​\(\(F−2\)​s\+s\)\|ℱi​j\)≤e−s\.\\mathbb\{P\}\\Bigg\(\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}^\{2\}\\leq\\sum\_\{t\\notin\\\{i,j\\\}\}\\mathbb\{E\}\[\\theta\_\{t\}^\{2\}\\mid\\mathcal\{F\}\_\{ij\}\]\-C\\,\\frac\{\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}^\{2\}\}\{d\-1\}\\big\(\\sqrt\{\(F\-2\)s\}\+s\\big\)\\ \\Bigg\|\\ \\mathcal\{F\}\_\{ij\}\\Bigg\)\\ \\leq\\ e^\{\-s\}\.Again the bound is uniform inℱi​j\\mathcal\{F\}\_\{ij\}, so it holds unconditionally after taking the expectation\. Since𝐯t\{\\mathbf\{v\}\}\_\{t\}is isotropic,𝔼​\[θt2∣ℱi​j\]=‖𝐰i​j‖22/d\\mathbb\{E\}\[\\theta\_\{t\}^\{2\}\\mid\\mathcal\{F\}\_\{ij\}\]=\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}^\{2\}/d, and hence∑t∉\{i,j\}𝔼​\[θt2∣ℱi​j\]=\(F−2\)​‖𝐰i​j‖22/d\\sum\_\{t\\notin\\\{i,j\\\}\}\\mathbb\{E\}\[\\theta\_\{t\}^\{2\}\\mid\\mathcal\{F\}\_\{ij\}\]=\(F\-2\)\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}^\{2\}/d\. Using1/d≥1/\(2​\(d−1\)\)1/d\\geq 1/\(2\(d\-1\)\)ford≥2d\\geq 2, the complement event implies

‖𝐘\(i​j\)‖22≥‖𝐰i​j‖22d−1​\(F−22−C​\(\(F−2\)​s\+s\)\)\.\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}^\{2\}\\geq\\frac\{\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}^\{2\}\}\{d\-1\}\\Big\(\\frac\{F\-2\}\{2\}\-C\\big\(\\sqrt\{\(F\-2\)s\}\+s\\big\)\\Big\)\.

##### A high\-probability ratio bound\.

On the intersection of the numerator and denominator events, ifF−2≥C0​sF\-2\\geq C\_\{0\}s, then‖𝐘\(i​j\)‖2≥c​‖𝐰i​j‖2​\(F−2\)/\(d−1\)\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\\geq c\\,\\\|\{\\mathbf\{w\}\}\_\{ij\}\\\|\_\{2\}\\sqrt\{\(F\-2\)/\(d\-1\)\}and hence

κ~i​j=\|Si​j\|‖𝐘\(i​j\)‖2≤C​sF−2\.\\widetilde\{\\kappa\}^\{ij\}=\\frac\{\|S\_\{ij\}\|\}\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\}\\ \\leq\\ C\\,\\sqrt\{\\frac\{s\}\{F\-2\}\}\.If insteadF−2<C0​sF\-2<C\_\{0\}s, thens/\(F−2\)≥c\\sqrt\{s/\(F\-2\)\}\\geq cand the same bound holds trivially sinceκ~i​j≤1\\widetilde\{\\kappa\}^\{ij\}\\leq 1\. Overall, for the fixed\(i,j\)\(i,j\)the bound holds with failure probability at most3​e−s3e^\{\-s\}\.

##### Choosessand union bound\.

Sets=L=log⁡\(C0​F2δ\)s=L=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)\. Then for a fixed\(i,j\)\(i,j\)the failure probability is≤3​e−L≤δ/F2\\leq 3e^\{\-L\}\\leq\\delta/F^\{2\}\(forC0C\_\{0\}large enough\)\. Union bounding over all pairs\(i,j\)\(i,j\)yields equation[66](https://arxiv.org/html/2607.10034#A2.E66)\. ∎

###### Lemma B\.43\(Centered coupling concentrationκ∘\\kappa^\{\\circ\}\(isotropic values\)\)\.

Assume𝐯1,…,𝐯F\{\\mathbf\{v\}\}\_\{1\},\\dots,\{\\mathbf\{v\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}\. Let𝐗∘\(i​j\)∈ℝF−2\\mathbf\{X\}^\{\\circ\(ij\)\}\\in\\mathbb\{R\}^\{F\-2\},i≠ji\\neq j, be deterministic vectors or random vectors independent of the values\. Define

κ∘\(i​j\):=\|⟨𝐗∘\(i​j\),𝐘\(i​j\)⟩\|‖𝐗∘\(i​j\)‖2​‖𝐘\(i​j\)‖2,κ∘:=maxi≠j⁡κ∘\(i​j\),\\kappa^\{\\circ\(ij\)\}\\vcentcolon=\\frac\{\\left\|\\left\\langle\\mathbf\{X\}^\{\\circ\(ij\)\},\\mathbf\{Y\}^\{\(ij\)\}\\right\\rangle\\right\|\}\{\\\|\\mathbf\{X\}^\{\\circ\(ij\)\}\\\|\_\{2\}\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\},\\qquad\\kappa^\{\\circ\}\\vcentcolon=\\max\_\{i\\neq j\}\\kappa^\{\\circ\(ij\)\},with the conventionκ∘\(i​j\)=0\\kappa^\{\\circ\(ij\)\}=0if a denominator vanishes\. Fixδ∈\(0,1\)\\delta\\in\(0,1\), and set

L:=log⁡\(C0​F2δ\)\.L\\vcentcolon=\\log\\left\(\\frac\{C\_\{0\}F^\{2\}\}\{\\delta\}\\right\)\.Then, with probability at least1−δ1\-\\delta,

κ∘≤C​LF−2\.\\kappa^\{\\circ\}\\leq C\\sqrt\{\\frac\{L\}\{F\-2\}\}\.

###### Proof\.

Condition on\{𝐗∘\(i​j\):i≠j\}\\\{\\mathbf\{X\}^\{\\circ\(ij\)\}:i\\neq j\\\}\. After conditioning, these columns are deterministic and independent of the values\. The proof of Lemma[B\.9\.3](https://arxiv.org/html/2607.10034#A2.SS9.SSS3)applies unchanged with𝐗\(i​j\)\\mathbf\{X\}^\{\(ij\)\}replaced by𝐗∘\(i​j\)\\mathbf\{X\}^\{\\circ\(ij\)\}\. Removing the conditioning proves the claim\. ∎

#### B\.9\.4Cross\-Talk Bounds: Isotropic Keys and Bilinear Random Features

###### Lemma B\.44\(Bilinear kernel mean and variance \(isotropic keys\)\)\.

Let𝐊^\\hat\{\\mathbf\{K\}\}be the bilinear random\-feature kernel from[SectionB\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2)withmmfeatures and Gaussian weights\(𝐚r,𝐛r\)r=1m\(\{\\mathbf\{a\}\}\_\{r\},\{\\mathbf\{b\}\}\_\{r\}\)\_\{r=1\}^\{m\}\. Assume keys𝐤1,…,𝐤F\{\\mathbf\{k\}\}\_\{1\},\\dots,\{\\mathbf\{k\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}\. Fixt≠it\\neq iand writeρ=⟨𝐤t,𝐤i⟩\\rho=\\langle\{\{\\mathbf\{k\}\}\_\{t\}\},\{\{\\mathbf\{k\}\}\_\{i\}\}\\rangle\. Then:

1. 1\.\(Conditional mean\)𝔼​\[𝐊^t​i∣ρ\]=ρ2\\mathbb\{E\}\[\\hat\{\\mathbf\{K\}\}\_\{ti\}\\mid\\rho\]=\\rho^\{2\}\.
2. 2\.\(Unconditional mean\)μ:=𝔼​\[𝐊^t​i\]=𝔼​\[ρ2\]=1/d\\mu\\vcentcolon=\\mathbb\{E\}\[\\hat\{\\mathbf\{K\}\}\_\{ti\}\]=\\mathbb\{E\}\[\\rho^\{2\}\]=1/d\.
3. 3\.\(Variance scale\) lettingσ2:=𝔼​\[\(𝐊^t​i−μ\)2\]\\sigma^\{2\}\\vcentcolon=\\mathbb\{E\}\[\(\\hat\{\\mathbf\{K\}\}\_\{ti\}\-\\mu\)^\{2\}\], we haveσ2=Θ​\(1d2\+1m\)\\sigma^\{2\}=\\Theta\(\\tfrac\{1\}\{d^\{2\}\}\+\\tfrac\{1\}\{m\}\)

###### Proof\.

Condition on\(𝐤t,𝐤i\)\(\{\\mathbf\{k\}\}\_\{t\},\{\\mathbf\{k\}\}\_\{i\}\)and hence onρ\\rho\. Write

𝐊^t​i=1m​∑r=1mUr,Ur:=\(𝐚r⊤​𝐤t\)​\(𝐚r⊤​𝐤i\)​\(𝐛r⊤​𝐤t\)​\(𝐛r⊤​𝐤i\),\\hat\{\\mathbf\{K\}\}\_\{ti\}=\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}U\_\{r\},\\qquad U\_\{r\}\\vcentcolon=\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\{\\mathbf\{k\}\}\_\{t\}\)\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\{\\mathbf\{k\}\}\_\{i\}\)\\,\(\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}\{\\mathbf\{k\}\}\_\{t\}\)\(\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}\{\\mathbf\{k\}\}\_\{i\}\),with\(𝐚r,𝐛r\)\(\{\\mathbf\{a\}\}\_\{r\},\{\\mathbf\{b\}\}\_\{r\}\)i\.i\.d\. standard Gaussian\. LetAr:=\(𝐚r⊤​𝐤t\)​\(𝐚r⊤​𝐤i\)A\_\{r\}\\vcentcolon=\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\{\\mathbf\{k\}\}\_\{t\}\)\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\{\\mathbf\{k\}\}\_\{i\}\)andBr:=\(𝐛r⊤​𝐤t\)​\(𝐛r⊤​𝐤i\)B\_\{r\}\\vcentcolon=\(\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}\{\\mathbf\{k\}\}\_\{t\}\)\(\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}\{\\mathbf\{k\}\}\_\{i\}\)so thatUr=Ar​BrU\_\{r\}=A\_\{r\}B\_\{r\}andArA\_\{r\}is independent ofBrB\_\{r\}\.

*Mean\.*Since\(𝐚r⊤​𝐤t,𝐚r⊤​𝐤i\)\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\{\\mathbf\{k\}\}\_\{t\},\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\{\\mathbf\{k\}\}\_\{i\}\)is centered bivariate Gaussian with covarianceρ\\rho,𝔼​\[Ar∣ρ\]=ρ\\mathbb\{E\}\[A\_\{r\}\\mid\\rho\]=\\rhoand likewise𝔼​\[Br∣ρ\]=ρ\\mathbb\{E\}\[B\_\{r\}\\mid\\rho\]=\\rho, hence𝔼​\[Ur∣ρ\]=ρ2\\mathbb\{E\}\[U\_\{r\}\\mid\\rho\]=\\rho^\{2\}and𝔼​\[𝐊^t​i∣ρ\]=ρ2\\mathbb\{E\}\[\\hat\{\\mathbf\{K\}\}\_\{ti\}\\mid\\rho\]=\\rho^\{2\}\. Averaging over isotropic keys yieldsμ=𝔼​\[ρ2\]=1/d\\mu=\\mathbb\{E\}\[\\rho^\{2\}\]=1/d\.

*Variance scale\.*A Wick/Isserlis calculation gives𝔼​\[Ar2∣ρ\]=1\+2​ρ2\\mathbb\{E\}\[A\_\{r\}^\{2\}\\mid\\rho\]=1\+2\\rho^\{2\}and henceVar​\(Ur∣ρ\)=1\+4​ρ2\+3​ρ4\\mathrm\{Var\}\(U\_\{r\}\\mid\\rho\)=1\+4\\rho^\{2\}\+3\\rho^\{4\}\. ThereforeVar​\(𝐊^t​i∣ρ\)=1m​\(1\+4​ρ2\+3​ρ4\)\\mathrm\{Var\}\(\\hat\{\\mathbf\{K\}\}\_\{ti\}\\mid\\rho\)=\\tfrac\{1\}\{m\}\(1\+4\\rho^\{2\}\+3\\rho^\{4\}\)\. Using the law of total variance givesσ2=𝔼​\[Var​\(𝐊^t​i∣ρ\)\]\+Var​\(ρ2\)=Θ​\(1m\)\+Θ​\(1d2\)\\sigma^\{2\}=\\mathbb\{E\}\[\\mathrm\{Var\}\(\\hat\{\\mathbf\{K\}\}\_\{ti\}\\mid\\rho\)\]\+\\mathrm\{Var\}\(\\rho^\{2\}\)=\\Theta\(\\tfrac\{1\}\{m\}\)\+\\Theta\(\\tfrac\{1\}\{d^\{2\}\}\), whereVar​\(ρ2\)=Θ​\(1/d2\)\\mathrm\{Var\}\(\\rho^\{2\}\)=\\Theta\(1/d^\{2\}\)follows from𝔼​\[ρ2\]=1/d\\mathbb\{E\}\[\\rho^\{2\}\]=1/dand𝔼​\[ρ4\]=3/\(d​\(d\+2\)\)\\mathbb\{E\}\[\\rho^\{4\}\]=3/\(d\(d\+2\)\)\. ∎

###### Lemma B\.45\(Gaussian\-chaos entry concentration, centered atρ2\\rho^\{2\}\)\.

Let𝐱,𝐲∈ℝd\\mathbf\{x\},\\mathbf\{y\}\\in\\mathbb\{R\}^\{d\}be deterministic unit vectors and setρ≔⟨𝐱,𝐲⟩\\rho\\coloneqq\\langle\\mathbf\{x\},\\mathbf\{y\}\\rangle\. Let\{𝐚r,𝐛r\}r=1m\\\{\{\\mathbf\{a\}\}\_\{r\},\{\\mathbf\{b\}\}\_\{r\}\\\}\_\{r=1\}^\{m\}be i\.i\.d\. with𝐚r,𝐛r∼𝒩​\(0,Id\)\{\\mathbf\{a\}\}\_\{r\},\{\\mathbf\{b\}\}\_\{r\}\\sim\\mathcal\{N\}\(0,I\_\{d\}\), and define the bilinear random\-feature kernel

K^​\(𝐱,𝐲\)≔1m​∑r=1m\(𝐚r⊤​𝐱\)​\(𝐚r⊤​𝐲\)​\(𝐛r⊤​𝐱\)​\(𝐛r⊤​𝐲\)\.\\hat\{K\}\(\\mathbf\{x\},\\mathbf\{y\}\)\\coloneqq\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\\mathbf\{y\}\)\(\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}\\mathbf\{y\}\)\.Then there exists a universal constantC\>0C\>0such that, for allu≥0u\\geq 0,

ℙ​\(\|K^​\(𝐱,𝐲\)−ρ2\|≥C​\(um\+u2m\)\)≤2​e−u\.\\mathbb\{P\}\\Big\(\\big\|\\hat\{K\}\(\\mathbf\{x\},\\mathbf\{y\}\)\-\\rho^\{2\}\\big\|\\geq C\\Big\(\\sqrt\{\\tfrac\{u\}\{m\}\}\+\\tfrac\{u^\{2\}\}\{m\}\\Big\)\\Big\)\\leq 2e^\{\-u\}\.

###### Proof\.

Fix𝐱,𝐲\\mathbf\{x\},\\mathbf\{y\}and write

Zr≔\(𝐚r⊤​𝐱\)​\(𝐚r⊤​𝐲\)​\(𝐛r⊤​𝐱\)​\(𝐛r⊤​𝐲\)−ρ2,r∈\[m\]\.Z\_\{r\}\\coloneqq\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\\mathbf\{y\}\)\(\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}\\mathbf\{x\}\)\(\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}\\mathbf\{y\}\)\-\\rho^\{2\},\\qquad r\\in\[m\]\.Then\{Zr\}r=1m\\\{Z\_\{r\}\\\}\_\{r=1\}^\{m\}are i\.i\.d\. and mean\-zero\. Moreover,ZrZ\_\{r\}is a centered degree\-44polynomial in jointly Gaussian random variables with bounded covariance, so‖Zr‖L2≤C\\\|Z\_\{r\}\\\|\_\{L\_\{2\}\}\\leq C; by Gaussian hypercontractivity \(Theorem[B\.50](https://arxiv.org/html/2607.10034#A2.Thmtheorem50)withk=4k=4\),‖Zr‖Lp≤\(p−1\)2​‖Zr‖L2≤C′​p2\\\|Z\_\{r\}\\\|\_\{L\_\{p\}\}\\leq\(p\-1\)^\{2\}\\\|Z\_\{r\}\\\|\_\{L\_\{2\}\}\\leq C^\{\\prime\}p^\{2\}for allp≥2p\\geq 2\. A standard moment\-to\-tail conversion then yields a sub\-Weibull\(12\)\(\\tfrac\{1\}\{2\}\)tail: there exists a universal constantC0\>0C\_\{0\}\>0such that, for allu≥0u\\geq 0,

ℙ​\(\|Zr\|≥C0​\(u\+u2\)\)≤2​e−u\.\\mathbb\{P\}\\Big\(\|Z\_\{r\}\|\\geq C\_\{0\}\(\\sqrt\{u\}\+u^\{2\}\)\\Big\)\\leq 2e^\{\-u\}\.Applying Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5)with weightswr≡1/mw\_\{r\}\\equiv 1/m\(so that‖𝐰‖2=m−1/2\\\|\\mathbf\{w\}\\\|\_\{2\}=m^\{\-1/2\}and‖𝐰‖∞=m−1\\\|\\mathbf\{w\}\\\|\_\{\\infty\}=m^\{\-1\}\) yields the claim\. ∎

###### Lemma B\.46\(Gaussian\-chaos entry concentration, centered atμ\\mu\)\.

Let𝐱,𝐲∼i\.i\.d\.Unif​\(𝕊d−1\)\\mathbf\{x\},\\mathbf\{y\}\\stackrel\{\{\\scriptstyle\\mathrm\{i\.i\.d\.\}\}\}\{\{\\sim\}\}\\mathrm\{Unif\}\(\\mathbb\{S\}^\{d\-1\}\)be independent of\{𝐚r,𝐛r\}r=1m\\\{\{\\mathbf\{a\}\}\_\{r\},\{\\mathbf\{b\}\}\_\{r\}\\\}\_\{r=1\}^\{m\}, and setμ≔1/d\\mu\\coloneqq 1/dandρ≔⟨𝐱,𝐲⟩\\rho\\coloneqq\\langle\\mathbf\{x\},\\mathbf\{y\}\\rangle\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and defineL≔log⁡\(6/δ\)L\\coloneqq\\log\(6/\\delta\)\. Assume

m≤d2L\.m\\leq\\frac\{d^\{2\}\}\{L\}\.\(67\)Then there exists a universal constantC\>0C\>0such that

ℙ​\(\|K^​\(𝐱,𝐲\)−μ\|≤C​\(Lm\+L2m\+Ld2\)\)≥1−δ\.\\mathbb\{P\}\\Big\(\\big\|\\hat\{K\}\(\\mathbf\{x\},\\mathbf\{y\}\)\-\\mu\\big\|\\leq C\\Big\(\\sqrt\{\\tfrac\{L\}\{m\}\}\+\\tfrac\{L^\{2\}\}\{m\}\+\\sqrt\{\\tfrac\{L\}\{d^\{2\}\}\}\\Big\)\\Big\)\\geq 1\-\\delta\.

###### Proof\.

By the triangle inequality,

\|K^​\(𝐱,𝐲\)−μ\|≤\|K^​\(𝐱,𝐲\)−ρ2\|\+\|ρ2−μ\|\.\|\\hat\{K\}\(\\mathbf\{x\},\\mathbf\{y\}\)\-\\mu\|\\leq\|\\hat\{K\}\(\\mathbf\{x\},\\mathbf\{y\}\)\-\\rho^\{2\}\|\+\|\\rho^\{2\}\-\\mu\|\.

##### Control random\-feature fluctuation\.

Condition on𝐱,𝐲\\mathbf\{x\},\\mathbf\{y\}and apply Lemma[B\.9\.4](https://arxiv.org/html/2607.10034#A2.SS9.SSS4)withu=Lu=Lto obtain

ℙ​\(\|K^​\(𝐱,𝐲\)−ρ2\|≤C​\(Lm\+L2m\)\|𝐱,𝐲\)≥1−δ/3\\mathbb\{P\}\\Big\(\|\\hat\{K\}\(\\mathbf\{x\},\\mathbf\{y\}\)\-\\rho^\{2\}\|\\leq C\\Big\(\\sqrt\{\\tfrac\{L\}\{m\}\}\+\\tfrac\{L^\{2\}\}\{m\}\\Big\)\\,\\Big\|\\,\\mathbf\{x\},\\mathbf\{y\}\\Big\)\\geq 1\-\\delta/3for a suitable universal constantC\>0C\>0\.

##### Control geometric fluctuation\.

Recall that if𝐱,𝐲∼i\.i\.d\.Unif​\(𝕊d−1\)\\mathbf\{x\},\\mathbf\{y\}\\stackrel\{\{\\scriptstyle\\text\{i\.i\.d\.\}\}\}\{\{\\sim\}\}\\mathrm\{Unif\}\(\\mathbb\{S\}^\{d\-1\}\), thenρ=dg1/‖g‖2\\rho\\stackrel\{\{\\scriptstyle d\}\}\{\{=\}\}g\_\{1\}/\\\|g\\\|\_\{2\}forg∼𝒩​\(0,Id\)g\\sim\\mathcal\{N\}\(0,I\_\{d\}\)\. LetX≔‖g‖22∼χd2X\\coloneqq\\\|g\\\|\_\{2\}^\{2\}\\sim\\chi^\{2\}\_\{d\}, so thatρ2=g12/X\\rho^\{2\}=g\_\{1\}^\{2\}/X\. Sinceg12∼χ12g\_\{1\}^\{2\}\\sim\\chi^\{2\}\_\{1\}andX∼χd2X\\sim\\chi^\{2\}\_\{d\}, Theorem[B\.53](https://arxiv.org/html/2607.10034#A2.Thmtheorem53)implies that, each with probability at least1−δ/31\-\\delta/3,

\|g12−1\|≤C​\(L\+L\),\|X−d\|≤C​\(d​L\+L\)\|g\_\{1\}^\{2\}\-1\|\\leq C\(\\sqrt\{L\}\+L\),\\qquad\|X\-d\|\\leq C\(\\sqrt\{dL\}\+L\)for a universal constantC\>0C\>0\. In particular, wheneverd≥C′​Ld\\geq C^\{\\prime\}Lfor a suitable absolute constantC′C^\{\\prime\}, the second estimate forcesX≥d/2X\\geq d/2; whend≲Ld\\lesssim Lthe final bound of this step is vacuous up to constants and can be absorbed by enlargingCC\. So assumeX≥d/2X\\geq d/2\. Writingρ2−μ=g12​d−XX​d\\rho^\{2\}\-\\mu=\\dfrac\{g\_\{1\}^\{2\}d\-X\}\{Xd\}and usingX≥d/2X\\geq d/2together with the triangle inequality\|g12​d−X\|≤d​\|g12−1\|\+\|X−d\|\|g\_\{1\}^\{2\}d\-X\|\\leq d\\,\|g\_\{1\}^\{2\}\-1\|\+\|X\-d\|,

\|ρ2−μ\|=\|g12​d−X\|X​d≤2d​\|g12−1\|\+2d2​\|X−d\|\.\|\\rho^\{2\}\-\\mu\|=\\frac\{\|g\_\{1\}^\{2\}d\-X\|\}\{Xd\}\\leq\\frac\{2\}\{d\}\\,\|g\_\{1\}^\{2\}\-1\|\+\\frac\{2\}\{d^\{2\}\}\\,\|X\-d\|\.Substituting the two chi\-square estimates and usingd≥1d\\geq 1,

\|ρ2−μ\|≤Cd​\(L\+L\)\+Cd2​\(d​L\+L\)≤Cd​\(L\+L\)\.\|\\rho^\{2\}\-\\mu\|\\leq\\frac\{C\}\{d\}\(\\sqrt\{L\}\+L\)\+\\frac\{C\}\{d^\{2\}\}\(\\sqrt\{dL\}\+L\)\\leq\\frac\{C\}\{d\}\\big\(\\sqrt\{L\}\+L\\big\)\.Thus, on the intersection of the two chi\-square events \(of probability at least1−2​δ/31\-2\\delta/3\),

\|ρ2−μ\|≤Cd​\(L\+L\)\.\|\\rho^\{2\}\-\\mu\|\\leq\\frac\{C\}\{d\}\\big\(\\sqrt\{L\}\+L\\big\)\.

##### Combine and simplify\.

Taking a union bound over both gives, with probability at least1−δ1\-\\delta,

\|K^​\(𝐱,𝐲\)−μ\|≤C​\(Lm\+L2m\+Ld\+Ld\)\.\|\\hat\{K\}\(\\mathbf\{x\},\\mathbf\{y\}\)\-\\mu\|\\leq C\\Big\(\\sqrt\{\\tfrac\{L\}\{m\}\}\+\\tfrac\{L^\{2\}\}\{m\}\+\\frac\{\\sqrt\{L\}\}\{d\}\+\\frac\{L\}\{d\}\\Big\)\.Finally, under equation[67](https://arxiv.org/html/2607.10034#A2.E67), we haveL/d≤L/mL/d\\leq\\sqrt\{L/m\}, so theL/dL/dterm is absorbed, yielding the stated bound\. ∎

###### Lemma B\.47\(Centered column\-energy concentrationEK∘E\_\{K\}^\{\\circ\}\(isotropic keys, bilinear kernel; sharpened\)\)\.

Assume keys𝐤1,…,𝐤F\{\\mathbf\{k\}\}\_\{1\},\\dots,\{\\mathbf\{k\}\}\_\{F\}are i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}, independent of the bilinear random features from[SectionB\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2)\. Let

μ:=1d,σ2:=𝔼​\[\(𝐊^t​i−μ\)2\]=Θ​\(1d2\+1m\),t≠i\.\\mu\\vcentcolon=\\frac\{1\}\{d\},\\qquad\\sigma^\{2\}\\vcentcolon=\\mathbb\{E\}\[\(\\hat\{\\mathbf\{K\}\}\_\{ti\}\-\\mu\)^\{2\}\]=\\Theta\\left\(\\frac\{1\}\{d^\{2\}\}\+\\frac\{1\}\{m\}\\right\),\\qquad t\\neq i\.Fixδ∈\(0,1\)\\delta\\in\(0,1\), and define

L:=log⁡\(C0​F2δ\)\.L\\vcentcolon=\\log\\left\(\\frac\{C\_\{0\}F^\{2\}\}\{\\delta\}\\right\)\.Assume

m≤d2L,L3≤c0​σ2​m2\.m\\leq\\frac\{d^\{2\}\}\{L\},\\qquad L^\{3\}\\leq c\_\{0\}\\sigma^\{2\}m^\{2\}\.Then, forc0\>0c\_\{0\}\>0sufficiently small andC0C\_\{0\}sufficiently large, with probability at least1−δ1\-\\delta,

EK∘≤C​σ2​\(\(F−2\)\+L\)\.E\_\{K\}^\{\\circ\}\\leq C\\sigma^\{2\}\\bigl\(\(F\-2\)\+L\\bigr\)\.Equivalently,

EK∘≤C​σ​\(F−2\)\+L\.\\sqrt\{E\_\{K\}^\{\\circ\}\}\\leq C\\sigma\\sqrt\{\(F\-2\)\+L\}\.In particular, ifF−2≥LF\-2\\geq L, then

EK∘≤C​σ​F−2\.\\sqrt\{E\_\{K\}^\{\\circ\}\}\\leq C\\sigma\\sqrt\{F\-2\}\.

###### Proof\.

AssumeF≥3F\\geq 3\. For eachi∈\[F\]i\\in\[F\], define

𝒞i:=∑t≠i\(𝐊^t​i−μ\)2\.\\mathcal\{C\}\_\{i\}\\vcentcolon=\\sum\_\{t\\neq i\}\(\\hat\{\\mathbf\{K\}\}\_\{ti\}\-\\mu\)^\{2\}\.For eachi≠ji\\neq j,

EK∘​\(i,j\)=∑t∉\{i,j\}\(𝐊^t​i−μ\)2≤𝒞i\.E\_\{K\}^\{\\circ\}\(i,j\)=\\sum\_\{t\\notin\\\{i,j\\\}\}\(\\hat\{\\mathbf\{K\}\}\_\{ti\}\-\\mu\)^\{2\}\\leq\\mathcal\{C\}\_\{i\}\.Thus

EK∘≤maxi⁡𝒞i\.E\_\{K\}^\{\\circ\}\\leq\\max\_\{i\}\\mathcal\{C\}\_\{i\}\.
Fixii, and setqi=𝐤iq\_\{i\}=\{\\mathbf\{k\}\}\_\{i\}\. Forq∈𝕊d−1q\\in\\mathbb\{S\}^\{d\-1\}, defineA​\(q\)A\(q\)as in Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5)\. Then, for everyx∈𝕊d−1x\\in\\mathbb\{S\}^\{d\-1\},

x⊤​A​\(q\)​x=1m​∑r=1m\(𝐚r⊤​x\)​\(𝐚r⊤​q\)​\(𝐛r⊤​x\)​\(𝐛r⊤​q\)=K^​\(x,q\)\.x^\{\\top\}A\(q\)x=\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}x\)\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}q\)\(\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}x\)\(\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}q\)=\\hat\{K\}\(x,q\)\.Therefore

𝐊^t​i=𝐤t⊤​A​\(qi\)​𝐤t,t≠i\.\\hat\{\\mathbf\{K\}\}\_\{ti\}=\{\\mathbf\{k\}\}\_\{t\}^\{\\top\}A\(q\_\{i\}\)\{\\mathbf\{k\}\}\_\{t\},\\qquad t\\neq i\.Define

τi:=tr⁡A​\(qi\)d,Bi:=A​\(qi\)−τi​Id,βi:=τi−μ\.\\tau\_\{i\}\\vcentcolon=\\frac\{\\operatorname\{tr\}A\(q\_\{i\}\)\}\{d\},\\qquad B\_\{i\}\\vcentcolon=A\(q\_\{i\}\)\-\\tau\_\{i\}I\_\{d\},\\qquad\\beta\_\{i\}\\vcentcolon=\\tau\_\{i\}\-\\mu\.Thentr⁡Bi=0\\operatorname\{tr\}B\_\{i\}=0, and

𝐊^t​i−μ=𝐤t⊤​Bi​𝐤t\+βi\.\\hat\{\\mathbf\{K\}\}\_\{ti\}\-\\mu=\{\\mathbf\{k\}\}\_\{t\}^\{\\top\}B\_\{i\}\{\\mathbf\{k\}\}\_\{t\}\+\\beta\_\{i\}\.Let

𝒢i:=σ​\(𝐤i,\{\(𝐚r,𝐛r\)\}r=1m\)\.\\mathcal\{G\}\_\{i\}\\vcentcolon=\\sigma\\left\(\{\\mathbf\{k\}\}\_\{i\},\\\{\(\{\\mathbf\{a\}\}\_\{r\},\{\\mathbf\{b\}\}\_\{r\}\)\\\}\_\{r=1\}^\{m\}\\right\)\.Conditional on𝒢i\\mathcal\{G\}\_\{i\}, the vectors\{𝐤t:t≠i\}\\\{\{\\mathbf\{k\}\}\_\{t\}:t\\neq i\\\}are independent uniform spherical vectors\. Define

vi:=2​‖Bi‖F2d​\(d\+2\)\+βi2,ri:=‖Bi‖opd\+\|βi\|\.v\_\{i\}\\vcentcolon=\\frac\{2\\\|B\_\{i\}\\\|\_\{F\}^\{2\}\}\{d\(d\+2\)\}\+\\beta\_\{i\}^\{2\},\\qquad r\_\{i\}\\vcentcolon=\\frac\{\\\|B\_\{i\}\\\|\_\{\\mathrm\{op\}\}\}\{d\}\+\|\\beta\_\{i\}\|\.By Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5), with probability at least1−δ/41\-\\delta/4, simultaneously for everyi∈\[F\]i\\in\[F\],

vi≤C​σ2,ri2​L2≤C​σ2​L\.v\_\{i\}\\leq C\\sigma^\{2\},\\qquad r\_\{i\}^\{2\}L^\{2\}\\leq C\\sigma^\{2\}L\.On this event, Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5), applied conditionally withn=F−1n=F\-1ands=Ls=L, gives

ℙ\(𝒞i\>C\(\(F−1\)σ2\+σ2L\)\|𝒢i\)≤e−L,\\mathbb\{P\}\\left\(\\mathcal\{C\}\_\{i\}\>C\\left\(\(F\-1\)\\sigma^\{2\}\+\\sigma^\{2\}L\\right\)\\,\\middle\|\\,\\mathcal\{G\}\_\{i\}\\right\)\\leq e^\{\-L\},becauseL≥log⁡\(e​\(F−1\)\)L\\geq\\log\(e\(F\-1\)\)after increasingC0C\_\{0\}\. Union bounding overi∈\[F\]i\\in\[F\],

F​e−L=F​δC0​F2≤δ/4Fe^\{\-L\}=F\\frac\{\\delta\}\{C\_\{0\}F^\{2\}\}\\leq\\delta/4forC0C\_\{0\}sufficiently large\. Combining the conditional square\-sum event with the parameter event yields, with probability at least1−δ1\-\\delta,

maxi⁡𝒞i≤C​σ2​\(\(F−1\)\+L\)\.\\max\_\{i\}\\mathcal\{C\}\_\{i\}\\leq C\\sigma^\{2\}\\bigl\(\(F\-1\)\+L\\bigr\)\.SinceF≥3F\\geq 3,F−1≤2​\(F−2\)F\-1\\leq 2\(F\-2\), so

EK∘≤C​σ2​\(\(F−2\)\+L\)\.E\_\{K\}^\{\\circ\}\\leq C\\sigma^\{2\}\\bigl\(\(F\-2\)\+L\\bigr\)\.Taking square roots proves

EK∘≤C​σ​\(F−2\)\+L\.\\sqrt\{E\_\{K\}^\{\\circ\}\}\\leq C\\sigma\\sqrt\{\(F\-2\)\+L\}\.IfF−2≥LF\-2\\geq L, then\(F−2\)\+L≤2​\(F−2\)\(F\-2\)\+L\\leq 2\(F\-2\), giving

EK∘≤C​σ​F−2\.\\sqrt\{E\_\{K\}^\{\\circ\}\}\\leq C\\sigma\\sqrt\{F\-2\}\.∎

###### Lemma B\.48\(Concentration of the effective couplingκeff∘\\kappa\_\{\\mathrm\{eff\}\}^\{\\circ\}\(isotropic keys, bilinear kernel\)\)\.

Assume the setting of Theorem[B\.19](https://arxiv.org/html/2607.10034#A2.Thmtheorem19): isotropic keys, bilinear kernel, and deterministic values/codes\. Letμ=1/d\\mu=1/dandσ2=Θ​\(1d2\+1m\)\\sigma^\{2\}=\\Theta\(\\tfrac\{1\}\{d^\{2\}\}\+\\tfrac\{1\}\{m\}\)\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and letL:=log⁡\(C0​F2δ\)L\\vcentcolon=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)\. Assume additionally thatm≤d2/Lm\\leq d^\{2\}/L\. Define

κeff∘:=maxi≠j⁡\|⟨𝐗∘\(i​j\),𝐘\(i​j\)⟩\|‖𝐘\(i​j\)‖2\\kappa\_\{\\mathrm\{eff\}\}^\{\\circ\}\\ \\vcentcolon=\\ \\max\_\{i\\neq j\}\\frac\{\|\\langle\{\\mathbf\{X\}^\{\\circ\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\|\}\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\}and

Lv:=maxi≠j⁡‖𝐘\(i​j\)‖12‖𝐘\(i​j\)‖22L\_\{v\}\\ \\vcentcolon=\\ \\max\_\{i\\neq j\}\\frac\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{1\}^\{2\}\}\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}^\{2\}\}Then with probability at least1−δ1\-\\delta\(over keys and features\),

κeff∘≤C​\(σ​L\+L2m\)​Lv\.\\kappa\_\{\\mathrm\{eff\}\}^\{\\circ\}\\ \\leq\\ C\\left\(\\sigma\\sqrt\{L\}\+\\frac\{L^\{2\}\}\{m\}\\right\)\\sqrt\{L\_\{v\}\}\.

###### Proof\.

Fix a pair\(i,j\)\(i,j\)withj≠ij\\neq i\. Writeθt:=𝐊^t​i−μ\\theta\_\{t\}\\vcentcolon=\\hat\{\\mathbf\{K\}\}\_\{ti\}\-\\mufort∉\{i,j\}t\\notin\\\{i,j\\\}, so that𝐗∘\(i​j\)=\(θt\)t∉\{i,j\}\\mathbf\{X\}^\{\\circ\(ij\)\}=\(\\theta\_\{t\}\)\_\{t\\notin\\\{i,j\\\}\}\. Then

\|⟨𝐗∘\(i​j\),𝐘\(i​j\)⟩\|‖𝐘\(i​j\)‖2=\|∑t∉\{i,j\}θt​𝐘t\(i​j\)\|‖𝐘\(i​j\)‖2≤\(maxt∉\{i,j\}⁡\|θt\|\)​‖𝐘\(i​j\)‖1‖𝐘\(i​j\)‖2\.\\frac\{\|\\langle\{\\mathbf\{X\}^\{\\circ\(ij\)\}\},\{\\mathbf\{Y\}^\{\(ij\)\}\}\\rangle\|\}\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\}\\ =\\ \\frac\{\\big\|\\sum\_\{t\\notin\\\{i,j\\\}\}\\theta\_\{t\}\\,\\mathbf\{Y\}^\{\(ij\)\}\_\{t\}\\big\|\}\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\}\\ \\leq\\ \\Big\(\\max\_\{t\\notin\\\{i,j\\\}\}\|\\theta\_\{t\}\|\\Big\)\\,\\frac\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{1\}\}\{\\\|\\mathbf\{Y\}^\{\(ij\)\}\\\|\_\{2\}\}\.Taking a maximum over\(i,j\)\(i,j\)and using the definition ofLvL\_\{v\}gives

κeff∘≤\(maxp≠q⁡\|𝐊^p​q−μ\|\)​Lv\.\\kappa\_\{\\mathrm\{eff\}\}^\{\\circ\}\\ \\leq\\ \\Big\(\\max\_\{p\\neq q\}\|\\hat\{\\mathbf\{K\}\}\_\{pq\}\-\\mu\|\\Big\)\\,\\sqrt\{L\_\{v\}\}\.Now apply Lemma[B\.9\.2](https://arxiv.org/html/2607.10034#A2.SS9.SSS2)with failure probabilityδ\\deltato boundmaxp≠q⁡\|𝐊^p​q−μ\|≤C​\(σ​L\+L2m\)\.\\max\_\{p\\neq q\}\|\\hat\{\\mathbf\{K\}\}\_\{pq\}\-\\mu\|\\leq C\(\\sigma\\sqrt\{L\}\+\\tfrac\{L^\{2\}\}\{m\}\)\.Multiplying byLv\\sqrt\{L\_\{v\}\}completes the proof\. ∎

#### B\.9\.5Additional Concentration Tools

###### Lemma B\.49\(Weighted Bernstein for sub\-Weibull\(1/2\)\(1/2\)\-type tails\)\.

LetZ1,…,ZNZ\_\{1\},\\dots,Z\_\{N\}be independent, mean\-zero random variables such that

ℙ​\(\|Zr\|≥C0​\(σ​u\+u2/m\)\)≤2​e−ufor all​u≥0,r∈\[N\]\.\\mathbb\{P\}\\\!\\left\(\|Z\_\{r\}\|\\geq C\_\{0\}\(\\sigma\\sqrt\{u\}\+u^\{2\}/m\)\\right\)\\leq 2e^\{\-u\}\\qquad\\text\{for all \}u\\geq 0,\\ r\\in\[N\]\.Then there exist absolute constantsC,c\>0C,c\>0such that for every deterministicw∈ℝNw\\in\\mathbb\{R\}^\{N\}and everyt≥0t\\geq 0,

ℙ​\(\|∑r=1Nwr​Zr\|≥C​\(σ​‖w‖2​t\+t2m​‖w‖∞\)\)≤2​e−c​t\.\\mathbb\{P\}\\\!\\left\(\\Big\|\\sum\_\{r=1\}^\{N\}w\_\{r\}Z\_\{r\}\\Big\|\\geq C\\Big\(\\sigma\\\|w\\\|\_\{2\}\\sqrt\{t\}\+\\frac\{t^\{2\}\}\{m\}\\\|w\\\|\_\{\\infty\}\\Big\)\\right\)\\leq 2e^\{\-ct\}\.Consequently, foru=log⁡\(C1/δ\)u=\\log\(C\_\{1\}/\\delta\), with probability at least1−δ1\-\\delta,

\|∑r=1Nwr​Zr\|≤C2​\(σ​‖w‖2​u\+u2m​‖w‖∞\)\.\\Big\|\\sum\_\{r=1\}^\{N\}w\_\{r\}Z\_\{r\}\\Big\|\\leq C\_\{2\}\\Big\(\\sigma\\\|w\\\|\_\{2\}\\sqrt\{u\}\+\\frac\{u^\{2\}\}\{m\}\\\|w\\\|\_\{\\infty\}\\Big\)\.

###### Proof\.

The assumed tail bound is equivalent up to absolute constants to the statement that eachZrZ\_\{r\}is sub\-Weibull of orderα=12\\alpha=\\tfrac\{1\}\{2\}with generalized Bernstein–Orlicz parametersν≍σ\\nu\\asymp\\sigmaandL≍\(σ​m\)−1L\\asymp\(\\sigma m\)^\{\-1\}\. Applying Theorem 2\.4 of\(Bong and Kuchibhotla,[2023](https://arxiv.org/html/2607.10034#bib.bib84)\)to the weighted sum∑r=1Nwr​Zr\\sum\_\{r=1\}^\{N\}w\_\{r\}Z\_\{r\}gives

ℙ​\(\|∑r=1Nwr​Zr\|≥C​\(σ​‖w‖2​t\+σ​L​t2​‖w‖∞\)\)≤2​e−c​t\.\\mathbb\{P\}\\\!\\left\(\\Big\|\\sum\_\{r=1\}^\{N\}w\_\{r\}Z\_\{r\}\\Big\|\\geq C\\Big\(\\sigma\\\|w\\\|\_\{2\}\\sqrt\{t\}\+\\sigma L\\,t^\{2\}\\\|w\\\|\_\{\\infty\}\\Big\)\\right\)\\leq 2e^\{\-ct\}\.Sinceσ​L≍1/m\\sigma L\\asymp 1/m, this becomes

ℙ​\(\|∑r=1Nwr​Zr\|≥C​\(σ​‖w‖2​t\+t2m​‖w‖∞\)\)≤2​e−c​t\.\\mathbb\{P\}\\\!\\left\(\\Big\|\\sum\_\{r=1\}^\{N\}w\_\{r\}Z\_\{r\}\\Big\|\\geq C\\Big\(\\sigma\\\|w\\\|\_\{2\}\\sqrt\{t\}\+\\frac\{t^\{2\}\}\{m\}\\\|w\\\|\_\{\\infty\}\\Big\)\\right\)\\leq 2e^\{\-ct\}\.Settingt=log⁡\(C1/δ\)t=\\log\(C\_\{1\}/\\delta\)yields the stated high\-probability bound\. ∎

###### Theorem B\.50\(Gaussian hypercontractivity; Nelson–Gross\)\.

LetG∼𝒩​\(0,In\)G\\sim\\mathcal\{N\}\(0,I\_\{n\}\), and letP​\(G\)P\(G\)be a polynomial of degree at mostkkin the standard Gaussian variables\. Then, for everyq≥2q\\geq 2,

‖P​\(G\)‖Lq≤\(q−1\)k/2​‖P​\(G\)‖L2\.\\\|P\(G\)\\\|\_\{L\_\{q\}\}\\leq\(q\-1\)^\{k/2\}\\\|P\(G\)\\\|\_\{L\_\{2\}\}\.

This is the standard degree\-kkpolynomial\-chaos corollary of the Gaussian hypercontractivity theorem; see O’Donnell\(O’Donnell,[2014](https://arxiv.org/html/2607.10034#bib.bib85), Theorem 11\.23\)\.

###### Theorem B\.51\(Sub\-Weibull average bound from moment growth\)\.

LetW1,…,WmW\_\{1\},\\dots,W\_\{m\}be independent mean\-zero random variables\. Suppose that, for someρ≥1\\rho\\geq 1andK\>0K\>0,

‖Wr‖Lp≤K​pρfor every​p≥2​and every​r∈\[m\]\.\\\|W\_\{r\}\\\|\_\{L\_\{p\}\}\\leq Kp^\{\\rho\}\\qquad\\text\{for every \}p\\geq 2\\text\{ and every \}r\\in\[m\]\.Then there is a constantCρ\>0C\_\{\\rho\}\>0, depending only onρ\\rho, such that for everyu≥1u\\geq 1,

ℙ​\(\|1m​∑r=1mWr\|\>Cρ​K​\(um\+uρm\)\)≤2​e−u\.\\mathbb\{P\}\\left\(\\left\|\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}W\_\{r\}\\right\|\>C\_\{\\rho\}K\\left\(\\sqrt\{\\frac\{u\}\{m\}\}\+\\frac\{u^\{\\rho\}\}\{m\}\\right\)\\right\)\\leq 2e^\{\-u\}\.In particular, ifW1,…,WmW\_\{1\},\\dots,W\_\{m\}are independent centered degree\-kkGaussian chaoses withk≥2k\\geq 2and

‖Wr‖L2≤K0,\\\|W\_\{r\}\\\|\_\{L\_\{2\}\}\\leq K\_\{0\},then

ℙ​\(\|1m​∑r=1mWr\|\>Ck​K0​\(um\+uk/2m\)\)≤2​e−u\.\\mathbb\{P\}\\left\(\\left\|\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}W\_\{r\}\\right\|\>C\_\{k\}K\_\{0\}\\left\(\\sqrt\{\\frac\{u\}\{m\}\}\+\\frac\{u^\{k/2\}\}\{m\}\\right\)\\right\)\\leq 2e^\{\-u\}\.

###### Proof\.

Setα:=1/ρ∈\(0,1\]\\alpha\\vcentcolon=1/\\rho\\in\(0,1\]\. By a standard argument, the moment growth condition implies the Orlicz\-norm bound

∥Wr∥ψα:=inf\{c\>0:𝔼exp\(\(\|Wr\|/c\)α\)≤2\}≤CρKfor everyr∈\[m\]:\\\|W\_\{r\}\\\|\_\{\\psi\_\{\\alpha\}\}\\vcentcolon=\\inf\\left\\\{c\>0:\\mathbb\{E\}\\exp\\left\(\(\|W\_\{r\}\|/c\)^\{\\alpha\}\\right\)\\leq 2\\right\\\}\\leq C\_\{\\rho\}K\\qquad\\text\{for every \}r\\in\[m\]:indeed, monotonicity ofp↦‖Wr‖Lpp\\mapsto\\\|W\_\{r\}\\\|\_\{L\_\{p\}\}and the assumption give‖Wr‖Lα​j≤Cρ​K​j1/α\\\|W\_\{r\}\\\|\_\{L\_\{\\alpha j\}\}\\leq C\_\{\\rho\}Kj^\{1/\\alpha\}for every integerj≥1j\\geq 1, and expanding the exponential and usingj\!≥\(j/e\)jj\!\\geq\(j/e\)^\{j\}shows𝔼​exp⁡\(\(\|Wr\|/\(Aρ​K\)\)α\)≤2\\mathbb\{E\}\\exp\\left\(\(\|W\_\{r\}\|/\(A\_\{\\rho\}K\)\)^\{\\alpha\}\\right\)\\leq 2forAρA\_\{\\rho\}sufficiently large\.

Now apply the generalized Bernstein–Orlicz inequality\(Kuchibhotla and Chakrabortty,[2022](https://arxiv.org/html/2607.10034#bib.bib86), Theorem 3\.1\)toZ:=1m​∑r=1mWrZ\\vcentcolon=\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}W\_\{r\}, with weightsar=1/ma\_\{r\}=1/mandbr:=ar​‖Wr‖ψαb\_\{r\}\\vcentcolon=a\_\{r\}\\\|W\_\{r\}\\\|\_\{\\psi\_\{\\alpha\}\}, so that‖b‖2≤Cρ​K/m\\\|b\\\|\_\{2\}\\leq C\_\{\\rho\}K/\\sqrt\{m\}and‖b‖∞≤Cρ​K/m\\\|b\\\|\_\{\\infty\}\\leq C\_\{\\rho\}K/m\. The defining tail property of their generalized Bernstein–Orlicz norm yields, for everyu≥1u\\geq 1,

ℙ​\(\|Z\|\>Cα​\(‖b‖2​u\+‖b‖∞​u1/α\)\)≤2​e−u\.\\mathbb\{P\}\\left\(\|Z\|\>C\_\{\\alpha\}\\left\(\\\|b\\\|\_\{2\}\\sqrt\{u\}\+\\\|b\\\|\_\{\\infty\}u^\{1/\\alpha\}\\right\)\\right\)\\leq 2e^\{\-u\}\.Substituting the bounds on‖b‖2\\\|b\\\|\_\{2\}and‖b‖∞\\\|b\\\|\_\{\\infty\}and using1/α=ρ1/\\alpha=\\rhogives the stated tail bound\.

Finally, ifWrW\_\{r\}is a centered degree\-kkGaussian chaos with‖Wr‖L2≤K0\\\|W\_\{r\}\\\|\_\{L\_\{2\}\}\\leq K\_\{0\}, Theorem[B\.50](https://arxiv.org/html/2607.10034#A2.Thmtheorem50)gives

‖Wr‖Lp≤\(p−1\)k/2​‖Wr‖L2≤Ck​K0​pk/2,p≥2\.\\\|W\_\{r\}\\\|\_\{L\_\{p\}\}\\leq\(p\-1\)^\{k/2\}\\\|W\_\{r\}\\\|\_\{L\_\{2\}\}\\leq C\_\{k\}K\_\{0\}p^\{k/2\},\\qquad p\\geq 2\.Apply the first part withρ=k/2\\rho=k/2andK=Ck​K0K=C\_\{k\}K\_\{0\}\. ∎

###### Theorem B\.52\(Operator norm from bilinear forms on nets\)\.

LetA∈ℝn×mA\\in\\mathbb\{R\}^\{n\\times m\}, let0<ε<1/20<\\varepsilon<1/2, and let𝒩⊂𝕊n−1\\mathcal\{N\}\\subset\\mathbb\{S\}^\{n\-1\},ℳ⊂𝕊m−1\\mathcal\{M\}\\subset\\mathbb\{S\}^\{m\-1\}be finiteε\\varepsilon\-nets\. Then

maxx∈𝒩,y∈ℳ⁡\|x⊤​A​y\|≤‖A‖op≤11−2​ε​maxx∈𝒩,y∈ℳ⁡\|x⊤​A​y\|\.\\max\_\{x\\in\\mathcal\{N\},\\;y\\in\\mathcal\{M\}\}\|x^\{\\top\}Ay\|\\leq\\\|A\\\|\_\{\\mathrm\{op\}\}\\leq\\frac\{1\}\{1\-2\\varepsilon\}\\max\_\{x\\in\\mathcal\{N\},\\;y\\in\\mathcal\{M\}\}\|x^\{\\top\}Ay\|\.In particular, forε=1/4\\varepsilon=1/4,

‖A‖op≤2​maxx∈𝒩,y∈ℳ⁡\|x⊤​A​y\|\.\\\|A\\\|\_\{\\mathrm\{op\}\}\\leq 2\\max\_\{x\\in\\mathcal\{N\},\\;y\\in\\mathcal\{M\}\}\|x^\{\\top\}Ay\|\.Moreover, for everyd≥1d\\geq 1, the sphere𝕊d−1\\mathbb\{S\}^\{d\-1\}admits a1/41/4\-net of cardinality at most9d9^\{d\}\. Hence one may choose such nets with

\|𝒩\|≤9nand\|ℳ\|≤9m\.\|\\mathcal\{N\}\|\\leq 9^\{n\}\\qquad\\text\{and\}\\qquad\|\\mathcal\{M\}\|\\leq 9^\{m\}\.

This is Vershynin\(Vershynin,[2026](https://arxiv.org/html/2607.10034#bib.bib89), Lemma 4\.4\.2 and Corollary 4\.2\.11\)\.

###### Theorem B\.53\(Laurent–Massart weighted chi\-square tail\)\.

LetY1,…,YDY\_\{1\},\\dots,Y\_\{D\}be independent standard Gaussian random variables, and leta1,…,aD≥0a\_\{1\},\\dots,a\_\{D\}\\geq 0\. Define

Z:=∑i=1Dai​\(Yi2−1\)\.Z\\vcentcolon=\\sum\_\{i=1\}^\{D\}a\_\{i\}\(Y\_\{i\}^\{2\}\-1\)\.Then, for everyx\>0x\>0,

ℙ​\(Z≥2​‖a‖2​x\+2​‖a‖∞​x\)≤e−x,\\mathbb\{P\}\\left\(Z\\geq 2\\\|a\\\|\_\{2\}\\sqrt\{x\}\+2\\\|a\\\|\_\{\\infty\}x\\right\)\\leq e^\{\-x\},and

ℙ​\(Z≤−2​‖a‖2​x\)≤e−x\.\\mathbb\{P\}\\left\(Z\\leq\-2\\\|a\\\|\_\{2\}\\sqrt\{x\}\\right\)\\leq e^\{\-x\}\.In particular, ifQ∼χD2Q\\sim\\chi^\{2\}\_\{D\}, then

ℙ​\(Q−D≥2​D​x\+2​x\)≤e−x,\\mathbb\{P\}\\left\(Q\-D\\geq 2\\sqrt\{Dx\}\+2x\\right\)\\leq e^\{\-x\},and

ℙ​\(D−Q≥2​D​x\)≤e−x\.\\mathbb\{P\}\\left\(D\-Q\\geq 2\\sqrt\{Dx\}\\right\)\\leq e^\{\-x\}\.

This is Laurent–Massart\(Laurent and Massart,[2000](https://arxiv.org/html/2607.10034#bib.bib83), Lemma 1\)\.

###### Lemma B\.54\(Weighted product\-Gaussian chaos\)\.

Let\(gr,hr\)r=1m\(g\_\{r\},h\_\{r\}\)\_\{r=1\}^\{m\}be independent pairs of independent standard Gaussians, and leta=\(a1,…,am\)∈ℝma=\(a\_\{1\},\\dots,a\_\{m\}\)\\in\\mathbb\{R\}^\{m\}be deterministic\. Then, for everyt≥0t\\geq 0,

ℙ​\(\|∑r=1mar​gr​hr\|\>C​\(‖a‖2​t\+‖a‖∞​t\)\)≤2​e−t\.\\mathbb\{P\}\\left\(\\left\|\\sum\_\{r=1\}^\{m\}a\_\{r\}g\_\{r\}h\_\{r\}\\right\|\>C\\left\(\\\|a\\\|\_\{2\}\\sqrt\{t\}\+\\\|a\\\|\_\{\\infty\}t\\right\)\\right\)\\leq 2e^\{\-t\}\.

###### Proof\.

For independentg,h∼N​\(0,1\)g,h\\sim N\(0,1\), conditioning ongggives

𝔼​eλ​g​h=𝔼​eλ2​g2/2=\(1−λ2\)−1/2,\|λ\|<1\.\\mathbb\{E\}e^\{\\lambda gh\}=\\mathbb\{E\}e^\{\\lambda^\{2\}g^\{2\}/2\}=\(1\-\\lambda^\{2\}\)^\{\-1/2\},\\qquad\|\\lambda\|<1\.Therefore, for\|λ\|≤c/‖a‖∞\|\\lambda\|\\leq c/\\\|a\\\|\_\{\\infty\},

log⁡𝔼​exp⁡\(λ​∑r=1mar​gr​hr\)=−12​∑r=1mlog⁡\(1−λ2​ar2\)≤C​λ2​‖a‖22\.\\log\\mathbb\{E\}\\exp\\left\(\\lambda\\sum\_\{r=1\}^\{m\}a\_\{r\}g\_\{r\}h\_\{r\}\\right\)=\-\\frac\{1\}\{2\}\\sum\_\{r=1\}^\{m\}\\log\(1\-\\lambda^\{2\}a\_\{r\}^\{2\}\)\\leq C\\lambda^\{2\}\\\|a\\\|\_\{2\}^\{2\}\.Ifa=0a=0, the claim is trivial\. Otherwise, Chernoff’s bound gives

ℙ​\(∑r=1mar​gr​hr\>ν\)≤exp⁡\(−λ​ν\+C​λ2​‖a‖22\)\\mathbb\{P\}\\left\(\\sum\_\{r=1\}^\{m\}a\_\{r\}g\_\{r\}h\_\{r\}\>\\nu\\right\)\\leq\\exp\\left\(\-\\lambda\\nu\+C\\lambda^\{2\}\\\|a\\\|\_\{2\}^\{2\}\\right\)for every0≤λ≤c/‖a‖∞0\\leq\\lambda\\leq c/\\\|a\\\|\_\{\\infty\}\. Optimizing with

λ=c​min⁡\{ν‖a‖22,1‖a‖∞\}\\lambda=c\\min\\left\\\{\\frac\{\\nu\}\{\\\|a\\\|\_\{2\}^\{2\}\},\\frac\{1\}\{\\\|a\\\|\_\{\\infty\}\}\\right\\\}gives

ℙ​\(∑r=1mar​gr​hr\>C​\(‖a‖2​t\+‖a‖∞​t\)\)≤e−t\.\\mathbb\{P\}\\left\(\\sum\_\{r=1\}^\{m\}a\_\{r\}g\_\{r\}h\_\{r\}\>C\(\\\|a\\\|\_\{2\}\\sqrt\{t\}\+\\\|a\\\|\_\{\\infty\}t\)\\right\)\\leq e^\{\-t\}\.Apply the same argument to−∑rar​gr​hr\-\\sum\_\{r\}a\_\{r\}g\_\{r\}h\_\{r\}and union bound\. ∎

###### Lemma B\.55\(Square\-sum concentration from a two\-level tail\)\.

LetX1,…,XnX\_\{1\},\\dots,X\_\{n\}be independent random variables\. Assume that, for constantsv\>0v\>0,r\>0r\>0, andC0≥1C\_\{0\}\\geq 1,

𝔼​Xi2≤v\\mathbb\{E\}X\_\{i\}^\{2\}\\leq vand, for everyu≥1u\\geq 1,

ℙ​\(\|Xi\|\>C0​\(v​u\+r​u\)\)≤2​e−u\.\\mathbb\{P\}\\left\(\|X\_\{i\}\|\>C\_\{0\}\(\\sqrt\{vu\}\+ru\)\\right\)\\leq 2e^\{\-u\}\.Assume also that

r2≤C0​v\.r^\{2\}\\leq C\_\{0\}v\.Then, for everys≥1s\\geq 1,

ℙ​\(∑i=1nXi2\>C​\(n​v\+v​s\+r2​s2\)\)≤C​e−s,\\mathbb\{P\}\\left\(\\sum\_\{i=1\}^\{n\}X\_\{i\}^\{2\}\>C\\left\(nv\+vs\+r^\{2\}s^\{2\}\\right\)\\right\)\\leq Ce^\{\-s\},whereCCdepends only onC0C\_\{0\}\.

###### Proof\.

We first reduce the problem to sums of variables with exponential tails\. For eachii, letRiR\_\{i\}have the same distribution as\|Xi\|\|X\_\{i\}\|, with the variablesR1,…,RnR\_\{1\},\\dots,R\_\{n\}independent\. Define

Ti:=inf\{u≥1:Ri≤C0​\(v​u\+r​u\)\}\.T\_\{i\}\\vcentcolon=\\inf\\left\\\{u\\geq 1:R\_\{i\}\\leq C\_\{0\}\(\\sqrt\{vu\}\+ru\)\\right\\\}\.Then, by definition,

Ri≤C0​\(v​Ti\+r​Ti\)\.R\_\{i\}\\leq C\_\{0\}\(\\sqrt\{vT\_\{i\}\}\+rT\_\{i\}\)\.Moreover, for everyu≥1u\\geq 1, ifTi\>uT\_\{i\}\>u, then

Ri\>C0​\(v​u\+r​u\),R\_\{i\}\>C\_\{0\}\(\\sqrt\{vu\}\+ru\),and hence the assumed tail bound gives

ℙ​\(Ti\>u\)≤2​e−u\.\\mathbb\{P\}\(T\_\{i\}\>u\)\\leq 2e^\{\-u\}\.Since the desired estimate depends only on the marginal laws and independence, we may work with this coupled representation and write, after changing the absolute constant,

\|Xi\|≤C​\(v​Ti\+r​Ti\),ℙ​\(Ti\>u\)≤2​e−u,u≥1\.\|X\_\{i\}\|\\leq C\(\\sqrt\{vT\_\{i\}\}\+rT\_\{i\}\),\\qquad\\mathbb\{P\}\(T\_\{i\}\>u\)\\leq 2e^\{\-u\},\\quad u\\geq 1\.
Squaring the domination gives

Xi2≤C​\(v​Ti\+r2​Ti2\)\.X\_\{i\}^\{2\}\\leq C\\left\(vT\_\{i\}\+r^\{2\}T\_\{i\}^\{2\}\\right\)\.Hence

∑i=1nXi2≤C​\(v​∑i=1nTi\+r2​∑i=1nTi2\)\.\\sum\_\{i=1\}^\{n\}X\_\{i\}^\{2\}\\leq C\\left\(v\\sum\_\{i=1\}^\{n\}T\_\{i\}\+r^\{2\}\\sum\_\{i=1\}^\{n\}T\_\{i\}^\{2\}\\right\)\.It remains to control the two sums involvingTiT\_\{i\}\.

The tail boundℙ​\(Ti\>u\)≤2​e−u\\mathbb\{P\}\(T\_\{i\}\>u\)\\leq 2e^\{\-u\}implies, for everyp≥2p\\geq 2,

‖Ti‖Lp≤C​p,‖Ti2‖Lp=‖Ti‖L2​p2≤C​p2\.\\\|T\_\{i\}\\\|\_\{L\_\{p\}\}\\leq Cp,\\qquad\\\|T\_\{i\}^\{2\}\\\|\_\{L\_\{p\}\}=\\\|T\_\{i\}\\\|\_\{L\_\{2p\}\}^\{2\}\\leq Cp^\{2\}\.Consequently,

‖Ti−𝔼​Ti‖Lp≤C​p,‖Ti2−𝔼​Ti2‖Lp≤C​p2\.\\\|T\_\{i\}\-\\mathbb\{E\}T\_\{i\}\\\|\_\{L\_\{p\}\}\\leq Cp,\\qquad\\\|T\_\{i\}^\{2\}\-\\mathbb\{E\}T\_\{i\}^\{2\}\\\|\_\{L\_\{p\}\}\\leq Cp^\{2\}\.Also𝔼​Ti≤C\\mathbb\{E\}T\_\{i\}\\leq Cand𝔼​Ti2≤C\\mathbb\{E\}T\_\{i\}^\{2\}\\leq C\. Applying Theorem[B\.51](https://arxiv.org/html/2607.10034#A2.Thmtheorem51)to the centered variablesTi−𝔼​TiT\_\{i\}\-\\mathbb\{E\}T\_\{i\}withρ=1\\rho=1gives

ℙ​\(∑i=1nTi\>C​\(n\+s\)\)≤C​e−s\.\\mathbb\{P\}\\left\(\\sum\_\{i=1\}^\{n\}T\_\{i\}\>C\(n\+s\)\\right\)\\leq Ce^\{\-s\}\.Similarly, applying Theorem[B\.51](https://arxiv.org/html/2607.10034#A2.Thmtheorem51)toTi2−𝔼​Ti2T\_\{i\}^\{2\}\-\\mathbb\{E\}T\_\{i\}^\{2\}withρ=2\\rho=2gives

ℙ​\(∑i=1nTi2\>C​\(n\+s2\)\)≤C​e−s\.\\mathbb\{P\}\\left\(\\sum\_\{i=1\}^\{n\}T\_\{i\}^\{2\}\>C\(n\+s^\{2\}\)\\right\)\\leq Ce^\{\-s\}\.On the intersection of these two events,

∑i=1nXi2≤C​\(v​\(n\+s\)\+r2​\(n\+s2\)\)\.\\sum\_\{i=1\}^\{n\}X\_\{i\}^\{2\}\\leq C\\left\(v\(n\+s\)\+r^\{2\}\(n\+s^\{2\}\)\\right\)\.Sincer2≤C0​vr^\{2\}\\leq C\_\{0\}v, the termr2​nr^\{2\}nis absorbed byv​nvn\. Therefore

∑i=1nXi2≤C​\(n​v\+v​s\+r2​s2\)\\sum\_\{i=1\}^\{n\}X\_\{i\}^\{2\}\\leq C\\left\(nv\+vs\+r^\{2\}s^\{2\}\\right\)with probability at least1−C​e−s1\-Ce^\{\-s\}\. ∎

###### Lemma B\.56\(Spherical quadratic\-form tail\)\.

Letx∼Unif​\(𝕊d−1\)x\\sim\\mathrm\{Unif\}\(\\mathbb\{S\}^\{d\-1\}\), and letB∈ℝd×dB\\in\\mathbb\{R\}^\{d\\times d\}be symmetric and trace\-free\. Then, for everyu≥1u\\geq 1,

ℙ​\(\|x⊤​B​x\|\>C​\(‖B‖Fd​u\+‖B‖opd​u\)\)≤2​e−u\.\\mathbb\{P\}\\left\(\|x^\{\\top\}Bx\|\>C\\left\(\\frac\{\\\|B\\\|\_\{F\}\}\{d\}\\sqrt\{u\}\+\\frac\{\\\|B\\\|\_\{\\mathrm\{op\}\}\}\{d\}u\\right\)\\right\)\\leq 2e^\{\-u\}\.

###### Proof\.

Write

x=g‖g‖2,g∼𝒩​\(0,Id\)\.x=\\frac\{g\}\{\\\|g\\\|\_\{2\}\},\\qquad g\\sim\\mathcal\{N\}\(0,I\_\{d\}\)\.DiagonalizeB=Q​diag⁡\(λ1,…,λd\)​Q⊤B=Q\\operatorname\{diag\}\(\\lambda\_\{1\},\\dots,\\lambda\_\{d\}\)Q^\{\\top\}\. By rotational invariance,

g⊤​B​g=d∑i=1dλi​gi2\.g^\{\\top\}Bg\\stackrel\{\{\\scriptstyle d\}\}\{\{=\}\}\\sum\_\{i=1\}^\{d\}\\lambda\_\{i\}g\_\{i\}^\{2\}\.Sincetr⁡B=∑iλi=0\\operatorname\{tr\}B=\\sum\_\{i\}\\lambda\_\{i\}=0,

g⊤​B​g=∑i=1dλi​\(gi2−1\)\.g^\{\\top\}Bg=\\sum\_\{i=1\}^\{d\}\\lambda\_\{i\}\(g\_\{i\}^\{2\}\-1\)\.Apply Theorem[B\.53](https://arxiv.org/html/2607.10034#A2.Thmtheorem53)to the positive and negative parts of the weightsλi\\lambda\_\{i\}\. Equivalently, apply it to the two weighted sums associated withλi\+\\lambda\_\{i\}^\{\+\}andλi−\\lambda\_\{i\}^\{\-\}\. This gives, for allu≥1u\\geq 1,

ℙ​\(\|g⊤​B​g\|\>C​\(‖B‖F​u\+‖B‖op​u\)\)≤2​e−u\.\\mathbb\{P\}\\left\(\|g^\{\\top\}Bg\|\>C\\left\(\\\|B\\\|\_\{F\}\\sqrt\{u\}\+\\\|B\\\|\_\{\\mathrm\{op\}\}u\\right\)\\right\)\\leq 2e^\{\-u\}\.Choose a numerical constanta≥1a\\geq 1, to be fixed below\. Applying the above Gaussian quadratic\-form bound witha​uauin place ofuu, we get

ℙ​\(\|g⊤​B​g\|\>Ca​\(‖B‖F​u\+‖B‖op​u\)\)≤2​e−a​u\.\\mathbb\{P\}\\left\(\|g^\{\\top\}Bg\|\>C\_\{a\}\\left\(\\\|B\\\|\_\{F\}\\sqrt\{u\}\+\\\|B\\\|\_\{\\mathrm\{op\}\}u\\right\)\\right\)\\leq 2e^\{\-au\}\.Also, by the lower\-tail part of Theorem[B\.53](https://arxiv.org/html/2607.10034#A2.Thmtheorem53)applied to‖g‖22∼χd2\\\|g\\\|\_\{2\}^\{2\}\\sim\\chi\_\{d\}^\{2\},

ℙ​\(‖g‖22<d/2\)≤e−c​d\.\\mathbb\{P\}\\left\(\\\|g\\\|\_\{2\}^\{2\}<d/2\\right\)\\leq e^\{\-cd\}\.Ifu≤c1​du\\leq c\_\{1\}d, withc1\>0c\_\{1\}\>0chosen small enough, thene−c​d≤e−2​ue^\{\-cd\}\\leq e^\{\-2u\}\. On the event‖g‖22≥d/2\\\|g\\\|\_\{2\}^\{2\}\\geq d/2,

\|x⊤​B​x\|=\|g⊤​B​g\|‖g‖22≤Ca​\(‖B‖Fd​u\+‖B‖opd​u\)\.\|x^\{\\top\}Bx\|=\\frac\{\|g^\{\\top\}Bg\|\}\{\\\|g\\\|\_\{2\}^\{2\}\}\\leq C\_\{a\}\\left\(\\frac\{\\\|B\\\|\_\{F\}\}\{d\}\\sqrt\{u\}\+\\frac\{\\\|B\\\|\_\{\\mathrm\{op\}\}\}\{d\}u\\right\)\.Thus, foru≤c1​du\\leq c\_\{1\}d, the failure probability is at most2​e−a​u\+e−2​u2e^\{\-au\}\+e^\{\-2u\}\. Choosingaalarge enough and enlarging the constant in the threshold gives failure probability at most2​e−u2e^\{\-u\}\. Ifu\>c1​du\>c\_\{1\}d, then the same inequality holds deterministically after enlargingCC, because

\|x⊤​B​x\|≤‖B‖op≤C​‖B‖opd​u\.\|x^\{\\top\}Bx\|\\leq\\\|B\\\|\_\{\\mathrm\{op\}\}\\leq C\\frac\{\\\|B\\\|\_\{\\mathrm\{op\}\}\}\{d\}u\.This proves the claim\. ∎

###### Lemma B\.57\(Spherical quadratic square\-sum bound\)\.

Letx1,…,xnx\_\{1\},\\dots,x\_\{n\}be i\.i\.d\. uniform on𝕊d−1⊂ℝd\\mathbb\{S\}^\{d\-1\}\\subset\\mathbb\{R\}^\{d\}\. LetB∈ℝd×dB\\in\\mathbb\{R\}^\{d\\times d\}be deterministic, symmetric, and trace\-free, and letβ∈ℝ\\beta\\in\\mathbb\{R\}\. Define

Xℓ:=xℓ⊤​B​xℓ\+β\.X\_\{\\ell\}\\vcentcolon=x\_\{\\ell\}^\{\\top\}Bx\_\{\\ell\}\+\\beta\.Set

v:=2​‖B‖F2d​\(d\+2\)\+β2,r:=‖B‖opd\+\|β\|\.v\\vcentcolon=\\frac\{2\\\|B\\\|\_\{F\}^\{2\}\}\{d\(d\+2\)\}\+\\beta^\{2\},\\qquad r\\vcentcolon=\\frac\{\\\|B\\\|\_\{\\mathrm\{op\}\}\}\{d\}\+\|\\beta\|\.Then, for everys≥1s\\geq 1,

ℙ​\(∑ℓ=1nXℓ2\>C​\(n​v\+v​s\+r2​s2\)\)≤e−s\.\\mathbb\{P\}\\left\(\\sum\_\{\\ell=1\}^\{n\}X\_\{\\ell\}^\{2\}\>C\\left\(nv\+vs\+r^\{2\}s^\{2\}\\right\)\\right\)\\leq e^\{\-s\}\.

###### Proof\.

Forx∼Unif​\(𝕊d−1\)x\\sim\\mathrm\{Unif\}\(\\mathbb\{S\}^\{d\-1\}\), isotropy gives

𝔼​\[x⊤​B​x\]=tr⁡Bd=0\.\\mathbb\{E\}\[x^\{\\top\}Bx\]=\\frac\{\\operatorname\{tr\}B\}\{d\}=0\.The fourth\-moment identity for the sphere gives

𝔼​\[\(x⊤​B​x\)2\]=\(tr⁡B\)2\+2​‖B‖F2d​\(d\+2\)=2​‖B‖F2d​\(d\+2\)\.\\mathbb\{E\}\[\(x^\{\\top\}Bx\)^\{2\}\]=\\frac\{\(\\operatorname\{tr\}B\)^\{2\}\+2\\\|B\\\|\_\{F\}^\{2\}\}\{d\(d\+2\)\}=\\frac\{2\\\|B\\\|\_\{F\}^\{2\}\}\{d\(d\+2\)\}\.Therefore

𝔼​Xℓ2=v\.\\mathbb\{E\}X\_\{\\ell\}^\{2\}=v\.Moreover,

\(‖B‖opd\)2≤‖B‖F2d2≤C​2​‖B‖F2d​\(d\+2\)≤C​v,\\left\(\\frac\{\\\|B\\\|\_\{\\mathrm\{op\}\}\}\{d\}\\right\)^\{2\}\\leq\\frac\{\\\|B\\\|\_\{F\}^\{2\}\}\{d^\{2\}\}\\leq C\\frac\{2\\\|B\\\|\_\{F\}^\{2\}\}\{d\(d\+2\)\}\\leq Cv,andβ2≤v\\beta^\{2\}\\leq v, so

By Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5), for everyu≥1u\\geq 1,

ℙ​\(\|x⊤​B​x\|\>C​\(‖B‖Fd​u\+‖B‖opd​u\)\)≤2​e−u\.\\mathbb\{P\}\\left\(\|x^\{\\top\}Bx\|\>C\\left\(\\frac\{\\\|B\\\|\_\{F\}\}\{d\}\\sqrt\{u\}\+\\frac\{\\\|B\\\|\_\{\\mathrm\{op\}\}\}\{d\}u\\right\)\\right\)\\leq 2e^\{\-u\}\.Since‖B‖F/d≤C​v\\\|B\\\|\_\{F\}/d\\leq C\\sqrt\{v\},‖B‖op/d≤r\\\|B\\\|\_\{\\mathrm\{op\}\}/d\\leq r, and\|β\|≤v\|\\beta\|\\leq\\sqrt\{v\}, this implies

ℙ​\(\|Xℓ\|\>C​\(v​u\+r​u\)\)≤2​e−u\.\\mathbb\{P\}\\left\(\|X\_\{\\ell\}\|\>C\(\\sqrt\{vu\}\+ru\)\\right\)\\leq 2e^\{\-u\}\.Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5)applies becauser2≤C​vr^\{2\}\\leq Cv\. It gives

ℙ​\(∑ℓ=1nXℓ2\>C​\(n​v\+v​s\+r2​s2\)\)≤C​e−s\.\\mathbb\{P\}\\left\(\\sum\_\{\\ell=1\}^\{n\}X\_\{\\ell\}^\{2\}\>C\\left\(nv\+vs\+r^\{2\}s^\{2\}\\right\)\\right\)\\leq Ce^\{\-s\}\.Replacingssin the equation above bys\+c0s\+c\_\{0\}, for a sufficiently large absolute constantc0c\_\{0\}, and enlargingCC, changes the right\-hand side toe−se^\{\-s\}and leaves the threshold in the same form\. This proves the claim\. ∎

###### Lemma B\.58\(Scalar weight regularity for one bilinear column\)\.

Letα1,…,αm,β1,…,βm\\alpha\_\{1\},\\dots,\\alpha\_\{m\},\\beta\_\{1\},\\dots,\\beta\_\{m\}be independent standard Gaussians, and define

γr=αr​βr,ηr=αr​βr2,ζr=αr2​βr\.\\gamma\_\{r\}=\\alpha\_\{r\}\\beta\_\{r\},\\qquad\\eta\_\{r\}=\\alpha\_\{r\}\\beta\_\{r\}^\{2\},\\qquad\\zeta\_\{r\}=\\alpha\_\{r\}^\{2\}\\beta\_\{r\}\.Ifm≥C∗​L3m\\geq C\_\{\*\}L^\{3\}, then, with probability at least1−C​e−L1\-Ce^\{\-L\},

\|1m​∑r=1m\(αr2​βr2−1\)\|≤C​\(Lm\+L2m\),\\left\|\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\(\\alpha\_\{r\}^\{2\}\\beta\_\{r\}^\{2\}\-1\)\\right\|\\leq C\\left\(\\sqrt\{\\frac\{L\}\{m\}\}\+\\frac\{L^\{2\}\}\{m\}\\right\),∑r=1mγr2≤C​m,∑r=1mηr2≤C​m,∑r=1mζr2≤C​m,\\sum\_\{r=1\}^\{m\}\\gamma\_\{r\}^\{2\}\\leq Cm,\\qquad\\sum\_\{r=1\}^\{m\}\\eta\_\{r\}^\{2\}\\leq Cm,\\qquad\\sum\_\{r=1\}^\{m\}\\zeta\_\{r\}^\{2\}\\leq Cm,and

‖γ‖∞2​L≤C​m\.\\\|\\gamma\\\|\_\{\\infty\}^\{2\}L\\leq Cm\.

###### Proof\.

The variableα2​β2−1\\alpha^\{2\}\\beta^\{2\}\-1is a centered degree\-four Gaussian polynomial with boundedL2L\_\{2\}\-norm\. By Theorem[B\.50](https://arxiv.org/html/2607.10034#A2.Thmtheorem50),

‖α2​β2−1‖Lp≤C​p2,p≥2\.\\\|\\alpha^\{2\}\\beta^\{2\}\-1\\\|\_\{L\_\{p\}\}\\leq Cp^\{2\},\\qquad p\\geq 2\.Applying Theorem[B\.51](https://arxiv.org/html/2607.10034#A2.Thmtheorem51)withρ=2\\rho=2gives

\|1m​∑r=1m\(αr2​βr2−1\)\|≤C​\(Lm\+L2m\)\\left\|\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\(\\alpha\_\{r\}^\{2\}\\beta\_\{r\}^\{2\}\-1\)\\right\|\\leq C\\left\(\\sqrt\{\\frac\{L\}\{m\}\}\+\\frac\{L^\{2\}\}\{m\}\\right\)with probability at least1−2​e−L1\-2e^\{\-L\}\.

Next,

𝔼​γr2=1,𝔼​ηr2=3,𝔼​ζr2=3\.\\mathbb\{E\}\\gamma\_\{r\}^\{2\}=1,\\qquad\\mathbb\{E\}\\eta\_\{r\}^\{2\}=3,\\qquad\\mathbb\{E\}\\zeta\_\{r\}^\{2\}=3\.The centered variables

γr2−1,ηr2−3,ζr2−3\\gamma\_\{r\}^\{2\}\-1,\\qquad\\eta\_\{r\}^\{2\}\-3,\\qquad\\zeta\_\{r\}^\{2\}\-3are centered Gaussian polynomials of degrees4,6,64,6,6, respectively, with boundedL2L\_\{2\}\-norms\. Applying Theorem[B\.51](https://arxiv.org/html/2607.10034#A2.Thmtheorem51)withρ=2\\rho=2forγr2−1\\gamma\_\{r\}^\{2\}\-1andρ=3\\rho=3for the degree\-six terms gives

1m​∑r=1mγr2≤C,1m​∑r=1mηr2≤C,1m​∑r=1mζr2≤C\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\\gamma\_\{r\}^\{2\}\\leq C,\\qquad\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\\eta\_\{r\}^\{2\}\\leq C,\\qquad\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\\zeta\_\{r\}^\{2\}\\leq Cwith probability at least1−C​e−L1\-Ce^\{\-L\}, providedm≥C∗​L3m\\geq C\_\{\*\}L^\{3\}\.

It remains to control‖γ‖∞\\\|\\gamma\\\|\_\{\\infty\}\. Since

2​\|α​β\|≤α2\+β2,2\|\\alpha\\beta\|\\leq\\alpha^\{2\}\+\\beta^\{2\},andα2\+β2∼χ22\\alpha^\{2\}\+\\beta^\{2\}\\sim\\chi^\{2\}\_\{2\}, there is an absolute constantc\>0c\>0such that

ℙ​\(\|α​β\|\>u\)≤2​e−c​u\.\\mathbb\{P\}\(\|\\alpha\\beta\|\>u\)\\leq 2e^\{\-cu\}\.Thus

ℙ​\(‖γ‖∞2​L\>C​m\)≤2​m​exp⁡\(−c​C​mL\)\.\\mathbb\{P\}\\left\(\\\|\\gamma\\\|\_\{\\infty\}^\{2\}L\>Cm\\right\)\\leq 2m\\exp\\left\(\-c\\sqrt\{\\frac\{Cm\}\{L\}\}\\right\)\.Sincem≥C∗​L3m\\geq C\_\{\*\}L^\{3\}, writem=L3​ym=L^\{3\}ywithy≥C∗y\\geq C\_\{\*\}\. Then

mL=L​y,\\sqrt\{\\frac\{m\}\{L\}\}=L\\sqrt\{y\},whereas

L\+log⁡\(2​m\)\+1=L\+log⁡\(2​L3​y\)\+1≤C​\(L\+log⁡y\)≤C​L​y\.L\+\\log\(2m\)\+1=L\+\\log\(2L^\{3\}y\)\+1\\leq C\(L\+\\log y\)\\leq CL\\sqrt\{y\}\.Choosing the absolute constantsC∗C\_\{\*\}andCCsufficiently large therefore ensures

c​C​mL≥L\+log⁡\(2​m\)\+1\.c\\sqrt\{\\frac\{Cm\}\{L\}\}\\geq L\+\\log\(2m\)\+1\.Therefore

2​m​exp⁡\(−c​C​mL\)≤e−L\.2m\\exp\\left\(\-c\\sqrt\{\\frac\{Cm\}\{L\}\}\\right\)\\leq e^\{\-L\}\.A union bound over the above events completes the proof\. ∎

###### Lemma B\.59\(Weighted Gaussian product bounds\)\.

LetU,V∈ℝn×mU,V\\in\\mathbb\{R\}^\{n\\times m\}have independent standard Gaussian entries, and letΓ=diag⁡\(γ1,…,γm\)\\Gamma=\\operatorname\{diag\}\(\\gamma\_\{1\},\\dots,\\gamma\_\{m\}\)be deterministic\. Then, for everys≥1s\\geq 1, with probability at least1−4​e−s1\-4e^\{\-s\},

‖U​Γ​V⊤‖op≤C​\(‖γ‖2​n\+s\+‖γ‖∞​\(n\+s\)\),\\\|U\\Gamma V^\{\\top\}\\\|\_\{\\mathrm\{op\}\}\\leq C\\left\(\\\|\\gamma\\\|\_\{2\}\\sqrt\{n\+s\}\+\\\|\\gamma\\\|\_\{\\infty\}\(n\+s\)\\right\),and

\|tr⁡\(U​Γ​V⊤\)\|≤C​\(n​‖γ‖2​s\+‖γ‖∞​s\)\.\\left\|\\operatorname\{tr\}\(U\\Gamma V^\{\\top\}\)\\right\|\\leq C\\left\(\\sqrt\{n\}\\,\\\|\\gamma\\\|\_\{2\}\\sqrt\{s\}\+\\\|\\gamma\\\|\_\{\\infty\}s\\right\)\.Furthermore, if

‖γ‖22≤C0​m,‖γ‖∞2​s≤C0​m,n≥s,\\\|\\gamma\\\|\_\{2\}^\{2\}\\leq C\_\{0\}m,\\qquad\\\|\\gamma\\\|\_\{\\infty\}^\{2\}s\\leq C\_\{0\}m,\\qquad n\\geq s,then, with probability at least1−6​e−s1\-6e^\{\-s\},

‖U​Γ​V⊤‖F2≤C​n2​m\.\\\|U\\Gamma V^\{\\top\}\\\|\_\{F\}^\{2\}\\leq Cn^\{2\}m\.

###### Proof\.

For fixedx,y∈𝕊n−1x,y\\in\\mathbb\{S\}^\{n\-1\},

x⊤​U​Γ​V⊤​y=∑r=1mγr​\(ur⊤​x\)​\(vr⊤​y\),x^\{\\top\}U\\Gamma V^\{\\top\}y=\\sum\_\{r=1\}^\{m\}\\gamma\_\{r\}\(u\_\{r\}^\{\\top\}x\)\(v\_\{r\}^\{\\top\}y\),whereur,vru\_\{r\},v\_\{r\}are the columns ofU,VU,V\. The variables\(ur⊤​x\)​\(vr⊤​y\)\(u\_\{r\}^\{\\top\}x\)\(v\_\{r\}^\{\\top\}y\)are independent products of independent standard Gaussians\. Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5)gives, for everyt≥0t\\geq 0,

ℙ​\(\|x⊤​U​Γ​V⊤​y\|\>C​\(‖γ‖2​t\+‖γ‖∞​t\)\)≤2​e−t\.\\mathbb\{P\}\\left\(\|x^\{\\top\}U\\Gamma V^\{\\top\}y\|\>C\(\\\|\\gamma\\\|\_\{2\}\\sqrt\{t\}\+\\\|\\gamma\\\|\_\{\\infty\}t\)\\right\)\\leq 2e^\{\-t\}\.Take1/41/4\-nets𝒩,ℳ\\mathcal\{N\},\\mathcal\{M\}of𝕊n−1\\mathbb\{S\}^\{n\-1\}with cardinalities at most9n9^\{n\}\. Sett=s\+2​n​log⁡9t=s\+2n\\log 9, union bound over𝒩×ℳ\\mathcal\{N\}\\times\\mathcal\{M\}, and apply Theorem[B\.52](https://arxiv.org/html/2607.10034#A2.Thmtheorem52)\. This proves the operator bound\.

For the trace,

tr⁡\(U​Γ​V⊤\)=∑ℓ=1n∑r=1mγr​Uℓ​r​Vℓ​r\.\\operatorname\{tr\}\(U\\Gamma V^\{\\top\}\)=\\sum\_\{\\ell=1\}^\{n\}\\sum\_\{r=1\}^\{m\}\\gamma\_\{r\}U\_\{\\ell r\}V\_\{\\ell r\}\.This is again a weighted sum of products of independent standard Gaussians, with weightsγr\\gamma\_\{r\}repeatednntimes\. Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5)therefore gives the stated trace bound\.

For the Frobenius bound, first note

‖U​Γ‖F2=∑r=1mγr2​‖ur‖22\.\\\|U\\Gamma\\\|\_\{F\}^\{2\}=\\sum\_\{r=1\}^\{m\}\\gamma\_\{r\}^\{2\}\\\|u\_\{r\}\\\|\_\{2\}^\{2\}\.This is a weighted chi\-square variable with meann​‖γ‖22≤C​n​mn\\\|\\gamma\\\|\_\{2\}^\{2\}\\leq Cnm\. By Theorem[B\.53](https://arxiv.org/html/2607.10034#A2.Thmtheorem53), using‖γ‖∞2​s≤C0​m\\\|\\gamma\\\|\_\{\\infty\}^\{2\}s\\leq C\_\{0\}m, we get

‖U​Γ‖F2≤C​n​m\\\|U\\Gamma\\\|\_\{F\}^\{2\}\\leq Cnmwith probability at least1−e−s1\-e^\{\-s\}\.

Condition onU,ΓU,\\Gamma, and set

R:=Γ​U⊤​U​Γ\.R\\vcentcolon=\\Gamma U^\{\\top\}U\\Gamma\.Then

‖U​Γ​V⊤‖F2=tr⁡\(V​R​V⊤\)\.\\\|U\\Gamma V^\{\\top\}\\\|\_\{F\}^\{2\}=\\operatorname\{tr\}\(VRV^\{\\top\}\)\.Writing the rows ofVVasg1,…,gn∈ℝmg\_\{1\},\\dots,g\_\{n\}\\in\\mathbb\{R\}^\{m\},

tr⁡\(V​R​V⊤\)=∑ℓ=1ngℓ⊤​R​gℓ\.\\operatorname\{tr\}\(VRV^\{\\top\}\)=\\sum\_\{\\ell=1\}^\{n\}g\_\{\\ell\}^\{\\top\}Rg\_\{\\ell\}\.DiagonalizeRR, and apply Theorem[B\.53](https://arxiv.org/html/2607.10034#A2.Thmtheorem53)\. With conditional probability at least1−e−s1\-e^\{\-s\},

tr⁡\(V​R​V⊤\)≤n​tr⁡R\+2​n​tr⁡\(R2\)​s\+2​‖R‖op​s\.\\operatorname\{tr\}\(VRV^\{\\top\}\)\\leq n\\operatorname\{tr\}R\+2\\sqrt\{n\\operatorname\{tr\}\(R^\{2\}\)s\}\+2\\\|R\\\|\_\{\\mathrm\{op\}\}s\.On\{‖U​Γ‖F2≤C​n​m\}\\\{\\\|U\\Gamma\\\|\_\{F\}^\{2\}\\leq Cnm\\\},

tr⁡R=‖U​Γ‖F2≤C​n​m,\\operatorname\{tr\}R=\\\|U\\Gamma\\\|\_\{F\}^\{2\}\\leq Cnm,‖R‖op≤tr⁡R≤C​n​m,\\\|R\\\|\_\{\\mathrm\{op\}\}\\leq\\operatorname\{tr\}R\\leq Cnm,and

tr⁡\(R2\)≤‖R‖op​tr⁡R≤C​n2​m2\.\\operatorname\{tr\}\(R^\{2\}\)\\leq\\\|R\\\|\_\{\\mathrm\{op\}\}\\operatorname\{tr\}R\\leq Cn^\{2\}m^\{2\}\.Sincen≥sn\\geq s, all three terms are bounded byC​n2​mCn^\{2\}m\. Thus

‖U​Γ​V⊤‖F2≤C​n2​m\.\\\|U\\Gamma V^\{\\top\}\\\|\_\{F\}^\{2\}\\leq Cn^\{2\}m\.Union bounding the events completes the proof\. ∎

###### Lemma B\.60\(Bilinear one\-column matrix regularity\)\.

Fixδ∈\(0,1\)\\delta\\in\(0,1\), and define

L:=log⁡\(C0​F2δ\)\.L\\vcentcolon=\\log\\left\(\\frac\{C\_\{0\}F^\{2\}\}\{\\delta\}\\right\)\.Assume

m≤d2L,m≥C∗​L3\.m\\leq\\frac\{d^\{2\}\}\{L\},\\qquad m\\geq C\_\{\*\}L^\{3\}\.Let𝐚1,𝐛1,…,𝐚m,𝐛m∈ℝd\{\\mathbf\{a\}\}\_\{1\},\{\\mathbf\{b\}\}\_\{1\},\\dots,\{\\mathbf\{a\}\}\_\{m\},\{\\mathbf\{b\}\}\_\{m\}\\in\\mathbb\{R\}^\{d\}be i\.i\.d\. standard Gaussian vectors—the bilinear random features of[SectionB\.2\.2](https://arxiv.org/html/2607.10034#A2.SS2.SSS2)\. Forq∈𝕊d−1q\\in\\mathbb\{S\}^\{d\-1\}, define

A​\(q\):=12​m​∑r=1m\(𝐚r⊤​q\)​\(𝐛r⊤​q\)​\(𝐚r​𝐛r⊤\+𝐛r​𝐚r⊤\),A\(q\)\\vcentcolon=\\frac\{1\}\{2m\}\\sum\_\{r=1\}^\{m\}\(\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}q\)\(\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}q\)\(\{\\mathbf\{a\}\}\_\{r\}\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}\+\{\\mathbf\{b\}\}\_\{r\}\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\),and

N​\(q\):=A​\(q\)−q​q⊤\.N\(q\)\\vcentcolon=A\(q\)\-qq^\{\\top\}\.Let𝐤1,…,𝐤F\{\\mathbf\{k\}\}\_\{1\},\\dots,\{\\mathbf\{k\}\}\_\{F\}be i\.i\.d\. uniform on𝕊d−1\\mathbb\{S\}^\{d\-1\}, independent of the features\. Then, with probability at least1−δ/41\-\\delta/4, simultaneously for alli∈\[F\]i\\in\[F\],

‖N​\(𝐤i\)‖F2≤C​d2m,\\\|N\(\{\\mathbf\{k\}\}\_\{i\}\)\\\|\_\{F\}^\{2\}\\leq C\\frac\{d^\{2\}\}\{m\},‖N​\(𝐤i\)‖op≤C​\(dm\+dm​L\),\\\|N\(\{\\mathbf\{k\}\}\_\{i\}\)\\\|\_\{\\mathrm\{op\}\}\\leq C\\left\(\\sqrt\{\\frac\{d\}\{m\}\}\+\\frac\{d\}\{\\sqrt\{mL\}\}\\right\),and

\|tr⁡N​\(𝐤i\)\|≤C​d​Lm\.\|\\operatorname\{tr\}N\(\{\\mathbf\{k\}\}\_\{i\}\)\|\\leq C\\sqrt\{\\frac\{dL\}\{m\}\}\.

###### Proof\.

It suffices to prove the result for a fixed deterministicq∈𝕊d−1q\\in\\mathbb\{S\}^\{d\-1\}with failure probability at mostC​e−LCe^\{\-L\}\. Conditional on the keys, the vectors𝐤i\{\\mathbf\{k\}\}\_\{i\}are deterministic and independent of the feature randomness\. A union bound overi∈\[F\]i\\in\[F\]gives failure probability at most

C​F​e−L=C​F​δC0​F2≤δ/4CFe^\{\-L\}=CF\\frac\{\\delta\}\{C\_\{0\}F^\{2\}\}\\leq\\delta/4forC0C\_\{0\}sufficiently large\.

Fixqq\. By rotational invariance, assumeq=e1q=e\_\{1\}\. Write

𝐚r=\(αr,ur\),𝐛r=\(βr,vr\),\{\\mathbf\{a\}\}\_\{r\}=\(\\alpha\_\{r\},u\_\{r\}\),\\qquad\{\\mathbf\{b\}\}\_\{r\}=\(\\beta\_\{r\},v\_\{r\}\),whereαr,βr∼N​\(0,1\)\\alpha\_\{r\},\\beta\_\{r\}\\sim N\(0,1\),ur,vr∼𝒩​\(0,Id−1\)u\_\{r\},v\_\{r\}\\sim\\mathcal\{N\}\(0,I\_\{d\-1\}\), and all variables are independent acrossrr\. Setn=d−1n=d\-1\. Therr\-th summand is

Sr=12​αr​βr​\(𝐚r​𝐛r⊤\+𝐛r​𝐚r⊤\),S\_\{r\}=\\frac\{1\}\{2\}\\alpha\_\{r\}\\beta\_\{r\}\(\{\\mathbf\{a\}\}\_\{r\}\{\\mathbf\{b\}\}\_\{r\}^\{\\top\}\+\{\\mathbf\{b\}\}\_\{r\}\{\\mathbf\{a\}\}\_\{r\}^\{\\top\}\),and𝔼​Sr=e1​e1⊤\\mathbb\{E\}S\_\{r\}=e\_\{1\}e\_\{1\}^\{\\top\}\. In block form,

N​\(e1\)=\(N11N1⟂⊤N1⟂N⟂⟂\),N\(e\_\{1\}\)=\\begin\{pmatrix\}N\_\{11\}&N\_\{1\\perp\}^\{\\top\}\\\\ N\_\{1\\perp\}&N\_\{\\perp\\perp\}\\end\{pmatrix\},where

N11=1m​∑r=1m\(αr2​βr2−1\),N\_\{11\}=\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\(\\alpha\_\{r\}^\{2\}\\beta\_\{r\}^\{2\}\-1\),N1⟂=12​m​∑r=1m\(ηr​ur\+ζr​vr\),N\_\{1\\perp\}=\\frac\{1\}\{2m\}\\sum\_\{r=1\}^\{m\}\(\\eta\_\{r\}u\_\{r\}\+\\zeta\_\{r\}v\_\{r\}\),and, withU,V∈ℝn×mU,V\\in\\mathbb\{R\}^\{n\\times m\}having columnsur,vru\_\{r\},v\_\{r\},Γ=diag⁡\(γ1,…,γm\)\\Gamma=\\operatorname\{diag\}\(\\gamma\_\{1\},\\dots,\\gamma\_\{m\}\),

N⟂⟂=12​m​\(U​Γ​V⊤\+V​Γ​U⊤\),N\_\{\\perp\\perp\}=\\frac\{1\}\{2m\}\(U\\Gamma V^\{\\top\}\+V\\Gamma U^\{\\top\}\),with

γr=αr​βr,ηr=αr​βr2,ζr=αr2​βr\.\\gamma\_\{r\}=\\alpha\_\{r\}\\beta\_\{r\},\\qquad\\eta\_\{r\}=\\alpha\_\{r\}\\beta\_\{r\}^\{2\},\\qquad\\zeta\_\{r\}=\\alpha\_\{r\}^\{2\}\\beta\_\{r\}\.
Let𝒢\\mathcal\{G\}be the good event from Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5)\. On𝒢\\mathcal\{G\},

\|N11\|≤C​\(Lm\+L2m\),\|N\_\{11\}\|\\leq C\\left\(\\sqrt\{\\frac\{L\}\{m\}\}\+\\frac\{L^\{2\}\}\{m\}\\right\),‖γ‖22≤C​m,‖γ‖∞2​L≤C​m,∑rηr2≤C​m,∑rζr2≤C​m\.\\\|\\gamma\\\|\_\{2\}^\{2\}\\leq Cm,\\qquad\\\|\\gamma\\\|\_\{\\infty\}^\{2\}L\\leq Cm,\\qquad\\sum\_\{r\}\\eta\_\{r\}^\{2\}\\leq Cm,\\qquad\\sum\_\{r\}\\zeta\_\{r\}^\{2\}\\leq Cm\.Alsoℙ​\(𝒢c\)≤C​e−L\\mathbb\{P\}\(\\mathcal\{G\}^\{c\}\)\\leq Ce^\{\-L\}\.

Conditional on the scalar weights,

Sη:=∑rηr​ur∼𝒩​\(0,\(∑rηr2\)​In\),Sζ:=∑rζr​vr∼𝒩​\(0,\(∑rζr2\)​In\)\.S\_\{\\eta\}\\vcentcolon=\\sum\_\{r\}\\eta\_\{r\}u\_\{r\}\\sim\\mathcal\{N\}\\left\(0,\\left\(\\sum\_\{r\}\\eta\_\{r\}^\{2\}\\right\)I\_\{n\}\\right\),\\qquad S\_\{\\zeta\}\\vcentcolon=\\sum\_\{r\}\\zeta\_\{r\}v\_\{r\}\\sim\\mathcal\{N\}\\left\(0,\\left\(\\sum\_\{r\}\\zeta\_\{r\}^\{2\}\\right\)I\_\{n\}\\right\)\.On𝒢\\mathcal\{G\}, both covariance scalars are at mostC​mCm\. Therefore‖Sη‖22/\(C​m\)\\\|S\_\{\\eta\}\\\|\_\{2\}^\{2\}/\(Cm\)and‖Sζ‖22/\(C​m\)\\\|S\_\{\\zeta\}\\\|\_\{2\}^\{2\}/\(Cm\)are stochastically dominated, up to an absolute constant, byχn2\\chi\_\{n\}^\{2\}\. By the chi\-square consequence of Theorem[B\.53](https://arxiv.org/html/2607.10034#A2.Thmtheorem53), with conditional failure probability at most2​e−L2e^\{\-L\},

‖Sη‖2≤C​m​\(n\+L\),‖Sζ‖2≤C​m​\(n\+L\)\.\\\|S\_\{\\eta\}\\\|\_\{2\}\\leq C\\sqrt\{m\}\(\\sqrt\{n\}\+\\sqrt\{L\}\),\\qquad\\\|S\_\{\\zeta\}\\\|\_\{2\}\\leq C\\sqrt\{m\}\(\\sqrt\{n\}\+\\sqrt\{L\}\)\.Hence

‖N1⟂‖2≤C​n\+Lm\.\\\|N\_\{1\\perp\}\\\|\_\{2\}\\leq C\\sqrt\{\\frac\{n\+L\}\{m\}\}\.Sincem≥C∗​L3m\\geq C\_\{\*\}L^\{3\}andm≤d2/Lm\\leq d^\{2\}/L,

d2≥m​L≥C∗​L4,d^\{2\}\\geq mL\\geq C\_\{\*\}L^\{4\},sod≥c​L2d\\geq cL^\{2\}andn\+L≤C​dn\+L\\leq Cd\. Therefore

‖N1⟂‖2≤C​dm\.\\\|N\_\{1\\perp\}\\\|\_\{2\}\\leq C\\sqrt\{\\frac\{d\}\{m\}\}\.
By Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5), on𝒢\\mathcal\{G\}, with conditional probability at least1−4​e−L1\-4e^\{\-L\},

‖U​Γ​V⊤‖op≤C​\(m​n\+L\+mL​\(n\+L\)\)\.\\\|U\\Gamma V^\{\\top\}\\\|\_\{\\mathrm\{op\}\}\\leq C\\left\(\\sqrt\{m\}\\sqrt\{n\+L\}\+\\sqrt\{\\frac\{m\}\{L\}\}\(n\+L\)\\right\)\.The same bound holds forV​Γ​U⊤V\\Gamma U^\{\\top\}\. Thus

‖N⟂⟂‖op≤C​\(n\+Lm\+n\+Lm​L\)≤C​\(dm\+dm​L\)\.\\\|N\_\{\\perp\\perp\}\\\|\_\{\\mathrm\{op\}\}\\leq C\\left\(\\sqrt\{\\frac\{n\+L\}\{m\}\}\+\\frac\{n\+L\}\{\\sqrt\{mL\}\}\\right\)\\leq C\\left\(\\sqrt\{\\frac\{d\}\{m\}\}\+\\frac\{d\}\{\\sqrt\{mL\}\}\\right\)\.Combining block estimates gives

‖N​\(e1\)‖op≤C​\(dm\+dm​L\)\.\\\|N\(e\_\{1\}\)\\\|\_\{\\mathrm\{op\}\}\\leq C\\left\(\\sqrt\{\\frac\{d\}\{m\}\}\+\\frac\{d\}\{\\sqrt\{mL\}\}\\right\)\.
For the Frobenius bound, Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5)gives

‖U​Γ​V⊤‖F2≤C​n2​m,\\\|U\\Gamma V^\{\\top\}\\\|\_\{F\}^\{2\}\\leq Cn^\{2\}m,because‖γ‖22≤C​m\\\|\\gamma\\\|\_\{2\}^\{2\}\\leq Cm,‖γ‖∞2​L≤C​m\\\|\\gamma\\\|\_\{\\infty\}^\{2\}L\\leq Cm, andn=d−1≥Ln=d\-1\\geq L\. Thus

‖N⟂⟂‖F2≤C​d2m\.\\\|N\_\{\\perp\\perp\}\\\|\_\{F\}^\{2\}\\leq C\\frac\{d^\{2\}\}\{m\}\.The scalar and vector blocks satisfy

\|N11\|2≤C​d2m,‖N1⟂‖22≤C​dm≤C​d2m\.\|N\_\{11\}\|^\{2\}\\leq C\\frac\{d^\{2\}\}\{m\},\\qquad\\\|N\_\{1\\perp\}\\\|\_\{2\}^\{2\}\\leq C\\frac\{d\}\{m\}\\leq C\\frac\{d^\{2\}\}\{m\}\.Therefore

‖N​\(e1\)‖F2≤C​d2m\.\\\|N\(e\_\{1\}\)\\\|\_\{F\}^\{2\}\\leq C\\frac\{d^\{2\}\}\{m\}\.
Finally,

tr⁡N​\(e1\)=N11\+tr⁡N⟂⟂=N11\+1m​tr⁡\(U​Γ​V⊤\)\.\\operatorname\{tr\}N\(e\_\{1\}\)=N\_\{11\}\+\\operatorname\{tr\}N\_\{\\perp\\perp\}=N\_\{11\}\+\\frac\{1\}\{m\}\\operatorname\{tr\}\(U\\Gamma V^\{\\top\}\)\.By Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5),

1m​\|tr⁡\(U​Γ​V⊤\)\|≤C​\(d​Lm\+Lm\)≤C​d​Lm\.\\frac\{1\}\{m\}\|\\operatorname\{tr\}\(U\\Gamma V^\{\\top\}\)\|\\leq C\\left\(\\sqrt\{\\frac\{dL\}\{m\}\}\+\\sqrt\{\\frac\{L\}\{m\}\}\\right\)\\leq C\\sqrt\{\\frac\{dL\}\{m\}\}\.The scalar termN11N\_\{11\}is absorbed by the same bound\. Hence

\|tr⁡N​\(e1\)\|≤C​d​Lm\.\|\\operatorname\{tr\}N\(e\_\{1\}\)\|\\leq C\\sqrt\{\\frac\{dL\}\{m\}\}\.This proves the fixed\-direction estimate\. Rotational invariance and the initial union bound over the keys complete the proof\. ∎

###### Lemma B\.61\(Bilinear one\-column quadratic\-form parameters\)\.

Assume the setting of Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5)\. Suppose also that

m≤d2L,L3≤c0​σ2​m2,m\\leq\\frac\{d^\{2\}\}\{L\},\\qquad L^\{3\}\\leq c\_\{0\}\\sigma^\{2\}m^\{2\},where

σ2=Θ​\(1d2\+1m\)\.\\sigma^\{2\}=\\Theta\\left\(\\frac\{1\}\{d^\{2\}\}\+\\frac\{1\}\{m\}\\right\)\.Forq∈𝕊d−1q\\in\\mathbb\{S\}^\{d\-1\}, define

τ​\(q\):=tr⁡A​\(q\)d,B​\(q\):=A​\(q\)−τ​\(q\)​Id,β​\(q\):=τ​\(q\)−1d\.\\tau\(q\)\\vcentcolon=\\frac\{\\operatorname\{tr\}A\(q\)\}\{d\},\\qquad B\(q\)\\vcentcolon=A\(q\)\-\\tau\(q\)I\_\{d\},\\qquad\\beta\(q\)\\vcentcolon=\\tau\(q\)\-\\frac\{1\}\{d\}\.Also define

v​\(q\):=2​‖B​\(q\)‖F2d​\(d\+2\)\+β​\(q\)2,r​\(q\):=‖B​\(q\)‖opd\+\|β​\(q\)\|\.v\(q\)\\vcentcolon=\\frac\{2\\\|B\(q\)\\\|\_\{F\}^\{2\}\}\{d\(d\+2\)\}\+\\beta\(q\)^\{2\},\\qquad r\(q\)\\vcentcolon=\\frac\{\\\|B\(q\)\\\|\_\{\\mathrm\{op\}\}\}\{d\}\+\|\\beta\(q\)\|\.Ifc0\>0c\_\{0\}\>0is sufficiently small andC0C\_\{0\}sufficiently large, then with probability at least1−δ/41\-\\delta/4, simultaneously for everyi∈\[F\]i\\in\[F\],

v​\(𝐤i\)≤C​σ2,r​\(𝐤i\)2​L2≤C​σ2​L\.v\(\{\\mathbf\{k\}\}\_\{i\}\)\\leq C\\sigma^\{2\},\\qquad r\(\{\\mathbf\{k\}\}\_\{i\}\)^\{2\}L^\{2\}\\leq C\\sigma^\{2\}L\.

###### Proof\.

Sincem≤d2/Lm\\leq d^\{2\}/L,

1d2≤1m​L≤1m\.\\frac\{1\}\{d^\{2\}\}\\leq\\frac\{1\}\{mL\}\\leq\\frac\{1\}\{m\}\.Thus

σ2=Θ​\(1d2\+1m\)≍1m\.\\sigma^\{2\}=\\Theta\\left\(\\frac\{1\}\{d^\{2\}\}\+\\frac\{1\}\{m\}\\right\)\\asymp\\frac\{1\}\{m\}\.The assumptionL3≤c0​σ2​m2L^\{3\}\\leq c\_\{0\}\\sigma^\{2\}m^\{2\}impliesm≥C∗​L3m\\geq C\_\{\*\}L^\{3\}, after choosingc0\>0c\_\{0\}\>0sufficiently small\. Hence Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5)applies\. Also

d2≥m​L≥C∗​L4,d^\{2\}\\geq mL\\geq C\_\{\*\}L^\{4\},sod≥c​L2d\\geq cL^\{2\}\.

Work on the event from Lemma[B\.9\.5](https://arxiv.org/html/2607.10034#A2.SS9.SSS5)\. Fixii, writeq=𝐤iq=\{\\mathbf\{k\}\}\_\{i\},A=A​\(q\)A=A\(q\), andN=N​\(q\)=A−q​q⊤N=N\(q\)=A\-qq^\{\\top\}\. Since

we have

tr⁡A=1\+tr⁡N,β​\(q\)=tr⁡Nd\.\\operatorname\{tr\}A=1\+\\operatorname\{tr\}N,\\qquad\\beta\(q\)=\\frac\{\\operatorname\{tr\}N\}\{d\}\.Using the trace bound,

\|β​\(q\)\|≤C​Ld​m\.\|\\beta\(q\)\|\\leq C\\sqrt\{\\frac\{L\}\{dm\}\}\.Sinced≥Ld\\geq L,

β​\(q\)2≤C​Ld​m≤C​1m≤C​σ2\.\\beta\(q\)^\{2\}\\leq C\\frac\{L\}\{dm\}\\leq C\\frac\{1\}\{m\}\\leq C\\sigma^\{2\}\.
Next,

B​\(q\)=\(q​q⊤−1d​Id\)\+\(N−tr⁡Nd​Id\)\.B\(q\)=\\left\(qq^\{\\top\}\-\\frac\{1\}\{d\}I\_\{d\}\\right\)\+\\left\(N\-\\frac\{\\operatorname\{tr\}N\}\{d\}I\_\{d\}\\right\)\.The trace\-removal map is an orthogonal projection in Frobenius norm, and

‖q​q⊤−1d​Id‖F2=1−1d≤1\.\\left\\\|qq^\{\\top\}\-\\frac\{1\}\{d\}I\_\{d\}\\right\\\|\_\{F\}^\{2\}=1\-\\frac\{1\}\{d\}\\leq 1\.Therefore

‖B​\(q\)‖F2≤C​\(1\+‖N‖F2\)≤C​\(1\+d2m\)\.\\\|B\(q\)\\\|\_\{F\}^\{2\}\\leq C\\left\(1\+\\\|N\\\|\_\{F\}^\{2\}\\right\)\\leq C\\left\(1\+\\frac\{d^\{2\}\}\{m\}\\right\)\.Hence

2​‖B​\(q\)‖F2d​\(d\+2\)≤C​\(1d2\+1m\)≤C​σ2\.\\frac\{2\\\|B\(q\)\\\|\_\{F\}^\{2\}\}\{d\(d\+2\)\}\\leq C\\left\(\\frac\{1\}\{d^\{2\}\}\+\\frac\{1\}\{m\}\\right\)\\leq C\\sigma^\{2\}\.Together withβ​\(q\)2≤C​σ2\\beta\(q\)^\{2\}\\leq C\\sigma^\{2\}, this provesv​\(q\)≤C​σ2v\(q\)\\leq C\\sigma^\{2\}\.

Forr​\(q\)r\(q\), use

‖B​\(q\)‖op≤1\+‖N‖op\+\|tr⁡N\|d\.\\\|B\(q\)\\\|\_\{\\mathrm\{op\}\}\\leq 1\+\\\|N\\\|\_\{\\mathrm\{op\}\}\+\\frac\{\|\\operatorname\{tr\}N\|\}\{d\}\.Therefore

‖B​\(q\)‖opd≤C​\(1d\+1d​m\+1m​L\+Ld3​m\)\.\\frac\{\\\|B\(q\)\\\|\_\{\\mathrm\{op\}\}\}\{d\}\\leq C\\left\(\\frac\{1\}\{d\}\+\\frac\{1\}\{\\sqrt\{dm\}\}\+\\frac\{1\}\{\\sqrt\{mL\}\}\+\\sqrt\{\\frac\{L\}\{d^\{3\}m\}\}\\right\)\.Sinced≥L2d\\geq L^\{2\}, all terms except1/d1/dare dominated byC/m​LC/\\sqrt\{mL\}\. Also

\|β​\(q\)\|≤C​Ld​m≤C​1m​L\.\|\\beta\(q\)\|\\leq C\\sqrt\{\\frac\{L\}\{dm\}\}\\leq C\\frac\{1\}\{\\sqrt\{mL\}\}\.Thus

r​\(q\)≤C​\(1d\+1m​L\)\.r\(q\)\\leq C\\left\(\\frac\{1\}\{d\}\+\\frac\{1\}\{\\sqrt\{mL\}\}\\right\)\.Squaring and multiplying byL2L^\{2\},

r​\(q\)2​L2≤C​\(L2d2\+Lm\)\.r\(q\)^\{2\}L^\{2\}\\leq C\\left\(\\frac\{L^\{2\}\}\{d^\{2\}\}\+\\frac\{L\}\{m\}\\right\)\.Sincem≤d2/Lm\\leq d^\{2\}/L,

L2d2≤Lm\.\\frac\{L^\{2\}\}\{d^\{2\}\}\\leq\\frac\{L\}\{m\}\.Therefore

r​\(q\)2​L2≤C​Lm≤C​σ2​L\.r\(q\)^\{2\}L^\{2\}\\leq C\\frac\{L\}\{m\}\\leq C\\sigma^\{2\}L\.The estimates hold simultaneously for allq=𝐤iq=\{\\mathbf\{k\}\}\_\{i\}\. ∎

### B\.10Hebbian MLPs within Transformers

#### B\.10\.1Noisy Margin

##### Clean and Noisy Margins

Given per\-item queries𝐪1,…,𝐪F\{\\mathbf\{q\}\}\_\{1\},\\dots,\{\\mathbf\{q\}\}\_\{F\}, define

γmin​\(𝐪\):=mini∈\[F\]⁡minj≠i⁡γi​j​\(𝐪i\)\.\\gamma\_\{\\min\}\(\{\\mathbf\{q\}\}\)\\ \\vcentcolon=\\ \\min\_\{i\\in\[F\]\}\\ \\min\_\{j\\neq i\}\\ \\gamma\_\{ij\}\(\{\\mathbf\{q\}\}\_\{i\}\)\.Notably, let the noisy margin, for noisy queries𝐤~\\tilde\{\{\\mathbf\{k\}\}\}, be

γmin​\(𝐤~\):=mini∈\[F\]⁡minj≠i⁡γi​j​\(𝐤~i\)\.\\gamma\_\{\\min\}\(\\tilde\{\{\\mathbf\{k\}\}\}\)\\ \\vcentcolon=\\ \\min\_\{i\\in\[F\]\}\\ \\min\_\{j\\neq i\}\\ \\gamma\_\{ij\}\(\\tilde\{\{\\mathbf\{k\}\}\}\_\{i\}\)\.and the clean margin for noiseless queries𝐤\{\\mathbf\{k\}\}be

γmin​\(𝐤\):=mini∈\[F\]⁡minj≠i⁡γi​j​\(𝐤i\)\.\\gamma\_\{\\min\}\(\{\\mathbf\{k\}\}\)\\ \\vcentcolon=\\ \\min\_\{i\\in\[F\]\}\\ \\min\_\{j\\neq i\}\\ \\gamma\_\{ij\}\(\{\\mathbf\{k\}\}\_\{i\}\)\.

##### Noisy queries\.

We query with𝐤~i\\widetilde\{\{\\mathbf\{k\}\}\}\_\{i\}\(instead of𝐤i\{\\mathbf\{k\}\}\_\{i\}\), assuming

‖𝐤~i−𝐤i‖2≤ϵ∀i∈\[F\]\.\\\|\\widetilde\{\{\\mathbf\{k\}\}\}\_\{i\}\-\{\\mathbf\{k\}\}\_\{i\}\\\|\_\{2\}\\ \\leq\\ \\epsilon\\qquad\\forall i\\in\[F\]\.\(68\)

##### Lipschitz stability in the query argument\.

Assume that for someLkL\_\{k\}and all queries of interest,

\|K​\(𝐤t,𝐪\)−K​\(𝐤t,𝐪′\)\|≤Lk​‖𝐪−𝐪′‖2∀t∈\[F\]\.\\big\|K\(\{\\mathbf\{k\}\}\_\{t\},\{\\mathbf\{q\}\}\)\-K\(\{\\mathbf\{k\}\}\_\{t\},\{\\mathbf\{q\}\}^\{\\prime\}\)\\big\|\\ \\leq\\ L\_\{k\}\\,\\\|\{\\mathbf\{q\}\}\-\{\\mathbf\{q\}\}^\{\\prime\}\\\|\_\{2\}\\qquad\\forall t\\in\[F\]\.\(69\)\(Here𝐪,𝐪′\{\\mathbf\{q\}\},\{\\mathbf\{q\}\}^\{\\prime\}range over a set containing\{𝐤i\}∪\{𝐤~i\}\\\{\{\\mathbf\{k\}\}\_\{i\}\\\}\\cup\\\{\\widetilde\{\{\\mathbf\{k\}\}\}\_\{i\}\\\}\.\)

###### Theorem B\.62\(Noised margin bound with Lipschitz stability \(isotropic values\)\)\.

Assume noisy queries \([Equation68](https://arxiv.org/html/2607.10034#A2.E68)\), lipschitz stability on the kernel \([Equation69](https://arxiv.org/html/2607.10034#A2.E69)\), and isotropic values\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and setL=log⁡\(C0​F2δ\)L=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)\. Then with probability at least1−δ1\-\\delta, simultaneously for alli∈\[F\]i\\in\[F\]and allj≠ij\\neq i,

γi​j​\(𝐤~i\)≥γi​j​\(𝐤i\)−2​\(1\+μv\)​Lk​ϵ−2​2​Lk​ϵ​F​Ld\.\\gamma\_\{ij\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\_\{i\}\)\\ \\geq\\ \\gamma\_\{ij\}\(\{\\mathbf\{k\}\}\_\{i\}\)\\ \-\\ 2\(1\+\\mu\_\{v\}\)\\,L\_\{k\}\\,\\epsilon\\ \-\\ 2\\sqrt\{2\}\\,L\_\{k\}\\,\\epsilon\\,\\sqrt\{\\frac\{FL\}\{d\}\}\.\(70\)Moreover, using the standard isotropic\-values coherence bound forμv\\mu\_\{v\}from[SectionB\.9\.1](https://arxiv.org/html/2607.10034#A2.SS9.SSS1), the same event implies the simplified form

γi​j​\(𝐤~i\)≥γi​j​\(𝐤i\)−2​Lk​ϵ−C1​Lk​ϵ​F​Ld,\\gamma\_\{ij\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\_\{i\}\)\\ \\geq\\ \\gamma\_\{ij\}\(\{\\mathbf\{k\}\}\_\{i\}\)\\ \-\\ 2\\,L\_\{k\}\\,\\epsilon\\ \-\\ C\_\{1\}\\,L\_\{k\}\\,\\epsilon\\,\\sqrt\{\\frac\{FL\}\{d\}\},\(71\)for an absolute constantC1\>0C\_\{1\}\>0\. Further, in the common regimeF​L≥dFL\\geq d\(so thatF​L/d≥1\\sqrt\{FL/d\}\\geq 1\), we have

γi​j​\(𝐤~i\)≥γi​j​\(𝐤i\)−C2​Lk​ϵ​F​Ld,\\gamma\_\{ij\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\_\{i\}\)\\ \\geq\\ \\gamma\_\{ij\}\(\{\\mathbf\{k\}\}\_\{i\}\)\\ \-\\ C\_\{\\mathrm\{2\}\}\\,L\_\{k\}\\,\\epsilon\\,\\sqrt\{\\frac\{FL\}\{d\}\},\(72\)for an absolute constantC2\>0C\_\{\\mathrm\{2\}\}\>0\.

###### Proof\.

Fixi∈\[F\]i\\in\[F\]andj≠ij\\neq i, and define

Δt,i:=K​\(𝐤t,𝐤~i\)−K​\(𝐤t,𝐤i\)\.\\Delta\_\{t,i\}\\ \\vcentcolon=\\ K\(\{\\mathbf\{k\}\}\_\{t\},\\widetilde\{\{\\mathbf\{k\}\}\}\_\{i\}\)\-K\(\{\\mathbf\{k\}\}\_\{t\},\{\\mathbf\{k\}\}\_\{i\}\)\.Subtracting the noisy and clean margin gives the exact expansion

γi​j​\(𝐤~i\)−γi​j​\(𝐤i\)=∑t=1FΔt,i​⟨𝐯i−𝐯j,𝐯t⟩\.\\gamma\_\{ij\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\_\{i\}\)\-\\gamma\_\{ij\}\(\{\\mathbf\{k\}\}\_\{i\}\)\\ =\\ \\sum\_\{t=1\}^\{F\}\\Delta\_\{t,i\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{t\}\}\\rangle\.\(73\)

##### BoundΔt,i\\Delta\_\{t,i\}by Lipschitzness\.

By[Equation69](https://arxiv.org/html/2607.10034#A2.E69)and[Equation68](https://arxiv.org/html/2607.10034#A2.E68),\|Δt,i\|≤Lk​ϵ\|\\Delta\_\{t,i\}\|\\leq L\_\{k\}\\epsilonfor alltt\.

##### Handlet=it=iandt=jt=j\.

Using\|⟨𝐯i−𝐯j,𝐯i⟩\|≤1\+μv\\big\|\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{i\}\}\\rangle\\big\|\\leq 1\+\\mu\_\{v\}and\|⟨𝐯i−𝐯j,𝐯j⟩\|≤1\+μv\\big\|\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\\big\|\\leq 1\+\\mu\_\{v\},

Δi,i​⟨𝐯i−𝐯j,𝐯i⟩\+Δj,i​⟨𝐯i−𝐯j,𝐯j⟩≥−2​\(1\+μv\)​Lk​ϵ\.\\Delta\_\{i,i\}\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{i\}\}\\rangle\+\\Delta\_\{j,i\}\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{j\}\}\\rangle\\ \\geq\\ \-\\,2\(1\+\\mu\_\{v\}\)\\,L\_\{k\}\\epsilon\.

##### Concentrate the sum overt∉\{i,j\}t\\notin\\\{i,j\\\}\.

Let

Si​j:=∑t∉\{i,j\}Δt,i​⟨𝐯i−𝐯j,𝐯t⟩\.S\_\{ij\}\\ \\vcentcolon=\\ \\sum\_\{t\\notin\\\{i,j\\\}\}\\Delta\_\{t,i\}\\,\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{t\}\}\\rangle\.Condition on\(𝐯i,𝐯j\)\(\{\\mathbf\{v\}\}\_\{i\},\{\\mathbf\{v\}\}\_\{j\}\)\. Then\(𝐯t\)t∉\{i,j\}\(\{\\mathbf\{v\}\}\_\{t\}\)\_\{t\\notin\\\{i,j\\\}\}are independent isotropic vectors, so by the*isotropic inner\-product sub\-Gaussianity lemma*\([SectionB\.9\.1](https://arxiv.org/html/2607.10034#A2.SS9.SSS1)\), each⟨𝐯i−𝐯j,𝐯t⟩\\langle\{\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\},\{\{\\mathbf\{v\}\}\_\{t\}\}\\rangleis mean\-zero and sub\-Gaussian with parameter≲‖𝐯i−𝐯j‖2/d≤2/d\\lesssim\\\|\{\\mathbf\{v\}\}\_\{i\}\-\{\\mathbf\{v\}\}\_\{j\}\\\|\_\{2\}/\\sqrt\{d\}\\leq 2/\\sqrt\{d\}\. By standard sub\-Gaussian closure under weighted sums \(see, e\.g\., the proof of[SectionB\.9\.3](https://arxiv.org/html/2607.10034#A2.SS9.SSS3)\),Si​jS\_\{ij\}is sub\-Gaussian with variance proxy at most

4d​∑t∉\{i,j\}Δt,i2≤4d⋅F​\(Lk​ϵ\)2\.\\frac\{4\}\{d\}\\sum\_\{t\\notin\\\{i,j\\\}\}\\Delta\_\{t,i\}^\{2\}\\ \\leq\\ \\frac\{4\}\{d\}\\cdot F\\,\(L\_\{k\}\\epsilon\)^\{2\}\.Hence the usual one\-sided sub\-Gaussian tail bound yields

ℙ​\(Si​j≤−u\)≤exp⁡\(−u2​d8​F​Lk2​ϵ2\)∀u≥0\.\\mathbb\{P\}\\\!\\left\(S\_\{ij\}\\leq\-u\\right\)\\ \\leq\\ \\exp\\\!\\left\(\-\\frac\{u^\{2\}\\,d\}\{8FL\_\{k\}^\{2\}\\epsilon^\{2\}\}\\right\)\\qquad\\forall u\\geq 0\.Withu:=2​2​Lk​ϵ​F​Ldu\\vcentcolon=2\\sqrt\{2\}\\,L\_\{k\}\\epsilon\\sqrt\{\\tfrac\{FL\}\{d\}\}, the right\-hand side equalse−L=δ/\(C0​F2\)e^\{\-L\}=\\delta/\(C\_\{0\}F^\{2\}\)\. A union bound over all ordered pairs\(i,j\)\(i,j\)gives that with probability at least1−δ/21\-\\delta/2, simultaneously for alliiandj≠ij\\neq i,

Si​j≥−2​2​Lk​ϵ​F​Ld\.S\_\{ij\}\\ \\geq\\ \-2\\sqrt\{2\}\\,L\_\{k\}\\epsilon\\sqrt\{\\frac\{FL\}\{d\}\}\.

##### Combine\.

Plugging the bounds for thet=i,jt=i,jterms and forSi​jS\_\{ij\}into[Equation73](https://arxiv.org/html/2607.10034#A2.E73)yields[Equation70](https://arxiv.org/html/2607.10034#A2.E70)\.

##### Absorb theμv\\mu\_\{v\}term\.

By the standard coherence bound for isotropic values \([SectionB\.9\.1](https://arxiv.org/html/2607.10034#A2.SS9.SSS1)\), with probability at least1−δ/21\-\\delta/2we haveμv≤C​Ld−1\\mu\_\{v\}\\leq C\\sqrt\{\\tfrac\{L\}\{d\-1\}\}for an absoluteCC\. SinceLd−1≤F​Ld\\sqrt\{\\tfrac\{L\}\{d\-1\}\}\\leq\\sqrt\{\\tfrac\{FL\}\{d\}\}, the term2​μv​Lk​ϵ2\\mu\_\{v\}L\_\{k\}\\epsiloncan be absorbed into theF​Ld\\sqrt\{\\tfrac\{FL\}\{d\}\}term, giving[Equation71](https://arxiv.org/html/2607.10034#A2.E71)\(after adjustingC1C\_\{1\}\)\. Finally, in the regimeF​L≥dFL\\geq d, we have2​Lk​ϵ≤2​Lk​ϵ​F​Ld2L\_\{k\}\\epsilon\\leq 2L\_\{k\}\\epsilon\\sqrt\{\\tfrac\{FL\}\{d\}\}, so the2​Lk​ϵ2L\_\{k\}\\epsilonterm can also be absorbed into theF​Ld\\sqrt\{\\tfrac\{FL\}\{d\}\}term, yielding[Equation72](https://arxiv.org/html/2607.10034#A2.E72)\. ∎

###### Corollary B\.63\(Noised min\-margin bound for bilinear\-MLP stored\-query retrieval \(isotropic values\)\)\.

Consider the isotropic\-keys/isotropic\-values bilinear\-MLP setting of[SectionB\.8\.4](https://arxiv.org/html/2607.10034#A2.SS8.SSS4), and assume the kernel satisfies[Equation69](https://arxiv.org/html/2607.10034#A2.E69)with Lipschitz constantLbilL\_\{\\mathrm\{bil\}\}\. Fixδ∈\(0,1\)\\delta\\in\(0,1\)and setL=log⁡\(C0​F2δ\)L=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)for an absoluteCC\. Assume also the common regimeF​L≥dFL\\geq d\. Then with probability at least1−δ1\-\\delta,

γmin​\(𝐤~\)≥γmin​\(𝐤\)−C2​Lbil​ϵ​F​Ld\.\\gamma\_\{\\min\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\)\\ \\geq\\ \\gamma\_\{\\min\}\(\{\\mathbf\{k\}\}\)\\ \-\\ C\_\{\\mathrm\{2\}\}\\,L\_\{\\mathrm\{bil\}\}\\,\\epsilon\\,\\sqrt\{\\frac\{FL\}\{d\}\}\.\(74\)

###### Proof\.

Apply[TheoremB\.62](https://arxiv.org/html/2607.10034#A2.Thmtheorem62)in the regimeF​L≥dFL\\geq dusing[Equation72](https://arxiv.org/html/2607.10034#A2.E72)withLk=LbilL\_\{k\}=L\_\{\\mathrm\{bil\}\}\. Since the bound holds*simultaneously*for all\(i,j\)\(i,j\), we may take the minimum overi∈\[F\]i\\in\[F\]andj≠ij\\neq i:

γmin​\(𝐤~\)=mini⁡minj≠i⁡γi​j​\(𝐤~i\)≥mini⁡minj≠i⁡γi​j​\(𝐤i\)−C2​Lbil​ϵ​F​Ld=γmin​\(𝐤\)−C2​Lbil​ϵ​F​Ld\.\\gamma\_\{\\min\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\)=\\min\_\{i\}\\min\_\{j\\neq i\}\\gamma\_\{ij\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\_\{i\}\)\\ \\geq\\ \\min\_\{i\}\\min\_\{j\\neq i\}\\gamma\_\{ij\}\(\{\\mathbf\{k\}\}\_\{i\}\)\\ \-\\ C\_\{\\mathrm\{2\}\}L\_\{\\mathrm\{bil\}\}\\epsilon\\sqrt\{\\frac\{FL\}\{d\}\}=\\gamma\_\{\\min\}\(\{\\mathbf\{k\}\}\)\\ \-\\ C\_\{\\mathrm\{2\}\}L\_\{\\mathrm\{bil\}\}\\epsilon\\sqrt\{\\frac\{FL\}\{d\}\}\.∎

###### Corollary B\.64\(Condition onϵ\\epsilonforγmin​\(𝐤~\)\>0\\gamma\_\{\\min\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\)\>0\)\.

Under the assumptions of[SectionB\.10\.1](https://arxiv.org/html/2607.10034#A2.SS10.SSS1.Px8), assume the clean margin is strictly positive:

γmin​\(𝐤\)\>0\.\\gamma\_\{\\min\}\(\{\\mathbf\{k\}\}\)\\ \>\\ 0\.\(75\)A sufficient condition forγmin​\(𝐤~\)\>0\\gamma\_\{\\min\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\)\>0is

ϵ<γmin​\(𝐤\)C2​Lbil​dF​L\.\\epsilon\\ <\\ \\frac\{\\gamma\_\{\\min\}\(\{\\mathbf\{k\}\}\)\}\{C\_\{\\mathrm\{2\}\}\\,L\_\{\\mathrm\{bil\}\}\}\\,\\sqrt\{\\frac\{d\}\{FL\}\}\.\(76\)

###### Proof\.

By[Equation74](https://arxiv.org/html/2607.10034#A2.E74),

γmin​\(𝐤~\)≥γmin​\(𝐤\)−C2​Lbil​ϵ​F​Ld\.\\gamma\_\{\\min\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\)\\ \\geq\\ \\gamma\_\{\\min\}\(\{\\mathbf\{k\}\}\)\\ \-\\ C\_\{\\mathrm\{2\}\}L\_\{\\mathrm\{bil\}\}\\epsilon\\sqrt\{\\frac\{FL\}\{d\}\}\.If[Equation76](https://arxiv.org/html/2607.10034#A2.E76)holds, the subtracted term is<γmin​\(𝐤\)<\\gamma\_\{\\min\}\(\{\\mathbf\{k\}\}\), so the right\-hand side is positive\. ∎

###### Corollary B\.65\(Noise\-robust fact\-storage capacity remains information\-theoretically optimal\)\.

Fixδ∈\(0,1\)\\delta\\in\(0,1\)and setL:=log⁡\(C0​F2δ\)L\\vcentcolon=\\log\(\\tfrac\{C\_\{0\}F^\{2\}\}\{\\delta\}\)for an absoluteC0C\_\{0\}\. Consider the iso–iso bilinear\-MLP model with dimensiondd\. There exist universal constantsc,C,c0\>0c,C,c\_\{0\}\>0such that if

d≥C​L,m≥C​L3,F≤c​min⁡\{d3L,m​dL\},d\\ \\geq\\ CL,\\qquad m\\ \\geq\\ CL^\{3\},\\qquad F\\ \\leq\\ c\\,\\min\\Big\\\{\\frac\{d^\{3\}\}\{L\},\\frac\{md\}\{L\}\\Big\\\},\(77\)thenγmin​\(𝐤\)≥c0\\gamma\_\{\\min\}\(\{\\mathbf\{k\}\}\)\\geq c\_\{0\}with probability at least1−δ1\-\\delta\(by the combined margin bound of[SectionB\.8\.4](https://arxiv.org/html/2607.10034#A2.SS8.SSS4)\)\. If additionally

ϵ≤c02​C2​Lbil​dF​L,\\epsilon\\ \\leq\\ \\frac\{c\_\{0\}\}\{2\\,C\_\{\\mathrm\{2\}\}\\,L\_\{\\mathrm\{bil\}\}\}\\,\\sqrt\{\\frac\{d\}\{FL\}\},\(78\)thenγmin​\(𝐤~\)≥c0/2\\gamma\_\{\\min\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\)\\geq c\_\{0\}/2with probability at least1−δ1\-\\delta\.

In the sketch\-limited regimeF≲m​d/LF\\lesssim md/L, the parameter countW≍m​dW\\asymp mdthus satisfiesW≍F​L≍F​log⁡\(F/δ\)W\\asymp FL\\asymp F\\log\(F/\\delta\), i\.e\. the \(noise\-robust\) fact\-storage scaling is information\-theoretically optimal up to constants \(subject to[Equation78](https://arxiv.org/html/2607.10034#A2.E78)\)\.

###### Proof\.

Under[Equation77](https://arxiv.org/html/2607.10034#A2.E77), the combined margin bound of[SectionB\.8\.4](https://arxiv.org/html/2607.10034#A2.SS8.SSS4)yieldsγmin​\(𝐤\)≥c0\\gamma\_\{\\min\}\(\{\\mathbf\{k\}\}\)\\geq c\_\{0\}with probability at least1−δ1\-\\delta\. Then apply[Equation74](https://arxiv.org/html/2607.10034#A2.E74):

γmin​\(𝐤~\)≥c0−C2​Lbil​ϵ​F​Ld\.\\gamma\_\{\\min\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\)\\ \\geq\\ c\_\{0\}\\ \-\\ C\_\{\\mathrm\{2\}\}L\_\{\\mathrm\{bil\}\}\\epsilon\\sqrt\{\\frac\{FL\}\{d\}\}\.If[Equation78](https://arxiv.org/html/2607.10034#A2.E78)holds, the subtracted term is at mostc0/2c\_\{0\}/2, givingγmin​\(𝐤~\)≥c0/2\\gamma\_\{\\min\}\(\\widetilde\{\{\\mathbf\{k\}\}\}\)\\geq c\_\{0\}/2\.

Finally, whenF≲m​d/LF\\lesssim md/L, we havem​d≍F​L≍F​log⁡\(F/δ\)md\\asymp FL\\asymp F\\log\(F/\\delta\)\. ∎

## Appendix CAdditional Related Work

##### Empirical studies: probing and editing LLM knowledge\.

Gevaet al\.\([2021](https://arxiv.org/html/2607.10034#bib.bib33);[2022](https://arxiv.org/html/2607.10034#bib.bib34)\)observed that knowledge is often stored within MLPs via key–value mappings, motivating a line of work that attempts to reverse engineer the facts encoded in MLPs\(Daiet al\.,[2022](https://arxiv.org/html/2607.10034#bib.bib39); Nandaet al\.,[2023](https://arxiv.org/html/2607.10034#bib.bib35)\)and to edit them\(Daiet al\.,[2022](https://arxiv.org/html/2607.10034#bib.bib39); Menget al\.,[2023a](https://arxiv.org/html/2607.10034#bib.bib38);[c](https://arxiv.org/html/2607.10034#bib.bib24); Guptaet al\.,[2024](https://arxiv.org/html/2607.10034#bib.bib26); Guet al\.,[2024](https://arxiv.org/html/2607.10034#bib.bib54); Fanget al\.,[2025](https://arxiv.org/html/2607.10034#bib.bib55); Sunet al\.,[2025](https://arxiv.org/html/2607.10034#bib.bib56)\)\. These studies provide strong empirical evidence that MLPs act as a locus of factual storage in large language models\.

##### Empirical studies: scaling factual knowledge\.

A related empirical line of work formalizes factual knowledge as associative recall over key–value stores and studies its scaling behavior\(Elhageet al\.,[2022](https://arxiv.org/html/2607.10034#bib.bib44); Allen\-Zhu and Li,[2024](https://arxiv.org/html/2607.10034#bib.bib41); Zucchetet al\.,[2025](https://arxiv.org/html/2607.10034#bib.bib37)\)\. These works consistently find that trained models store facts at the asymptotically optimal rate implied by[Theorem2\.4](https://arxiv.org/html/2607.10034#S2.Thmtheorem4)\(Allen\-Zhu and Li,[2024](https://arxiv.org/html/2607.10034#bib.bib41); Zucchetet al\.,[2025](https://arxiv.org/html/2607.10034#bib.bib37); Morriset al\.,[2025](https://arxiv.org/html/2607.10034#bib.bib42)\), which motivates the search for explicit constructions with comparable parameter efficiency\.