Kernel Ridge Attention: Nonlinear Memory with Selective Forgetting
Summary
Kernel Ridge Attention (KRA) is a recurrent sequence mixer that uses recursive ridge regression with covariance-guided memory updates and an Ornstein-Uhlenbeck transition for selective forgetting, achieving constant-size inference state and improved language-modeling performance over attention and recurrent baselines.
View Cached Full Text
Cached at: 11/19/26, 09:03 AM
# Kernel Ridge Attention:Nonlinear Memory with Selective Forgetting
Source: [https://arxiv.org/html/2611.23896](https://arxiv.org/html/2611.23896)
Yichen WangAffiliation:Cornell UniversityEmail:[yw2598@cornell\.edu](mailto:)Congfu Xu††thanks:Corresponding author\.Affiliation:Zhejiang UniversityEmail:[xucongfu@zju\.edu\.cn](mailto:)
###### Abstract
Kernel Ridge Attention \(KRA\) is a recurrent sequence mixer that combines learned nonlinear features with covariance\-guided memory updates\. It stores associations through recursive ridge regression, using correlations among past keys to determine how new observations modify memory\. A content\-dependent Ornstein–Uhlenbeck transition controls retention and restores uncertainty for future writes\. This construction admits an exact temporal\-kernel interpretation and maintains an inference state whose size is independent of context length\. Associative block maps and explicit adjoints support parallel training while preserving the recursive update\. In controlled language\-modeling experiments, KRA improves prediction quality over attention and recurrent baselines, strengthens natural associative recall over the evaluated recurrent models, and achieves practical distributed training throughput\. These properties provide a principled foundation for nonlinear, adaptive sequence memory\.
## 1Introduction
Efficient sequence models must repeatedly decide what to retain, what to overwrite, and how to retrieve it\. Softmax attention preserves a separate key and value for every past token\. Recurrent alternatives write these associations into a fixed\-size state\([Katharopoulos et al\., 2020](https://arxiv.org/html/2611.23896#bib.bib7);[Dao & Gu, 2024](https://arxiv.org/html/2611.23896#bib.bib3)\)\. This changes the central modeling problem: memory updates must manage interference among associations that share the same state\.
The delta rule addresses interference by writing a prediction residual\. Gated DeltaNet additionally controls retention with an input\-dependent decay\([Yang et al\., 2025](https://arxiv.org/html/2611.23896#bib.bib17)\)\. These mechanisms make recurrent memory selective in both writing and forgetting\. A further question concerns the*geometry*of a write: when past keys are correlated, which directions should a new observation modify? A scalar write strength treats those directions uniformly\. Ridge regression provides a different answer by using the inverse regularized key covariance to shape the update\.
We develop this idea intoKernel Ridge Attention \(KRA\)\. Each head constructs unit\-norm nonlinear Fourier features and maintains a regression matrix together with its posterior covariance\. The covariance transforms a key into a write direction, suppressing directions already constrained by previous observations\. A learned content clock evolves this memory through an Ornstein–Uhlenbeck \(OU\) process\. The transition both attenuates old coefficients and replenishes covariance toward a stationary prior, coupling selective forgetting with the capacity to learn new associations\.
This construction has a direct kernel interpretation\. Conditioned on the realized feature and clock sequence, the recurrent output equals prefix kernel ridge regression with a product of a learned content kernel and an exact exponential time kernel\. The content representation has finite dimension, while the temporal kernel is implemented through Markov state transitions\. Consequently, the model retains a constant\-size inference state through Markov transitions that summarize the prefix\.
The architecture applies short causal convolutions, projections, and SiLU to the query, key, and value paths\. Query and key then enter a shared learned Fourier map\. We express the covariance evolution as a composition of block maps and the coefficient evolution as an affine recurrence\. Associative prefix scans compute block boundaries, and explicit joint adjoints provide the corresponding backward computation\. This organization preserves recursive least squares while exposing parallelism across sequence blocks\.
We evaluate KRA using a shared backbone, token stream, and training budget\. At roughly 31M parameters and 100M training tokens, KRA reaches 88\.00 perplexity, improving over Transformer\+\+ by 9\.47% and Gated DeltaNet by 20\.68%\. Natural rare\-repeat top\-1 accuracy reaches 25\.41%, compared with 6\.69% for Gated DeltaNet\. The optimized eight\-GPU training step takes 31\.77 ms, corresponding to 1\.03M tokens/s\. The recurrent inference state occupies 1\.165 MiB at both 4K and 8K prefixes\.
Our contributions are threefold: \(1\) a nonlinear recurrent attention layer with covariance\-guided writes and selective OU forgetting; \(2\) an exact prefix\-kernel interpretation with associative block maps and explicit joint adjoints; and \(3\) a controlled evaluation of language modeling, associative recall, feature components, and distributed training efficiency\.
Figure 1:Kernel Ridge Attention\.Left: a pre\-norm decoder layer with KRA and a SwiGLU MLP\. Right: the expanded mixer\. Query, key, and value paths apply causal convolution, projection, and SiLU\. Query and key additionally share a learned Fourier map\. The content clock controls OU prediction\. Each head carries covariancePPand regression coefficientsΘ\\Theta, writes the current value, and reads with the query\.
## 2Kernel Ridge Attention
We describe a single head; the layer applies the construction independently across heads\. The head dimension and value dimension ared=dv=64d=d\_\{v\}=64\. The content feature dimension isD=2m=64D=2m=64, withm=32m=32learned frequencies\. Vectors are columns, andxtx\_\{t\}denotes the normalized layer input\.
### 2\.1Learned content features
Let𝒞Q\\mathcal\{C\}\_\{Q\},𝒞K\\mathcal\{C\}\_\{K\}, and𝒞V\\mathcal\{C\}\_\{V\}be independent causal depthwise convolutions of width four\. Each path extracts local input\-channel patterns before projection and SiLU\. The value isvt=SiLU\(WV\(𝒞Vx\)t\)v\_\{t\}=\\operatorname\{SiLU\}\\\!\\left\(W\_\{V\}\(\\mathcal\{C\}\_\{V\}x\)\_\{t\}\\right\), and the query and key are
qt=SiLU\(WQ\(𝒞Qx\)t\),kt=SiLU\(WK\(𝒞Kx\)t\)\.q\_\{t\}=\\operatorname\{SiLU\}\\\!\\left\(W\_\{Q\}\(\\mathcal\{C\}\_\{Q\}x\)\_\{t\}\\right\),\\qquad k\_\{t\}=\\operatorname\{SiLU\}\\\!\\left\(W\_\{K\}\(\\mathcal\{C\}\_\{K\}x\)\_\{t\}\\right\)\.\(1\)For a frequency matrixΩ∈ℝm×d\\Omega\\in\\mathbb\{R\}^\{m\\times d\}, define
ϕΩ\(z\)=1m\[cos\(Ωz\)sin\(Ωz\)\],at=ϕΩ\(kt\),bt=ϕΩ\(qt\)\.\\phi\_\{\\Omega\}\(z\)=\\frac\{1\}\{\\sqrt\{m\}\}\\begin\{bmatrix\}\\cos\(\\Omega z\)\\\\ \\sin\(\\Omega z\)\\end\{bmatrix\},\\qquad a\_\{t\}=\\phi\_\{\\Omega\}\(k\_\{t\}\),\\quad b\_\{t\}=\\phi\_\{\\Omega\}\(q\_\{t\}\)\.\(2\)The map has unit norm and induces the positive semidefinite kernel
κΩ\(z,z′\)=ϕΩ\(z\)⊤ϕΩ\(z′\)=1m∑r=1mcos\(ωr⊤\(z−z′\)\)\.\\kappa\_\{\\Omega\}\(z,z^\{\\prime\}\)=\\phi\_\{\\Omega\}\(z\)^\{\\top\}\\phi\_\{\\Omega\}\(z^\{\\prime\}\)=\\frac\{1\}\{m\}\\sum\_\{r=1\}^\{m\}\\cos\\\!\\left\(\\omega\_\{r\}^\{\\top\}\(z\-z^\{\\prime\}\)\\right\)\.\(3\)Query and key share the bank within a head; different heads and layers have independent banks\. We initialize orthogonal directions with Gaussian QR and use scrambled Sobol samples for the radialχd\\chi\_\{d\}quantiles, following the motivation of orthogonal random features\([Rahimi & Recht, 2007](https://arxiv.org/html/2611.23896#bib.bib9);[Yu et al\., 2016](https://arxiv.org/html/2611.23896#bib.bib18)\)\. The initial spectral scale isβ0=1/8\\beta\_\{0\}=1/8\. During training, both the bank and a positive bandwidth offset are learned:Ωeff=exp\(ℓβ/2\)Ωparam\\Omega\_\{\\mathrm\{eff\}\}=\\exp\(\\ell\_\{\\beta\}/2\)\\Omega\_\{\\mathrm\{param\}\}, where the initial bank already containsβ0\\sqrt\{\\beta\_\{0\}\}\. Equation \([3](https://arxiv.org/html/2611.23896#S2.E3)\) defines the learned kernel exactly\.
### 2\.2Selective time and uncertainty recovery
Each head computes a positive content\-dependent step and a retention factor,
ρt=exp\(Alog\)softplus\(wa⊤xt\+bΔt\),δt=exp\(−ρt\),ut=∑j=1tρj\.\\rho\_\{t\}=\\exp\(A\_\{\\log\}\)\\operatorname\{softplus\}\(w\_\{a\}^\{\\top\}x\_\{t\}\+b\_\{\\Delta t\}\),\\qquad\\delta\_\{t\}=\\exp\(\-\\rho\_\{t\}\),\\qquad u\_\{t\}=\\sum\_\{j=1\}^\{t\}\\rho\_\{j\}\.\(4\)The cumulative coordinateutu\_\{t\}supports the kernel interpretation, and the recurrent transition uses the currentδt\\delta\_\{t\}\. A learned positive ridgeλ=exp\(ℓλ\)\\lambda=\\exp\(\\ell\_\{\\lambda\}\)is initialized to one\. The recurrent state consists ofΘt∈ℝD×dv\\Theta\_\{t\}\\in\\mathbb\{R\}^\{D\\times d\_\{v\}\}andPt∈ℝD×DP\_\{t\}\\in\\mathbb\{R\}^\{D\\times D\}, initialized asΘ0=0\\Theta\_\{0\}=0andP0=λ−1𝐈P\_\{0\}=\\lambda^\{\-1\}\\mathbf\{I\}\. Before observing the current value, the model predicts
Θ¯t=δtΘt−1,P¯t=δt2Pt−1\+\(1−δt2\)λ−1𝐈\.\\bar\{\\Theta\}\_\{t\}=\\delta\_\{t\}\\Theta\_\{t\-1\},\\qquad\\bar\{P\}\_\{t\}=\\delta\_\{t\}^\{2\}P\_\{t\-1\}\+\(1\-\\delta\_\{t\}^\{2\}\)\\lambda^\{\-1\}\\mathbf\{I\}\.\(5\)These are the exact mean and covariance transitions of a stationary OU process sampled at the learned steps\. The second term restores uncertainty toward the prior: forgetting old evidence also reopens directions for future writes\.
### 2\.3Covariance\-guided writing
The observation update uses the residual under the predicted regression matrix:
zt\\displaystyle z\_\{t\}=P¯tat,\\displaystyle=\\bar\{P\}\_\{t\}a\_\{t\},st\\displaystyle s\_\{t\}=1\+at⊤zt,\\displaystyle=1\+a\_\{t\}^\{\\top\}z\_\{t\},Gt\\displaystyle G\_\{t\}=zt/st,\\displaystyle=z\_\{t\}/s\_\{t\},et\\displaystyle e\_\{t\}=vt−Θ¯t⊤at,\\displaystyle=v\_\{t\}\-\\bar\{\\Theta\}\_\{t\}^\{\\top\}a\_\{t\},Θt\\displaystyle\\Theta\_\{t\}=Θ¯t\+Gtet⊤,\\displaystyle=\\bar\{\\Theta\}\_\{t\}\+G\_\{t\}e\_\{t\}^\{\\top\},Pt\\displaystyle P\_\{t\}=P¯t−ztzt⊤/st\.\\displaystyle=\\bar\{P\}\_\{t\}\-z\_\{t\}z\_\{t\}^\{\\top\}/s\_\{t\}\.\(6\)We read after writing,ot=Θt⊤bto\_\{t\}=\\Theta\_\{t\}^\{\\top\}b\_\{t\}\. This inclusive update predicts the next token from the current causal prefix\. Each value channel has its own regression coefficients and shares the same covariance\.
The update is the unique solution of a local regularized regression problem:
Θt=argminΘ\{12‖vt−Θ⊤at‖22\+12tr\[\(Θ−Θ¯t\)⊤P¯t−1\(Θ−Θ¯t\)\]\}\.\\Theta\_\{t\}=\\arg\\min\_\{\\Theta\}\\left\\\{\\frac\{1\}\{2\}\\\|v\_\{t\}\-\\Theta^\{\\top\}a\_\{t\}\\\|\_\{2\}^\{2\}\+\\frac\{1\}\{2\}\\operatorname\{tr\}\\\!\\left\[\(\\Theta\-\\bar\{\\Theta\}\_\{t\}\)^\{\\top\}\\bar\{P\}\_\{t\}^\{\-1\}\(\\Theta\-\\bar\{\\Theta\}\_\{t\}\)\\right\]\\right\\\}\.\(7\)ThusP¯t−1\\bar\{P\}\_\{t\}^\{\-1\}supplies the metric of the write\. In the isotropic caseP¯t=c𝐈\\bar\{P\}\_\{t\}=c\\mathbf\{I\},Gt=cat/\(1\+c‖at‖2\)G\_\{t\}=ca\_\{t\}/\(1\+c\\\|a\_\{t\}\\\|^\{2\}\)is a scalar multiple of the key\. The full covariance rotates and rescales this direction according to past observations\. This is the structural distinction from a scalar\-gated residual write\.
###### Proposition 1\(Bounded covariance and writing\)\.
Forλ\>0\\lambda\>0and0<δt≤10<\\delta\_\{t\}\\leq 1, the recurrence satisfies0≺Pt⪯λ−1𝐈0\\prec P\_\{t\}\\preceq\\lambda^\{\-1\}\\mathbf\{I\}\. For unit\-norm features, the self\-write coefficient obeys
0<at⊤P¯tat1\+at⊤P¯tat≤11\+λ\.0<\\frac\{a\_\{t\}^\{\\top\}\\bar\{P\}\_\{t\}a\_\{t\}\}\{1\+a\_\{t\}^\{\\top\}\\bar\{P\}\_\{t\}a\_\{t\}\}\\leq\\frac\{1\}\{1\+\\lambda\}\.\(8\)
The current value’s coefficient in the query output isbt⊤Gtb\_\{t\}^\{\\top\}G\_\{t\}, which depends on the query–key alignment\. This separates the capacity to write an observation from the ability of a particular query to retrieve it\.
### 2\.4Exact prefix kernel regression
Fix a realized sequence of features and clock values\. On the prefix1:t1\{:\}t, define the key Gram matrix and query cross\-covariance
Cij=ai⊤aje−\|ui−uj\|,dt,i=bt⊤aie−\(ut−ui\)\.C\_\{ij\}=a\_\{i\}^\{\\top\}a\_\{j\}\\,e^\{\-\|u\_\{i\}\-u\_\{j\}\|\},\\qquad d\_\{t,i\}=b\_\{t\}^\{\\top\}a\_\{i\}\\,e^\{\-\(u\_\{t\}\-u\_\{i\}\)\}\.\(9\)
###### Proposition 2\(Recurrent–dual equivalence\)\.
LetV1:tV\_\{1:t\}stack the observed value vectors as rows\. The output of Equations \([5](https://arxiv.org/html/2611.23896#S2.E5)\)–\([6](https://arxiv.org/html/2611.23896#S2.E6)\) equals
ot⊤=dt⊤\(C1:t,1:t\+λ𝐈\)−1V1:t\.o\_\{t\}^\{\\top\}=d\_\{t\}^\{\\top\}\(C\_\{1:t,1:t\}\+\\lambda\\mathbf\{I\}\)^\{\-1\}V\_\{1:t\}\.\(10\)
The content Gram and OU time Gram are positive semidefinite, so their entrywise productCCis positive semidefinite\. Positive ridge makes the solve strictly positive definite\. The time factor is exact: a finite Markov state realizes its exponential covariance\. This uses the state\-space interpretation of temporal Gaussian processes\([Hartikainen & Särkkä, 2010](https://arxiv.org/html/2611.23896#bib.bib5);[Särkkä & Hartikainen, 2012](https://arxiv.org/html/2611.23896#bib.bib11)\)\.
The regression system is built from keys; queries evaluate the learned function at new feature points\. DistinctWQW\_\{Q\}andWKW\_\{K\}therefore fit naturally into this construction\. Positive definiteness belongs to the feature\-domain kernel and its key Gram, while the query–key cross\-covariance supplies the readout\. The kernel identity is conditional on the realized layer\-input path, including its content\-dependent clock\. Appendix[A](https://arxiv.org/html/2611.23896#A1)gives the proofs and the corresponding dynamic least\-squares objective\.
## 3Parallel Recursive Computation
KRA’s token update usesO\(D2\+Ddv\)O\(D^\{2\}\+Dd\_\{v\}\)arithmetic and carriesD2\+DdvD^\{2\}\+Dd\_\{v\}scalars per head\. We organize a sequence into blocks and precompute their state transformations\. The resulting composition laws expose parallelism in both the forward recurrence and its adjoints\.
### 3\.1Covariance and coefficient prefix maps
A block acts on an incoming covariance through a Riccati map
Φ\(P\)=C\+A\(P−1\+J\)−1A⊤,\\Phi\(P\)=C\+A\(P^\{\-1\}\+J\)^\{\-1\}A^\{\\top\},\(11\)whereC,J⪰0C,J\\succeq 0\. The block statistics\(A,C,J\)\(A,C,J\)follow from local Sherman–Morrison updates and incorporate the OU process noise\. These maps are closed under composition, following the associative filtering formulation of[Särkkä & García\-Fernández \(2019\)](https://arxiv.org/html/2611.23896#bib.bib10)\. For an earlier blocklland a later blockrr, letN=\(𝐈\+ClJr\)−1N=\(\\mathbf\{I\}\+C\_\{l\}J\_\{r\}\)^\{\-1\}\. The composed map has
Ar∘l\\displaystyle A\_\{r\\circ l\}=ArNAl,\\displaystyle=A\_\{r\}NA\_\{l\},Cr∘l\\displaystyle C\_\{r\\circ l\}=Cr\+ArNClAr⊤,\\displaystyle=C\_\{r\}\+A\_\{r\}NC\_\{l\}A\_\{r\}^\{\\top\},Jr∘l\\displaystyle J\_\{r\\circ l\}=Jl\+Al⊤JrNAl\.\\displaystyle=J\_\{l\}\+A\_\{l\}^\{\\top\}J\_\{r\}NA\_\{l\}\.\(12\)Associativity supplies all block\-boundary covariances through a prefix scan\. Each block then evaluates Equation \([6](https://arxiv.org/html/2611.23896#S2.E6)\) from its boundary to obtain the gainsGiG\_\{i\}\.
Given these gains, writeTi=δi\(𝐈−Giai⊤\)T\_\{i\}=\\delta\_\{i\}\(\\mathbf\{I\}\-G\_\{i\}a\_\{i\}^\{\\top\}\)\. The coefficient recurrence is affine:
Θi=TiΘi−1\+Givi⊤,Θout=FΘin\+B\.\\Theta\_\{i\}=T\_\{i\}\\Theta\_\{i\-1\}\+G\_\{i\}v\_\{i\}^\{\\top\},\\qquad\\Theta\_\{\\rm out\}=F\\Theta\_\{\\rm in\}\+B\.\(13\)Two blocks compose as\(FrFl,FrBl\+Br\)\(F\_\{r\}F\_\{l\},F\_\{r\}B\_\{l\}\+B\_\{r\}\), giving a second prefix scan\. The within\-block outputs can also be written as causal matrix products\. Withci=∏j≤iδjc\_\{i\}=\\prod\_\{j\\leq i\}\\delta\_\{j\}and the innovationseje\_\{j\}from Equation \([6](https://arxiv.org/html/2611.23896#S2.E6)\),
oi⊤=cibi⊤Θin\+∑j≤icicj\(bi⊤Gj\)ej⊤\.o\_\{i\}^\{\\top\}=c\_\{i\}b\_\{i\}^\{\\top\}\\Theta\_\{\\rm in\}\+\\sum\_\{j\\leq i\}\\frac\{c\_\{i\}\}\{c\_\{j\}\}\(b\_\{i\}^\{\\top\}G\_\{j\}\)e\_\{j\}^\{\\top\}\.\(14\)Decay ratios are evaluated from sums of log retention\. Across blocks, the global dependency is organized into logarithmically many scan stages; each block retains its short recursive update\.
### 3\.2Explicit joint adjoints
LetD=∂ℒ/∂ΘD=\\partial\\mathcal\{L\}/\\partial\\Thetaand letSSbe the symmetric covariance adjoint,S=∂ℒ/∂P\+\(∂ℒ/∂P\)⊤S=\\partial\\mathcal\{L\}/\\partial P\+\(\\partial\\mathcal\{L\}/\\partial P\)^\{\\top\}\. DefineFi=Ti⋯T1F\_\{i\}=T\_\{i\}\\cdots T\_\{1\},F0=𝐈F\_\{0\}=\\mathbf\{I\}, and the block statistic
wi=δiFi−1⊤ai,E=∑iwiei⊤/si\.w\_\{i\}=\\delta\_\{i\}F\_\{i\-1\}^\{\\top\}a\_\{i\},\\qquad E=\\sum\_\{i\}w\_\{i\}e\_\{i\}^\{\\top\}/s\_\{i\}\.\(15\)Perturbations of the incoming states obey
ΔPout=FΔPinF⊤,ΔΘout=FΔΘin\+FΔPinE\.\\Delta P\_\{\\rm out\}=F\\Delta P\_\{\\rm in\}F^\{\\top\},\\qquad\\Delta\\Theta\_\{\\rm out\}=F\\Delta\\Theta\_\{\\rm in\}\+F\\Delta P\_\{\\rm in\}E\.\(16\)Consequently, the block backward map is
Din\\displaystyle D\_\{\\rm in\}=F⊤Dout\+Dlocal,\\displaystyle=F^\{\\top\}D\_\{\\rm out\}\+D\_\{\\rm local\},Sin\\displaystyle S\_\{\\rm in\}=F⊤SoutF\+F⊤DoutE⊤\+EDout⊤F\+Slocal\.\\displaystyle=F^\{\\top\}S\_\{\\rm out\}F\+F^\{\\top\}D\_\{\\rm out\}E^\{\\top\}\+ED\_\{\\rm out\}^\{\\top\}F\+S\_\{\\rm local\}\.\(17\)The local source terms collect the block’s output gradients\. Causal block products compute these terms in parallel, and a reverse prefix scan supplies the incoming adjoints for each local backward recurrence\. Appendix[B](https://arxiv.org/html/2611.23896#A2)derives the statistics and the explicit token adjoints\.
### 3\.3Working\-precision evaluation
The covariance maps require fixed\-size state\-space inverses\. Their positive spectral structure provides bounds for polynomial inverse evaluation; a recomputed residualR=𝐈−WXR=\\mathbf\{I\}\-WXthen corrects an approximate inverse throughX←X\+XRX\\leftarrow X\+XR\. The stationary prior also simplifies the backward computation: forP0=r𝐈P\_\{0\}=r\\mathbf\{I\}withr=λ−1r=\\lambda^\{\-1\}, its initial\-state contribution is12tr\(S0\)\\tfrac\{1\}\{2\}\\operatorname\{tr\}\(S\_\{0\}\)\.
We use 64\-token blocks, BF16 network activations, and FP32 features, recurrent states, and adjoints\. Block matrix products use compensated arithmetic and residual correction\. The implementation follows the explicit recurrences above, with saved boundary states and local recomputation\. Numerical validation covers outputs, all state and input gradients, causality, and streaming continuation\.
## 4Experiments
We study language\-modeling quality, associative recall, training efficiency, and the learned content features\. All reported models use the same training\-token stream and final validation windows\.
### 4\.1Experimental setup
Models and training\.The common decoder has six layers, width 384, six heads, a head dimension of 64, and a SwiGLU feed\-forward width of 1152\. It uses pre\-RMSNorm residual blocks and a tied input/output embedding with 50,257 tokens\. KRA has 30\.92M parameters\. We compare it with Transformer\+\+, Gated DeltaNet, and two recurrence controls that replace the Gated DeltaNet core with KDA or GDN\-2 updates\([Kimi Team, 2025](https://arxiv.org/html/2611.23896#bib.bib8);[Hatamizadeh et al\., 2026](https://arxiv.org/html/2611.23896#bib.bib6)\)\. The latter controls preserve the Gated DeltaNet outer layer, use direct full\-rank gate projections, and initialize the expanded gates to reproduce its scalar\-gate function\. Their parameter counts are reported explicitly\.
Each run trains for 3,052 updates on 100,007,936 prediction tokens from a fixed SlimPajama sample\([Cerebras, 2023](https://arxiv.org/html/2611.23896#bib.bib2)\), tokenized with GPT\-2\. The global batch is 16 sequences of length 2,048, with microbatch two on each of eight RTX 5090 GPUs\. We use AdamW, a peak learning rate of10−310^\{\-3\}, 100 warmup steps, a cosine decay to10−410^\{\-4\}, weight decay 0\.1, and gradient clipping at one\. The last checkpoint is evaluated on 512 fixed windows containing 1,048,576 prediction tokens\. Curves use a shared subset of 128 windows\. All runs use one training seed and one shared data order; Appendix[C](https://arxiv.org/html/2611.23896#A3)records initialization, data construction, and measurement details\.
Recall\.Following the focus on natural associative recall in[Arora et al\. \(2024\)](https://arxiv.org/html/2611.23896#bib.bib1), we identify repeated token bigrams within the same document and window, selecting bigrams seen at most twelve times in the training stream\. This yields 31,319 rare\-repeat targets, including 2,749 with a previous occurrence at distance at least 512\. We also use 3,584 controlled prompts that balance two target values, matched and mismatched cues, and seven cue–target distances\. We report both balanced choice accuracy and association gain, a log\-probability contrast defined in Appendix[D](https://arxiv.org/html/2611.23896#A4)\.
### 4\.2Language modeling and recall
Table 1:Language modeling at a shared 100M\-token budget\.KDA and GDN\-2 are recurrence controls in the Gated DeltaNet backbone\. Lower NLL and perplexity are better\.KRA reaches a validation NLL of 4\.4774 and perplexity of 88\.00 \(Table[1](https://arxiv.org/html/2611.23896#S4.T1)\)\. Perplexity improves by 9\.47% over Transformer\+\+, 20\.68% over Gated DeltaNet, and 17\.76% over the GDN\-2 recurrence control\. Figure[2](https://arxiv.org/html/2611.23896#S4.F2)shows the validation trajectories and final checkpoint comparison\.
Figure 2:Language\-modeling comparison\.Left: NLL on the shared curve subset, starting at the first evaluation after initialization\. Right: perplexity of the final checkpoint on the complete validation stream\. Curves and final values use different, explicitly fixed evaluation sizes\.Table 2:Recall within 2K\-token windows\.Rare\-repeat top\-1 and controlled\-copy balanced accuracy are percentages\. “Far” denotes distance≥512\\geq 512; copy gain is measured in nats\.On natural rare repeats, KRA attains 25\.41% top\-1 accuracy, compared with 6\.69% for Gated DeltaNet and 8\.95% for the GDN\-2 recurrence control \(Table[2](https://arxiv.org/html/2611.23896#S4.T2)\)\. KRA reaches 8\.69% top\-1 on rare repeats at distance at least 512\. Transformer\+\+ reaches 32\.00% on the full rare\-repeat slice and the highest scores on distant targets\. The controlled copy probe in Figure[3](https://arxiv.org/html/2611.23896#S4.F3)measures association gain and balanced answer choice across cue distances\. Together, the natural and controlled probes characterize the retrieval behavior alongside overall language\-modeling quality\.
Figure 3:Natural and controlled recall\.The natural slice measures repeated content in held\-out text\. The controlled probe varies cue–target distance while balancing answer and cue identity; its dashed line marks chance accuracy\.
### 4\.3Core feature ablations
Learning the Fourier bank\.A matched bank comparison uses SiLU content features with pre\-projection Q/K convolutions and a linear value path with post\-projection convolution\. Learning the spectral directions yields 92\.36 perplexity and 25\.20% rare\-repeat top\-1, compared with 92\.96 and 23\.60% for a fixed initial bank \(Table[3](https://arxiv.org/html/2611.23896#S4.T3)\)\. Both arms share their initial function and learn the bandwidth\. Bank learning adds 73,728 trainable parameters, or 0\.24% of the model\.
The activation before the Fourier map\.The Q/K activation pair shares post\-projection convolutions in both arms\. SiLU yields 94\.95 perplexity and 20\.46% rare\-repeat top\-1; the identity yields 96\.56 and 18\.06%\. The Fourier bank, bandwidth, ridge, and content clock remain learned throughout\.
Value activation\.With pre\-projection convolutions on all three paths, SiLU values yield 88\.00 perplexity, compared with 93\.16 for linear values\. Far\-repeat top\-1 rises from 8\.00% to 8\.69%\. These matched comparisons connect feature and value construction to prediction and retrieval behavior\.
Table 3:Core content\-feature ablations\.Each group shares its front end, initial tensors, clock, ridge, and bandwidth\. The bank and Q/K activation groups use the architectures described in the text; the value\-activation group uses pre\-projection convolutions on Q/K/V\.
### 4\.4Training efficiency and inference state
Table[4](https://arxiv.org/html/2611.23896#S4.T4)measures the optimized implementations at sequence length 2,048 and microbatch two\. KRA completes full\-model forward and backward in 21\.08 ms\. Its eight\-GPU step takes 31\.77 ms, or 1\.136 times Gated DeltaNet’s 27\.97 ms, and sustains 1\.03M prediction tokens/s\. The step includes gradient communication, clipping, and the optimizer\. Both models share the same backbone and loss optimizations\. Appendix[C](https://arxiv.org/html/2611.23896#A3)specifies the timing protocol\.
Table 4:Optimized training latency\.Medians over 50 samples on RTX 5090 GPUs\. Full\-model forward–backward uses one GPU; complete steps use eight GPUs with global batch 16\.The six\-layer KRA inference state occupies 1\.165 MiB at both 4K and 8K prefixes \(Table[5](https://arxiv.org/html/2611.23896#S4.T5)\)\. The attention cache occupies 36 and 72 MiB, respectively; at 8K, KRA uses 61\.8 times less persistent storage\. The recurrent state holds the covariance, coefficient matrix, and short\-convolution histories, and its size is independent of prefix length\.
Table 5:Persistent inference state\.Batch\-one storage after 4K and 8K prefixes, including convolution histories for recurrent models and the key/value cache for attention\.
## 5Related Work
Recurrent attention and selective updates\.Linear attention converts feature\-based attention into recurrent accumulation\([Katharopoulos et al\., 2020](https://arxiv.org/html/2611.23896#bib.bib7)\)\. DeltaNet uses residual writes and supports hardware\-efficient sequence processing\([Yang et al\., 2024](https://arxiv.org/html/2611.23896#bib.bib16)\)\. Gated DeltaNet combines decay and the delta rule\([Yang et al\., 2025](https://arxiv.org/html/2611.23896#bib.bib17)\); KDA adds channel\-wise retention\([Kimi Team, 2025](https://arxiv.org/html/2611.23896#bib.bib8)\), and GDN\-2 separates erase and write gates\([Hatamizadeh et al\., 2026](https://arxiv.org/html/2611.23896#bib.bib6)\)\. KRA shares the residual\-write structure, with a full covariance determining the write direction and an OU transition restoring its uncertainty\.
Regression as recurrent computation\.Mesa layers formulate sequence mixing as an inner regression problem\([von Oswald et al\., 2023](https://arxiv.org/html/2611.23896#bib.bib14)\), and MesaNet develops an efficient solver\-based implementation for language modeling\([von Oswald et al\., 2026](https://arxiv.org/html/2611.23896#bib.bib15)\)\. Test\-Time Training treats the recurrent state as a learned model updated on the current sequence\([Sun et al\., 2024](https://arxiv.org/html/2611.23896#bib.bib12)\)\. Preconditioned DeltaNet explicitly incorporates curvature, using diagonal approximations in practical recurrent operators\([Tumma et al\., 2026](https://arxiv.org/html/2611.23896#bib.bib13)\)\. KRA combines learned nonlinear features with a full covariance and an exact OU product kernel\. These components connect adaptive memory geometry to a finite\-state realization of nonlinear kernel regression\.
Kernel features and state\-space Gaussian processes\.Random Fourier features approximate stationary kernels with finite\-dimensional maps\([Rahimi & Recht, 2007](https://arxiv.org/html/2611.23896#bib.bib9)\); orthogonal features improve the initialization geometry\([Yu et al\., 2016](https://arxiv.org/html/2611.23896#bib.bib18)\)\. We use this construction as an initialization and learn the frequencies end to end\. Temporal Gaussian processes admit exact state\-space representations for kernels with finite Markov realizations\([Hartikainen & Särkkä, 2010](https://arxiv.org/html/2611.23896#bib.bib5);[Särkkä & Hartikainen, 2012](https://arxiv.org/html/2611.23896#bib.bib11)\)\. KRA uses the OU realization to retain an exact temporal factor while learning a finite content kernel\.
Associative filtering and smoothing express temporal inference as parallel prefix operations\([Särkkä & García\-Fernández, 2019](https://arxiv.org/html/2611.23896#bib.bib10)\)\. KRA applies this structure to its covariance and coefficient maps and derives explicit joint adjoints for end\-to\-end sequence\-model training\.
## 6Conclusion
Kernel Ridge Attention combines learned nonlinear features, covariance\-guided writes, and a content\-dependent OU clock in one recurrent regression layer\. Its finite content state realizes an exact temporal kernel\. Associative block maps and explicit adjoints organize the recurrence into parallel training computations\. In the controlled language\-modeling study, KRA improves perplexity over attention and delta\-rule baselines and strengthens natural associative recall over the evaluated recurrent models\. These results establish nonlinear regression and uncertainty recovery as useful components of fixed\-state language\-model memory, with a formulation that supports further evaluation across model scales and sequence tasks\.
## Acknowledgments
We thank Jiahui Wang \(Alibaba Cloud\)\.
## References
- Arora et al\. \(2024\)Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré\.Zoology: Measuring and improving recall in efficient language models\.In*International Conference on Learning Representations*, 2024\.URL[https://arxiv\.org/abs/2312\.04927](https://arxiv.org/abs/2312.04927)\.
- Cerebras \(2023\)Cerebras\.Slimpajama\-627b\.Dataset, 2023\.URL[https://huggingface\.co/datasets/cerebras\-archive/SlimPajama\-627B](https://huggingface.co/datasets/cerebras-archive/SlimPajama-627B)\.
- Dao & Gu \(2024\)Tri Dao and Albert Gu\.Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality\.In*International Conference on Machine Learning*, 2024\.URL[https://arxiv\.org/abs/2405\.21060](https://arxiv.org/abs/2405.21060)\.
- FLA Contributors \(2024\)FLA Contributors\.Flash linear attention\.Software, 2024\.URL[https://github\.com/fla\-org/flash\-linear\-attention](https://github.com/fla-org/flash-linear-attention)\.
- Hartikainen & Särkkä \(2010\)Jouni Hartikainen and Simo Särkkä\.Kalman filtering and smoothing solutions to temporal gaussian process regression models\.In*IEEE International Workshop on Machine Learning for Signal Processing*, 2010\.URL[https://users\.aalto\.fi/~ssarkka/pub/gp\-ts\-kfrts\.pdf](https://users.aalto.fi/~ssarkka/pub/gp-ts-kfrts.pdf)\.
- Hatamizadeh et al\. \(2026\)Ali Hatamizadeh, Yejin Choi, and Jan Kautz\.Gated DeltaNet\-2: Decoupling erase and write in linear attention\.*arXiv preprint arXiv:2605\.22791*, 2026\.URL[https://arxiv\.org/abs/2605\.22791](https://arxiv.org/abs/2605.22791)\.
- Katharopoulos et al\. \(2020\)Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret\.Transformers are RNNs: Fast autoregressive transformers with linear attention\.In*International Conference on Machine Learning*, pp\. 5156–5165, 2020\.URL[https://proceedings\.mlr\.press/v119/katharopoulos20a\.html](https://proceedings.mlr.press/v119/katharopoulos20a.html)\.
- Kimi Team \(2025\)Kimi Team\.Kimi Linear: An expressive, efficient attention architecture\.*arXiv preprint arXiv:2510\.26692*, 2025\.URL[https://arxiv\.org/abs/2510\.26692](https://arxiv.org/abs/2510.26692)\.
- Rahimi & Recht \(2007\)Ali Rahimi and Benjamin Recht\.Random features for large\-scale kernel machines\.In*Advances in Neural Information Processing Systems*, 2007\.URL[https://pages\.cs\.wisc\.edu/~brecht/papers/07\.rah\.rec\.nips\.pdf](https://pages.cs.wisc.edu/~brecht/papers/07.rah.rec.nips.pdf)\.
- Särkkä & García\-Fernández \(2019\)Simo Särkkä and Ángel F\. García\-Fernández\.Temporal parallelization of bayesian smoothers\.*arXiv preprint arXiv:1905\.13002*, 2019\.URL[https://arxiv\.org/abs/1905\.13002](https://arxiv.org/abs/1905.13002)\.
- Särkkä & Hartikainen \(2012\)Simo Särkkä and Jouni Hartikainen\.Infinite\-dimensional kalman filtering approach to spatio\-temporal gaussian process regression\.In*International Conference on Artificial Intelligence and Statistics*, pp\. 993–1001, 2012\.URL[https://proceedings\.mlr\.press/v22/sarkka12\.html](https://proceedings.mlr.press/v22/sarkka12.html)\.
- Sun et al\. \(2024\)Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin\.Learning to \(learn at test time\): RNNs with expressive hidden states\.*arXiv preprint arXiv:2407\.04620*, 2024\.URL[https://arxiv\.org/abs/2407\.04620](https://arxiv.org/abs/2407.04620)\.
- Tumma et al\. \(2026\)Neehal Tumma, Noel Loo, and Daniela Rus\.Preconditioned DeltaNet: Curvature\-aware sequence modeling for linear recurrences\.*arXiv preprint arXiv:2604\.21100*, 2026\.URL[https://arxiv\.org/abs/2604\.21100](https://arxiv.org/abs/2604.21100)\.
- von Oswald et al\. \(2023\)Johannes von Oswald, Maximilian Schlegel, Alexander Meulemans, Seijin Kobayashi, Eyvind Niklasson, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Blaise Agüera y Arcas, Max Vladymyrov, Razvan Pascanu, and João Sacramento\.Uncovering mesa\-optimization algorithms in transformers\.*arXiv preprint arXiv:2309\.05858*, 2023\.URL[https://arxiv\.org/abs/2309\.05858](https://arxiv.org/abs/2309.05858)\.
- von Oswald et al\. \(2026\)Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Sarthak Mittal, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A\. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Agüera y Arcas, and João Sacramento\.MesaNet: Sequence modeling by locally optimal test\-time training\.In*International Conference on Learning Representations*, 2026\.URL[https://arxiv\.org/abs/2506\.05233](https://arxiv.org/abs/2506.05233)\.
- Yang et al\. \(2024\)Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim\.Parallelizing linear transformers with the delta rule over sequence length\.In*Advances in Neural Information Processing Systems*, 2024\.URL[https://arxiv\.org/abs/2406\.06484](https://arxiv.org/abs/2406.06484)\.
- Yang et al\. \(2025\)Songlin Yang, Jan Kautz, and Ali Hatamizadeh\.Gated delta networks: Improving Mamba2 with delta rule\.In*International Conference on Learning Representations*, 2025\.URL[https://arxiv\.org/abs/2412\.06464](https://arxiv.org/abs/2412.06464)\.
- Yu et al\. \(2016\)Felix X\. Yu, Ananda Theertha Suresh, Krzysztof Choromanski, Daniel Holtmann\-Rice, and Sanjiv Kumar\.Orthogonal random features\.In*Advances in Neural Information Processing Systems*, 2016\.URL[https://arxiv\.org/abs/1610\.09072](https://arxiv.org/abs/1610.09072)\.
## Appendix ADerivations
### A\.1The local regression update
Differentiating Equation \([7](https://arxiv.org/html/2611.23896#S2.E7)\) with respect toΘ\\Thetagives
\(P¯t−1\+atat⊤\)Θ=P¯t−1Θ¯t\+atvt⊤\.\(\\bar\{P\}\_\{t\}^\{\-1\}\+a\_\{t\}a\_\{t\}^\{\\top\}\)\\Theta=\\bar\{P\}\_\{t\}^\{\-1\}\\bar\{\\Theta\}\_\{t\}\+a\_\{t\}v\_\{t\}^\{\\top\}\.\(18\)Sherman–Morrison gives
Pt=\(P¯t−1\+atat⊤\)−1=P¯t−P¯tatat⊤P¯t1\+at⊤P¯tat\.P\_\{t\}=\(\\bar\{P\}\_\{t\}^\{\-1\}\+a\_\{t\}a\_\{t\}^\{\\top\}\)^\{\-1\}=\\bar\{P\}\_\{t\}\-\\frac\{\\bar\{P\}\_\{t\}a\_\{t\}a\_\{t\}^\{\\top\}\\bar\{P\}\_\{t\}\}\{1\+a\_\{t\}^\{\\top\}\\bar\{P\}\_\{t\}a\_\{t\}\}\.\(19\)Substitution yieldsΘt=Θ¯t\+Gt\(vt−Θ¯t⊤at\)⊤\\Theta\_\{t\}=\\bar\{\\Theta\}\_\{t\}\+G\_\{t\}\(v\_\{t\}\-\\bar\{\\Theta\}\_\{t\}^\{\\top\}a\_\{t\}\)^\{\\top\}\. The Hessian is positive definite becauseP¯t\\bar\{P\}\_\{t\}is positive definite, establishing uniqueness\.
### A\.2Covariance and self\-write bounds
Assume0≺Pt−1⪯λ−1𝐈0\\prec P\_\{t\-1\}\\preceq\\lambda^\{\-1\}\\mathbf\{I\}\. Equation \([5](https://arxiv.org/html/2611.23896#S2.E5)\) is a convex combination ofPt−1P\_\{t\-1\}andλ−1𝐈\\lambda^\{\-1\}\\mathbf\{I\}, so0≺P¯t⪯λ−1𝐈0\\prec\\bar\{P\}\_\{t\}\\preceq\\lambda^\{\-1\}\\mathbf\{I\}\. The precision update isPt−1=P¯t−1\+atat⊤P\_\{t\}^\{\-1\}=\\bar\{P\}\_\{t\}^\{\-1\}\+a\_\{t\}a\_\{t\}^\{\\top\}, giving0≺Pt⪯P¯t0\\prec P\_\{t\}\\preceq\\bar\{P\}\_\{t\}\. Induction fromP0=λ−1𝐈P\_\{0\}=\\lambda^\{\-1\}\\mathbf\{I\}proves the covariance bound\. For a unit feature,ht=at⊤P¯tat∈\(0,λ−1\]h\_\{t\}=a\_\{t\}^\{\\top\}\\bar\{P\}\_\{t\}a\_\{t\}\\in\(0,\\lambda^\{\-1\}\]\. Sinceh/\(1\+h\)h/\(1\+h\)is increasing for positivehh, the self\-write coefficient is at most1/\(1\+λ\)1/\(1\+\\lambda\)\.
### A\.3OU state and the prefix kernel
For one value channel, consider the stationary process
dθ\(u\)=−θ\(u\)du\+2/λdW\(u\),θ\(0\)∼𝒩\(0,λ−1𝐈D\)\.d\\theta\(u\)=\-\\theta\(u\)\\,du\+\\sqrt\{2/\\lambda\}\\,dW\(u\),\\qquad\\theta\(0\)\\sim\\mathcal\{N\}\(0,\\lambda^\{\-1\}\\mathbf\{I\}\_\{D\}\)\.\(20\)Its exact transition over a positive intervalρt\\rho\_\{t\}is
θt=δtθt−1\+ξt,ξt∼𝒩\(0,\(1−δt2\)λ−1𝐈D\),δt=e−ρt\.\\theta\_\{t\}=\\delta\_\{t\}\\theta\_\{t\-1\}\+\\xi\_\{t\},\\qquad\\xi\_\{t\}\\sim\\mathcal\{N\}\(0,\(1\-\\delta\_\{t\}^\{2\}\)\\lambda^\{\-1\}\\mathbf\{I\}\_\{D\}\),\\quad\\delta\_\{t\}=e^\{\-\\rho\_\{t\}\}\.\(21\)The stationary covariance isCov\(θi,θj\)=λ−1e−\|ui−uj\|𝐈D\\operatorname\{Cov\}\(\\theta\_\{i\},\\theta\_\{j\}\)=\\lambda^\{\-1\}e^\{\-\|u\_\{i\}\-u\_\{j\}\|\}\\mathbf\{I\}\_\{D\}\. Under observationsvi=ai⊤θi\+ϵiv\_\{i\}=a\_\{i\}^\{\\top\}\\theta\_\{i\}\+\\epsilon\_\{i\}with independent unit\-variance noise, the observation covariance isC/λ\+𝐈C/\\lambda\+\\mathbf\{I\}\. The querybt⊤θtb\_\{t\}^\{\\top\}\\theta\_\{t\}has cross\-covariancedt⊤/λd\_\{t\}^\{\\top\}/\\lambdawith those observations\. Gaussian conditioning therefore yields
𝔼\[bt⊤θt∣v1:t\]=dt⊤λ\(Cλ\+𝐈\)−1v1:t=dt⊤\(C\+λ𝐈\)−1v1:t\.\\mathbb\{E\}\[b\_\{t\}^\{\\top\}\\theta\_\{t\}\\mid v\_\{1:t\}\]=\\frac\{d\_\{t\}^\{\\top\}\}\{\\lambda\}\\left\(\\frac\{C\}\{\\lambda\}\+\\mathbf\{I\}\\right\)^\{\-1\}v\_\{1:t\}=d\_\{t\}^\{\\top\}\(C\+\\lambda\\mathbf\{I\}\)^\{\-1\}v\_\{1:t\}\.\(22\)The Kalman prediction and observation updates for the same model are exactly Equations \([5](https://arxiv.org/html/2611.23896#S2.E5)\)–\([6](https://arxiv.org/html/2611.23896#S2.E6)\)\. Applying the argument to each value channel proves recurrent–dual equivalence\.
The positive semidefinite product kernel lives on feature\-time points:
κ\(\(z,u\),\(z′,u′\)\)=ϕΩ\(z\)⊤ϕΩ\(z′\)e−\|u−u′\|\.\\kappa\(\(z,u\),\(z^\{\\prime\},u^\{\\prime\}\)\)=\\phi\_\{\\Omega\}\(z\)^\{\\top\}\\phi\_\{\\Omega\}\(z^\{\\prime\}\)e^\{\-\|u\-u^\{\\prime\}\|\}\.\(23\)Keys provide the regression points\(ki,ui\)\(k\_\{i\},u\_\{i\}\); the query is\(qt,ut\)\(q\_\{t\},u\_\{t\}\)\. This formulation allows separate query and key projections\. The key Gram is symmetric and positive semidefinite, and the query cross\-covariance evaluates the learned function through its distinct query map\. Causality follows by conditioning on each prefix separately\.
### A\.4A dynamic least\-squares objective
For0<δi<10<\\delta\_\{i\}<1, the same state is obtained by minimizing over a coefficient trajectory,
ℒ\(θ0:t\)=\\displaystyle\\mathcal\{L\}\(\\theta\_\{0:t\}\)=\{\}λ2‖θ0‖2\+12∑i=1t\(vi−ai⊤θi\)2\\displaystyle\\frac\{\\lambda\}\{2\}\\\|\\theta\_\{0\}\\\|^\{2\}\+\\frac\{1\}\{2\}\\sum\_\{i=1\}^\{t\}\(v\_\{i\}\-a\_\{i\}^\{\\top\}\\theta\_\{i\}\)^\{2\}\+λ2∑i=1t‖θi−δiθi−1‖21−δi2\.\\displaystyle\+\\frac\{\\lambda\}\{2\}\\sum\_\{i=1\}^\{t\}\\frac\{\\\|\\theta\_\{i\}\-\\delta\_\{i\}\\theta\_\{i\-1\}\\\|^\{2\}\}\{1\-\\delta\_\{i\}^\{2\}\}\.\(24\)The final component of the minimizer equals the filtered posterior mean at timett\. This objective shows what selective forgetting changes: it permits the regression function itself to evolve, with a step\-dependent penalty\. In the unit\-retention limit, the trajectory becomes constant and the objective reduces to ordinary ridge regression on the accumulated features\. In the zero\-retention limit, the recurrent regression state uses only the current observation; the local convolution still supplies short\-range context\.
## Appendix BBlock Maps and Their Adjoints
### B\.1Constructing the covariance map
InitializeA=𝐈A=\\mathbf\{I\},C=0C=0, andJ=0J=0for an empty block\. For one observation with featureaa, retentionδ\\delta, and prior variancer=λ−1r=\\lambda^\{\-1\}, form
C−\\displaystyle C^\{\-\}=δ2C\+\(1−δ2\)r𝐈,\\displaystyle=\\delta^\{2\}C\+\(1\-\\delta^\{2\}\)r\\mathbf\{I\},z\\displaystyle z=C−a,\\displaystyle=C^\{\-\}a,s\\displaystyle s=1\+a⊤z,\\displaystyle=1\+a^\{\\top\}z,w\\displaystyle w=δA⊤a\.\\displaystyle=\\delta A^\{\\top\}a\.\(25\)The block statistics update by
A←δA−zw⊤/s,C←C−−zz⊤/s,J←J\+ww⊤/s\.A\\leftarrow\\delta A\-zw^\{\\top\}/s,\\qquad C\\leftarrow C^\{\-\}\-zz^\{\\top\}/s,\\qquad J\\leftarrow J\+ww^\{\\top\}/s\.\(26\)Substitution into Equation \([11](https://arxiv.org/html/2611.23896#S3.E11)\) gives the covariance produced by the token recurrence for any incoming positive definitePP\. The Woodbury identity gives Equation \([12](https://arxiv.org/html/2611.23896#S3.E12)\)\. Composition is associative because each triple represents a function of the incoming covariance\. Applying prefix maps toP0P\_\{0\}therefore yields the same boundary states as sequential filtering\.
### B\.2Causal source statistics
Within a block, writeℓi=∑j≤ilogδj\\ell\_\{i\}=\\sum\_\{j\\leq i\}\\log\\delta\_\{j\}\. The coefficient transport and cross statistic at positioniisatisfy
Fi=eℓi𝐈−∑j≤ieℓi−ℓjGjwj⊤,Ei=∑j≤iwjej⊤/sj\.F\_\{i\}=e^\{\\ell\_\{i\}\}\\mathbf\{I\}\-\\sum\_\{j\\leq i\}e^\{\\ell\_\{i\}\-\\ell\_\{j\}\}G\_\{j\}w\_\{j\}^\{\\top\},\\qquad E\_\{i\}=\\sum\_\{j\\leq i\}w\_\{j\}e\_\{j\}^\{\\top\}/s\_\{j\}\.\(27\)For an output adjointo¯i=∂ℒ/∂oi\\bar\{o\}\_\{i\}=\\partial\\mathcal\{L\}/\\partial o\_\{i\}, definefi=Fi⊤bif\_\{i\}=F\_\{i\}^\{\\top\}b\_\{i\}andzi=Eio¯iz\_\{i\}=E\_\{i\}\\bar\{o\}\_\{i\}\. Then
Dlocal=∑ifio¯i⊤,Slocal=∑i\(fizi⊤\+zifi⊤\)\.D\_\{\\rm local\}=\\sum\_\{i\}f\_\{i\}\\bar\{o\}\_\{i\}^\{\\top\},\\qquad S\_\{\\rm local\}=\\sum\_\{i\}\(f\_\{i\}z\_\{i\}^\{\\top\}\+z\_\{i\}f\_\{i\}^\{\\top\}\)\.\(28\)Each sum can be expressed using block Gram products and inclusive causal masks\. These statistics combine with the terminal adjoints according to Equation \([17](https://arxiv.org/html/2611.23896#S3.E17)\)\.
### B\.3The local handwritten backward recurrence
At a token, addbio¯i⊤b\_\{i\}\\bar\{o\}\_\{i\}^\{\\top\}toDD, and setdG=Deid\_\{G\}=De\_\{i\}andde=D⊤Gid\_\{e\}=D^\{\\top\}G\_\{i\}\. Withzi=P¯iaiz\_\{i\}=\\bar\{P\}\_\{i\}a\_\{i\}as in Equation \([6](https://arxiv.org/html/2611.23896#S2.E6)\), compute
ds\\displaystyle d\_\{s\}=12zi⊤Szi−dG⊤zisi2,\\displaystyle=\\frac\{\\tfrac\{1\}\{2\}z\_\{i\}^\{\\top\}Sz\_\{i\}\-d\_\{G\}^\{\\top\}z\_\{i\}\}\{s\_\{i\}^\{2\}\},ζi\\displaystyle\\zeta\_\{i\}=dG−Szisi\+dsai,\\displaystyle=\\frac\{d\_\{G\}\-Sz\_\{i\}\}\{s\_\{i\}\}\+d\_\{s\}a\_\{i\},S−\\displaystyle S^\{\-\}=S\+ζiai⊤\+aiζi⊤,\\displaystyle=S\+\\zeta\_\{i\}a\_\{i\}^\{\\top\}\+a\_\{i\}\\zeta\_\{i\}^\{\\top\},D−\\displaystyle D^\{\-\}=D−aide⊤\.\\displaystyle=D\-a\_\{i\}d\_\{e\}^\{\\top\}\.\(29\)The state adjoints propagate asS←δi2S−S\\leftarrow\\delta\_\{i\}^\{2\}S^\{\-\}andD←δiD−D\\leftarrow\\delta\_\{i\}D^\{\-\}\. The value gradient isded\_\{e\}, and the remaining input gradients are
b¯i\\displaystyle\\bar\{b\}\_\{i\}=Θio¯i,\\displaystyle=\\Theta\_\{i\}\\bar\{o\}\_\{i\},a¯i\\displaystyle\\bar\{a\}\_\{i\}=dszi\+P¯iζi−Θ¯ide,\\displaystyle=d\_\{s\}z\_\{i\}\+\\bar\{P\}\_\{i\}\\zeta\_\{i\}\-\\bar\{\\Theta\}\_\{i\}d\_\{e\},g¯i\\displaystyle\\bar\{g\}\_\{i\}=δi2tr\[S−\(Pi−1−r𝐈\)\]\+δitr\[\(D−\)⊤Θi−1\],\\displaystyle=\\delta\_\{i\}^\{2\}\\operatorname\{tr\}\\\!\\left\[S^\{\-\}\(P\_\{i\-1\}\-r\\mathbf\{I\}\)\\right\]\+\\delta\_\{i\}\\operatorname\{tr\}\\\!\\left\[\(D^\{\-\}\)^\{\\top\}\\Theta\_\{i\-1\}\\right\],gi\\displaystyle g\_\{i\}=logδi\.\\displaystyle=\\log\\delta\_\{i\}\.\(30\)The prior receives12\(1−δi2\)tr\(S−\)\\tfrac\{1\}\{2\}\(1\-\\delta\_\{i\}^\{2\}\)\\operatorname\{tr\}\(S^\{\-\}\)per transition and12tr\(S0\)\\tfrac\{1\}\{2\}\\operatorname\{tr\}\(S\_\{0\}\)from the initial covariance\. Block products evaluate the contractions with forward states from their saved boundaries and rank\-one updates\. These expressions propagate gradients through the complete training window\.
### B\.4Inverse residual correction
ForW=𝐈\+ClJrW=\\mathbf\{I\}\+C\_\{l\}J\_\{r\}, the eigenvalues are positive becauseCl,Jr⪰0C\_\{l\},J\_\{r\}\\succeq 0\. A spectral upper bound is
λmax\(ClJr\)≤min\{‖ClJr‖∞,tr\(ClJr\),tr\(\(ClJr\)2\)\}\.\\lambda\_\{\\max\}\(C\_\{l\}J\_\{r\}\)\\leq\\min\\\!\\left\\\{\\\|C\_\{l\}J\_\{r\}\\\|\_\{\\infty\},\\operatorname\{tr\}\(C\_\{l\}J\_\{r\}\),\\sqrt\{\\operatorname\{tr\}\(\(C\_\{l\}J\_\{r\}\)^\{2\}\)\}\\right\\\}\.\(31\)This bounds the interval used by a polynomial inverse approximation\. RecomputingR=𝐈−WXR=\\mathbf\{I\}\-WXin working precision and updatingX\+=X\+XRX^\{\+\}=X\+XRgives𝐈−WX\+=R2\\mathbf\{I\}\-WX^\{\+\}=R^\{2\}in exact arithmetic\. The block recurrence uses two such residual corrections\.
Algorithm 1Sequential Kernel Ridge Attention, one head1:Features
at,bt∈ℝDa\_\{t\},b\_\{t\}\\in\\mathbb\{R\}^\{D\}, values
vt∈ℝdvv\_\{t\}\\in\\mathbb\{R\}^\{d\_\{v\}\}, retention
δt\\delta\_\{t\}, ridge
λ\>0\\lambda\>0
2:
P←λ−1𝐈DP\\leftarrow\\lambda^\{\-1\}\\mathbf\{I\}\_\{D\},
Θ←0D×dv\\Theta\\leftarrow 0\_\{D\\times d\_\{v\}\}
3:for
t=1,…,Tt=1,\\ldots,Tdo
4:
Θ¯←δtΘ\\bar\{\\Theta\}\\leftarrow\\delta\_\{t\}\\Theta
5:
P¯←δt2P\+\(1−δt2\)λ−1𝐈D\\bar\{P\}\\leftarrow\\delta\_\{t\}^\{2\}P\+\(1\-\\delta\_\{t\}^\{2\}\)\\lambda^\{\-1\}\\mathbf\{I\}\_\{D\}
6:
z←P¯atz\\leftarrow\\bar\{P\}a\_\{t\},
s←1\+at⊤zs\\leftarrow 1\+a\_\{t\}^\{\\top\}z
7:
e←vt−Θ¯⊤ate\\leftarrow v\_\{t\}\-\\bar\{\\Theta\}^\{\\top\}a\_\{t\}
8:
Θ←Θ¯\+\(z/s\)e⊤\\Theta\\leftarrow\\bar\{\\Theta\}\+\(z/s\)e^\{\\top\}
9:
P←P¯−zz⊤/sP\\leftarrow\\bar\{P\}\-zz^\{\\top\}/s
10:
ot←Θ⊤bto\_\{t\}\\leftarrow\\Theta^\{\\top\}b\_\{t\}
11:endfor
12:return
o1:T,P,Θo\_\{1:T\},P,\\Theta
## Appendix CTraining and Reproduction Details
### C\.1Data and common training protocol
We use the archived SlimPajama repository at revisionc60a39f1a375and the GPT\-2 tokenizer at revision607a30d783df; the accompanying manifest records both full revisions\. A fixed permutation of source shard names determines reading order, and rows retain their source order\. The SHA256 hash of each original UTF\-8 document, interpreted as an integer modulo 1,000, assigns residues 0–9 to validation and all other residues to training\. The resulting streams contain 92,783 training documents and 941 validation documents, with zero overlap in document hashes\.
We append an end\-of\-text separator and pack contiguous streams into 2,049\-token windows with stride 2,048, using the last 2,048 tokens as shifted labels\. Each window resets the recurrent state; separators within a window preserve model state\. The training stream contains 100,007,936 prediction tokens, and validation contains 1,048,576\. The run manifest records the exact token\-stream and index hashes\. The reported validation stream serves as the shared development split for model construction and evaluation\.
Table 6:Shared architecture and optimization settings\.SettingValueSettingValueLayers / width6 / 384Heads / head width6 / 64FFN width1,152Vocabulary50,257Sequence length2,048Global / microbatch16 / 2Updates3,052Devices8×8\\timesRTX 5090Peak / final LR10−310^\{\-3\}/10−410^\{\-4\}Warmup updates100Adam\(β1,β2\)\(\\beta\_\{1\},\\beta\_\{2\}\)\(0\.9,0\.98\)\(0\.9,0\.98\)Adamϵ\\epsilon10−810^\{\-8\}Weight decay0\.1Gradient clip1\.0RMSNormϵ\\epsilon10−610^\{\-6\}Logit softcap30Parameter init\. std\.0\.02Feature dimension64Frequency count32Initialβ\\beta/λ\\lambda1/81/8/ 1The first warmup learning rate is10−610^\{\-6\}\. Output and FFN down\-projections use residual scaling1/2L1/\\sqrt\{2L\}\. Dropout is zero\. The language\-model head applies the shared logit softcap30tanh\(ℓ/30\)30\\tanh\(\\ell/30\)\. Weight decay excludes the tied embedding, tensors of dimension below two, and explicitly exempt decay parameters\. Parameters, gradients, and optimizer state are FP32; ordinary network activations use BF16 autocast\. All models backpropagate through complete 2,048\-token windows\.
Model, data, bank, and numerical\-check seeds are respectively 20261109, 20261110, 20261111, and 20261112\. Common embeddings, projections, normalizations, and FFN parameters are loaded from a shared initial snapshot\. Model\-specific parameters use separate random streams\. The content clock starts from the same scalar decay\-gate initialization as the Gated DeltaNet baseline\. On the fixed initial diagnostic inputs, the median head half\-life,ln2/mean\(ρ\)\\ln 2/\\operatorname\{mean\}\(\\rho\), is 14\.37 tokens\. The clock remains trainable throughout\.
### C\.2Backbones and recurrence controls
Transformer\+\+ uses causal softmax multi\-head attention in the common residual backbone\. Gated DeltaNet uses width\-four Q/K/V convolutions, SiLU, normalized query/key vectors, and its native head normalization and output gate\. The KDA and GDN\-2 controls retain this outer layer and change the recurrence\. Writing states in feature\-by\-value orientation, their updates are
StKDA\\displaystyle S\_\{t\}^\{\\mathrm\{KDA\}\}=\(𝐈−ηtktkt⊤\)diag\(αt\)St−1\+ηtktvt⊤,\\displaystyle=\(\\mathbf\{I\}\-\\eta\_\{t\}k\_\{t\}k\_\{t\}^\{\\top\}\)\\operatorname\{diag\}\(\\alpha\_\{t\}\)S\_\{t\-1\}\+\\eta\_\{t\}k\_\{t\}v\_\{t\}^\{\\top\},\(32\)StGDN2\\displaystyle S\_\{t\}^\{\\mathrm\{GDN2\}\}=\(𝐈−kt\(bterase⊙kt\)⊤\)diag\(αt\)St−1\+kt\(wtwrite⊙vt\)⊤\.\\displaystyle=\(\\mathbf\{I\}\-k\_\{t\}\(b\_\{t\}^\{\\rm erase\}\\odot k\_\{t\}\)^\{\\top\}\)\\operatorname\{diag\}\(\\alpha\_\{t\}\)S\_\{t\-1\}\+k\_\{t\}\(w\_\{t\}^\{\\rm write\}\\odot v\_\{t\}\)^\{\\top\}\.\(33\)Expanded decay channels initially copy the scalar gate\. Erase and write projections in the GDN\-2 control begin equal and subsequently learn independently\. Gates use the sigmoid range\. The comparisons use direct gate projections in the common recurrent backbone\.
The recurrent baselines use Flash Linear Attention\([FLA Contributors, 2024](https://arxiv.org/html/2611.23896#bib.bib4)\), pinned to commit98193113ecb7\(full revision in the manifest\)\. Gated DeltaNet and the two controls use chunk size 64 for training and recurrent kernels for decode\. Recurrence gradients are checked against FP64 references\. KRA uses the 64\-token block\-map and explicit\-adjoint formulation of Section[3](https://arxiv.org/html/2611.23896#S3)\. The environment uses PyTorch 2\.9\.1 with CUDA 12\.8 and Triton 3\.5\.1\.
### C\.3Ablation controls
The SiLU\-versus\-identity comparison uses the same post\-projection convolution and initial tensors in both arms\. The Fourier\-bank pair shares pre\-projection Q/K convolutions, SiLU content features, and a linear value path with post\-projection convolution\. Both arms have the same initial function; the learned\-bank arm optimizes the spectral directions\. The value\-activation pair shares pre\-projection convolutions on all three paths and compares SiLU values with linear values\. Positive bandwidth and ridge remain learned across these comparisons\. Table[3](https://arxiv.org/html/2611.23896#S4.T3)separates the three groups and identifies the common front end for each contrast\.
### C\.4Timing and state accounting
The training\-latency benchmark uses the final checkpoints and the shared sequence length and microbatch\. Both models use the same normalization, feed\-forward, and loss optimizations, plus captured execution\. Gated DeltaNet also uses its native fused convolutions, gates, and output normalization\. Full\-model forward–backward latency is the median of 50 samples after five warmup iterations\. The eight\-GPU step includes distributed gradient reduction, gradient clipping, and AdamW\. After capture warmup, five steps are discarded and 50 maximum\-rank wall times are retained\. Synchronization barriers lie outside the timed interval\. A zero learning rate preserves each checkpoint during timing; every rank’s parameters are checked afterward\.
Persistent\-state measurements use batch one after 4K and 8K prefixes\. They include the recurrent matrices and convolution history, or the attention key/value cache\. The accompanying numerical data contains the measurements for each model\.
The six\-layer KRA core stores
6×6×\(642\+64×64\)×4=1,179,648bytes\.6\\times 6\\times\(64^\{2\}\+64\\times 64\)\\times 4=1\{,\}179\{,\}648\\ \\text\{bytes\}\.\(34\)Its three short\-convolution histories add 41,472 bytes, yielding 1,221,120 bytes, or 1\.164551 MiB\. The Fourier bank is a learned parameter and contributes no additional recurrent buffer\. This storage is independent of prefix length\. Training activations and saved boundary states scale with the training sequence length\.
## Appendix DRecall Definitions
Natural recall annotations identify the nearest preceding occurrence of an exact token bigram in the same document and window\. A rare bigram has training frequency at most twelve\. Accuracy predicts the second token from the current prefix; NLL uses the full vocabulary\. Distances are measured between the matched occurrences\. Annotations reset at the end\-of\-text separator, independently of the model\-state reset convention\.
For controlled recall, each of 128 cases supplies a 32\-token natural cue, a distractor cue, a background, and two answer tokensA,BA,B\. For each distanced∈\{64,128,256,512,1024,1536,1984\}d\\in\\\{64,128,256,512,1024,1536,1984\\\}, we insert an earlier cue followed by eitherAAorBB, and end the prompt with the query cue\. The earlier cue either matches the query or is a distractor\. This gives128×7×2×2=3,584128\\times 7\\times 2\\times 2=3,584prompts\. Letℓc,s\(y\)\\ell\_\{c,s\}\(y\)be the log probability of targetyywith cue conditionccand stored answerss\. Define
Mc=12\[ℓc,A\(A\)−ℓc,B\(A\)\+ℓc,B\(B\)−ℓc,A\(B\)\],Gain=Mmatched−Mdistractor\.M\_\{c\}=\\frac\{1\}\{2\}\\left\[\\ell\_\{c,A\}\(A\)\-\\ell\_\{c,B\}\(A\)\+\\ell\_\{c,B\}\(B\)\-\\ell\_\{c,A\}\(B\)\\right\],\\qquad\\mathrm\{Gain\}=M\_\{\\rm matched\}\-M\_\{\\rm distractor\}\.\(35\)Balanced accuracy compares the two target probabilities for each stored answer under matched cues, averages the two directions, and awards half credit for ties\. It is a two\-choice score; natural top\-1 uses the full vocabulary\. “Far” copy results average the four distances at least 512\. This custom diagnostic measures association strength and answer choice separately\.
## Appendix ENumerical Validation
We validate recurrent outputs, covariance and coefficient states, all input gradients, the learned prior, and complete model gradients\. Eleven core cases cover non\-diagonal positive definite initial covariance, tail blocks, long zero/weak forgetting, and captured 2K inputs from the first and last model layers\. Six additional cases cover uniform strong and zero forgetting\. Seven checks verify the shared prior’s initial\-state and transition contributions\. Twenty\-eight model checks cover FP32/BF16 execution, every learned parameter, causal suffix invariance, state capacity, and uneven streaming continuation\.
An independent FP64 check verifies the recurrent–dual identity, covariance\-map composition, causal block read, and joint adjoint formulas across 60 cases\. It covers unit retention, near\-unit retention, nonuniform content steps, strong forgetting, and tail blocks\. The optimized complete model has a maximum parameter\-gradient relative error of7\.47×10−67\.47\\times 10^\{\-6\}in the FP32 comparison\. Its million\-token NLL differs from the reference by2\.18×10−62\.18\\times 10^\{\-6\}\. The accompanying manifest records source hashes, checkpoint hashes, formula\-check results, and the data used to generate the figures and tables\.Similar Articles
RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention
RIS-Kernel introduces a model-agnostic sparse attention architecture (RIS) that reduces self-attention complexity from O(N^2) to O(N log N) for long-context LLM inference, enabling operation on commodity CPU hardware without GPU acceleration.
AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
AWARe is a fine-tuning method that mitigates catastrophic forgetting in multimodal large language models by selectively freezing important parameters based on activation patterns, preserving upstream capabilities while adapting to downstream tasks.
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
This paper explores how an exponentially decaying memory module from RAT+ can improve query-aware sparse inference methods for long-context language models, demonstrating consistent accuracy gains across various sparse budgets on needle-in-a-haystack tasks.
Dynamic Linear Attention
This paper proposes DLA, a dynamic memory modeling framework for multi-state linear attention that adaptively merges states based on token information variation and maintains a fixed-size state cache, enabling better long-context representation without the quadratic complexity of standard attention.
REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression
REAL introduces an Attention Behavior Matrix to analyze attention head behaviors in both success and failure cases, enabling more effective KV cache eviction. It achieves comparable accuracy to the strongest baseline while requiring 32x less space.