Forgetful Attention: A Trainable Support-Vector Memory with Certified Selection and Exact Unlearning

# A Trainable Support-Vector Memory with Certified Selection and Exact Unlearning
Source: [https://arxiv.org/html/2607.12204](https://arxiv.org/html/2607.12204)
###### Abstract

Attention can be viewed as an online learner over context, yet existing test\-time memories cannot certify that dropping a token leaves outputs unchanged or delete its influence outright\. We introduceSupport Vector Attention \(SV\-Attention\), a max\-margin memory whose weights are support coefficients of a one\-class SVM with fixed box parameterCC\. Its active\-set partition gives reserve tokens exactly zero weight, certifying output\-preserving eviction; a reversible incremental solver deletes a token to recover the state produced by retraining without it under the sameCC\. In fp64 experiments, decrement and refit recover identical partitions whenever the optimum is unique, and their decision functions match to a median deviation of about10−910^\{\-9\}\(10−1310^\{\-13\}on learned keys\); the10−210^\{\-2\}worst case is confined to ill\-conditioned duplicates and remains below coefficient decay in every regime\. The exact path reuses the maintained KKT inverse in a custom backward\. Training uses a separate stabilized batched approximation and does not carry the exact\-deletion certificate; it reaches9,1259\{,\}125tokens/s on a3\.223\.22M\-parameter model, while remaining35\.835\.8times slower than an MPS softmax reference\. At matched budgets, certified selection reaches0\.860\.86vs\.0\.320\.32rare\-item recall and retains0\.800\.80vs\.0\.050\.05deterioration hours on real MIMIC\-IV streams\. We also demonstrate surgical forgetting, exact editing, patient\-record deletion, and a forgettable retrieval memory over real sentence embeddings\. On enwik8, the hybrid obtains2\.1782\.178BPC vs\.2\.3832\.383for a matched\-state sliding\-window Transformer across seven seeds \(8\.6%8\.6\\%paired improvement,p=0\.001p=0\.001\); a three\-seed TinyStories result is directionally positive but not significant \(p=0\.057p=0\.057\)\.

## 1Introduction

Attention is increasingly understood not as a similarity heuristic but as a learning algorithm that runs at inference time\. In this test\-time\-learning \(TTL\) view, a layer maintains a memory fit online to the \(key, value\) pairs of the context and answers queries by reading that memory out\(Katharopouloset al\.,[2020](https://arxiv.org/html/2607.12204#bib.bib1); Yanget al\.,[2024](https://arxiv.org/html/2607.12204#bib.bib2); von Oswaldet al\.,[2023](https://arxiv.org/html/2607.12204#bib.bib5)\)\. The members of this family are distinguished by the inner problem they solve: DeltaNet takes a single online gradient \(delta\-rule\) step per token\(Yanget al\.,[2024](https://arxiv.org/html/2607.12204#bib.bib2)\), the mesa layer solves an online ridge regression in closed form\(von Oswaldet al\.,[2023](https://arxiv.org/html/2607.12204#bib.bib5);[2025](https://arxiv.org/html/2607.12204#bib.bib6)\), Titans trains a deep memory by gradient descent with a surprise gate and decay\-based forgetting\(Behrouzet al\.,[2025](https://arxiv.org/html/2607.12204#bib.bib7)\), and selective state\-space models gate what enters a fixed recurrent state\(Gu and Dao,[2024](https://arxiv.org/html/2607.12204#bib.bib3)\)\. The family has a conspicuous gap\. Every member fits an*unconstrained*, dense least\-squares\-style or recurrent memory in which every token retains some weight, and the only lever for removing a token is an exponential decay that erodes it slowly and never fully\. Two questions therefore have no principled answer:*which context tokens can be dropped with no effect on the output*, and*how a single token’s contribution can be deleted outright*\.

Both questions are practical\. Long\-context inference is memory\-bound, and production systems prune the key–value cache with scored heuristics that come with no guarantee about what was discarded\(Zhanget al\.,[2023](https://arxiv.org/html/2607.12204#bib.bib8); Liet al\.,[2024b](https://arxiv.org/html/2607.12204#bib.bib9)\)\. Separately, the EU GDPR’s right to erasure and clinical consent withdrawal can require a specific record to be deleted from a deployed model on demand; the usual remedies \(clearing a buffer, gradient ascent, scaling a coefficient toward zero\) leave a residual that is hard to bound and therefore hard to defend to an auditor\(Bourtouleet al\.,[2021](https://arxiv.org/html/2607.12204#bib.bib11); Huanget al\.,[2024](https://arxiv.org/html/2607.12204#bib.bib13)\)\. Fast attention has, by default, accepted both approximations\. We show that for a well\-chosen member of the TTL family neither is necessary\.

The lever is a classical algorithm\.Cauwenberghs and Poggio \([2000](https://arxiv.org/html/2607.12204#bib.bib17)\)showed that the support vector machine solution can be constructed*exactly*one point at a time: an adiabatic increment raises a point’s coefficient while preserving the Karush–Kuhn–Tucker \(KKT\) conditions on all previous points, and the procedure is reversible, so a point can be decrementally removed, leaving the solution identical to one never trained on it\. In the maintained fp64 implementation, the algorithm keeps the inverse of the bordered KKT matrix over the margin support vectors\. We observe that this single object pays for itself three times: it drives the forward solve, it certifies which tokens are inert, and by the implicit function theorem it is precisely the operator needed by a custom backward pass\. The stabilized batched implementation used for training is a separate numerical path, described in Section[3\.4](https://arxiv.org/html/2607.12204#S3.SS4)\.

We make this concrete inSupport Vector Attention \(SV\-Attention\), the constrained, max\-margin member of the TTL family: a layer whose attention weights over context tokens are the support coefficients of a one\-class SVM fit to the keys\. Tokens the margin classifies as*reserve*receive exactly zero weight; tokens on or beyond the margin carry the readout\. From this single construction the layer inherits two properties that gradient\-based memories structurally lack, certified selection and exact forgetting, along with a third property that makes it trainable\. Our contributions are as follows:

1. 1\.Exact forgetting in the maintained solver\.Exact \(“perfect”\) unlearning is equivalence to retraining from scratch without the data\(Cao and Yang,[2015](https://arxiv.org/html/2607.12204#bib.bib12); Bourtouleet al\.,[2021](https://arxiv.org/html/2607.12204#bib.bib11)\)\. We test the fp64 decremental path against a retrain\-without run of the same algorithm with the same primitive box hyperparameterCCheld fixed\. It recovers the*identical*support\-vector partition wherever the optimum is unique \(9898–100%100\\%of trials on Gaussian, real MIMIC\-IV\(Johnsonet al\.,[2023](https://arxiv.org/html/2607.12204#bib.bib25)\), and trained\-model keys\), and its decision function, the metric invariant to the non\-unique optimum on duplicates, matches that refit to a median of∼10−9\{\\sim\}10^\{\-9\}\(10−1310^\{\-13\}on learned keys\)\. The worst case of1\.1×10−21\.1\\times 10^\{\-2\}is confined to ill\-conditioned near\-duplicate streams and remains below coefficient decay’s deviation in every regime \(Section[4](https://arxiv.org/html/2607.12204#S4)\)\.
2. 2\.Certified selection\.The active\-set partition is a certificate rather than a score: complementary slackness makes reserve eviction provably output\-preserving\. At a matched token budget under skewed redundancy, certified selection beats H2O\-style heavy\-hitter eviction on rare\-item recall by a wide margin \(0\.860\.86vs\.0\.320\.32, even with H2O given oracle access to the evaluation queries\), and an adaptive support set retains0\.950\.95recall at4040distinct items where a matched\-state DeltaNet recurrence collapses to0\.060\.06\(Section[5](https://arxiv.org/html/2607.12204#S5)\)\.
3. 3\.A trainable max\-margin memory with two numerical paths\.For the exact maintained solver, a custom vector–Jacobian product reuses the forward\-maintained bordered\-KKT inverse with no second solve and no unrolling \(gradients match finite differences under double\-precisiongradcheck\)\. The separate batched training path finds approximate partitions for all \(sequence, head, chunk\) problems at once, then uses a10−310^\{\-3\}\-ridgedtorch\.linalg\.solveand its autograd backward\. Its coefficients are a stabilized differentiable approximation, not exact coefficients and not an exact\-deletion certificate\. In a full\-model benchmark this path reaches9,1259\{,\}125tokens/s,6\.1×6\.1\\timesslower than an unfused PyTorch CPU softmax reference and35\.8×35\.8\\timesslower than its MPS reference \(Section[7](https://arxiv.org/html/2607.12204#S7)\)\.
4. 4\.Capabilities demonstrated on real data\.Using the exact fp64 solver on learned keys and on records we did not construct, we show surgical fact forgetting that leaves other facts intact, exact knowledge editing, a right\-to\-be\-forgotten patient deletion on real MIMIC\-IV streams \(zero leakage;0\.800\.80vs\.0\.050\.05retention of critical hours against heavy\-hitter eviction\), and a certified, forgettable retrieval memory over real sentence embeddings \(Section[6](https://arxiv.org/html/2607.12204#S6)\)\.

#### Scope of the claims\.

The contribution is a pair of*properties*, certified selection and exact fixed\-CCforgetting, for the long\-range gate evaluated with the maintained fp64 solver, together with a custom inverse\-reusing backward for that solver\. No standard sequence layer offers either property, and both hold on real data independent of language\-model scale\. The batched language\-model path is a stabilized training approximation and does not itself certify exact deletion; the hybrid’s dense local path is deleted by masking and renormalization, not by an SV certificate \(Section[3\.5](https://arxiv.org/html/2607.12204#S3.SS5)\)\. The layer is not positioned as a throughput\- or accuracy\-competitive general\-purpose replacement for attention: on non\-redundant context the gate is near\-uniform and the layer behaves as plain kernel attention, and on raw predictive accuracy against standard sequence baselines it is competitive rather than dominant\. A language\-model probe \(Section[8](https://arxiv.org/html/2607.12204#S8)\) confirms the layer is a usable sequence layer, with a matched\-state advantage specific to small scale\. On redundant, anomaly\-driven streams with deletion obligations, the long\-range certificates are worth their cost\.

## 2Related Work

Test\-time learning and fast weights\.Linear attention reads the context as an associative memory updated by outer products\(Katharopouloset al\.,[2020](https://arxiv.org/html/2607.12204#bib.bib1); Schlaget al\.,[2021](https://arxiv.org/html/2607.12204#bib.bib18)\); DeltaNet replaces the Hebbian update with an online delta rule\(Yanget al\.,[2024](https://arxiv.org/html/2607.12204#bib.bib2)\); the mesa layer solves the regularized least\-squares problem exactly each step\(von Oswaldet al\.,[2023](https://arxiv.org/html/2607.12204#bib.bib5);[2025](https://arxiv.org/html/2607.12204#bib.bib6)\); Titans trains a deep memory online with momentum and a forgetting gate\(Behrouzet al\.,[2025](https://arxiv.org/html/2607.12204#bib.bib7)\); and selective state\-space models gate what enters a fixed recurrent state\(Gu and Dao,[2024](https://arxiv.org/html/2607.12204#bib.bib3)\)\. All maintain an*unconstrained*least\-squares\-style or recurrent state, which is why they lack the active\-set structure, and hence the certificates, of our constrained, max\-margin gate\. Our experiments compare against DeltaNet and linear attention, the constant\-state members with standard reference implementations; the mesa layer and Titans differ in their inner objective but share the dense, decay\-only state the contrast targets, so the structural argument, though not the empirical comparison, carries over to them unchanged\.

Attention and support vector machines\.Tarzanaghet al\.\([2023](https://arxiv.org/html/2607.12204#bib.bib19)\)showed that gradient descent on a one\-layer attention model converges in direction to a hard\-margin SVM that separates optimal from non\-optimal tokens, a*descriptive*result about training dynamics\. Ours is*constructive*: we place a one\-class SVM in the layer, solve it with an exact fp64 maintained path for the certificates, and use a separate stabilized differentiable approximation for batched training\.

Machine unlearning and knowledge editing\.Exact unlearning by retraining shards \(SISA,Bourtouleet al\.,[2021](https://arxiv.org/html/2607.12204#bib.bib11)\) is costly, and weight\-space approximate unlearning \(gradient ascent, scrubbing\) leaves residual influence that is hard to bound; decremental SVM unlearning was revisited for deep classifiers by ECO\(Huanget al\.,[2024](https://arxiv.org/html/2607.12204#bib.bib13)\), which removes*training*data through a final\-layer SVM\. The current benchmarks for language\-model unlearning \(TOFU,Mainiet al\.,[2024](https://arxiv.org/html/2607.12204#bib.bib14); MUSE,Shiet al\.,[2025](https://arxiv.org/html/2607.12204#bib.bib15); WMDP,Liet al\.,[2024a](https://arxiv.org/html/2607.12204#bib.bib16)\) evaluate*parametric*forgetting \(editing weights so a fact is no longer recalled\), and report a stubborn forget\-versus\-utility trade: MUSE finds approximate unlearning degrades retained\-task utility by2424–100%100\\%\. Our scope is different: SV\-Attention forgets from the*context memory*rather than from weights, and its decrement is zero\-collateral by construction, because the retained tokens are re\-solved under the same fixedCCto the optimum they would have reached without the deleted one, so there is no utility\-for\-forgetting trade to tune\. We hold the fp64 result to the exact\-unlearning definition ofCao and Yang \([2015](https://arxiv.org/html/2607.12204#bib.bib12)\)\(equivalence to retraining with the same algorithm and hyperparameters\), whose in\-layer analogue is coefficient decay, our baseline\. Knowledge\-editing methods such as ROME\(Menget al\.,[2022](https://arxiv.org/html/2607.12204#bib.bib27)\)and MEMIT\(Menget al\.,[2023](https://arxiv.org/html/2607.12204#bib.bib28)\)edit facts by modifying weights; our edit is an exact remove\-and\-add in the context memory whose old binding leaves a residual at machine precision\.

KV\-cache eviction and long context\.H2O\(Zhanget al\.,[2023](https://arxiv.org/html/2607.12204#bib.bib8)\)and SnapKV\(Liet al\.,[2024b](https://arxiv.org/html/2607.12204#bib.bib9)\)rank cache entries by accumulated attention mass, a frequency signal with no guarantee about what eviction discards\. A more recent line restores losslessness by*keeping*the full cache: VeriCache\(Yaoet al\.,[2026](https://arxiv.org/html/2607.12204#bib.bib10)\)drafts from a compressed cache and verifies each token against the retained full cache, guaranteeing identical outputs\. We differ on both axes\. Our certified reserve is*structurally*inert by complementary slackness rather than by a verification pass, so its eviction cannot change the output, and we retain no full cache to fall back on\. The same machinery supports exact*deletion*, which a verify\-against\-full\-cache scheme does not address\. The hybrid layer’s global\-plus\-local pattern follows the long\-context literature\(Beltagyet al\.,[2020](https://arxiv.org/html/2607.12204#bib.bib29)\); its exactly solved global memory can be certified, while its local current\-chunk path is deleted directly by masking and carries no SV certificate\.

Differentiable optimization\.A layer that contains an optimization problem can be differentiated by the implicit function theorem on its KKT system, as in OptNet\(Amos and Kolter,[2017](https://arxiv.org/html/2607.12204#bib.bib20)\)and differentiable convex layers\(Agrawalet al\.,[2019](https://arxiv.org/html/2607.12204#bib.bib21)\); the usual price is a matrix factorization in the backward pass\. The exact maintained path avoids that price, because its forward state carries the relevant inverse\(Cauwenberghs and Poggio,[2000](https://arxiv.org/html/2607.12204#bib.bib17)\), reducing its custom backward to a multiply\. The batched training path \(Section[3\.4](https://arxiv.org/html/2607.12204#S3.SS4)\) instead callstorch\.linalg\.solveon a ridged system and relies on autograd; it does not share the no\-second\-solve claim\.

## 3Method

### 3\.1The support\-vector gate

Given context keysxi∈ℝdx\_\{i\}\\in\\mathbb\{R\}^\{d\}and valuesvi∈ℝdvv\_\{i\}\\in\\mathbb\{R\}^\{d\_\{v\}\},i=1,…,ni=1,\\dots,n, and a queryqq, SV\-Attention computes a kernel\-weighted average gated by support coefficientsα\\alpha,

o​\(q\)=∑iαi​κ​\(q,xi\)​vi∑iαi​κ​\(q,xi\),κ​\(a,b\)=exp⁡\(−∥a−b∥2/σ2\),o\(q\)=\\frac\{\\sum\_\{i\}\\alpha\_\{i\}\\,\\kappa\(q,x\_\{i\}\)\\,v\_\{i\}\}\{\\sum\_\{i\}\\alpha\_\{i\}\\,\\kappa\(q,x\_\{i\}\)\},\\qquad\\kappa\(a,b\)=\\exp\(\-\\lVert a\-b\\rVert^\{2\}/\\sigma^\{2\}\),\(1\)whereσ\\sigmais the kernel bandwidth andα=\(α1,…,αn\)\\alpha=\(\\alpha\_\{1\},\\dots,\\alpha\_\{n\}\)solves the one\-class support vector data description\(Schölkopfet al\.,[2001](https://arxiv.org/html/2607.12204#bib.bib22); Tax and Duin,[2004](https://arxiv.org/html/2607.12204#bib.bib23)\)over the keys,

minα⁡α⊤​K​α−diag​\(K\)⊤​αs\.t\.∑iαi=1,0≤αi≤C,\\min\_\{\\alpha\}\\;\\alpha^\{\\top\}K\\alpha\-\\mathrm\{diag\}\(K\)^\{\\top\}\\alpha\\quad\\text\{s\.t\.\}\\quad\\textstyle\\sum\_\{i\}\\alpha\_\{i\}=1,\\;0\\leq\\alpha\_\{i\}\\leq C,\(2\)withKi​j=κ​\(xi,xj\)K\_\{ij\}=\\kappa\(x\_\{i\},x\_\{j\}\)the key Gram matrix\. The boxCCis the primitive solver hyperparameter and remains fixed as tokens are admitted or deleted\. When we report the familiar budget parameterν∈\(0,1\]\\nu\\in\(0,1\], it is used only once to calibrateC:=1/\(ν​n0\)C:=1/\(\\nu n\_\{0\}\)at a declared admission/reference sizen0n\_\{0\}; we do*not*replacen0n\_\{0\}by the changing context size\. Feasibility requiresn​C≥1nC\\geq 1; the causal implementation uses an ungated warm\-up until at least⌈1/C⌉\+2\\lceil 1/C\\rceil\+2keys are available and issues no SV certificate during that warm\-up\. The attention weight of a token is its query similarity*gated by its support coefficient in a max\-margin description of the key distribution*\. The KKT conditions partition the context into themargin setSS\(coefficients strictly inside the box,0<αi<C0<\\alpha\_\{i\}<C\), theerror setEE\(coefficients atCC; boundary and outlier keys\), and thereserve setRR\(coefficients exactly zero; keys interior to the description\)\. A reserve token’s coefficient is not merely small; it is identically zero, and by complementary slackness removing it leaves the optimum, and hence the readout, unchanged\. This is the certificate: eviction ofRRprovably cannot alter the output \(Figure[1](https://arxiv.org/html/2607.12204#S3.F1)\)\.

![Refer to caption](https://arxiv.org/html/2607.12204v1/figs/svattn_method.png)Figure 1:*The SV\-Attention layer, on a real ICU stay\.*Context keys are fit by a one\-class SVM whose active\-set partition gates the readout\.*Left:*the120120hourly vital tokens of one MIMIC\-IV ICU stay, projected onto their top two principal components \(the gate is fit in the full66\-D vital space; the curve is the real boundary sliced through the PC plane\)\. The solver certifies most hours asreserve\(α=0\\alpha=0, gray\), a few asmarginsupport vectors \(SS, blue\), and atypical hours as theerror set\(EE, red\); the rare deterioration hours \(ringed\) fall on the support set \(1616of2222\)\.*Center:*the coefficientsαi\\alpha\_\{i\}are sparse \(3838of120120nonzero\) and capped at fixedCC, calibrated once at the figure’s reference sizen0=120n\_\{0\}=120\.*Right:*the readout weightsκ​\(q,xi\)\\kappa\(q,x\_\{i\}\)byαi\\alpha\_\{i\}; right of the dashed line \(the support/reserve cutoff\), a reserve token’s weight is identically zero, so it contributes nothing and is safe to evict\. In the exact maintained path, the single bordered\-KKT inverseℛ=M−1\\mathcal\{R\}=M^\{\-1\}pays three times: certified selection, exact forgetting, and the custom backward pass\.
### 3\.2Exact fp64 increments and decrements

For certificate experiments we computeα\\alphain fp64 with the maintained incremental algorithm ofCauwenberghs and Poggio \([2000](https://arxiv.org/html/2607.12204#bib.bib17)\)\(the same exact add/remove machinery was developed for support vector regression byMaet al\.,[2003](https://arxiv.org/html/2607.12204#bib.bib24)\), which we ported and verified \(Section[7](https://arxiv.org/html/2607.12204#S7)\)\. The solver receivesCCdirectly and keeps it unchanged for its lifetime\. It adds one key at a time, raising its coefficient adiabatically while preserving the KKT conditions of all previous keys, with set migrations handled by rank\-one updates ofℛ=M−1\\mathcal\{R\}=M^\{\-1\}, the inverse of the bordered KKT matrix

M=\[0𝟏⊤𝟏2​KS​S\],M=\\begin\{bmatrix\}0&\\mathbf\{1\}^\{\\top\}\\\\ \\mathbf\{1\}&2K\_\{SS\}\\end\{bmatrix\},\(3\)whereKS​SK\_\{SS\}is the Gram matrix restricted to the margin set\. WithCCheld fixed, the procedure is reversible: driving a coefficient back to zero along the same path removes the key*exactly*, returning the solver to the state it would occupy had the key never entered under the same algorithm and hyperparameters\. In our fp64 verification suite a decrement composed with the increment that preceded it restored the solution to within∼10−16\{\\sim\}10^\{\-16\}on well\-conditioned data\. All exact\-deletion claims use this path, not the batched training approximation below\.

### 3\.3The inverse\-reusing custom backward

For the exact maintained path, within a fixed partition the free variablesu=\[ρ;αS\]u=\[\\rho;\\,\\alpha\_\{S\}\]satisfyM​u=cMu=c, whereρ\\rhois the multiplier of the equality constraint andcccollects the constant terms contributed byEE\. By the implicit function theorem, a custom vector–Jacobian product for an upstream cotangentu¯\\bar\{u\}reusesℛ\\mathcal\{R\}: withw=ℛ⊤​u¯w=\\mathcal\{R\}^\{\\top\}\\bar\{u\}, the gradient∂L/∂K\\partial L/\\partial Kis a sum of rank\-one outer products ofwwandαS\\alpha\_\{S\}\(full expressions in Appendix[A](https://arxiv.org/html/2607.12204#A1)\)\. Because the forward pass already maintainsℛ\\mathcal\{R\}, this custom backward performs no second system solve\. Set migrations are measure\-zero, piecewise\-smooth boundaries, which we treat as ReLU\-style kinks and do not differentiate through\. The expressions match finite differences under double\-precisiongradcheck\(Section[7](https://arxiv.org/html/2607.12204#S7)\)\. This no\-second\-solve statement does not apply to the batched path\.

### 3\.4A stabilized batched training path

Solved per \(sequence, head\) and, in the causal layer, per chunk boundary, a sequential active\-set walk in Python does not use the GPU and is the layer’s throughput bottleneck\. We therefore use a separate, stabilized approximation for training \(Figure[2](https://arxiv.org/html/2607.12204#S3.F2)a\)\. First, a single*batched*accelerated projected\-gradient solver \(FISTA,Beck and Teboulle,[2009](https://arxiv.org/html/2607.12204#bib.bib33)\) estimates the partition for all problems at once: \([2](https://arxiv.org/html/2607.12204#S3.E2)\) is a box\- and sum\-constrained convex quadratic program, so each iteration is a dense batched matrix product plus a vectorized capped\-simplex projection, fused into one compiled graph and masked so problems of different prefix length pack into one batch \(Algorithm[1](https://arxiv.org/html/2607.12204#alg1), Appendix[B](https://arxiv.org/html/2607.12204#A2)\)\. We run this partition finder in single precision on Apple\-silicon GPUs through MLX\(Hannunet al\.,[2023](https://arxiv.org/html/2607.12204#bib.bib31)\)\. Second, the PyTorch path constructs the bordered KKT system for those masks, adds a ridge of10−310^\{\-3\}to the margin block, and callstorch\.linalg\.solve; standard autograd differentiates that solve\. We denote the resulting coefficientsα~\\widetilde\{\\alpha\}\. The ridge stabilizes redundant\-key batches, but it also meansα~\\widetilde\{\\alpha\}is an approximation to the coefficients of \([2](https://arxiv.org/html/2607.12204#S3.E2)\)\. This path neither uses the inverse\-reusing custom VJP nor supplies an exact\-deletion certificate\. Its throughput consequence is in Section[7](https://arxiv.org/html/2607.12204#S7.SS0.SSS0.Px2)\.

### 3\.5A causal layer and a local path

We lift the block gate to a causal, multi\-head sequence layer with a chunk\-frozen update \(Titans\-MAC style\): the sequence is processed in chunks, and queries in a chunk read against the gate state fit on the causal prefix\. Under exact inference, each boundary admits the new keys to the fixed\-CCmaintained solver and may evict certified\-inert reserve tokens; batched training instead recomputes the approximate partition\. Within a fixed estimated partition the per\-chunk readout is differentiable through the ridged batched solve and autograd, so the layer trains end\-to\-end; the gate state is detached across chunk boundaries \(a chunk\-frozen approximation controlled by the chunk length\), which costs0\.04%0\.04\\%bits\-per\-character against a token\-level update ablation at small scale\.

Trained as a pure chunk\-frozen gate the layer plateaus as a language\-model layer\. We localize the cause by ablation \(Section[8](https://arxiv.org/html/2607.12204#S8)\): it is neither the kernel nor the gate but the chunk\-frozen*contract*itself, which makes a query read the gate fit on the prefix*before*its chunk and therefore never see the most recent tokens, the local context that dominates next\-token prediction\. We add a local path \(Figure[2](https://arxiv.org/html/2607.12204#S3.F2)b\): a query at positionttattends, with a learnedq⋅kq\\\!\\cdot\\\!ksoftmax, over all of\[0,t\]\[0,t\], where the chunk\-frozen long\-range prefix is weighted by the gate coefficient and the recent tokens inside its own chunk are read directly\. This is a gated\-global, dense\-local pattern in the spirit of long\-context sparse attention\(Beltagyet al\.,[2020](https://arxiv.org/html/2607.12204#bib.bib29)\); it preserves causality and differentiability \(verified by gradient check\)\.

#### Hybrid deletion semantics\.

The support\-vector certificate applies to the*long\-range*prefix when it is solved by the exact fixed\-CCmaintained solver: deleting an admitted token follows the decremental reverse path and re\-solves the remaining gate\. A token in the*current chunk*has not yet been admitted to that gate\. Deleting it masks its raw key/value slot and its attention weight, then renormalizes the remaining weights; the mask also excludes it at future admissions\. This is direct local\-cache deletion, not a support\-vector certificate\. During training, the long\-range gate uses the stabilized batched approximation while the local path remains dense; exact deletion is certified only when the long\-range gate is recomputed by the fp64 maintained path\.

![Refer to caption](https://arxiv.org/html/2607.12204v1/figs/arch_layer.png)Figure 2:*The trainable layer and its claim boundary\.**\(a\)*The batched training path stacks allB⋅HB\\\!\\cdot\\\!H\(sequence, head, chunk\) problems\. FISTA estimates the active\-set masks\(S,E\)\(S,E\); atorch\.linalg\.solveof the bordered system with a10−310^\{\-3\}margin\-block ridge returns stabilized coefficientsα~\\widetilde\{\\alpha\}, and autograd differentiates the solve\. This path is not the exact\-deletion solver; its end\-to\-end throughput is reported in Section[7](https://arxiv.org/html/2607.12204#S7.SS0.SSS0.Px2)\.*\(b\)*The hybrid SV\-memory layer: a query at positionttreads two paths\. Under the exact fp64 fixed\-CCsolver, the*long\-range memory*has certified reserve tokens \(α=0\\alpha=0\) and supports exact decremental deletion\. The*local window*\(the current chunk up tott\) is read directly by learnedq⋅kq\\\!\\cdot\\\!ksoftmax\. A local deletion masks the raw slot/weight and renormalizes; it has no SV certificate\. The pure chunk\-frozen gate plateaus because it cannot see this local window, so the hybrid adds the recent path while keeping the certificate boundary explicit\.

## 4Exact forgetting

We test exact forgetting twice: first as a concrete, legible removal, then against the formal definition of exact unlearning\. Every deletion and selection certificate in this and the next two sections uses the maintained CPU solver in fp64\. For trained\-model experiments, the model supplies frozen learned keys, which are then solved and deleted with that fp64 path\. Language\-model training and the throughput benchmark use the stabilized batched path and are not used to certify exact deletion\.

#### Removing a planted secret\.

We add a distinctive \(key, value\) pair to1414ordinary context tokens \(d=4d=4,σ=1\.5\\sigma=1\.5\), giving admission sizen0=15n\_\{0\}=15\. The reporting parameterν=0\.5\\nu=0\.5calibratesC=1/\(ν​n0\)=2/15C=1/\(\\nu n\_\{0\}\)=2/15once, and thatCCis held fixed for deletion and the never\-saw\-it reference\. We probe near every key over3030trials\. Before forgetting, a probe near the secret retrieved it in80%80\\%of trials\. The exact decrement left a maximum readout deviation of8\.9×10−58\.9\\times 10^\{\-5\}from a layer that never contained the secret, and the secret was retrieved in0%0\\%of trials\. Decay\-style forgetting \(scaling the secret’s coefficient byγ\\gamma\) left deviations of0\.570\.57,0\.340\.34, and0\.160\.16atγ=0\.5,0\.1,0\.01\\gamma=0\.5,0\.1,0\.01, and still surfaced the secret in80%80\\%,40%40\\%, and7%7\\%of trials\. Even a99%99\\%decay leaves the layer measurably shaped by the token; the decremental path removes the shape itself, because the remaining coefficients are re\-solved to the optimum they would have reached without it\.

#### Equivalence to retraining\.

Exact \(“perfect”\) unlearning is defined as equivalence to retraining from scratch without the data\(Cao and Yang,[2015](https://arxiv.org/html/2607.12204#bib.bib12); Bourtouleet al\.,[2021](https://arxiv.org/html/2607.12204#bib.bib11)\)\. That reference must use the same learning algorithm and hyperparameters\. HereCCis the primitive hyperparameter: decrement and retrain\-without both retain theCCcalibrated atn0n\_\{0\}\. Replacing it by1/\(ν​\(n0−1\)\)1/\(\\nu\(n\_\{0\}\-1\)\)after one deletion would change the optimization problem and would not be a same\-algorithm unlearning test\. We evaluate the fixed\-CCdecrement on two levels\.*State\-level\.*Where the SVDD optimum is unique, the decremental reverse path recovers the same active\-set partition\(S,E,R\)\(S,E,R\)and coefficientsα\\alpha, up to fp64 numerical tolerance, as a from\-scratch refit on the retained tokens\. The observed partition\-match rates are99%99\\%,98%98\\%, and100%100\\%on Gaussian, real MIMIC\-IV, and trained\-model keys respectively \(Table[1](https://arxiv.org/html/2607.12204#S4.T1)\)\. On exact duplicates the optimum is non\-unique, soα\\alphaand the partition can legitimately differ \(52%52\\%match\) without any forgetting failure, which is why we also measure at the function level\.*Function\-level \(the duplicate\-invariant metric\)\.*We compare the decision functionsf​\(x\)=2​∑iαi​κ​\(x,xi\)−ρf\(x\)=2\\sum\_\{i\}\\alpha\_\{i\}\\kappa\(x,x\_\{i\}\)\-\\rhoof the decrement and the refit on probes spanning the retained, forgotten, and jittered points, over300300trials per regime\. The deviation has a median of4\.5×10−134\.5\\times 10^\{\-13\}on the trained model’s learned keys and∼10−9\{\\sim\}10^\{\-9\}on well\-conditioned Gaussian and real\-ICU keys, rising to a worst case of1\.1×10−21\.1\\times 10^\{\-2\}confined to the ill\-conditioned near\-duplicate tail, which deduplication removes \(Figure[3](https://arxiv.org/html/2607.12204#S4.F3)b\); the mean is pulled up by that tail, so the median is the representative figure\. Coefficient decay, on the identical metric, deviates by2\.2×10−22\.2\\times 10^\{\-2\}to2\.4×10−12\.4\\times 10^\{\-1\}\(median\), larger than our*worst*case in every regime and five to eleven orders of magnitude larger at the median, because it never re\-solves and its gate still reflects the forgotten token\. The precise boundary of the claim is therefore algorithmic equivalence to the same fixed\-CCretrain\-without problem, realized numerically in fp64 and reported with the measured function deviations above\. The threat model is decision/readout influence, not membership inference\.

#### The cost of a deletion\.

The decrement is also cheap: an active\-token removal uses rank\-one updates on the maintained KKT inverse, while a certified\-inert reserve token deletes directly\. In our deterministic benchmark, a uniformly selected deletion completes in0\.310\.31–1\.231\.23ms on an Apple M3 Ultra CPU for contexts of120120–512512tokens \(the regime of our ICU streams\),1818–197×197\\timesfaster than a from\-scratch retained\-set QP refit with the same primitive boxCCheld fixed \(median over3030trials per context length;experiments/deletion\_latency\.py\)\. Across all trials, the maximum decision\-function deviation from that refit was5\.3×10−45\.3\\times 10^\{\-4\}\.

Table 1:Exact forgetting as equivalence to retraining \(300300trials/regime\)\. The decrement and from\-scratch reference run the same fp64 algorithm with the primitive boxCCheld fixed\. “partition match” is the fraction of trials in which they recover the same active\-set partition\(S,E,R\)\(S,E,R\); where the optimum is unique this is the state\-level equivalence test\. “decrement f\-dev” is the decision\-function deviation of the decrement from that refit; “decay f\-dev” is the same metric for coefficient decay\. The mean is pulled up by the ill\-conditioned tail, so the median is the typical figure\. Partition match below100%100\\%on duplicates reflects the non\-unique optimum, not a forgetting failure; the decision function still matches, and decay exceeds even our worst case everywhere\.![Refer to caption](https://arxiv.org/html/2607.12204v1/figs/forgetting_rigor.png)Figure 3:*Fixed\-CCfp64 decrement equals retraining; decay does not\.**\(a\)*Decision\-function deviation from a from\-scratch refit: the decrement’s*worst case*\(blue\) against coefficient decay’s*median*\(red\), on the same metric, across four regimes\. This is the conservative comparison, and the decrement’s worst case still falls below decay’s typical everywhere \(reaching∼10−11\{\\sim\}10^\{\-11\}on the trained model’s learned keys\)\.*\(b\)*The functional deviation as a function of the gate Gram condition number:∼10−13\{\\sim\}10^\{\-13\}for well\-conditioned keys \(left cluster\),∼10−9\{\\sim\}10^\{\-9\}in the typical regime, rising only into an ill\-conditioned tail \(near\-duplicate streams\) that deduplication removes\.

## 5Certified selection

Certified selection is the second property the active\-set partition confers: because reserve tokens are provably inert, evicting them to a bounded budget cannot change the output\. We establish it on the smallest tasks that can falsify it, against the eviction baselines \(H2O\-style heavy hitters, recency, random\) and the efficient\-attention baselines \(linear attention, DeltaNet\) a deployed selector must beat\.

#### The gate earns its keep under redundancy\.

A control experiment first scopes where the gate does and does not help\. On a causal multi\-query associative\-recall sweep\(Aroraet al\.,[2024](https://arxiv.org/html/2607.12204#bib.bib4)\)over*distinct*keys \(Table[2](https://arxiv.org/html/2607.12204#S5.T2)\), the distance\-kernel readout holds recall@1 at0\.960\.96–0\.990\.99while softmax decays from0\.870\.87to0\.140\.14and the matched\-state inner\-product recurrences \(DeltaNet, linear\) collapse to chance: the distance kernel retrieves where dot\-product memories cannot\. But a gate\-off ablation \(uniform weights, same readout\)*matches*the gate on distinct keys and slightly*exceeds*it at the highest load \(0\.990\.99vs\.0\.880\.88at6464items\): on clean, non\-redundant keys the max\-margin gate can drop a still\-needed key for no benefit\. The advantage there is the distance kernel, not the gate\. The gate earns its keep under redundancy, which the next results isolate\.

Table 2:Causal multi\-query recall@1 vs\. standard sequence baselines \(chunk\-frozen training\), sweeping the number of distinct key–value pairs \(nquery=8n\_\{\\text\{query\}\}=8,400400held\-out instances; bold = best per row\)\. “SV \(no gate\)” is the uniform\-weight ablation\. Inner\-product memories of matched state collapse as distinct items grow; the gate≈\\approxthe ablation on distinct keys and slightly trails it at the highest load \(the advantage there is the distance kernel, not the gate\)\.
#### The gate is load\-bearing on selective copying\.

The selective\-copying task\(Gu and Dao,[2024](https://arxiv.org/html/2607.12204#bib.bib3)\)is the opposite regime: a handful of informative \(content, value\) tokens must be recovered from a growing field of near\-duplicate zero\-value filler\. With88informative tokens andFFfiller tokens \(Table[3](https://arxiv.org/html/2607.12204#S5.T3)\), SV\-Attention holds recall@1 at∼0\.99\{\\sim\}0\.99asFFgrows from88to120120, while softmax is diluted to0\.480\.48–0\.760\.76and the fixed\-state recurrences fade\. The mechanism is the certified reserve: the support set stays at1111–1414tokens regardless of length, so the working state does not grow with the distractor mass\.

Table 3:Selective copying\(Gu and Dao,[2024](https://arxiv.org/html/2607.12204#bib.bib3)\):88informative \(content, value\) tokens buried inFFnear\-duplicate zero\-value filler tokens; the probe reproduces one informative token’s value \(recall@1,300300held\-out instances, chance0\.1250\.125\)\. “SV state” is the mean certified support size; filler is sent to the inert reserve, so the working state stays bounded as lengthLLgrows\.
#### Matched\-budget selection beats heavy\-hitter eviction\.

We construct contexts with skewed redundancy:66rare groups of one key each and66dense groups of1010near\-duplicate keys \(n=66n=66tokens,1212effective items\); two queries probe each group\. We fix a token budget equal to the gate’s support size and compare selection policies with an*identical*uniform\-RBF readout, so only the selection differs: the certified support set; an H2O\-style policy keeping the top\-kktokens by accumulated attention mass\(Zhanget al\.,[2023](https://arxiv.org/html/2607.12204#bib.bib8)\), given an*oracle*form \(scoring with the evaluation queries\) and a*warm*form \(scoring with an early half, then serving unseen queries\); recency; and random\. Over6060trials at a∼56%\{\\sim\}56\\%budget \(Table[4](https://arxiv.org/html/2607.12204#S5.T4)\), the certified gate beats every baseline; the gap is starkest on rare items \(0\.860\.86vs\.0\.320\.32for H2O\), because under redundancy the heavy hitters are the duplicated keys\. The gate also exceeds full\-context recall \(0\.930\.93vs\.0\.860\.86\), as evicting certified\-inert duplicates removes mass that crowds rare keys out of the kernel sum, and the advantage grows as the budget tightens: at a41%41\\%budget, rare\-item recall is0\.940\.94\(gate\) vs\.0\.210\.21\(H2O\)\. Streaming a redundant context through the causal layer with an online bounded budget \(deciding eviction before future queries are known\), certified selection retains rare items at0\.930\.93versus0\.300\.30for streaming heavy\-hitter eviction, confirming the result survives causal operation\.

Table 4:Matched\-budget eviction, multi\-query recall under skewed redundancy \(6060trials,∼56%\{\\sim\}56\\%budget\)\. Only the selection policy differs \(identical uniform\-RBF readout\)\.
#### An adaptive bounded state outlasts a fixed\-state recurrence\.

A fair capacity comparison against DeltaNet must isolate state*capacity*from the similarity function, since the distance kernel retrieves on non\-orthogonal keys where inner\-product memories cannot\. We therefore use near\-orthogonal content keys, on which every method retrieves perfectly at low load, and grow the number of*distinct*items at a fixed block of2020filler tokens \(Table[5](https://arxiv.org/html/2607.12204#S5.T5)\)\. All methods ace88items \(confirming fairness\), then the fixed\-state baselines fade as distinct items exceed capacity \(DeltaNet0\.93→0\.060\.93\\\!\\to\\\!0\.06\), while SV\-Attention holds near1\.01\.0through2424items and0\.950\.95at4040, its support set growing adaptively \(9→379\\\!\\to\\\!37\) but well below full context\. DeltaNet’s capacity scales with its state dimension, so this is a*matched\-state*claim, not an absolute one\.

Table 5:Fair capacity comparison on near\-orthogonal keys with2020filler tokens\. Recall@1 \(nearest value\),300300held\-out instances\. “SV state” is the mean support\-set size\.

## 6Capabilities on real data

We now demonstrate the two properties as concrete capabilities on a trained model, on real intensive\-care records, and on real text, rather than as aggregate metrics alone\. In each case, the certificate is computed by the exact fixed\-CCfp64 solver\.

#### Surgical forgetting and exact editing on a trained model

\(Figure[4](https://arxiv.org/html/2607.12204#S6.F4)\)\. We freeze the learned keys of a trained multi\-query associative\-recall model\(Aroraet al\.,[2024](https://arxiv.org/html/2607.12204#bib.bib4)\)\(recall@1=1\.0=1\.0\) and pass them to the fp64 maintained solver\. Forgetting one fact drops its retrieval from1\.01\.0to chance while every other fact remains perfectly answerable: the unlearning is targeted, with no collateral damage\. An exact fixed\-CCremove\-and\-add edits a fact \(“carol→\\tocairo” becomes “carol→\\totokyo”\); the old binding’s residual influence is0to machine precision, versus1\.151\.15for decay\. Under redundancy, certified eviction keeps a queried rare fact100%100\\%of the time at a token budget where heavy\-hitter \(H2O\) eviction keeps it21%21\\%\.

![Refer to caption](https://arxiv.org/html/2607.12204v1/figs/interpretable_demos.png)Figure 4:*Auditable fp64 memory on a trained recall model\.*Frozen learned keys are solved with the exact fixed\-CCmaintained path\.*\(a\)*Forgetting one fact \(carol\) flips only its own retrieval; the other seven facts remain answerable, so the unlearning is targeted with no collateral damage\.*\(b\)*An exact remove\-and\-add corrects a fact; the old binding leaves residual influence0\(machine precision\) versus1\.151\.15for decay\.*\(c\)*Under redundancy, certified eviction keeps a queried rare fact at a budget where heavy\-hitter \(H2O\), recency, and random eviction discard it\.
#### Right\-to\-be\-forgotten and certified retention on real ICU streams

\(Figure[5](https://arxiv.org/html/2607.12204#S6.F5)\)\. Intensive\-care vital\-sign records are a near\-ideal stress test: a patient is monitored for days, most hours are physiologically stable and nearly identical to their neighbors, and the decisive moments are rare and irregular\. We built hourly token streams from the MIMIC\-IV v3\.1 ICU module\(Johnsonet al\.,[2023](https://arxiv.org/html/2607.12204#bib.bib25)\):1,4651\{,\}465stays of at least4848hours, six routine vitals binned to hourly means, forward\-filled and standardized, a Harutyunyan\-style feature set\(Harutyunyanet al\.,[2019](https://arxiv.org/html/2607.12204#bib.bib26)\)\. Solving the gate per stay, the certified\-inert reserve fraction is0\.6670\.667and stable with length: roughly two of every three monitored hours are provably redundant\. At a matched budget equal to the support\-set size \(mean33%33\\%of the stream\), with an identical uniform\-RBF readout, the certified gate retained0\.800\.80of the rare deterioration hours a clinician must not lose, against0\.050\.05for heavy\-hitter eviction \(Table[6](https://arxiv.org/html/2607.12204#S6.T6)\); deterioration hours are physiologic outliers, and the one\-class support set is outlier\-aware by construction, whereas heavy\-hitter mass concentrates on the dense, stable hours\. Separately, fixed\-CCfp64 decremental forgetting of one of several patients sharing a context removed only that patient \(no longer retrievable; the others untouched and the retained state an exact solution, KKT residual∼10−12\{\\sim\}10^\{\-12\}\), and the forgotten patient’s record leaked zero weight into the readout, against0\.290\.29and0\.090\.09under decay: a right\-to\-be\-forgotten deletion with a numerical certificate, on real records\.

Table 6:Matched\-budget selection on real MIMIC\-IV ICU streams \(∼33%\{\\sim\}33\\%budget,1,3291\{,\}329stays\)\. Retention: fraction of deterioration hours kept\. Retrieval: a deterioration hour’s nearest retained neighbor is also one\.![Refer to caption](https://arxiv.org/html/2607.12204v1/figs/clinical_forget_demo.png)Figure 5:*Right\-to\-be\-forgotten on real MIMIC\-IV ICU records with the exact fixed\-CCfp64 solver\.**\(a\)*Deleting one patient removes only that patient \(no longer retrievable\); the others are untouched and the retained state stays exact \(KKT∼10−12\{\\sim\}10^\{\-12\}\)\.*\(b\)*The forgotten patient’s record leaks nothing after exact deletion, versus coefficient decay\.*\(c\)*At a matched token budget the certified gate keeps0\.800\.80of the rare deterioration hours, versus0\.050\.05for heavy\-hitter eviction \(computed on a400400\-stay subset; full\-cohort numbers in Table[6](https://arxiv.org/html/2607.12204#S6.T6)\)\.
#### Downstream prediction is competitive\.

Beyond selection, we ask the head\-to\-head question directly: at each hour, predict whether any vital crosses a deterioration threshold within the next six hours, a*future\-outcome*task, so the label and the gate do not read the same hour’s vitals\. Training the causal SV\-Attention layer and the sequence baselines \(Table[7](https://arxiv.org/html/2607.12204#S6.T7)\), SV\-Attention is competitive but not best: AUROC0\.700\.70versus0\.780\.78for an LSTM and0\.750\.75for softmax\. A2×22\\times 2ablation crossing the gate update \(chunk\-frozen vs\. per\-token\) with the selection mechanism \(max\-margin gate vs\. uniform\-RBF readout\) localizes the gap: a uniform\-RBF readout with per\-token updates*matches*softmax \(0\.7510\.751\), while the chunk\-frozen blind spot \(a query misses the most recent hours\) and the max\-margin selection \(weight on atypical rather than recently\-trending tokens\) each cap AUROC near0\.700\.70and are partly redundant\. The selection that wins certified retention is the same selection that mildly costs prediction, here∼0\.05\{\\sim\}0\.05AUROC\.

Table 7:Downstream next\-66h deterioration prediction on held\-out MIMIC\-IV ICU stays \(AUROC/AUPRC\)\. SV\-Attention is competitive but not the best raw predictor; its additional value is certified selection and exact forgetting when its long\-range gate is run with the fixed\-CCfp64 solver\.
#### A certified, forgettable retrieval memory over real text

\(Figure[6](https://arxiv.org/html/2607.12204#S6.F6)\)\. We build a retrieval memory whose fact keys are real MiniLM sentence embeddings and whose gate is the exact fixed\-CCfp64 one\-class SVM; it answers7/77/7natural\-language questions\. Deleting “Alice’s password” makes the system return “no record—forgotten” on that query while other facts answer unchanged, and under boilerplate redundancy certified eviction keeps a rare fact answerable \(1\.001\.00\) where heavy\-hitter eviction does not \(0\.000\.00\)\. This is the deployable framing: a retrieval\-augmented memory\(Lewiset al\.,[2020](https://arxiv.org/html/2607.12204#bib.bib32)\)for a language model whose retention and deletion are auditable rather than scored\.

![Refer to caption](https://arxiv.org/html/2607.12204v1/figs/sv_rag_demo.png)Figure 6:*A certified, forgettable retrieval memory over real text*\(MiniLM sentence embeddings; exact fixed\-CCfp64 solver\)\.*\(a\)*Deleting “Alice’s password” makes the system answer “no record—forgotten” on that query while other facts answer unchanged\.*\(b\)*Under boilerplate redundancy, certified eviction keeps a rare fact answerable at a budget where heavy\-hitter, recency, and random eviction do not\.

## 7The gate is correct and trainable

#### Maintained\-solver and gradient verification\.

We verify the exact path’s custom implicit backward with double\-precisiongradcheck\(ϵ=10−6\\epsilon=10^\{\-6\}\), differentiating through the kernel map so perturbations preserve the symmetry ofKKand the partition is locally stable; gradcheck passes for a scalarization of\(α,ρ\)\(\\alpha,\\rho\), for the decision scores, and for the full layer readouto​\(q\)o\(q\)with respect to keys, values, and queries\. A student layer trained by Adam to match a teacher SV\-Attention mapping reduced loss by99%99\\%in200200steps, confirming that custom\-VJP gradients are usable, not merely correct\. We separately gradcheck the batchedtorch\.linalg\.solve/autograd path end to end; that check validates the derivative of the ridged approximation, not exact coefficients of \([2](https://arxiv.org/html/2607.12204#S3.E2)\)\. We verified the maintained incremental solver against an independent interior\-point QP: on well\-conditioned data it matched the batch optimum at every point of a4545\-point stream \(worst decision\-value gap1\.6×10−61\.6\\times 10^\{\-6\}\), removal matched the batch optimum on the retained set \(2\.7×10−72\.7\\times 10^\{\-7\}\), and decrement∘\\circincrement restored the state to4\.4×10−164\.4\\times 10^\{\-16\}\. On near\-duplicate points the key Gram matrix can reach condition number∼1011\{\\sim\}10^\{11\}and the optimum is effectively non\-unique, so fixed\-CCretraining equivalence is reported at the decision\-function level with a logged numerical tolerance\.

#### The batched solver makes the gate trainable at scale\.

Table[8](https://arxiv.org/html/2607.12204#S7.T8)times a full forward/backward step at the3\.223\.22M configuration \(four layers,dmodel=256d\_\{\\text\{model\}\}\{=\}256, block256256, batch1616\)\. Each direct row is the median of seven synchronized repetitions after three warm\-ups\. The current hybrid path spans PyTorch on CPU \(model, ridged KKT solve, and autograd\) and MLX on the Apple GPU \(FISTA partition finder\), including transfers between them\. It reaches9,1259\{,\}125tokens/s:6\.11×6\.11\\timesthe step time of an unfused PyTorch CPU softmax reference and35\.80×35\.80\\timesthe step time of the corresponding MPS reference\. The two exact active\-set rows are historical RBF\-only diagnostics, measured directly at batch22and linearly extrapolated to batch1616; they are not like\-for\-like hybrid measurements\. A5050k\-step projection for the directly measured mixed path is6\.236\.23hours, excluding optimizer updates, data loading, evaluation, and checkpoints\. The1010M model we study \(Section[8](https://arxiv.org/html/2607.12204#S8);dmodel=384d\_\{\\text\{model\}\}\{=\}384, block512512\) runs at∼2,400\{\\sim\}2\{,\}400tokens/s and trains for6,0006\{,\}000steps in under six hours\. The comparison establishes practical small\-scale training, not throughput parity; a fused softmax kernel would widen the accelerator gap\.

Table 8:Full\-model training\-step throughput at the3\.223\.22M configuration \(forward\+\+backward; Apple M3 Ultra;4,0964\{,\}096tokens/step\)\. Direct rows report batch1616medians; starred exact\-solver rows extrapolate batch22measurements linearly by8×8\\timesand implement the RBF, not hybrid, readout\. Relative time is against the CPU softmax row\. All accelerator work is synchronized\.

## 8A language\-model usability probe

This section asks whether the hybrid layer is a usable sequence layer at a matched token budget\. We use byte\-level corpora with no tokenizer, where a controlled comparison is cleanest, and report on both standard enwik8 and the contemporary TinyStories corpus\(Eldan and Li,[2023](https://arxiv.org/html/2607.12204#bib.bib30)\)at two model scales\. The only fair state\-matched comparison is against a sliding\-window softmax Transformer \(“Transformer\+\+,” swa\), whose window we set to the hybrid’s measured state; recurrent baselines \(DeltaNet, linear\) carry their own state and are reported at it \(full protocol in Appendix[C](https://arxiv.org/html/2607.12204#A3)\)\.

At3\.223\.22M parameters \(Table[9](https://arxiv.org/html/2607.12204#S8.T9)\), the seven\-seed mean BPC is2\.1782\.178for the hybrid and2\.3832\.383for the matched\-state Transformer\+\+\. Every paired seed favors the hybrid, with a mean relative improvement of8\.6%8\.6\\%\(95%95\\%CI\[5\.2,12\.0\]%\[5\.2,12\.0\]\\%; pairedt=5\.95t=5\.95, two\-sidedp=0\.001p=0\.001; Appendix[D](https://arxiv.org/html/2607.12204#A4)\)\. The DeltaNet, linear\-attention, and full\-softmax entries are single\-seed reference runs, so we do not use them for multi\-seed inference; they suggest that DeltaNet, with its larger matrix state, is the stronger raw predictor\. On TinyStories, all three paired seeds favor the hybrid by14\.2%14\.2\\%on average, but this is a directional replication atn=3n=3, not a statistically confirmed result \(p=0\.057p=0\.057\)\. The hybrid’s distinction is the certifiable exact\-solver memory that the recurrences cannot provide, not dominance in raw perplexity\.

The small\-scale edge does not extend to larger models\.Repeating the matched\-state comparison at1010M parameters \(block512512, five seeds\) is a wash: SV is2\.5%2\.5\\%*worse*than swa on average with very high seed variance \(sd∼15%\{\\sim\}15\\%; not significant\), and a single3232M run is similar\. Two effects compound\. At block512512the matched window is most of the context, so the sliding\-window baseline is already near\-full attention and leaves little for a selective long\-range memory to win\. More importantly, the chunk\-frozen gate optimizes more slowly and less stably at scale: several1010M seeds \(and the3232M run\) had not converged within our6,0006\{,\}000\-step budget \(their loss was still descending\), so the fixed\-budget comparison penalizes them\. We therefore scope the language\-model edge to the small\-scale regime and treat scalable, stable optimization of the gate as an open problem\. This is a limitation of the*language\-modeling*use; certified selection and exact deletion are properties of the separate fp64 maintained solve, established on real data and independent of language\-model scale\.

Table 9:Matched\-state language modeling \(3\.223\.22M params, four layers, block256256, enwik8 BPC; lower is better\)\. SV and matched swa are seven\-seed means and form the paired significance comparison \(Appendix[D](https://arxiv.org/html/2607.12204#A4)\)\. DeltaNet, linear attention, and full softmax are single\-seed references only\. “State” is the token budget each layer reads\. The supported comparison is the small\-scale matched\-state SV–swa result; the other rows provide directional context, not multi\-seed claims\.
## 9Discussion

The experiments support a single thesis: when the inner problem of a test\-time\-learning layer is constrained and max\-margin rather than dense and least\-squares, the layer acquires structure that can be*certified*rather than merely measured\. For the exact fixed\-CCsolve, the active\-set partition is not a score to be thresholded; it is a statement, backed by complementary slackness, about which tokens the readout provably ignores\. This is the right lens on the empirical results\. Heavy\-hitter eviction lost on rare items not because its scoring is poorly tuned but because accumulated attention mass is a*popularity*statistic: under redundancy, popularity concentrates on duplicates, and the rare singletons that carry the information are exactly what gets evicted\. The one\-class gate inverts this failure mode by construction, spending its budget on the boundary of the key distribution, and the MIMIC\-IV result is the same mechanism on data we did not design: deterioration hours are physiologic outliers, so a gate that keeps outliers keeps them, at a0\.800\.80vs\.0\.050\.05retention gap\.

Exactness of forgetting deserves a separate comment, because its value is regulatory as much as technical\. Every gradient\-based memory forgets by decay, which leaves behind an influence one can only bound empirically and argue about; the fp64 decremental path instead returns the solver to the state it would hold without the token under the same fixedCC, to a tolerance we log\. We checked this not against a convenient probe but against the formal definition of exact unlearning, equivalence to same\-algorithm retraining, on real ICU records and on a trained model’s own keys, and we measured coefficient decay on the identical yardstick so the gap is not an artifact of metric choice\. When deleting a record is a legal or clinical obligation, “the residual was small” and “the model was provably re\-solved without it” are not the same statement, and only the second is one an auditor can act on\. This also marks where the layer does*not*help: the certificate is a statement about redundancy and atypicality, and a context with neither gives the gate nothing to certify, so it falls back to ordinary kernel attention\. Fortunately the workloads that motivate long\-context memory \(boilerplate documents, vitals stable for hours, dialogue that repeats\) are redundant in precisely this sense\. In the hybrid, this certificate stops at the long\-range gate: current\-chunk tokens belong to the dense local path and are removed by masking and renormalization\.

## 10Limitations

- •The advantage is regime\-specific\.The selection advantage requires the informative tokens to be distributional outliers, or the context to be redundant; the gate is a one\-class anomaly detector and confers no benefit where importance does not coincide with atypicality\. It is an active\-set solve, not a fused kernel\. The current mixed CPU–MLX implementation is6\.1×6\.1\\timesslower than an unfused CPU softmax reference and35\.8×35\.8\\timesslower than its MPS reference, so it is not a throughput\-competitive general\-purpose attention layer\.
- •The language\-model edge is confined to small scale\.The significant enwik8 matched\-state win \(8\.6%8\.6\\%,p=0\.001p=0\.001, seven seeds\) is at3\.223\.22M parameters\. TinyStories is a three\-seed directional replication \(14\.2%14\.2\\%mean improvement,p=0\.057p=0\.057\), not statistical confirmation\. At1010M \(five seeds\) the comparison is a wash and a3232M run is similar\. The chunk\-frozen gate converges markedly slower and less stably than softmax at1010M–3232M; a stable, budget\-efficient optimizer for the gate at scale is an open problem we do not solve here\. This is the central obstacle to the language\-modeling use; it does not affect the exact\-solver certificate experiments\.
- •The training path is approximate and the hybrid certificate is long\-range only\.Batched training uses an estimated partition, a10−310^\{\-3\}ridge,torch\.linalg\.solve, and autograd\. It does not produce exact SVDD coefficients, does not reuse a maintained inverse in backward, and carries no exact\-deletion certificate\. Exact deletion requires the fixed\-CCfp64 maintained solver\. Within the hybrid, that certificate covers admitted long\-range tokens; deleting a current\-chunk local token masks its raw slot/weight and renormalizes, with no SV certificate\.
- •Exactness carries a logged tolerance on ill\-conditioned streams\.Where the optimum is unique, the fixed\-CCdecrement and retrain\-without target the same state; in fp64 we report the observed partition match and function deviation\. On near\-duplicate keys the optimum is non\-unique and conditioning degrades, with a logged worst\-case decision\-function deviation of1\.1×10−21\.1\\times 10^\{\-2\}; deduplication is required on maximally redundant streams\. We also found one edge case \(the margin set emptying mid\-decrement\) unhandled in both our solver and the reference, which deduplication avoids but a production decrement path should cover\. Finally, fixed\-CCfeasibility requiresn​C≥1nC\\geq 1after deletion; below the implementation’s⌈1/C⌉\+2\\lceil 1/C\\rceil\+2admission threshold it falls back to the ungated path and makes no SV certificate\.
- •Forgetting threat model\.We measure decision/readout influence, not membership inference; deploying for compliance would require a privacy evaluation appropriate to the regulation\.
- •The contribution is scale\-independent\.Certified selection and exact deletion are properties of the fixed\-CCfp64 solve, established on real MIMIC\-IV records and a trained model’s learned representations; they do not depend on language\-model scale, so the small\-scale probe caveat above does not weaken those solver\-level results\.

## 11Conclusion

We added the constrained, max\-margin member to the test\-time\-learning family of attention layers\. Its exact fixed\-CCfp64 maintained solver provides certified selection, exact forgetting relative to same\-algorithm retraining, and a custom backward that reuses the forward\-maintained KKT inverse\. A separate10−310^\{\-3\}\-ridged batchedtorch\.linalg\.solve/autograd path makes training practical, but its coefficients are approximate and it does not inherit the exact\-deletion certificate\. We verified the solver\-level properties on real clinical streams and learned representations and made the hybrid boundary explicit: the certificate covers the long\-range gate, while a current\-chunk deletion masks and renormalizes the local path\. As attention systems take on longer contexts and stricter retention obligations, such precisely scoped guarantees can be evaluated alongside predictive and throughput benchmarks\.

## Broader Impact

Exact, auditable deletion of a token’s influence is a capability with an obvious upside for deployed systems that must honor erasure requests: a mechanism whose removals can be demonstrated is preferable to one whose removals can only be claimed\. We are deliberate about its limits\. The guarantee applies to the exact fixed\-CCfp64 long\-range gate, not the ridged batched training path or the hybrid’s local\-path mask\. It concerns decision and readout behavior, not membership inference, so a deleted long\-range token is “forgotten” in the sense that the memory’s outputs match a same\-hyperparameter model that never held it, not in the sense of a formal privacy bound; a compliance use would need to be paired with a privacy evaluation matched to the relevant regulation\. The clinical study is retrospective on de\-identified MIMIC\-IV data used under its credentialed agreement, the deterioration labels are alarm thresholds rather than validated endpoints, and none of this constitutes a clinical tool absent prospective validation\.

## Data Availability and Reproducibility

The clinical experiments use the MIMIC\-IV v3\.1 ICU module\(Johnsonet al\.,[2023](https://arxiv.org/html/2607.12204#bib.bib25)\), a de\-identified record dataset released by the MIT Laboratory for Computational Physiology through PhysioNet and available to credentialed researchers under the PhysioNet Credentialed Health Data Use Agreement; the data were used in accordance with that agreement\. Because the agreement prohibits redistribution of the dataset or its derivatives, we release the full preprocessing and experiment code, which reproduces every clinical result from the raw PhysioNet files, but not the data or any derived caches\. The verified solver, the batched GPU gate, the hybrid layer, the full verification suite, and every synthetic experiment and figure are released as code with fixed seeds\. Source and reproducibility materials are available at[https://github\.com/vishrmsh/sv\-attention](https://github.com/vishrmsh/sv-attention)\.

## References

- A\. Agrawal, B\. Amos, S\. Barratt, S\. Boyd, S\. Diamond, and J\. Z\. Kolter \(2019\)Differentiable convex optimization layers\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/1910.12430)Cited by:[§2](https://arxiv.org/html/2607.12204#S2.p5.1)\.
- B\. Amos and J\. Z\. Kolter \(2017\)OptNet: differentiable optimization as a layer in neural networks\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/1703.00443)Cited by:[§2](https://arxiv.org/html/2607.12204#S2.p5.1)\.
- S\. Arora, S\. Eyuboglu, A\. Timalsina, I\. Johnson, M\. Poli, J\. Zou, A\. Rudra, and C\. Ré \(2024\)Zoology: measuring and improving recall in efficient language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2312.04927)Cited by:[§5](https://arxiv.org/html/2607.12204#S5.SS0.SSS0.Px1.p1.7),[§6](https://arxiv.org/html/2607.12204#S6.SS0.SSS0.Px1.p1.9)\.
- A\. Beck and M\. Teboulle \(2009\)A fast iterative shrinkage\-thresholding algorithm for linear inverse problems\.SIAM Journal on Imaging Sciences2\(1\),pp\. 183–202\.External Links:[Link](https://doi.org/10.1137/080716542)Cited by:[§3\.4](https://arxiv.org/html/2607.12204#S3.SS4.p1.3)\.
- A\. Behrouz, P\. Zhong, and V\. Mirrokni \(2025\)Titans: learning to memorize at test time\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2501.00663)Cited by:[§1](https://arxiv.org/html/2607.12204#S1.p1.1),[§2](https://arxiv.org/html/2607.12204#S2.p1.1)\.
- I\. Beltagy, M\. E\. Peters, and A\. Cohan \(2020\)Longformer: the long\-document transformer\.InarXiv preprint arXiv:2004\.05150,External Links:[Link](https://arxiv.org/abs/2004.05150)Cited by:[§2](https://arxiv.org/html/2607.12204#S2.p4.1),[§3\.5](https://arxiv.org/html/2607.12204#S3.SS5.p2.3)\.
- L\. Bourtoule, V\. Chandrasekaran, C\. A\. Choquette\-Choo, H\. Jia, A\. Travers, B\. Zhang, D\. Lie, and N\. Papernot \(2021\)Machine unlearning\.InIEEE Symposium on Security and Privacy \(S&P\),External Links:[Link](https://arxiv.org/abs/1912.03817)Cited by:[item 1](https://arxiv.org/html/2607.12204#S1.I1.i1.p1.6),[§1](https://arxiv.org/html/2607.12204#S1.p2.1),[§2](https://arxiv.org/html/2607.12204#S2.p3.3),[§4](https://arxiv.org/html/2607.12204#S4.SS0.SSS0.Px2.p1.20)\.
- Y\. Cao and J\. Yang \(2015\)Towards making systems forget with machine unlearning\.InIEEE Symposium on Security and Privacy \(S&P\),External Links:[Link](https://doi.org/10.1109/SP.2015.35)Cited by:[item 1](https://arxiv.org/html/2607.12204#S1.I1.i1.p1.6),[§2](https://arxiv.org/html/2607.12204#S2.p3.3),[§4](https://arxiv.org/html/2607.12204#S4.SS0.SSS0.Px2.p1.20)\.
- G\. Cauwenberghs and T\. Poggio \(2000\)Incremental and decremental support vector machine learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://papers.nips.cc/paper_files/paper/2000/hash/155fa09596c7e18e50b58eb7e0c6ccb4-Abstract.html)Cited by:[§1](https://arxiv.org/html/2607.12204#S1.p3.1),[§2](https://arxiv.org/html/2607.12204#S2.p5.1),[§3\.2](https://arxiv.org/html/2607.12204#S3.SS2.p1.3)\.
- R\. Eldan and Y\. Li \(2023\)TinyStories: how small can language models be and still speak coherent english?\.arXiv preprint arXiv:2305\.07759\.External Links:[Link](https://arxiv.org/abs/2305.07759)Cited by:[Appendix D](https://arxiv.org/html/2607.12204#A4.p2.7),[§8](https://arxiv.org/html/2607.12204#S8.p1.1)\.
- A\. Gu and T\. Dao \(2024\)Mamba: linear\-time sequence modeling with selective state spaces\.InConference on Language Modeling \(COLM\),External Links:[Link](https://arxiv.org/abs/2312.00752)Cited by:[§1](https://arxiv.org/html/2607.12204#S1.p1.1),[§2](https://arxiv.org/html/2607.12204#S2.p1.1),[§5](https://arxiv.org/html/2607.12204#S5.SS0.SSS0.Px2.p1.10),[Table 3](https://arxiv.org/html/2607.12204#S5.T3)\.
- A\. Hannun, J\. Digani, A\. Katharopoulos, and R\. Collobert \(2023\)MLX: efficient and flexible machine learning on apple silicon\.External Links:[Link](https://github.com/ml-explore/mlx)Cited by:[Appendix B](https://arxiv.org/html/2607.12204#A2.p1.1),[§3\.4](https://arxiv.org/html/2607.12204#S3.SS4.p1.3)\.
- H\. Harutyunyan, H\. Khachatrian, D\. C\. Kale, G\. Ver Steeg, and A\. Galstyan \(2019\)Multitask learning and benchmarking with clinical time series data\.Scientific Data6\(1\),pp\. 96\.External Links:[Link](https://doi.org/10.1038/s41597-019-0103-9)Cited by:[Appendix C](https://arxiv.org/html/2607.12204#A3.SS0.SSS0.Px4.p1.9),[§6](https://arxiv.org/html/2607.12204#S6.SS0.SSS0.Px2.p1.10)\.
- Y\. Huang, P\. Wu, and C\. Wang \(2024\)ECO: efficient computational optimization for exact machine unlearning in deep neural networks\.InICML 2024 Workshop on Advancing Neural Network Training \(WANT\),External Links:[Link](https://openreview.net/forum?id=SeBVP0zxKp)Cited by:[§1](https://arxiv.org/html/2607.12204#S1.p2.1),[§2](https://arxiv.org/html/2607.12204#S2.p3.3)\.
- A\. E\. W\. Johnson, L\. Bulgarelli, L\. Shen, A\. Gayles, A\. Shammout, S\. Horng, T\. J\. Pollard, S\. Hao, B\. Moody, B\. Gow, L\. H\. Lehman, L\. A\. Celi, and R\. G\. Mark \(2023\)MIMIC\-IV, a freely accessible electronic health record dataset\.Scientific Data10\(1\)\.External Links:[Link](https://doi.org/10.1038/s41597-022-01899-x)Cited by:[item 1](https://arxiv.org/html/2607.12204#S1.I1.i1.p1.6),[§6](https://arxiv.org/html/2607.12204#S6.SS0.SSS0.Px2.p1.10),[Data Availability and Reproducibility](https://arxiv.org/html/2607.12204#Sx2.p1.1)\.
- A\. Katharopoulos, A\. Vyas, N\. Pappas, and F\. Fleuret \(2020\)Transformers are RNNs: fast autoregressive transformers with linear attention\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2006.16236)Cited by:[§1](https://arxiv.org/html/2607.12204#S1.p1.1),[§2](https://arxiv.org/html/2607.12204#S2.p1.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel, S\. Riedel, and D\. Kiela \(2020\)Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2005.11401)Cited by:[§6](https://arxiv.org/html/2607.12204#S6.SS0.SSS0.Px4.p1.4)\.
- N\. Li, A\. Pan, A\. Gopal, S\. Yue, D\. Berrios,et al\.\(2024a\)The WMDP benchmark: measuring and reducing malicious use with unlearning\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2403.03218)Cited by:[§2](https://arxiv.org/html/2607.12204#S2.p3.3)\.
- Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen \(2024b\)SnapKV: LLM knows what you are looking for before generation\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2404.14469)Cited by:[§1](https://arxiv.org/html/2607.12204#S1.p2.1),[§2](https://arxiv.org/html/2607.12204#S2.p4.1)\.
- J\. Ma, J\. Theiler, and S\. Perkins \(2003\)Accurate on\-line support vector regression\.Neural Computation15\(11\),pp\. 2683–2703\.External Links:[Link](https://doi.org/10.1162/089976603322385117)Cited by:[§3\.2](https://arxiv.org/html/2607.12204#S3.SS2.p1.3)\.
- P\. Maini, Z\. Feng, A\. Schwarzschild, Z\. C\. Lipton, and J\. Z\. Kolter \(2024\)TOFU: a task of fictitious unlearning for LLMs\.arXiv preprint arXiv:2401\.06121\.External Links:[Link](https://arxiv.org/abs/2401.06121)Cited by:[§2](https://arxiv.org/html/2607.12204#S2.p3.3)\.
- K\. Meng, D\. Bau, A\. Andonian, and Y\. Belinkov \(2022\)Locating and editing factual associations in GPT\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2202.05262)Cited by:[§2](https://arxiv.org/html/2607.12204#S2.p3.3)\.
- K\. Meng, A\. S\. Sharma, A\. Andonian, Y\. Belinkov, and D\. Bau \(2023\)Mass\-editing memory in a transformer\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2210.07229)Cited by:[§2](https://arxiv.org/html/2607.12204#S2.p3.3)\.
- I\. Schlag, K\. Irie, and J\. Schmidhuber \(2021\)Linear transformers are secretly fast weight programmers\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2102.11174)Cited by:[§2](https://arxiv.org/html/2607.12204#S2.p1.1)\.
- B\. Schölkopf, J\. C\. Platt, J\. Shawe\-Taylor, A\. J\. Smola, and R\. C\. Williamson \(2001\)Estimating the support of a high\-dimensional distribution\.Neural Computation13\(7\),pp\. 1443–1471\.External Links:[Link](https://doi.org/10.1162/089976601750264965)Cited by:[§3\.1](https://arxiv.org/html/2607.12204#S3.SS1.p1.7)\.
- W\. Shi, J\. Lee, Y\. Huang, S\. Malladi, J\. Zhao, A\. Holtzman, D\. Liu, L\. Zettlemoyer, N\. A\. Smith, and C\. Zhang \(2025\)MUSE: machine unlearning six\-way evaluation for language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2407.06460)Cited by:[§2](https://arxiv.org/html/2607.12204#S2.p3.3)\.
- D\. A\. Tarzanagh, Y\. Li, C\. Thrampoulidis, and S\. Oymak \(2023\)Transformers as support vector machines\.arXiv preprint arXiv:2308\.16898\.External Links:[Link](https://arxiv.org/abs/2308.16898)Cited by:[§2](https://arxiv.org/html/2607.12204#S2.p2.1)\.
- D\. M\. J\. Tax and R\. P\. W\. Duin \(2004\)Support vector data description\.Machine Learning54\(1\),pp\. 45–66\.External Links:[Link](https://doi.org/10.1023/B:MACH.0000008084.60811.49)Cited by:[§3\.1](https://arxiv.org/html/2607.12204#S3.SS1.p1.7)\.
- J\. von Oswald, N\. Scherrer, S\. Kobayashi, L\. Versari, S\. Yang, M\. Schlegel, K\. Maile, Y\. Schimpf, O\. Sieberling, A\. Meulemans, R\. A\. Saurous, G\. Lajoie, C\. Frenkel, R\. Pascanu, B\. Agüera y Arcas, and J\. Sacramento \(2025\)MesaNet: sequence modeling by locally optimal test\-time training\.arXiv preprint arXiv:2506\.05233\.External Links:[Link](https://arxiv.org/abs/2506.05233)Cited by:[§1](https://arxiv.org/html/2607.12204#S1.p1.1),[§2](https://arxiv.org/html/2607.12204#S2.p1.1)\.
- J\. von Oswald, M\. Schlegel, A\. Meulemans, S\. Kobayashi, E\. Niklasson, N\. Zucchet, N\. Scherrer, N\. Miller, M\. Sandler, B\. Agüera y Arcas, M\. Vladymyrov, and J\. Sacramento \(2023\)Uncovering mesa\-optimization algorithms in transformers\.arXiv preprint arXiv:2309\.05858\.External Links:[Link](https://arxiv.org/abs/2309.05858)Cited by:[§1](https://arxiv.org/html/2607.12204#S1.p1.1),[§2](https://arxiv.org/html/2607.12204#S2.p1.1)\.
- S\. Yang, B\. Wang, Y\. Zhang, Y\. Shen, and Y\. Kim \(2024\)Parallelizing linear transformers with the delta rule over sequence length\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2406.06484)Cited by:[§1](https://arxiv.org/html/2607.12204#S1.p1.1),[§2](https://arxiv.org/html/2607.12204#S2.p1.1)\.
- J\. Yao, S\. Shen, K\. Du, S\. Feng, D\. Seo, R\. Zhang, Y\. Huang, Y\. Liu, S\. Lu, and J\. Jiang \(2026\)VeriCache: turning lossy KV cache into lossless LLM inference\.arXiv preprint arXiv:2605\.17613\.External Links:[Link](https://arxiv.org/abs/2605.17613)Cited by:[§2](https://arxiv.org/html/2607.12204#S2.p4.1)\.
- Z\. Zhang, Y\. Sheng, T\. Zhou, T\. Chen, L\. Zheng, R\. Cai, Z\. Song, Y\. Tian, C\. Ré, C\. Barrett, Z\. Wang, and B\. Chen \(2023\)H2O: heavy\-hitter oracle for efficient generative inference of large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2306.14048)Cited by:[§1](https://arxiv.org/html/2607.12204#S1.p2.1),[§2](https://arxiv.org/html/2607.12204#S2.p4.1),[§5](https://arxiv.org/html/2607.12204#S5.SS0.SSS0.Px3.p1.17)\.

## Appendix AThe implicit vector–Jacobian product

This section describes the custom backward for the exact fp64 maintained solver; it is not the backward used by the stabilized batched training path\. Within a fixed active\-set partition\(S,E,R\)\(S,E,R\)the free variablesu=\[ρ;αS\]u=\[\\rho;\\,\\alpha\_\{S\}\]satisfy the bordered KKT systemM​u=cMu=c, withM=\[\[0,𝟏⊤\],\[𝟏,2​KS​S\]\]M=\[\[0,\\mathbf\{1\}^\{\\top\}\],\[\\mathbf\{1\},2K\_\{SS\}\]\]and

c=\[1−\|E\|​Cdiag​\(K\)S−2​C​KS​E​𝟏\],u=ℛ​c,ℛ=M−1\.c=\\begin\{bmatrix\}1\-\|E\|\\,C\\\\ \\mathrm\{diag\}\(K\)\_\{S\}\-2C\\,K\_\{SE\}\\mathbf\{1\}\\end\{bmatrix\},\\qquad u=\\mathcal\{R\}c,\\quad\\mathcal\{R\}=M^\{\-1\}\.\(4\)For a lossLLwith upstream cotangentsρ¯,α¯S\\bar\{\\rho\},\\bar\{\\alpha\}\_\{S\}, letu¯=\[ρ¯;α¯S\]\\bar\{u\}=\[\\bar\{\\rho\};\\,\\bar\{\\alpha\}\_\{S\}\]andw=ℛ⊤​u¯w=\\mathcal\{R\}^\{\\top\}\\bar\{u\}with margin blockwSw\_\{S\}\. Differentiatingu=ℛ​cu=\\mathcal\{R\}cand usingd​ℛ=−ℛ​d​M​ℛd\\mathcal\{R\}=\-\\mathcal\{R\}\\,dM\\,\\mathcal\{R\}givesd​L=w⊤​d​c−w⊤​d​M​udL=w^\{\\top\}dc\-w^\{\\top\}dM\\,u; collecting the coefficients ofd​Ki​jdK\_\{ij\},

∂L∂Ki​i\\displaystyle\\frac\{\\partial L\}\{\\partial K\_\{ii\}\}\+=\[wS\]i\\displaystyle\\mathrel\{\+\}=\[w\_\{S\}\]\_\{i\}\(i∈S\),\\displaystyle\(i\\in S\),∂L∂Ki​e\\displaystyle\\frac\{\\partial L\}\{\\partial K\_\{ie\}\}\+=−2C\[wS\]i\\displaystyle\\mathrel\{\+\}=\-2C\\,\[w\_\{S\}\]\_\{i\}\(i∈S,e∈E\),\\displaystyle\(i\\in S,\\,e\\in E\),∂L∂Ki​j\\displaystyle\\frac\{\\partial L\}\{\\partial K\_\{ij\}\}\+=−2\[wS\]i\[αS\]j\\displaystyle\\mathrel\{\+\}=\-2\\,\[w\_\{S\}\]\_\{i\}\\,\[\\alpha\_\{S\}\]\_\{j\}\(i,j∈S\)\.\\displaystyle\(i,j\\in S\)\.\(5\)Every term is a rank\-one assembly, and the custom backward only multiplies byℛ⊤\\mathcal\{R\}^\{\\top\}, which the exact forward pass already maintains; it performs no second system solve\. Changes in the active set happen on a measure\-zero set of inputs across which the map is piecewise smooth; as with the kink in a ReLU, we do not route gradient through the discrete change in set membership\. The expressions are verified against finite differences by double\-precisiongradcheck\. This inverse\-reuse/no\-second\-solve statement does not apply to Appendix[B](https://arxiv.org/html/2607.12204#A2)\.

## Appendix BThe batched projected\-gradient solver

The language\-model training forward uses a stabilized approximation for every \(sequence, head, chunk\) problem\. A sequential active\-set walk solves one problem at a time; instead, the single\-precision accelerated projected gradient of Algorithm[1](https://arxiv.org/html/2607.12204#alg1)estimates all partitions at once\. Because \([2](https://arxiv.org/html/2607.12204#S3.E2)\) is a box\- and sum\-constrained convex quadratic program, each FISTA step is a dense batched matrix product and a vectorized projection onto the capped simplex\{a:∑iai=1,0≤ai≤C\}\\\{a:\\sum\_\{i\}a\_\{i\}=1,\\,0\\leq a\_\{i\}\\leq C\\\}, computed by bisection on the dual variable\. The projection and step are fused into one compiled graph and run through MLX\(Hannunet al\.,[2023](https://arxiv.org/html/2607.12204#bib.bib31)\); a per\-row mask lets problems of different prefix length share one batch\.

The estimated coefficients define masks\(S,E\)\(S,E\)\. In PyTorch, the implementation packs their bordered KKT systems, adds a fixed ridgeλ=10−3\\lambda=10^\{\-3\}to each valid margin diagonal, and computes

M~​u=c,M~=M\+λ​diag​\(0,𝟏S\),α~S=uS\\widetilde\{M\}u=c,\\qquad\\widetilde\{M\}=M\+\\lambda\\,\\mathrm\{diag\}\(0,\\mathbf\{1\}\_\{S\}\),\\qquad\\widetilde\{\\alpha\}\_\{S\}=u\_\{S\}\(6\)withtorch\.linalg\.solve\. Standard autograd differentiates this solve\. Thusα~\\widetilde\{\\alpha\}is the exact solution of the*ridged packed system*, but only a stabilized approximation to the unregularized coefficients of \([2](https://arxiv.org/html/2607.12204#S3.E2)\)\. This path neither invokes the custom VJP of Appendix[A](https://arxiv.org/html/2607.12204#A1)nor inherits its no\-second\-solve property, and it is not used for an exact\-deletion certificate\.

Algorithm 1Stabilized batched SVDD training approximation \(one FISTA step shown; all problems in parallel\)\.1:inputGram tensors

K∈ℝG×n×nK\\in\\mathbb\{R\}^\{G\\times n\\times n\}, box

CC, mask

μ∈\{0,1\}G×n\\mu\\in\\\{0,1\\\}^\{G\\times n\}
2:

L←2​λmax​\(K\)L\\leftarrow 2\\,\\lambda\_\{\\max\}\(K\)⊳\\trianglerightLipschitz constant; batched power iteration

3:

a←μ/∑μ,y←a,t←1a\\leftarrow\\mu/\\\!\\sum\\mu,\\;y\\leftarrow a,\\;t\\leftarrow 1
4:for

k=1,…,Tk=1,\\dots,Tdo

5:

g←2​K​y−diag​\(K\)g\\leftarrow 2Ky\-\\mathrm\{diag\}\(K\)⊳\\trianglerightbatched gradient

6:

a′←Π△C​\(y−g/L,μ\)a^\{\\prime\}\\leftarrow\\Pi\_\{\\triangle\_\{C\}\}\\\!\\big\(y\-g/L,\\,\\mu\\big\)⊳\\trianglerightcapped\-simplex projection \(bisection\)

7:

t′←12​\(1\+1\+4​t2\),y←a′\+t−1t′​\(a′−a\)t^\{\\prime\}\\leftarrow\\tfrac\{1\}\{2\}\\big\(1\+\\sqrt\{1\+4t^\{2\}\}\\big\),\\quad y\\leftarrow a^\{\\prime\}\+\\tfrac\{t\-1\}\{t^\{\\prime\}\}\(a^\{\\prime\}\-a\)
8:

a←a′,t←t′a\\leftarrow a^\{\\prime\},\\;t\\leftarrow t^\{\\prime\}
9:endfor

10:derive estimated masks

\(S,E\)\(S,E\)from

aa
11:

M~←M\+10−3​diag​\(0,𝟏S\)\\widetilde\{M\}\\leftarrow M\+10^\{\-3\}\\,\\mathrm\{diag\}\(0,\\mathbf\{1\}\_\{S\}\)
12:

u←torch\.linalg\.solve​\(M~,c\)u\\leftarrow\\texttt\{torch\.linalg\.solve\}\(\\widetilde\{M\},c\)⊳\\trianglerightautograd differentiates the ridged solve

13:returnstabilized

α~\\widetilde\{\\alpha\}

## Appendix CExperimental details

#### Models and training\.

The backbone is a pre\-norm decoder with token and learned positional embeddings, a weight\-tied output head, and a pluggable attention module so every variant shares an identical backbone\. The3\.223\.22M configuration isdmodel=256d\_\{\\text\{model\}\}\{=\}256,44layers,44heads, block256256; the1010M configuration isdmodel=384d\_\{\\text\{model\}\}\{=\}384,66layers,66heads, block512512\. All variants train with AdamW \(β=\(0\.9,0\.95\)\\beta=\(0\.9,0\.95\), weight decay0\.10\.1\), gradient clip1\.01\.0, a cosine schedule, batch1616, on enwik8 bytes \(vocabulary256256\); we report the best validation BPC over training\. The primitive solver hyperparameter is the boxCC\. The language\-model runs setC=0\.25C=0\.25directly\. In studies that reportν\\nu, we calibrateC:=1/\(ν​n0\)C:=1/\(\\nu n\_\{0\}\)once at the declared admission/reference sizen0n\_\{0\}, record that scalar, and do not recompute it as the context size changes\. Because the constrained problem requiresn​C≥1nC\\geq 1, the causal implementation uses an ungated warm\-up below⌈1/C⌉\+2\\lceil 1/C\\rceil\+2available keys and issues no SV certificate there\. The gate uses an RBF kernel, chunk length6464, and8080FISTA iterations in the batched solver\.

All selection and exact\-forgetting certificates use the maintained CPU solver in fp64 with fixedCC\. The language\-model training and throughput paths use the single\-precision batched partition finder followed by the10−310^\{\-3\}\-ridgedtorch\.linalg\.solve/autograd path; their coefficients are a stabilized training approximation\. When a certificate is reported on a trained model, its frozen learned keys are re\-solved by the fp64 maintained solver\. Clinical experiments use MIMIC\-IV under its credentialed data use agreement\.

#### Throughput protocol\.

We runpython \-m experiments\.g1\_throughput \-\-paperon an Apple M3 Ultra \(32 physical CPU cores, 512 GiB unified memory\), macOS 26\.5, Python 3\.11\.14, PyTorch 2\.12\.1, NumPy 2\.4\.6, and MLX 0\.31\.2\. A timed step iszero\_gradplus full\-model forward, cross\-entropy, and backward; it excludes the optimizer update, data loading, evaluation, and checkpointing\. Direct rows use batch1616and report the median and interquartile range of seven separately synchronized samples after three warm\-ups\. The mixed SV row includes PyTorch CPU work, host/device transfers, MLX\-GPU FISTA, the ridged PyTorch KKT solve, and backward\. The two exact active\-set diagnostics implement the RBF readout, are measured at batch22, and are linearly extrapolated by8×8\\times; they are retained as historical diagnostics rather than like\-for\-like hybrid comparisons\. Softmax is the repository’s unfused masked implementation, not FlashAttention\.

#### Matched\-state protocol\.

For each seed we train the SV hybrid, measure its mean active state\|S∪E\|\|S\\cup E\|on held\-out data, and set the sliding\-window Transformer’s window to that state plus the local\-window length, so the two layers see a matched token budget\. The recurrent baselines \(DeltaNet, linear attention\) carry their own fixed state and are reported at it; full softmax is the unbounded reference\. The seven\-seed paired comparison is only SV versus matched swa; the DeltaNet, linear, and full\-softmax rows are single\-seed references\. Every variant uses the identical backbone, optimizer, schedule, and training token budget\.

#### Exact\-forgetting protocol\.

For each trial we build a context, calibrate or setCConce, solve the gate in fp64, and forget a token two ways: \(i\) the Cauwenberghs–Poggio decremental reverse path, and \(ii\) a from\-scratch refit on the retained tokens with that sameCC\(the “retrain\-without” reference\)\. This is required by the same\-algorithm definition: recomputingCCfrom the smaller retained\-set size would change a primitive hyperparameter\. We compare their decision functionsf​\(x\)=2​∑iαi​κ​\(x,xi\)−ρf\(x\)=2\\sum\_\{i\}\\alpha\_\{i\}\\kappa\(x,x\_\{i\}\)\-\\rhoon probes spanning the retained, forgotten, and jittered points, and report the maximum deviation\. We use the decision function rather than the raw coefficients because on exact duplicates the SVDD optimum is non\-unique \(α\\alphaand the partition may differ while the function is identical\), so the function is the correct equivalence test\. The decay baseline scales the forgotten token’s coefficient byγ\\gammaand is evaluated on the same metric\. Regimes: Gaussian keys \(d=6d\{=\}6\), redundant near\-duplicate clusters, real MIMIC\-IV hourly vitals \(six Harutyunyan\-style core vitals\(Harutyunyanet al\.,[2019](https://arxiv.org/html/2607.12204#bib.bib26)\), deduplicated within0\.10\.1standardized units\), and a trained recall model’s learned key projections;300300trials each\.

## Appendix DPer\-seed matched\-state results

Table[10](https://arxiv.org/html/2607.12204#A4.T10)gives the per\-seed bits\-per\-character for the SV hybrid and the matched\-state sliding\-window Transformer \(swa\) at3\.223\.22M parameters, the basis for the significance test in Section[8](https://arxiv.org/html/2607.12204#S8)\. Every one of the seven seeds favors the hybrid\. Mean BPC is2\.1782\.178for SV and2\.3832\.383for swa; the mean paired relative improvement is8\.6%8\.6\\%\(95%95\\%CI\[5\.2,12\.0\]%\[5\.2,12\.0\]\\%\), with a pairedtt\-testt=5\.95t=5\.95\(p=0\.001p=0\.001\), a distribution\-free Wilcoxon signed\-rankp=0\.016p=0\.016, and Cohendz=2\.25d\_\{z\}=2\.25\.

Table 10:Per\-seed matched\-state language modeling \(enwik8 BPC;3\.223\.22M params\)\. Each seed sets the model initialization, data order, and matched window; the hybrid wins on all seven\. The final column gives mean BPC for the model rows and mean paired relative improvement for the final row\.On the contemporary TinyStories corpus\(Eldan and Li,[2023](https://arxiv.org/html/2607.12204#bib.bib30)\)\(also3\.223\.22M, byte\-level\), the hybrid wins directionally on all three seeds: BPC1\.329/1\.185/1\.2691\.329/1\.185/1\.269versus matched swa1\.434/1\.451/1\.5261\.434/1\.451/1\.526, a mean improvement of14\.2%14\.2\\%\(pairedt=4\.0t=4\.0,p=0\.057p=0\.057atn=3n=3\)\. This is a small directional replication on a second corpus, not statistical confirmation\.

#### Larger scales\.

At1010M parameters \(block512512, five seeds\) the matched\-state comparison does not favor the hybrid: per\-seed SV vs\. swa BPC is2\.406/2\.5022\.406/2\.502,2\.249/2\.6612\.249/2\.661,2\.415/2\.4262\.415/2\.426,2\.475/2\.2862\.475/2\.286,3\.127/2\.5243\.127/2\.524, a mean of\+2\.5%\+2\.5\\%\(SV worse\) with a standard deviation of∼15%\{\\sim\}15\\%\(not significant\)\. The SV runs are themselves high\-variance: the two worst seeds had not converged within the6,0006\{,\}000\-step budget \(their loss was still descending steeply\), so the comparison is partly a fixed\-budget artifact rather than a clean quality verdict\. A single3232M run tells the same story: the SV\-hybrid reached only4\.4564\.456BPC and was still descending at step6,0006\{,\}000, versus a converged swa at3\.1063\.106\. The chunk\-frozen gate optimizes slower and less stably at scale, and we scope the language\-model edge to the small\-scale regime accordingly \(Section[10](https://arxiv.org/html/2607.12204#S10)\)\.