Kernel-based Interpolatory Methods for Context Compression of Attention Mechanisms

# Kernel-based Interpolatory Methods for Context Compression of Attention Mechanisms
Source: [https://arxiv.org/html/2610.40943](https://arxiv.org/html/2610.40943)
Michel Fabrice SerretAffiliation:Center for Scientific Computing, Theory and Data, Paul Scherrer Institute, Switzerland\.Affiliation:Corresponding Author \([michel\.serret@psi\.ch](mailto:[email protected])\)Laura GrigoriAffiliation:Center for Scientific Computing, Theory and Data, Paul Scherrer Institute, Switzerland\.Affiliation:Institute of Mathematics, EPFL, Switzerland

October 29, 2026

###### Abstract

The bottleneck of attention mechanisms, which are the fundamental building block of modern language models, is their quadratic complexity in terms of context size\. There has thus been an increasing interest in the compression of the context, also called KV\-cache compression\. In this article, we consider the problem of compressing the context of attention mechanisms by first formulating it in the Hilbert space associated with its kernel\. In this formalism, attention can be represented as the ratio of two linear maps whose interpolant is also the ratio of interpolants, this allows us to consider the interpolation of the linear maps directly\. We thus introduce two novel KV\-cache compression methods which we call GAIM and Q\-DAIM, based on the Generalized Empirical Interpolation Method \(GEIM\) and the Q\-Discrete Empirical Interpolation Method \(Q\-DEIM\), respectively\. As the cost of these methods is quadratic in terms of context length, we propose surrogate models based on the Taylor series, the Performer and the Random Fourier Features method which allow the compression to be done in linear time\. Finally, we extend this result to the streaming attention setting, providing new methods which allow one to penalize the selection of interpolation nodes based on the previously interpolated context and to update the previous interpolation based on new information\. We then provide numerical results on the convergence of the residuals and the output error in future queries and preliminary results on the LongBench\-E benchmark showing performance comparable to the state\-of\-the\-art methods\. These experiments suggest that the range revealed by our Q\-DAIM method could help identify where information is concentrated in the KV cache, thus enabling an efficient compression\.

## 1Introduction

The attention mechanism, one of the main building blocks of the Transformer architecture\[[Vas\+23](https://arxiv.org/html/2610.40943#bib.bibx50)\], is based on functions that involve pairwise interactions of its inputs\. Indeed, attention mechanisms are used to construct functions that can change depending on the context111For a more detailed description of attention in the context of LLMs, see\[[Ser26](https://arxiv.org/html/2610.40943#bib.bibx40)\]\., given by their keyK∈ℝN×dK\\in\\mathbb\{R\}^\{N\\times d\}and value matricesV∈ℝN×dV\\in\\mathbb\{R\}^\{N\\times d\}withN∈ℕ∗N\\in\\mathbb\{N\}^\{\*\}\. They are applied to an input known as a query vectorq∈ℝdq\\in\\mathbb\{R\}^\{d\}\. In the context of Large Language Models, this context corresponds to a set ofNNtokens, sets of characters, and their associated query, key, and value vectors\. The attention mechanism allows the models to encode the semantic information contained in the relationships of the input token with the context tokens\. To do so, the rows of the key matrices,\(Ki\)i∈\[N\]\(K\_\{i\}\)\_\{i\\in\[N\]\}, are used to calculate a similarity between the input token’s query vector and each contextual key, via a kernel functionκexp:x,y↦exp⁡\(⟨x,y⟩/d\)\\kappa\_\{\\exp\}:x,y\\mapsto\\exp\\big\(\\langle x,y\\rangle/\\sqrt\{d\}\\big\)\. This distribution of similarities over the keys is then normalised resulting in the normalised attention scores given by the softmax function

softmax⁡\(q,K\)=\(exp⁡\(⟨q,Ki⟩/d\)∑j∈\[N\]exp⁡\(⟨q,Kj⟩/d\)\)i∈\[N\]∈ℝN\.\\operatorname\{softmax\}\(q,K\)=\\left\(\\frac\{\\exp\(\\langle q,K\_\{i\}\\rangle/\\sqrt\{d\}\)\}\{\\sum\_\{j\\in\[N\]\}\\exp\(\\langle q,K\_\{j\}\\rangle/\\sqrt\{d\}\)\}\\right\)\_\{i\\in\[N\]\}\\in\\mathbb\{R\}^\{N\}\.For each context tokeni∈\[N\]i\\in\[N\], the associated value rowViV\_\{i\}is weighted by the attention score associated with its key\. The attention on a single queryq∈ℝdq\\in\\mathbb\{R\}^\{d\}is then given by

Att⁡\(q,K,V\)=softmax⁡\(q,K\)𝖳​V=∑i∈\[N\]κexp​\(q,Ki\)​Vi∑j∈\[N\]κexp​\(q,Kj\)∈ℝd,\\operatorname\{Att\}\(q,K,V\)=\\operatorname\{softmax\}\(q,K\)^\{\\mathsf\{T\}\}V=\\sum\_\{i\\in\[N\]\}\\frac\{\\kappa\_\{\\exp\}\(q,K\_\{i\}\)V\_\{i\}\}\{\\sum\_\{j\\in\[N\]\}\\kappa\_\{\\exp\}\(q,K\_\{j\}\)\}\\in\\mathbb\{R\}^\{d\},\(1\)whereκexp\\kappa\_\{\\exp\}is the exponential kernel of the scaled dot\-product\.

However, computing the attention has a quadratic cost in the sequence lengthNNand, at inference time, requires caching the key and value embeddings of all past tokens\. Attention is hence the main computational and memory bottleneck of modern Large Language Models \(LLMs\)\. Implementations such as FlashAttention\[[Dao\+22](https://arxiv.org/html/2610.40943#bib.bibx9),[Dao23](https://arxiv.org/html/2610.40943#bib.bibx10)\]compute attention exactly while reducing memory traffic; however, it still has a quadratic scaling\. Ever since the rise of LLMs in the last few years, there has been heightened interest in the development of sub\-quadratic approximations of the attention mechanism\. The methods proposed come from several fields of research, ranging from clustering and sparsity\[[KKL20](https://arxiv.org/html/2610.40943#bib.bibx26),[Zan\+23](https://arxiv.org/html/2610.40943#bib.bibx57),[Han\+23](https://arxiv.org/html/2610.40943#bib.bibx21),[Hoo\+25](https://arxiv.org/html/2610.40943#bib.bibx25)\]to low\-rank and subspace projection\-based methods\[[Wan\+20](https://arxiv.org/html/2610.40943#bib.bibx51),[Sin\+24](https://arxiv.org/html/2610.40943#bib.bibx43)\]\. Recent works also used sketching and tensor decompositions\[[PP25](https://arxiv.org/html/2610.40943#bib.bibx35)\]as well as kernel\-based approximations, which replace the exponential kernel of attention with finite\-dimensional features\[[Cho\+22](https://arxiv.org/html/2610.40943#bib.bibx6),[KMZ24](https://arxiv.org/html/2610.40943#bib.bibx28)\]or by Nyström approximations\[[Che\+21](https://arxiv.org/html/2610.40943#bib.bibx4),[LLS24](https://arxiv.org/html/2610.40943#bib.bibx31),[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]of the attention scores\. A classification of these methods through the lens of numerical analysis can be found in\[[Ser\+26](https://arxiv.org/html/2610.40943#bib.bibx39)\]\. Other methods attempt to compress the KV cache of a pretrained model directly, removing tokens from the KV cache depending on their associated attention\-score statistics, discrepancy measures, or following heuristics based on the norm of the key and value vectors\[[Zha\+23](https://arxiv.org/html/2610.40943#bib.bibx59),[Li\+24](https://arxiv.org/html/2610.40943#bib.bibx30),[Xia\+24](https://arxiv.org/html/2610.40943#bib.bibx55),[Yan\+24](https://arxiv.org/html/2610.40943#bib.bibx56),[Han\+25a](https://arxiv.org/html/2610.40943#bib.bibx23),[Che\+26](https://arxiv.org/html/2610.40943#bib.bibx5)\]\. The observation at the basis of our method is shared with the WildCat method\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\], attention can be written as the ratio of two linear maps, and the orthogonal projections of these maps onto a well\-chosen subspace provide an approximation of the attention mechanism\. Indeed, formulated in the Reproducing Kernel Hilbert Space \(RKHS\),HH, of the exponential kernelκexp\\kappa\_\{\\exp\}underlying scaled dot\-product attention, attention becomes a ratio of linear maps in the query featureϕ⁡\[q\]∈H\\phi\[q\]\\in H,

Att⁡\(q,K,V\)=ϕ​\[q\]𝖳​Kϕ𝖳​Vϕ​\[q\]𝖳​Kϕ𝖳​𝟏N,\\operatorname\{Att\}\(q,K,V\)=\\frac\{\\phi\[q\]^\{\\mathsf\{T\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}V\}\{\\phi\[q\]^\{\\mathsf\{T\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}\\mathbf\{1\}\_\{N\}\},whereKϕ:H→ℝNK\_\{\\phi\}:H\\to\\mathbb\{R\}^\{N\}denotes the linear map whose rows are the featuresϕ⁡\[Ki\]\\phi\[K\_\{i\}\],i∈\[N\]i\\in\[N\]\. The numerator is given by the feature\-value operatorKϕ𝖳​VK\_\{\\phi\}^\{\\mathsf\{T\}\}Vand the denominator by the feature\-normalization operatorKϕ𝖳​𝟏NK\_\{\\phi\}^\{\\mathsf\{T\}\}\\mathbf\{1\}\_\{N\}\. As they share their left factorKϕ𝖳K\_\{\\phi\}^\{\\mathsf\{T\}\}, both are generated by the same key features and have ranges in the sameNN\-dimensional subspaceSpan⁡\(Kϕ\)\\operatorname\{Span\}\(K\_\{\\phi\}\)ofHH\. They can thus be approximated jointly through the augmented operatorM:=Kϕ𝖳​VaugM:=K\_\{\\phi\}^\{\\mathsf\{T\}\}V^\{\\mathrm\{aug\}\}, withVaug:=\[V∣𝟏N\]∈ℝN×\(d\+1\)V^\{\\mathrm\{aug\}\}:=\[V\\mid\\mathbf\{1\}\_\{N\}\]\\in\\mathbb\{R\}^\{N\\times\(d\+1\)\}the augmented value matrix\.

Given a matrixL∈ℝn×dL\\in\\mathbb\{R\}^\{n\\times d\}ofn<Nn<Nkeys, letΠLϕ\\Pi\_\{L\_\{\\phi\}\}be the orthogonal projector ontoSpan⁡\(Lϕ\):=\{∑jαj​ϕ​\[Lj\]∈H\|αj∈ℝ\}\\operatorname\{Span\}\(L\_\{\\phi\}\):=\\big\\\{\\sum\_\{j\}\\alpha\_\{j\}\\phi\[L\_\{j\}\]\\in H\\,\\big\|\\,\\alpha\_\{j\}\\in\\mathbb\{R\}\\big\\\}and letWaug:=\[W∣WNorm\]∈ℝn×\(d\+1\)W^\{\\mathrm\{aug\}\}:=\[W\\mid W\_\{\\mathrm\{Norm\}\}\]\\in\\mathbb\{R\}^\{n\\times\(d\+1\)\}be the weights such thatLϕ𝖳​Waug=ΠLϕ​ML\_\{\\phi\}^\{\\mathsf\{T\}\}W^\{\\mathrm\{aug\}\}=\\Pi\_\{L\_\{\\phi\}\}M\. Projecting both linear maps yields the approximation

Att~L​\(q,K,V\)=ϕ​\[q\]𝖳​ΠLϕ​Kϕ𝖳​Vϕ​\[q\]𝖳​ΠLϕ​Kϕ𝖳​𝟏N=ϕ​\[q\]𝖳​Lϕ𝖳​Wϕ​\[q\]𝖳​Lϕ𝖳​WNorm,\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(q,K,V\)=\\frac\{\\phi\[q\]^\{\\mathsf\{T\}\}\\Pi\_\{L\_\{\\phi\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}V\}\{\\phi\[q\]^\{\\mathsf\{T\}\}\\Pi\_\{L\_\{\\phi\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}\\mathbf\{1\}\_\{N\}\}=\\frac\{\\phi\[q\]^\{\\mathsf\{T\}\}L\_\{\\phi\}^\{\\mathsf\{T\}\}W\}\{\\phi\[q\]^\{\\mathsf\{T\}\}L\_\{\\phi\}^\{\\mathsf\{T\}\}W\_\{\\mathrm\{Norm\}\}\},which WildCat\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]interprets as a Nyström\-based approximation of the attention mechanism\.

In this article, we adopt instead the point of view of interpolation\. The rows ofLLare the*interpolation nodes*, which play the role of the magic points of the generalized empirical interpolation method \(GEIM\)\[[MM13](https://arxiv.org/html/2610.40943#bib.bibx33)\],ℬ:=Span⁡\(Lϕ\)\\mathcal\{B\}:=\\operatorname\{Span\}\(L\_\{\\phi\}\)is the basis space onto which we interpolate, andWaugW^\{\\mathrm\{aug\}\}is the projection ofMMontoℬ\\mathcal\{B\}, expressed in the node features\. The pair\(L,Waug\)\(L,W^\{\\mathrm\{aug\}\}\)then replaces\(K,Vaug\)\(K,V^\{\\mathrm\{aug\}\}\)as the compressed KV cache\. SinceKϕ𝖳​VK\_\{\\phi\}^\{\\mathsf\{T\}\}VandKϕ𝖳​𝟏NK\_\{\\phi\}^\{\\mathsf\{T\}\}\\mathbf\{1\}\_\{N\}are interpolated on the same basis space, the ratio of their interpolants is the interpolant of their ratio: it reproduces attention exactly whenever the query is one of the interpolation nodes\. The compression of attention thus divides into two tasks: the*selection*of nodesLL, which determinesℬ\\mathcal\{B\}, and the*projection*ofMMontoℬ\\mathcal\{B\}, which determinesWaugW^\{\\mathrm\{aug\}\}\. The selection phase, described in §[3\.1\.2](https://arxiv.org/html/2610.40943#S3.SS1.SSS2)and §[3\.1\.3](https://arxiv.org/html/2610.40943#S3.SS1.SSS3), uses the information contained in the keys and the values, throughMM, to choose the interpolation nodesLLthat minimize a given residual norm\. The natural residual is the worst\-case attention error in thepp\-norm,ℛp​\(L\)=maxq∈ℝd⁡‖Att⁡\(q,K,V\)−Att~L​\(q,K,V\)‖p\\mathcal\{R\}\_\{p\}\(L\)=\\max\_\{q\\in\\mathbb\{R\}^\{d\}\}\\\|\\operatorname\{Att\}\(q,K,V\)\-\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(q,K,V\)\\\|\_\{p\}\. However, this residual is nonlinear and very sensitive to small errors in the denominator\. We therefore consider five variants of the attention residual, based on different approximations ofℛp\\mathcal\{R\}\_\{p\}\. The projection phase, described in §[3\.2](https://arxiv.org/html/2610.40943#S3.SS2), is formalized as a Petrov\-Galerkin problem: it computes the weightsWaugW^\{\\mathrm\{aug\}\}for whichLϕ𝖳​WaugL\_\{\\phi\}^\{\\mathsf\{T\}\}W^\{\\mathrm\{aug\}\}best approximatesMMon a given test space𝒯\\mathcal\{T\}, the orthogonal projection above being the Galerkin case𝒯=ℬ\\mathcal\{T\}=\\mathcal\{B\}\.

However, classical empirical interpolation methods such as GEIM\[[MM13](https://arxiv.org/html/2610.40943#bib.bibx33)\]or Q\-DEIM\[[DG16](https://arxiv.org/html/2610.40943#bib.bibx13),[DS18](https://arxiv.org/html/2610.40943#bib.bibx15)\]cannot be applied directly to this problem for three reasons\. First, due to the infinite\-dimensional nature of the RKHS, operators such asMMand features such as the rows ofKϕK\_\{\\phi\}cannot be constructed explicitly, and hence the usual pivot\-based matrix factorizations cannot, in general, be applied\. We have to resort instead to methods relying on kernel evaluations rather than on the elements ofHH\. Second, the kernel norms of the keys,‖ϕ⁡\[Ki\]‖H=exp⁡\(‖Ki‖2/\(2​d\)\)\\\|\\phi\[K\_\{i\}\]\\\|\_\{H\}=\\exp\\big\(\\\|K\_\{i\}\\\|^\{2\}/\(2\\sqrt\{d\}\)\\big\), span several orders of magnitude, so that the exponential kernel Gram matrices involved in the projection are ill\-conditioned\. Third, evaluating the exact residual and its updates through kernel evaluations has a cost quadratic in the number of keys, which defeats the purpose of compression\.

Our main contribution is to introduce two KV\-cache compression methods for attention interpolation that overcome these obstacles, being numerically stable and scaling linearly with the context sizeNN:

- •GAIM, a greedy selection in the style of GEIM\[[MM13](https://arxiv.org/html/2610.40943#bib.bibx33),[Mad\+07](https://arxiv.org/html/2610.40943#bib.bibx32)\], which greedily picks the interpolation nodes minimizing theL∞L^\{\\infty\}\-norm of the chosen residual, by selecting at each step the node with the largest current residual\. It evaluates the augmented residual operatorMΔ:=M−ΠLϕ​M=ΠLϕ⟂​MM\_\{\\Delta\}:=M\-\\Pi\_\{L\_\{\\phi\}\}M=\\Pi\_\{L\_\{\\phi\}\}^\{\\perp\}Mon the key features,Kϕ​MΔK\_\{\\phi\}M\_\{\\Delta\}and selects the row with the largestL∞L^\{\\infty\}\-norm as the next interpolation node and updates the residual through a rank\-one update\.
- •Q\-DAIM, a Q\-DEIM\-based selection\[[DG16](https://arxiv.org/html/2610.40943#bib.bibx13),[DS18](https://arxiv.org/html/2610.40943#bib.bibx15)\], which minimizes theL2L^\{2\}\-norm of the residual in the metric induced by the map using the orthogonal basis obtained through a CholeskyQR / Cholesky whitening of the augmented operator\. It relies on a QR\-like factorizationM=U​RM=UR, whereU:ℝr→HU:\\mathbb\{R\}^\{r\}\\to His an orthonormal basis inHHof the range ofMM, of rankr≤d\+1r\\leq d\+1, andR∈ℝr×\(d\+1\)R\\in\\mathbb\{R\}^\{r\\times\(d\+1\)\}\. AsUUcannot be constructed explicitly, we instead compute a Cholesky factorization of its Gram matrix,G:=M𝖳​M=R𝖳​R∈ℝ\(d\+1\)×\(d\+1\)G:=M^\{\\mathsf\{T\}\}M=R^\{\\mathsf\{T\}\}R\\in\\mathbb\{R\}^\{\(d\+1\)\\times\(d\+1\)\}which only requires kernel evaluations, to obtainRR\. The orthogonal operator onto the range ofMMis then given byU=M​R†=Kϕ𝖳​Vaug​R†U=MR^\{\\dagger\}=K\_\{\\phi\}^\{\\mathsf\{T\}\}V^\{\\mathrm\{aug\}\}R^\{\\dagger\}\. By evaluatingUUon the keys,F:=Kϕ​U∈ℝN×rF:=K\_\{\\phi\}U\\in\\mathbb\{R\}^\{N\\times r\}, we obtain a matrix whose rows are the coordinates of the key features in the revealed range, then we select the nodes to be the pivots of a column\-pivoted QR factorization ofF𝖳F^\{\\mathsf\{T\}\}, which greedily maximizes theL2L^\{2\}norm of the reconstructed basis of the revealed range\. Additionally, the Q\-DEIM interpolation operator then provides us with an alternative Petrov\-Galerkin projection whose test space is given by the revealed range ofMM\.

In practice, both selections are carried out in the Gaussian reformulation described below\. Both methods are designed for streaming attention, where the interpolation is applied on successive segments of the context as they come in, and where each interpolation can be improved by using information from previously interpolated segments\. In §[4](https://arxiv.org/html/2610.40943#S4), we decompose the interpolation over a stream into a four\-stage process\. An orthogonalization of the residual against a basis constructed from previously selected interpolation nodes, which we call deflation for brevity, the selection of new interpolation nodes, the projection of the values, and an optional update of the previously interpolated weights\.

The contributions that make these methods numerically stable and computationally feasible are threefold\.

1. \(i\.\)A Gaussian\-kernel and scaled key norm reformulation\.§[2](https://arxiv.org/html/2610.40943#S2)develops attention interpolation in the RKHS of the exponential kernel\. To address the ill\-conditioning of this formulation, we provide an exact Gaussian reformulation of the attention mechanism and highlight several of its invariants, namely query norm scaling, global key norm scaling, and global key translation\. Combined, they separate the RKHS interpolation problem into two parts, a well\-conditioned linear problem in translation\-invariant Gaussian features and a kernel norm rescaling, which stabilizes our methods\.
2. \(ii\.\)Kernel\-based selection and Petrov\-Galerkin projections\.Splitting the compression into selection and projection allows both phases to be carried out through kernel evaluations only\. For the projection, we provide several alternative Petrov\-Galerkin methods, which differ by their test space𝒯\\mathcal\{T\}, and hence by the metric, based on the interpolation nodesLL, in whichLϕ𝖳​WaugL\_\{\\phi\}^\{\\mathsf\{T\}\}W^\{\\mathrm\{aug\}\}approximatesMM\.
3. \(iii\.\)Finite\-dimensional surrogate residuals\.The exact residual and its update are costly,O⁡\(Nseg2​d\)O\(N\_\{\\mathrm\{seg\}\}^\{2\}d\), to evaluate directly, whereNsegN\_\{\\mathrm\{seg\}\}denotes the size of a segment\. We therefore introduce finite\-dimensional*surrogate*residuals based on finite\-dimensional approximations of the exponential kernel’s feature map\. We namely consider feature maps constructed using the first\-order Taylor approximation, Random Fourier Features\[[RR07](https://arxiv.org/html/2610.40943#bib.bibx38)\]or from the Performer FAVOR\+ features\[[Cho\+22](https://arxiv.org/html/2610.40943#bib.bibx6)\]\. Due to the finite\-dimensional nature of these maps, key features can be represented as matrices and this allows for a change in the order of operations that reduces the cost of selection and residual updates toO⁡\(Nseg​d2\)O\(N\_\{\\mathrm\{seg\}\}d^\{2\}\)\. These surrogate residuals are used as alternative selection criterion for our magic points\. However, to limit the approximation error, the projection step is still obtained through an exact Petrov\-Galerkin projection of the operator on the chosen nodes, which costs on the order ofO⁡\(Nseg​d2\+d3\)O\(N\_\{\\mathrm\{seg\}\}d^\{2\}\+d^\{3\}\)for a single segment of sizeNsegN\_\{\\mathrm\{seg\}\}with a number of interpolation nodes proportional todd,nseg=O⁡\(d\)n\_\{\\mathrm\{seg\}\}=O\(d\)\.

We provide a complexity analysis for segments of sizeNseg∈ℕN\_\{\\mathrm\{seg\}\}\\in\\mathbb\{N\}withnsegn\_\{\\mathrm\{seg\}\}interpolation nodes per segment of our streaming interpolation methods in §[4\.3](https://arxiv.org/html/2610.40943#S4.SS3)\. We also provide a numerical study of our proposed interpolation methods in §[5](https://arxiv.org/html/2610.40943#S5)on Qwen2\.5\-7B\-Instruct\[[Qwe\+25](https://arxiv.org/html/2610.40943#bib.bibx37)\], in terms of the convergence of the residuals, of the attention error on future queries of the same text, and of the performance on the LongBench\-E benchmark\[[Bai\+24](https://arxiv.org/html/2610.40943#bib.bibx2)\]\. Overall, in a preliminary LongBench\-E evaluation at25%25\\%context compression, Q\-DAIM obtains the best average score of all the methods we evaluated, retaining87\.1%87\.1\\%of the score of the uncompressed model and slightly outperforming both GAIM and the RP\-Cholesky selection of WildCat\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]implemented in the same framework\. For this benchmark, we computed several variants of the model to isolate the impact of the different parameters of the methods on the result\. We observe that a lower attention error does not necessarily translate into better benchmark scores, the selection of the interpolation nodes having a more decisive impact on performance than the projection\. Interestingly, it appears that the revealed range of Q\-DAIM provides both better selection and better projection performance than the GAIM or RP\-Cholesky methods\. Indeed, by looking at the variants where no projection is used, i\.e\.Waug=VaugW^\{\\mathrm\{aug\}\}=V^\{\\mathrm\{aug\}\}, Q\-DAIM selection appears to outperform all other methods, and on the other hand the best performing variant on the most tasks for Q\-DAIM is the one using the revealed\-range projection, i\.e\. the Q\-DEIM interpolation operator\. This suggests that the range of the augmented operator revealed by Q\-DAIM is a better indicator of the information retained by a method than the residuals and that the revealed range provides additional information which seems to not be accessible to the other methods\. As the revealed range is only approximately reconstructed in this setting, being limited toddvectors for its reconstruction but requiring the complete set of keys for a perfect reconstruction, this motivates future research for the direct reconstruction ofUU\.

The present work belongs to the kernel\-based family of attention approximation methods and is most closely related to the WildCat method\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\], which also approximates the augmented linear map similarly but whose selection uses a Randomly Pivoted Cholesky \(RP\-Cholesky\) on the Gram matrix of the keys for a rescaled kernel\. WildCat was also recently combined with the Thinformer approximation inside the causal Express wrapper of\[[Gon\+26](https://arxiv.org/html/2610.40943#bib.bibx19)\]\. Compared to the WildCat\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]method, whose random selection is optimal to reconstruct the span of the keys in the Hilbert spaceHH, our method takes into account the value matrix and focuses on minimizing the residual of attention\.

## 2Attention in the Hilbert space of the scaled dot\-product exponential kernel

In this section, we first define the attention mechanism and highlight its formulation in the Reproducing Kernel Hilbert Space setting in §[2\.1](https://arxiv.org/html/2610.40943#S2.SS1), and proceed to define its interpolation on a set of keys in §[2\.2](https://arxiv.org/html/2610.40943#S2.SS2)\. Then we consider the Gaussian reformulation of the linear maps in §[2\.4](https://arxiv.org/html/2610.40943#S2.SS4)and describe the invariance properties of the attention mechanism in §[2\.3](https://arxiv.org/html/2610.40943#S2.SS3)which are the basis of our numerical implementations\. Indeed, they allow us to improve the conditioning of the interpolation problem and provide better numerical stability\.

#### Notation

In the paper, we use the following notation\. We writeℕ\\mathbb\{N\}for the set of non\-negative integers,ℕ∗=ℕ∖\{0\}\\mathbb\{N\}^\{\*\}=\\mathbb\{N\}\\setminus\\\{0\\\}, and\[n\]:=\{1,…,n\}\[n\]:=\\\{1,\\dots,n\\\}forn∈ℕ∗n\\in\\mathbb\{N\}^\{\*\}\. We denote arbitrary vectors usingx∈ℝdx\\in\\mathbb\{R\}^\{d\}andy∈ℝdy\\in\\mathbb\{R\}^\{d\}\. We set⟨⋅,⋅⟩\\langle\\cdot,\\cdot\\rangleand∥⋅∥\\\|\\cdot\\\|to denote theL2L^\{2\}inner product and norm, and𝟏n∈ℝn\\mathbf\{1\}\_\{n\}\\in\\mathbb\{R\}^\{n\}to be the vector of ones for anyn∈ℕ∗n\\in\\mathbb\{N\}^\{\*\}\. For any matrixA∈ℝm×nA\\in\\mathbb\{R\}^\{m\\times n\}, we denote byA𝖳A^\{\\mathsf\{T\}\}its transpose and byAi∈ℝnA\_\{i\}\\in\\mathbb\{R\}^\{n\}the vector corresponding to itsii\-th row, and we also setA†A^\{\\dagger\}to be the pseudo\-inverse\.‖A‖F\\\|A\\\|\_\{F\}denotes the Frobenius norm andTr⁡\(A\)\\operatorname\{Tr\}\(A\)the trace of the matrixAA\. Given a Hilbert spaceHH, we denote by⟨⋅,⋅⟩H\\langle\\cdot,\\cdot\\rangle\_\{H\}and∥⋅∥H\\\|\\cdot\\\|\_\{H\}its associated inner product and norm\. We denote using the notationΠ𝒮\\Pi\_\{\\mathcal\{S\}\}the orthogonal projector onto a closed subspace𝒮⊆H\\mathcal\{S\}\\subseteq H, withΠ𝒮⟂:=Id−Π𝒮\\Pi\_\{\\mathcal\{S\}\}^\{\\perp\}:=\\mathrm\{Id\}\-\\Pi\_\{\\mathcal\{S\}\}\. We also use by abuse of notationΠΨ\\Pi\_\{\\Psi\}for a linear mapΨ:H→ℝn\\Psi:H\\to\\mathbb\{R\}^\{n\}to denote the orthogonal projector onto its row\-spaceSpan⁡\(Ψ\)\\operatorname\{Span\}\(\\Psi\)\. For any feature mapϕ:ℝd→H\\phi:\\mathbb\{R\}^\{d\}\\to H, we denote for anyx∈ℝdx\\in\\mathbb\{R\}^\{d\}the mapped feature asϕ⁡\[x\]∈H\\phi\[x\]\\in H\. Occasionally, for clarity, we use Python notation to identify the matrix rows, columns and elements:A⁡\[i,j\]A\[i,j\]is used to denote the entry in rowiiand columnjj,A\[i,:\]A\[i,:\]denotes theii\-th row andA\[:,j\]A\[:,j\]thejj\-th column\. More generally, given index setsI⊆\[m\]I\\subseteq\[m\]andJ⊆\[n\]J\\subseteq\[n\], we setA⁡\[I,J\]A\[I,J\]to be the submatrix ofAAcomposed of the rows indexed byIIand columns indexed byJJ\. We also set, when used in python notation,i:j=\{i,…,j\}i:j=\\\{i,\\dots,j\\\}, for anyi,j∈ℕ∗i,j\\in\\mathbb\{N\}^\{\*\}withi≤ji\\leq j\. For anyx∈ℝdx\\in\\mathbb\{R\}^\{d\}, we setDiag⁡\(x\)∈ℝd×d\\operatorname\{Diag\}\(x\)\\in\\mathbb\{R\}^\{d\\times d\}to be the diagonal matrix whose diagonal is given by the elements ofxx\. On the other hand, when used with a matrixA∈ℝn×nA\\in\\mathbb\{R\}^\{n\\times n\}, it corresponds to the diagonal of the matrix inDiag⁡\(A\)=\(A⁡\[i,i\]\)i∈\[n\]∈ℝn\\operatorname\{Diag\}\(A\)=\(A\[i,i\]\)\_\{i\\in\[n\]\}\\in\\mathbb\{R\}^\{n\}\. The Hadamard, or elementwise, product is denoted by⊙\\odot\. Theexp\\expandlog\\logfunctions are considered to be applied elementwise to vectors and matrices\.

### 2\.1Attention as a ratio of two linear maps

As described in the context of the Nyström approximation used in WildCat method\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\], the above formulation of attention \([1](https://arxiv.org/html/2610.40943#S1.E1)\) can be rewritten as the ratio of two linear maps through the kernel functionκexp\\kappa\_\{\\exp\}\. In our case, it is more useful to describe attention as a ratio of linear maps in the Reproducing Kernel Hilbert Space,HH, ofκexp\\kappa\_\{\\exp\}by its explicit feature map,ϕ:ℝd→H\\phi:\\mathbb\{R\}^\{d\}\\to H\. Indeed, letH1:=ℝdH\_\{1\}:=\\mathbb\{R\}^\{d\}, the RKHS associated toκexp\\kappa\_\{\\exp\}is given byH=⨁r∈ℕSym⁡\(H1⊗r\)H=\\bigoplus\_\{r\\in\\mathbb\{N\}\}\\mathrm\{Sym\}\(H\_\{1\}^\{\\otimes r\}\), the direct sum of the Hilbert spaces of symmetric tensors\[[Com\+08](https://arxiv.org/html/2610.40943#bib.bibx8)\]of powers ofH1H\_\{1\}222With the convention thatH1⊗0:=ℝH\_\{1\}^\{\\otimes 0\}:=\\mathbb\{R\}andx⊗0=1x^\{\\otimes 0\}=1\. Its associated feature map,ϕ:ℝd→H\\phi:\\mathbb\{R\}^\{d\}\\to H, relies on the Taylor expansion, given, for anyx,y∈ℝdx,y\\in\\mathbb\{R\}^\{d\}, as

⟨ϕ⁡\[x\],ϕ⁡\[y\]⟩H=∑r∈ℕ⟨x,y⟩rr\!​dr=∑r∈ℕ⟨x⊗r,y⊗r⟩r\!​dr\.\\langle\\phi\[x\],\\phi\[y\]\\rangle\_\{H\}=\\sum\_\{r\\in\\mathbb\{N\}\}\\frac\{\\langle x,y\\rangle^\{r\}\}\{r\!\\sqrt\{d^\{r\}\}\}=\\sum\_\{r\\in\\mathbb\{N\}\}\\frac\{\\langle x^\{\\otimes r\},y^\{\\otimes r\}\\rangle\}\{r\!\\sqrt\{d^\{r\}\}\}\.\(2\)For anyr∈ℕr\\in\\mathbb\{N\}and anyx∈ℝdx\\in\\mathbb\{R\}^\{d\}, we setϕr​\[x\]:=x⊗rr\!​dr/2\\phi\_\{r\}\[x\]:=\\frac\{x^\{\\otimes r\}\}\{\\sqrt\{r\!d^\{r/2\}\}\}\. Then the feature map satisfying \([2](https://arxiv.org/html/2610.40943#S2.E2)\), isϕ⁡\[x\]=\(ϕr​\[x\]\)r∈ℕ\.\\phi\[x\]=\(\\phi\_\{r\}\[x\]\)\_\{r\\in\\mathbb\{N\}\}\.

We remark here that any sum of features generated byϕ\\phiis a direct sum of symmetric tensors of degreerr, one for eachr∈ℕr\\in\\mathbb\{N\}\. We denote byKϕ:H→ℝNK\_\{\\phi\}:H\\to\\mathbb\{R\}^\{N\}the linear map whose rows are the vectorsϕ⁡\[Kj\]∈H\\phi\[K\_\{j\}\]\\in H,j∈\[N\]j\\in\[N\]\. Being bounded and of finite rank,KϕK\_\{\\phi\}is a Hilbert\-Schmidt operator, which allows us to manipulate it as we would with a matrix\. Attention can then be expressed as

Att⁡\(q,K,V\)=ϕ​\[q\]𝖳​Kϕ𝖳​Vϕ​\[q\]𝖳​Kϕ𝖳​𝟏N=ϕ​\[q\]𝖳​ΠKϕ​Kϕ𝖳​Vϕ​\[q\]𝖳​ΠKϕ​Kϕ𝖳​𝟏N,ΠKϕ=Kϕ𝖳​\(Kϕ​Kϕ𝖳\)†​Kϕ,\\operatorname\{Att\}\(q,K,V\)=\\frac\{\\phi\[q\]^\{\\mathsf\{T\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}V\}\{\\phi\[q\]^\{\\mathsf\{T\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}\\mathbf\{1\}\_\{N\}\}=\\frac\{\\phi\[q\]^\{\\mathsf\{T\}\}\\Pi\_\{K\_\{\\phi\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}V\}\{\\phi\[q\]^\{\\mathsf\{T\}\}\\Pi\_\{K\_\{\\phi\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}\\mathbf\{1\}\_\{N\}\},\\qquad\\Pi\_\{K\_\{\\phi\}\}=K\_\{\\phi\}^\{\\mathsf\{T\}\}\\big\(K\_\{\\phi\}K\_\{\\phi\}^\{\\mathsf\{T\}\}\\big\)^\{\\dagger\}K\_\{\\phi\},\(3\)whereΠKϕ\\Pi\_\{K\_\{\\phi\}\}is the orthogonal projector ontoSpan⁡\(Kϕ\)\\operatorname\{Span\}\(K\_\{\\phi\}\)inHH\. As already mentioned in the introduction, the attention is thus a ratio of two linear maps ofϕ⁡\[q\]\\phi\[q\], both of which act on the sameNN\-dimensional subspace ofHH,Span⁡\(Kϕ\)\\operatorname\{Span\}\(K\_\{\\phi\}\)determined by the key vectors\. Consider the two following sets of elements ofHHconstructed fromKK, the elements of the first consists in the feature map of a sum of vectors while those of the second consists of a sum of feature maps\.

ϕ⁡\[Span⁡\(K\)\]\\displaystyle\\phi\\left\[\\operatorname\{Span\}\(K\)\\right\]:=ϕ⁡\[\{∑jαj​Kj∈ℝd\|αj∈ℝ\}\],\\displaystyle:=\\phi\\left\[\\left\\\{\\sum\_\{j\}\\alpha\_\{j\}K\_\{j\}\\in\\mathbb\{R\}^\{d\}\\Big\|\\,\\alpha\_\{j\}\\in\\mathbb\{R\}\\right\\\}\\right\],Span⁡\(Kϕ\)\\displaystyle\\operatorname\{Span\}\(K\_\{\\phi\}\):=\{∑jαj​ϕ​\[Kj\]∈H\|αj∈ℝ\}\.\\displaystyle:=\\left\\\{\\sum\_\{j\}\\alpha\_\{j\}\\phi\[K\_\{j\}\]\\in H\\Big\|\\,\\alpha\_\{j\}\\in\\mathbb\{R\}\\right\\\}\.\(4\)The first set is a submanifold while the second is a vector subspace, and they only match on the feature maps of the keys themselves\. Indeed,ϕ:ℝd→H\\phi:\\mathbb\{R\}^\{d\}\\to His differentiable, injective, and a homeomorphism overϕ⁡\[ℝd\]\\phi\[\\mathbb\{R\}^\{d\}\]using the restriction of the feature map toH⊗1H^\{\\otimes 1\},ϕ1:ℝd→ℝd\\phi\_\{1\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\}\.

###### Proposition 1\.

LetK∈ℝN×dK\\in\\mathbb\{R\}^\{N\\times d\}, with distinct and nonzero rows\. Then the intersection of the two sets is given by

Span⁡\(Kϕ\)∩ϕ⁡\[Span⁡\(K\)\]=Span⁡\(Kϕ\)∩ϕ⁡\[ℝd\]=\{ϕ⁡\[Kj\]:j∈\[N\]\}\.\\operatorname\{Span\}\(K\_\{\\phi\}\)\\cap\\phi\[\\operatorname\{Span\}\(K\)\]=\\operatorname\{Span\}\(K\_\{\\phi\}\)\\cap\\phi\[\\mathbb\{R\}^\{d\}\]=\\\{\\phi\[K\_\{j\}\]:j\\in\[N\]\\\}\.

The proof, given in Appendix[A](https://arxiv.org/html/2610.40943#A1), relies on the following lemma\.

###### Lemma 1\.

For anyu∈ℝd∖\{0\}u\\in\\mathbb\{R\}^\{d\}\\setminus\\\{0\\\}and anyα∈ℝ∖\{0,1\}\\alpha\\in\\mathbb\{R\}\\setminus\\\{0,1\\\},α​ϕ​\[u\]∉ϕ⁡\[ℝd\]\\alpha\\phi\[u\]\\notin\\phi\[\\mathbb\{R\}^\{d\}\]\. Furthermore, letu∈ℝd∖\{0\}u\\in\\mathbb\{R\}^\{d\}\\setminus\\\{0\\\},k∈ℕ∗k\\in\\mathbb\{N\}^\{\*\}, andui=αi​uu\_\{i\}=\\alpha\_\{i\}ufori∈\[k\]i\\in\[k\], with pairwise distinctα1,…,αk∈ℝ∖\{0\}\\alpha\_\{1\},\\dots,\\alpha\_\{k\}\\in\\mathbb\{R\}\\setminus\\\{0\\\}\. Then, for everyR∈ℕR\\in\\mathbb\{N\}, the truncated features\(ϕr​\[ui\]\)r=RR\+k−1\\big\(\\phi\_\{r\}\[u\_\{i\}\]\\big\)\_\{r=R\}^\{R\+k\-1\},i∈\[k\]i\\in\[k\], are linearly independent and thusdimSpan⁡\{ϕ⁡\[u1\],…,ϕ⁡\[uk\]\}=k\\dim\\operatorname\{Span\}\\\{\\phi\[u\_\{1\}\],\\dots,\\phi\[u\_\{k\}\]\\\}=k\.

The proof of[Lemma1](https://arxiv.org/html/2610.40943#Thmlemma1)is also given in Appendix[A](https://arxiv.org/html/2610.40943#A1)\.

The consequence we shall use is that the only elements ofSpan⁡\(Kϕ\)\\operatorname\{Span\}\(K\_\{\\phi\}\)which coincide with the actual feature map of a vector ofℝd\\mathbb\{R\}^\{d\}are in the keys themselves\.

### 2\.2Interpolating attention

We are interested in interpolating attention on a given reduced set of key vectors\. LetL∈ℝn×dL\\in\\mathbb\{R\}^\{n\\times d\}withn≪Nn\\ll Nbe a matrix formed by a set of key vectors, whose rows we call the*interpolation nodes*, or magic points, onto which we wish to interpolate the mapAtt⁡\(⋅,K,V\):ℝd→ℝd\\operatorname\{Att\}\(\\cdot,K,V\):\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\}, and letLϕ:H→ℝnL\_\{\\phi\}:H\\to\\mathbb\{R\}^\{n\}be the operator with rowsϕ⁡\[Li\]\\phi\[L\_\{i\}\]\. The reduced set of key features\{ϕ⁡\[Li\]\}i∈\[n\]\\\{\\phi\[L\_\{i\}\]\\\}\_\{i\\in\[n\]\}then span the basis spaceℬ\\mathcal\{B\}onto which we interpolate, i\.e\.

ℬ:=Span⁡\(Lϕ\)⊂Span⁡\(Kϕ\)⊂H\.\\mathcal\{B\}:=\\operatorname\{Span\}\(L\_\{\\phi\}\)\\subset\\operatorname\{Span\}\(K\_\{\\phi\}\)\\subset H\.\(5\)We now discuss an approach for interpolating the attention on this reduced set of keys\. For anyqqwithϕ⁡\[q\]∈ℬ\\phi\[q\]\\in\\mathcal\{B\}, we can insertΠLϕ\\Pi\_\{L\_\{\\phi\}\}, the orthogonal projector ontoℬ\\mathcal\{B\}, into \([3](https://arxiv.org/html/2610.40943#S2.E3)\) without changing its value,

Att⁡\(q,K,V\)=ϕ​\[q\]𝖳​ΠLϕ​ΠKϕ​Kϕ𝖳​Vϕ​\[q\]𝖳​ΠLϕ​ΠKϕ​Kϕ𝖳​𝟏N,\\operatorname\{Att\}\(q,K,V\)=\\frac\{\\phi\[q\]^\{\\mathsf\{T\}\}\\Pi\_\{L\_\{\\phi\}\}\\Pi\_\{K\_\{\\phi\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}V\}\{\\phi\[q\]^\{\\mathsf\{T\}\}\\Pi\_\{L\_\{\\phi\}\}\\Pi\_\{K\_\{\\phi\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}\\mathbf\{1\}\_\{N\}\},and by[Proposition1](https://arxiv.org/html/2610.40943#Thmproposition1)the nodesq∈\{L1,…,Ln\}q\\in\\\{L\_\{1\},\\dots,L\_\{n\}\\\}are the only queries for whichϕ⁡\[q\]∈ℬ\\phi\[q\]\\in\\mathcal\{B\}\. The interpolation equality thus holds exactly at the nodes, although it may also hold at other queries, as attention only depends onϕ⁡\[q\]\\phi\[q\]through its projectionΠKϕ​ϕ​\[q\]\\Pi\_\{K\_\{\\phi\}\}\\phi\[q\]ontoSpan⁡\(Kϕ\)\\operatorname\{Span\}\(K\_\{\\phi\}\), which may lie inℬ\\mathcal\{B\}even whenϕ⁡\[q\]\\phi\[q\]does not\. For any two matricesA∈ℝa×dA\\in\\mathbb\{R\}^\{a\\times d\}andB∈ℝb×dB\\in\\mathbb\{R\}^\{b\\times d\}, we define

EA​B\[i,j\]:=⟨ϕ\[Ai\],ϕ\[Bj\]⟩H=exp\(⟨Ai,Bj⟩/d\),i∈\[a\],j∈\[b\],i\.e\.EA​B=exp\(AB𝖳/d\),E\_\{AB\}\[i,j\]:=\\langle\\phi\[A\_\{i\}\],\\phi\[B\_\{j\}\]\\rangle\_\{H\}=\\exp\(\\langle A\_\{i\},B\_\{j\}\\rangle/\\sqrt\{d\}\),\\quad i\\in\[a\],j\\in\[b\],\\qquad\\text\{i\.e\.\}\\quad E\_\{AB\}=\\exp\(AB^\{\\mathsf\{T\}\}/\\sqrt\{d\}\),where the element of the matrix in theii\-th row and thejj\-th column is denoted using square brackets and where the exponential is applied entrywise, we obtain,

ΠLϕ​Kϕ𝖳=Lϕ𝖳​\(Lϕ​Lϕ𝖳\)†​Lϕ​Kϕ𝖳=Lϕ𝖳​EL​L†​EL​K\.\\Pi\_\{L\_\{\\phi\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}=L\_\{\\phi\}^\{\\mathsf\{T\}\}\(L\_\{\\phi\}L\_\{\\phi\}^\{\\mathsf\{T\}\}\)^\{\\dagger\}L\_\{\\phi\}K\_\{\\phi\}^\{\\mathsf\{T\}\}=L\_\{\\phi\}^\{\\mathsf\{T\}\}E\_\{LL\}^\{\\dagger\}E\_\{LK\}\.Hence, for a givenLL, we have \(using a similar method to the one used in\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]\) that the associated interpolated value vectorW∈ℝn×dW\\in\\mathbb\{R\}^\{n\\times d\}and the normalization weightWNormW\_\{\\mathrm\{Norm\}\}are the solutions to the following equations

EL​L​W=EL​K​VandEL​L​WNorm=EL​K​𝟏N\.E\_\{LL\}W=E\_\{LK\}V\\quad\\text\{ and \}\\quad E\_\{LL\}W\_\{\\mathrm\{Norm\}\}=E\_\{LK\}\\mathbf\{1\}\_\{N\}\.\(6\)SettingWaug∈ℝn×\(d\+1\)W^\{\\mathrm\{aug\}\}\\in\\mathbb\{R\}^\{n\\times\(d\+1\)\}, and assumingEL​LE\_\{LL\}is invertible, we can write, for simplicity, the solution to be

Waug:=EL​L†​EL​K​Vaug=\[W∣WNorm\],Vaug:=\[V∣𝟏N\]\.W^\{\\mathrm\{aug\}\}:=E\_\{LL\}^\{\\dagger\}E\_\{LK\}V^\{\\mathrm\{aug\}\}=\[W\\mid W\_\{\\mathrm\{Norm\}\}\],\\qquad V^\{\\mathrm\{aug\}\}:=\[V\\mid\\mathbf\{1\}\_\{N\}\]\.\(7\)Indeed, as the numerator and denominator share their left factorKϕ𝖳K\_\{\\phi\}^\{\\mathsf\{T\}\}, we use a single basis space for the interpolation of both the numerator and denominator by considering the jointℝd\+1\\mathbb\{R\}^\{d\+1\}augmented interpolation problem given by

EL​L​Waug=EL​K​Vaug\.E\_\{LL\}W^\{\\mathrm\{aug\}\}=E\_\{LK\}V^\{\\mathrm\{aug\}\}\.\(8\)The interpolant of attention onℬ\\mathcal\{B\}is then given by

Att~L​\(q,K,V\)=ϕ​\[q\]𝖳​ΠLϕ​Kϕ𝖳​Vϕ​\[q\]𝖳​ΠLϕ​Kϕ𝖳​𝟏N=ϕ​\[q\]𝖳​Lϕ𝖳​Wϕ​\[q\]𝖳​Lϕ𝖳​WNorm\.\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(q,K,V\)=\\frac\{\\phi\[q\]^\{\\mathsf\{T\}\}\\Pi\_\{L\_\{\\phi\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}V\}\{\\phi\[q\]^\{\\mathsf\{T\}\}\\Pi\_\{L\_\{\\phi\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}\\mathbf\{1\}\_\{N\}\}=\\frac\{\\phi\[q\]^\{\\mathsf\{T\}\}L\_\{\\phi\}^\{\\mathsf\{T\}\}W\}\{\\phi\[q\]^\{\\mathsf\{T\}\}L\_\{\\phi\}^\{\\mathsf\{T\}\}W\_\{\\mathrm\{Norm\}\}\}\.\(9\)Hence,*the interpolant of the attention mechanism is the ratio of the interpolants of the two linear maps*\.

The residual we ideally want to minimise in the context of interpolation is the maximum error inp∈ℕ∗p\\in\\mathbb\{N\}^\{\*\}norm over all possible queriesq∈ℝdq\\in\\mathbb\{R\}^\{d\},

ℛp​\(L\)=maxq∈ℝd⁡∥Att⁡\(q,K,V\)−Att~L​\(q,K,V\)∥p\.\\mathcal\{R\}\_\{p\}\(L\)=\\max\_\{q\\in\\mathbb\{R\}^\{d\}\}\\lVert\\operatorname\{Att\}\(q,K,V\)\-\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(q,K,V\)\\rVert\_\{p\}\.We note that, contrary to the exact denominator, the interpolated denominatorϕ​\[q\]𝖳​Lϕ𝖳​WNorm\\phi\[q\]^\{\\mathsf\{T\}\}L\_\{\\phi\}^\{\\mathsf\{T\}\}W\_\{\\mathrm\{Norm\}\}is not guaranteed to be positive and may be equal to zero for some queries\. In that caseAtt~L\\widetilde\{\\operatorname\{Att\}\}\_\{L\}has a pole andℛp​\(L\)\\mathcal\{R\}\_\{p\}\(L\)is infinite, we introduce a mean vector approximation in §[4\.1\.1](https://arxiv.org/html/2610.40943#S4.SS1.SSS1)to reduce the impact of this problem\.

### 2\.3Invariants of attention

However, the linear problem \([6](https://arxiv.org/html/2610.40943#S2.E6)\) is extremely ill\-conditioned and computing its numerical solution is therefore very unstable\. Indeed, the RKHS norms of the keys vary by orders of magnitude, resulting in ill\-conditionedEL​KE\_\{LK\}andEK​KE\_\{KK\}matrices\. However, using the invariants of the attention mechanism, we can solve an equivalent but better\-conditioned problem through a Gaussian kernel formulation\.

In this subsection, we consider three invariants of the attention mechanism and describe a Gaussian kernel reformulation of the attention mechanism \(§[2\.4](https://arxiv.org/html/2610.40943#S2.SS4)\)\. We then show that recentering the keys \([20](https://arxiv.org/html/2610.40943#S2.E20)\), also used in WildCat\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\], gives us an additional degree of freedom and finally combine these results in §[2\.4\.1](https://arxiv.org/html/2610.40943#S2.SS4.SSS1)to improve the conditioning of the interpolation procedure\.

###### Proposition 2\(Invariants of attention\)\.

Considerq∈ℝdq\\in\\mathbb\{R\}^\{d\},K∈ℝN×dK\\in\\mathbb\{R\}^\{N\\times d\}andV∈ℝN×dV\\in\\mathbb\{R\}^\{N\\times d\}\.

1. \(i\.\)Query kernel\-norm scaling: Givenλ∈ℝ∖\{0\}\\lambda\\in\\mathbb\{R\}\\setminus\\\{0\\\}, the attention as defined in \([3](https://arxiv.org/html/2610.40943#S2.E3)\) is invariant under the transformϕ⁡\[q\]→λ​ϕ​\[q\]\\phi\[q\]\\to\\lambda\\phi\[q\]\. Attention thus depends only on the direction ofϕ⁡\[q\]\\phi\[q\]inHH, not on its magnitude\.
2. \(ii\.\)Global key mass scaling: Givenλ∈ℝ∖\{0\}\\lambda\\in\\mathbb\{R\}\\setminus\\\{0\\\}, the attention output is invariant under the rescaling of every key feature by the sameλ\\lambda, i\.e\.ϕ⁡\[Ki\]→λ​ϕ​\[Ki\]\\phi\[K\_\{i\}\]\\to\\lambda\\phi\[K\_\{i\}\]for alli∈\[N\]i\\in\[N\]\.
3. \(iii\.\)Global key translation: For anyδ∈ℝd\\delta\\in\\mathbb\{R\}^\{d\}, the attention output is invariant under the translationKi→Ki\+δK\_\{i\}\\to K\_\{i\}\+\\deltafor alli∈\[N\]i\\in\[N\], i\.e\.Att⁡\(q,K\+δ,V\)=Att⁡\(q,K,V\)\\operatorname\{Att\}\(q,K\+\\delta,V\)=\\operatorname\{Att\}\(q,K,V\)\.

The proof is given in Appendix[A](https://arxiv.org/html/2610.40943#A1)\. Each of these properties of attention allows us to consider for our interpolation a chosen representative of the ‘equivalence’ class they create\.

1. I\.Kernel\-normalized queries\. As we approximate the ratio of linear maps by a ratio of two interpolated linear maps, we can restrict ourselves to kernel\-normalized query vectors to reproduce the effect of attention333Note however that this does not imply any restriction on the norm ofqq\. Indeed, the only kernel\-normalized vector ofϕ⁡\[ℝd\]\\phi\[\\mathbb\{R\}^\{d\}\]isϕ⁡\[0\]\\phi\[0\]\. In the Gaussian rewriting §[2\.4](https://arxiv.org/html/2610.40943#S2.SS4), this corresponds to restricting ourselves to the directional vectors given by the Gaussian feature mapϕ˘\\breve\{\\phi\}, simplifying our Gaussian kernel reformulation in §[2\.4](https://arxiv.org/html/2610.40943#S2.SS4)\.
2. II\.Key normalization gauge\. As attention is invariant by a global rescaling of the kernel norms of the keys,‖ϕ⁡\[Ki\]‖H\\\|\\phi\[K\_\{i\}\]\\\|\_\{H\}, only their ratios as defined later in equation \([22](https://arxiv.org/html/2610.40943#S2.E22)\) carry information\. In §[2\.4](https://arxiv.org/html/2610.40943#S2.SS4)we use this to define the max\-normalised gauge which allows us to restrict the numerical range of all kernel norms\(‖ϕ⁡\[Ki\]‖H\)i∈\[N\]\\left\(\\\|\\phi\[K\_\{i\}\]\\\|\_\{H\}\\right\)\_\{i\\in\[N\]\}to\(0,1\]\(0,1\], and hence their logarithms, the shifted scaled squaredL2L^\{2\}norms\(‖Ki‖2−maxj∈\[N\]⁡‖Kj‖2\)/\(2​d\)\(\\\|K\_\{i\}\\\|^\{2\}\-\\max\_\{j\\in\[N\]\}\\\|K\_\{j\}\\\|^\{2\}\)/\(2\\sqrt\{d\}\), to\[−maxj∈\[N\]∥Kj∥2/\(2d\),0\]\[\-\\max\_\{j\\in\[N\]\}\\\|K\_\{j\}\\\|^\{2\}/\(2\\sqrt\{d\}\),0\]\.
3. III\.Key centering gauge\. As attention is invariant to the translation of the keys by a same vector, the keys can be recentered\. In \([21](https://arxiv.org/html/2610.40943#S2.E21)\), we consider the mean of the key vectors as the origin, allowing us to reduce the spread of the mass ratios\.

Query translation, on the other hand, does change the output of attention \(otherwise, it would be constant\)\.

###### Proposition 3\(Query translation\)\.

Givenq,δ∈ℝdq,\\delta\\in\\mathbb\{R\}^\{d\}, the effect of a translation ofδ\\deltaon the output of attention is given by

Att⁡\(q\+δ,K,V\)=\[exp⁡\(q𝖳​K𝖳/d\)⊙exp⁡\(δ𝖳​K𝖳/d\)\]​V\[exp⁡\(q𝖳​K𝖳/d\)⊙exp⁡\(δ𝖳​K𝖳/d\)\]​𝟏N=ϕ​\[q\]𝖳​Kϕ𝖳​ΞK,δ​Vϕ​\[q\]𝖳​Kϕ𝖳​ΞK,δ​𝟏N,\\operatorname\{Att\}\(q\+\\delta,K,V\)=\\frac\{\\left\[\\exp\\left\(q^\{\\mathsf\{T\}\}K^\{\\mathsf\{T\}\}/\\sqrt\{d\}\\right\)\\odot\\exp\\left\(\\delta^\{\\mathsf\{T\}\}K^\{\\mathsf\{T\}\}/\\sqrt\{d\}\\right\)\\right\]V\}\{\\left\[\\exp\\left\(q^\{\\mathsf\{T\}\}K^\{\\mathsf\{T\}\}/\\sqrt\{d\}\\right\)\\odot\\exp\\left\(\\delta^\{\\mathsf\{T\}\}K^\{\\mathsf\{T\}\}/\\sqrt\{d\}\\right\)\\right\]\\mathbf\{1\}\_\{N\}\}=\\frac\{\\phi\[q\]^\{\\mathsf\{T\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}\\Xi\_\{K,\\delta\}V\}\{\\phi\[q\]^\{\\mathsf\{T\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}\\Xi\_\{K,\\delta\}\\mathbf\{1\}\_\{N\}\},where we setΞK,δ=Diag⁡\(\(⟨ϕ⁡\[δ\],ϕ⁡\[Ki\]⟩\)i∈\[N\]\)\\Xi\_\{K,\\delta\}=\\operatorname\{Diag\}\\left\(\(\\langle\\phi\[\\delta\],\\phi\[K\_\{i\}\]\\rangle\)\_\{i\\in\[N\]\}\\right\)\.

### 2\.4Gaussian formulation and centering of the keys

The main tool that allows us to make use of these invariants is the reformulation of attention using the Gaussian kernel\. We denote the Gaussian kernel \(or Radial Basis Function kernel\) byκ˘\(x,y\)=exp\(−∥x−y∥2/2d\)=∥ϕ\[x−y\]∥H−1\\breve\{\\kappa\}\(x,y\)=\\exp\\left\(\{\-\\\|x\-y\\\|^\{2\}/2\\sqrt\{d\}\}\\right\)=\\\|\\phi\[x\-y\]\\\|\_\{H\}^\{\-1\}; we denote with the breve symbol⋅˘\\breve\{\\cdot\}the objects attached to this Gaussian kernel\. For anyx,y∈ℝdx,y\\in\\mathbb\{R\}^\{d\}, the exponential kernel can be rewritten as

κexp​\(x,y\)=‖ϕ⁡\[x\]‖H​κ˘​\(x,y\)​‖ϕ⁡\[y\]‖H\.\\kappa\_\{\\exp\}\(x,y\)=\\\|\\phi\[x\]\\\|\_\{H\}\\breve\{\\kappa\}\(x,y\)\\\|\\phi\[y\]\\\|\_\{H\}\.\(13\)This can equivalently be expressed by defining the Gaussian feature mapϕ˘​\[x\]:=ϕ⁡\[x\]‖ϕ⁡\[x\]‖H\\breve\{\\phi\}\[x\]:=\\frac\{\\phi\[x\]\}\{\\\|\\phi\[x\]\\\|\_\{H\}\}, such that, for anyx,y∈ℝdx,y\\in\\mathbb\{R\}^\{d\}, we have⟨ϕ˘​\[x\],ϕ˘​\[y\]⟩H=κ˘​\(x,y\)\\langle\\breve\{\\phi\}\[x\],\\breve\{\\phi\}\[y\]\\rangle\_\{H\}=\\breve\{\\kappa\}\(x,y\)\. As discussed later, a useful property of this reformulation is that it separates the norm‖ϕ⁡\[x\]‖H\\\|\\phi\[x\]\\\|\_\{H\}, and directionϕ˘​\[x\]\\breve\{\\phi\}\[x\], of the exponential kernel featuresϕ⁡\[x\]\\phi\[x\]\. Then, for any Gaussian kernel evaluations on the matricesA∈ℝa×dA\\in\\mathbb\{R\}^\{a\\times d\}andB∈ℝb×dB\\in\\mathbb\{R\}^\{b\\times d\}, we set

E˘A​B​\[i,j\]=⟨ϕ⁡\[Ai\],ϕ⁡\[Bj\]⟩H∥ϕ⁡\[Ai\]∥H​∥ϕ⁡\[Bj\]∥H=⟨ϕ˘​\[Ai\],ϕ˘​\[Bj\]⟩H,i∈\[a\],j∈\[b\],\\breve\{E\}\_\{AB\}\[i,j\]=\\frac\{\\langle\\phi\[A\_\{i\}\],\\phi\[B\_\{j\}\]\\rangle\_\{H\}\}\{\\lVert\\phi\[A\_\{i\}\]\\rVert\_\{H\}\\lVert\\phi\[B\_\{j\}\]\\rVert\_\{H\}\}=\\langle\\breve\{\\phi\}\[A\_\{i\}\],\\breve\{\\phi\}\[B\_\{j\}\]\\rangle\_\{H\},\\quad i\\in\[a\],j\\in\[b\],\(14\)where we denote the element of the matrix in theii\-th row andjj\-th column using square brackets\. The Gaussian gram matrix can hence be related to the exponential kernel one as

EL​L=DLϕ1/2​E˘L​L​DLϕ1/2,EK​K=DKϕ1/2​E˘K​K​DKϕ1/2E\_\{LL\}=D\_\{L\_\{\\phi\}\}^\{1/2\}\\breve\{E\}\_\{LL\}D\_\{L\_\{\\phi\}\}^\{1/2\},\\quad E\_\{KK\}=D\_\{K\_\{\\phi\}\}^\{1/2\}\\breve\{E\}\_\{KK\}D\_\{K\_\{\\phi\}\}^\{1/2\}\(15\)where we set, for anyB∈ℝb×dB\\in\\mathbb\{R\}^\{b\\times d\},DBϕ=Diag⁡\(exp⁡\(‖Bi‖22/d\)\)i∈\[b\]=Diag⁡\(‖ϕ⁡\[Bi\]‖H2\)i∈\[b\]D\_\{B\_\{\\phi\}\}=\\operatorname\{Diag\}\\left\(\\exp\(\{\\\|B\_\{i\}\\\|^\{2\}\_\{2\}/\\sqrt\{d\}\}\)\\right\)\_\{i\\in\[b\]\}=\\operatorname\{Diag\}\(\\\|\\phi\[B\_\{i\}\]\\\|\_\{H\}^\{2\}\)\_\{i\\in\[b\]\}, abbreviatedDBD\_\{B\}when no ambiguity arises\. WithB˘ϕ:=DB−1/2Bϕ\\breve\{B\}\_\{\\phi\}:=D\_\{B\}^\{\-1/2\}B\_\{\\phi\}the operator with rowsϕ˘​\[Bi\]\\breve\{\\phi\}\[B\_\{i\}\]for anyB∈ℝb×dB\\in\\mathbb\{R\}^\{b\\times d\}, we obtain

Att⁡\(q,K,V\)=ϕ˘​\[q\]𝖳​K˘ϕ𝖳​DKϕ1/2​Vϕ˘​\[q\]𝖳​K˘ϕ𝖳​DKϕ1/2​𝟏N=E˘q​K​DKϕ1/2​VE˘q​K​DKϕ1/2​𝟏N\.\\operatorname\{Att\}\(q,K,V\)=\\frac\{\\breve\{\\phi\}\[q\]^\{\\mathsf\{T\}\}\\breve\{K\}\_\{\\phi\}^\{\\mathsf\{T\}\}D\_\{K\_\{\\phi\}\}^\{1/2\}V\}\{\\breve\{\\phi\}\[q\]^\{\\mathsf\{T\}\}\\breve\{K\}\_\{\\phi\}^\{\\mathsf\{T\}\}D\_\{K\_\{\\phi\}\}^\{1/2\}\\mathbf\{1\}\_\{N\}\}=\\frac\{\\breve\{E\}\_\{qK\}D\_\{K\_\{\\phi\}\}^\{1/2\}V\}\{\\breve\{E\}\_\{qK\}D\_\{K\_\{\\phi\}\}^\{1/2\}\\mathbf\{1\}\_\{N\}\}\.\(16\)We also obtain for the interpolation,

Att~L​\(q,K,V\)=ϕ˘​\[q\]𝖳​L˘ϕ𝖳​DLϕ1/2​Wϕ˘​\[q\]𝖳​L˘ϕ𝖳​DLϕ1/2​WNorm=E˘q​L​DLϕ1/2​WE˘q​L​DLϕ1/2​WNorm\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(q,K,V\)=\\frac\{\\breve\{\\phi\}\[q\]^\{\\mathsf\{T\}\}\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}D\_\{L\_\{\\phi\}\}^\{1/2\}W\}\{\\breve\{\\phi\}\[q\]^\{\\mathsf\{T\}\}\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}D\_\{L\_\{\\phi\}\}^\{1/2\}W\_\{\\mathrm\{Norm\}\}\}=\\frac\{\\breve\{E\}\_\{qL\}D\_\{L\_\{\\phi\}\}^\{1/2\}W\}\{\\breve\{E\}\_\{qL\}D\_\{L\_\{\\phi\}\}^\{1/2\}W\_\{\\mathrm\{Norm\}\}\}\(17\)Hence, the interpolated weights \([7](https://arxiv.org/html/2610.40943#S2.E7)\) are given as

W=EL​L†EL​KV=DL−1/2E˘L​L†E˘L​KDK1/2V,WNorm=EL​L†EL​K𝟏N=DL−1/2E˘L​L†E˘L​KDK1/2𝟏N\.W=E\_\{LL\}^\{\\dagger\}E\_\{LK\}V=D\_\{L\}^\{\-1/2\}\\breve\{E\}\_\{LL\}^\{\\dagger\}\\breve\{E\}\_\{LK\}D\_\{K\}^\{1/2\}V,\\quad W\_\{\\mathrm\{Norm\}\}=E\_\{LL\}^\{\\dagger\}E\_\{LK\}\\mathbf\{1\}\_\{N\}=D\_\{L\}^\{\-1/2\}\\breve\{E\}\_\{LL\}^\{\\dagger\}\\breve\{E\}\_\{LK\}D\_\{K\}^\{1/2\}\\mathbf\{1\}\_\{N\}\.\(18\)
Furthermore, due to invariant*\(ii\.\)*and Gaussian formulation, we can replaceDKϕD\_\{K\_\{\\phi\}\}in Equation \([16](https://arxiv.org/html/2610.40943#S2.E16)\) by itsL∞L^\{\\infty\}\-normalisation, which we denote as

ηi:=∥ϕ⁡\[Ki\]∥Hmaxj⁡∥ϕ⁡\[Kj\]∥H=exp⁡\(∥Ki∥2−maxj⁡∥Kj∥22​d\)=DK,i​i1/2∥DK1/2∥∞∈\(0,1\]\.\\eta\_\{i\}:=\\frac\{\\lVert\\phi\[K\_\{i\}\]\\rVert\_\{H\}\}\{\\max\_\{j\}\\lVert\\phi\[K\_\{j\}\]\\rVert\_\{H\}\}=\\exp\\\!\\Big\(\\frac\{\\lVert K\_\{i\}\\rVert^\{2\}\-\\max\_\{j\}\\lVert K\_\{j\}\\rVert^\{2\}\}\{2\\sqrt\{d\}\}\\Big\)=\\frac\{D\_\{K,ii\}^\{1/2\}\}\{\\lVert D\_\{K\}^\{1/2\}\\rVert\_\{\\infty\}\}\\in\(0,1\]\.\(19\)
The global key translation invariance*\(iii\)*of[Proposition2](https://arxiv.org/html/2610.40943#Thmproposition2)represents an additional degree of freedom for interpolation\. As we have seen before, the Gaussian formulation allows us to separateϕ⁡\[x\]\\phi\[x\]into its norm from the direction given by the feature mapϕ˘\\breve\{\\phi\}, i\.e\. the normalised vector in the RKHS\. Due to the translation invariance of the Gaussian kernel, only the norm of the keys depends on the recentering\.

###### Proposition 4\(Gram matrices for translated keys\)\.

For anyA∈ℝa×dA\\in\\mathbb\{R\}^\{a\\times d\},B∈ℝb×dB\\in\\mathbb\{R\}^\{b\\times d\}and anyδ∈ℝd\\delta\\in\\mathbb\{R\}^\{d\}, we haveE˘A−δ,B−δ=E˘A​B\\breve\{E\}\_\{A\-\\delta,B\-\\delta\}=\\breve\{E\}\_\{AB\}and hence

EA−δ,B−δ=D\(A−δ\)ϕ1/2​E˘A​B​D\(B−δ\)ϕ1/2=ΞA,δ−1​EA​B​ΞB,δ−1⋅exp⁡\(‖δ‖2d\)\.E\_\{A\-\\delta,B\-\\delta\}=D\_\{\(A\-\\delta\)\_\{\\phi\}\}^\{1/2\}\\breve\{E\}\_\{AB\}D\_\{\(B\-\\delta\)\_\{\\phi\}\}^\{1/2\}=\\Xi^\{\-1\}\_\{A,\\delta\}E\_\{AB\}\\Xi\_\{B,\\delta\}^\{\-1\}\\cdot\\exp\\left\(\\frac\{\\\|\\delta\\\|^\{2\}\}\{\\sqrt\{d\}\}\\right\)\.whereΞ\\Xiis defined as in Proposition[3](https://arxiv.org/html/2610.40943#Thmproposition3)and we set

D\(A−δ\)ϕ1/2\[i,i\]=∥ϕ\[Ai−δ\]∥H=e‖Ai‖2/2​de−⟨Ai,δ⟩/de‖δ‖2/2​d\.D\_\{\(A\-\\delta\)\_\{\\phi\}\}^\{1/2\}\[i,i\]=\\\|\\phi\[A\_\{i\}\-\\delta\]\\\|\_\{H\}=e^\{\\\|A\_\{i\}\\\|^\{2\}/2\\sqrt\{d\}\}e^\{\-\\langle A\_\{i\},\\delta\\rangle/\\sqrt\{d\}\}e^\{\\\|\\delta\\\|^\{2\}/2\\sqrt\{d\}\}\.

The proof is given in Appendix[A](https://arxiv.org/html/2610.40943#A1)\. In our case, this leads to the following,

EL−δ,K−δ=e‖δ‖2/d⋅ΞL,δ−1​EL​K​ΞK,δ−1andEK−δ,K−δ=e‖δ‖2/d⋅ΞK,δ−1​EK​K​ΞK,δ−1\.E\_\{L\-\\delta,K\-\\delta\}=e^\{\\\|\\delta\\\|^\{2\}/\\sqrt\{d\}\}\\cdot\\Xi\_\{L,\\delta\}^\{\-1\}E\_\{LK\}\\Xi\_\{K,\\delta\}^\{\-1\}\\quad\\text\{ and \}\\quad E\_\{K\-\\delta,K\-\\delta\}=e^\{\\\|\\delta\\\|^\{2\}/\\sqrt\{d\}\}\\cdot\\Xi\_\{K,\\delta\}^\{\-1\}E\_\{KK\}\\Xi\_\{K,\\delta\}^\{\-1\}\.\(20\)
ComparingDKϕD\_\{K\_\{\\phi\}\}andD\(K−δ\)ϕD\_\{\(K\-\\delta\)\_\{\\phi\}\}, we see that the effect of recentering on the ratio of norms is given by

‖ϕ⁡\[Ki−δ\]‖H‖ϕ⁡\[Kj−δ\]‖H=e‖Ki‖2−‖Kj‖2−2​⟨Ki−Kj,δ⟩2​d=‖ϕ⁡\[Ki\]‖H‖ϕ⁡\[Kj\]‖H⋅e−⟨Ki−Kj,δ⟩d\.\\frac\{\\\|\\phi\[K\_\{i\}\-\\delta\]\\\|\_\{H\}\}\{\\\|\\phi\[K\_\{j\}\-\\delta\]\\\|\_\{H\}\}=e^\{\\frac\{\\\|K\_\{i\}\\\|^\{2\}\-\\\|K\_\{j\}\\\|^\{2\}\-2\\langle K\_\{i\}\-K\_\{j\},\\delta\\rangle\}\{2\\sqrt\{d\}\}\}=\\frac\{\\\|\\phi\[K\_\{i\}\]\\\|\_\{H\}\}\{\\\|\\phi\[K\_\{j\}\]\\\|\_\{H\}\}\\cdot e^\{\-\\frac\{\\langle K\_\{i\}\-K\_\{j\},\\delta\\rangle\}\{\\sqrt\{d\}\}\}\.We note that instead of being exact at the chosen keys, the interpolation is exact at the translated nodesLi−δL\_\{i\}\-\\delta\.

###### Proposition 5\(Interpolation with translated keys\)\.

Givenδ∈ℝd\\delta\\in\\mathbb\{R\}^\{d\}, the interpolation of attention onδ\\delta\-translated keys gives

Att~L−δ​\(q,K−δ,V\)=Eq​L​ΞL,δ​EL​L†​EL​K​ΞK,δ−1​VEq​L​ΞL,δ​EL​L†​EL​K​ΞK,δ−1​𝟏N\.\\widetilde\{\\operatorname\{Att\}\}\_\{L\-\\delta\}\(q,K\-\\delta,V\)=\\frac\{E\_\{qL\}\\Xi\_\{L,\\delta\}E\_\{LL\}^\{\\dagger\}E\_\{LK\}\\Xi^\{\-1\}\_\{K,\\delta\}V\}\{E\_\{qL\}\\Xi\_\{L,\\delta\}E\_\{LL\}^\{\\dagger\}E\_\{LK\}\\Xi^\{\-1\}\_\{K,\\delta\}\\mathbf\{1\}\_\{N\}\}\.The interpolation is hence exact at the translated nodesLi−δL\_\{i\}\-\\delta,i∈\[n\]i\\in\[n\], whose features\{ϕ⁡\[Li−δ\],i∈\[n\]\}\\\{\\phi\[L\_\{i\}\-\\delta\],i\\in\[n\]\\\}are the only points ofϕ⁡\[ℝd\]\\phi\[\\mathbb\{R\}^\{d\}\]in the interpolation basis, and the interpolation basis is

ℬδ:=Span⁡\(\(L−δ\)ϕ\)≠ℬ\.\\mathcal\{B\}\_\{\\delta\}:=\\operatorname\{Span\}\\big\(\(L\-\\delta\)\_\{\\phi\}\\big\)\\neq\\mathcal\{B\}\.Translation invariance is in particular recovered whenΠLϕ​Kϕ𝖳=Kϕ𝖳\\Pi\_\{L\_\{\\phi\}\}K^\{\\mathsf\{T\}\}\_\{\\phi\}=K^\{\\mathsf\{T\}\}\_\{\\phi\}\.

The proof is given in Appendix[A](https://arxiv.org/html/2610.40943#A1)\. Let us now consider which point of origin we want to choose\. As we have seen above, the selection of the centering pointδ\\deltaonly affects the distribution of the scaled squaredL2L^\{2\}norms,‖Ki−δ‖2/\(2​d\)=log⁡‖ϕ⁡\[Ki−δ\]‖H\\\|K\_\{i\}\-\\delta\\\|^\{2\}/\(2\\sqrt\{d\}\)=\\log\\\|\\phi\[K\_\{i\}\-\\delta\]\\\|\_\{H\}fori∈\[N\]i\\in\[N\]\. Therefore, for stability, our choice is to minimize the sum of the scaled squaredL2L^\{2\}norms, in general improving the conditioning of the problem while making the norms of the vectors more comparable by concentrating the distribution of the norm of the keys\. Indeed, as the Gaussian Gram matrix is constructed from normalised vectors and the Gaussian scalar product decreases exponentially fast with distance, the conditioning of the problem depends mainly on theDKϕD\_\{K\_\{\\phi\}\}matrices\. Similarly to\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\], we choose to recenter the key vectors on the mean vector

μ=1N​∑j=1NKj\.\\mu=\\frac\{1\}\{N\}\\sum\_\{j=1\}^\{N\}K\_\{j\}\.\(21\)This choice of the centerμ\\muis motivated by the fact that, as the mean of the key distribution, it minimizes the sum of squared distancesμ=arg⁡minδ∈ℝd​∑i∈\[N\]‖Ki−δ‖2\\mu=\\underset\{\\delta\\in\\mathbb\{R\}^\{d\}\}\{\\arg\\min\}\\sum\_\{i\\in\[N\]\}\\\|K\_\{i\}\-\\delta\\\|^\{2\}, and thus the sum of the scaled squaredL2L^\{2\}norms,∑i∈\[N\]‖Ki−δ‖2/\(2​d\)\\sum\_\{i\\in\[N\]\}\\\|K\_\{i\}\-\\delta\\\|^\{2\}/\(2\\sqrt\{d\}\)\. We note however that this does not in general minimize the spread of these norms, which may increase after centering for particular distributions of the keys\.

#### 2\.4\.1Stabilizing interpolation through the Gaussian formulation and centering of the keys

The Gaussian reformulation described in §[2\.4](https://arxiv.org/html/2610.40943#S2.SS4)and the centering of the key vectors with respect to the mean described in equation \([21](https://arxiv.org/html/2610.40943#S2.E21)\) now allow us to construct a better conditioned linear system equivalent to equation \([6](https://arxiv.org/html/2610.40943#S2.E6)\)\. As discussed above, the Gaussian kernel matricesE˘L​L\\breve\{E\}\_\{LL\}andE˘L​K\\breve\{E\}\_\{LK\}are in general much better conditioned than their exponential counterparts \(as shown in Appendix[D](https://arxiv.org/html/2610.40943#A4)\)\. This means that the ill\-conditioning is almost entirely due to the matricesDLϕD\_\{L\_\{\\phi\}\}andDKϕD\_\{K\_\{\\phi\}\}, whose conditioning depends only on the spread of the scaled squaredL2L^\{2\}norms of the keys,‖Ki‖2/\(2​d\)\\\|K\_\{i\}\\\|^\{2\}/\(2\\sqrt\{d\}\)fori∈\[N\]i\\in\[N\]\. We now look at how the invariants allow us to stabilize the problem\. LetηL\(c\)∈\(0,1\]n\\eta^\{\(c\)\}\_\{L\}\\in\(0,1\]^\{n\}andηK\(c\)∈\(0,1\]N\\eta^\{\(c\)\}\_\{K\}\\in\(0,1\]^\{N\}be the gauged norms, which we will refer to as masses, of the centered interpolation nodes and keys,

ηL,i\(c\)=‖ϕ⁡\[Li−μ\]‖Hmaxj⁡‖ϕ⁡\[Kj−μ\]‖H,ηK,i\(c\)=‖ϕ⁡\[Ki−μ\]‖Hmaxj⁡‖ϕ⁡\[Kj−μ\]‖H,\\eta^\{\(c\)\}\_\{L,i\}=\\frac\{\\\|\\phi\[L\_\{i\}\-\\mu\]\\\|\_\{H\}\}\{\\max\_\{j\}\\\|\\phi\[K\_\{j\}\-\\mu\]\\\|\_\{H\}\},\\qquad\\eta^\{\(c\)\}\_\{K,i\}=\\frac\{\\\|\\phi\[K\_\{i\}\-\\mu\]\\\|\_\{H\}\}\{\\max\_\{j\}\\\|\\phi\[K\_\{j\}\-\\mu\]\\\|\_\{H\}\},\(22\)both evaluated in log space\. Then, the augmented Galerkin weights \([18](https://arxiv.org/html/2610.40943#S2.E18)\) can be obtained by the two\-step solve and scale method

E˘L​L​X=E˘L​K​Diag⁡\(ηK\(c\)\)​Vaug,Waug\(c\)=Diag⁡\(ηL\(c\)\)−1​X\.\\breve\{E\}\_\{LL\}X=\\breve\{E\}\_\{LK\}\\operatorname\{Diag\}\\big\(\\eta^\{\(c\)\}\_\{K\}\\big\)V^\{\\mathrm\{aug\}\},\\qquad\{W^\{\\mathrm\{aug\}\}\}^\{\(c\)\}=\\operatorname\{Diag\}\\big\(\\eta^\{\(c\)\}\_\{L\}\\big\)^\{\-1\}X\.\(23\)Then, the interpolated attention onℬμ\\mathcal\{B\}\_\{\\mu\}is given by

Att~L\(c\)​\(q,K,V\)=E˘q,L−μ​Diag⁡\(ηL\(c\)\)​W\(c\)E˘q,L−μ​Diag⁡\(ηL\(c\)\)​WNorm\(c\)=Eq​L​W\(c\)Eq​L​WNorm\(c\),\\widetilde\{\\operatorname\{Att\}\}\_\{L^\{\(c\)\}\}\(q,K,V\)=\\frac\{\\breve\{E\}\_\{q,L\-\\mu\}\\operatorname\{Diag\}\(\\eta^\{\(c\)\}\_\{L\}\)W^\{\(c\)\}\}\{\\breve\{E\}\_\{q,L\-\\mu\}\\operatorname\{Diag\}\(\\eta^\{\(c\)\}\_\{L\}\)W^\{\(c\)\}\_\{\\mathrm\{Norm\}\}\}=\\frac\{E\_\{qL\}W^\{\(c\)\}\}\{E\_\{qL\}W\_\{\\mathrm\{Norm\}\}^\{\(c\)\}\},\(24\)where we note that the second equality usesLLand notL−μL\-\\mu\. The masses of the nodes are kept bounded in\(0,1\]\(0,1\]thanks to theη\\etaformulation, the Gaussian kernel matrix has in general a moderate condition number \(see Appendix[D](https://arxiv.org/html/2610.40943#A4)\) and the right hand side’s rescaling is controlled thanks to the mean recentering\.*From now on, unless otherwise specified, we will consider the problems in theμ\\mu\-centered context and drop the\(c\)\(c\)\.*Note also thatWaugW^\{\\mathrm\{aug\}\}is remultiplied byDiag⁡\(ηL\(c\)\)−1\\operatorname\{Diag\}\(\\eta^\{\(c\)\}\_\{L\}\)^\{\-1\}after the solve, rescaling it to be in the same gauge asVaugV^\{\\mathrm\{aug\}\}\. Indeed, as the two terms in Eq\.\([22](https://arxiv.org/html/2610.40943#S2.E22)\) share their denominators, the gauge is cancelled by construction\. While having different denominators would have no impact on the interpolated attention output, it would affect the scaling of the resulting interpolated value matrix\. We choose to keep it comparable toVV, notably for range correction which is described in §[4\.1\.1](https://arxiv.org/html/2610.40943#S4.SS1.SSS1)\.

Hence, while the key centering can be seen as a hyperparameter which allows us to select our exactness set throughℬδ\\mathcal\{B\}\_\{\\delta\}, it also affects our capacity to solve the interpolation problem and we will restrict ourselves to theμ\\mu\-centering in the rest of this article\. Indeed, while equivalent in exact arithmetic, the origin centering would require too much precision due to the redundant linear term while the mean centering keeps every direction resolvable \([21](https://arxiv.org/html/2610.40943#S2.E21)\)\. Furthermore, the mean centering allows to have an interpolation basis which may be more representative of the future queries as they tend to not align exactly with the keys\.

## 3Empirical interpolation in infinite\-dimensional spaces

As discussed in the previous section, interpolation can be divided into two distinct but connected tasks, the*Selection*of the basis keys, i\.e\. constructing a basis spaceℬ\\mathcal\{B\}, and the*Projection*of the augmented linear map onto it, i\.e\. approximatingKϕ𝖳​VK\_\{\\phi\}^\{\\mathsf\{T\}\}Vthrough a Petrov\-Galerkin*Projection*by minimising the residual on a test space𝒯\\mathcal\{T\}that is potentially different fromℬ\\mathcal\{B\}\.

### 3\.1Basis selection

The selection chooses the basis spaceℬ\\mathcal\{B\}of the RKHS, as previously described in §[2\.2](https://arxiv.org/html/2610.40943#S2.SS2)\. Projection determines the linear operator that maps the input \(projected onto the basis\), and thus the output space\. In this section, we first define the residual operators and error terms \(§[3\.1\.1](https://arxiv.org/html/2610.40943#S3.SS1.SSS1)\), and present two selectors:

- •The GAIM selector in §[3\.1\.2](https://arxiv.org/html/2610.40943#S3.SS1.SSS2)is based on greedily selecting one after the other the interpolation nodes with the highest residuals in theL∞L^\{\\infty\}norm, iteratively removing the components with the highest error in any output dimension, following the GEIM method\[[MM13](https://arxiv.org/html/2610.40943#bib.bibx33)\]\. We also provide anL2L^\{2\}norm alternative that chooses the directions that minimize the sum of squared errors\.
- •The Q\-DAIM selector in §[3\.1\.3](https://arxiv.org/html/2610.40943#S3.SS1.SSS3)is based on the Q\-DEIM\[[DS18](https://arxiv.org/html/2610.40943#bib.bibx15),[DG16](https://arxiv.org/html/2610.40943#bib.bibx13)\]method which reframes theL2L^\{2\}formulation of the residual minimization problem in the metric induced by the linear map\. This is done by constructing an orthogonal basis for the linear map through either a CholeskyQR\[[SW02](https://arxiv.org/html/2610.40943#bib.bibx47)\]or through the Singular Value Decomposition, a method known in the data\-science literature as Cholesky or PCA whitening respectively\[[KLS18](https://arxiv.org/html/2610.40943#bib.bibx27)\]\. The algorithm then selects nodes to maximise the volume of the approximation in the obtained basis\.

We also present an approach for taking into account multiple query heads inL2L^\{2\}norm using a multi\-head scoring metric in §[3\.1\.4](https://arxiv.org/html/2610.40943#S3.SS1.SSS4)\. Finally, we present approximate finite\-dimensional surrogates for the computation of the residuals that allow a linear complexity in terms of context size in §[3\.1\.5](https://arxiv.org/html/2610.40943#S3.SS1.SSS5)and thus allow for the compression of longer contexts\. Let us now take a look at the residuals we want to minimize\.

#### 3\.1\.1Approximation of the residual error

Consider the numerator and denominator of the attention output that we denoteg⁡\(q\)=ϕ​\[q\]𝖳​Mg∈ℝdg\(q\)=\\phi\[q\]^\{\\mathsf\{T\}\}M^\{g\}\\in\\mathbb\{R\}^\{d\}andh⁡\(q\)=ϕ​\[q\]𝖳​Mh∈ℝ\+h\(q\)=\\phi\[q\]^\{\\mathsf\{T\}\}M^\{h\}\\in\\mathbb\{R\}^\{\+\}, where we setMg:=Kϕ𝖳​VM^\{g\}:=K\_\{\\phi\}^\{\\mathsf\{T\}\}VandMh=Kϕ𝖳​𝟏NM^\{h\}=K\_\{\\phi\}^\{\\mathsf\{T\}\}\\mathbf\{1\}\_\{N\}\. The ratiog⁡\(q\)/h⁡\(q\)=ϕ​\[q\]𝖳​Mgϕ​\[q\]𝖳​Mhg\(q\)/h\(q\)=\\frac\{\\phi\[q\]^\{\\mathsf\{T\}\}M^\{g\}\}\{\\phi\[q\]^\{\\mathsf\{T\}\}M^\{h\}\}is therefore determined by applying the combined*augmented*linear operator toϕ⁡\[q\]\\phi\[q\], this operator is defined as

M:=Kϕ𝖳​Vaug=\[Mg∣Mh\]\.M:=K\_\{\\phi\}^\{\\mathsf\{T\}\}V^\{\\mathrm\{aug\}\}=\\big\[M^\{g\}\\mid M^\{h\}\\big\]\.\(25\)For interpolated attention, we denoteg~=ϕ​\[q\]𝖳​MLg\\tilde\{g\}=\\phi\[q\]^\{\\mathsf\{T\}\}M\_\{L\}^\{g\}andh~=ϕ​\[q\]𝖳​MLh\\tilde\{h\}=\\phi\[q\]^\{\\mathsf\{T\}\}M\_\{L\}^\{h\}similarly, whereML:=ΠLϕ​M=Lϕ𝖳​WaugM\_\{L\}:=\\Pi\_\{L\_\{\\phi\}\}M=L\_\{\\phi\}^\{\\mathsf\{T\}\}W^\{\\mathrm\{aug\}\}\. We can then write that

Att⁡\(q,K,V\)=g⁡\(q\)/h⁡\(q\)andAtt~L​\(q,K,V\)=g~​\(q\)/h~​\(q\)\.\\operatorname\{Att\}\(q,K,V\)=g\(q\)/h\(q\)\\quad\\text\{ and \}\\quad\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(q,K,V\)=\\tilde\{g\}\(q\)/\\tilde\{h\}\(q\)\.UsingΠLϕ​Kϕ𝖳=Lϕ𝖳​EL​L†​EL​K\\Pi\_\{L\_\{\\phi\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}=L\_\{\\phi\}^\{\\mathsf\{T\}\}E\_\{LL\}^\{\\dagger\}E\_\{LK\}and the weights \([7](https://arxiv.org/html/2610.40943#S2.E7)\), the decomposition \([12](https://arxiv.org/html/2610.40943#S2.E12)\) splitsMMinto its component in the basis spaceℬ\\mathcal\{B\}and its orthogonal complement inSpan⁡\(Kϕ\)\\operatorname\{Span\}\(K\_\{\\phi\}\),

M=ML\+MΔ,withMΔ:=ΠKϕ⊖Lϕ​M=M−Lϕ𝖳​Waug\.M=M\_\{L\}\+M\_\{\\Delta\},\\qquad\\text\{ with \}\\qquad M\_\{\\Delta\}:=\\Pi\_\{K\_\{\\phi\}\\ominus L\_\{\\phi\}\}M=M\-L\_\{\\phi\}^\{\\mathsf\{T\}\}W^\{\\mathrm\{aug\}\}\.\(26\)We callMΔM\_\{\\Delta\}the*augmented residual operator*\. Indeed, the residual error ofggandhhdue to the interpolation method are given by the firstddcolumns and the last column ofMΔ=\[MΔg∣MΔh\]M\_\{\\Delta\}=\\big\[M^\{g\}\_\{\\Delta\}\\mid M^\{h\}\_\{\\Delta\}\\big\]respectively\. The residual error is then given by

Δ​g=g−g~=ϕ​\[q\]𝖳​MΔg,and​Δ​h=h−h~=ϕ​\[q\]𝖳​MΔh\.\\Delta g=g\-\\tilde\{g\}=\\phi\[q\]^\{\\mathsf\{T\}\}M^\{g\}\_\{\\Delta\},\\quad\\text\{ and \}\\Delta h=h\-\\tilde\{h\}=\\phi\[q\]^\{\\mathsf\{T\}\}M^\{h\}\_\{\\Delta\}\.The residual error on the attention output is then obtained as

Att−Att~L=gh−g~h~=\(gh−gh~\)\+\(gh~−g~h~\)=−gh​h~​Δ​h\+Δ​gh~=1h~​\(Δ​g−gh​Δ​h\)\.\\operatorname\{Att\}\-\\widetilde\{\\operatorname\{Att\}\}\_\{L\}=\\frac\{g\}\{h\}\-\\frac\{\\tilde\{g\}\}\{\\tilde\{h\}\}=\\Big\(\\frac\{g\}\{h\}\-\\frac\{g\}\{\\tilde\{h\}\}\\Big\)\+\\Big\(\\frac\{g\}\{\\tilde\{h\}\}\-\\frac\{\\tilde\{g\}\}\{\\tilde\{h\}\}\\Big\)=\-\\frac\{g\}\{h\\tilde\{h\}\}\\Delta h\+\\frac\{\\Delta g\}\{\\tilde\{h\}\}=\\frac\{1\}\{\\tilde\{h\}\}\\Big\(\\Delta g\-\\frac\{g\}\{h\}\\Delta h\\Big\)\.This admits the closed form

Att−Att~L=1ϕ​\[q\]𝖳​Lϕ𝖳​WNorm⏟T3​\(ϕ​\[q\]𝖳​MΔg⏟T1−ϕ​\[q\]𝖳​Mgϕ​\[q\]𝖳​Mh​ϕ​\[q\]𝖳​MΔh⏟T2\)\.\\operatorname\{Att\}\-\\widetilde\{\\operatorname\{Att\}\}\_\{L\}=\\frac\{1\}\{\\underbrace\{\\phi\[q\]^\{\\mathsf\{T\}\}L\_\{\\phi\}^\{\\mathsf\{T\}\}W\_\{\\mathrm\{Norm\}\}\}\_\{T\_\{3\}\}\}\\Big\(\\underbrace\{\\phi\[q\]^\{\\mathsf\{T\}\}M^\{g\}\_\{\\Delta\}\}\_\{T\_\{1\}\}\-\\underbrace\{\\frac\{\\phi\[q\]^\{\\mathsf\{T\}\}M^\{g\}\}\{\\phi\[q\]^\{\\mathsf\{T\}\}M^\{h\}\}\\phi\[q\]^\{\\mathsf\{T\}\}M^\{h\}\_\{\\Delta\}\}\_\{T\_\{2\}\}\\Big\)\.\(27\)
This formulation allows the usage of the residual given in \([27](https://arxiv.org/html/2610.40943#S3.E27)\) for interpolation\. Indeed, each of the three terms depends only on the query through the prefactorϕ​\[q\]𝖳\\phi\[q\]^\{\\mathsf\{T\}\}and is linear inMΔM\_\{\\Delta\},MΔhM\_\{\\Delta\}^\{h\}andMLhM\_\{L\}^\{h\}, respectively\. Hence, for any new nodex∈ℝdx\\in\\mathbb\{R\}^\{d\}, we can update the residual to the new setL′=L∪\{x\}L^\{\\prime\}=L\\cup\\\{x\\\}using rank\-11updates ofMΔM\_\{\\Delta\}andMLM\_\{L\}\. Indeed, letρ⁡\[x\]:=ΠKϕ⊖Lϕ​ϕ​\[x\]\\rho\[x\]:=\\Pi\_\{K\_\{\\phi\}\\ominus L\_\{\\phi\}\}\\phi\[x\]be the projection of the new node onSpan⁡\(Kϕ\)⊖Span⁡\(Lϕ\)\\operatorname\{Span\}\(K\_\{\\phi\}\)\\ominus\\operatorname\{Span\}\(L\_\{\\phi\}\)and setsL​\(a,b\):=⟨a,ΠKϕ⊖Lϕ​b⟩H=Ea​b−Ea​L​EL​L†​EL​bs\_\{L\}\(a,b\):=\\langle a,\\Pi\_\{K\_\{\\phi\}\\ominus L\_\{\\phi\}\}b\\rangle\_\{H\}=E\_\{ab\}\-E\_\{aL\}E\_\{LL\}^\{\\dagger\}E\_\{Lb\}, the scalar product inHHrestricted toSpan⁡\(Kϕ\)⊖Span⁡\(Lϕ\)\\operatorname\{Span\}\(K\_\{\\phi\}\)\\ominus\\operatorname\{Span\}\(L\_\{\\phi\}\)\. The norm ofρ⁡\[x\]\\rho\[x\]is given by the norm ofsL​\(x,x\)=Ex​x−Ex​L​EL​L†​EL​xs\_\{L\}\(x,x\)=E\_\{xx\}\-E\_\{xL\}E\_\{LL\}^\{\\dagger\}E\_\{Lx\}\. Then the projection ontoSpan⁡\(Lϕ′\)\\operatorname\{Span\}\(L^\{\\prime\}\_\{\\phi\}\)is given byΠLϕ′=ΠLϕ\+ρ⁡\[x\]​ρ​\[x\]𝖳/sL​\(x,x\)\\Pi\_\{L^\{\\prime\}\_\{\\phi\}\}=\\Pi\_\{L\_\{\\phi\}\}\+\\rho\[x\]\\rho\[x\]^\{\\mathsf\{T\}\}/s\_\{L\}\(x,x\)and we have

ML′=ML\+ρ⁡\[x\]​rxsL​\(x,x\),MΔ′=MΔ−ρ⁡\[x\]​rxsL​\(x,x\),rx:=ϕ​\[x\]𝖳​ΠKϕ⊖Lϕ​M=ϕ​\[x\]𝖳​MΔ=\[rxgrxh\]\.M\_\{L^\{\\prime\}\}=M\_\{L\}\+\\frac\{\\rho\[x\]r\_\{x\}\}\{s\_\{L\}\(x,x\)\},\\qquad M\_\{\\Delta\}^\{\\prime\}=M\_\{\\Delta\}\-\\frac\{\\rho\[x\]r\_\{x\}\}\{s\_\{L\}\(x,x\)\},\\qquad r\_\{x\}:=\\phi\[x\]^\{\\mathsf\{T\}\}\\Pi\_\{K\_\{\\phi\}\\ominus L\_\{\\phi\}\}M=\\phi\[x\]^\{\\mathsf\{T\}\}M\_\{\\Delta\}=\\begin\{bmatrix\}r^\{g\}\_\{x\}&r^\{h\}\_\{x\}\\end\{bmatrix\}\.At a queryqq, the updates to the three blocks are then obtained through the scalarsL​\(q,x\):=ϕ​\[q\]𝖳​ΠLϕ⟂​ϕ​\[x\]=ϕ​\[q\]𝖳​ρ​\[x\]s\_\{L\}\(q,x\):=\\phi\[q\]^\{\\mathsf\{T\}\}\\Pi\_\{L\_\{\\phi\}\}^\{\\perp\}\\phi\[x\]=\\phi\[q\]^\{\\mathsf\{T\}\}\\rho\[x\]:

T1′=T1−sL​\(q,x\)sL​\(x,x\)​rxg,T2′=T2−gh​sL​\(q,x\)sL​\(x,x\)​rxh,T3′=T3\+sL​\(q,x\)sL​\(x,x\)​rxh,T\_\{1\}^\{\\prime\}=T\_\{1\}\-\\frac\{s\_\{L\}\(q,x\)\}\{s\_\{L\}\(x,x\)\}r^\{g\}\_\{x\},\\qquad T\_\{2\}^\{\\prime\}=T\_\{2\}\-\\frac\{g\}\{h\}\\frac\{s\_\{L\}\(q,x\)\}\{s\_\{L\}\(x,x\)\}r^\{h\}\_\{x\},\\qquad T\_\{3\}^\{\\prime\}=T\_\{3\}\+\\frac\{s\_\{L\}\(q,x\)\}\{s\_\{L\}\(x,x\)\}r^\{h\}\_\{x\},\(28\)where we note that the values that need to be computed are the evaluations of

rx=\(Ex​K−Ex​LEL​L†EL​K\)Vaug,sL\(x,x\)=Ex​x−Ex​LEL​L†EL​xandsL\(q,x\)=Eq​x−Eq​LEL​L†EL​x\.r\_\{x\}=\\left\(E\_\{xK\}\-E\_\{xL\}E\_\{LL\}^\{\\dagger\}E\_\{LK\}\\right\)V^\{\\mathrm\{aug\}\},\\quad s\_\{L\}\(x,x\)=E\_\{xx\}\-E\_\{xL\}E\_\{LL\}^\{\\dagger\}E\_\{Lx\}\\quad\\text\{ and \}\\quad s\_\{L\}\(q,x\)=E\_\{qx\}\-E\_\{qL\}E\_\{LL\}^\{\\dagger\}E\_\{Lx\}\.Hence,T1T\_\{1\},T2T\_\{2\}andT3T\_\{3\}can be updated*separately*for each rank\-one extension ofLL, which is what makes our GEIM\-based greedy selection computationally feasible\. We note that the above quantities cannot be computed throughρ⁡\[x\]\\rho\[x\], which lives inHH\. They are thus obtained through kernel evaluations and projected through either the inverse of the node or using a solve from its Cholesky factorization\. Thus, we keep in memory either the inverse or a Cholesky factorization of the Gram matrix for the projection step\. With the addition of a new nodex∈ℝdx\\in\\mathbb\{R\}^\{d\}, we therefore use a rank\-11update for the addition of the new border

EL′​L′=\(EL​LEL​xEx​LEx​x\)\.E\_\{L^\{\\prime\}L^\{\\prime\}\}=\\begin\{pmatrix\}E\_\{LL\}&E\_\{Lx\}\\\\ E\_\{xL\}&E\_\{xx\}\\end\{pmatrix\}\.SettingsL​\(x,x\)=Ex​x−Ex​L​EL​L†​EL​xs\_\{L\}\(x,x\)=E\_\{xx\}\-E\_\{xL\}E\_\{LL\}^\{\\dagger\}E\_\{Lx\},u=EL​L†​EL​xu=E\_\{LL\}^\{\\dagger\}E\_\{Lx\}andΓL​L​ΓL​L𝖳=EL​L\\Gamma\_\{LL\}\\Gamma\_\{LL\}^\{\\mathsf\{T\}\}=E\_\{LL\}the Cholesky factorization, we can write

EL′​L′†=\(EL​L†\+u​u𝖳sL​\(x,x\)−usL​\(x,x\)−u𝖳sL​\(x,x\)1sL​\(x,x\)\),ΓL′​L′=\(ΓL​L0γ𝖳sL​\(x,x\)\),γ:=ΓL​L†​EL​x,E\_\{L^\{\\prime\}L^\{\\prime\}\}^\{\\dagger\}=\\begin\{pmatrix\}E\_\{LL\}^\{\\dagger\}\+\\dfrac\{uu^\{\\mathsf\{T\}\}\}\{s\_\{L\}\(x,x\)\}&\-\\dfrac\{u\}\{s\_\{L\}\(x,x\)\}\\\\\[6\.0pt\] \-\\dfrac\{u^\{\\mathsf\{T\}\}\}\{s\_\{L\}\(x,x\)\}&\\dfrac\{1\}\{s\_\{L\}\(x,x\)\}\\end\{pmatrix\},\\qquad\\Gamma\_\{L^\{\\prime\}L^\{\\prime\}\}=\\begin\{pmatrix\}\\Gamma\_\{LL\}&0\\\\ \\gamma^\{\\mathsf\{T\}\}&\\sqrt\{s\_\{L\}\(x,x\)\}\\end\{pmatrix\},\\quad\\gamma:=\\Gamma\_\{LL\}^\{\\dagger\}E\_\{Lx\},\(29\)both atO⁡\(n2\)O\(n^\{2\}\)cost in memory per node\. We note that in practice, the computations are done using the Gaussian formulation from equation \([15](https://arxiv.org/html/2610.40943#S2.E15)\), which gives the following\.

EL​L†=DLϕ−1/2E˘L​L†DLϕ−1/2\.E\_\{LL\}^\{\\dagger\}=D\_\{L\_\{\\phi\}\}^\{\-1/2\}\\breve\{E\}\_\{LL\}^\{\\dagger\}D\_\{L\_\{\\phi\}\}^\{\-1/2\}\.Thus, we can compute using the same formulae as the evaluations replaced by their Gaussian counterpartss˘L​\(a,b\)=sL​\(a,b\)‖ϕ⁡\[a\]‖H​‖ϕ⁡\[b\]‖H\\breve\{s\}\_\{L\}\(a,b\)=\\frac\{s\_\{L\}\(a,b\)\}\{\\\|\\phi\[a\]\\\|\_\{H\}\\\|\\phi\[b\]\\\|\_\{H\}\}andρ˘​\[x\]=ρ⁡\[x\]/‖ϕ⁡\[x\]‖H\\breve\{\\rho\}\[x\]=\\rho\[x\]/\\\|\\phi\[x\]\\\|\_\{H\}\. A useful corollary is that the terms˘L​\(x,x\):=1−E˘x​L​E˘L​L†​E˘L​x\\breve\{s\}\_\{L\}\(x,x\):=1\-\\breve\{E\}\_\{xL\}\\breve\{E\}\_\{LL\}^\{\\dagger\}\\breve\{E\}\_\{Lx\}can then be used to determine the collinearity of a new node in the selection loop\.

The residual is composed of three terms,T1=Δ​gT\_\{1\}=\\Delta gassociated with the linear residual of the numeratorMΔgM^\{g\}\_\{\\Delta\},T2=gh⋅Δ​hT\_\{2\}=\\frac\{g\}\{h\}\\cdot\\Delta hcorresponding to the linear effect of the residual of the denominatorMΔhM\_\{\\Delta\}^\{h\}on the error and the prefactor1T3\\frac\{1\}\{T\_\{3\}\}that corresponds to the multiplicative effect related to the interpolated denominator\. For our method, we consider five variants of the residual combining these terms:

- *\(v1\.\)*The value residualT1T\_\{1\},
- *\(v2\.\)*the unnormalised error residualT1−T2T\_\{1\}\-T\_\{2\},
- *\(v3\.\)*the full residualT1−T2T3\\frac\{T\_\{1\}\-T\_\{2\}\}\{T\_\{3\}\},
- *\(v4\.\)*thehh\-normalised full residualT1−T2h\\frac\{T\_\{1\}\-T\_\{2\}\}\{h\}, replacing the denominator by its exact value,
- *\(v5\.\)*and the augmented\-moment residualϕ​\[q\]𝖳​MΔ\\phi\[q\]^\{\\mathsf\{T\}\}M\_\{\\Delta\}, which corresponds to the residual of the interpolation map itself\.

We also consider two possible norms,L∞L^\{\\infty\}andL2L^\{2\}, the first one more suited to the GEIM\-inspired method and the second one to the Q\-DEIM method of the next subsection\.

#### 3\.1\.2Greedy selection of the magic points by GEIM

Following the generalized empirical interpolation method\[[MM13](https://arxiv.org/html/2610.40943#bib.bibx33)\]\(and the original empirical interpolation method\), we constructLLby greedily selecting the nodes from a dictionary of vectors that iteratively best minimize one of the residual variants v1\-v5, evaluated at the Gaussian directions of the keys,ϕ˘​\[Ki\]\\breve\{\\phi\}\[K\_\{i\}\]\. We name this the*greedy attention interpolation method*\(GAIM\)\. In this paper, we restrict the dictionary to the rows ofKK, which allows the error to be restricted toSpan⁡\(Kϕ\)⊖Span⁡\(Lϕ\)\\operatorname\{Span\}\(K\_\{\\phi\}\)\\ominus\\operatorname\{Span\}\(L\_\{\\phi\}\), as mentioned in §[2\.2](https://arxiv.org/html/2610.40943#S2.SS2)\.

In practice, while we choose our dictionary of possible directions as the set of keys itself, it should be noted that any arbitrary set of vectors ofℝd\\mathbb\{R\}^\{d\}may be chosen for this method\. Furthermore, one might ideally want to obtain the vector inHHmaximizing the residual\. However, constructing such a vector often requires access to all the keys as we can only reconstruct it through the feature mapϕ\\phi\.

LetVηaug:=Diag⁡\(η\)​Vaug=\[Vη∣η\]V^\{\\mathrm\{aug\}\}\_\{\\eta\}:=\\operatorname\{Diag\}\(\\eta\)V^\{\\mathrm\{aug\}\}=\[V\_\{\\eta\}\\mid\\eta\], whereη\\etaare the gauged masses defined in \([19](https://arxiv.org/html/2610.40943#S2.E19)\)\. We consider theη\\eta\-gauged augmented operator and residual operator equivalent to \([25](https://arxiv.org/html/2610.40943#S3.E25)\),

Mη:=K˘ϕ𝖳​Vηaug=M∥DK1/2∥∞,MΔ,η:=ΠKϕ⊖Lϕ​Mη=MΔ∥DK1/2∥∞\.M\_\{\\eta\}:=\\breve\{K\}\_\{\\phi\}^\{\\mathsf\{T\}\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}=\\frac\{M\}\{\\lVert D\_\{K\}^\{1/2\}\\rVert\_\{\\infty\}\},\\qquad M\_\{\\Delta,\\eta\}:=\\Pi\_\{K\_\{\\phi\}\\ominus L\_\{\\phi\}\}M\_\{\\eta\}=\\frac\{M\_\{\\Delta\}\}\{\\lVert D\_\{K\}^\{1/2\}\\rVert\_\{\\infty\}\}\.Thus, in the following proposition, every residual score is expressed in theL∞L^\{\\infty\}gauge\.

###### Proposition 6\(Residual scores\)\.

LetS˘L:=E˘K​K−E˘K​L​E˘L​L†​E˘L​K\\breve\{S\}\_\{L\}:=\\breve\{E\}\_\{KK\}\-\\breve\{E\}\_\{KL\}\\breve\{E\}\_\{LL\}^\{\\dagger\}\\breve\{E\}\_\{LK\}be the projection onto the orthogonal complement ofSpan⁡\(Lϕ\)\\operatorname\{Span\}\(L\_\{\\phi\}\)of the Gaussian Gram matrixE˘K​K\\breve\{E\}\_\{KK\}\. ThenS˘L=K˘ϕ​ΠKϕ⊖Lϕ​K˘ϕ𝖳⪰0\\breve\{S\}\_\{L\}=\\breve\{K\}\_\{\\phi\}\\Pi\_\{K\_\{\\phi\}\\ominus L\_\{\\phi\}\}\\breve\{K\}\_\{\\phi\}^\{\\mathsf\{T\}\}\\succeq 0and for everyi∈\[N\]i\\in\[N\], the residual of the linear mapMMevaluated on the keysKKis given by

RΔ\[i,:\]:=⟨ϕ˘\[Ki\],MΔ,η⟩H=\[S˘LVηaug\]i,:∈ℝd\+1\.R\_\{\\Delta\}\[i,:\]:=\\Big\\langle\\breve\{\\phi\}\[K\_\{i\}\],M\_\{\\Delta,\\eta\}\\Big\\rangle\_\{H\}=\\big\[\\breve\{S\}\_\{L\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}\\big\]\_\{i,:\}\\in\\mathbb\{R\}^\{d\+1\}\.

The proof is given in Appendix[A](https://arxiv.org/html/2610.40943#A1)\. Then every residual variant can be assembled fromRΔR\_\{\\Delta\}, and fromR0:=E˘K​K​VηaugR^\{0\}:=\\breve\{E\}\_\{KK\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}, the evaluation of the augmented operator on the keys\. Indeed, the rowiiof these matrices gives us

gi=R0\[i,:d\],hi=R0\[i,d\+1\]andΔgi=RΔ\[i,:d\],Δhi=RΔ\[i,d\+1\]\.g\_\{i\}=R^\{0\}\[i,:d\],\\quad h\_\{i\}=R^\{0\}\[i,d\+1\]\\quad\\text\{ and \}\\quad\\Delta g\_\{i\}=R\_\{\\Delta\}\[i,:d\],\\quad\\Delta h\_\{i\}=R\_\{\\Delta\}\[i,d\+1\]\.Hence, we can updateT1​\(i\)=Δ​giT\_\{1\}\(i\)=\\Delta g\_\{i\},T2​\(i\)=\(gi/hi\)​Δ​hiT\_\{2\}\(i\)=\(g\_\{i\}/h\_\{i\}\)\\Delta h\_\{i\}andT3​\(i\)=hi−Δ​hiT\_\{3\}\(i\)=h\_\{i\}\-\\Delta h\_\{i\}, at a costO⁡\(N⁡\(d\+1\)\)O\(N\(d\+1\)\)per pick\.

LetRselR\_\{\\mathrm\{sel\}\}be the matrix of the chosen residual variant and the normp∈\{∞,2\}p\\in\\\{\\infty,2\\\}, the next node is obtained using

i⋆=argmaxi∉L∥Rsel\[i,:\]∥p,L←L∪\{Ki⋆\},i^\{\\star\}=\\arg\\max\_\{i\\notin L\}\\big\\lVert R\_\{\\mathrm\{sel\}\}\[i,:\]\\big\\rVert\_\{p\},\\qquad L\\leftarrow L\\cup\\\{K\_\{i^\{\\star\}\}\\\},\(30\)where for simplicity, we denote, for anyv∈ℝdv\\in\\mathbb\{R\}^\{d\}andA∈ℝa×dA\\in\\mathbb\{R\}^\{a\\times d\}, byA∪vA\\cup vthe concatenation of the vectorvvat the end ofAA\. We note that, similarly to GEIM, this is a greedy method which selects the node with the largest current residual, which is not guaranteed to be the optimal selection to reduce the residual\. Indeed, for the Galerkin residual, adding the keyKiK\_\{i\}decreases‖MΔ,η‖HS2\\\|M\_\{\\Delta,\\eta\}\\\|\_\{\\mathrm\{HS\}\}^\{2\}by∥RΔ\[i,:\]∥22/S˘L\[i,i\]\\\|R\_\{\\Delta\}\[i,:\]\\\|\_\{2\}^\{2\}/\\breve\{S\}\_\{L\}\[i,i\]\.

In practice, we note that for the pairing \(v5\.,L∞L^\{\\infty\}\) the addition of the denominator column does not seem to work well with the infinity norm\.

The exact\-kernel selection loop is summarized in Algorithm[1](https://arxiv.org/html/2610.40943#alg1)\. The rank\-1 residual updates, computed forq∈\(Ki\)i∈\[N\]q\\in\(K\_\{i\}\)\_\{i\\in\[N\]\}using \([28](https://arxiv.org/html/2610.40943#S3.E28)\), have only a linear cost inNN\. The overall complexity is therefore of orderO⁡\(N2​d\+N​n​\(d\+n\)\)O\(N^\{2\}d\+Nn\(d\+n\)\)\. Indeed, we only maintain the residual rows andS˘L\\breve\{S\}\_\{L\}, although at the cost of keeping then×nn\\times nmatrixEL​L†E\_\{LL\}^\{\\dagger\}in memory\. The quadratic cost of the method comes from the initialization ofR0R^\{0\}which requires computingEK​KE\_\{KK\}and hence the scalar product between each pair of keys for a total ofN2N^\{2\}kernel evaluations\.

Algorithm 1Exact\-kernel greedy GAIM selection1:keys

K∈ℝN×dK\\in\\mathbb\{R\}^\{N\\times d\}, values

V∈ℝN×dV\\in\\mathbb\{R\}^\{N\\times d\}, budget

nn, residual variant

RselR\_\{\\mathrm\{sel\}\}, norm

pp, collinearity floor

τσ\\tau\_\{\\sigma\}
2:

Vaug←\[V∣𝟏N\]V^\{\\mathrm\{aug\}\}\\leftarrow\[V\\mid\\mathbf\{1\}\_\{N\}\];

η←\\eta\\leftarrowgauged masses of \([19](https://arxiv.org/html/2610.40943#S2.E19)\), in log space;

𝔏←∅\\mathfrak\{L\}\\leftarrow\\varnothing;

E˘L​L†←\[\]\\breve\{E\}\_\{LL\}^\{\\dagger\}\\leftarrow\[\]
3:

E˘K​K←\[exp\(−∥Ki−Kj∥2/2d\)\]i,j∈\[N\]\\breve\{E\}\_\{KK\}\\leftarrow\\big\[\\exp\(\-\\\|K\_\{i\}\-K\_\{j\}\\\|^\{2\}/2\\sqrt\{d\}\)\\big\]\_\{i,j\\in\[N\]\};

R0←E˘K​K​Diag⁡\(η\)​VaugR^\{0\}\\leftarrow\\breve\{E\}\_\{KK\}\\operatorname\{Diag\}\(\\eta\)V^\{\\mathrm\{aug\}\};

RΔ←R0R\_\{\\Delta\}\\leftarrow R^\{0\}⊳\\trianglerightGram, evaluations and residual rows atL=∅L=\\varnothing

4:for

t=1,…,nt=1,\\dots,ndo

5:Update

RselR\_\{\\mathrm\{sel\}\}from

\(RΔ,R0\)\(R\_\{\\Delta\},R^\{0\}\)⊳\\trianglerightchosen residual variant, §[3\.1\.2](https://arxiv.org/html/2610.40943#S3.SS1.SSS2)

6:

i⋆←argmaxi∉𝔏∥\[Rsel\]i,:∥pi^\{\\star\}\\leftarrow\\arg\\max\_\{i\\notin\\mathfrak\{L\}\}\\lVert\[R\_\{\\mathrm\{sel\}\}\]\_\{i,:\}\\rVert\_\{p\}⊳\\trianglerightselection in chosen norm

7:

ui⋆←\[E˘K​K\]:,i⋆−E˘K​LE˘L​L†\[E˘L​K\]:,i⋆u\_\{i^\{\\star\}\}\\leftarrow\[\\breve\{E\}\_\{KK\}\]\_\{:,i^\{\\star\}\}\-\\breve\{E\}\_\{KL\}\\breve\{E\}\_\{LL\}^\{\\dagger\}\[\\breve\{E\}\_\{LK\}\]\_\{:,i^\{\\star\}\};

σ←\[ui⋆\]i⋆\\sigma\\leftarrow\[u\_\{i^\{\\star\}\}\]\_\{i^\{\\star\}\}; check

σ≥τσ\\sigma\\geq\\tau\_\{\\sigma\}⊳\\trianglerightSchur complement column, collinearity check

8:

RΔ←RΔ−ui⋆\[RΔ\]i⋆,:/σR\_\{\\Delta\}\\leftarrow R\_\{\\Delta\}\-u\_\{i^\{\\star\}\}\[R\_\{\\Delta\}\]\_\{i^\{\\star\},:\}/\\sigma⊳\\trianglerightOrthogonalisation

9:Grow

E˘L​L†\\breve\{E\}\_\{LL\}^\{\\dagger\}by the border of

i⋆i^\{\\star\}with pivot

σ\\sigma;

𝔏←𝔏∪\{i⋆\}\\mathfrak\{L\}\\leftarrow\\mathfrak\{L\}\\cup\\\{i^\{\\star\}\\\}⊳\\trianglerightrank\-11bordered update, §[3\.1\.1](https://arxiv.org/html/2610.40943#S3.SS1.SSS1)

10:endfor

11:returnnodes

L=K\[𝔏,:\]L=K\[\\mathfrak\{L\},:\], inverse

E˘L​L†\\breve\{E\}\_\{LL\}^\{\\dagger\}

#### 3\.1\.3Greedy selection by rank\-revealed Q\-DEIM

While the GAIM method greedily minimizes the residual for any given norm, the*Q\-DEIM\-based Attention Interpolation Method*\(Q\-DAIM\) only targets the residual in the normL2L^\{2\}, similar to Q\-DEIM\[[DG16](https://arxiv.org/html/2610.40943#bib.bibx13)\]\. To do so, instead of considering the residual in its original space, we consider the Cholesky\-whitened\[[KLS18](https://arxiv.org/html/2610.40943#bib.bibx27)\]counterpart of the augmented operator, i\.e\. the orthogonal projector onto the range ofMηM\_\{\\eta\}obtained through a CholeskyQR\. Then we follow the Q\-DEIM methodology and iteratively choose our nodes greedily to maximize the volume spanned in the orthogonal basis, i\.e\. we choose the keys having the highest overlap with the basis vectors via the pivots of a greedily pivoted QR \(or for more stability a rank\-revealing pivoted QR with theoretical guarantees\)\. Furthermore, as we have the projection of the keys in an orthogonal basis, compared to the GAIM method, there is no need to computeΠL\\Pi\_\{L\}for the orthogonalisation procedure of the operator and all nodes can be selected sequentially without having to update a factorisation of the Gram matrixEL​LE\_\{LL\}\. However, this comes at the cost of only being able to consider linear residuals, thus being restricted to v1 and v5\. Note that we use this method on the Gaussian formulation of the problem from \([16](https://arxiv.org/html/2610.40943#S2.E16)\), allowing it to take into account the metric induced by bothVaugV^\{\\mathrm\{aug\}\}and by the mass ratiosη\\eta, resulting in a better conditioned problem\.

Let us now look at how we implement this in the context of attention\. As mentioned before, we wish to find the orthogonal projector onto the range ofMηM\_\{\\eta\}\. As the augmented operatorMηM\_\{\\eta\}has rankrrbounded bymin⁡\(N,d\+1\)\\min\(N,d\+1\), the range ofMηM\_\{\\eta\}is a subspace ofHHof dimensionrrand there exists a decomposition

Mη=K˘ϕ𝖳​Vηaug=U​RM\_\{\\eta\}=\\breve\{K\}^\{\\mathsf\{T\}\}\_\{\\phi\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}=URwhereU:ℝr→HU:\\mathbb\{R\}^\{r\}\\to His an isometry whose columns span the range ofMηM\_\{\\eta\}, andR∈ℝr×\(d\+1\)R\\in\\mathbb\{R\}^\{r\\times\(d\+1\)\}\. Ideally, we would obtain this decomposition from a rank\-revealing QR decomposition ofMηM\_\{\\eta\}\. However, due to the infinite\-dimensional nature of the problem, we will need to resort to alternative formulations to useUU\.

To obtain the factorisation, we consider a method known as whitening\[[KLS18](https://arxiv.org/html/2610.40943#bib.bibx27)\], which groups CholeskyQR\-like\[[Hig90](https://arxiv.org/html/2610.40943#bib.bibx24)\]methods where, given the Gram matrix

G=Mη𝖳​Mη=Vηaug𝖳​E˘K​K​Vηaug=R𝖳​R∈ℝ\(d\+1\)×\(d\+1\),G=M\_\{\\eta\}^\{\\mathsf\{T\}\}M\_\{\\eta\}=\{V^\{\\mathrm\{aug\}\}\_\{\\eta\}\}^\{\\mathsf\{T\}\}\\breve\{E\}\_\{KK\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}=R^\{\\mathsf\{T\}\}R\\in\\mathbb\{R\}^\{\(d\+1\)\\times\(d\+1\)\},\(31\)we compute a factorization with permutationsPR𝖳​G​PR=R𝖳​RP\_\{R\}^\{\\mathsf\{T\}\}GP\_\{R\}=R^\{\\mathsf\{T\}\}Rand apply the \(pseudo\-\)inverse ofRRtoMηM\_\{\\eta\}to obtainUU\. Indeed,GGis symmetric and positive semi\-definite as the Gram of the column ‘vectors’ ofMηM\_\{\\eta\}, furthermore its rank is upper bounded bymin⁡\(N,d\+1\)\\min\(N,d\+1\)\. To obtainR∈ℝr×\(d\+1\)R\\in\\mathbb\{R\}^\{r\\times\(d\+1\)\}, we thus consider a*rank\-revealing*diagonally pivoted Cholesky\[[Hig90](https://arxiv.org/html/2610.40943#bib.bibx24)\]where we pivot on the largest remaining element on the diagonal of the residual, and stop when it falls belowτchol\\tau\_\{\\mathrm\{chol\}\}times the largest diagonal entry ofGG\. Then, the whitened coefficient matrix is given by

C=Vηaug​PR​R†∈ℝN×r,whereC𝖳​E˘K​K​C=Ir\.C=V^\{\\mathrm\{aug\}\}\_\{\\eta\}P\_\{R\}R^\{\\dagger\}\\in\\mathbb\{R\}^\{N\\times r\},\\quad\\text\{ where \}\\quad C^\{\\mathsf\{T\}\}\\breve\{E\}\_\{KK\}C=I\_\{r\}\.\(32\)This then gives us the orthogonal operator

U=K˘ϕ𝖳​C,withU𝖳​U=Ir\.U=\\breve\{K\}\_\{\\phi\}^\{\\mathsf\{T\}\}C,\\quad\\text\{ with \}\\quad U^\{\\mathsf\{T\}\}U=I\_\{r\}\.\(33\)We note that the identitiesC𝖳​E˘K​K​C=IrC^\{\\mathsf\{T\}\}\\breve\{E\}\_\{KK\}C=I\_\{r\}andU𝖳​U=IrU^\{\\mathsf\{T\}\}U=I\_\{r\}hold when the Cholesky factorization reveals the exact rank ofGG\. When the factorization is stopped earlier, we instead havePR𝖳​G​PR=R𝖳​R\+EP\_\{R\}^\{\\mathsf\{T\}\}GP\_\{R\}=R^\{\\mathsf\{T\}\}R\+EwithE⪰0E\\succeq 0, such thatU𝖳​U=Ir\+R†𝖳​E​R†U^\{\\mathsf\{T\}\}U=I\_\{r\}\+\{R^\{\\dagger\}\}^\{\\mathsf\{T\}\}ER^\{\\dagger\}\. In this caseUUis only approximately orthonormal\. An exactly orthonormal basis can then be recovered from the pivoted columns only,U=MηPR\[:,1:r\]R11−1U=M\_\{\\eta\}P\_\{R\}\[:,1:r\]R\_\{11\}^\{\-1\}, whereR11R\_\{11\}denotes the leadingr×rr\\times rblock ofRR\. An error is then introduced due to the truncation procedure, it corresponds to‖ΠU⟂​Mη‖2=‖\(I−U​U𝖳\)​Mη‖2\\\|\\Pi\_\{U\}^\{\\perp\}M\_\{\\eta\}\\\|^\{2\}=\\\|\(I\-UU^\{\\mathsf\{T\}\}\)M\_\{\\eta\}\\\|^\{2\}\.

In practice, sinceGGis of size\(d\+1\)×\(d\+1\)\(d\+1\)\\times\(d\+1\), the rank revealed by the Cholesky factorization always satisfiesr≤d\+1r\\leq d\+1\. When the budget is small and we have2​n\+p≤d\+12n\+p\\leq d\+1, for a chosen oversamplingp≥0p\\geq 0, we stop the Cholesky factorization atr=2​n\+pr=2n\+p, even if the threshold has not yet been reached, in which caseUUis only approximately orthonormal, as noted above\. Furthermore, while it might seem reasonable to take small thresholds, this can sometimes amplify noise as each vector in the revealed range provides an equal weight in the selection criterion\. We also note that any alternative square\-root ofGGmay be used for the whitening, the most relevant alternative consisting in directly computing the counterpart toR†R^\{\\dagger\}obtained from the eigendecomposition ofGG\.

Compared to the unwhitened selection ofL2L^\{2\}GAIM\. Q\-DAIM appears to be less sensitive to the keys with the largest kernel norms which tend to monopolize the pivots in GAIM\. In the same manner, applying the method with the augmented value matrix \(i\.e\. v5\.\) is also impactful, as it allows the map of the denominator to be taken into account in the metric\.

For the selection procedure, we consider the projection of the key vectors onto the range ofMηM\_\{\\eta\},

F=K˘ϕU=E˘K​KC∈ℝN×r,F\[i,:\]=⟨ϕ˘\[Ki\],U⟩,F=\\breve\{K\}\_\{\\phi\}U=\\breve\{E\}\_\{KK\}C\\in\\mathbb\{R\}^\{N\\times r\},\\qquad F\[i,:\]=\\big\\langle\\breve\{\\phi\}\[K\_\{i\}\],U\\big\\rangle,\(34\)and a column\-pivoted QR \(Businger\-Golub\[[BG65](https://arxiv.org/html/2610.40943#bib.bibx3)\]\) is applied toF𝖳F^\{\\mathsf\{T\}\}\. We setP=\(pi\)i∈\[n\]⊂\[N\]P=\(p\_\{i\}\)\_\{i\\in\[n\]\}\\subset\[N\]to be the indices of thennselected pivots such thatK\[P,:\]=LK\[P,:\]=L\. However, when the node budgetn≥rn\\geq r, the column\-pivoted QR described above only gives us at mostrrinformative pivots, the remaining selections being chosen on numerical noise\. Hence, for budgetsn\>rn\>r, we consider optional restart procedures, as described in Appendix[E](https://arxiv.org/html/2610.40943#A5)\.

An alternative to the fixed number of selected nodes for interpolation is to instead consider keeping pivots as long as the addition of a new pivot improves the reconstruction above a certain threshold, i\.e\. keep pivots only whileνℓ2/ν12≥τ\\nu\_\{\\ell\}^\{2\}/\\nu\_\{1\}^\{2\}\\geq\\tau, whereνℓ\\nu\_\{\\ell\}denotes the remaining norm of theℓ\\ell\-th pivot of the QR\-CP ofF𝖳F^\{\\mathsf\{T\}\}, or alternatively while the relative operator energy missed by the nodes remains above a threshold\.

Finally, our Q\-DAIM interpolant444Note that, writingℐL∗=UFP,:†L˘ϕ=U\(L˘ϕU\)†L˘ϕ\\mathcal\{I\}\_\{L\}^\{\*\}=UF\_\{P,:\}^\{\\dagger\}\\breve\{L\}\_\{\\phi\}=U\(\\breve\{L\}\_\{\\phi\}U\)^\{\\dagger\}\\breve\{L\}\_\{\\phi\}for the usual DEIM operator onHH, the operator we use for the interpolation is thus the adjoint of the usual DEIM operator, i\.e\.ℐL=\(ℐL∗\)∗\\mathcal\{I\}\_\{L\}=\(\\mathcal\{I\}\_\{L\}^\{\*\}\)^\{\*\}\. Indeed, this arises because in the classical DEIM method the convention is for multiplication to occur on the right, i\.e\. givenM=VSU𝖳∈ℝd×N\\mathrm\{M\}=\\mathrm\{V\}\\mathrm\{S\}\\mathrm\{U\}^\{\\mathsf\{T\}\}\\in\\mathbb\{R\}^\{d\\times N\}, the transposed equivalent to our operatorMηM\_\{\\eta\},x∈ℝN\\mathrm\{x\}\\in\\mathbb\{R\}^\{N\}and a pivoting matrixP\\mathrm\{P\}, Q\-DEIM approximates asMx≈MU​\(P𝖳​U\)†​P𝖳​x\\mathrm\{M\}\\mathrm\{x\}\\approx\\mathrm\{M\}\\mathrm\{U\}\{\(\\mathrm\{P\}^\{\\mathsf\{T\}\}\\mathrm\{U\}\)\}^\{\\dagger\}\\mathrm\{P\}^\{\\mathsf\{T\}\}\\mathrm\{x\}\. Whereas we apply multiplication on the left,ϕ\[q\]𝖳K˘ϕ𝖳Vηaug≈ϕ\[q\]𝖳\(K˘ϕ\[P,:\]\)𝖳\(U𝖳K˘ϕ\[P,:\]𝖳\)†U𝖳K˘ϕ𝖳Vηaug\\phi\[q\]^\{\\mathsf\{T\}\}\\breve\{K\}\_\{\\phi\}^\{\\mathsf\{T\}\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}\\approx\\phi\[q\]^\{\\mathsf\{T\}\}\(\\breve\{K\}\_\{\\phi\}\[P,:\]\)^\{\\mathsf\{T\}\}\(U^\{\\mathsf\{T\}\}\\breve\{K\}\_\{\\phi\}\[P,:\]^\{\\mathsf\{T\}\}\)^\{\\dagger\}U^\{\\mathsf\{T\}\}\\breve\{K\}\_\{\\phi\}^\{\\mathsf\{T\}\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}, but the method is the same in both cases, we project the input,ϕ⁡\[q\]\\phi\[q\]in our case, onto the reduced basis determined by the pivots\.is then given by the oblique projection

ℐLMη=L˘ϕ𝖳FP,:†𝖳F𝖳Vηaug=L˘ϕ𝖳FP,:†𝖳R†𝖳PR𝖳G=L˘ϕ𝖳FP,:†𝖳RPR𝖳,ℐL=L˘ϕ𝖳\(U𝖳L˘ϕ𝖳\)†U𝖳,\\mathcal\{I\}\_\{L\}M\_\{\\eta\}=\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}F\_\{P,:\}^\{\\dagger\\mathsf\{T\}\}F^\{\\mathsf\{T\}\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}=\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}F\_\{P,:\}^\{\\dagger\\mathsf\{T\}\}\{\{R^\{\\dagger\}\}^\{\\mathsf\{T\}\}P\_\{R\}^\{\\mathsf\{T\}\}\}G=\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}F\_\{P,:\}^\{\\dagger\\mathsf\{T\}\}RP\_\{R\}^\{\\mathsf\{T\}\},\\qquad\\mathcal\{I\}\_\{L\}=\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}\(U^\{\\mathsf\{T\}\}\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}\)^\{\\dagger\}U^\{\\mathsf\{T\}\},\(35\)where we use†\\daggerfor the pseudo\-inverse\. We note thatℐL\\mathcal\{I\}\_\{L\}has rank at mostrr\. Hence, whenn\>rn\>r, it does not project onto the whole basis spaceℬ\\mathcal\{B\}but only onto a subspace ofℬ\\mathcal\{B\}of dimension at mostrr, the pseudo\-inverse selecting the minimum\-norm coefficients\. Furthermore, as its test spaceΠU​ℬ\\Pi\_\{U\}\\mathcal\{B\}differs fromℬ\\mathcal\{B\}, this projection is in general not exact at the interpolation nodes, but instead at the projections of their features ontoSpan⁡\(U\)\\operatorname\{Span\}\(U\), i\.e\.\(ΠU​ϕ˘​\[Li\]\)𝖳​ℐL​Mη=\(ΠU​ϕ˘​\[Li\]\)𝖳​Mη\(\\Pi\_\{U\}\\breve\{\\phi\}\[L\_\{i\}\]\)^\{\\mathsf\{T\}\}\\mathcal\{I\}\_\{L\}M\_\{\\eta\}=\(\\Pi\_\{U\}\\breve\{\\phi\}\[L\_\{i\}\]\)^\{\\mathsf\{T\}\}M\_\{\\eta\}for alli∈\[n\]i\\in\[n\]\.

We note that an improvement to our implementation would be the use of randomisation\-based QR methods suited to the caseN≫d≥rN\\gg d\\geq r\. Such methods include randomised methods preserving rank\-revealing properties such as\[[FG26](https://arxiv.org/html/2610.40943#bib.bibx18)\]and recent methods specifically suitable for the Q\-DEIM method such as\[[CK26](https://arxiv.org/html/2610.40943#bib.bibx7)\]\.

Algorithm 2Batch Q\-DAIM selection1:keys

K∈ℝN×dK\\in\\mathbb\{R\}^\{N\\times d\}\(centered\), values

V∈ℝN×dV\\in\\mathbb\{R\}^\{N\\times d\}, budget

nn, oversampling

pp, Cholesky tolerance

τchol\\tau\_\{\\mathrm\{chol\}\}, linear residual variant \(v1\. or v5\. of §[3\.1\.2](https://arxiv.org/html/2610.40943#S3.SS1.SSS2)\)

2:

Vaug←\[V∣𝟏N\]V^\{\\mathrm\{aug\}\}\\leftarrow\[V\\mid\\mathbf\{1\}\_\{N\}\];

η←\\eta\\leftarrowgauged masses of \([19](https://arxiv.org/html/2610.40943#S2.E19)\)

3:

Vsel←Diag⁡\(η\)​VV\_\{\\mathrm\{sel\}\}\\leftarrow\\operatorname\{Diag\}\(\\eta\)V\(v1\.\), or

Diag⁡\(η\)​Vaug\\operatorname\{Diag\}\(\\eta\)V^\{\\mathrm\{aug\}\}\(v5\.\)

4:

G←Vsel𝖳​E˘K​K​VselG\\leftarrow\{V\_\{\\mathrm\{sel\}\}\}^\{\\mathsf\{T\}\}\\breve\{E\}\_\{KK\}V\_\{\\mathrm\{sel\}\}⊳\\trianglerightoperator Gram \([31](https://arxiv.org/html/2610.40943#S3.E31)\)

5:

\(PR,R\)←\(P\_\{R\},R\)\\leftarrowdiagonally pivoted Cholesky of

GG⊳\\trianglerightrank\-revealing factorisation

6:

C←Vsel​PR​R†C\\leftarrow V\_\{\\mathrm\{sel\}\}P\_\{R\}\{R^\{\\dagger\}\}⊳\\trianglerightwhitened coefficients \([33](https://arxiv.org/html/2610.40943#S3.E33)\)

7:

F←E˘K​K​CF\\leftarrow\\breve\{E\}\_\{KK\}C⊳\\trianglerightevaluation matrix \([34](https://arxiv.org/html/2610.40943#S3.E34)\)

8:

𝔏←\\mathfrak\{L\}\\leftarrowfirst

nnpivots of the QR\-CP of

F𝖳F^\{\\mathsf\{T\}\}⊳\\trianglerightor volume\-thresholded stop, see the remark above

9:return

L=K\[𝔏,:\]L=K\[\\mathfrak\{L\},:\]⊳\\trianglerightweights: any projection of §[3\.2](https://arxiv.org/html/2610.40943#S3.SS2)

The full Q\-DAIM batch selector is summarised in Algorithm[2](https://arxiv.org/html/2610.40943#alg2)\. Similarly to the GAIM method, the Q\-DAIM selector also has a complexity inN2N^\{2\}due to the construction of the Gram matrixE˘K​K\\breve\{E\}\_\{KK\}\.

#### 3\.1\.4Multi\-headedL2L^\{2\}weighted residuals

One issue that arises in the context of multi\-headed attention is that the attention residual for a key head does not take into account the attention residual summed over the query heads\. GivenNheadsN\_\{\\mathrm\{heads\}\}query heads, we setOh=Att⁡\(Qh,Kh,Vh\)O\_\{h\}=\\operatorname\{Att\}\(Q\_\{h\},K\_\{h\},V\_\{h\}\)to be the output of query headhhand the associated output projection matrixWhO∈ℝd×dW^\{O\}\_\{h\}\\in\\mathbb\{R\}^\{d\\times d\}555We useddfor both dimensions for simplicity, see\[[Ser\+26](https://arxiv.org/html/2610.40943#bib.bibx39)\]for more details on multi\-headed attention\. The output of a single layer of attention is given byO=∑h∈\[Nheads\]Oh​WhOO=\\sum\_\{h\\in\[N\_\{\\mathrm\{heads\}\}\]\}O\_\{h\}W^\{O\}\_\{h\}\. Under grouped\-query attention \(GQA\)\[[Ain\+23](https://arxiv.org/html/2610.40943#bib.bibx1)\]theNheadsN\_\{\\mathrm\{heads\}\}query heads shareNgroups<NheadsN\_\{\\mathrm\{groups\}\}<N\_\{\\mathrm\{heads\}\}key\-value heads through a surjectionh↦𝔤h∈\[Ngroups\]h\\mapsto\\mathfrak\{g\}\_\{h\}\\in\[N\_\{\\mathrm\{groups\}\}\]such that

Oh=Att\(Qh,K𝔤h,V𝔤h\),∑h∈\[Nheads\]OhWhO=∑k∈\[Ngroups\]∑h:𝔤h=kAtt\(Qh,K𝔤h,V𝔤h\)WhOO\_\{h\}=\\operatorname\{Att\}\(Q\_\{h\},K\_\{\\mathfrak\{g\}\_\{h\}\},V\_\{\\mathfrak\{g\}\_\{h\}\}\),\\quad\\sum\_\{h\\in\[N\_\{\\mathrm\{heads\}\}\]\}O\_\{h\}W^\{O\}\_\{h\}=\\sum\_\{k\\in\[N\_\{\\mathrm\{groups\}\}\]\}\\sum\_\{h:\\mathfrak\{g\}\_\{h\}=k\}\\operatorname\{Att\}\(Q\_\{h\},K\_\{\\mathfrak\{g\}\_\{h\}\},V\_\{\\mathfrak\{g\}\_\{h\}\}\)W^\{O\}\_\{h\}
In practice, as we shall see in §[4](https://arxiv.org/html/2610.40943#S4), we interpolate on different nodes for each*KV head*, hence the output error ofkk\-th KV head is given by the sum of query heads of its group, i\.e\. heads such that𝔤h=k\\mathfrak\{g\}\_\{h\}=k\.

For anL2L^\{2\}norm residual, we can take this into account by weighting the output dimensions ofVVthrough the Cholesky factor

Ωk=∑h:𝔤h=kWhOWhO𝖳∈ℝd×d,for which∑h:𝔤h=k∥eWhO∥22=eΩke𝖳=∥eΩk1/2∥22\\Omega\_\{k\}=\\sum\_\{h:\\mathfrak\{g\}\_\{h\}=k\}W^\{O\}\_\{h\}\{W^\{O\}\_\{h\}\}^\{\\mathsf\{T\}\}\\in\\mathbb\{R\}^\{d\\times d\},\\qquad\\text\{for which\}\\qquad\\sum\_\{h:\\mathfrak\{g\}\_\{h\}=k\}\\big\\lVert eW^\{O\}\_\{h\}\\big\\rVert\_\{2\}^\{2\}=e\\Omega\_\{k\}e^\{\\mathsf\{T\}\}=\\big\\lVert e\\Omega\_\{k\}^\{1/2\}\\big\\rVert\_\{2\}^\{2\}\(36\)given an error terme∈ℝde\\in\\mathbb\{R\}^\{d\}shared by the group\. We note that, as the query heads of a group have different errorsehe\_\{h\}and the output error∑heh​WhO\\sum\_\{h\}e\_\{h\}W^\{O\}\_\{h\}contains cross terms between heads, this weighting is a proxy for the error of the layer output rather than its exact value\. We thus reweight the value columns of the residual byΩk1/2\\Omega\_\{k\}^\{1/2\}, scoring∥\[S˘LDiag\(η\)Vaug\]i,1:dΩk1/2∥2\\lVert\[\\breve\{S\}\_\{L\}\\operatorname\{Diag\}\(\\eta\)V^\{\\mathrm\{aug\}\}\]\_\{i,1:d\}\\Omega\_\{k\}^\{1/2\}\\rVert\_\{2\}in place of \([30](https://arxiv.org/html/2610.40943#S3.E30)\)\. Two things to note, first is that this method can only be applied to theL2L^\{2\}norm, and that as it does not change the range, it is only applicable to GAIM\. Second, eachΩk\\Omega\_\{k\}only needs to be computed once at initialisation as it only depends on the head and the model being used\.

#### 3\.1\.5The finite\-dimensional surrogate

The main bottleneck of both the GAIM and Q\-DAIM methods resides in the quadratic cost of the computation of the Gram matrixE˘K​K\\breve\{E\}\_\{KK\}\. As we multiply it byVηV\_\{\\eta\}afterwards, if the attention was linear, a simple matrix product reordering would allow us to go fromO⁡\(N2​d\)→O⁡\(N​d2\)O\(N^\{2\}d\)\\to O\(Nd^\{2\}\)\. To enable this re\-ordering, we resort to approximatingϕ˘\\breve\{\\phi\}using finite feature mapsψ:ℝd→ℝD\\psi:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{D\}, withD∈ℕ∗D\\in\\mathbb\{N\}^\{\*\}\. Indeed, as we know thatη\\etacarries some important information for the selection of key nodes, we choose to preserve it\. SetKψ=\(ψ⁡\[Kj\]\)j∈\[N\]K\_\{\\psi\}=\(\\psi\[K\_\{j\}\]\)\_\{j\\in\[N\]\}andLψ=\(ψ⁡\[Lk\]\)k∈\[n\]L\_\{\\psi\}=\(\\psi\[L\_\{k\}\]\)\_\{k\\in\[n\]\}the feature matrices andDKψ=Diag⁡\(Kψ​Kψ𝖳\)D\_\{K\_\{\\psi\}\}=\\operatorname\{Diag\}\(K\_\{\\psi\}K\_\{\\psi\}^\{\\mathsf\{T\}\}\), andK˘ψ:=DKψ−1/2Kψ\\breve\{K\}\_\{\\psi\}:=D\_\{K\_\{\\psi\}\}^\{\-1/2\}K\_\{\\psi\}then, our method consists in replacing

K˘ϕ𝖳ηK⟶Kψ𝖳DKψ−1/2ηK,L˘ϕ𝖳ηL⟶Lψ𝖳DLψ−1/2ηL,\\breve\{K\}\_\{\\phi\}^\{\\mathsf\{T\}\}\\eta\_\{K\}\\longrightarrow K\_\{\\psi\}^\{\\mathsf\{T\}\}D\_\{K\_\{\\psi\}\}^\{\-1/2\}\\eta\_\{K\},\\qquad\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}\\eta\_\{L\}\\longrightarrow L\_\{\\psi\}^\{\\mathsf\{T\}\}D\_\{L\_\{\\psi\}\}^\{\-1/2\}\\eta\_\{L\},\(37\)and equivalently for their transposes\. In practice we consider three main methods to obtain such aψ\\psi:

- •The linear Taylor mapψ⁡\(x\)=\(1,x/d1/4\)∈ℝd\+1\\psi\(x\)=\(1,x/d^\{1/4\}\)\\in\\mathbb\{R\}^\{d\+1\}, corresponding to the first order Taylor approximation ofκexp\\kappa\_\{\\exp\},
- •the Performer FAVOR\+\+randomised feature map\[[Cho\+22](https://arxiv.org/html/2610.40943#bib.bibx6)\]approximatingκexp\\kappa\_\{\\exp\},
- •and the Random Fourier Features\[[RR07](https://arxiv.org/html/2610.40943#bib.bibx38)\]which approximates the Gaussian kernel directly\.

For the GAIM method, we consider the linear\-Taylor map and the Performer method, whereas for the Q\-DAIM method, we use the random Fourier features for the Gaussian matrix which approximate the Gaussian gram directly\. For clarity, we denote byψ˘\\breve\{\\psi\}, these features, and the approximation thus readsE˘K​K≈K˘ψ˘​K˘ψ˘𝖳\\breve\{E\}\_\{KK\}\\approx\\breve\{K\}\_\{\\breve\{\\psi\}\}\\breve\{K\}\_\{\\breve\{\\psi\}\}^\{\\mathsf\{T\}\}\.

##### GAIM linear\-complexity surrogates\.

For each residual operatorM∗M\_\{\*\}described in §[3\.1\.1](https://arxiv.org/html/2610.40943#S3.SS1.SSS1), we writeM∗,ψM\_\{\*,\\psi\}for their*finite\-dimensional surrogate operator*, obtained through substitution as in \([37](https://arxiv.org/html/2610.40943#S3.E37)\), thus we obtain

Mψ:=K˘ψ𝖳​Vηaug=\[Mψg∣Mψh\]M\_\{\\psi\}:=\\breve\{K\}\_\{\\psi\}^\{\\mathsf\{T\}\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}=\\big\[M^\{g\}\_\{\\psi\}\\mid M^\{h\}\_\{\\psi\}\\big\]\(38\)We introduce a first variant GAIM\-LIN\-O, where we compute the residual operators using theLϕL\_\{\\phi\}\-orthogonal projection of the map, and substituteKϕK\_\{\\phi\}in the same way as above,

M^Δ,ψ:=Mψ−L˘ψ𝖳​Wηaug=:\[M^Δ,ψg∣M^Δ,ψh\],\\hat\{M\}\_\{\\Delta,\\psi\}:=M\_\{\\psi\}\-\\breve\{L\}\_\{\\psi\}^\{\\mathsf\{T\}\}W^\{\\mathrm\{aug\}\}\_\{\\eta\}=:\\big\[\\hat\{M\}^\{g\}\_\{\\Delta,\\psi\}\\mid\\hat\{M\}^\{h\}\_\{\\Delta,\\psi\}\\big\],\(39\)whereWηaug:=E˘L​L†​E˘L​K​VηaugW^\{\\mathrm\{aug\}\}\_\{\\eta\}:=\\breve\{E\}\_\{LL\}^\{\\dagger\}\\breve\{E\}\_\{LK\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}is the gauged Galerkin solutionXXof \([23](https://arxiv.org/html/2610.40943#S2.E23)\)\. However, thisϕ\\phi\-orthogonal surrogate requires the computation of the cross\-Gram and Gram matrices,EL​KE\_\{LK\}andEL​LE\_\{LL\}, as well as the inverse of the latter\. To avoid this, we define a second variant whose residual operators are given as

MΔ,ψ:=Mψ−ΠLψ​Mψ=:\[MΔ,ψg∣MΔ,ψh\],\\displaystyle M\_\{\\Delta,\\psi\}:=M\_\{\\psi\}\-\\Pi\_\{L\_\{\\psi\}\}M\_\{\\psi\}=:\\big\[M^\{g\}\_\{\\Delta,\\psi\}\\mid M^\{h\}\_\{\\Delta,\\psi\}\\big\],\(40\)whereΠLψ\\Pi\_\{L\_\{\\psi\}\}is the orthogonal projector ontoSpan⁡\(Lψ\)⊂ℝD\\operatorname\{Span\}\(L\_\{\\psi\}\)\\subset\\mathbb\{R\}^\{D\}\. To approximate this, we consider a Matching Pursuit approach similar to\[[PRK93](https://arxiv.org/html/2610.40943#bib.bibx36),[Dav97](https://arxiv.org/html/2610.40943#bib.bibx11),[TGS06](https://arxiv.org/html/2610.40943#bib.bibx49)\], which we call GAIM\-LIN, where after each greedy selection indexed byi∗∈\[N\]i^\{\*\}\\in\[N\]for the chosen residual, we set instead

MΔ,ψ\(t\)=MΔ,ψ\(t−1\)−\(K˘ψ\[i∗,:\]𝖳K˘ψ\[i∗,:\]\)MΔ,ψ\(t−1\),M\_\{\\Delta,\\psi\}^\{\(t\)\}=M^\{\(t\-1\)\}\_\{\\Delta,\\psi\}\-\\left\(\\breve\{K\}\_\{\\psi\}\[i^\{\*\},:\]^\{\\mathsf\{T\}\}\\breve\{K\}\_\{\\psi\}\[i^\{\*\},:\]\\right\)M^\{\(t\-1\)\}\_\{\\Delta,\\psi\},orthogonalizing the residual only with respect to the last selection\.

Algorithm 3Surrogate greedy GAIM selection : GAIM\-LIN1:keys

K∈ℝN×dK\\in\\mathbb\{R\}^\{N\\times d\}, values

V∈ℝN×dV\\in\\mathbb\{R\}^\{N\\times d\}, budget

nn, feature map

ψ\\psi, residual variant

RselR\_\{\\mathrm\{sel\}\}, norm

pp
2:

Vaug←\[V∣𝟏N\]V^\{\\mathrm\{aug\}\}\\leftarrow\[V\\mid\\mathbf\{1\}\_\{N\}\];

η←\\eta\\leftarrowgauged masses of \([19](https://arxiv.org/html/2610.40943#S2.E19)\);

𝔏←∅\\mathfrak\{L\}\\leftarrow\\varnothing
3:

K˘ψ←DKψ−1/2Kψ\\breve\{K\}\_\{\\psi\}\\leftarrow D\_\{K\_\{\\psi\}\}^\{\-1/2\}K\_\{\\psi\};

Mψ←K˘ψ𝖳​Diag⁡\(η\)​Vaug∈ℝD×\(d\+1\)M\_\{\\psi\}\\leftarrow\\breve\{K\}\_\{\\psi\}^\{\\mathsf\{T\}\}\\operatorname\{Diag\}\(\\eta\)V^\{\\mathrm\{aug\}\}\\in\\mathbb\{R\}^\{D\\times\(d\+1\)\}⊳\\trianglerightnormalised features and moment

4:

R0←K˘ψ​MψR^\{0\}\\leftarrow\\breve\{K\}\_\{\\psi\}M\_\{\\psi\};

RΔ←R0R\_\{\\Delta\}\\leftarrow R^\{0\}⊳\\trianglerightevaluations and residual rows atL=∅L=\\varnothing

5:for

t=1,…,nt=1,\\dots,ndo

6:Update

RselR\_\{\\mathrm\{sel\}\}from

\(RΔ,R0\)\(R\_\{\\Delta\},R^\{0\}\);

i⋆←argmaxi∉𝔏∥\[Rsel\]i,:∥pi^\{\\star\}\\leftarrow\\arg\\max\_\{i\\notin\\mathfrak\{L\}\}\\lVert\[R\_\{\\mathrm\{sel\}\}\]\_\{i,:\}\\rVert\_\{p\}⊳\\trianglerightchosen variant, §[3\.1\.2](https://arxiv.org/html/2610.40943#S3.SS1.SSS2)

7:

RΔ←RΔ−K˘ψK˘ψ,i⋆𝖳\[RΔ\]i∗,:R\_\{\\Delta\}\\leftarrow R\_\{\\Delta\}\-\\breve\{K\}\_\{\\psi\}\\breve\{K\}\_\{\\psi,i^\{\\star\}\}^\{\\mathsf\{T\}\}\[R\_\{\\Delta\}\]\_\{i^\{\*\},:\};

𝔏←𝔏∪\{i⋆\}\\mathfrak\{L\}\\leftarrow\\mathfrak\{L\}\\cup\\\{i^\{\\star\}\\\}⊳\\trianglerightdeflation of the moment, updated residual rows

8:endfor

9:return

L=K\[𝔏,:\]L=K\[\\mathfrak\{L\},:\]⊳\\trianglerightweights: any projection of §[3\.2](https://arxiv.org/html/2610.40943#S3.SS2)

##### Q\-DAIM Random Fourier Features linear complexity surrogate\.

We tackle the quadratic cost of Q\-DAIM by building a surrogate Gram matrixGψ˘G\_\{\\breve\{\\psi\}\}which allows changing the order of multiplication and computingMψ˘=K˘ψ˘𝖳​Vηaug∈ℝD×\(d\+1\)≈MηM\_\{\\breve\{\\psi\}\}=\\breve\{K\}\_\{\\breve\{\\psi\}\}^\{\\mathsf\{T\}\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}\\in\\mathbb\{R\}^\{D\\times\(d\+1\)\}\\approx M\_\{\\eta\}first\. We use the notationψ˘\\breve\{\\psi\}for the Random Fourier Feature map to highlight the fact that the feature map approximates the Gaussian features directly and thus does not require renormalization throughDKψD\_\{K\_\{\\psi\}\}\. This then gives us

Gψ˘:=Mψ˘𝖳​Mψ˘=Vηaug𝖳​K˘ψ˘​K˘ψ˘𝖳​Vηaug∈ℝ\(d\+1\)×\(d\+1\)\.G\_\{\\breve\{\\psi\}\}:=M\_\{\\breve\{\\psi\}\}^\{\\mathsf\{T\}\}M\_\{\\breve\{\\psi\}\}=\{V^\{\\mathrm\{aug\}\}\_\{\\eta\}\}^\{\\mathsf\{T\}\}\\breve\{K\}\_\{\\breve\{\\psi\}\}\\breve\{K\}\_\{\\breve\{\\psi\}\}^\{\\mathsf\{T\}\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}\\in\\mathbb\{R\}^\{\(d\+1\)\\times\(d\+1\)\}\.The surrogate GramGψ˘G\_\{\\breve\{\\psi\}\}is still positive semi\-definite regardless ofψ˘\\breve\{\\psi\}, and the remaining Q\-DAIM procedure, whitening followed by selection of nodes based on QRCP pivots, remains unchanged\. In particular, the rank\-revealing pivoted Cholesky factorization used for whitening \([33](https://arxiv.org/html/2610.40943#S3.E33)\) is now applied toGψ˘G\_\{\\breve\{\\psi\}\}instead ofGG\. The evaluation matrix \([34](https://arxiv.org/html/2610.40943#S3.E34)\) is formed similarly throughMψ˘M\_\{\\breve\{\\psi\}\}as

Fψ˘=K˘ψ˘​\(K˘ψ˘𝖳​Cψ˘\),F\_\{\\breve\{\\psi\}\}=\\breve\{K\}\_\{\\breve\{\\psi\}\}\\big\(\\breve\{K\}\_\{\\breve\{\\psi\}\}^\{\\mathsf\{T\}\}C\_\{\\breve\{\\psi\}\}\\big\),withCψ˘=Vηaug​PRψ˘​Rψ˘†C\_\{\\breve\{\\psi\}\}=V^\{\\mathrm\{aug\}\}\_\{\\eta\}P\_\{R\_\{\\breve\{\\psi\}\}\}R\_\{\\breve\{\\psi\}\}^\{\\dagger\}andPRψ˘𝖳​Gψ˘​PRψ˘=Rψ˘𝖳​Rψ˘P\_\{R\_\{\\breve\{\\psi\}\}\}^\{\\mathsf\{T\}\}G\_\{\\breve\{\\psi\}\}P\_\{R\_\{\\breve\{\\psi\}\}\}=R\_\{\\breve\{\\psi\}\}^\{\\mathsf\{T\}\}R\_\{\\breve\{\\psi\}\}\. Hence, the overall complexity is reduced fromO⁡\(N​d​\(N\+d\)\)O\(Nd\(N\+d\)\)toO⁡\(D⁡\(d\+1\)​\(N\+d\+1\)\)O\(D\(d\+1\)\(N\+d\+1\)\), the second term coming from the surrogate GramGψ˘G\_\{\\breve\{\\psi\}\}\.

### 3\.2Alternative projections

The methods we have developed above all possess a natural associated projection, i\.e\., a way to constructWaugW^\{\\mathrm\{aug\}\}, used to actually select the nodes\. As we shall see, each of these methods can be classified as a specific case of a Petrov\-Galerkin projection\. Indeed, Petrov\-Galerkin methods approximate a continuous operatorMMin a spaceHHby an operatorMLM\_\{L\}defined in a finite\-dimensional subspace,ℬ⊂H\\mathcal\{B\}\\subset H, of functions called the trial space, which corresponds to our basis spaceℬ\\mathcal\{B\}\. The approximated operator is then obtained by choosing a set of functions𝒯\\mathcal\{T\}, called the test space, and its associated norm in which the residual is minimized\.

In the case of direct interpolation/GAIM, the approximated linear map is obtained by setting the residual to zero at the interpolation nodes\. This corresponds to a Bubnov\-Galerkin method, as we test the function on the basis space𝒯=ℬ\\mathcal\{T\}=\\mathcal\{B\}and consider the orthogonal projection of our linear maps onto the basis space\. For Q\-DAIM, the test space corresponds to the projection ofℬ\\mathcal\{B\}on the revealed range of the operator, i\.e\.,𝒯=ΠU​ℬ\\mathcal\{T\}=\\Pi\_\{U\}\\mathcal\{B\}, which coincides withℬ\\mathcal\{B\}only ifℬ⊆Span⁡\(U\)\\mathcal\{B\}\\subseteq\\operatorname\{Span\}\(U\), which requiresn≤rn\\leq r\.

The above methods can equivalently be seen as orthogonality constraints\. On the other hand, the least\-square approximation𝒯=Span⁡\(Kϕ\)\\mathcal\{T\}=\\operatorname\{Span\}\(K\_\{\\phi\}\)consists in a case where the test space contains directions orthogonal to the trial space, i\.e\.𝒯∩ℬ⟂≠\{0\}\\mathcal\{T\}\\cap\\mathcal\{B\}^\{\\perp\}\\neq\\\{0\\\}, and where the dimensions of𝒯\\mathcal\{T\}andℬ\\mathcal\{B\}may not match\. In our case, it is overdetermined\. To handle this as an orthogonality constraint, we consider the least\-square Petrov\-Galerkin consisting in solving the orthogonality constraints arising from the normal equations instead\.

Finally, we also consider the projection obtained by a direct minimization of the overdetermined system’s residual\. We note that, while this projection and the least\-square Petrov\-Galerkin coincide in exact arithmetic, their numerical solutions need not be the same, the normal equations squaring the condition number of the system\. In the following we describe in more detail these different types of projection\. We now formalise how these methods can be framed as Galerkin approximations ofMη=K˘ϕ𝖳​Vηaug∈Hd\+1M\_\{\\eta\}=\\breve\{K\}\_\{\\phi\}^\{\\mathsf\{T\}\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}\\in H^\{d\+1\}, the gauged augmented operator, on the trial spaceℬ=Span⁡\{ϕ˘​\[ℓ1\],…,ϕ˘​\[ℓn\]\}⊂H\\mathcal\{B\}=\\operatorname\{Span\}\\\{\\breve\{\\phi\}\[\\ell\_\{1\}\],\\dots,\\breve\{\\phi\}\[\\ell\_\{n\}\]\\\}\\subset H\.

Let𝒲\\mathcal\{W\}be a symmetric positive semi\-definite operator acting onHH\. The Petrov\-Galerkin projection findsX∈ℝn×\(d\+1\)X\\in\\mathbb\{R\}^\{n\\times\(d\+1\)\}such that the residualL˘ϕ𝖳​X−Mη\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}X\-M\_\{\\eta\}is𝒲\\mathcal\{W\}\-orthogonal toℬ\\mathcal\{B\}, i\.e\.

⟨ϕ˘\[ℓi\]𝖳,\(L˘ϕ𝖳X−Mη\)\[:,j\]⟩𝒲=0for alli∈\[n\]andj∈\[d\+1\],\\big\\langle\\breve\{\\phi\}\[\\ell\_\{i\}\]^\{\\mathsf\{T\}\},\(\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}X\-M\_\{\\eta\}\)\[:,j\]\\big\\rangle\_\{\\mathcal\{W\}\}=0\\quad\\text\{for all \}i\\in\[n\]\\text\{ and \}j\\in\[d\+1\],\(41\)where we set⟨f,g⟩𝒲:=⟨f,𝒲​g⟩H\\langle f,g\\rangle\_\{\\mathcal\{W\}\}:=\\langle f,\\mathcal\{W\}g\\rangle\_\{H\}\. The solution is obtained by solving then×nn\\times nlinear system,

A𝒲​X=B𝒲,A𝒲:=L˘ϕ​𝒲​L˘ϕ𝖳,B𝒲:=L˘ϕ​𝒲​Mη\.A\_\{\\mathcal\{W\}\}X=B\_\{\\mathcal\{W\}\},\\qquad A\_\{\\mathcal\{W\}\}:=\\breve\{L\}\_\{\\phi\}\\mathcal\{W\}\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\},\\qquad B\_\{\\mathcal\{W\}\}:=\\breve\{L\}\_\{\\phi\}\\mathcal\{W\}M\_\{\\eta\}\.\(42\)The projected weights are then obtained by

Waug=Diag⁡\(ηL\)−1​X\.W^\{\\mathrm\{aug\}\}=\\operatorname\{Diag\}\(\\eta\_\{L\}\)^\{\-1\}X\.\(43\)
The projection methods correspond to the following choices of𝒲\\mathcal\{W\}, where we setΣK=K˘ϕ𝖳​K˘ϕ\\Sigma\_\{K\}=\\breve\{K\}\_\{\\phi\}^\{\\mathsf\{T\}\}\\breve\{K\}\_\{\\phi\}andΠU=U​U𝖳\\Pi\_\{U\}=UU^\{\\mathsf\{T\}\}the projector onto𝒰\\mathcal\{U\}\. We provide the equivalent linear systems for each projection method in Table[1](https://arxiv.org/html/2610.40943#S3.T1), as well as the formulation of the WildCat\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]temperature\-scaled projection in the context of Petrov\-Galerkin, more details can be found in Appendix[B](https://arxiv.org/html/2610.40943#A2)\. We note that, for the Q\-DAIM and WildCat rows of the table, the𝒲\\mathcal\{W\}column gives the associated oblique projectorsℐL\\mathcal\{I\}\_\{L\}andPτP\_\{\\tau\}, which are not symmetric\. These projections correspond to Petrov\-Galerkin methods with distinct trial and test spaces rather than to a symmetric positive semi\-definite𝒲\\mathcal\{W\}\. Furthermore, the solution of \([42](https://arxiv.org/html/2610.40943#S3.E42)\) is unique only whenA𝒲A\_\{\\mathcal\{W\}\}is nonsingular\.

Table 1:Matrices and projectors associated with the different projection methods\.
### 3\.3Positioning with respect to kernel\-based selection methods

We now discuss how our two selectors relate to WildCat\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]and, more generally, to greedy kernel interpolation methods\[[WSH21](https://arxiv.org/html/2610.40943#bib.bibx53),[WSH23](https://arxiv.org/html/2610.40943#bib.bibx54)\]\. WildCat selects its nodes through a randomly pivoted Cholesky\[[ETW25](https://arxiv.org/html/2610.40943#bib.bibx17)\]of the key Gram matrixEK​KE\_\{KK\}alone, i\.e\. the next node is sampled with probabilityi∼‖ΠLϕ⟂​ϕ​\[Ki\]‖H2i\\sim\\\|\\Pi^\{\\perp\}\_\{L\_\{\\phi\}\}\\phi\[K\_\{i\}\]\\\|\_\{H\}^\{2\}\. Hence, it aims at reconstructing the span ofKϕK\_\{\\phi\}and does not take into account the effect ofVVon the interpolated output\. In the context of kernel interpolation methods\[[DSW05](https://arxiv.org/html/2610.40943#bib.bibx16),[SH17](https://arxiv.org/html/2610.40943#bib.bibx41)\], this corresponds to a randomized version of aPP\-greedy method, setting the power functionPL​\(x\)=∥ΠLϕ⟂​ϕ​\[x\]∥HP\_\{L\}\(x\)=\\lVert\\Pi\_\{L\_\{\\phi\}\}^\{\\perp\}\\phi\[x\]\\rVert\_\{H\}\. Similarly, Q\-DAIM can be seen as aPP\-greedy method for the revealed\-range projection normPL​\(x\)=‖ΠΠU​Lϕ⟂​ΠU​ϕ˘​\[x\]‖HP\_\{L\}\(x\)=\\\|\\Pi\_\{\\Pi\_\{U\}L\_\{\\phi\}\}^\{\\perp\}\\Pi\_\{U\}\\breve\{\\phi\}\[x\]\\\|\_\{H\}, which to our knowledge has not been studied in the context of kernel interpolation\. GAIM, on the other hand, corresponds to what is referred to as anff\-greedy method in the case of linear variants based onggandhh\. Note, however, that the convergence theory of the greedy kernel methods concerns the interpolation of functions of the native space, hereggandhh, whereas the error on attention itself is only related to them through \([27](https://arxiv.org/html/2610.40943#S3.E27)\)\. As our method consists in considering the ratiog/hg/hof two such interpolants, we have the following rudimentary bound

‖Att⁡\(q,K,V\)−Att~L​\(q,K,V\)‖2≤‖MΔg‖HS\+‖Att⁡\(q,K,V\)‖2​‖MΔh‖H\|ϕ˘​\[q\]𝖳​MLh\|,\\left\\\|\\operatorname\{Att\}\(q,K,V\)\-\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(q,K,V\)\\right\\\|\_\{2\}\\leq\\frac\{\\\|M^\{g\}\_\{\\Delta\}\\\|\_\{\\mathrm\{HS\}\}\+\\\|\\operatorname\{Att\}\(q,K,V\)\\\|\_\{2\}\\\|M^\{h\}\_\{\\Delta\}\\\|\_\{H\}\}\{\\left\|\\breve\{\\phi\}\[q\]^\{\\mathsf\{T\}\}M^\{h\}\_\{L\}\\right\|\},\(44\)where we set‖MΔg‖HS=Tr⁡\(MΔg𝖳​MΔg\)\\\|M^\{g\}\_\{\\Delta\}\\\|\_\{\\mathrm\{HS\}\}=\\sqrt\{\\operatorname\{Tr\}\(\{M^\{g\}\_\{\\Delta\}\}^\{\\mathsf\{T\}\}M^\{g\}\_\{\\Delta\}\)\}to be the Hilbert\-Schmidt norm of the numerator residual operator\. We leave a more detailed analysis of the error bounds for future work\.

## 4Interpolation for streaming attention

The interpolation methods we have defined above consider a fixed numberNNof tokens and associatedKKandV∈ℝN×dV\\in\\mathbb\{R\}^\{N\\times d\}matrices to be compressed\. In modern attention\-based language models, each layer of the model contains a multi\-headed attention mechanism\. For each of these heads of attention, the key and value vectors for the text that has been seen until a given moment in time are stored in memory to allow faster computation once a new sentence is provided to, or generated by, the model\. This is known as KV caching\. Once the input context is read, the key and value vectors of a given token impact the construction of the query, key and values of subsequent tokens in the same head\. Additionally, as the layers are sequential, a given token’s key and value vector also affects all the layers that follow\.

We therefore choose to read the context in segments ofNsegN\_\{\\mathrm\{seg\}\}tokens and alternate two steps\. First, the model is computed on the new segment using the compressed cache\. Then, we compress the keys and values produced by the model using one of our methods\. Beyond the first layer, the keys and values of a segment thus depend on the compressed attention of the previous layers and the errors due to the interpolation of the previous segments propagate both to the generation and compression of the vectors of the layers that follow\. Only the first segment’s key and value matrices are generated using exact attention\. For each layer and head, the cache thus holds, afterNtotN\_\{\\mathrm\{tot\}\}tokens have been read and compressed:

- •tsink∈ℕt\_\{\\mathrm\{sink\}\}\\in\\mathbb\{N\}fixed sink tokens at the beginning of the context, kept uncompressed;
- •trec∈ℕt\_\{\\mathrm\{rec\}\}\\in\\mathbb\{N\}recent tokens, corresponding to the lasttrect\_\{\\mathrm\{rec\}\}tokens read, kept uncompressed;
- •the key nodesL∈ℝn×dL\\in\\mathbb\{R\}^\{n\\times d\}and associated weightsWaug∈ℝn×\(d\+1\)W^\{\\mathrm\{aug\}\}\\in\\mathbb\{R\}^\{n\\times\(d\+1\)\}corresponding to the compressedN=Ntot−tsink−trecN=N\_\{\\mathrm\{tot\}\}\-t\_\{\\mathrm\{sink\}\}\-t\_\{\\mathrm\{rec\}\}tokens in between\.

After a new sentence’s key and values vectors are computed, its keys and values are appended to the cache\. The lasttrect\_\{\\mathrm\{rec\}\}of these new vectors form the new recent window and the other remaining vectors correspond to the segment\(Kseg,Vseg\)\(K\_\{\\mathrm\{seg\}\},V\_\{\\mathrm\{seg\}\}\)which, once it reaches a given sizeNsegN\_\{\\mathrm\{seg\}\}, is then compressed intonsegn\_\{\\mathrm\{seg\}\}nodes, for every layer and head at once\. Once the context has been read, the generated tokens are appended raw toKraw,VrawK\_\{\\mathrm\{raw\}\},V\_\{\\mathrm\{raw\}\}and are not compressed\.

Although the compression of the first segment is done using the exact same process as before, once a setL∈ℝn×dL\\in\\mathbb\{R\}^\{n\\times d\}andWaug∈ℝn×\(d\+1\)W^\{\\mathrm\{aug\}\}\\in\\mathbb\{R\}^\{n\\times\(d\+1\)\}have been instantiated, we slightly modify the interpolation process to take into account the compressed data\. In order to preserve the coherence of the interpolation between the different segments, i\.e\. the impact of the previous segments on the compression of a new one, we consider two additional steps to the interpolation’s selection and projection, an orthogonalisation step based on the compressed cache which we call the deflation step, and an associated weight update step on previous nodes after projection which we call the refit step\. Additionally, the centering of the keys needs to be refined, since the mean of the keys changes as new segments are added\.

### 4\.1Bridging the gap between global and local interpolation

LetKseg,Vseg∈ℝNseg×dK\_\{\\mathrm\{seg\}\},V\_\{\\mathrm\{seg\}\}\\in\\mathbb\{R\}^\{N\_\{\\mathrm\{seg\}\}\\times d\}be the key and value matrices of a given segment that we will compress intonseg∈ℕ∗n\_\{\\mathrm\{seg\}\}\\in\\mathbb\{N\}^\{\*\}nodesLsegL\_\{\\mathrm\{seg\}\}and associated weightsWsegaugW^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\}and letL∈ℝn×dL\\in\\mathbb\{R\}^\{n\\times d\}, withn∈ℕn\\in\\mathbb\{N\}, be the previously compressed nodes and weightsWaug∈ℝn×\(d\+1\)W^\{\\mathrm\{aug\}\}\\in\\mathbb\{R\}^\{n\\times\(d\+1\)\}\. Then, we can interpolate the segment using the selection and projection techniques defined in the fixed context case, with some optional modifications, based on the previously selected nodes, which allow to reduce the gap between the global and segment interpolation\. The first consideration to take into account is the centering of the keys\. Indeed, the mean key vectorμ\\muis not fixed in the context of the streaming attention, and we thus define two alternatives for the interpolation centering:

- •Global centering: Setμg=1N​∑i=1NKtsink\+i\\mu\_\{g\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}K\_\{t\_\{\\mathrm\{sink\}\}\+i\}the mean of the current global, sinkless, context\.
- •Segment centering: Setμseg=1Nseg​∑i=1NsegKseg,i\\mu\_\{\\mathrm\{seg\}\}=\\frac\{1\}\{N\_\{\\mathrm\{seg\}\}\}\\sum\_\{i=1\}^\{N\_\{\\mathrm\{seg\}\}\}K\_\{\\mathrm\{seg\},i\}the mean of the current segment\.

Once this is fixed, along with the choice of selection and projection method, we modify the selection method to remove the redundancy with the historical nodes through a deflation method\. In the same way as the basis deflation that is used after the selection of a node, this consists in considering only the part of the residual orthogonal to a given span inHH\. We consider either the revealed span, throughΠU\\Pi\_\{U\}, or the selected nodes, viaΠL\\Pi\_\{L\}, subtracting the part of the residual explained by the historical nodes/linear map\. Note that, in the rest of this section, every object \(masses, values and weights\) is handled in theη\\etagauge of \([19](https://arxiv.org/html/2610.40943#S2.E19)\); we drop theη\\etasubscripts for simplicity\. This results in two types of deflation which can be obtained as

- •Basis deflationΠLϕ⟂\\Pi\_\{L\_\{\\phi\}\}^\{\\perp\}: For any selection method, we consider the projection onto the orthogonal complement of the basis spaceℬ=Span⁡\(Lϕ\)\\mathcal\{B\}=\\operatorname\{Span\}\(L\_\{\\phi\}\), obtained as ΠLϕ⟂​K˘seg,ϕ𝖳=K˘seg,ϕ𝖳−L˘ϕ𝖳​E˘L​L†​E˘L​Kseg\.\\Pi\_\{L\_\{\\phi\}\}^\{\\perp\}\\breve\{K\}\_\{\\mathrm\{seg\},\\phi\}^\{\\mathsf\{T\}\}=\\breve\{K\}\_\{\\mathrm\{seg\},\\phi\}^\{\\mathsf\{T\}\}\-\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}\\breve\{E\}\_\{LL\}^\{\\dagger\}\\breve\{E\}\_\{LK\_\{\\mathrm\{seg\}\}\}\.The Gram matrixE˘K​K\\breve\{E\}\_\{KK\}is then replaced by its orthogonal projection, the deflated Gram matrix given by S˘L=K˘seg,ϕ​ΠLϕ⟂​K˘seg,ϕ𝖳=E˘Kseg​Kseg−E˘Kseg​L​E˘L​L†​E˘L​Kseg\.\\breve\{S\}\_\{L\}=\\breve\{K\}\_\{\\mathrm\{seg\},\\phi\}\\Pi^\{\\perp\}\_\{L\_\{\\phi\}\}\{\\breve\{K\}\_\{\\mathrm\{seg\},\\phi\}\}^\{\\mathsf\{T\}\}=\\breve\{E\}\_\{K\_\{\\mathrm\{seg\}\}K\_\{\\mathrm\{seg\}\}\}\-\\breve\{E\}\_\{K\_\{\\mathrm\{seg\}\}L\}\\breve\{E\}\_\{LL\}^\{\\dagger\}\\breve\{E\}\_\{LK\_\{\\mathrm\{seg\}\}\}\.
- •Range deflationΠU⟂\\Pi\_\{U\}^\{\\perp\}: For any selection method, we consider its projection onto the orthogonal complement ofSpan⁡\(U\)\\operatorname\{Span\}\(U\), obtained as ΠU⟂​K˘seg,ϕ𝖳=K˘seg,ϕ𝖳−L˘ϕ𝖳​Waug​GL†​Waug𝖳​E˘L​Kseg,\\Pi\_\{U\}^\{\\perp\}\\breve\{K\}\_\{\\mathrm\{seg\},\\phi\}^\{\\mathsf\{T\}\}=\\breve\{K\}\_\{\\mathrm\{seg\},\\phi\}^\{\\mathsf\{T\}\}\-\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}W^\{\\mathrm\{aug\}\}G\_\{L\}^\{\\dagger\}\{W^\{\\mathrm\{aug\}\}\}^\{\\mathsf\{T\}\}\\breve\{E\}\_\{LK\_\{\\mathrm\{seg\}\}\},whereGL=Waug𝖳​E˘L​L​Waug∈ℝ\(d\+1\)×\(d\+1\)G\_\{L\}=\{W^\{\\mathrm\{aug\}\}\}^\{\\mathsf\{T\}\}\\breve\{E\}\_\{LL\}W^\{\\mathrm\{aug\}\}\\in\\mathbb\{R\}^\{\(d\+1\)\\times\(d\+1\)\}\. This gives us the deflated Gram matrix, S˘U=K˘seg,ϕ​ΠU⟂​K˘seg,ϕ𝖳=E˘Kseg​Kseg−E˘Kseg​L​Waug​GL†​Waug𝖳​E˘L​Kseg,\\breve\{S\}\_\{U\}=\\breve\{K\}\_\{\\mathrm\{seg\},\\phi\}\\Pi^\{\\perp\}\_\{U\}\\breve\{K\}\_\{\\mathrm\{seg\},\\phi\}^\{\\mathsf\{T\}\}=\\breve\{E\}\_\{K\_\{\\mathrm\{seg\}\}K\_\{\\mathrm\{seg\}\}\}\-\\breve\{E\}\_\{K\_\{\\mathrm\{seg\}\}L\}W^\{\\mathrm\{aug\}\}G\_\{L\}^\{\\dagger\}\{W^\{\\mathrm\{aug\}\}\}^\{\\mathsf\{T\}\}\\breve\{E\}\_\{LK\_\{\\mathrm\{seg\}\}\},

In practice, the range deflation may also be computed using the Cholesky/eigendecomposition ofGLG\_\{L\}in the same way as the Q\-DAIM deflation, i\.e\. throughFU,c​h=K˘seg,ϕ​L˘ϕ𝖳​Waug​PRL​RL†F\_\{U,ch\}=\\breve\{K\}\_\{\\mathrm\{seg\},\\phi\}\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}W^\{\\mathrm\{aug\}\}P\_\{R\_\{L\}\}R\_\{L\}^\{\\dagger\}withPRL𝖳​GL​PRL=RL𝖳​RLP\_\{R\_\{L\}\}^\{\\mathsf\{T\}\}G\_\{L\}P\_\{R\_\{L\}\}=R\_\{L\}^\{\\mathsf\{T\}\}R\_\{L\}\. Furthermore, instead of being recomputed fromLLandWaugW^\{\\mathrm\{aug\}\}, the matrixGLG\_\{L\}can be updated for each new segment as,

G\[L∪Lseg\]=GL\+GLseg\+Waug𝖳​E˘L​Lseg​Wsegaug\+Wsegaug𝖳​E˘Lseg​L​Waug,G\_\{\[L\\cup L\_\{\\mathrm\{seg\}\}\]\}=G\_\{L\}\+G\_\{L\_\{\\mathrm\{seg\}\}\}\+\{W^\{\\mathrm\{aug\}\}\}^\{\\mathsf\{T\}\}\\breve\{E\}\_\{LL\_\{\\mathrm\{seg\}\}\}W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\}\+\{W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\}\}^\{\\mathsf\{T\}\}\\breve\{E\}\_\{L\_\{\\mathrm\{seg\}\}L\}W^\{\\mathrm\{aug\}\},which requires computing only the matrixE˘L​Lseg\\breve\{E\}\_\{LL\_\{\\mathrm\{seg\}\}\}coupling the previously and newly selected nodes, at a cost ofO⁡\(n​nseg​\(d\+1\)\)O\(nn\_\{\\mathrm\{seg\}\}\(d\+1\)\), the products with the weights addingO⁡\(nseg​\(d\+1\)​\(nseg\+d\+1\)\)O\(n\_\{\\mathrm\{seg\}\}\(d\+1\)\(n\_\{\\mathrm\{seg\}\}\+d\+1\)\)\. We note that this update assumes that the historical weightsWaugW^\{\\mathrm\{aug\}\}are unchanged\. When a refit replaces them byWaug\+WrefitaugW^\{\\mathrm\{aug\}\}\+W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{refit\}\}, the first term becomes\(Waug\+Wrefitaug\)𝖳​E˘L​L​\(Waug\+Wrefitaug\)\(W^\{\\mathrm\{aug\}\}\+W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{refit\}\}\)^\{\\mathsf\{T\}\}\\breve\{E\}\_\{LL\}\(W^\{\\mathrm\{aug\}\}\+W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{refit\}\}\)and the cross terms must be computed with the refitted weights\.

The modified operators for each deflation method are described in table[2](https://arxiv.org/html/2610.40943#S4.T2)\.

Table 2:Modified matrices for each deflation type\.We also note that for the finite\-dimensional surrogate methods, we replaceΠLϕ⟂​K˘seg,ϕ𝖳\\Pi\_\{L\_\{\\phi\}\}^\{\\perp\}\\breve\{K\}\_\{\\mathrm\{seg\},\\phi\}^\{\\mathsf\{T\}\}by

ΠLϕ,ψ⟂​K˘seg,ψ𝖳=K˘seg,ψ𝖳−L˘ψ𝖳​E˘L​L†​E˘L​Kseg,\\Pi\_\{L\_\{\\phi\},\\psi\}^\{\\perp\}\\breve\{K\}\_\{\\mathrm\{seg\},\\psi\}^\{\\mathsf\{T\}\}=\\breve\{K\}\_\{\\mathrm\{seg\},\\psi\}^\{\\mathsf\{T\}\}\-\\breve\{L\}\_\{\\psi\}^\{\\mathsf\{T\}\}\\breve\{E\}\_\{LL\}^\{\\dagger\}\\breve\{E\}\_\{LK\_\{\\mathrm\{seg\}\}\},andΠU⟂​K˘seg,ϕ𝖳\\Pi\_\{U\}^\{\\perp\}\\breve\{K\}\_\{\\mathrm\{seg\},\\phi\}^\{\\mathsf\{T\}\}by

ΠU,ψ⟂​K˘seg,ψ𝖳:=K˘seg,ψ𝖳−L˘ψ𝖳​Waug​GL†​Waug𝖳​E˘L​Kseg\.\\Pi\_\{U,\\psi\}^\{\\perp\}\\breve\{K\}\_\{\\mathrm\{seg\},\\psi\}^\{\\mathsf\{T\}\}:=\\breve\{K\}\_\{\\mathrm\{seg\},\\psi\}^\{\\mathsf\{T\}\}\-\\breve\{L\}\_\{\\psi\}^\{\\mathsf\{T\}\}W^\{\\mathrm\{aug\}\}G\_\{L\}^\{\\dagger\}\{W^\{\\mathrm\{aug\}\}\}^\{\\mathsf\{T\}\}\\breve\{E\}\_\{LK\_\{\\mathrm\{seg\}\}\}\.
After the selection, the projection is done on the undeflated operator, as in fixed context interpolation, and the resulting residual operatorMΔ=Kseg,ϕ𝖳​Vsegaug−Lseg,ϕ𝖳​WsegaugM\_\{\\Delta\}=K\_\{\\mathrm\{seg\},\\phi\}^\{\\mathsf\{T\}\}V^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\}\-L\_\{\\mathrm\{seg\},\\phi\}^\{\\mathsf\{T\}\}W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\}is used to update the weights of the historical nodes\. To do so, although any projection method may be used for the refit, the machinery used for the deflation allows a refit counterpart consisting in updatingWaugW^\{\\mathrm\{aug\}\}with the weights associated to the orthogonal projection used for the deflation, making it less costly\. Thus, denotingWrefitaugW^\{\\mathrm\{aug\}\}\_\{\\mathrm\{refit\}\}the update to the weights of the historical nodes, we obtain the updates

L←\[LLseg\],Waug←\[Waug0\]\+\[WrefitaugWsegaug\]\.L\\leftarrow\\begin\{bmatrix\}L\\\\ L\_\{\\mathrm\{seg\}\}\\end\{bmatrix\},\\qquad W^\{\\mathrm\{aug\}\}\\leftarrow\\begin\{bmatrix\}W^\{\\mathrm\{aug\}\}\\\\ 0\\end\{bmatrix\}\+\\begin\{bmatrix\}W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{refit\}\}\\\\ W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\}\\end\{bmatrix\}\.Thanks to the linearity of the attention’s numerator and denominator, we can directly append the new segments nodes and weights,LsegL\_\{\\mathrm\{seg\}\}andWsegaugW^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\}, toLLandWaugW^\{\\mathrm\{aug\}\}\. Indeed, once all the segments have been compressed intoL∈ℝn×dL\\in\\mathbb\{R\}^\{n\\times d\}andWaug∈ℝn×\(d\+1\)W^\{\\mathrm\{aug\}\}\\in\\mathbb\{R\}^\{n\\times\(d\+1\)\}, we can approximate, denoting byKrawK\_\{\\mathrm\{raw\}\}andVrawV\_\{\\mathrm\{raw\}\}the key and values associated with the sink and recent tokens, the attention’s output as

Att~L​\(Q,K,V\)=Eq​Kraw​Vraw\+Eq​L​WEq​Kraw​𝟏Nraw\+Eq​L​WNorm\.\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(Q,K,V\)=\\frac\{E\_\{qK\_\{\\mathrm\{raw\}\}\}V\_\{\\mathrm\{raw\}\}\+E\_\{qL\}W\}\{E\_\{qK\_\{\\mathrm\{raw\}\}\}\\mathbf\{1\}\_\{N\_\{\\mathrm\{raw\}\}\}\+E\_\{qL\}W\_\{\\mathrm\{Norm\}\}\}\.
In practice, an issue that may arise due to the interpolation is the ill\-conditioning of the ratio due to the interpolated denominator\. To mitigate this, we introduce a mean vector approximation of the remaining keys and values which allows us to better approximate the range of the operator, we hence call this method range correction\.

#### 4\.1\.1Mean vector approximation and range correction

GivenK,V∈ℝN×dK,V\\in\\mathbb\{R\}^\{N\\times d\}, its associated compressed set of nodesL∈ℝn×dL\\in\\mathbb\{R\}^\{n\\times d\}, and their weightsWaug=\[W∣WNorm\]∈ℝn×\(d\+1\)W^\{\\mathrm\{aug\}\}=\[W\\mid W\_\{\\mathrm\{Norm\}\}\]\\in\\mathbb\{R\}^\{n\\times\(d\+1\)\}\. Range correction then consists in completing the ’missing weights’ at origin associated toSpan⁡\(Kϕ\)⊖Span⁡\(Lϕ\)\\operatorname\{Span\}\(K\_\{\\phi\}\)\\ominus\\operatorname\{Span\}\(L\_\{\\phi\}\)using a ‘mean\-node’ approximation of the remaining nodes\. To do so, we consider the average of the remaining nodes and weights, given by

L¯:=∑i=1NKi−∑j=1nLjN−n,W¯:=∑i=1NVi−∑j=1nWjN−n,andW¯Norm:=N−∑j=1nWNorm,jN−n\.\\bar\{L\}:=\\frac\{\\sum\_\{i=1\}^\{N\}K\_\{i\}\-\\sum\_\{j=1\}^\{n\}L\_\{j\}\}\{N\-n\},\\quad\\bar\{W\}:=\\frac\{\\sum\_\{i=1\}^\{N\}V\_\{i\}\-\\sum\_\{j=1\}^\{n\}W\_\{j\}\}\{N\-n\},\\quad\\text\{ and \}\\quad\\bar\{W\}\_\{\\mathrm\{Norm\}\}:=\\frac\{N\-\\sum\_\{j=1\}^\{n\}W\_\{\\mathrm\{Norm\},j\}\}\{N\-n\}\.Given theNraw:=tsink\+trecN\_\{\\mathrm\{raw\}\}:=t\_\{\\mathrm\{sink\}\}\+t\_\{\\mathrm\{rec\}\}raw tokensKraw,Vraw∈ℝNraw×dK\_\{\\mathrm\{raw\}\},V\_\{\\mathrm\{raw\}\}\\in\\mathbb\{R\}^\{N\_\{\\mathrm\{raw\}\}\\times d\}corresponding to the sink and recent windows\. The range correction procedure consists in computing

Att~R​C,L​\(Q,K,V\)=Eq​Kraw​Vraw\+Eq​L​W\+N¯​Eq​L¯​W¯Eq​Kraw​𝟏Nraw\+Eq​L​WNorm\+N¯​Eq​L¯​W¯Norm,\\widetilde\{\\operatorname\{Att\}\}\_\{RC,L\}\(Q,K,V\)=\\frac\{E\_\{qK\_\{\\mathrm\{raw\}\}\}V\_\{\\mathrm\{raw\}\}\+E\_\{qL\}W\+\\bar\{N\}E\_\{q\\bar\{L\}\}\\bar\{W\}\}\{E\_\{qK\_\{\\mathrm\{raw\}\}\}\\mathbf\{1\}\_\{N\_\{\\mathrm\{raw\}\}\}\+E\_\{qL\}W\_\{\\mathrm\{Norm\}\}\+\\bar\{N\}E\_\{q\\bar\{L\}\}\\bar\{W\}\_\{\\mathrm\{Norm\}\}\},where we setN¯:=N−n\\bar\{N\}:=N\-n\. Using this, we can approximate each unselected node by the average of the unselected nodes, giving us at the origin

Att~R​C,L​\(0,K,V\)=𝟏Nraw𝖳​Vraw\+𝟏n𝖳​W\+N¯​W¯Nraw\+𝟏n𝖳​WNorm\+N¯​W¯Norm=Att⁡\(0,K,V\)\.\\widetilde\{\\operatorname\{Att\}\}\_\{RC,L\}\(0,K,V\)=\\frac\{\\mathbf\{1\}\_\{N\_\{\\mathrm\{raw\}\}\}^\{\\mathsf\{T\}\}V\_\{\\mathrm\{raw\}\}\+\\mathbf\{1\}\_\{n\}^\{\\mathsf\{T\}\}W\+\\bar\{N\}\\bar\{W\}\}\{N\_\{\\mathrm\{raw\}\}\+\\mathbf\{1\}\_\{n\}^\{\\mathsf\{T\}\}W\_\{\\mathrm\{Norm\}\}\+\\bar\{N\}\\bar\{W\}\_\{\\mathrm\{Norm\}\}\}=\\operatorname\{Att\}\(0,K,V\)\.

#### 4\.1\.2Positional encoding removal

Additionally, for sufficiently distant tokens, an issue that arises is that it appears that the RoPE positional encoding\[[Su\+23](https://arxiv.org/html/2610.40943#bib.bibx45)\]of tokens adversely affects the selection of interpolation nodes\. To reduce this effect, we also consider a ’RoPE Stripping’ method, wherein we consider the RoPE\-less key vectors for selection only\. Alternatively, we also consider stripping only the rotation matrices with frequencies such that their period is higher than the size of the segment\.

### 4\.2Streaming interpolation methods

Our streaming compression techniques are hence completely defined by the choice of the key centers and of the method used at each of the four stages, summarised in table[3](https://arxiv.org/html/2610.40943#S4.T3)\.

Table 3:Parameters for interpolation of streaming attention\.ChoiceOptionsKey centeringSegment \(c\) / Global \(g\)§[4](https://arxiv.org/html/2610.40943#S4)*\(1\)*DeflationNone / Basis \(S˘L\\breve\{S\}\_\{L\}\) / Range \(S˘U\\breve\{S\}\_\{U\}\)§[4](https://arxiv.org/html/2610.40943#S4)*\(2\)*SelectionGAIM / Q\-DAIM§[3\.1\.2](https://arxiv.org/html/2610.40943#S3.SS1.SSS2)featuresExact / Linear\-Taylor / Performer / RFF§[3\.1\.5](https://arxiv.org/html/2610.40943#S3.SS1.SSS5)residualv1\(T1T\_\{1\}\) / v2\(T1−T2T\_\{1\}\{\-\}T\_\{2\}\) / v3\(T1−T2T3\\tfrac\{T\_\{1\}\-T\_\{2\}\}\{T\_\{3\}\}\) / v4\(T1−T2h\\tfrac\{T\_\{1\}\-T\_\{2\}\}\{h\}\) / v5\(ϕ​\[q\]𝖳​MΔ\\phi\[q\]^\{\\mathsf\{T\}\}M\_\{\\Delta\}\)§[3\.1\.2](https://arxiv.org/html/2610.40943#S3.SS1.SSS2)*\(3\)*ProjectionBasis \(Galerkin\) / Range / PG least squares / least squares§[3\.2](https://arxiv.org/html/2610.40943#S3.SS2)*\(4\)*RefitNone / any projection method onLL§[4](https://arxiv.org/html/2610.40943#S4)Additional parametersRange correctionNone / Mean vector \(§[4\.1\.1](https://arxiv.org/html/2610.40943#S4.SS1.SSS1)\)§[4](https://arxiv.org/html/2610.40943#S4)RoPE StrippingAll frequencies/ frequency truncation§[4](https://arxiv.org/html/2610.40943#S4)One thing to note is that deflation and refit both require keeping, or recomputing, information on the previously compressed nodes\. Indeed, the basis variants require the cached inverse, or Cholesky factorisation, ofE˘L​L\\breve\{E\}\_\{LL\}at a memory cost ofO⁡\(n2\)O\(n^\{2\}\)or a recomputation cost inO⁡\(n3\)O\(n^\{3\}\)\. The range variants on the other hand require the GramGLG\_\{L\}and its Cholesky factorisation or inverse, at a memory cost ofO⁡\(\(d\+1\)2\)O\(\(d\+1\)^\{2\}\), or a recomputation cost inO⁡\(n⁡\(d\+1\)​\(n\+d\+1\)\)O\(n\(d\+1\)\(n\+d\+1\)\)\. Furthermore, not all combinations of choices are possible, Q\-DAIM only admits the linear residuals v1 and v5\. We thus define our two main strategies:

- •Streaming interpolation: The sequential application of the above methods to successive segments, the deflation and refit of each segment being based on the nodes and weights accumulated so far\.
- •Block interpolation: A fully parallelizable strategy in which, given a block consisting of multiple segments, each segment is interpolated in parallel, the deflation and refit being based on theLLandWaugW^\{\\mathrm\{aug\}\}of the previous blocks only, and not on the segments of the current block\.

Algorithm 4Compression of one segment, for one layer and one KV head\. The four stages are given as input functions, chosen among the combinations of Table[3](https://arxiv.org/html/2610.40943#S4.T3); the range correction \(§[4\.1\.1](https://arxiv.org/html/2610.40943#S4.SS1.SSS1)\) is omitted\.1:segment

\(Kseg,Vseg\)\(K\_\{\\mathrm\{seg\}\},V\_\{\\mathrm\{seg\}\}\); nodes and weights

\(L,Waug\)\(L,W^\{\\mathrm\{aug\}\}\); budget

nsegn\_\{\\mathrm\{seg\}\}; stage functionsDeflate,Select,Project,Refit

2:

\(S˘,R,cache\)←Deflate​\(L,Waug,Kseg,Vsegaug\)\(\\breve\{S\},R,\\mathrm\{cache\}\)\\leftarrow\\textsc\{Deflate\}\(L,W^\{\\mathrm\{aug\}\},K\_\{\\mathrm\{seg\}\},V^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\}\)⊳\\triangleright*\(1\)*: deflated Gram and residual, cached factorisations

3:

Lseg←Select​\(S˘,R,nseg\)L\_\{\\mathrm\{seg\}\}\\leftarrow\\textsc\{Select\}\(\\breve\{S\},R,n\_\{\\mathrm\{seg\}\}\)⊳\\triangleright*\(2\)*: at mostnsegn\_\{\\mathrm\{seg\}\}nodes ofKsegK\_\{\\mathrm\{seg\}\}

4:

Wsegaug←Project​\(Lseg,Kseg,Vsegaug\)W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\}\\leftarrow\\textsc\{Project\}\(L\_\{\\mathrm\{seg\}\},K\_\{\\mathrm\{seg\}\},V^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\}\)⊳\\triangleright*\(3\)*: on the undeflated operator

5:

Wrefitaug←Refit​\(cache,L,Lseg,Wsegaug,Kseg,Vsegaug\)W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{refit\}\}\\leftarrow\\textsc\{Refit\}\(\\mathrm\{cache\},L,L\_\{\\mathrm\{seg\}\},W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\},K\_\{\\mathrm\{seg\}\},V^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\}\);

Waug←Waug\+WrefitaugW^\{\\mathrm\{aug\}\}\\leftarrow W^\{\\mathrm\{aug\}\}\+W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{refit\}\}⊳\\triangleright*\(4\)*: refit on the residual of the projected segment

6:

\(L,Waug\)←\(L∪Lseg,\[Waug;Wsegaug\]\)\(L,W^\{\\mathrm\{aug\}\}\)\\leftarrow\(L\\cup L\_\{\\mathrm\{seg\}\},\[W^\{\\mathrm\{aug\}\};W^\{\\mathrm\{aug\}\}\_\{\\mathrm\{seg\}\}\]\)⊳\\trianglerightappend the new nodes and weights

7:return

\(L,Waug\)\(L,W^\{\\mathrm\{aug\}\}\)

### 4\.3Complexity and memory

Let us now discuss the complexity and memory requirements of the different interpolation strategies, to do so, we first recall the following notation:

In Tables[4](https://arxiv.org/html/2610.40943#S4.T4)and[5](https://arxiv.org/html/2610.40943#S4.T5)we list the memory and computational complexity associated to each step of the streaming interpolation methods defined above\. Here, the advantage of the surrogate methods is clear, while the cost of exact GAIM and Q\-DAIM selection are dominated by theO⁡\(Nseg2​d\)O\(N\_\{\\mathrm\{seg\}\}^\{2\}d\)construction of the Gram matrix, the surrogate methods are able to reduce it toO⁡\(Nseg​d​D\)O\(N\_\{\\mathrm\{seg\}\}dD\), making them usable even when segment sizes become prohibitive for the exact methods\. We recall that, as stated in the introduction, we consider a number of interpolation nodes per segment proportional to the head dimension,nseg=O⁡\(d\)n\_\{\\mathrm\{seg\}\}=O\(d\), so that these costs are linear inNsegN\_\{\\mathrm\{seg\}\}\. For a fixed compression ratio, i\.e\.nseg=θ​Nsegn\_\{\\mathrm\{seg\}\}=\\theta N\_\{\\mathrm\{seg\}\}, the terms inNseg​nsegN\_\{\\mathrm\{seg\}\}n\_\{\\mathrm\{seg\}\}would instead be quadratic in the segment size\. Furthermore, a new issue arises with basis deflation and refit, indeed the requirement to keep in memory then2n^\{2\}matrix associated to the Gram’s Cholesky factorisation \(or its inverse\) can quickly become prohibitive and grow beyond the initial KV cache’s footprint\. The computation of the historical Gram itself is also a bottleneck as it costsO⁡\(n2​nseg\)O\(n^\{2\}n\_\{\\mathrm\{seg\}\}\)per segment for a totalO⁡\(n3\)O\(n^\{3\}\)cost\. These increased costs make the basis deflation and refit unusable in practice\. Their Range\-revealed counterparts however, represent a less costly and more memory\-preserving alternative\. Their computational cost and memory requirements are more manageable, scaling linearly in the number of historical nodes\. We now look at some numerical results which will enable us to shed some light on the performance of our interpolation methods, the impact of deflation and refit and a look into how the variants compare with each other\.

MatrixSize*Post\-Compression*NodesLLn×dn\\times dWeightsWaugW^\{\\mathrm\{aug\}\}n×\(d\+1\)n\\times\(d\+1\)Running meanμg\\mu\_\{g\}ddRange\-correction weightsW¯,W¯Norm\\bar\{W\},\\bar\{W\}\_\{\\mathrm\{Norm\}\}d\+1d\+1Sink\(K,V\)sink\(K,V\)\_\{\\mathrm\{sink\}\}2​tsink​d2t\_\{\\mathrm\{sink\}\}dRecent\(K,V\)rec\(K,V\)\_\{\\mathrm\{rec\}\}2​trec​d2t\_\{\\mathrm\{rec\}\}d*Deflation/Refit*Basis projectionE˘L​L†\\breve\{E\}\_\{LL\}^\{\\dagger\}orΓL​L\\Gamma\_\{LL\}n2n^\{2\}Range projectionGLG\_\{L\},RLR\_\{L\}\(d\+1\)2\(d\+1\)^\{2\}*Post\-Compression:2​\(n\+1\+tsink\+trec\)​d\+n\+12\(n\+1\+t\_\{\\mathrm\{sink\}\}\+t\_\{\\mathrm\{rec\}\}\)d\+n\+1**Compression Ratio:θ≈n/N\\theta\\approx n/N*Table 4:Data in memory per head and per layer kept throughout the context compression\.Table 5:Computational cost of each step, for a single head, per segment\. The selection and residual update costs are summed over thensegn\_\{\\mathrm\{seg\}\}picks, and the surrogate costs assumeD≥dD\\geq d\.∗Galerkin coefficients updated by bordering at each pick, so that each residual update is rank\-1\.

†Single pass of Algorithm[2](https://arxiv.org/html/2610.40943#alg2),r≤min⁡\(Nseg,d\+1\)r\\leq\\min\(N\_\{\\mathrm\{seg\}\},d\+1\)\. The restarts of Appendix[E](https://arxiv.org/html/2610.40943#A5)\(nseg\>rn\_\{\\mathrm\{seg\}\}\>r\) addO⁡\(Nseg​nseg​\(nseg\+r\)\)O\(N\_\{\\mathrm\{seg\}\}n\_\{\\mathrm\{seg\}\}\(n\_\{\\mathrm\{seg\}\}\+r\)\)to the exact selector\.

‡Absorbed whenE˘L​L†\\breve\{E\}\_\{LL\}^\{\\dagger\}is maintained by bordered updates \(exact GAIM, GAIM\-LIN\-O, streaming basis variants\)\.

§R0=E˘K​K​VηaugR^\{0\}=\\breve\{E\}\_\{KK\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}costsO⁡\(Nseg2​d\)O\(N\_\{\\mathrm\{seg\}\}^\{2\}d\)unless reused from an exact selection, as assumed for the refit\.

## 5Numerical results

In this section, we discuss the efficiency of our method by considering two different aspects\. First, we focus on the convergence of the interpolation method in terms of the norm of the different residuals considered in this work and in terms of the downstream exact attention error for queries that appear after the context is compressed\. Second, we focus on the score of the methods on a set of benchmarks, LongBench\-E\[[Bai\+24](https://arxiv.org/html/2610.40943#bib.bibx2)\], used in the literature to determine the performance of compression methods\. Our method uses the following setting:

- •Model: Qwen2\.5\-7B\-Instruct\[[Qwe\+25](https://arxiv.org/html/2610.40943#bib.bibx37)\]:2828layers,Nheads=28N\_\{\\mathrm\{heads\}\}=28query heads,Nkv=4N\_\{\\mathrm\{kv\}\}=4KV heads,d=128d=128with bf16 weights\.
- •Library: Huggingface Transformers Library\[[Wol\+20](https://arxiv.org/html/2610.40943#bib.bibx52)\]with PyTorch 2\.12 backend\.
- •Backend: Model forward: bf16 on Metal\(MPS\), Compression: float64 on CPU\.
- •Hardware: Apple M4 Pro with 48GB of Unified Memory\.

### 5\.1Interpolation Residuals

To study the convergence of the GAIM and Q\-DAIM approaches described in §[3\.1\.2](https://arxiv.org/html/2610.40943#S3.SS1.SSS2)and §[3\.1\.3](https://arxiv.org/html/2610.40943#S3.SS1.SSS3), we use a single long document from the Wiki\-40b\[[Guo\+20](https://arxiv.org/html/2610.40943#bib.bibx20)\]dataset: the English Wikipedia article on the American Civil War \(Wikidata Q8676, index 449 of the English\-language train split\)\. After thetsink=32t\_\{\\mathrm\{sink\}\}=32sink tokens, as introduced in §[4](https://arxiv.org/html/2610.40943#S4)this corresponds to the first tokens of the context, we compress the key and value vectors of the nextN=4096N=4096tokens and evaluate the query vectors of the following40964096tokens, which we call*future queries*\. Unless stated otherwise, every metric is the median over all KV heads666In certain types of attention called Grouped Query Attention \(GQA\), multiple heads of attention share the same set of key and value vectors but possess different query vectors, this shared head of attention is what we refer to here as a KV head\. In MHA every head of attention is a KV head\.of the layers\{0,7,14,21\}\\\{0,7,14,21\\\}\. Recall that Q\-DAIM only admits the linear residuals v1 and v5, which explains the missing Q\-DAIM curves for v2, v3 and v4 in the following figures\.

#### 5\.1\.1Direct Interpolation residuals

As a first experiment, we look at the interpolation of the complete context \(N=4096N=4096\) using our different methods and their variants\. We look at three metrics which depend only on the keys and values, all expressed with the basis deflated Gram matrixS˘L\\breve\{S\}\_\{L\}of §[4](https://arxiv.org/html/2610.40943#S4)and the Hilbert–Schmidt norm:

- •relative norm of the augmented residual operator:‖MΔ,η‖22‖Mη‖22=Tr⁡\(Vηaug𝖳​S˘L​Vηaug\)Tr⁡\(Vηaug𝖳​E˘K​K​Vηaug\)\\frac\{\\\|M\_\{\\Delta,\\eta\}\\\|\_\{2\}^\{2\}\}\{\\\|M\_\{\\eta\}\\\|\_\{2\}^\{2\}\}=\\frac\{\\operatorname\{Tr\}\(\{V^\{\\mathrm\{aug\}\}\_\{\\eta\}\}^\{\\mathsf\{T\}\}\\breve\{S\}\_\{L\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}\)\}\{\\operatorname\{Tr\}\(\{V^\{\\mathrm\{aug\}\}\_\{\\eta\}\}^\{\\mathsf\{T\}\}\\breve\{E\}\_\{KK\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}\)\}\.
- •relative norm of the numerator residual operator:‖MΔ,ηg‖22‖Mηg‖22=Tr⁡\(Vη𝖳​S˘L​Vη\)Tr⁡\(Vη𝖳​E˘K​K​Vη\)\\frac\{\\\|M^\{g\}\_\{\\Delta,\\eta\}\\\|\_\{2\}^\{2\}\}\{\\\|M^\{g\}\_\{\\eta\}\\\|\_\{2\}^\{2\}\}=\\frac\{\\operatorname\{Tr\}\(V\_\{\\eta\}^\{\\mathsf\{T\}\}\\breve\{S\}\_\{L\}V\_\{\\eta\}\)\}\{\\operatorname\{Tr\}\(V\_\{\\eta\}^\{\\mathsf\{T\}\}\\breve\{E\}\_\{KK\}V\_\{\\eta\}\)\}\.
- •relative norm of the denominator residual operator:‖MΔ,ηh‖22‖Mηh‖22=Tr⁡\(η𝖳​S˘L​η\)Tr⁡\(η𝖳​E˘K​K​η\)\\frac\{\\\|M^\{h\}\_\{\\Delta,\\eta\}\\\|\_\{2\}^\{2\}\}\{\\\|M^\{h\}\_\{\\eta\}\\\|\_\{2\}^\{2\}\}=\\frac\{\\operatorname\{Tr\}\(\\eta^\{\\mathsf\{T\}\}\\breve\{S\}\_\{L\}\\eta\)\}\{\\operatorname\{Tr\}\(\\eta^\{\\mathsf\{T\}\}\\breve\{E\}\_\{KK\}\\eta\)\}\.

Figure[1](https://arxiv.org/html/2610.40943#S5.F1)displays the median squared norms of the augmented and denominator operators, the numerator being indistinguishable from the augmented one\. We observe that, as expected, variants v1 and v5, which directly optimize these residuals, perform better than the other variants in these metrics\. A surprising result is the fact that the residual norm ofMΔ,ηhM^\{h\}\_\{\\Delta,\\eta\}is also minimised by the selection of the un\-augmented variant v1\.

Figure 1:Residual operator convergence for the different methods\. Relative squared norm of the augmented residual operator \(top\) and of the denominator residual operator \(bottom\) against the number of nodesnn, median over the1616KV heads\. Each panel shows the variants of one method in colour and the curves of the other methods in grey\. The norm of the numerator residual operatorMΔ,ηgM^\{g\}\_\{\\Delta,\\eta\}stays within about3%3\\%of that of the augmented one and is not shown\.Figures[2](https://arxiv.org/html/2610.40943#S5.F2)and[3](https://arxiv.org/html/2610.40943#S5.F3)display the median error for the numerator, denominator, and exact attention on future queries\. To make heads comparable, we divide the errors by the mean output norm over the 10% of future queries with the largest attention on the same head, which illustrates how the error compares with the heavy hitters of that head, i\.e\. the set of queries with high attention output norms relative to the rest of the queries\. We define this as the median heavy\-hitter relative error, given the future queries matrixQ∈ℝNQ×dQ\\in\\mathbb\{R\}^\{N\_\{Q\}\\times d\},

εg10%=medq∈Q​‖Δ​g​\(q\)‖2meanq∈Qg10%​‖g⁡\(q\)‖2,εh10%=medq∈Q​\|Δ​h​\(q\)\|meanq∈Qh10%​\|h⁡\(q\)\|,εAtt10%=medq∈Q​‖Att⁡\(q,K,V\)−Att~L​\(q,K,V\)‖2meanq∈QAtt10%​‖Att⁡\(q,K,V\)‖2,\\varepsilon\_\{g\}^\{10\\%\}=\\frac\{\\mathrm\{med\}\_\{q\\in Q\}\\,\\\|\\Delta g\(q\)\\\|\_\{2\}\}\{\\mathrm\{mean\}\_\{q\\in Q^\{10\\%\}\_\{g\}\}\\,\\\|g\(q\)\\\|\_\{2\}\},\\quad\\varepsilon\_\{h\}^\{10\\%\}=\\frac\{\\mathrm\{med\}\_\{q\\in Q\}\\,\|\\Delta h\(q\)\|\}\{\\mathrm\{mean\}\_\{q\\in Q^\{10\\%\}\_\{h\}\}\\,\|h\(q\)\|\},\\quad\\varepsilon\_\{\\operatorname\{Att\}\}^\{10\\%\}=\\frac\{\\mathrm\{med\}\_\{q\\in Q\}\\,\\\|\\operatorname\{Att\}\(q,K,V\)\-\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(q,K,V\)\\\|\_\{2\}\}\{\\mathrm\{mean\}\_\{q\\in Q^\{10\\%\}\_\{\\operatorname\{Att\}\}\}\\,\\\|\\operatorname\{Att\}\(q,K,V\)\\\|\_\{2\}\},\(45\)whereQf10%⊂QQ^\{10\\%\}\_\{f\}\\subset Qdenotes the10%10\\%of queries on which‖f⁡\(q\)‖2\\\|f\(q\)\\\|\_\{2\}is largest\.

The dash\-dotted line indicates the*mean\-vector approximation*, i\.e\. the range\-corrected estimate \(described in §[4\.1\.1](https://arxiv.org/html/2610.40943#S4.SS1.SSS1)\) withn=0n=0nodes, and the dotted line indicates the median output norm, i\.e\. the zero\-prediction error\. As expected, the v2, v3 and v4 variants outperform v1 and v5 in terms of attention error for all GAIM\-based methods while, by design, the opposite is true for the residual norms, as can be expected given that v1 and v5 directly minimise the residual norms\.

Figure 2:Heavy hitter relative error \([45](https://arxiv.org/html/2610.40943#S5.E45)\) in numerator and denominator withtsink=32t\_\{\\mathrm\{sink\}\}=32and without range correction\. Top:εg10%\\varepsilon\_\{g\}^\{10\\%\}; bottom:εh10%\\varepsilon\_\{h\}^\{10\\%\}; both on the future queries\. Each curve is the median of this ratio over the112112query heads; each panel shows the variants of one method in colour and the curves of the other methods in grey\. The interquartile range over the heads is shown in Figure[11](https://arxiv.org/html/2610.40943#A3.F11)of Appendix[C](https://arxiv.org/html/2610.40943#A3)\.Figure 3:Attention error inL2L^\{2\}norm on future queries withtsink=32t\_\{\\mathrm\{sink\}\}=32andtrec=0t\_\{\\mathrm\{rec\}\}=0\. Heavy hitter relative attention errorεAtt10%\\varepsilon^\{10\\%\}\_\{\\operatorname\{Att\}\}without \(top\) and with \(bottom\) range correction\. Each curve is the median of this ratio over the112112query heads; each panel shows the variants of one method in colour and the curves of the other methods in grey\. For both rows, the zero\-readout errors are at least 3 times higher and have thus not been drawn in the figures\. The interquartile range over the heads is shown in Figure[12](https://arxiv.org/html/2610.40943#A3.F12)of Appendix[C](https://arxiv.org/html/2610.40943#A3)\.
#### 5\.1\.2Impact of range\-correction on a long context example

Table 6:Number of query heads per layer \(out of2828\) on which range correction lowers the maximum error inL∞L^\{\\infty\}norm,maxq∈Q⁡‖Att⁡\(q,K,V\)−Att~L​\(q,K,V\)‖∞\\max\_\{q\\in Q\}\\\|\\operatorname\{Att\}\(q,K,V\)\-\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(q,K,V\)\\\|\_\{\\infty\}\. GAIM exact v2, nodes selected in theL∞L^\{\\infty\}score norm,tsink=32t\_\{\\mathrm\{sink\}\}=32\. In parentheses we give the heads for which the error decreases by more than10%10\\%\. Last column: heads improved under every variant and every budget\.Table[6](https://arxiv.org/html/2610.40943#S5.T6)shows the number of heads of each layer that were improved by range correction in terms ofL∞L^\{\\infty\}attention error in future queries\. The decisive advantage of the range\-corrected estimators on the median error observed in Figure[3](https://arxiv.org/html/2610.40943#S5.F3)is less clear once we look at specific heads\. Indeed, as can be seen from Table[6](https://arxiv.org/html/2610.40943#S5.T6), while most heads are improved, few of them are improved by more than10%10\\%\. To illustrate this, Figure[4](https://arxiv.org/html/2610.40943#S5.F4)and Figure[5](https://arxiv.org/html/2610.40943#S5.F5)show two failure modes for the interpolation, one for the range\-corrected case and one for the raw interpolation\. Each figure plots the norm of the attention error for each future query, ordered according to the norm of the attention of the query; the solid black line indicates the norm of the attention output vector while the dotted line corresponds to the mean vector approximation\. As can be seen from Figure[4](https://arxiv.org/html/2610.40943#S5.F4), for some heads range correction adversely affects the performance of the interpolation\. Indeed, as the mean vector approximation has a high error, range\-corrected interpolation performs poorly until a sufficient number of interpolation nodes are chosen, compensating the mean value approximation, whereas the uncorrected interpolation performs well even with few nodes\. It is thus not a panacea but a parameter which may improve performance in certain cases, and for certain tasks as we shall see on the benchmarks\. On the other hand, we can see in Figure[5](https://arxiv.org/html/2610.40943#S5.F5)a case where, when the chosen interpolation nodes are not able to represent attention at the queries correctly, range correction allows to mitigate the failure of interpolation and, given a sufficient number of nodes, to extract some additional performance from the interpolation\.

Figure 4:Worst\-case error for range\-corrected GAIM interpolation inL∞L^\{\\infty\}norm\. Layer 0, query head 4\. GAIM exact with the nodes selected in theL∞L^\{\\infty\}score norm; each curve is a sliding mean over205205consecutive queries\.Figure 5:Worst\-case GAIM interpolation error and its improvement with range\-correction inL∞L^\{\\infty\}norm\. Layer 1, query head 16; this layer lies outside the layers\{0,7,14,21\}\\\{0,7,14,21\\\}used for the median curves\. GAIM exact with the nodes selected in theL∞L^\{\\infty\}score norm; each curve is a sliding mean over205205consecutive queries\. Range correction enables some of the variants to improve their error levels, especially on the tokens with higher attention scores\.

### 5\.2Alternative projection comparison

Figures[6](https://arxiv.org/html/2610.40943#S5.F6)and[7](https://arxiv.org/html/2610.40943#S5.F7)illustrate the impact of alternative projections described in §[3\.2](https://arxiv.org/html/2610.40943#S3.SS2)on the residual error of the GAIM v4 and Q\-DAIM v5 selection methods described above\. These show that on future queries, the oblique projections of Q\-DAIM \(in the case of Q\-DAIM selection\) and of WildCat provide a definite improvement in terms of residual attention error\. As we will see in the LongBench\-E benchmarks, this reduction in attention error with respect to the uncompressed attention does not correlate one\-to\-one with better performance on tasks in practice\. Although it is worth noting that very poor attention reconstruction error correlates with poor task results, generally because they result in garbled nonsensical text\. Table[7](https://arxiv.org/html/2610.40943#S5.T7)displays the number of heads for which WildCat temperature scaling improves the attention error on future queries\.

Table 7:Number of query heads per layer \(out of2828\) on which the temperature\-scaled projection lowers the maximum error inL∞L^\{\\infty\}norm,maxq∈Q⁡‖Att⁡\(q,K,V\)−Att~L​\(q,K,V\)‖∞\\max\_\{q\\in Q\}\\\|\\operatorname\{Att\}\(q,K,V\)\-\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(q,K,V\)\\\|\_\{\\infty\}\. GAIM exact v4 with the nodes selected in theL∞L^\{\\infty\}score norm, and Q\-DAIM exact v5,tsink=32t\_\{\\mathrm\{sink\}\}=32, RC on, the projection is solved in the kernel rescaled as in\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\],τ≈2\.04\\tau\\approx 2\.04\. In parentheses we give the heads for which the error decreases by more than10%10\\%\. The last column is the number of heads that improved for allnn\.For the rest of the residual\-based experiments, we will consider the Galerkin, i\.e\. the orthogonal projection, onto the interpolation nodes\.

Figure 6:Heavy\-hitter relative attention errorεAtt10%\\varepsilon^\{10\\%\}\_\{\\operatorname\{Att\}\}\([45](https://arxiv.org/html/2610.40943#S5.E45)\) using GAIM exact v4 interpolation nodes for the different projection methods described in Table[1](https://arxiv.org/html/2610.40943#S3.T1), and without projection, i\.e\. keeping the values of the selected keys,W=V\[𝔏,:\]W=V\[\\mathfrak\{L\},:\]\. Median over the112112query heads; the interquartile range over the heads is shown in Figure[13](https://arxiv.org/html/2610.40943#A3.F13)of Appendix[C](https://arxiv.org/html/2610.40943#A3)\. The methodology used is the same as the experiments in §[5\.1](https://arxiv.org/html/2610.40943#S5.SS1)\.Figure 7:Heavy\-hitter relative attention errorεAtt10%\\varepsilon^\{10\\%\}\_\{\\operatorname\{Att\}\}\([45](https://arxiv.org/html/2610.40943#S5.E45)\) using Q\-DAIM exact v5 interpolation nodes for the different projection methods described in Table[1](https://arxiv.org/html/2610.40943#S3.T1), and without projection, i\.e\. keeping the values of the selected keys,W=V\[𝔏,:\]W=V\[\\mathfrak\{L\},:\]\. Median over the112112query heads; the interquartile range over the heads is shown in Figure[14](https://arxiv.org/html/2610.40943#A3.F14)of Appendix[C](https://arxiv.org/html/2610.40943#A3)\. The methodology used is the same as the experiments in §[5\.1](https://arxiv.org/html/2610.40943#S5.SS1)\.#### 5\.2\.1Block solve with global projection

We focus now on the same document on the effect of a projection restricted to a segment versus a global projection on a complete interpolation basis\. To do so, we select the nodes with the best\-performing methods on the above metrics, in a per\-segment interpolation without deflation or refit\. For each of these methods, we compare the projection restricted to each segment with the projection against all interpolation nodes\. In Figure[8](https://arxiv.org/html/2610.40943#S5.F8), we plot, for each method, the ratio of the error obtained with the global projection to the error obtained with the segment\-local one, computed head by head\. Interestingly, in most cases the residual error of the segment local interpolation is within5%5\\%of the global interpolation\.

Figure 8:Comparison of segment\-local projection versus global interpolation node projection on attention residual\. Ratio, head by head, of the error obtained with the global projection to the error obtained with the segment\-local one, where the error is the median attention errormedq​‖Att⁡\(q\)−Att~L​\(q\)‖2\\mathrm\{med\}\_\{q\}\\,\\\|\\operatorname\{Att\}\(q\)\-\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(q\)\\\|\_\{2\}over the future queries \(left, centre\) or the fit residual‖E˘K​K​Vηaug−E˘K​L​ηL​Waug‖F\\\|\\breve\{E\}\_\{KK\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}\-\\breve\{E\}\_\{KL\}\\,\\eta\_\{L\}W^\{\\mathrm\{aug\}\}\\\|\_\{F\}at the keys \(right\)\. The dot is the median over the heads, the thick bar spans the quartiles and the thin line the 10th to 90th percentiles; a ratio below11favours the global projection, and “worse in” gives the share of heads on which it has the larger error\.112112query heads for the future queries and1616KV heads for the fit at the keys, withNseg=256N\_\{\\mathrm\{seg\}\}=256,nseg=64n\_\{\\mathrm\{seg\}\}=64andtsink=32t\_\{\\mathrm\{sink\}\}=32\.
#### 5\.2\.2Comparison between streaming and block methods

Finally, we compare the streaming method described in §[4](https://arxiv.org/html/2610.40943#S4)with range deflation and refit to block methods without deflation or refit\. In order for the comparison to reflect the interpolation methods as used in practice, we proceed as follows\. First, we keep the raw keys and values of the sink tokenstsink=32t\_\{\\mathrm\{sink\}\}=32, then we compress the key and value vectors of the subsequent tokensN=4096N=4096and keep the key and value vectors of the next tokenstrec=32t\_\{\\mathrm\{rec\}\}=32\. Then, for each query after thetsink\+N\+trect\_\{\\mathrm\{sink\}\}\+N\+t\_\{\\mathrm\{rec\}\}tokens, we compute the attention with the additional raw keys and values of the tokens up to the query token, as would occur in practice if one were to compress only theNNtokens\.

Figure 9:Norm of the augmented residual operator for segmented interpolation\. Top: relative squared norm‖MΔ,η‖22/‖Mη‖22\\\|M\_\{\\Delta,\\eta\}\\\|\_\{2\}^\{2\}/\\\|M\_\{\\eta\}\\\|\_\{2\}^\{2\}of the block method, in percent and shaded on a logarithmic scale \(median over the heads\)\. Bottom: difference of this norm between the streaming \(with range deflation and refit\) and the block method, streaming minus block, in percentage points \(blue: lower with streaming, red: higher\)\. Range correction on,tsink=32t\_\{\\mathrm\{sink\}\}=32\.Figure 10:Impact of segment\-interpolation parameters on best\-performing GAIM and Q\-DAIM variants\. Top: relative attention errormedq​‖Att⁡\(q\)−Att~L​\(q\)‖2/meanq∈top​10%​‖Att⁡\(q\)‖2\\mathrm\{med\}\_\{q\}\\,\\\|\\operatorname\{Att\}\(q\)\-\\widetilde\{\\operatorname\{Att\}\}\_\{L\}\(q\)\\\|\_\{2\}\\,/\\,\\mathrm\{mean\}\_\{q\\in\\mathrm\{top\}\\,10\\%\}\\,\\\|\\operatorname\{Att\}\(q\)\\\|\_\{2\}on the future queries for the block method, in percent \(median over the heads\)\. Bottom: difference of this error between the streaming \(with range deflation and refit\) and the block method, streaming minus block, in percentage points \(blue: lower with streaming, red: higher\)\. Range correction on,tsink=32t\_\{\\mathrm\{sink\}\}=32,trec=32t\_\{\\mathrm\{rec\}\}=32\.Some observations are in order\. First, the impact of the streaming configuration on the norm of the augmented residuals in Figure[9](https://arxiv.org/html/2610.40943#S5.F9)is small in most cells \(for the variants v1 or v5 that converge on the operator norms\)\. For future queries attention residual, it appears to be a small but net positive for the smaller segment\-size and smaller compression ratio cells of the GAIM\-exact method, while its effect on the other methods is unclear\. All in all, it appears that while the impact of range deflation and refit is small, it is nonetheless there\. And as we shall see in the next experiments, the relationship between the performance in language processing tasks and the remaining residual norm/future query attention error is unclear\. Indeed, while a high residual norm or future query error implies a poor performance, the lowest residual does not imply the best performance, which indicates that further investigations into a better metric to optimise might be necessary\.

### 5\.3LongBench\-E experiments

Indeed, while the residual error allows us to compare the output of the interpolated attention with the exact attention, this does not necessarily correlate with actual performance on natural language processing tasks\. In order to illustrate the impact of the compression on the performance of the LLM on given tasks, we provide in Table[8](https://arxiv.org/html/2610.40943#S5.T8)a preliminary result on the LongBench\-E\[[Bai\+24](https://arxiv.org/html/2610.40943#bib.bibx2)\]benchmark\. For a few variants of our GAIM and Q\-DAIM methods, we report, for each LongBench\-E task, the mean score over2525samples per task\. The bold scores correspond to the best\-performing compression method, and their distribution illustrates that the different parameters introduced in the article favour different types of tasks\. While the exact GAIM and Q\-DAIM methods were restricted to segment sizes on the order ofdddue to their quadratic scaling, we observe that the surrogate methods perform on par with the exact ones even though they scale linearly withNsegN\_\{\\mathrm\{seg\}\}\. For comparison, we include the performance of CompressKV, the WildCat\-based compression technique reported in\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]on the complete benchmark and for one\-to\-one comparison we also show*RP\-Cholesky*, a Randomly\-Pivoted Cholesky selection in the WildCatτ\\tau\-scaled kernel, similar to WildCat but using our segment\-based numerical framework\. Additionally, we also computed GAIM\-LIN and Q\-DAIM exact in the WildCatτ\\tau\-scaled kernel formulation, for both projection and selection, which provided a considerable improvement in residual error experiments\. In these benchmarks, it appears to mostly reduce the performance of GAIM while slightly improving Q\-DAIM on some specific tasks\. Most notably, some question answering tasks, MFQA and HotpotQA, scores seem to be positively impacted by the temperature scaling both in terms of selection and projection\.

Table 8:LongBench\-E \- 25 prompts per task at 25% context compression\. Key centers are given as the tuple \(selection, solve\), c = segment mean, g = global mean\. The Exact andCompressKVrows are the values reported in Table 4 of\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]\(25% compression\) on the complete benchmark \(compared to our 25 samples/task\)\. For comparison, the percentages rows give, per task, our best method on average \(Q\-DAIM exact,Nseg=64N\_\{\\mathrm\{seg\}\}\{=\}64\) and the best score per task relative to Dense, andCompressKVrelative to Exact\. Proj\.: Galerkin/basis projection \(Gal\.\), none \(each node keeps its own value with a unit weight,\[VL\|1\]\[V\_\{L\}\\,\|\\,1\]\), or Q\-DAIM’s revealed\-range projection restricted to the revealed rank \(ranger\)\. Methods optionally use theτ\\tautemperature scaled kernel for comparison with WildCat\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]\. RP\-Cholesky corresponds to an implementation of the WildCat selection and projection on our harness\.†: run on PyTorch 2\.14, whose GPU \(MPS\) attention kernels round differently\.
### 5\.4Observations

As the experiments above show, the attention residual does not seem to be correlated to the benchmark scores directly\. Indeed, as we have seen in Figure[6](https://arxiv.org/html/2610.40943#S5.F6)and Figure[7](https://arxiv.org/html/2610.40943#S5.F7)the temperature\-scaled projection lowers the attention error, yet loses 12 to 14 points on qasper for the GAIM\-LIN selection and 3 points for the Q\-DAIM selection\. The selection metric however does seem to determine which type of task performs better\. At the same budget, Q\-DAIM and GAIM\-LIN reach nearly the same LongBench\-E mean, albeit with different profiles\. Comparing their Galerkin variants with range correction, Q\-DAIM performs better on summarisation \(GovRep, MultiNews\), on HotpotQA and on code completion in LCC, while GAIM\-LIN leads on single\-document question answering \(Qasper, MFQA\) and, by a smaller margin, on 2Wiki and RepoB\-P\. This suggests that the selection criterion impacts which information the interpolation preserves, and that the range of the augmented operator, revealed by Q\-DAIM, helps identify the features that contain it\. It is also interesting to note that the selection of interpolation nodes has the most decisive impact on output performance\. While projection does help in some respects, simply using the values associated with the nodes without projection, also works well\. For instance, the projection\-free Q\-DAIM variant is the best performer on average \(50\.4050\.40,87\.1%87\.1\\%of Dense\), although it wins mainly by a large margin on a single task, 2Wiki, where it reaches60\.560\.5\(almost on par with the uncompressed model\) against35\.035\.0for the Galerkin variant with range correction \(45\.945\.9without\), while performing less well than the former on eight of the twelve other tasks\. Without projection or range correction, each node keeps its own key and value, so the score depends only on which tokens are kept; the fact that Q\-DAIM still outperforms GAIM\-LIN in this setting therefore suggests that its selection chooses better nodes\.

Finally, Q\-DAIM’s revealed\-range projection restricted to the revealed rank \(ranger\) performs very well\. Obtaining50\.0250\.02on average \(86\.5%86\.5\\%of Dense\), with the best scores on TriviaQA and SAMsum and the shared best on PRetr\. The restriction to the revealed rank, which is not necessary for the other projections, is due to the fact that on the span revealed byUUof dimensionrr, the firstrrpivots reduce the residual on this subspace ofHHto zero and the projected values on the remaining pivots are numerical noise\. Indeed, under range deflation, the operator in a later segment can have fewer revealed directionsrrthan the node budgetnsegn\_\{\\mathrm\{seg\}\}, down to one or two fornseg=16n\_\{\\mathrm\{seg\}\}=16in some heads of the middle layers\. Unrestricted range projection then reduces the performance dramatically due to the induced noise \(32\.132\.1on average, with garbled outputs on several prompts\)\. Restricting the projection to the firstrrpivots, removes this failure but leaves part of the budget unused in those segments\. Our restart method, consisting in projecting onto the orthogonal complement of the chosen nodes, allows us to find new directions that improve the reconstruction ofUUonHH, but with diminishing returns as all the keys are required to constructUUcompletely\. Finding a method that would allow one to represent the revealed rangeUUmore fully, could use this budget to further improve the method \(§[6](https://arxiv.org/html/2610.40943#S6)\)\.

We also note that removing the RoPE positional embedding on the key vectors has a noticeable impact on the performance of our methods, illustrated by the best\-performing PCount score\. This provides some weight to the argument that better compression may require taking into account positional information separately from semantic information\.

We highlight again the two cases illustrated in Figures[5](https://arxiv.org/html/2610.40943#S5.F5)and[4](https://arxiv.org/html/2610.40943#S5.F4), namely the fact that several regimes appear to occur in different heads, corresponding to the ability of either mean vector approximation or kernel based interpolation to provide an effective compression of the context and the interaction between the two methods\. These cases provide some evidence to support the idea that several different structures emerge in different attention heads and that the best methods of approximation for each structure differ\. It seems reasonable to expect that such structures not only appear in the studied model but that similar structures occur in most learned models of attention and that the correct way to compress the information contained in the data must take these structures into account\.

## 6Discussion and Future work

The experimental results we obtained are promising, showing a convergence in terms of residual norm and a reasonable LongBench\-E performance, comparable to state of the art methods \(see\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]\), which may, given more experimentation on the choice of method and associated best set of parameters, allow for further improvements in performance\. Furthermore, the interpolation methods we developed do not rely on the textual nature of the context, but only on the kernel underlying scaled dot\-product attention\. They are thus also suited to other types of attention mechanisms, such as the self\-attention and cross\-attention layers found in models aimed at vision, such as Vision Transformer\[[Dos\+21](https://arxiv.org/html/2610.40943#bib.bibx14)\]based models\[[Sim\+25](https://arxiv.org/html/2610.40943#bib.bibx42),[Tea\+25](https://arxiv.org/html/2610.40943#bib.bibx48)\]or graph attention models\[[Sun\+23](https://arxiv.org/html/2610.40943#bib.bibx46)\]\. In these non\-causal settings, the complete context is usually available at once while the context size is then set by a fixed resolution input\. Furthermore, given the fact that the nature of the output is different, the performance of the interpolated context might be more directly related to its residual operator norm or attention error\. The interpolatory framework developed in this article motivates several extensions, which will be the subject of future work\. First, since the revealed range of Q\-DAIM𝒰=Span⁡\(U\)\\mathcal\{U\}=\\operatorname\{Span\}\(U\)exists in the infinite dimensional RKHS, we never manipulate it directly but only through its evaluations of key featuresKϕK\_\{\\phi\}, requiring the storage of the complete set ofNNkeys\. The natural extension is thus to try to representUUdirectly in order to reduce this costly dependence on the keys\. An avenue to achieve this might be through the use of a random range finder for operators in Hilbert spaces\[[KPU25](https://arxiv.org/html/2610.40943#bib.bibx29)\]\. Second, our experiments provide support for the fact that different heads of attention possess different structures for the information they contain\. These structures respond differently to interpolation and range correction, and thus motivate the construction of an efficiently computable metric to identify the methods suitable for the approximation\. One such candidate is clustering, as it has been studied extensively for attention\[[Zan\+23](https://arxiv.org/html/2610.40943#bib.bibx57),[Han\+23](https://arxiv.org/html/2610.40943#bib.bibx21),[Hoo\+25](https://arxiv.org/html/2610.40943#bib.bibx25)\]but not for kernel interpolation, and a direct extension of our methods would be to interpolate each cluster separately along with its own mean vector range correction\. A related finding which appears to affect the approximability of these structures is positional encoding, and applying our methods to Latent Attention\[[Dee\+24](https://arxiv.org/html/2610.40943#bib.bibx12)\]might shed some light on its impact due to its separation of positional from semantic information\. Furthermore, applying our methods to it through conversion methods such as TransMLA\[[MYZ25](https://arxiv.org/html/2610.40943#bib.bibx34)\]would let us compare how the same data interpolate in latent and regular attention\.

Finally, the methods developed in this article reduce the context sizeNN, and can thus be complementary to quantization\-based methods such as TurboQuant\[[Zan\+25](https://arxiv.org/html/2610.40943#bib.bibx58)\]or PolarQuant\[[Han\+25](https://arxiv.org/html/2610.40943#bib.bibx22)\]\. Further improvements may also be obtained by using quantization to develop a more efficient way of selecting keys and projecting values by concentrating the information contained in the key and value vectors\.

## 7Acknowledgments

The first author \(M\.F\.S\.\) would like to thank Alexandre Pasco for the lively discussions that provided additional insights on the attention mechanism, Johannes Laute for the many conversations on the numerical implementation of our methods, and Benjamin Carrel for the numerous insightful discussions concerning interpolation\.

### Code and Data

The code used for the numerical experiments in Section[5](https://arxiv.org/html/2610.40943#S5)was written with the assistance of Claude Opus 4\.8 and Fable 5\.1 \(Anthropic\), accessed through Claude Code in 2026\. The authors designed the experiments, reviewed and tested all generated code, and take full responsibility for the results\. The experimental code is based upon the Huggingface transformers\[[Wol\+20](https://arxiv.org/html/2610.40943#bib.bibx52)\]python library and all experiments were run on Qwen2\.5\-7B\-Instruct\[[Qwe\+25](https://arxiv.org/html/2610.40943#bib.bibx37)\]\. The data used for the experiments comes from the Wiki\-40b\[[Guo\+20](https://arxiv.org/html/2610.40943#bib.bibx20)\]dataset and the LongBench benchmark\[[Bai\+24](https://arxiv.org/html/2610.40943#bib.bibx2)\]dataset\.

### A\.I\. Usage for typesetting and literature review

In addition to the use of AI for coding, AI assistance using the Claude Code Harness \(Opus 4\.8, 5\.5 and Fable 5\.1\) was used to search the literature, to generate diagrams, and to format and typeset parts of the text and ensure coherence of notation, as well as provide feedback on the text\.

## References

- \[Ain\+23\]Joshua Ainslie et al\.“GQA: Training Generalized Multi\-Query Transformer Models from Multi\-Head Checkpoints”arXiv, 2023DOI:[10\.48550/arXiv\.2305\.13245](https://dx.doi.org/10.48550/arXiv.2305.13245)
- \[Bai\+24\]Yushi Bai et al\.“LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding”In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*Bangkok, Thailand: Association for Computational Linguistics, 2024, pp\. 3119–3137DOI:[10\.18653/v1/2024\.acl\-long\.172](https://dx.doi.org/10.18653/v1/2024.acl-long.172)
- \[BG65\]Peter Businger and Gene\. Golub“Linear Least Squares Solutions by Householder Transformations”In*Numerische Mathematik*7\.3, 1965, pp\. 269–276DOI:[10\.1007/BF01436084](https://dx.doi.org/10.1007/BF01436084)
- \[Che\+21\]Yifan Chen, Qi Zeng, Heng Ji and Yun Yang“Skyformer: Remodel Self\-Attention with Gaussian Kernel and Nyström Method”arXiv, 2021DOI:[10\.48550/arXiv\.2111\.00035](https://dx.doi.org/10.48550/arXiv.2111.00035)
- \[Che\+26\]Justin\. Chen et al\.“Towards Tight Bounds for Streaming Attention”, 2026arXiv:[https://arxiv\.org/abs/2606\.07205](https://arxiv.org/abs/2606.07205)
- \[Cho\+22\]Krzysztof Choromanski et al\.“Rethinking Attention with Performers”arXiv, 2022DOI:[10\.48550/arXiv\.2009\.14794](https://dx.doi.org/10.48550/arXiv.2009.14794)
- \[CK26\]Alice Cortinovis and Daniel Kressner“Adaptive Randomized Pivoting for Column Subset Selection, DEIM, and Low\-Rank Approximation”In*SIAM Journal on Matrix Analysis and Applications*47\.1, 2026, pp\. 25–47DOI:[10\.1137/24M1719189](https://dx.doi.org/10.1137/24M1719189)
- \[Com\+08\]Pierre Comon, Gene Golub, Lek\-Heng Lim and Bernard Mourrain“Symmetric Tensors and Symmetric Tensor Rank”arXiv, 2008DOI:[10\.48550/arXiv\.0802\.1681](https://dx.doi.org/10.48550/arXiv.0802.1681)
- \[Dao\+22\]Tri Dao et al\.“FlashAttention: Fast and Memory\-Efficient Exact Attention with IO\-Awareness”arXiv, 2022DOI:[10\.48550/arXiv\.2205\.14135](https://dx.doi.org/10.48550/arXiv.2205.14135)
- \[Dao23\]Tri Dao“FlashAttention\-2: Faster Attention with Better Parallelism and Work Partitioning”arXiv, 2023DOI:[10\.48550/arXiv\.2307\.08691](https://dx.doi.org/10.48550/arXiv.2307.08691)
- \[Dav97\]Geoffrey Davis“Greedy adaptive approximation”In*Journal of Constructive Approximation*13\.1, 1997, pp\. 57–98
- \[Dee\+24\]DeepSeek\-AI et al\.“DeepSeek\-V2: A Strong, Economical, and Efficient Mixture\-of\-Experts Language Model”arXiv, 2024DOI:[10\.48550/arXiv\.2405\.04434](https://dx.doi.org/10.48550/arXiv.2405.04434)
- \[DG16\]Zlatko Drmač and Serkan Gugercin“A New Selection Operator for the Discrete Empirical Interpolation Method—Improved A Priori Error Bound and Extensions”In*SIAM Journal on Scientific Computing*38\.2, 2016, pp\. A631–A648DOI:[10\.1137/15M1019271](https://dx.doi.org/10.1137/15M1019271)
- \[Dos\+21\]Alexey Dosovitskiy et al\.“An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, 2021arXiv:[https://arxiv\.org/abs/2010\.11929](https://arxiv.org/abs/2010.11929)
- \[DS18\]Zlatko Drmač and Arvind\. Saibaba“The Discrete Empirical Interpolation Method: Canonical Structure and Formulation in Weighted Inner Product Spaces”arXiv, 2018DOI:[10\.48550/arXiv\.1704\.06606](https://dx.doi.org/10.48550/arXiv.1704.06606)
- \[DSW05\]Stefano De, Robert Schaback and Holger Wendland“Near\-Optimal Data\-Independent Point Locations for Radial Basis Function Interpolation”In*Advances in Computational Mathematics*23\.3, 2005, pp\. 317–330DOI:[10\.1007/s10444\-004\-1829\-1](https://dx.doi.org/10.1007/s10444-004-1829-1)
- \[ETW25\]Ethan\. Epperly, Joel\. Tropp and Robert\. Webber“Embrace Rejection: Kernel Matrix Approximation by Accelerated Randomly Pivoted Cholesky”arXiv, 2025DOI:[10\.48550/arXiv\.2410\.03969](https://dx.doi.org/10.48550/arXiv.2410.03969)
- \[FG26\]Israa Fakih and Laura Grigori“Efficient QR\-based Column Subset Selection through Randomized Sparse Embeddings”, 2026arXiv:[https://arxiv\.org/abs/2509\.03198](https://arxiv.org/abs/2509.03198)
- \[Gon\+26\]Albert Gong, Annabelle Carrell, Raaz Dwivedi and Lester Mackey“Express Language Modeling”arXiv, 2026DOI:[10\.48550/arXiv\.2606\.10944](https://dx.doi.org/10.48550/arXiv.2606.10944)
- \[Guo\+20\]Mandy Guo, Zihang Dai, Denny Vrandečić and Rami Al\-Rfou“Wiki\-40B: Multilingual Language Model Dataset”In*Proceedings of the Twelfth Language Resources and Evaluation Conference*Marseille, France: European Language Resources Association, 2020, pp\. 2440–2452URL:[https://aclanthology\.org/2020\.lrec\-1\.297/](https://aclanthology.org/2020.lrec-1.297/)
- \[Han\+23\]Insu Han et al\.“HyperAttention: Long\-context Attention in Near\-Linear Time”arXiv, 2023DOI:[10\.48550/arXiv\.2310\.05869](https://dx.doi.org/10.48550/arXiv.2310.05869)
- \[Han\+25\]Insu Han et al\.“PolarQuant: Quantizing KV Caches with Polar Transformation”, 2025arXiv:[https://arxiv\.org/abs/2502\.02617](https://arxiv.org/abs/2502.02617)
- \[Han\+25a\]Insu Han et al\.“Streaming Attention Approximation via Discrepancy Theory”arXiv, 2025DOI:[10\.48550/arXiv\.2502\.07861](https://dx.doi.org/10.48550/arXiv.2502.07861)
- \[Hig90\]Nicholas Higham“Analysis of the Cholesky decomposition of a semi\-definite matrix”In*Reliable Numerical Commputation*Oxford University Press, 1990DOI:[10\.1093/oso/9780198535645\.003\.0010](https://dx.doi.org/10.1093/oso/9780198535645.003.0010)
- \[Hoo\+25\]Coleman Hooper et al\.“Multipole Attention for Efficient Long Context Reasoning”arXiv, 2025DOI:[10\.48550/arXiv\.2506\.13059](https://dx.doi.org/10.48550/arXiv.2506.13059)
- \[KKL20\]Nikita Kitaev, Łukasz Kaiser and Anselm Levskaya“Reformer: The Efficient Transformer”arXiv, 2020DOI:[10\.48550/arXiv\.2001\.04451](https://dx.doi.org/10.48550/arXiv.2001.04451)
- \[KLS18\]Agnan Kessy, Alex Lewin and Korbinian Strimmer“Optimal Whitening and Decorrelation”In*The American Statistician*72\.4Taylor & Francis, 2018, pp\. 309–314DOI:[10\.1080/00031305\.2016\.1277159](https://dx.doi.org/10.1080/00031305.2016.1277159)
- \[KMZ24\]Praneeth Kacham, Vahab Mirrokni and Peilin Zhong“PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels”In*Proceedings of the 41st International Conference on Machine Learning*235, Proceedings of Machine Learning ResearchPMLR, 2024, pp\. 22748–22770URL:[https://proceedings\.mlr\.press/v235/kacham24a\.html](https://proceedings.mlr.press/v235/kacham24a.html)
- \[KPU25\]Daniel Kressner, David Persson and André Uschmajew“On the Randomized SVD in Infinite Dimensions”arXiv, 2025DOI:[10\.48550/arXiv\.2506\.06882](https://dx.doi.org/10.48550/arXiv.2506.06882)
- \[Li\+24\]Yuhong Li et al\.“SnapKV: LLM Knows What You Are Looking for Before Generation”arXiv, 2024DOI:[10\.48550/arXiv\.2404\.14469](https://dx.doi.org/10.48550/arXiv.2404.14469)
- \[LLS24\]Sangho Lee, Hayun Lee and Dongkun Shin“Proxyformer: Nyström\-Based Linear Transformer with Trainable Proxy Tokens”In*AAAI*38\.12, 2024, pp\. 13418–13426DOI:[10\.1609/aaai\.v38i12\.29244](https://dx.doi.org/10.1609/aaai.v38i12.29244)
- \[Mad\+07\]Yvon Maday, Ngoc Nguyen, Anthony\. Patera and George\.H\. Pau“A general, multipurpose interpolation procedure: the magic points” working paper or preprint, 2007URL:[https://hal\.science/hal\-00174797](https://hal.science/hal-00174797)
- \[MM13\]Y\. Maday and O\. Mula“A Generalized Empirical Interpolation Method: Application of Reduced Basis Techniques to Data Assimilation”, 2013, pp\. 221–235DOI:[10\.1007/978\-88\-470\-2592\-9\_13](https://dx.doi.org/10.1007/978-88-470-2592-9_13)
- \[MYZ25\]Fanxu Meng, Zengwei Yao and Muhan Zhang“TransMLA: Multi\-Head Latent Attention Is All You Need”arXiv, 2025DOI:[10\.48550/arXiv\.2502\.07864](https://dx.doi.org/10.48550/arXiv.2502.07864)
- \[PP25\]Ninh Pham and Rasmus Pagh“Tensor Sketch: Fast and Scalable Polynomial Kernel Approximation”arXiv, 2025DOI:[10\.48550/arXiv\.2505\.08146](https://dx.doi.org/10.48550/arXiv.2505.08146)
- \[PRK93\]Y\.C\. Pati, R\. Rezaiifar and P\.S\. Krishnaprasad“Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition”In*Proceedings of 27th Asilomar Conference on Signals, Systems and Computers*, 1993, pp\. 40–44 vol\.1DOI:[10\.1109/ACSSC\.1993\.342465](https://dx.doi.org/10.1109/ACSSC.1993.342465)
- \[Qwe\+25\]Qwen et al\.“Qwen2\.5 Technical Report”arXiv, 2025DOI:[10\.48550/arXiv\.2412\.15115](https://dx.doi.org/10.48550/arXiv.2412.15115)
- \[RR07\]Ali Rahimi and Benjamin Recht“Random Features for Large\-Scale Kernel Machines”In*Advances in Neural Information Processing Systems*20Curran Associates, Inc\., 2007URL:[https://proceedings\.neurips\.cc/paper\_files/paper/2007/file/013a006f03dbc5392effeb8f18fda755\-Paper\.pdf](https://proceedings.neurips.cc/paper_files/paper/2007/file/013a006f03dbc5392effeb8f18fda755-Paper.pdf)
- \[Ser\+26\]Michel Serret et al\.“Attention Mechanisms Through the Lens of Numerical Methods: Approximation Methods and Alternative Formulations”, 2026arXiv:[https://arxiv\.org/abs/2604\.01757](https://arxiv.org/abs/2604.01757)
- \[Ser26\]Michel Serret“Understanding Transformers and Attention Mechanisms: An Introduction for Applied Mathematicians”, 2026arXiv:[https://arxiv\.org/abs/2604\.00965](https://arxiv.org/abs/2604.00965)
- \[SH17\]Gabriele Santin and Bernard Haasdonk“Convergence Rate of the Data\-Independent P\-Greedy Algorithm in Kernel\-Based Approximation”In*Dolomites Research Notes on Approximation*10\.Special Issue, 2017, pp\. 68–78DOI:[10\.14658/PUPJ\-DRNA\-2017\-Special\_Issue\-9](https://dx.doi.org/10.14658/PUPJ-DRNA-2017-Special_Issue-9)
- \[Sim\+25\]Oriane Siméoni et al\.“DINOv3”, 2025arXiv:[https://arxiv\.org/abs/2508\.10104](https://arxiv.org/abs/2508.10104)
- \[Sin\+24\]Prajwal Singhania et al\.“Loki: Low\-rank Keys for Efficient Sparse Attention”, 2024arXiv:[https://arxiv\.org/abs/2406\.02542](https://arxiv.org/abs/2406.02542)
- \[SM26\]Tobias Schröder and Lester Mackey“WildCat: Near\-Linear Attention in Theory and Practice”, 2026arXiv:[https://arxiv\.org/abs/2602\.10056](https://arxiv.org/abs/2602.10056)
- \[Su\+23\]Jianlin Su et al\.“RoFormer: Enhanced Transformer with Rotary Position Embedding”, 2023arXiv:[https://arxiv\.org/abs/2104\.09864](https://arxiv.org/abs/2104.09864)
- \[Sun\+23\]Chengcheng Sun et al\.“Attention\-based graph neural networks: a survey”In*Artificial Intelligence Review*56\.S2Springer ScienceBusiness Media LLC, 2023, pp\. 2263–2310DOI:[10\.1007/s10462\-023\-10577\-2](https://dx.doi.org/10.1007/s10462-023-10577-2)
- \[SW02\]Andreas Stathopoulos and Kesheng Wu“A Block Orthogonalization Procedure with Constant Synchronization Requirements”In*SIAM Journal on Scientific Computing*23\.6, 2002, pp\. 2165–2182DOI:[10\.1137/S1064827500370883](https://dx.doi.org/10.1137/S1064827500370883)
- \[Tea\+25\]Gemma Team et al\.“Gemma 3 Technical Report”, 2025arXiv:[https://arxiv\.org/abs/2503\.19786](https://arxiv.org/abs/2503.19786)
- \[TGS06\]Joel\. Tropp, Anna\. Gilbert and Martin\. Strauss“Algorithms for simultaneous sparse approximation\. Part I: Greedy pursuit” Sparse Approximations in Signal and Image ProcessingIn*Signal Processing*86\.3, 2006, pp\. 572–588DOI:[https://doi\.org/10\.1016/j\.sigpro\.2005\.05\.030](https://dx.doi.org/https://doi.org/10.1016/j.sigpro.2005.05.030)
- \[Vas\+23\]Ashish Vaswani et al\.“Attention Is All You Need”, 2023arXiv:[https://arxiv\.org/abs/1706\.03762](https://arxiv.org/abs/1706.03762)
- \[Wan\+20\]Sinong Wang et al\.“Linformer: Self\-Attention with Linear Complexity”, 2020arXiv:[https://arxiv\.org/abs/2006\.04768](https://arxiv.org/abs/2006.04768)
- \[Wol\+20\]Thomas Wolf et al\.“Transformers: State\-of\-the\-Art Natural Language Processing”In*Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*Online: Association for Computational Linguistics, 2020, pp\. 38–45DOI:[10\.18653/v1/2020\.emnlp\-demos\.6](https://dx.doi.org/10.18653/v1/2020.emnlp-demos.6)
- \[WSH21\]Tizian Wenzel, Gabriele Santin and Bernard Haasdonk“A novel class of stabilized greedy kernel approximation algorithms: Convergence, stability and uniform point distribution”In*Journal of Approximation Theory*262, 2021, pp\. 105508DOI:[https://doi\.org/10\.1016/j\.jat\.2020\.105508](https://dx.doi.org/https://doi.org/10.1016/j.jat.2020.105508)
- \[WSH23\]Tizian Wenzel, Gabriele Santin and Bernard Haasdonk“Analysis of Target Data\-Dependent Greedy Kernel Algorithms: Convergence Rates forff\-,f⋅Pf\\cdot P\- andf/Pf/P\-Greedy”In*Constructive Approximation*57\.1, 2023, pp\. 45–74DOI:[10\.1007/s00365\-022\-09592\-3](https://dx.doi.org/10.1007/s00365-022-09592-3)
- \[Xia\+24\]Guangxuan Xiao et al\.“Efficient Streaming Language Models with Attention Sinks”, 2024arXiv:[https://arxiv\.org/abs/2309\.17453](https://arxiv.org/abs/2309.17453)
- \[Yan\+24\]Dongjie Yang et al\.“PyramidInfer: Pyramid KV Cache Compression for High\-throughput LLM Inference”, 2024arXiv:[https://arxiv\.org/abs/2405\.12532](https://arxiv.org/abs/2405.12532)
- \[Zan\+23\]Amir Zandieh, Insu Han, Majid Daliri and Amin Karbasi“KDEformer: Accelerating Transformers via Kernel Density Estimation”, 2023arXiv:[https://arxiv\.org/abs/2302\.02451](https://arxiv.org/abs/2302.02451)
- \[Zan\+25\]Amir Zandieh, Majid Daliri, Majid Hadian and Vahab Mirrokni“TurboQuant: Online Vector Quantization with Near\-optimal Distortion Rate”, 2025arXiv:[https://arxiv\.org/abs/2504\.19874](https://arxiv.org/abs/2504.19874)
- \[Zha\+23\]Zhenyu Zhang et al\.“H2O: Heavy\-Hitter Oracle for Efficient Generative Inference of Large Language Models”, 2023arXiv:[https://arxiv\.org/abs/2306\.14048](https://arxiv.org/abs/2306.14048)

## Appendix AProofs

###### Proof of[Proposition1](https://arxiv.org/html/2610.40943#Thmproposition1)\.

We first prove the result when the rows ofKKare, in addition, pairwise non\-collinear, i\.e\. such that, for all distincti,j∈\[N\]i,j\\in\[N\], there exists noα\\alphafor whichKi=α​KjK\_\{i\}=\\alpha K\_\{j\}\. LetΨ∈Span⁡\(Kϕ\)∩ϕ⁡\[Span⁡\(K\)\]\\Psi\\in\\operatorname\{Span\}\(K\_\{\\phi\}\)\\cap\\phi\\left\[\\operatorname\{Span\}\(K\)\\right\], we can then set, dropping the prefactor

Ψ=∑i=1Nαi​ϕ​\[Ki\]=\(\(dr/4​r\!\)−1​∑i=1Nαi​Ki⊗r\)r∈ℕ,\\Psi=\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}\\phi\[K\_\{i\}\]=\\left\(\(d^\{r/4\}\\sqrt\{r\!\}\)^\{\-1\}\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}K\_\{i\}^\{\\otimes r\}\\right\)\_\{r\\in\\mathbb\{N\}\},where, for alli∈\[N\]i\\in\[N\],αi∈ℝ\\alpha\_\{i\}\\in\\mathbb\{R\}\. SinceΨ∈ϕ⁡\[Span⁡\(K\)\]\\Psi\\in\\phi\[\\operatorname\{Span\}\(K\)\]there also exists av∈Span⁡\(K\)v\\in\\operatorname\{Span\}\(K\)such thatΨ=ϕ⁡\[v\]\\Psi=\\phi\[v\], so the degree\-rrcomponent isΨr=\(dr/4​r\!\)−1​v⊗r\\Psi\_\{r\}=\(d^\{r/4\}\\sqrt\{r\!\}\)^\{\-1\}v^\{\\otimes r\}\. For the casev=0v=0, we haveϕ0​\[v\]=1=∑i=1Nαi\\phi\_\{0\}\[v\]=1=\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}and for anyr≥1r\\geq 1,ϕr​\[v\]=∑i=1Nαi​ϕr​\[Ki\]=0\\phi\_\{r\}\[v\]=\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}\\phi\_\{r\}\[K\_\{i\}\]=0, which thanks to the linear independence we shall prove below implies that, for alli∈\[N\]i\\in\[N\],αi=0\\alpha\_\{i\}=0, and hence1=∑i=1Nαi=01=\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}=0\. This contradiction then requiresv≠0v\\neq 0\. Forv≠0v\\neq 0, we have for everyr≥1r\\geq 1, that the matricization ofΨ2​r\\Psi\_\{2r\}has rank11\. Let us show that this leads to a unique non\-zero coefficientαi=1\\alpha\_\{i\}=1for somei∈\[N\]i\\in\[N\]\. Indeed, for sufficiently largerr, the rank of the matricization ofΨ2​r=∑i=1Nαi​Ki⊗r⊗Ki⊗r\\Psi\_\{2r\}=\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}K\_\{i\}^\{\\otimes r\}\\otimes K\_\{i\}^\{\\otimes r\}is equal to the number of non\-zero coefficients\. Indeed, as the rows ofKKare pairwise linearly independent vectors, we have that, therr\-th tensor power of the keysKi⊗rK\_\{i\}^\{\\otimes r\}, for alli∈\[N\]i\\in\[N\], are linearly independent\[[Com\+08](https://arxiv.org/html/2610.40943#bib.bibx8), Corollary 4\.4\]\. Hence, the rank ofΨ2​r\\Psi\_\{2r\}is equal to the number of non\-zero coefficients\. Thus, as the rank needs to be equal to 1, we have that there exists a single non\-zero coefficientαj∈ℝ∗\\alpha\_\{j\}\\in\\mathbb\{R\}^\{\*\}such thatΨ=αj​ϕ​\[Kj\]\\Psi=\\alpha\_\{j\}\\phi\[K\_\{j\}\]for somejj\. The first result of[Lemma1](https://arxiv.org/html/2610.40943#Thmlemma1)then gives us thatαj=1\\alpha\_\{j\}=1\. Finally, thanks to the second result of Lemma[1](https://arxiv.org/html/2610.40943#Thmlemma1), we can extend the result toKKwith distinct, non\-zero rows\. Indeed, consider a key matrix whose rows formp∈\[N\]p\\in\[N\]equivalence classesC1,…,CpC\_\{1\},\\dots,C\_\{p\}for the pairwise collinearity, with representativesu1,…,upu\_\{1\},\\dots,u\_\{p\}, and setKi=ci​ukK\_\{i\}=c\_\{i\}u\_\{k\}for alli∈Cki\\in C\_\{k\}, where thecic\_\{i\}are non\-zero and distinct inside each equivalence class\. Letm∈\[N\]m\\in\[N\]be the cardinality of the biggest collinearity class,R:=max⁡\(1,p−1\)R:=\\max\(1,p\-1\),ℜ:=\{R,…,R\+m−1\}\\mathfrak\{R\}:=\\\{R,\\dots,R\+m\-1\\\}and, for anyx∈ℝdx\\in\\mathbb\{R\}^\{d\}, letϕℜ​\[x\]:=\(ϕr​\[x\]\)r∈ℜ\\phi\_\{\\mathfrak\{R\}\}\[x\]:=\(\\phi\_\{r\}\[x\]\)\_\{r\\in\\mathfrak\{R\}\}denote the truncated features\. First, the truncated featuresϕℜ​\[Ki\]\\phi\_\{\\mathfrak\{R\}\}\[K\_\{i\}\],i∈\[N\]i\\in\[N\], are linearly independent\. Indeed, letγ∈ℝN\\gamma\\in\\mathbb\{R\}^\{N\}be such that∑i=1Nγi​ϕℜ​\[Ki\]=0\\sum\_\{i=1\}^\{N\}\\gamma\_\{i\}\\phi\_\{\\mathfrak\{R\}\}\[K\_\{i\}\]=0, i\.e\. such that, for allr∈ℜr\\in\\mathfrak\{R\},

∑k=1p\(∑i∈Ckγi​cir\)​ϕr​\[uk\]=0\.\\sum\_\{k=1\}^\{p\}\\Big\(\\sum\_\{i\\in C\_\{k\}\}\\gamma\_\{i\}c\_\{i\}^\{r\}\\Big\)\\phi\_\{r\}\[u\_\{k\}\]=0\.Asr≥p−1r\\geq p\-1, therr\-th tensor powers of the representativesuk⊗ru\_\{k\}^\{\\otimes r\}are linearly independent\[[Com\+08](https://arxiv.org/html/2610.40943#bib.bibx8), Corollary 4\.4\], so that∑i∈Ckγi​cir=0\\sum\_\{i\\in C\_\{k\}\}\\gamma\_\{i\}c\_\{i\}^\{r\}=0for allk∈\[p\]k\\in\[p\]andr∈ℜr\\in\\mathfrak\{R\}, and the second result of Lemma[1](https://arxiv.org/html/2610.40943#Thmlemma1), applied inside each equivalence class, gives us thatγ=0\\gamma=0\. In particular, the casev=0v=0is excluded in the same way as above, as it would give∑i=1Nαi​ϕℜ​\[Ki\]=0\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}\\phi\_\{\\mathfrak\{R\}\}\[K\_\{i\}\]=0\. Then, in the same way as above, consider the matricization of∑i=1Nαi​ϕℜ​\[Ki\]⊗ϕℜ​\[Ki\]\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}\\phi\_\{\\mathfrak\{R\}\}\[K\_\{i\}\]\\otimes\\phi\_\{\\mathfrak\{R\}\}\[K\_\{i\}\]\. Since, for anyx∈ℝdx\\in\\mathbb\{R\}^\{d\}andr,s∈ℜr,s\\in\\mathfrak\{R\},ϕr​\[x\]⊗ϕs​\[x\]=\(r\+sr\)1/2​ϕr\+s​\[x\]\\phi\_\{r\}\[x\]\\otimes\\phi\_\{s\}\[x\]=\\binom\{r\+s\}\{r\}^\{1/2\}\\phi\_\{r\+s\}\[x\], its\(r,s\)\(r,s\)block is given by

∑i=1Nαi​ϕr​\[Ki\]⊗ϕs​\[Ki\]=\(r\+sr\)1/2​Ψr\+s=\(r\+sr\)1/2​ϕr\+s​\[v\]=ϕr​\[v\]⊗ϕs​\[v\],\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}\\phi\_\{r\}\[K\_\{i\}\]\\otimes\\phi\_\{s\}\[K\_\{i\}\]=\\binom\{r\+s\}\{r\}^\{1/2\}\\Psi\_\{r\+s\}=\\binom\{r\+s\}\{r\}^\{1/2\}\\phi\_\{r\+s\}\[v\]=\\phi\_\{r\}\[v\]\\otimes\\phi\_\{s\}\[v\],so that∑i=1Nαi​ϕℜ​\[Ki\]⊗ϕℜ​\[Ki\]=ϕℜ​\[v\]⊗ϕℜ​\[v\]\\sum\_\{i=1\}^\{N\}\\alpha\_\{i\}\\phi\_\{\\mathfrak\{R\}\}\[K\_\{i\}\]\\otimes\\phi\_\{\\mathfrak\{R\}\}\[K\_\{i\}\]=\\phi\_\{\\mathfrak\{R\}\}\[v\]\\otimes\\phi\_\{\\mathfrak\{R\}\}\[v\], whose matricization has rank11asv≠0v\\neq 0\. Now, as before, since the truncated feature vectors are all linearly independent, we have the same rank argument as in the non\-collinear case and thus the result, there is a uniquej∈\[N\]j\\in\[N\]such thatϕ⁡\[v\]=ϕ⁡\[Kj\]\\phi\[v\]=\\phi\[K\_\{j\}\]\. ∎

###### Proof of[Lemma1](https://arxiv.org/html/2610.40943#Thmlemma1)\.

Supposeϕ⁡\[v\]=α​ϕ​\[u\]\\phi\[v\]=\\alpha\\phi\[u\]for somev∈ℝdv\\in\\mathbb\{R\}^\{d\}andα∉\{0,1\}\\alpha\\notin\\\{0,1\\\}\. The equality of the degree\-11components givesv=α​uv=\\alpha u\. The equality of the degree\-22components then requiresα​ϕ2​\[u\]=ϕ2​\[α​u\]=α2​ϕ2​\[u\]\\alpha\\phi\_\{2\}\[u\]=\\phi\_\{2\}\[\\alpha u\]=\\alpha^\{2\}\\phi\_\{2\}\[u\]withϕ2​\[u\]≠0\\phi\_\{2\}\[u\]\\neq 0, soα∈\{0,1\}\\alpha\\in\\\{0,1\\\}, contradicting the supposition\.

For the second claim, letβ∈ℝk\\beta\\in\\mathbb\{R\}^\{k\}be such that, for allr∈\{R,…,R\+k−1\}r\\in\\\{R,\\dots,R\+k\-1\\\},

∑i=1kβi​ϕr​\[ui\]=\(∑i=1kβi​αir\)​ϕr​\[u\]=0\.\\sum\_\{i=1\}^\{k\}\\beta\_\{i\}\\phi\_\{r\}\[u\_\{i\}\]=\\Big\(\\sum\_\{i=1\}^\{k\}\\beta\_\{i\}\\alpha\_\{i\}^\{r\}\\Big\)\\phi\_\{r\}\[u\]=0\.Sinceu≠0u\\neq 0,ϕr​\[u\]≠0\\phi\_\{r\}\[u\]\\neq 0, this is equivalent to

∑i=1kβi​αir=0,r∈\{R,…,R\+k−1\}\.\\sum\_\{i=1\}^\{k\}\\beta\_\{i\}\\alpha\_\{i\}^\{r\}=0,\\qquad r\\in\\\{R,\\dots,R\+k\-1\\\}\.\(46\)We show by induction onkkthat this impliesβ=0\\beta=0, for anyR∈ℕR\\in\\mathbb\{N\}, any pairwise distinctα1,…,αk∈ℝ∖\{0\}\\alpha\_\{1\},\\dots,\\alpha\_\{k\}\\in\\mathbb\{R\}\\setminus\\\{0\\\}and anyβ∈ℝk\\beta\\in\\mathbb\{R\}^\{k\}\. Fork=1k=1, \([46](https://arxiv.org/html/2610.40943#A1.E46)\) impliesβ1​α1R=0\\beta\_\{1\}\\alpha\_\{1\}^\{R\}=0, henceβ1=0\\beta\_\{1\}=0asα1≠0\\alpha\_\{1\}\\neq 0\. Fork≥2k\\geq 2, assume the claim holds for thek−1k\-1vectors\. Let us considerβ\\betasuch that for allr∈\[R,R\+k−1\]r\\in\[R,R\+k\-1\],

∑i=1kβi​αir=∑i=1k−1βi​αir\+βk​αkr=0\.\\sum\_\{i=1\}^\{k\}\\beta\_\{i\}\\alpha\_\{i\}^\{r\}=\\sum\_\{i=1\}^\{k\-1\}\\beta\_\{i\}\\alpha\_\{i\}^\{r\}\+\\beta\_\{k\}\\alpha\_\{k\}^\{r\}=0\.For eachr∈\{R,…,R\+k−2\}r\\in\\\{R,\\dots,R\+k\-2\\\}, subtractingαk\\alpha\_\{k\}times the equation of degreerrfrom that of degreer\+1r\+1eliminates the last term, giving us

∑i=1kβi​αir\+1−αk​∑i=1kβi​αir=∑i=1k−1βi​\(αi−αk\)​αir=0\.\\sum\_\{i=1\}^\{k\}\\beta\_\{i\}\\alpha\_\{i\}^\{r\+1\}\-\\alpha\_\{k\}\\sum\_\{i=1\}^\{k\}\\beta\_\{i\}\\alpha\_\{i\}^\{r\}=\\sum\_\{i=1\}^\{k\-1\}\\beta\_\{i\}\(\\alpha\_\{i\}\-\\alpha\_\{k\}\)\\alpha\_\{i\}^\{r\}=0\.We identify this as the property \([46](https://arxiv.org/html/2610.40943#A1.E46)\) forα1,…,αk−1\\alpha\_\{1\},\\dots,\\alpha\_\{k\-1\}on thek−1k\-1consecutive degreesR,…,R\+k−2R,\\dots,R\+k\-2, with coefficientsβi​\(αi−αk\)\\beta\_\{i\}\(\\alpha\_\{i\}\-\\alpha\_\{k\}\)\. Hence, by the induction hypothesis,βi​\(αi−αk\)=0\\beta\_\{i\}\(\\alpha\_\{i\}\-\\alpha\_\{k\}\)=0, and since theαi\\alpha\_\{i\}are pairwise distinct,βi=0\\beta\_\{i\}=0for alli∈\[k−1\]i\\in\[k\-1\]\. The equation of degreeRRthen reduces toβk​αkR=0\\beta\_\{k\}\\alpha\_\{k\}^\{R\}=0, soβk=0\\beta\_\{k\}=0\. This proves the claim at rankkk, and thus for allk∈ℕ∗k\\in\\mathbb\{N\}^\{\*\}by induction\. The truncated features are thus linearly independent, hence so are theϕ⁡\[ui\]\\phi\[u\_\{i\}\], anddimSpan⁡\{ϕ⁡\[u1\],…,ϕ⁡\[uk\]\}=k\\dim\\operatorname\{Span\}\\\{\\phi\[u\_\{1\}\],\\dots,\\phi\[u\_\{k\}\]\\\}=k\. ∎

###### Proof of[Proposition2](https://arxiv.org/html/2610.40943#Thmproposition2)\.

1. \(i\.\)The numerator and denominator are linear inϕ⁡\[q\]\\phi\[q\], so any scalar cancels the ratio\.
2. \(ii\.\)The numerator and denominator are also linear in the key features\(ϕ⁡\[Ki\]\)i∈\[N\]\(\\phi\[K\_\{i\}\]\)\_\{i\\in\[N\]\}, so rescaling all of them byλ\\lambdamultiplies both byλ\\lambda\.
3. \(iii\.\)Translating all keys byδ\\deltamultiplies each evaluation of the kernel by the single factor per\-querye⟨q,δ⟩/de^\{\\langle q,\\delta\\rangle/\\sqrt\{d\}\}, which cancels in the ratio, Att⁡\(q,K\+δ,V\)=e⟨q,δ⟩/d​exp⁡\(q𝖳​K𝖳/d\)​Ve⟨q,δ⟩/d​exp⁡\(q𝖳​K𝖳/d\)​𝟏N=exp⁡\(q𝖳​K𝖳/d\)​Vexp⁡\(q𝖳​K𝖳/d\)​𝟏N=Att⁡\(q,K,V\)\.\\operatorname\{Att\}\(q,K\+\\delta,V\)=\\frac\{e^\{\\langle q,\\delta\\rangle/\\sqrt\{d\}\}\\exp\\left\(q^\{\\mathsf\{T\}\}K^\{\\mathsf\{T\}\}/\\sqrt\{d\}\\right\)V\}\{e^\{\\langle q,\\delta\\rangle/\\sqrt\{d\}\}\\exp\\left\(q^\{\\mathsf\{T\}\}K^\{\\mathsf\{T\}\}/\\sqrt\{d\}\\right\)\\mathbf\{1\}\_\{N\}\}=\\frac\{\\exp\\left\(q^\{\\mathsf\{T\}\}K^\{\\mathsf\{T\}\}/\\sqrt\{d\}\\right\)V\}\{\\exp\\left\(q^\{\\mathsf\{T\}\}K^\{\\mathsf\{T\}\}/\\sqrt\{d\}\\right\)\\mathbf\{1\}\_\{N\}\}=\\operatorname\{Att\}\(q,K,V\)\.\(47\)

∎

###### Proof of[Proposition4](https://arxiv.org/html/2610.40943#Thmproposition4)\.

κ˘\(x−δ,y−δ\)=exp\(−∥x−y∥2/2d\)=κ˘\(x,y\)\\breve\{\\kappa\}\(x\-\\delta,y\-\\delta\)=\\exp\(\-\\\|x\-y\\\|^\{2\}/2\\sqrt\{d\}\)=\\breve\{\\kappa\}\(x,y\), and the second identity corresponds to applying equation \([15](https://arxiv.org/html/2610.40943#S2.E15)\) to the translated matrices\. ∎

###### Proof of[Proposition5](https://arxiv.org/html/2610.40943#Thmproposition5)\.

The interpolated attention can be written as

Att~L−δ​\(q,K−δ,V\)=Eq,L−δ​EL−δ,L−δ†​EL−δ,K−δ​VEq,L−δ​EL−δ,L−δ†​EL−δ,K−δ​𝟏N\.\\widetilde\{\\operatorname\{Att\}\}\_\{L\-\\delta\}\(q,K\-\\delta,V\)=\\frac\{E\_\{q,L\-\\delta\}E^\{\\dagger\}\_\{L\-\\delta,L\-\\delta\}E\_\{L\-\\delta,K\-\\delta\}V\}\{E\_\{q,L\-\\delta\}E^\{\\dagger\}\_\{L\-\\delta,L\-\\delta\}E\_\{L\-\\delta,K\-\\delta\}\\mathbf\{1\}\_\{N\}\}\.Using[Proposition4](https://arxiv.org/html/2610.40943#Thmproposition4)and equation \([20](https://arxiv.org/html/2610.40943#S2.E20)\), we obtain

Att~L−δ​\(q,K−δ,V\)=Ξq,δ−1Ξq,δ−1​Eq,L​ΞL,δ​EL,L†​ΞL,δ​ΞL,δ−1​EL,K​ΞK,δ−1​VEq,L​ΞL,δ​EL,L†​ΞL,δ​ΞL,δ−1​EL,K​ΞK,δ−1​𝟏N\.\\widetilde\{\\operatorname\{Att\}\}\_\{L\-\\delta\}\(q,K\-\\delta,V\)=\\frac\{\\Xi\_\{q,\\delta\}^\{\-1\}\}\{\\Xi\_\{q,\\delta\}^\{\-1\}\}\\frac\{E\_\{q,L\}\\Xi\_\{L,\\delta\}E^\{\\dagger\}\_\{L,L\}\\Xi\_\{L,\\delta\}\\Xi\_\{L,\\delta\}^\{\-1\}E\_\{L,K\}\\Xi^\{\-1\}\_\{K,\\delta\}V\}\{E\_\{q,L\}\\Xi\_\{L,\\delta\}E^\{\\dagger\}\_\{L,L\}\\Xi\_\{L,\\delta\}\\Xi\_\{L,\\delta\}^\{\-1\}E\_\{L,K\}\\Xi\_\{K,\\delta\}^\{\-1\}\\mathbf\{1\}\_\{N\}\}\.Simplifying theΞ\\Xiterms then gives us

Att~L−δ​\(q,K−δ,V\)=Eq,L​ΞL,δ​EL,L†​EL,K​ΞK,δ−1​VEq,L​ΞL,δ​EL,L†​EL,K​ΞK,δ−1​𝟏N=Eq\+δ,L​EL,L†​EL−δ,K​VEq\+δ,L​EL,L†​EL−δ,K​𝟏N,\\widetilde\{\\operatorname\{Att\}\}\_\{L\-\\delta\}\(q,K\-\\delta,V\)=\\frac\{E\_\{q,L\}\\Xi\_\{L,\\delta\}E^\{\\dagger\}\_\{L,L\}E\_\{L,K\}\\Xi\_\{K,\\delta\}^\{\-1\}V\}\{E\_\{q,L\}\\Xi\_\{L,\\delta\}E^\{\\dagger\}\_\{L,L\}E\_\{L,K\}\\Xi\_\{K,\\delta\}^\{\-1\}\\mathbf\{1\}\_\{N\}\}=\\frac\{E\_\{q\+\\delta,L\}E^\{\\dagger\}\_\{L,L\}E\_\{L\-\\delta,K\}V\}\{E\_\{q\+\\delta,L\}E^\{\\dagger\}\_\{L,L\}E\_\{L\-\\delta,K\}\\mathbf\{1\}\_\{N\}\},where the first equality gives us the identity of the proposition\. For theii\-th rowℓδ\\ell\_\{\\delta\}ofL−δL\-\\delta,Eℓδ\+δ,L​EL,L†=ei𝖳E\_\{\\ell\_\{\\delta\}\+\\delta,L\}E^\{\\dagger\}\_\{L,L\}=e\_\{i\}^\{\\mathsf\{T\}\}and the equation simplifies to

Att~L−δ​\(ℓ−δ,K−δ,V\)=Eℓ−δ,K​VEℓ−δ,K​𝟏N=Att⁡\(ℓ−δ,K,V\),\\widetilde\{\\operatorname\{Att\}\}\_\{L\-\\delta\}\(\\ell\-\\delta,K\-\\delta,V\)=\\frac\{E\_\{\\ell\-\\delta,K\}V\}\{E\_\{\\ell\-\\delta,K\}\\mathbf\{1\}\_\{N\}\}=\\operatorname\{Att\}\(\\ell\-\\delta,K,V\),which shows that the interpolation is exact at the translated nodes, whose features are, by[Proposition1](https://arxiv.org/html/2610.40943#Thmproposition1), the only points ofϕ⁡\[ℝd\]\\phi\[\\mathbb\{R\}^\{d\}\]inℬδ\\mathcal\{B\}\_\{\\delta\}\. ∎

###### Proof of[Proposition6](https://arxiv.org/html/2610.40943#Thmproposition6)\.

Using the identities of \([15](https://arxiv.org/html/2610.40943#S2.E15)\), we haveEL​L†=DLϕ−1/2E˘L​L†DLϕ−1/2E\_\{LL\}^\{\\dagger\}=D\_\{L\_\{\\phi\}\}^\{\-1/2\}\\breve\{E\}\_\{LL\}^\{\\dagger\}D\_\{L\_\{\\phi\}\}^\{\-1/2\}andEL​K=DLϕ1/2​E˘L​K​DKϕ1/2=EK​L𝖳E\_\{LK\}=D\_\{L\_\{\\phi\}\}^\{1/2\}\\breve\{E\}\_\{LK\}D\_\{K\_\{\\phi\}\}^\{1/2\}=E\_\{KL\}^\{\\mathsf\{T\}\}\. Applying this to each block of the Schur complementSL:=EK​K−EK​L​EL​L†​EL​KS\_\{L\}:=E\_\{KK\}\-E\_\{KL\}E\_\{LL\}^\{\\dagger\}E\_\{LK\}, theDLϕD\_\{L\_\{\\phi\}\}terms cancel withinEK​L​EL​L†​EL​KE\_\{KL\}E\_\{LL\}^\{\\dagger\}E\_\{LK\}and thusSL=DKϕ1/2​S˘L​DKϕ1/2S\_\{L\}=D\_\{K\_\{\\phi\}\}^\{1/2\}\\breve\{S\}\_\{L\}D\_\{K\_\{\\phi\}\}^\{1/2\}\. Hence, usingEK​L​Waug=EK​L​EL​L†​EL​K​VaugE\_\{KL\}W^\{\\mathrm\{aug\}\}=E\_\{KL\}E\_\{LL\}^\{\\dagger\}E\_\{LK\}V^\{\\mathrm\{aug\}\}and∥ϕ⁡\[Ki\]∥H=κexp​\(Ki,Ki\)1/2=DKϕ1/2​\[i,i\]\\lVert\\phi\[K\_\{i\}\]\\rVert\_\{H\}=\\kappa\_\{\\exp\}\(K\_\{i\},K\_\{i\}\)^\{1/2\}=D\_\{K\_\{\\phi\}\}^\{1/2\}\[i,i\], we can write

K˘ϕMΔ=DK−1/2\(EK​KVaug−EK​LWaug\)=DK−1/2SLVaug=S˘LDK1/2Vaug,\\breve\{K\}\_\{\\phi\}M\_\{\\Delta\}=D\_\{K\}^\{\-1/2\}\\big\(E\_\{KK\}V^\{\\mathrm\{aug\}\}\-E\_\{KL\}W^\{\\mathrm\{aug\}\}\\big\)=D\_\{K\}^\{\-1/2\}S\_\{L\}V^\{\\mathrm\{aug\}\}=\\breve\{S\}\_\{L\}D\_\{K\}^\{1/2\}V^\{\\mathrm\{aug\}\},dividing both sides by∥DK1/2∥∞\\lVert D\_\{K\}^\{1/2\}\\rVert\_\{\\infty\}, gives us the formula\. The identityS˘L=K˘ϕ​ΠKϕ⊖Lϕ​K˘ϕ𝖳\\breve\{S\}\_\{L\}=\\breve\{K\}\_\{\\phi\}\\Pi\_\{K\_\{\\phi\}\\ominus L\_\{\\phi\}\}\\breve\{K\}\_\{\\phi\}^\{\\mathsf\{T\}\}follows fromΠKϕ⊖Lϕ​Kϕ𝖳=Kϕ𝖳−Lϕ𝖳​EL​L†​EL​K\\Pi\_\{K\_\{\\phi\}\\ominus L\_\{\\phi\}\}K\_\{\\phi\}^\{\\mathsf\{T\}\}=K\_\{\\phi\}^\{\\mathsf\{T\}\}\-L\_\{\\phi\}^\{\\mathsf\{T\}\}E\_\{LL\}^\{\\dagger\}E\_\{LK\}, and the positive semi\-definiteness fromΠKϕ⊖Lϕ\\Pi\_\{K\_\{\\phi\}\\ominus L\_\{\\phi\}\}being an orthogonal projector\. ∎

## Appendix BWildCat\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]temperature scaling as an oblique projection

WildCat introduces an alternative rescaled scalar product in the RKHS given byκτ​\(x,y\)=exp⁡\(⟨xτ,yτ⟩/d\)\\kappa\_\{\\tau\}\(x,y\)=\\exp\\left\(\\langle\\frac\{x\}\{\\tau\},\\frac\{y\}\{\\tau\}\\rangle/\\sqrt\{d\}\\right\)for a temperatureτ\>0\\tau\>0\. The selection and projection steps, ofLLandWWin our formulation, of WildCat, then consist in considering the orthogonal projection of the linear map in this metric\. In this subsection, we provide an equivalent formulation in terms of an oblique projection in the original kernel, allowing us to represent the WildCat temperature method as an oblique Petrov\-Galerkin projection\. Letα∈ℝ∖\{0\}\\alpha\\in\\mathbb\{R\}\\setminus\\\{0\\\}and for anyA∈ℝa×dA\\in\\mathbb\{R\}^\{a\\times d\}, setAϕα=\(ϕ⁡\[α​Ai\]\)i∈\[a\]A\_\{\\phi\}^\{\\alpha\}=\\left\(\\phi\[\\alpha A\_\{i\}\]\\right\)\_\{i\\in\[a\]\}and𝔇α=\(αr​Ir\)r∈ℕ\\mathfrak\{D\}\_\{\\alpha\}=\(\\alpha^\{r\}I\_\{r\}\)\_\{r\\in\\mathbb\{N\}\}\. We then have, for anyx∈ℝdx\\in\\mathbb\{R\}^\{d\},ϕ⁡\[α​x\]=𝔇α​ϕ​\[x\]\\phi\[\\alpha x\]=\\mathfrak\{D\}\_\{\\alpha\}\\phi\[x\], henceAϕα=Aϕ​𝔇αA\_\{\\phi\}^\{\\alpha\}=A\_\{\\phi\}\\mathfrak\{D\}\_\{\\alpha\}\. We note that𝔇α\\mathfrak\{D\}\_\{\\alpha\}is unbounded onHHfor\|α\|\>1\|\\alpha\|\>1, so that, forτ≠1\\tau\\neq 1, either𝔇τ\\mathfrak\{D\}\_\{\\tau\}or its inverse is unbounded\. The identities below are thus to be understood on the finite spans of feature vectors, the final expression ofPτP\_\{\\tau\}being bounded and used as its definition\. Let

ΠLϕα=Lϕα𝖳​Eα​L,α​L−1​Lϕα\\Pi\_\{L\_\{\\phi\}^\{\\alpha\}\}=\{L\_\{\\phi\}^\{\\alpha\}\}^\{\\mathsf\{T\}\}E\_\{\\alpha L,\\alpha L\}^\{\-1\}L\_\{\\phi\}^\{\\alpha\}be the orthogonal projector ontoSpan⁡\(Lϕα\)\\operatorname\{Span\}\(L\_\{\\phi\}^\{\\alpha\}\)\. Then the WildCatτ\\tauprojector is given by the oblique projector:

Pτ:=𝔇τ​ΠLϕτ−1​𝔇τ−1=Lϕ𝖳​Eτ−2​L,L−1​Lϕτ−2,P\_\{\\tau\}:=\\mathfrak\{D\}\_\{\\tau\}\\Pi\_\{L\_\{\\phi\}^\{\\tau^\{\-1\}\}\}\\mathfrak\{D\}\_\{\\tau\}^\{\-1\}=L\_\{\\phi\}^\{\\mathsf\{T\}\}E\_\{\\tau^\{\-2\}L,L\}^\{\-1\}L^\{\\tau^\{\-2\}\}\_\{\\phi\},and the set of exact interpolation nodes is given byℬLϕτ−2\\mathcal\{B\}\_\{L\_\{\\phi\}^\{\\tau^\{\-2\}\}\}\. In our Petrov\-Galerkin formulation, this amounts to a trial space given bySpan⁡\(Lϕ\)\\operatorname\{Span\}\(L\_\{\\phi\}\)and a test space given bySpan⁡\(Lϕτ−2\)\\operatorname\{Span\}\(L\_\{\\phi\}^\{\\tau^\{\-2\}\}\)\. Thanks to the obliqueness of the projector, it extrapolates differently than the orthogonal projector and as shown in\[[SM26](https://arxiv.org/html/2610.40943#bib.bibx44)\]its error can be bounded under reasonable assumptions\. For the weights, this gives us,

W=EL,τ−2​L−1​EL,τ−2​K​V\.W=E\_\{L,\\tau^\{\-2\}L\}^\{\-1\}E\_\{L,\\tau^\{\-2\}K\}V\.In the Gaussian reformulation, we have, for anyA∈ℝa×dA\\in\\mathbb\{R\}^\{a\\times d\},B∈ℝb×dB\\in\\mathbb\{R\}^\{b\\times d\}and anyα,β∈ℝ\\alpha,\\beta\\in\\mathbb\{R\},

Eα​A,β​B=DAα2/2​E˘α​A,β​B​DBβ2/2\.E\_\{\\alpha A,\\beta B\}=D^\{\\alpha^\{2\}/2\}\_\{A\}\\breve\{E\}\_\{\\alpha A,\\beta B\}D\_\{B\}^\{\\beta^\{2\}/2\}\.Using this, the fact thatEτ−2​L,L=Eτ−1​L,τ−1​LE\_\{\\tau^\{\-2\}L,L\}=E\_\{\\tau^\{\-1\}L,\\tau^\{\-1\}L\}and that, for anyα∈ℝ\\alpha\\in\\mathbb\{R\},E˘α​A,α​B=\(E˘A​B​\[i,j\]α2\)i∈\[a\],j∈\[b\]\\breve\{E\}\_\{\\alpha A,\\alpha B\}=\(\\breve\{E\}\_\{AB\}\[i,j\]^\{\\alpha^\{2\}\}\)\_\{i\\in\[a\],j\\in\[b\]\}, we obtain that

W=DL−τ−2/2E˘τ−1​L,τ−1​L−1E˘τ−1​L,τ−1​KDKτ−2/2VW=D\_\{L\}^\{\-\\tau^\{\-2\}/2\}\\breve\{E\}\_\{\\tau^\{\-1\}L,\\tau^\{\-1\}L\}^\{\-1\}\\breve\{E\}\_\{\\tau^\{\-1\}L,\\tau^\{\-1\}K\}\\,D\_\{K\}^\{\\tau^\{\-2\}/2\}\\,V\(48\)In theη\\etagauge with centering, the two\-step solve \([23](https://arxiv.org/html/2610.40943#S2.E23)\) then becomes

E˘τ−1​L,τ−1​LX=E˘τ−1​L,τ−1​KDiag⁡\(ηK\(c\)\)τ−2​Vaug⏟=:Vη,τaug,Waug\(c\)=Diag\(ηL\(c\)\)−τ−2X,\\breve\{E\}\_\{\\tau^\{\-1\}L,\\tau^\{\-1\}L\}X=\\breve\{E\}\_\{\\tau^\{\-1\}L,\\tau^\{\-1\}K\}\\underbrace\{\\operatorname\{Diag\}\\big\(\\eta^\{\(c\)\}\_\{K\}\\big\)^\{\\tau^\{\-2\}\}V^\{\\mathrm\{aug\}\}\}\_\{=:V^\{\\mathrm\{aug\}\}\_\{\\eta,\\tau\}\},\\qquad\{W^\{\\mathrm\{aug\}\}\}^\{\(c\)\}=\\operatorname\{Diag\}\\big\(\\eta^\{\(c\)\}\_\{L\}\\big\)^\{\-\\tau^\{\-2\}\}X,\(49\)while the readout \([24](https://arxiv.org/html/2610.40943#S2.E24)\) is unchanged\.

## Appendix CAdditional figures

Figures[11](https://arxiv.org/html/2610.40943#A3.F11)–[14](https://arxiv.org/html/2610.40943#A3.F14)repeat Figures[2](https://arxiv.org/html/2610.40943#S5.F2),[3](https://arxiv.org/html/2610.40943#S5.F3),[6](https://arxiv.org/html/2610.40943#S5.F6)and[7](https://arxiv.org/html/2610.40943#S5.F7)with, for every curve, the interquartile range over the112112query heads in dashed lines of the same colour\. For each head, the error is the median over the future queries of \([45](https://arxiv.org/html/2610.40943#S5.E45)\), as in the main text\. Reference levels lying far above the curves are not drawn and their values are given in the legend\.

Figure 11:Heavy hitter relative error \([45](https://arxiv.org/html/2610.40943#S5.E45)\) in numerator \(top,εg10%\\varepsilon\_\{g\}^\{10\\%\}\) and denominator \(bottom,εh10%\\varepsilon\_\{h\}^\{10\\%\}\) on the future queries, withtsink=32t\_\{\\mathrm\{sink\}\}=32and without range correction, as in Figure[2](https://arxiv.org/html/2610.40943#S5.F2)\. Solid: median over the112112query heads; dashed: interquartile range over the heads\.Figure 12:Heavy hitter relative attention errorεAtt10%\\varepsilon^\{10\\%\}\_\{\\operatorname\{Att\}\}on the future queries without \(top\) and with \(bottom\) range correction, withtsink=32t\_\{\\mathrm\{sink\}\}=32andtrec=0t\_\{\\mathrm\{rec\}\}=0, as in Figure[3](https://arxiv.org/html/2610.40943#S5.F3)\. Solid: median over the112112query heads; dashed: interquartile range over the heads\.Figure 13:Heavy hitter relative attention errorεAtt10%\\varepsilon^\{10\\%\}\_\{\\operatorname\{Att\}\}using GAIM exact v4 interpolation nodes for the projection methods of Table[1](https://arxiv.org/html/2610.40943#S3.T1)and without projection \(W=V\[𝔏,:\]W=V\[\\mathfrak\{L\},:\]\), as in Figure[6](https://arxiv.org/html/2610.40943#S5.F6)\. Solid: median over the112112query heads; dashed: interquartile range over the heads\.Figure 14:Heavy hitter relative attention errorεAtt10%\\varepsilon^\{10\\%\}\_\{\\operatorname\{Att\}\}using Q\-DAIM exact v5 interpolation nodes for the projection methods of Table[1](https://arxiv.org/html/2610.40943#S3.T1)and without projection \(W=V\[𝔏,:\]W=V\[\\mathfrak\{L\},:\]\), as in Figure[7](https://arxiv.org/html/2610.40943#S5.F7)\. Solid: median over the112112query heads; dashed: interquartile range over the heads\.Figures[15](https://arxiv.org/html/2610.40943#A3.F15)and[16](https://arxiv.org/html/2610.40943#A3.F16)give, for the same projections, the residual operator they leave on the keys, i\.e\. the relative squared norm‖Mη−L˘ϕ𝖳​X‖H2/‖Mη‖H2\\\|M\_\{\\eta\}\-\\breve\{L\}\_\{\\phi\}^\{\\mathsf\{T\}\}X\\\|\_\{H\}^\{2\}/\\\|M\_\{\\eta\}\\\|\_\{H\}^\{2\}of the augmented operator and of its denominator part, computed fromE˘K​K\\breve\{E\}\_\{KK\},E˘L​K\\breve\{E\}\_\{LK\}andE˘L​L\\breve\{E\}\_\{LL\}on the context keys, the sink excluded\. The GAIM exact nodes are those of the v1 variant: with the v4 nodes of Figure[6](https://arxiv.org/html/2610.40943#S5.F6), every projection leaves the whole operator, about0\.99990\.9999of its norm atn=512n=512\. The Galerkin projection, the orthogonal projection inHH, gives the smallest residual by construction, and the least\-squares projection and the values of the selected keys,W=V\[𝔏,:\]W=V\[\\mathfrak\{L\},:\], are within0\.1%0\.1\\%of it\. The revealed\-range projection is larger and peaks aroundn=129n=129, where the number of nodes reaches the revealed rank\. The WildCat temperature projection leaves the largest residual, above the norm of the denominator operator itself for smallnn, although it gives the lowest attention error at the future queries in Figures[6](https://arxiv.org/html/2610.40943#S5.F6)and[7](https://arxiv.org/html/2610.40943#S5.F7)\.

Figure 15:Relative squared norm of the residual operator on the keys left by each projection of Table[1](https://arxiv.org/html/2610.40943#S3.T1), and without projection \(W=V\[𝔏,:\]W=V\[\\mathfrak\{L\},:\]\), on GAIM exact v1 interpolation nodes \(L∞L^\{\\infty\}score\),tsink=32t\_\{\\mathrm\{sink\}\}=32\. Left: augmented operator; right: denominator\. Median over the1616KV heads\.Figure 16:Relative squared norm of the residual operator on the keys left by each projection of Table[1](https://arxiv.org/html/2610.40943#S3.T1), and without projection \(W=V\[𝔏,:\]W=V\[\\mathfrak\{L\},:\]\), on the Q\-DAIM exact v5 interpolation nodes of Figure[7](https://arxiv.org/html/2610.40943#S5.F7),tsink=32t\_\{\\mathrm\{sink\}\}=32\. Left: augmented operator; right: denominator\. Median over the1616KV heads\.
## Appendix DConditioning of the key Gram matrices

In this appendix, we illustrate the conditioning argument of §[2\.4\.1](https://arxiv.org/html/2610.40943#S2.SS4.SSS1)on the document used in §[5\.1](https://arxiv.org/html/2610.40943#S5.SS1)\. Figure[17](https://arxiv.org/html/2610.40943#A4.F17)shows, for each layer of the model and each of its four KV heads, the condition numberκ2\\kappa\_\{2\}of the Gram matrix of the context keys,EK​KE\_\{KK\}, and of the Gram matrix of then=512n=512interpolation nodes,EL​LE\_\{LL\}\. In both cases thetsink=32t\_\{\\mathrm\{sink\}\}=32sink keys are excluded, and the context consists of the first40964096tokens of the document, i\.e\.N=4064N=4064keys once the sinks are removed, rather than the40964096tokens following the sinks used in §[5\.1](https://arxiv.org/html/2610.40943#S5.SS1)\. The interpolation nodes are selected either by GAIM exact, using the v2 residual inL2L^\{2\}norm, or by Q\-DAIM exact, where the nodes past the revealed rank are chosen as described in Appendix[E](https://arxiv.org/html/2610.40943#A5)\.

For each of these sets of keys, we compare three Gram matrices\. The first one is the Gaussian Gram matrixE˘\\breve\{E\}which, by Proposition[4](https://arxiv.org/html/2610.40943#Thmproposition4), does not depend on the centering of the keys\. The two others correspond to the exponential Gram matrixE=D1/2​E˘​D1/2E=D^\{1/2\}\\breve\{E\}D^\{1/2\}of \([15](https://arxiv.org/html/2610.40943#S2.E15)\), computed either with the keys taken at the origin or with the keys centered on their mean\. The condition numbers are obtained from the eigenvalues computed in double precision, which only allows us to resolveκ2\\kappa\_\{2\}up to1/\(n​ϵ\)1/\(n\\epsilon\)\. Beyond this level, we instead give the lower bound

κ⁡\(D\)=maxi⁡Ei​imini⁡Ei​i=exp⁡\(maxi⁡‖Ki‖2−mini⁡‖Ki‖2d\),\\kappa\(D\)=\\frac\{\\max\_\{i\}E\_\{ii\}\}\{\\min\_\{i\}E\_\{ii\}\}=\\exp\\Big\(\\frac\{\\max\_\{i\}\\\|K\_\{i\}\\\|^\{2\}\-\\min\_\{i\}\\\|K\_\{i\}\\\|^\{2\}\}\{\\sqrt\{d\}\}\\Big\),which can be computed exactly as it only involves the diagonal ofEE, together with the upper boundκ⁡\(D\)​κ​\(E˘\)\\kappa\(D\)\\,\\kappa\(\\breve\{E\}\)\.

Over the112112KV heads of the model, the Gaussian Gram matrix of the context keys remains well conditioned, with a medianlog10⁡κ2\\log\_\{10\}\\kappa\_\{2\}of0\.410\.41and a maximum of4\.14\.1, reached in layer00\. The Gaussian Gram matrices of the512512interpolation nodes all stay below102\.410^\{2\.4\}, with medians of100\.1410^\{0\.14\}for Q\-DAIM and100\.3310^\{0\.33\}for GAIM exact\. On the other hand, the exponential Gram matrix of the keys taken at the origin is beyond the double\-precision resolution on105105of the112112heads, its diagonal bound having a median of1016\.810^\{16\.8\}and reaching1035410^\{354\}in layer2727\. Centering the keys on their mean lowers these values to a median of about1011\.510^\{11\.5\}and to at most103410^\{34\}, although4747heads still remain beyond the resolution\. For the512512interpolation nodes, the centered exponential Gram matrix has a median condition number of105\.610^\{5\.6\}for Q\-DAIM and1011\.010^\{11\.0\}for GAIM exact\. As discussed in §[2\.4\.1](https://arxiv.org/html/2610.40943#S2.SS4.SSS1), the ill\-conditioning of the interpolation problem is thus carried by the norm termsDD, and the solves in the Gaussian formulation remain well conditioned in every head\.

Figure 17:Condition numberκ2\\kappa\_\{2\}per layer of the Gram matrices of the context keys,EK​KE\_\{KK\}\(N=4064N=4064, left\), and of then=512n=512interpolation nodes selected by GAIM exact \(centre\) and Q\-DAIM exact \(right\),EL​LE\_\{LL\};tsink=32t\_\{\\mathrm\{sink\}\}=32sink keys excluded\. Blue: Gaussian GramE˘\\breve\{E\}\(any centering\); red: exponential GramEEwith the keys at the origin; orange: exponential Gram with the keys centred on their mean\. Lines: median over the four KV heads of the layer; small marks: one KV head\. Past the double\-precision resolution1/\(n​ϵ\)1/\(n\\epsilon\)\(dotted\), hollow marks give the lower boundκ⁡\(D\)\\kappa\(D\)and the bars extend toκ⁡\(D\)​κ​\(E˘\)\\kappa\(D\)\\,\\kappa\(\\breve\{E\}\); triangles mark the heads beyond104010^\{40\}, with the largest exponent of the layer\.
## Appendix EQ\-DAIM past the revealed rank

The operator Gram matrixGGof \([31](https://arxiv.org/html/2610.40943#S3.E31)\) is of size\(d\+1\)×\(d\+1\)\(d\+1\)\\times\(d\+1\)\. Hence, the revealed rangeUUhas rankr≤d\+1r\\leq d\+1and the evaluation matrixFFof \([34](https://arxiv.org/html/2610.40943#S3.E34)\) hasrrcolumns\. As the column\-pivoted QR of Algorithm[2](https://arxiv.org/html/2610.40943#alg2)deflates the rows ofFFin theserrcoordinates, oncerrpivots have been selected, the remaining rows then give zero up to numerical error and any further pivot is thus chosen on numerical noise\. For the document of §[5\.1](https://arxiv.org/html/2610.40943#S5.SS1), we observe that the norm of the pivot drops by about fifteen orders of magnitude after the129129\-th selected node, and even earlier in heads of lower numerical rank, for example after7777nodes in a head of layer00and after1010nodes in a head of layer1414\. However, the nodes beyondrrremain useful for the interpolation\. Indeed, each direction of the revealed range is in general a combination of the features of many keys\. We thus select these additional nodes by restarting the selection, in one of the two following ways\.

##### Q\-DAIM exact\.

In this case, the rangeU=K˘ϕ𝖳​CU=\\breve\{K\}\_\{\\phi\}^\{\\mathsf\{T\}\}Cis revealed only once and the selection proceeds in blocks\. Within each block, the nodes are given by the pivots of the column\-pivoted QR ofF𝖳F^\{\\mathsf\{T\}\}, so that the first block coincides with Algorithm[2](https://arxiv.org/html/2610.40943#alg2)\. Once the residual of the block collapses, i\.e\. once the largest remaining squared row norm falls below10−1210^\{\-12\}times that of the first pivot of the block, we replaceFFby

FL=K˘ϕ​ΠLϕ⟂​U=S˘L​C,F\_\{L\}=\\breve\{K\}\_\{\\phi\}\\,\\Pi^\{\\perp\}\_\{L\_\{\\phi\}\}\\,U=\\breve\{S\}\_\{L\}\\,C,whereS˘L\\breve\{S\}\_\{L\}is the projection of the Gram matrixEK​KE\_\{KK\}onto the orthogonal complement ofSpan⁡\(Lϕ\)\\operatorname\{Span\}\(L\_\{\\phi\}\)of Proposition[6](https://arxiv.org/html/2610.40943#Thmproposition6)\. The rows ofFLF\_\{L\}are the coordinates, in the revealed range, of the key features orthogonalized inHHagainst all the selected nodes, and a new block is started by applying the column\-pivoted QR toFL𝖳F\_\{L\}^\{\\mathsf\{T\}\}\. Furthermore,FLF\_\{L\}can be updated at a costO⁡\(N​r\)O\(Nr\)per selected node\. Indeed, settingσ=S˘L​\[i,i\]\\sigma=\\breve\{S\}\_\{L\}\[i,i\]andg=S˘L\[:,i\]/σg=\\breve\{S\}\_\{L\}\[:,i\]/\\sqrt\{\\sigma\}for the selected nodeii, we haveFL←FL−gFL\[i,:\]/σF\_\{L\}\\leftarrow F\_\{L\}\-g\\,F\_\{L\}\[i,:\]/\\sqrt\{\\sigma\}, where the columnsggform the partial Cholesky factor ofE˘K​K\\breve\{E\}\_\{KK\}, which allows us to obtain the next column ofS˘L\\breve\{S\}\_\{L\}at a costO⁡\(N​n\)O\(Nn\)\.

##### Q\-DAIM RFF\.

For the surrogate selection of §[3\.1\.5](https://arxiv.org/html/2610.40943#S3.SS1.SSS5), we instead proceed in rounds ofddnodes\. After each round, the coefficients of the selected nodes are computed using the exact kernel,X=E˘L​L−1​E˘L​K​VηaugX=\\breve\{E\}\_\{LL\}^\{\-1\}\\breve\{E\}\_\{LK\}V^\{\\mathrm\{aug\}\}\_\{\\eta\}\. The next round then reveals the range of, and selects the pivots on, the surrogate residual operatorMψ˘−L˘ψ˘𝖳​XM\_\{\\breve\{\\psi\}\}\-\\breve\{L\}\_\{\\breve\{\\psi\}\}^\{\\mathsf\{T\}\}Xinstead ofMψ˘M\_\{\\breve\{\\psi\}\}, so that each round has up tod\+1d\+1new directions to reveal\.

Figure[18](https://arxiv.org/html/2610.40943#A5.F18)compares both restart procedures with the single pass of Algorithm[2](https://arxiv.org/html/2610.40943#alg2), for the v1 and v5 variants\. The curves coincide up to about129129nodes, after which the restarts lower the norm of the residual operator on the keys\. Atn=512n=512, this norm decreases from8\.9×10−38\.9\\times 10^\{\-3\}\(v1\) and4\.3×10−34\.3\\times 10^\{\-3\}\(v5\) to3\.8×10−33\.8\\times 10^\{\-3\}for Q\-DAIM exact, and from1\.05×10−21\.05\\times 10^\{\-2\}\(v1\) and1\.47×10−21\.47\\times 10^\{\-2\}\(v5\) to3\.9×10−33\.9\\times 10^\{\-3\}for Q\-DAIM RFF, which then matches the exact selector\. The attention error on the future queries, on the other hand, changes little\. With range correction, the restarted selections and the single pass of Q\-DAIM exact reach0\.1470\.147–0\.1480\.148, compared to0\.1440\.144for the single pass of Q\-DAIM RFF\. Without range correction, the restarted selections reach0\.1840\.184–0\.1870\.187, compared to0\.1780\.178–0\.1880\.188for the single pass of Q\-DAIM exact and0\.1760\.176–0\.1770\.177for that of Q\-DAIM RFF\. As observed in §[5](https://arxiv.org/html/2610.40943#S5), a lower residual on the keys therefore does not necessarily translate into a lower error on the future queries\.

Figure 18:Q\-DAIM exact \(left\) and Q\-DAIM RFF \(right\) past the revealed rank, single pass of Algorithm[2](https://arxiv.org/html/2610.40943#alg2)\(dashed\) against the restarted selections of Appendix[E](https://arxiv.org/html/2610.40943#A5)\(solid\), for the v1 and v5 targets\. Top: relative squared norm of the augmented residual operator on the keys, without sink, as in Figure[1](https://arxiv.org/html/2610.40943#S5.F1)\. Middle and bottom: heavy hitter relative attention errorεAtt10%\\varepsilon^\{10\\%\}\_\{\\operatorname\{Att\}\}on the future queries withtsink=32t\_\{\\mathrm\{sink\}\}=32, without and with range correction, as in Figure[3](https://arxiv.org/html/2610.40943#S5.F3)\. Each curve is the median over the heads\. Dotted vertical lines: restarts, everyd=128d=128nodes for Q\-DAIM RFF; for Q\-DAIM exact, where at least half of the KV heads resetFF, i\.e\. after128128,256256and384384nodes for the v1 target and one node later per block for v5 \(on1414of the1616KV heads; the other two, of lower numerical rank, reset more often\)\.