The Query Knows What to Forget: A Second Erase Direction for Linear Attention
Summary
Introduces a Query-derived Erase Direction (QED) for linear attention models to improve retrieval and about double the usable context length, as demonstrated on the S-NIAH-1 benchmark.
View Cached Full Text
Cached at: 08/17/26, 10:11 AM
# The Query Knows What to Forget: A Second Erase Direction for Linear Attention Source: [https://arxiv.org/abs/2608.13668](https://arxiv.org/abs/2608.13668) [View PDF](https://arxiv.org/pdf/2608.13668) > Abstract:Linear attention keeps a state of fixed size\. At long context, many stored items share this state, and interference between them degrades retrieval\. Gated DeltaNet\-2 \(GDN\-2\), like every delta\-rule model before it, derives its erase vector from the key of the current token\. However, the interference in its reads is measured through the query, and the erase step cannot reach it\. We introduce the Query\-derived Erase Direction \(QED\)\. QED adds a second erase direction derived from the query and orthogonal to the key\. In the fast\-weight view, a key\-directed delta edit cannot change the key\-orthogonal part of a read\. It uses the editable part to cancel old\-state content measured along the query\. It also improves retrieval at every length past the training window, and it about doubles the usable context length on S\-NIAH\-1\. ## Submission history From: Dhruman Gupta \[[view email](https://arxiv.org/show-email/160615c3/2608.13668)\] **\[v1\]**Thu, 13 Aug 2026 18:04:53 UTC \(321 KB\)
Similar Articles
Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention
Proposes Erase-then-Delta Attention (EDA), a memory update rule for linear attention that decouples erase and write addresses to selectively suppress stale information before writing new content. Experiments on 2.5B dense and 25B MoE models demonstrate consistent gains in standard and long-context evaluations.
Subquadratic AI introduces SubQ-1.1-Small, a new model using Smart Sparse Attention
Subquadratic AI introduces SubQ-1.1-Small, a model leveraging Smart Sparse Attention to achieve near-perfect long-context retrieval up to 12M tokens with up to 1,000x attention compute reduction. It balances long-context optimization with strong general reasoning, outperforming baselines on benchmarks like NIAH and RULER.
Faster Query-Key Learning Sharpens Attention in Self-Attention Models
This paper analyzes how the parameterization of query-key and output-value circuits in self-attention models affects attention sharpness during training. Through gradient-flow analysis, they show that faster query-key learning relative to output-value learning produces sharper attention, improving interpretability without sacrificing predictive performance.
QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decoding
This paper introduces QEvict, a KV-cache management scheme for LLMs that uses recoverable quantized eviction to handle attention drift during long-context decoding, improving memory efficiency while preserving important historical context.
Q-Interference: Memory-Efficient Phase-Aware Quantum-Inspired Attention
This paper proposes Q-Interference, a memory-efficient quantum-inspired attention mechanism for GPT models that uses phase-aware scoring and an exact trigonometric factorization to improve token interaction without increasing memory overhead.