Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Summary
Sparse Delta Memory extends gated linear RNNs with sparse addressing to dramatically increase hidden state capacity for improved long-context learning and retrieval while maintaining computational efficiency.
View Cached Full Text
Cached at: 07/09/26, 03:40 PM
Paper page - Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Source: https://huggingface.co/papers/2607.07386
Abstract
Sparse Delta Memory extends gated linear RNNs with sparse addressing to dramatically increase hidden state capacity for improved long-context learning and retrieval while maintaining computational efficiency.
Linear attention modelsallow a fixed state size and a fixed amount of compute per token. However, due to their limited state size,linear attention modelsfall behind in long-context recall compared tosoftmax-attention-basedtransformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state ofgated linear RNNsto orders of magnitude higher capacity using asparse addressing scheme. SDM extends theGated DeltaNetarchitecture by replacing the dense key-value outer product with sparse reads and writes to a largeexplicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance onin-context learningandlong-context retrievaltasks. Moreover, by learning the initial state of the SDM memory and therefore using it as aparametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2607\.07386
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.07386 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.07386 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.07386 in a Space README.md to link it from this page.
Collections including this paper1
Similar Articles
Δ-Mem: Efficient Online Memory for Large Language Models
Proposes delta-Mem, a lightweight online memory mechanism that uses a compact state matrix updated by delta-rule learning to improve long-context performance of frozen LLMs without full fine-tuning or context extension.
SinkRec: Mitigating Semantic State Sink in Long Sequence Recommendation with Memory-Conditioned Gated Delta Networks
SinkRec introduces a hybrid memory-transition architecture to mitigate semantic state sink in long sequence recommendation, using memory-conditioned gated delta networks to decouple pattern storage from dynamic modeling, achieving linear-time efficiency.
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
This paper explores how an exponentially decaying memory module from RAT+ can improve query-aware sparse inference methods for long-context language models, demonstrating consistent accuracy gains across various sparse budgets on needle-in-a-haystack tasks.
Dynamic Linear Attention
DLA introduces adaptive state merging and capacity-bounded memory modeling for multi-state linear attention, improving long-context LLM performance.
Dynamic Linear Attention
This paper proposes DLA, a dynamic memory modeling framework for multi-state linear attention that adaptively merges states based on token information variation and maintains a fixed-size state cache, enabling better long-context representation without the quadratic complexity of standard attention.