MARCH: Scaling Recurrent Memory with Content-Routed State Anchors

# MARCH: Scaling Recurrent Memory with Content-Routed State Anchors
Source: [https://arxiv.org/html/2608.12435](https://arxiv.org/html/2608.12435)
\\setheadertext

MARCH\\setheaderlogos![[Uncaptioned image]](https://arxiv.org/html/2608.12435v1/assets/logo_lab.png)![[Uncaptioned image]](https://arxiv.org/html/2608.12435v1/assets/THUEE-logo.png)\\reportnumber

Kaisen Yang2∗Shu Yu1,3Ermo Hua1,2Ning Ding1,2Xia Hu1Bowen Zhou1,2Chaochao Lu1†♯Youbang Sun1,2‡♯ 1Shanghai AI Laboratory2Tsinghua University3Fudan University ∗Equal Contribution\.†Project Lead\.‡Technical Lead\.♯Corresponding Authors\.

###### Abstract

Transformers owe much of their strong long\-context retrieval capability to a token\-level memory that grows with context length\. This flexibility, however, incurs a quadratic computation complexity during training and a key–value cache that grows linearly during autoregressive inference\. Recurrent alternatives offer efficient decoding by compressing the entire history into a fixed\-size state, but often underperform on recall\-intensive tasks since earlier associations usually get overwritten by subsequent updates, and only the most recent contextual information is retained\. In this paper, we introduceMemory\-AnchorRouting acrossContextHistory \(MARCH\), a network architecture that effectively scales state\-space models beyond a fixed\-size dimension, while maintaining computational efficiency over long\-sequences\. MARCH periodically caches cumulative recurrent\-state checkpoints as*state anchors*and associates each anchor with a compact, content\-conditioned anchor key\. This lets MARCH maintain a memory bank, which can grow as context length increases, providing a controllable trade\-off between historical resolution and memory cost\. At each token, MARCH produces an anchor query to attend all causally available state anchors, and the output is calculated as an attention\-style aggregation over all historical anchors along the current state\. We show that after standard pretraining, MARCH consistently outperforms multiple linear attention variants across commonsense reasoning, LongBench, and in\-context retrieval\. These results demonstrate that content\-routed state caching substantially strengthens recurrent long\-range memory while preserving its native computation path\.

![Refer to caption](https://arxiv.org/html/2608.12435v1/x1.png)Figure 1:Overview of MARCH\. Left: long\-context retrieval performance at a context length of 8K for MARCH and Gated DeltaNet\. Right: State anchors enable content\-based retrieval of historical recurrent states\.## 1Introduction

Large language models \(LLMs\) have demonstrated remarkable capabilities across a wide range of language understanding and generation tasks\. However, many real\-world applications—including long\-document understanding, multi\-turn interaction, and in\-context learning—require models to integrate information distributed across extended sequences\[bai2025longbench,kwan2024mt,zou2025many\]\. Supporting such contexts involves more than simply increasing the number of input tokens: models must retain relevant information across long intervening spans and reliably retrieve it when needed\[liu2023lost,hsieh2024ruler\]\. Effective long\-context modeling therefore hinges on a model’s ability to manage memory—determining what information to preserve, how to represent it, and when to retrieve it\[wang2023augmenting,behrouz2024titans\]\. A useful perspective is to view a sequence model as a memory system with two basic operations:*writing*, which incorporates each new input into memory, and*reading*, which retrieves information relevant to the current input\[behrouz2024titans\]\. Under this view, standard self\-attention maintains a growing token\-level memory in its key–value cache\[vaswani2017attention\]: it writes by appending each new key–value pair without compressing the existing cache, and reads by matching the current query against all stored keys and combining their associated values\. This uncompressed, token\-level memory provides a direct path to every preceding token, enabling accurate recall of fine\-grained details and distant dependencies\. Its flexibility, however, entails quadratic computation during training and a key–value cache that grows linearly with sequence length during autoregressive inference, making self\-attention increasingly costly as context windows expand\.

Linear attention and modern recurrent sequence models make the opposite trade\-off\[katharopoulos2020transformers,dao2024transformers\]\. They compress the causal prefix into a fixed\-size, matrix\-valued recurrent state, update this state with each new input, and read from it by applying the current query\. This design enables constant\-memory recurrent decoding, but the same compressed write that makes it efficient also limits its long\-range memory\. At each step, information from the new token is written into a state already shared by the entire history\. Much recent work has therefore focused on improving the write operation\. Selective state\-space models such as Mamba introduce input\-dependent state transitions and forgetting\[gu2023mamba\], whereas DeltaNet and Gated DeltaNet use data\-dependent delta\-rule updates to revise existing associations before incorporating new information\[schlag2021lineartransformersselfretrievalcompressive,yang2025gateddelta\]\. These mechanisms improve state tracking and mitigate indiscriminate accumulation, but they do not eliminate the underlying fixed\-state bottleneck: the entire history must still share one evolving state, and only its latest version remains available for reading\. Indeed, although recent linear recurrent models can match or surpass softmax attention in short\-context settings, their performance often degrades as the evaluation context grows\[arora2024zoology,wang2026dynamic\]\. Once an earlier association has been weakened by forgetting or modified by subsequent writes, the model has no direct path to its earlier representation and cannot recover it from the latest state alone\.

Recent works have relaxed the fixed\-state bottleneck along two broad directions\. One approach increases the memory capacity available at each step through partitioned sparse states, large memories with sparse reads and writes, or routed mixtures of independent states\[pan25SSE,cabannes26SDM,du2025mom\]\. A second line preserves a temporally structured collection of compressed states through logarithmic hierarchies, adaptive state construction and merging, or recurrent\-state caching\[guo2026loglinear,wang2026dynamic,behrouz26memory\]\. Collectively, these approaches demonstrate that expanding memory capacity or temporal coverage can improve long\-range recall\. However, with the exception of certain instances in\[behrouz26memory\], all of the existing works still maintain a finite or upper\-constrained state space dimension, though the dimensionality of which is increased\. Moreover, as multiple states become available, the primary bottleneck shifts from memory construction to memory retrieval: the model must determine which state retains the information most relevant to the current token\. In particular, how to construct context\-dependent representations for historical checkpoints that facilitate effective and efficient query\-dependent retrieval remains underexplored\.

In this work, we introduceMemory\-AnchorRouting acrossContextHistory \(MARCH\), a memory\-augmented recurrent architecture that enables selective retrieval from earlier versions of recurrent memory\. Without modifying the underlying recurrence, MARCH periodically preserves cumulative states as*state anchors*, giving later tokens access to earlier versions of the evolving memory\. Each anchor is associated with a compact learned descriptor, allowing the model to route each token to relevant historical states when additional context is needed and combine their contents with the current\-state readout\. MARCH thereby complements efficient recurrent processing with selective access to preserved historical memory\. The resulting mechanism remains causal and is trained end to end with the standard language\-modeling objective\.

Our main contributions are summarized as follows:

- •Routable memory\-state bank\.We introduce MARCH, which expands fixed\-state recurrence into a growing bank of historical memory states which increases its capacity as context grows, alleviating the single\-state memory bottleneck without modifying the underlying recurrence\.
- •Content\-conditioned historical retrieval\.MARCH brings attention\-style content routing to recurrent memory by applying a standard softmax over compact keys for a temporally sparse set of state anchors rather than token\-level key–value pairs\. A learned null route allows the model to suppress the historical branch when the current recurrent state is sufficient, while residual fusion preserves the original recurrent path and supports end\-to\-end training\.
- •Extensive empirical validation\.We demonstrate consistent improvements over strong recurrent baselines across commonsense reasoning, LongBench, in\-context retrieval, and NIAH evaluations, including robust extrapolation beyond the training context length\.

## 2Preliminaries

#### Full and Linear Attention as Memory\.

Let𝐱t∈ℝd\\mathbf\{x\}\_\{t\}\\in\\mathbb\{R\}^\{d\}denote the hidden representation at positiontt\. The corresponding query, key, and value vectors are obtained through learned linear projections:

𝐪t=𝐖q​𝐱t,𝐤t=𝐖k​𝐱t,𝐯t=𝐖v​𝐱t,\\mathbf\{q\}\_\{t\}=\\mathbf\{W\}\_\{q\}\\mathbf\{x\}\_\{t\},\\qquad\\mathbf\{k\}\_\{t\}=\\mathbf\{W\}\_\{k\}\\mathbf\{x\}\_\{t\},\\qquad\\mathbf\{v\}\_\{t\}=\\mathbf\{W\}\_\{v\}\\mathbf\{x\}\_\{t\},\(1\)where𝐖q,𝐖k∈ℝdk×d\\mathbf\{W\}\_\{q\},\\mathbf\{W\}\_\{k\}\\in\\mathbb\{R\}^\{d\_\{k\}\\times d\}and𝐖v∈ℝdv×d\\mathbf\{W\}\_\{v\}\\in\\mathbb\{R\}^\{d\_\{v\}\\times d\}, such that𝐪t,𝐤t∈ℝdk\\mathbf\{q\}\_\{t\},\\mathbf\{k\}\_\{t\}\\in\\mathbb\{R\}^\{d\_\{k\}\}and𝐯t∈ℝdv\\mathbf\{v\}\_\{t\}\\in\\mathbb\{R\}^\{d\_\{v\}\}\. Following the memory\-system perspective adopted in prior work\[behrouz2024titans\], we view a causal sequence mixer as an online memory system\. Letℳt\\mathcal\{M\}\_\{t\}denote the memory state after processing the firsttttokens, withℳ0\\mathcal\{M\}\_\{0\}denoting its initial state\. At each position, the current key–value pair is first written into memory, after which the updated memory is queried using the current query:

ℳt=Write⁡\(ℳt−1;𝐤t,𝐯t\),𝐨t=Read⁡\(ℳt;𝐪t\),\\mathcal\{M\}\_\{t\}=\\operatorname\{Write\}\\bigl\(\\mathcal\{M\}\_\{t\-1\};\\mathbf\{k\}\_\{t\},\\mathbf\{v\}\_\{t\}\\bigr\),\\qquad\\mathbf\{o\}\_\{t\}=\\operatorname\{Read\}\\bigl\(\\mathcal\{M\}\_\{t\};\\mathbf\{q\}\_\{t\}\\bigr\),\(2\)where𝐨t∈ℝdv\\mathbf\{o\}\_\{t\}\\in\\mathbb\{R\}^\{d\_\{v\}\}denotes the memory readout at positiontt\. For causal softmax attention, the memory explicitly retains all projected key–value pairs observed up to positiontt:

ℳt=\(𝐊≤t,𝐕≤t\),\\mathcal\{M\}\_\{t\}=\\bigl\(\\mathbf\{K\}\_\{\\leq t\},\\mathbf\{V\}\_\{\\leq t\}\\bigr\),\(3\)where𝐊≤t∈ℝt×dk\\mathbf\{K\}\_\{\\leq t\}\\in\\mathbb\{R\}^\{t\\times d\_\{k\}\}and𝐕≤t∈ℝt×dv\\mathbf\{V\}\_\{\\leq t\}\\in\\mathbb\{R\}^\{t\\times d\_\{v\}\}stack the keys and values row\-wise, respectively\. Writing appends\(𝐤t,𝐯t\)\(\\mathbf\{k\}\_\{t\},\\mathbf\{v\}\_\{t\}\)to these matrices, whereas reading performs content\-based retrieval:

𝐨t=𝐕≤t⊤​softmax⁡\(𝐊≤t​𝐪tdk\)\.\\mathbf\{o\}\_\{t\}=\\mathbf\{V\}\_\{\\leq t\}^\{\\top\}\\operatorname\{softmax\}\\left\(\\frac\{\\mathbf\{K\}\_\{\\leq t\}\\mathbf\{q\}\_\{t\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)\.\(4\)This explicit storage keeps individual tokens directly retrievable and enables fine\-grained retrieval from the entire causal prefix\. However, processing a sequence of lengthTTrequires𝒪​\(T2\)\\mathcal\{O\}\(T^\{2\}\)query–key interactions, while autoregressive decoding maintains a key–value cache of size𝒪​\(T​\(dk\+dv\)\)\\mathcal\{O\}\\\!\\left\(T\(d\_\{k\}\+d\_\{v\}\)\\right\)per attention head\[vaswani2017attention\]\.

Linear attention instantiates the memory stateℳt\\mathcal\{M\}\_\{t\}as a fixed\-size matrix𝐒t∈ℝdv×dk\\mathbf\{S\}\_\{t\}\\in\\mathbb\{R\}^\{d\_\{v\}\\times d\_\{k\}\}\. Its write and read operations are given by

𝐒t=𝐒t−1\+𝐯t​𝐤t⊤,𝐨t=𝐒t​𝐪t\.\\mathbf\{S\}\_\{t\}=\\mathbf\{S\}\_\{t\-1\}\+\\mathbf\{v\}\_\{t\}\\mathbf\{k\}\_\{t\}^\{\\top\},\\qquad\\mathbf\{o\}\_\{t\}=\\mathbf\{S\}\_\{t\}\\mathbf\{q\}\_\{t\}\.\(5\)Each write therefore adds a rank\-one key–value association to the shared matrix, while each read retrieves a query\-dependent superposition of the stored values\. This enables constant\-memory recurrent decoding, but introduces interference as the compressed history grows\.

#### Gated DeltaNet \(GDN\)\.

To mitigate the interference caused by the additive write rule of linear attention, GDN retains the same state\-based read operation but introduces input\-dependent retention and a targeted delta\-rule write\[yang2025gateddelta\]:

𝐒t=αt​𝐒t−1\+βt​\(𝐯t−αt​𝐒t−1​𝐤t\)​𝐤t⊤,𝐨t=𝐒t​𝐪t,\\mathbf\{S\}\_\{t\}=\{\\color\[rgb\]\{1,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{1,0,0\}\\alpha\_\{t\}\}\\mathbf\{S\}\_\{t\-1\}\+\{\\color\[rgb\]\{0,0,1\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0,1\}\\beta\_\{t\}\}\\left\(\\mathbf\{v\}\_\{t\}\-\{\\color\[rgb\]\{1,0,0\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{1,0,0\}\\alpha\_\{t\}\}\\mathbf\{S\}\_\{t\-1\}\\mathbf\{k\}\_\{t\}\\right\)\\mathbf\{k\}\_\{t\}^\{\\top\},\\qquad\\mathbf\{o\}\_\{t\}=\\mathbf\{S\}\_\{t\}\\mathbf\{q\}\_\{t\},\(6\)whereαt∈\(0,1\)\\alpha\_\{t\}\\in\(0,1\)is an input\-dependent retention gate, andβt∈\[0,1\]\\beta\_\{t\}\\in\[0,1\]modulates the strength of the targeted delta update\. Despite its more adaptive state dynamics, GDN still compresses the entire causal history into a single fixed\-size recurrent state\. Because all associations share this evolving state, information weakened or overwritten by subsequent updates has no direct retrieval path, limiting reliable long\-context recall\.

#### Scaling recurrent memory\.

Recent work has sought to relax the fixed\-state bottleneck along two broad directions\. Capacity\-expansion methods enlarge the current recurrent state, whereas temporal\-expansion methods retain multiple versions of the state along its trajectory:

ℳtcap\\displaystyle\\mathcal\{M\}\_\{t\}^\{\\mathrm\{cap\}\}≔𝐒~t=\[𝐒t\(1\)​∣⋯∣​𝐒t\(P\)\]∈ℝdv×Dmem,Dmem=∑p=1Pdp,\\displaystyle\\coloneqq\\widetilde\{\\mathbf\{S\}\}\_\{t\}=\\bigl\[\\mathbf\{S\}\_\{t\}^\{\(1\)\}\\mid\\cdots\\mid\\mathbf\{S\}\_\{t\}^\{\(P\)\}\\bigr\]\\in\\mathbb\{R\}^\{d\_\{v\}\\times D\_\{\\mathrm\{mem\}\}\},\\qquad D\_\{\\mathrm\{mem\}\}=\\sum\_\{p=1\}^\{P\}d\_\{p\},\(7\)ℳttemp\\displaystyle\\mathcal\{M\}\_\{t\}^\{\\mathrm\{temp\}\}≔\(𝐒¯t,1,…,𝐒¯t,Mt\),𝐒¯t,m∈ℝdv×dk,1≤τt,1<⋯<τt,Mt≤t\.\\displaystyle\\coloneqq\\bigl\(\\overline\{\\mathbf\{S\}\}\_\{t,1\},\\ldots,\\overline\{\\mathbf\{S\}\}\_\{t,M\_\{t\}\}\\bigr\),\\qquad\\overline\{\\mathbf\{S\}\}\_\{t,m\}\\in\\mathbb\{R\}^\{d\_\{v\}\\times d\_\{k\}\},\\qquad 1\\leq\\tau\_\{t,1\}<\\cdots<\\tau\_\{t,M\_\{t\}\}\\leq t\.In the capacity formulation,PPis the fixed number of state partitions andDmemD\_\{\\mathrm\{mem\}\}is their total memory dimension\. Such methods increaseDmemD\_\{\\mathrm\{mem\}\}while using sparse access to keep computation tractable\[pan25SSE,cabannes26SDM\]\. In the temporal formulation,MtM\_\{t\}is the number of retained state representations, each associated with a temporal boundaryτt,m\\tau\_\{t,m\}\. These methods preserve states from distinct temporal regions or earlier stages of the recurrent trajectory\[guo2026loglinear,wang2026dynamic,behrouz26memory\]\. MARCH follows the latter direction by retaining cumulative snapshots of a continuously evolving recurrent state\.

## 3Method

![Refer to caption](https://arxiv.org/html/2608.12435v1/x2.png)Figure 2:Architecture of MARCH\. Left: Text tokens update one continuous Gated DeltaNet state, while periodic checkpoints form state anchors\. Token\-dependent routing combines causally visible anchors, and the resulting historical readout is added to the current\-state readout\. Right: MARCH augments every recurrent layer and rebuilds state\-aware routing keys across layers\.Figure[2](https://arxiv.org/html/2608.12435#S3.F2)illustrates MARCH, a content\-routed recurrent memory framework that enables selective retrieval from earlier versions of an evolving recurrent state\. Rather than routing over predefined state indices or temporal scales, MARCH matches each query against individual historical states based on their contents\. As tokens are processed, MARCH periodically checkpoints the cumulative recurrent state, producing a bank of state anchors\. Each checkpoint is paired with an occurrence of a shared learned anchor token, whose hidden representation yields a compact*routing key*\. For each text token, a*routing query*scores all causally visible anchors alongside a learned null option, allowing the model to use historical memory only when useful\. The resulting routing probabilities define a weighted combination of the visible anchor states, which is read using the token’s standard recurrent query\. This historical readout is added to the current\-state readout, preserving the native recurrent path while introducing a content\-dependent route to earlier memory\. Together, state anchoring and content\-routed retrieval turn the otherwise transient state trajectory into a persistent source of long\-range memory\.

### 3\.1Continuous Recurrent\-State Anchoring

#### Anchor placement\.

Let𝒯=\[t1,…,tL\]\\mathcal\{T\}=\[t\_\{1\},\\ldots,t\_\{L\}\]be a sequence ofLLtext tokens\. An anchoring policy specifies an ordered set of text boundariesℬ=\{bm\}m=1M\\mathcal\{B\}=\\\{b\_\{m\}\\\}\_\{m=1\}^\{M\}, where0=b0<b1<⋯<bM≤L0=b\_\{0\}<b\_\{1\}<\\cdots<b\_\{M\}\\leq L\. We insert an anchor position after each boundary:

𝒯^=∥m=1M\(\[tbm−1\+1,…,tbm\]∥\[ξm\]\)∥\[tbM\+1,…,tL\]\.\\widehat\{\\mathcal\{T\}\}=\\mathop\{\\mathbin\{\\\|\}\}\_\{m=1\}^\{M\}\\Big\(\[t\_\{b\_\{m\-1\}\+1\},\\ldots,t\_\{b\_\{m\}\}\]\\mathbin\{\\\|\}\[\\xi\_\{m\}\]\\Big\)\\mathbin\{\\\|\}\[t\_\{b\_\{M\}\+1\},\\ldots,t\_\{L\}\]\.\(8\)where∥\\\|denotes sequence concatenation, andξm\\xi\_\{m\}is themm\-th occurrence of a shared learned anchor embeddingξ\\xi\. Text and anchor positions serve different computational roles\. Text positions apply the base recurrent update, allowing the matrix\-valued state to evolve continuously across anchor boundaries\. Immediately after processingtbmt\_\{b\_\{m\}\}, MARCH checkpoints the resulting cumulative state to form themm\-th state anchor\. The following anchor positionξm\\xi\_\{m\}does not modify the recurrent state; instead, its hidden representation provides the routing metadata associated with that checkpoint\. Thus, each anchor boundary produces two coupled objects: a snapshot of the recurrent memory and a compact representation through which that snapshot can later be retrieved\. We formalize these two operations next\.

#### Cumulative recurrent\-state checkpointing\.

At layerℓ\\ell, MARCH leaves the underlying recurrent update unchanged and carries the state𝐒t\(ℓ\)∈ℝdv×dk\\mathbf\{S\}^\{\(\\ell\)\}\_\{t\}\\in\\mathbb\{R\}^\{d\_\{v\}\\times d\_\{k\}\}continuously across anchor boundaries\. At each boundarybmb\_\{m\}, it snapshots the current state:

𝐀\(m,ℓ\)=𝐒bm\(ℓ\)∈ℝdv×dk,m=1,…,M\.\\mathbf\{A\}^\{\(m,\\ell\)\}=\\mathbf\{S\}^\{\(\\ell\)\}\_\{b\_\{m\}\}\\in\\mathbb\{R\}^\{d\_\{v\}\\times d\_\{k\}\},\\qquad m=1,\\ldots,M\.\(9\)Because the recurrence is not reset between anchor boundaries,𝐀\(m,ℓ\)\\mathbf\{A\}^\{\(m,\\ell\)\}encodes the cumulative prefix up to positionbmb\_\{m\}, rather than only the segment since the preceding boundary\. We therefore refer to it as a*state anchor*\. The ordered bank\{𝐀\(1,ℓ\),…,𝐀\(M,ℓ\)\}\\\{\\mathbf\{A\}^\{\(1,\\ell\)\},\\ldots,\\mathbf\{A\}^\{\(M,\\ell\)\}\\\}traces the temporal evolution of a single recurrent memory, preserving earlier versions before subsequent decay and delta updates attenuate or modify their contents\.

#### Content\-conditioned anchor metadata\.

Let𝐮m\(ℓ\)\\mathbf\{u\}^\{\(\\ell\)\}\_\{m\}denote the normalized input representation of anchor positionξm\\xi\_\{m\}at layerℓ\\ell\. The anchor position reads only its aligned state checkpoint:

𝐪m\(ℓ\)=𝐖q\(ℓ\)​𝐮m\(ℓ\),𝐨m\(ℓ\)=𝐀\(m,ℓ\)​𝐪m\(ℓ\)\.\\mathbf\{q\}^\{\(\\ell\)\}\_\{m\}=\\mathbf\{W\}^\{\(\\ell\)\}\_\{q\}\\mathbf\{u\}^\{\(\\ell\)\}\_\{m\},\\qquad\\mathbf\{o\}^\{\(\\ell\)\}\_\{m\}=\\mathbf\{A\}^\{\(m,\\ell\)\}\\mathbf\{q\}^\{\(\\ell\)\}\_\{m\}\.\(10\)The same input representation is projected into a compact routing key:

𝜿m\(ℓ\)=𝐖k\(ℓ\)​𝐮m\(ℓ\)∈ℝdr\.\\boldsymbol\{\\kappa\}^\{\(\\ell\)\}\_\{m\}=\\mathbf\{W\}^\{\(\\ell\)\}\_\{k\}\\mathbf\{u\}^\{\(\\ell\)\}\_\{m\}\\in\\mathbb\{R\}^\{d\_\{r\}\}\.\(11\)The aligned readout is incorporated into the anchor position through the standard output projection and residual pathway\. Consequently,𝐮m\(ℓ\+1\)\\mathbf\{u\}^\{\(\\ell\+1\)\}\_\{m\}depends on𝐀\(m,ℓ\)\\mathbf\{A\}^\{\(m,\\ell\)\}, and the routing key produced at layerℓ\+1\\ell\+1becomes conditioned on the content retained by the aligned state anchor\. Thus, although all anchor positions share the same learned input embedding, they acquire distinct, state\-dependent representations after the first layer\. This cross\-layer construction makes routing explicitly dependent on what each state anchor contains, rather than only on its temporal index\.

### 3\.2Content\-Routed Historical Reading

#### Content\-based routing\.

For a text token at positiontt, the causally available state anchors are indexed by𝒱t=\{m∈\{1,…,M\}∣bm<t\}\\mathcal\{V\}\_\{t\}=\\\{m\\in\\\{1,\\ldots,M\\\}\\mid b\_\{m\}<t\\\}\. MARCH projects the normalized hidden state𝐱t\\mathbf\{x\}\_\{t\}into a routing query and scores it against the key of each visible anchor:

𝝆t=𝐖R​𝐱t,at,m=𝝆t⊤​𝜿m,m∈𝒱t\.\\boldsymbol\{\\rho\}\_\{t\}=\\mathbf\{W\}\_\{R\}\\mathbf\{x\}\_\{t\},\\qquad a\_\{t,m\}=\\boldsymbol\{\\rho\}\_\{t\}^\{\\top\}\\boldsymbol\{\\kappa\}\_\{m\},\\quad m\\in\\mathcal\{V\}\_\{t\}\.\(12\)To allow the model to bypass historical memory, we augment the visible anchor set with a null option∅\\varnothing, whose payload is fixed to zero,𝐀\(∅\)=𝟎\\mathbf\{A\}^\{\(\\varnothing\)\}=\\mathbf\{0\}\. Its query\-dependent logit isnt=𝐰∅⊤​𝐱t\+b∅n\_\{t\}=\\mathbf\{w\}\_\{\\varnothing\}^\{\\top\}\\mathbf\{x\}\_\{t\}\+b\_\{\\varnothing\}\. Let𝒱~t=𝒱t∪\{∅\}\\widetilde\{\\mathcal\{V\}\}\_\{t\}=\\mathcal\{V\}\_\{t\}\\cup\\\{\\varnothing\\\}denote the augmented candidate set\. We define the logit of each candidatej∈𝒱~tj\\in\\widetilde\{\\mathcal\{V\}\}\_\{t\}as

st,j=\{at,j,j∈𝒱t,nt,j=∅,πt,j=exp⁡\(st,j\)∑r∈𝒱~texp⁡\(st,r\)\.s\_\{t,j\}=\\begin\{cases\}a\_\{t,j\},&j\\in\\mathcal\{V\}\_\{t\},\\\\ n\_\{t\},&j=\\varnothing,\\end\{cases\}\\qquad\\pi\_\{t,j\}=\\frac\{\\exp\(s\_\{t,j\}\)\}\{\\displaystyle\\sum\_\{r\\in\\widetilde\{\\mathcal\{V\}\}\_\{t\}\}\\exp\(s\_\{t,r\}\)\}\.\(13\)Since the selected routing probabilities directly weight the historical state readouts, their scores remain jointly optimized by the language\-modeling objective\. The routing query𝝆t\\boldsymbol\{\\rho\}\_\{t\}determines which anchors to retrieve, whereas the state\-read query𝐪t\\mathbf\{q\}\_\{t\}reads their matrix\-valued contents\. If no anchor is visible, the null option receives all probability mass\.

We note that the aggregation formulation in Equation \([13](https://arxiv.org/html/2608.12435#S3.E13)\) readily admits a sparse variant by restricting aggregation to theKKhighest\-scoring visible anchors \(Top\-KK\)\. This sparse approach exhibits natural connections with the hierarchical sparse attention approaches\[lu2025moba,HiLS\], while preserving dense token\-level processing rather than relying on hard token\-level pruning\. Our ablation studies in Section[5](https://arxiv.org/html/2608.12435#S5)show that sparse routing substantially reduces aggregation cost with minimal performance degradation\.

#### Historical retrieval and residual fusion\.

Given the routing probabilities, the causally visible state anchors are aggregated into a query\-dependent historical state, which is read using the same state\-read query as the current state\. The resulting historical readout is then added to the current\-state readout:

𝐨t=𝐒t​𝐪t\+∑j∈𝒱~tπt,j​𝐀\(j\)​𝐪t\.\\mathbf\{o\}\_\{t\}=\\mathbf\{S\}\_\{t\}\\mathbf\{q\}\_\{t\}\+\\sum\_\{j\\in\\widetilde\{\\mathcal\{V\}\}\_\{t\}\}\\pi\_\{t,j\}\\mathbf\{A\}^\{\(j\)\}\\mathbf\{q\}\_\{t\}\.\(14\)This additive formulation preserves the original recurrent path and introduces historical retrieval as an auxiliary residual branch, without modifying the underlying recurrent update\. Since the routing probabilities directly affect the layer output, the routing queries and anchor\-derived keys are optimized end\-to\-end with the language\-modeling objective\.

### 3\.3Implementation

We implement MARCH as a two\-stage producer–reader computation\. Following the hardware\-efficient chunkwise formulation of Gated DeltaNet\[yang2025gateddelta\], the producer processes recurrent updates in blocks amenable to tensor\-core acceleration, computes each token’s current\-state output, and checkpoints the recurrent state at each anchor boundary\. The resulting state anchors are consumed by the historical reader\. Inspired by the I/O\-aware principles of FlashAttention\[dao2022flashattention\], the reader jointly tiles query tokens and state anchors, reuses each anchor tile across a block of queries, and fuses routing\-score computation, online softmax updates, and the accumulation of weighted state readouts into a streaming reduction\. This fused schedule avoids materializing either the dense token\-to\-anchor routing matrix or the substantially larger tensor of per\-anchor candidate readouts, thereby reducing intermediate storage and the associated HBM traffic\. As shown in[Figure˜4](https://arxiv.org/html/2608.12435#S4.F4), despite the cost of historical retrieval, our fused dense implementation exceeds FlashAttention\-2 in throughput at 64K and above and incurs lower core runtime from 32K onward\.

## 4Experiments

MARCH is designed to extend the long\-range memory of recurrent models while preserving their general language capabilities\. In this paper, we verify the effectiveness of MARCH by training from scratch, and evaluate across a diverse suite of benchmarks spanning zero\-shot commonsense reasoning, long\-context understanding, and in\-context retrieval\. Across these tasks, MARCH consistently outperforms existing recurrent baselines, with particularly strong gains on retrieval\-intensive and long\-context benchmarks\.

### 4\.1Experimental Setup

#### Training configuration\.

Following the academic\-scale protocol used by Log\-Linear Attention\[guo2026loglinear\], we pretrain the models from scratch on 50B tokens from the Long\-Data\-Collections dataset, using a sequence length of 16K\. The main configurations use 21 layers and a hidden size of 1536\. The Transformer \(693M\) uses 16 attention heads and a RoPE base of 500K, while Gated DeltaNet \(793M\) and its variants use six value heads\. To control for parameter count in addition to model depth, we also include a2424\-layer Transformer with \(778M\) parameters, closely matching the size of the Gated DeltaNet\. For MARCH, we set the routing dimension todr=64d\_\{r\}=64and use a periodic anchoring interval ofC=512C=512text tokens\. We train all models with a global batch size of approximately4\.24\.2M tokens using the fused AdamW optimizer, withβ1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95,ϵ=10−8\\epsilon=10^\{\-8\}, and a weight decay of0\.10\.1\. The peak learning rate is set to4×10−44\\times 10^\{\-4\}with a warmup\-stable\-decay schedule\. All models use the same training data, token budget, context length, and optimization configuration\.

#### Baselines\.

Our primary comparisons are against standard GDN\[yang2025gateddelta\]and GDN augmented with Log\-Linear Attention\[guo2026loglinear\]\. MARCH and these two baselines use matched architectural configurations and the same pretraining setup, enabling a controlled comparison of their memory mechanisms\. To contextualize their performance against full attention, we additionally include two Transformer baselines: a2121\-layer model matched in depth to the recurrent models and a2424\-layer model approximately matched to them in parameter count\.

#### Evaluation tasks\.

For short\-context generalization, we use eight zero\-shot commonsense benchmarks: LAMBADA\[paperno\_lambada\_2016\], PIQA\[bisk\_piqa\_2020\], HellaSwag\[zellers\_hellaswag\_2019\], WinoGrande\[sakaguchi\_winogrande\_2021\], ARC\-Easy and ARC\-Challenge\[clark\_think\_2018\], OpenBookQA\[OpenBookQA2018\], and CommonsenseQA\[talmor2019commonsenseqaquestionansweringchallenge\]\. We additionally evaluate long\-context understanding on LongBench\[bai2024longbench\], covering single\-document QA, multi\-document QA, summarization, and few\-shot learning\. The long\-context retrieval evaluation covers six single\-neddle and multi\-needle tasks from RULER\[hsieh2024ruler\]at 4K, 8K, and 16K context lengths\. Finally, the in\-context retrieval suite contains SQuAD\[rajpurkar\_know\_2018\], TriviaQA\[JoshiTriviaQA2017\], SWDE\[lockard\_openceres\_2019\], FDA\[arora\_language\_2023\], Natural Questions\[kwiatkowski\-etal\-2019\-natural\], and DROP\[dua2019drop\]\. We follow the evaluation protocol of prior work\[wang2026dynamic\]and use the LM\-Evaluation\-Harness\[eval\-harness\]\.

Table 1:Zero\-shot performance of MARCH and baseline models on eight commonsense reasoning benchmarks\. Results are reported using accuracy \(acc\) or normalized accuracy \(acc\_n\), as indicated in the column headers; higher is better \(↑\\uparrow\)\. The best result among Gated DeltaNet variants in each column is highlighted in bold\.

### 4\.2Main Results

#### Commonsense reasoning\.

As shown in[Table˜1](https://arxiv.org/html/2608.12435#S4.T1), MARCH consistently outperforms both the vanilla and Log\-Linear variants of Gated DeltaNet across all eight zero\-shot commonsense reasoning benchmarks\. It improves the average accuracy from40\.140\.1and40\.040\.0to41\.541\.5, respectively, with the largest gain over the vanilla baseline observed on OpenBookQA \(\+2\.8\+2\.8points\), aligning with our findings in retrieval tasks presented below\. Moreover, MARCH achieves a higher average score than both Transformer baselines, surpassing the standard Transformer on six of eight tasks and the 24\-layer Transformer on four\. These results indicate that MARCH consistently strengthens the Gated DeltaNet backbone while remaining competitive with comparable full\-attention models on short\-context language understanding tasks\.

![Refer to caption](https://arxiv.org/html/2608.12435v1/x3.png)Figure 3:NIAH performance on three single\-needle and three multi\-needle tasks\. The Transformer achieves perfect accuracy on both S\-NIAH\-1 and S\-NIAH\-2 at context lengths of 4K, 8K, and 16K\.
#### Needle\-in\-a\-haystack retrieval\.

We evaluate long\-context associative retrieval using the needle\-in\-a\-haystack \(NIAH\) suite from RULER\[hsieh2024ruler\], where a model must recover values associated with keys embedded among irrelevant context\. All models are trained with a maximum context length of 16K;[Figure˜3](https://arxiv.org/html/2608.12435#S4.F3)reports results from 4K to 32K, making 32K a zero\-shot length\-extrapolation setting\. Across the 24 task–length combinations, MARCH outperforms the stronger recurrent baseline in 19 settings and matches it in the remaining five\. On the multi\-needle tasks, it wins in 11 of 12 settings\. At 32K, MARCH achieves the best result on all six tasks, retaining perfect accuracy on S\-NIAH\-1 and nonzero accuracy on the remaining tasks, whereas both Transformer variants and Log\-Linear Gated\-DeltaNet score zero throughout\. This contrast is consistent with RoPE extrapolation in the Transformers and the state\-index\-dependent coefficients of Log\-Linear Gated\-DeltaNet\. MARCH instead shares the same content\-based router across all anchors, allowing longer contexts to introduce additional anchors without requiring new anchor\-specific routing parameters\.

Table 2:Results on twelve LongBench tasks\. The best result among Gated DeltaNet and its variants is shown in bold for each task\. The relative average gain over the Gated DeltaNet is shown in green parentheses\.Single\-Doc QAMulti\-Doc QASummarizationFew\-shot LearningModelNQAQQAMFQHQA2WMMusGvRQMSMNsTRCTQASSMAvg\.↑\\uparrowTransformer4\.44\.115\.97\.59\.94\.110\.711\.614\.521\.033\.928\.313\.8w/*24 Layers*3\.511\.118\.37\.89\.84\.111\.312\.912\.822\.547\.523\.215\.4Gated DeltaNet3\.04\.813\.35\.68\.72\.12\.611\.313\.018\.038\.621\.911\.9w/*Log\-Linear*3\.66\.213\.67\.18\.23\.36\.313\.213\.517\.032\.825\.112\.5w/ MARCH4\.27\.814\.67\.411\.54\.88\.217\.414\.119\.043\.126\.314\.9 \(↑25%\\uparrow 25\\%\)
#### Long\-context understanding\.

[Table˜2](https://arxiv.org/html/2608.12435#S4.T2)reports results across four LongBench task categories\. MARCH consistently outperforms both vanilla Gated DeltaNet and its log\-linear variant on all twelve tasks\. The improvements are particularly pronounced on multi\-document QA: relative to the stronger of the vanilla and log\-linear Gated DeltaNet baselines, MARCH raises the 2WikiMultihopQA score from 8\.7 to 11\.5 and the MuSiQue score from 3\.3 to 4\.8, corresponding to relative gains of 32% and 45%, respectively\. The benefits also extend to summarization, where the QMSum score increases from 13\.2 to 17\.4 \(32%\), and to all three few\-shot learning tasks\. These results show that content\-routed state anchors improve long\-context understanding across diverse task formats, rather than benefiting only retrieval\-oriented question answering\.

Table 3:In\-context retrieval accuracy \(↑\\uparrow\)\. The best result among Gated DeltaNet and its variants is marked in bold for each benchmark, with the relative gain over the stronger of the two Gated DeltaNet baselines shown in green parentheses\.
#### In\-Context Retrieval\.

Following\[arora2024simple\], we evaluate in\-context retrieval on six real\-world, recall\-intensive benchmarks\. As shown in[Table˜3](https://arxiv.org/html/2608.12435#S4.T3), MARCH consistently outperforms both vanilla Gated DeltaNet and its Log\-Linear variant across all tasks\. Relative to the stronger of the vanilla and log\-linear Gated DeltaNet baselines on each benchmark, MARCH yields relative improvements ranging from8%8\\%on SQuAD to23%23\\%on TriviaQA and raises the average accuracy from20\.520\.5to23\.323\.3, corresponding to a14%14\\%relative improvement\. These consistent gains across heterogeneous retrieval tasks demonstrate that MARCH improves the retrieval capability of the Gated DeltaNet backbone beyond a particular dataset or input format\. Together, these results establish content\-routed state anchors as an effective mechanism for strengthening fine\-grained retrieval in recurrent models\.

![[Uncaptioned image]](https://arxiv.org/html/2608.12435v1/x4.png)

Figure 4:Training efficiency across sequence lengths\. Left: end\-to\-end training throughput in tokens per second \(higher is better\)\. Right: forward–backward runtime of the core sequence\-mixing operation in milliseconds \(lower is better\)\. MARCH \(Top\-44\) retains only the four highest\-scoring state anchors for each token and head during historical retrieval\.

#### Training efficiency\.

[Figure˜4](https://arxiv.org/html/2608.12435#S4.F4)compares the end\-to\-end throughput and core forward–backward runtime of FlashAttention\-2, Gated DeltaNet, dense MARCH, and its Top\-44implementation\. Sparse routing becomes increasingly beneficial as the context grows\. At 128K tokens, Top\-44MARCH more than doubles the training throughput of dense MARCH and reduces its core runtime by roughly an order of magnitude\. It also achieves higher throughput than FlashAttention\-2 at this length, although vanilla Gated DeltaNet remains faster because it incurs no historical\-retrieval overhead\.

## 5Ablation Studies

#### Effect of chunk size\.

The training chunk sizeCCdetermines how frequently MARCH checkpoints the recurrent state\. Smaller chunks create denser candidate anchors and offer finer temporal resolution, at the cost of a larger anchor cache and higher historical\-routing overhead\. We varyCCfrom 256 to 2048 while keeping all other model and training settings fixed\.[Table˜4](https://arxiv.org/html/2608.12435#S5.T4)reports performance on three in\-context retrieval benchmarks and the average over the six NIAH tasks at context lengths of 4K, 8K, and 16K\. As an additional inference test, we organize the MARCH state bank according to the Fenwick tree scheme used by Log\-Linear Attention\[guo2026loglinear,Fenwick1994AND\]\. This scheme arranges state anchors hierarchically and retains𝒪​\(log⁡T\)\\mathcal\{O\}\(\\log T\)anchors as the context grows\. We report performance close to\[guo2026loglinear\]\. This demonstrates that the learned router in MARCH shows great generalizability and flexibility across various state bank organization schemes\.

Table 4:Effect of chunk size on in\-context retrieval and NIAH\. Panel \(a\) evaluates checkpoints using the same chunk size during training and inference\. Panel \(b\) varies the inference\-time chunk size for the checkpoint trained with chunk size 512\. NIAH scores are averaged over six tasks at each context length\. Bold indicates the selected setting in each panel and the best result in each column\.
In Panel \(a\),C=512C=512provides the best overall balance between retrieval quality and anchor count\. Smaller chunks improve some long\-context results but incur higher memory and routing costs, whereas larger chunks generally degrade retrieval because the resulting checkpoints are too sparse\. We therefore adoptC=512C=512as the default\. Panel \(b\) shows that changing the chunk size at inference provides a flexible accuracy–memory trade\-off: denser anchors generally improve retrieval at higher cost, while overly sparse anchors lead to substantial degradation\. The Fenwick tree row additionally evaluates hierarchical organization of the state bank at inference\.

#### Routing design\.

We ablate the router’s query–key dimensiondrd\_\{r\}, routing sparsity, and learned null option\.[Table˜5](https://arxiv.org/html/2608.12435#S5.T5)reports aggregate results across general language understanding, long\-context benchmarks, and NIAH\. The default uses dense routing withdr=64d\_\{r\}=64and includes the null option\.

Table 5:Routing\-design ablations\. The first row is the default; each subsequent row changes one component\. Commonsense \(CS\), LongBench, and Retrieval are macro\-averages over 8, 12, and 6 benchmarks, respectively\. NIAH scores are averaged over six tasks, and Avg\. over the three context lengths\. Column\-wise best results are bold\.
Increasingdrd\_\{r\}to 192 yields higher retrieval capability but reduces general performance across other tasks, makingdr=64d\_\{r\}=64a more balanced choice overall\. Top\-44nearly matches dense routing on commonsense and retrieval but trails on NIAH, making it an efficiency\-oriented operating point when considered alongside[Figure˜4](https://arxiv.org/html/2608.12435#S4.F4)\. Removing the null option degrades every aggregate, confirming the benefit of bypassing irrelevant historical states\.

## 6Related Work

#### Efficient Attention Mechanisms\.

Efficient attention reduces the quadratic cost of full self\-attention through local windows, kernelization, or systems optimization\. Local sliding\-window attention limits each query to a bounded neighborhood\[wang2025rattentionminimalslidingwindow,cabannes2025shortwindowattentionenables\]\. Performer, Nystromformer, and Linear Attention replace the softmax kernel with feature maps and exploit associativity for linear\-time computation\[performer,xiong2021nystromformer,katharopoulos2020transformers\]\. FlashAttention\-2, sequence parallelism, and chunkwise algorithms instead improve hardware efficiency without changing the dense attention pattern\[flashattention2,Sun2024LinearAS\]\.

#### Sparse Attention\.

Sparse attention retains content\-based softmax retrieval but restricts each query to a small subset of token\-level key–value pairs\. Early methods rely on predefined connectivity: Sparse Transformer factorizes the attention pattern, while Longformer and BigBird combine local windows with global or random links\[child2019generating,beltagy2020longformer,zaheer2020big\]\. Later methods make the sparse pattern input dependent: Routing Transformer clusters tokens by content, H2O evicts low\-utility cache entries, and Quest selects KV\-cache pages conditioned on the current query\[roy2021efficient,zhang2023h2o,tang2024quest\]\. More recent trainable designs route queries to relevant blocks, as in MoBA, or combine compressed, selectively retrieved, and local branches with hardware\-aligned kernels, as in Native Sparse Attention\[lu2025moba,yuan2025native\]\. These approaches reduce attention\-score computation or memory traffic, but their accuracy hinges on token or block selection and they still store or manipulate token\-level KV memories\. In contrast, MARCH routes over compact keys associated with historical recurrent\-state snapshots, retrieving compressed prefix states rather than sparsifying token\-to\-token attention\.

#### State Space Models and Gated Linear Recurrences\.

State space models \(SSMs\) and linear recurrent networks compress the prefix into a recurrent state\. Linear Attention and its kernelized variants share this view through decayed outer\-product updates and query\-based reads\[katharopoulos2020transformers,chou2024metala,dao2024transformers\]\. S4 uses structured linear dynamics, while Mamba and Mamba\-2 use selective transitions; RetNet, RWKV, HGRN, and LRU combine associative memories with structured recurrences\[gu2021efficiently,gu2023mamba,dao2024transformers,sun2023retentive,peng2023rwkv,qin2024hgrn2,orvieto2023resurrecting,longhorn\]\. GLA introduces input\-dependent decay, DeltaNet and GDN use delta\-rule corrections, and GDN\-2 decouples erase and write through channel\-wise gates\[yang2025gateddelta,yang2024parallelizing,hatamizadeh2026gateddeltanet2,siems2025deltaproduct,Grazzi2024UnlockingSI\]\. Despite these advances, most models retain one fixed\-capacity state, whose dimension directly controls update and read cost; this bottleneck contributes to the retrieval gap with Transformers\[arora2024simple,wen\_rnns\_2024\]\. We preserve efficient recurrence while expanding memory into selectively accessed states\.

#### State Expansion and Associative Memory\.

Long\-context studies identify recurrent state capacity as a central limitation of Linear Attention models\[arora2024zoology,arora2024simple\]\. Multi\-State RNNs, HGRN2, and Log\-Linear Attention expand or hierarchically organize recurrent states\[Oren2024TransformersAM,qin2024hgrn2,guo2026loglinear\]\. Mixture\-of\-Memories, Sparse State Expansion, Product Key Memory, and Fast\-weight Product Key Memory use memory experts or sparse banks, while Sparse Delta Memory sparsifies GDN reads and writes\[du2025mom,pan25SSE,lample2019largememorylayersproduct,berges2024memorylayersscale,zhao2026fastweightproductkeymemory,cabannes26SDM,afzal2026raven\]\. Context\-compression methods instead retrieve at the token or chunk level, using learned summary tokens or selective chunk reopening\[chevalier2023adapting,mu2023learning,zhang2024long,deng2025unigist,petrov2025long,mao2026gisttokens\]\. Our method treats recurrent states as retrieval units, expands total capacity, and reads only selected states, decoupling capacity from dense per\-token updates\.

## 7Limitations and Future Work

MARCH adopts periodic checkpointing and organizes all historical states in a single homogeneous anchor bank\. Although this design is simple and efficient, fixed\-interval anchoring does not account for the non\-uniform evolution of recurrent memory: it may create redundant anchors in stable regions while providing insufficient resolution when the state changes rapidly\. A natural extension is to develop adaptive anchoring mechanisms according to state novelty or update magnitude and to consolidate or evict redundant anchors given a memory budget\. More broadly, MARCH improves access to earlier states but does not explicitly increase or specialize the capacity of the underlying memory\. Future work could combine state anchoring with larger\-capacity memory and multiple memory partitions specialized for different temporal scales or information types\. For example, short\-term context, salient episodic events, and slowly consolidated knowledge could be maintained through distinct write, retention, and forgetting mechanisms, while a hierarchical router determines both which memory partition and which stored state should serve each query\. In addition, MARCH has the potential to support external memory modules for optimized performance over specific downstream tasks, knowledge consolidation from experience to parametric information, and other memory manipulation mechanisms, opening up new scaling directions for test\-time training and continual learning\.

## 8Conclusion

We introduce MARCH, a novel attention architecture which augments recurrent models with content\-routed state anchors\. By preserving cumulative state checkpoints, MARCH enables selective access to earlier recurrent states without modifying the underlying recurrence\. It consistently outperforms strong recurrent baselines across commonsense reasoning, LongBench, in\-context retrieval, and NIAH\. Ablations further show a controllable retrieval–efficiency trade\-off through checkpoint density and sparse routing\. These results establish historical\-state retrieval as a practical approach to scaling recurrent memory beyond a single evolving state\.

## References