Maglev: Sliding Recurrent Memory

# Maglev: Sliding Recurrent Memory
Source: [https://arxiv.org/html/2608.02870](https://arxiv.org/html/2608.02870)
###### Abstract

We introduce Maglev, a recurrent Transformer architecture with fixed\-size memory that generalizes sliding\-window attention while remaining parallelizable during training\. Maglev consists of two coupled models: a prefillerQQ, which leverages full attention111In practice, we use interleaved full and sliding\-window attention forQQ, as this yields stronger performance\. The essential requirement is thatQQbe more expressive thanPP, with access to the full history\.to produce memory targetsmt′m^\{\\prime\}\_\{t\}, and a decoderPP, which uses only sliding\-window attention and recurrent K/V injection to produce decoder memoriesmtm\_\{t\}for next\-token prediction\. We train Maglev with a memory consistency loss that alignsmtm\_\{t\}withmt′m^\{\\prime\}\_\{t\}, allowing inference to usePPalone\. Empirically, Maglev improves validation loss and downstream pretraining benchmarks over sliding\-window and latent recurrent transformer baselines\. Moreover, sharing parameters betweenPPandQQreduces parameter memory while preserving most of the gains\.![Refer to caption](https://arxiv.org/html/2608.02870v1/figures/maglev.png)Figure 1:Maglev: a prefillerQQproduces memory targetsmt′m^\{\\prime\}\_\{t\}from the observed sequence, and a decoderPPconsumes the shiftedmt−1′m^\{\\prime\}\_\{t\-1\}while predictingxt\+1x\_\{t\+1\}and producing its own memorymtm\_\{t\}\. The consistency loss alignsmtm\_\{t\}withmt′m^\{\\prime\}\_\{t\}\. At inference, the prefiller is discarded and the decoder runs recurrently using its own memories\. In practice, we find that we could largely share parameters betweenPPandQQwithout degrading much performance\.

## 1Introduction

Transformers have a simple memory strategy: keep every token\. Full causal self\-attention lets each prediction revisit the entire prefix, enabling powerful nonlinear retrieval, copying, and in\-context computation\(Vaswaniet al\.,[2017](https://arxiv.org/html/2608.02870#bib.bib1)\); its cost, however, grows with context\. Sliding\-window attention bounds both attention and KV\-cache costs, but discards distant information and therefore provides no persistent long\-term memory\(Beltagyet al\.,[2020](https://arxiv.org/html/2608.02870#bib.bib3); Jianget al\.,[2023](https://arxiv.org/html/2608.02870#bib.bib4)\)\. Classical recurrent models such as LSTMs compress history with nonlinear token\-wise updates, but propagate a relatively small state sequentially, limiting both capacity and sequence parallelism\(Hochreiter and Schmidhuber,[1997](https://arxiv.org/html/2608.02870#bib.bib2)\)\. Memory Transformers offer richer states, yet commonly update them only at segment or block boundaries, imposing an artificial granularity on memory\(Daiet al\.,[2019](https://arxiv.org/html/2608.02870#bib.bib28); Raeet al\.,[2020](https://arxiv.org/html/2608.02870#bib.bib20); Hutchinset al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib31); Bulatovet al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib30)\)\. Linear attention and state\-space models remove these boundaries and parallelize token\-wise recurrence, but do so through structured linear or affine updates\(Katharopouloset al\.,[2020](https://arxiv.org/html/2608.02870#bib.bib5); Guet al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib6); Gu and Dao,[2023](https://arxiv.org/html/2608.02870#bib.bib7); Dao and Gu,[2024](https://arxiv.org/html/2608.02870#bib.bib8); Liuet al\.,[2024a](https://arxiv.org/html/2608.02870#bib.bib26); Yanget al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib13)\)\. The missing piece is a practical way to give a nonlinear Transformer persistent memory that it can rewrite after every token\.

#### Goal: token\-wise recurrent memory with parallel pretraining\.

We study language models with*token\-wise latent memory*\. After processingxtx\_\{t\}, the model writes a latent memory vectormtm\_\{t\}and retains the bounded window

ℳt=\{mt−W\+1,…,mt\},\\mathcal\{M\}\_\{t\}=\\\{m\_\{t\-W\+1\},\\ldots,m\_\{t\}\\\},\(1\)ignoring boundary effects\. Writing all trainable parameters asΘ\\Theta, the desired recurrent interface predicts the next token and writes the current memory from only a recent token window and the preceding memory window,

pΘ​\(xt\+1,mt∣xt−W\+1:t,ℳt−1\)\.p\_\{\\Theta\}\(x\_\{t\+1\},m\_\{t\}\\mid x\_\{t\-W\+1:t\},\\mathcal\{M\}\_\{t\-1\}\)\.\(2\)A practical token\-wise memory model should combine three properties:

- •Token\-wise nonlinear memory\.Unlike the structured, layer\-local linear or affine updates of linear attention and state\-space models\(Katharopouloset al\.,[2020](https://arxiv.org/html/2608.02870#bib.bib5); Guet al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib6); Gu and Dao,[2023](https://arxiv.org/html/2608.02870#bib.bib7); Dao and Gu,[2024](https://arxiv.org/html/2608.02870#bib.bib8)\), each memory update traverses the full nonlinear Transformer depth, providing looped\-Transformer\-like recurrent depth without extra Transformer iterations\(Dehghaniet al\.,[2019](https://arxiv.org/html/2608.02870#bib.bib27); Giannouet al\.,[2023](https://arxiv.org/html/2608.02870#bib.bib32)\)\. Because every token’s decoder state serves as memory through its ordinary K/V entry, the recurrence also requires no dedicated memory tokens or cache and introduces no artificial segment or block boundaries\(Daiet al\.,[2019](https://arxiv.org/html/2608.02870#bib.bib28); Raeet al\.,[2020](https://arxiv.org/html/2608.02870#bib.bib20); Hutchinset al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib31); Bulatovet al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib30)\)\.
- •Bounded recurrent inference\.The token and memory windows remain fixed as the sequence grows\. As in sliding\-window attention\(Beltagyet al\.,[2020](https://arxiv.org/html/2608.02870#bib.bib3); Jianget al\.,[2023](https://arxiv.org/html/2608.02870#bib.bib4)\), inference uses a bounded local KV cache; the recurrent memories carry information beyond that window without increasing its size\.
- •Parallel large\-scale pretraining\.Although inference is recurrent, ideally pretraining should process all positions in parallel, preserving standard Transformer throughput\(Vaswaniet al\.,[2017](https://arxiv.org/html/2608.02870#bib.bib1)\)\. Achieving this parallelism is nontrivial for nonlinear recurrence\(Limet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib25); Gonzalezet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib24)\)\.

The interface provides the first two properties by construction: it writesmtm\_\{t\}at every position while keeping both conditioning windows fixed\. Parallel pretraining, however, is not automatic\. Text providesx1:Tx\_\{1:T\}but not the latent trajectorym1:Tm\_\{1:T\}; because each update consumes preceding memories, direct training must generate this trajectory sequentially\. The central question is how to preserve the recurrent interface at deployment without unrolling it during pretraining\.

#### Maglev: parallel supervision for recurrent memory\.

Maglev breaks this dependency by separating memory construction during pretraining from memory propagation at inference\. A stronger causal prefillerQQproduces target memoriesm1:T′m^\{\\prime\}\_\{1:T\}from the observed sequence in parallel\. A sliding\-window decoderPPconsumes the shifted targetsm0:T−1′m^\{\\prime\}\_\{0:T\-1\}, predicts the next tokens, and emits memoriesm1:Tm\_\{1:T\}\. A consistency loss aligns eachmtm\_\{t\}withmt′m^\{\\prime\}\_\{t\}, teachingPPto produce the memory needed at the next step\. Training therefore requires two sequence\-parallel passes\. At inference,QQis removed andPPcloses the loop with its own memories\.

FigureLABEL:fig:headerillustrates the architecture\. The deployed decoder remains close to an ordinary sliding\-window Transformer, augmenting the same local attention pattern with a gated K/V pathway for shifted recurrent memory\. The decoder’s final normalized state is both the memorymtm\_\{t\}and the input to the language\-modeling head\. Fixed token and memory windows keep inference costs bounded\.

#### Contributions\.

We propose a token\-wise nonlinear recurrent\-memory Transformer that folds shifted memories into sliding\-window K/V entries, preserving bounded inference without extra memory tokens\. We introduce lifted parallel training, where a causal prefillerQQgenerates targets in parallel and consistency training teaches decoderPPto reproduce them without sequential recurrent unrolling\. With a435435M\-scale deployed decoder trained on43\.5243\.52B tokens, our best model improves FineWeb\-Edu validation BPB from0\.74130\.7413to0\.72510\.7251and average downstream accuracy from54\.154\.1to56\.456\.4over a matched sliding\-window Transformer, while also outperforming matched latent recurrent Transformer\. Moreover,QQandPPcan share most parameters while preserving most gains\.

## 2Method

Maglev replaces sequential latent\-state training with two sequence\-parallel Transformer passes\. As summarized in FigureLABEL:fig:header, a causal prefillerQQconstructs an auxiliary memory trajectory, and a deployable decoderPPlearns to use and reproduce it one step at a time\. At inference,QQis discarded andPPfeeds back its own memories, recovering token\-wise recurrence\. We first formalize this construction and then describe our Transformer instantiation\.

### 2\.1Lifted Parallel Training

Direct recurrent training would generatem1,m2,…,mTm\_\{1\},m\_\{2\},\\ldots,m\_\{T\}in order\.*Lifted parallel training*instead constructs an explicit auxiliary trajectory from the observed sequence in one causal, sequence\-parallel pass\. The prefillerQQmay have a larger receptive field than the decoder, but remains causal so that its memories contain no future information\.

Letx1:Tx\_\{1:T\}be a token sequence andddthe hidden dimension\. Letϕ\\phiandθ\\thetadenote the parameters ofQQandPP, respectively, and letΘ=ϕ∪θ\\Theta=\\phi\\cup\\thetadenote all unique trainable parameters\. The setsϕ\\phiandθ\\thetamay overlap\. Training consists of

𝒎1:T′=Qϕ​\(x1:T\)⏟prefiller pass,𝒎1:T=Pθ​\(x1:T,𝒎0:T−1′\)⏟decoder pass,\\underbrace\{\{\\color\[rgb\]\{0,0\.17578125,0\.4453125\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.17578125,0\.4453125\}\\boldsymbol\{m\}^\{\\prime\}\_\{1:T\}\}=Q\_\{\\phi\}\(x\_\{1:T\}\)\}\_\{\\text\{prefiller pass\}\},\\qquad\\underbrace\{\{\\color\[rgb\]\{0,0\.17578125,0\.4453125\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.17578125,0\.4453125\}\\boldsymbol\{m\}\_\{1:T\}\}=P\_\{\\theta\}\(x\_\{1:T\},\{\\color\[rgb\]\{0,0\.17578125,0\.4453125\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.17578125,0\.4453125\}\\boldsymbol\{m\}^\{\\prime\}\_\{0:T\-1\}\}\)\}\_\{\\text\{decoder pass\}\},\(3\)wherem0′=0m^\{\\prime\}\_\{0\}=0\. Heremt′m^\{\\prime\}\_\{t\}andmtm\_\{t\}are the prefiller and decoder outputs after all Transformer blocks and the final RMS normalization\. At positiontt, the shifted input givesPPa memory window ending atmt−1′m^\{\\prime\}\_\{t\-1\}, from which it producesmtm\_\{t\}\. BecauseQQis causal, this window depends only onx≤t−1x\_\{\\leq t\-1\}and matches the information available at inference\. Despite the sequence notation in Equation[3](https://arxiv.org/html/2608.02870#S2.E3), causal sliding\-window attention restricts each position to its local token context and corresponding shifted\-memory window\.

During training,PPreceives the shifted prefiller trajectorym0:T−1′m^\{\\prime\}\_\{0:T\-1\}; during inference, it receives its own preceding memoriesm0:T−1m\_\{0:T\-1\}through the same channel\. BecauseQQconstructs the complete target trajectory in one causal parallel pass, all shifted targets are available before the decoder pass, allowingPPto evaluate every position simultaneously\. Maglev thus replaces a length\-TTunroll with two sequence\-parallel training passes while retaining recurrent inference\.

Only the decoder predicts tokens\. Its final memorymtm\_\{t\}is also the state read by the language\-modeling head:

ot=U​mt,pΘ​\(xt\+1∣x≤t\)=softmax⁡\(15​tanh⁡\(ot/15\)\)\.o\_\{t\}=Um\_\{t\},\\qquad p\_\{\\Theta\}\(x\_\{t\+1\}\\mid x\_\{\\leq t\}\)=\\operatorname\{softmax\}\(15\\tanh\(o\_\{t\}/15\)\)\.\(4\)For valid target positionsℐ\\mathcal\{I\}, letyt=xt\+1y\_\{t\}=x\_\{t\+1\}\. We optimize

ℒ​\(Θ\)=1\|ℐ\|​∑t∈ℐ\[CE\(pΘ\(⋅∣x≤t\),yt\)⏟ℒCE\+λ​∥mt−mt′∥2d⏟ℒcons\]\.\\mathcal\{L\}\(\\Theta\)=\\frac\{1\}\{\|\\mathcal\{I\}\|\}\\sum\_\{t\\in\\mathcal\{I\}\}\\bigg\[\\underbrace\{\\operatorname\{CE\}\\bigl\(p\_\{\\Theta\}\(\\cdot\\mid x\_\{\\leq t\}\),y\_\{t\}\\bigr\)\}\_\{\\mathcal\{L\}\_\{\\rm CE\}\}\+\\lambda\\underbrace\{\\frac\{\\lVert m\_\{t\}\-m^\{\\prime\}\_\{t\}\\rVert\_\{2\}\}\{\\sqrt\{d\}\}\}\_\{\\mathcal\{L\}\_\{\\rm cons\}\}\\bigg\]\.\(5\)The first term trains next\-token prediction from decoder memorymtm\_\{t\}\. The second alignsmtm\_\{t\}with prefiller targetmt′m^\{\\prime\}\_\{t\}\. From a shifted prefiller\-memory window,PPlearns both to predict the next token and to produce the memory for the next window\. This consistency enables recurrent inference using decoder memories in place of prefiller memories\.

### 2\.2Transformer Instantiation

#### Prefiller, decoder, and parameter sharing\.

The prefiller is a training mechanism rather than a prescribed architecture: any causal model that produces allmt′m^\{\\prime\}\_\{t\}in parallel can serve asQQ\. OurQQandPPare Transformers following the nanochat layer\-pattern convention\(Karpathy,[2025](https://arxiv.org/html/2608.02870#bib.bib34)\), in which a pattern lists successive attention types and repeats through the stack\. For example,SLSLalternates sliding\-window \(S\) and full causal \(L\) attention\. The prefiller usesSLSL, with a512512\-token sliding window and full attention over the20482048\-token training context\. The decoder usesSSSS, restricting every layer toW=512W=512\. Thus,QQhas more context for constructing targets, whilePPretains the bounded receptive field required for deployment\.

By default,QQandPPshare Transformer blocks, with separate residual\-scaling parameters for the decoder path\. We also evaluate a variant with a separateQQstack\. Both use the same two\-pass objective in Equations[3](https://arxiv.org/html/2608.02870#S2.E3)–[5](https://arxiv.org/html/2608.02870#S2.E5); only the block sharing differs\.

#### Recurrent K/V injection\.

Following the latent recurrent Transformer\(Huanget al\.,[2026](https://arxiv.org/html/2608.02870#bib.bib69)\), the decoder injects shifted memory through K/V features rather than dedicated memory tokens\. At decoder layerℓ\\ell, letatℓa\_\{t\}^\{\\ell\}be the incoming residual stream andqtℓ,ktℓ,vtℓq\_\{t\}^\{\\ell\},k\_\{t\}^\{\\ell\},v\_\{t\}^\{\\ell\}the local query, key, and value\. During training, shared projections map the shifted prefiller memory to recurrent K/V features:

ktrec=RMSNorm⁡\(Wkrec​mt−1′\),vtrec=Wvrec​mt−1′\.k\_\{t\}^\{\\rm rec\}=\\operatorname\{RMSNorm\}\(W\_\{k\}^\{\\rm rec\}m^\{\\prime\}\_\{t\-1\}\),\\qquad v\_\{t\}^\{\\rm rec\}=W\_\{v\}^\{\\rm rec\}m^\{\\prime\}\_\{t\-1\}\.\(6\)At inference, decoder memorymt−1m\_\{t\-1\}replacesmt−1′m^\{\\prime\}\_\{t\-1\}\. Layer\-specific gates mix local and recurrent features,

gloc,tℓ=2​σ​\(Glocℓ​atℓ\),grec,tℓ=2​σ​\(Grecℓ​atℓ\),g\_\{\\rm loc,t\}^\{\\ell\}=2\\sigma\(G\_\{\\rm loc\}^\{\\ell\}a\_\{t\}^\{\\ell\}\),\\qquad g\_\{\\rm rec,t\}^\{\\ell\}=2\\sigma\(G\_\{\\rm rec\}^\{\\ell\}a\_\{t\}^\{\\ell\}\),\(7\)k¯tℓ=gloc,tℓ⊙ktℓ\+grec,tℓ⊙ktrec,v¯tℓ=gloc,tℓ⊙vtℓ\+grec,tℓ⊙vtrec\.\\bar\{k\}\_\{t\}^\{\\ell\}=g\_\{\\rm loc,t\}^\{\\ell\}\\odot k\_\{t\}^\{\\ell\}\+g\_\{\\rm rec,t\}^\{\\ell\}\\odot k\_\{t\}^\{\\rm rec\},\\qquad\\bar\{v\}\_\{t\}^\{\\ell\}=g\_\{\\rm loc,t\}^\{\\ell\}\\odot v\_\{t\}^\{\\ell\}\+g\_\{\\rm rec,t\}^\{\\ell\}\\odot v\_\{t\}^\{\\rm rec\}\.\(8\)The factor of two sets each gate to the neutral value one at zero pre\-activation, while allowing independent scaling of the two channels\. Sliding\-window attention operates on the mixed K/V cache,

ctℓ=Attention⁡\(qtℓ,k¯max⁡\(1,t−W\+1\):tℓ,v¯max⁡\(1,t−W\+1\):tℓ\)\.c\_\{t\}^\{\\ell\}=\\operatorname\{Attention\}\\\!\\left\(q\_\{t\}^\{\\ell\},\\bar\{k\}\_\{\\max\(1,t\-W\+1\):t\}^\{\\ell\},\\bar\{v\}\_\{\\max\(1,t\-W\+1\):t\}^\{\\ell\}\\right\)\.\(9\)Each cached entry at positionjjtherefore combines features of local tokenxjx\_\{j\}and shifted memorymj−1′m^\{\\prime\}\_\{j\-1\}during training, ormj−1m\_\{j\-1\}during inference\. Attention over the lastWWmixed entries exposes the corresponding memory window without allocating additional sequence positions\. When enabled, RoPE is applied to both local and recurrent keys before mixing\. After all Transformer blocks and the final normalization, the resulting decoder state ismtm\_\{t\}; it serves both as the next recurrent memory and as the input to the language\-modeling head\.

### 2\.3Inference

At inference,QQis discarded andPPruns recurrently,

𝒎t=Pθ​\(xt−W\+1:t,ℳt−1\)⏟recurrent inference,ℳt−1=\{mt−W,…,mt−1\}\.\\underbrace\{\{\\color\[rgb\]\{0,0\.17578125,0\.4453125\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.17578125,0\.4453125\}\\boldsymbol\{m\}\_\{t\}\}=P\_\{\\theta\}\(x\_\{t\-W\+1:t\},\{\\color\[rgb\]\{0,0\.17578125,0\.4453125\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.17578125,0\.4453125\}\\mathcal\{M\}\_\{t\-1\}\}\)\}\_\{\\text\{recurrent inference\}\},\\qquad\{\\color\[rgb\]\{0,0\.17578125,0\.4453125\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.17578125,0\.4453125\}\\mathcal\{M\}\_\{t\-1\}\}=\\\{\{\\color\[rgb\]\{0,0\.17578125,0\.4453125\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.17578125,0\.4453125\}m\_\{t\-W\}\},\\ldots,\{\\color\[rgb\]\{0,0\.17578125,0\.4453125\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0,0\.17578125,0\.4453125\}m\_\{t\-1\}\}\\\}\.\(10\)The language\-modeling head usesmtm\_\{t\}to predictxt\+1x\_\{t\+1\}, andmtm\_\{t\}is appended to the recurrent window\. TheWWexisting K/V entries represent both token and memory histories, so Maglev matches the cache size and attention cost of ordinary sliding\-window attention, independent of sequence length\.

## 3Related Work

#### Efficient and long\-context attention\.

Transformers rely on softmax attention over past tokens\(Vaswaniet al\.,[2017](https://arxiv.org/html/2608.02870#bib.bib1)\)\. Long\-context variants reduce this cost with local attention, compressed memory, recurrence, retrieval, or hybrid memory mechanisms\. Transformer\-XL and Compressive Transformers reuse or compress segment\-level activations\(Daiet al\.,[2019](https://arxiv.org/html/2608.02870#bib.bib28); Raeet al\.,[2020](https://arxiv.org/html/2608.02870#bib.bib20)\); Infini\-attention and landmark\-style methods add compressed or random\-access memory for longer contexts\(Munkhdalaiet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib19); Mohtashami and Jaggi,[2023](https://arxiv.org/html/2608.02870#bib.bib55)\); Memorizing Transformers augment attention with external retrieval\(Wuet al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib36)\)\. Sliding\-window attention is especially practical because it bounds compute and cache size, but it drops information outside the window\. Maglev keeps the sliding window and adds a shifted memory that carries information forward through the decoder\.

#### Linear recurrent sequence models\.

Linear attention and state space models replace the full key\-value cache with a fixed\-size recurrent state\(Katharopouloset al\.,[2020](https://arxiv.org/html/2608.02870#bib.bib5); Guet al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib6); Gu and Dao,[2023](https://arxiv.org/html/2608.02870#bib.bib7); Dao and Gu,[2024](https://arxiv.org/html/2608.02870#bib.bib8)\)\. Related convolutional and state\-space models such as H3, S5, Hyena, and Mamba use efficient recurrent or convolutional sequence operators\(Fuet al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib58); Smithet al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib61); Poliet al\.,[2023](https://arxiv.org/html/2608.02870#bib.bib59); Gu and Dao,[2023](https://arxiv.org/html/2608.02870#bib.bib7)\)\. More recent gated recurrent architectures, including RetNet, RWKV, Griffin/Hawk, gated delta networks, and xLSTM, add data\-dependent gates or learned update rules for stronger memory control\(Sunet al\.,[2023](https://arxiv.org/html/2608.02870#bib.bib9); Penget al\.,[2023](https://arxiv.org/html/2608.02870#bib.bib56); Deet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib17); Yanget al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib13); Becket al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib60)\)\. These models are efficient, but their state updates are often linear, affine, or specialized recurrent operators\. Maglev also uses a bounded state, but the state is produced by a nonlinear Transformer decoder and trained through prefiller consistency\.

#### Hybrid attention and recurrence\.

Hybrid models combine local attention with recurrent or state\-space layers so recent tokens remain easy to access while older context is compressed\(Deet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib17); Renet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib18); Lieberet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib57)\)\. Feedback Transformers and TransformerFAM expose high\-level past representations to future computation\(Fanet al\.,[2021](https://arxiv.org/html/2608.02870#bib.bib29); Hwanget al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib35)\); Recurrent Memory Transformers and Block\-Recurrent Transformers carry memory tokens or block states across segments\(Bulatovet al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib30); Hutchinset al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib31)\); block\-state and retention\-style models balance recurrence and parallelism through blockwise computation\(Pilaultet al\.,[2023](https://arxiv.org/html/2608.02870#bib.bib37); Sunet al\.,[2023](https://arxiv.org/html/2608.02870#bib.bib9)\)\. Maglev shares the goal of bounded memory, but keeps the deployed model close to a sliding\-window Transformer by injecting a shifted memory through the decoder’s K/V pathway\.

#### Online and test\-time memory\.

Several recent works view sequence models as online learners or test\-time memory systems\(Liuet al\.,[2024b](https://arxiv.org/html/2608.02870#bib.bib21); Sunet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib22); Behrouzet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib23)\)\. Other work studies how to parallelize nonlinear recurrent computation\(Limet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib25); Gonzalezet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib24)\)\. Maglev is closest in spirit to this line because its recurrent update is nonlinear, but it avoids sequential training by learning from a parallel prefiller rather than by directly unrolling the decoder recurrence\.

#### Latent recurrence and extra computation\.

Depth\-recurrent and latent\-thinking methods add computation by looping blocks or inserting auxiliary tokens before prediction\(Dehghaniet al\.,[2019](https://arxiv.org/html/2608.02870#bib.bib27); Giannouet al\.,[2023](https://arxiv.org/html/2608.02870#bib.bib32); Geipinget al\.,[2026](https://arxiv.org/html/2608.02870#bib.bib50); Goyalet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib38); Pfauet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib51); Herel and Mikolov,[2024](https://arxiv.org/html/2608.02870#bib.bib52); Zelikmanet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib53); Haoet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib54)\)\. Concurrently, LRT\(Huanget al\.,[2026](https://arxiv.org/html/2608.02870#bib.bib69)\)passes a previous\-token memory state into the next token and trains this recurrence through parallel refinement passes\. Other concurrent work also uses parallel supervision for recurrent memory: Supervised Memory Training\(Kumar and Isola,[2026](https://arxiv.org/html/2608.02870#bib.bib70)\)trains general RNNs from Transformer\-generated memory labels, while Rec2PM\(Chenet al\.,[2026](https://arxiv.org/html/2608.02870#bib.bib71)\)trains compact preference\-memory updates for long\-sequence generative recommendation\. Maglev applies a similar idea to language\-modeling Transformers: it injects shifted token memories through the K/V pathway, retaining the attention\-window and cache profile of ordinary sliding\-window attention while carrying a richer recurrent state\. It also supports sharing parameters between the prefillerQQand decoderPP, tying the parallel training signal directly to the recurrent Transformer rather than using a fully separate teacher–student system\.

## 4Experiments

We evaluate Maglev in the nanochat pretraining stack\(Karpathy,[2025](https://arxiv.org/html/2608.02870#bib.bib34)\)\. All models use the same tokenizer, data pipeline, optimizer family, sequence length, and evaluation scripts; architecture\-specific differences are described below\.

#### Training setup\.

We use the d20 nanochat architecture:L=20L=20layers, widthd=1280d=1280, head dimension128128,1010attention heads, and maximum sequence length20482048\. The short\-window size isW=512W=512\. The standard d20 model has435,159,040435\{,\}159\{,\}040non\-embedding scaling parameters, and we train for43\.5243\.52B tokens, corresponding to a100×100\\timesChinchilla\-style token budget when embedding parameters are excluded\(Hoffmannet al\.,[2022](https://arxiv.org/html/2608.02870#bib.bib41)\)\. The optimizer batch contains524,288524\{,\}288tokens per step, and we use the MuonAdamW training recipe\(Jordanet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib42)\)\.

#### Models\.

We compare against an interleaved full/sliding\-window Transformer with layer patternSLSL, a purely sliding\-window model with layer patternSSSS, and LRT variants using the same two layer patterns\. Both LRT and Maglev use the same shared recurrent K/V injection and residual/input\-skip update\. In Maglev, the prefillerQQusesSLSLand the decoderPPusesSSSS\. We evaluate shared\-parameter Maglev and a separate\-parameter variant, each with consistency weightλ∈\{0\.1,1\.0\}\\lambda\\in\\\{0\.1,1\.0\\\}\.

#### Evaluation\.

We report FineWeb\-Edu validation bits per byte \(FW BPB\)\(Lozhkovet al\.,[2024](https://arxiv.org/html/2608.02870#bib.bib43)\), LAMBADA perplexity and accuracy\(Papernoet al\.,[2016](https://arxiv.org/html/2608.02870#bib.bib62)\), and common pretraining downstream benchmarks: PIQA\(Bisket al\.,[2020](https://arxiv.org/html/2608.02870#bib.bib63)\), HellaSwag\(Zellerset al\.,[2019](https://arxiv.org/html/2608.02870#bib.bib64)\), WinoGrande\(Sakaguchiet al\.,[2021](https://arxiv.org/html/2608.02870#bib.bib65)\), ARC\-Easy and ARC\-Challenge\(Clarket al\.,[2018](https://arxiv.org/html/2608.02870#bib.bib66)\), SocialIQA\(Sapet al\.,[2019](https://arxiv.org/html/2608.02870#bib.bib67)\), and BoolQ\(Clarket al\.,[2019](https://arxiv.org/html/2608.02870#bib.bib68)\)\. The average column is the mean of the reported downstream accuracies\.

![Refer to caption](https://arxiv.org/html/2608.02870v1/figures/training_consistency.png)Figure 2:Training dynamics for the Maglev consistency objective\. The prefillerQQsupplies memory targetsmt′m^\{\\prime\}\_\{t\}, and the decoderPPis trained to produce matching memoriesmtm\_\{t\}while optimizing next\-token prediction\.ModelFWLMDLMDPIQAHellaWinoARC\-EARC\-CSIQABoolQAvgBPB↓\\downarrowPPL↓\\downarrowAcc↑\\uparrowAcc↑\\uparrowAccn↑\\uparrowAcc↑\\uparrowAcc↑\\uparrowAccn↑\\uparrowAcc↑\\uparrowAcc↑\\uparrowAcc↑\\uparrowTransformer \(SLSL\)0\.73730\.73738\.448\.4445\.945\.973\.753\.053\.057\.757\.768\.668\.638\.838\.841\.741\.756\.956\.954\.554\.5LRT \(SLSL\)0\.72920\.72927\.947\.9447\.747\.772\.472\.454\.954\.958\.158\.170\.170\.139\.439\.440\.740\.763\.663\.655\.955\.9SWA \(SSSS\)0\.74130\.74138\.548\.5446\.246\.270\.570\.553\.353\.357\.157\.168\.468\.440\.140\.141\.341\.356\.256\.254\.154\.1LRT \(SSSS\)0\.73310\.73317\.927\.9247\.347\.372\.372\.354\.954\.958\.870\.439\.839\.840\.740\.756\.256\.255\.055\.0Maglev \(λ=0\.1\\lambda=0\.1\)0\.72950\.72958\.068\.0647\.347\.372\.072\.055\.258\.758\.770\.070\.041\.341\.341\.641\.663\.663\.656\.256\.2Maglev \(λ=1\\lambda=1\)0\.73200\.73208\.278\.2746\.346\.372\.472\.453\.853\.856\.456\.469\.769\.739\.839\.841\.241\.251\.151\.153\.953\.9Maglev \(sep\.λ=0\.1\\lambda=0\.1\)0\.72760\.72767\.7348\.772\.272\.254\.754\.757\.957\.969\.269\.240\.540\.542\.762\.062\.056\.056\.0Maglev \(sep\.λ=1\\lambda=1\)0\.72518\.068\.0647\.447\.472\.672\.655\.255\.257\.457\.469\.769\.742\.742\.142\.164\.056\.4

Table 1:Pretraining benchmark results after43\.5243\.52B training tokens\. FW BPB is evaluated on FineWeb\-Edu validation data; LMD denotes LAMBADA\. HellaSwag and ARC\-Challenge use normalized accuracy\. Higher is better except for BPB and perplexity\.
#### Results\.

Maglev improves the fixed\-window decoder without relying on full attention at inference\. The shared Maglev model withλ=0\.1\\lambda=0\.1reaches0\.72950\.7295FW BPB and56\.256\.2average downstream accuracy, improving over both the SSSS sliding\-window baseline and the corresponding LRT baseline\. The separate\-parameter variant withλ=1\\lambda=1gives the best FW BPB \(0\.72510\.7251\) and average downstream score \(56\.456\.4\)\. The comparison suggests that additional prefiller capacity can improve the consistency target, while the shared model uses less parameter memory and keeps the two paths tightly coupled\.

The results also show that the consistency weight is not purely monotone\. A largerλ\\lambdaimproves the separate\-parameter model, but hurts the shared model on several downstream tasks\. This is consistent with the role ofQQ: whenQQandPPshare most parameters, an overly strong memory\-consistency term can constrain the decoder representation; whenQQhas separate capacity, the stronger target can provide a more useful training signal\.

## 5Conclusion and Future Work

We introduced Maglev, a fixed\-memory recurrent Transformer trained through a prefiller–decoder consistency objective\. The prefiller provides parallel memory targetsmt′m^\{\\prime\}\_\{t\}, while the decoder learns to predict tokens and produce its own memoriesmtm\_\{t\}using sliding\-window attention with recurrent K/V injection\. At inference, the prefiller is removed and the decoder runs as a bounded\-memory recurrent model\. Our experiments show that this scheme improves validation BPB and downstream pretraining benchmarks over sliding\-window and LRT baselines in the nanochat d20 setting\. More broadly, Maglev provides a way to train a nonlinear recurrent model with fixed inference memory while preserving parallel training\.

Because our experiments were constrained by available compute, Maglev remains a preliminary investigation rather than a definitive study of this design space\. Several directions remain open\. First, scaling Maglev will require studying the tradeoff between prefiller strength and decoder capacity, together with kernels that make recurrent injection efficient in deployment\. Second,QQneed not be trained from scratch withPP: a pretrained or lightly fine\-tuned language model could provide memory targets while onlyPPis trained, distilling its representations into a compact recurrent decoder\. Third, our results show that substantial sharing betweenQQandPPis possible, but the best sharing pattern remains unclear; intermediate designs could share embeddings, MLPs, attention projections, or selected layers while retaining task\-specific components\. Finally, recurrent K/V injection is only one way to expose past memories\. Future work should compare alternatives such as residual\-stream injection, recurrent tokens, cross\-attention, and layer\-specific memory projections, characterizing their tradeoffs in expressivity, stability, parameter cost, cache size, and inference throughput\.

## References

- M\. Beck, K\. Pöppel, M\. Spanring, A\. Auer, O\. Prudnikova, M\. Kopp, G\. Klambauer, J\. Brandstetter, and S\. Hochreiter \(2024\)xLSTM: extended long short\-term memory\.Advances in Neural Information Processing Systems37,pp\. 107547–107603\.Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px2.p1.1)\.
- A\. Behrouz, P\. Zhong, and V\. Mirrokni \(2024\)Titans: learning to memorize at test time\.External Links:2501\.00663,[Link](https://arxiv.org/abs/2501.00663)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px4.p1.1)\.
- I\. Beltagy, M\. E\. Peters, and A\. Cohan \(2020\)Longformer: the long\-document transformer\.arXiv preprint arXiv:2004\.05150\.External Links:2004\.05150,[Link](https://arxiv.org/abs/2004.05150)Cited by:[2nd item](https://arxiv.org/html/2608.02870#S1.I1.i2.p1.1),[§1](https://arxiv.org/html/2608.02870#S1.p1.1)\.
- Y\. Bisk, R\. Zellers, R\. L\. Bras, J\. Gao, and Y\. Choi \(2020\)PIQA: reasoning about physical commonsense in natural language\.Proceedings of the AAAI Conference on Artificial Intelligence34\(05\),pp\. 7432–7439\.Cited by:[§4](https://arxiv.org/html/2608.02870#S4.SS0.SSS0.Px3.p1.1)\.
- A\. Bulatov, Y\. Kuratov, and M\. S\. Burtsev \(2022\)Recurrent memory transformer\.InAdvances in Neural Information Processing Systems,Cited by:[1st item](https://arxiv.org/html/2608.02870#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2608.02870#S1.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px3.p1.1)\.
- Y\. Chen, Y\. Wang, Y\. Liu, Q\. Wang, K\. Cheng, X\. Xu, J\. Yan, S\. Yang, M\. Guo, J\. Zhang, H\. Yu, and J\. Jiang \(2026\)Recurrent preference memory for efficient long\-sequence generative recommendation\.arXiv preprint arXiv:2602\.11605\.External Links:[Link](https://arxiv.org/abs/2602.11605)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px5.p1.2)\.
- C\. Clark, K\. Lee, M\. Chang, T\. Kwiatkowski, M\. Collins, and K\. Toutanova \(2019\)BoolQ: exploring the surprising difficulty of natural yes/no questions\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics,pp\. 2924–2936\.External Links:[Link](https://aclanthology.org/N19-1300/)Cited by:[§4](https://arxiv.org/html/2608.02870#S4.SS0.SSS0.Px3.p1.1)\.
- P\. Clark, I\. Cowhey, O\. Etzioni, T\. Khot, A\. Sabharwal, C\. Schoenick, and O\. Tafjord \(2018\)Think you have solved question answering? try ARC, the AI2 reasoning challenge\.InarXiv preprint arXiv:1803\.05457,External Links:[Link](https://arxiv.org/abs/1803.05457)Cited by:[§4](https://arxiv.org/html/2608.02870#S4.SS0.SSS0.Px3.p1.1)\.
- Z\. Dai, Z\. Yang, Y\. Yang, J\. Carbonell, Q\. V\. Le, and R\. Salakhutdinov \(2019\)Transformer\-XL: attentive language models beyond a fixed\-length context\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 2978–2988\.External Links:[Link](https://aclanthology.org/P19-1285/)Cited by:[1st item](https://arxiv.org/html/2608.02870#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2608.02870#S1.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px1.p1.1)\.
- T\. Dao and A\. Gu \(2024\)Transformers are SSMs: generalized models and efficient algorithms through structured state space duality\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 10041–10071\.External Links:[Link](https://proceedings.mlr.press/v235/dao24a.html)Cited by:[1st item](https://arxiv.org/html/2608.02870#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2608.02870#S1.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px2.p1.1)\.
- S\. De, S\. L\. Smith, A\. Fernando, A\. Botev, G\. Cristian\-Muraru, A\. Gu, R\. Haroun, L\. Berrada, Y\. Chen, S\. Srinivasan, G\. Desjardins, A\. Doucet, D\. Budden, Y\. W\. Teh, R\. Pascanu, N\. De Freitas, and C\. Gulcehre \(2024\)Griffin: mixing gated linear recurrences with local attention for efficient language models\.External Links:2402\.19427,[Link](https://arxiv.org/abs/2402.19427)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px3.p1.1)\.
- M\. Dehghani, S\. Gouws, O\. Vinyals, J\. Uszkoreit, and Ł\. Kaiser \(2019\)Universal transformers\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HyzdRiR9Y7)Cited by:[1st item](https://arxiv.org/html/2608.02870#S1.I1.i1.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px5.p1.2)\.
- A\. Fan, T\. Lavril, E\. Grave, A\. Joulin, and S\. Sukhbaatar \(2021\)Addressing some limitations of transformers with feedback memory\.arXiv preprint arXiv:2002\.09402\.External Links:[Link](https://arxiv.org/abs/2002.09402)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px3.p1.1)\.
- D\. Y\. Fu, T\. Dao, K\. K\. Saab, A\. W\. Thomas, A\. Rudra, and C\. Ré \(2022\)Hungry hungry hippos: towards language modeling with state space models\.arXiv preprint arXiv:2212\.14052\.External Links:[Link](https://arxiv.org/abs/2212.14052)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px2.p1.1)\.
- J\. Geiping, S\. McLeish, N\. Jain, J\. Kirchenbauer, S\. Singh, B\. Bartoldson, B\. Kailkhura, A\. Bhatele, and T\. Goldstein \(2026\)Scaling up test\-time compute with latent reasoning: a recurrent depth approach\.Advances in Neural Information Processing Systems38,pp\. 41340–41391\.Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px5.p1.2)\.
- A\. Giannou, S\. Rajput, J\. Sohn, K\. Lee, J\. D\. Lee, and D\. Papailiopoulos \(2023\)Looped transformers as programmable computers\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 11398–11442\.External Links:[Link](https://proceedings.mlr.press/v202/giannou23a.html)Cited by:[1st item](https://arxiv.org/html/2608.02870#S1.I1.i1.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px5.p1.2)\.
- X\. Gonzalez, A\. Warrington, J\. T\. H\. Smith, and S\. Linderman \(2024\)Towards scalable and stable parallelization of nonlinear RNNs\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=hBCxxVQDBw)Cited by:[3rd item](https://arxiv.org/html/2608.02870#S1.I1.i3.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px4.p1.1)\.
- S\. Goyal, Z\. Ji, A\. S\. Rawat, A\. K\. Menon, S\. Kumar, and V\. Nagarajan \(2024\)Think before you speak: training language models with pause tokens\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ph04CRkPdC)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px5.p1.2)\.
- A\. Gu and T\. Dao \(2023\)Mamba: linear\-time sequence modeling with selective state spaces\.External Links:2312\.00752,[Link](https://arxiv.org/abs/2312.00752)Cited by:[1st item](https://arxiv.org/html/2608.02870#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2608.02870#S1.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px2.p1.1)\.
- A\. Gu, K\. Goel, and C\. Ré \(2022\)Efficiently modeling long sequences with structured state spaces\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=uYLFoz1vlAC)Cited by:[1st item](https://arxiv.org/html/2608.02870#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2608.02870#S1.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px2.p1.1)\.
- S\. Hao, S\. Sukhbaatar, D\. Su, X\. Li, Z\. Hu, J\. Weston, and Y\. Tian \(2024\)Training large language models to reason in a continuous latent space\.arXiv preprint arXiv:2412\.06769\.External Links:[Link](https://arxiv.org/abs/2412.06769)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px5.p1.2)\.
- D\. Herel and T\. Mikolov \(2024\)Thinking tokens for language modeling\.arXiv preprint arXiv:2405\.08644\.External Links:[Link](https://arxiv.org/abs/2405.08644)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px5.p1.2)\.
- S\. Hochreiter and J\. Schmidhuber \(1997\)Long short\-term memory\.Neural Computation9\(8\),pp\. 1735–1780\.External Links:[Document](https://dx.doi.org/10.1162/neco.1997.9.8.1735),[Link](https://doi.org/10.1162/neco.1997.9.8.1735)Cited by:[§1](https://arxiv.org/html/2608.02870#S1.p1.1)\.
- J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. d\. L\. Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.\(2022\)Training compute\-optimal large language models\.arXiv preprint arXiv:2203\.15556\.External Links:[Link](https://arxiv.org/abs/2203.15556)Cited by:[§4](https://arxiv.org/html/2608.02870#S4.SS0.SSS0.Px1.p1.10)\.
- Z\. Huang, X\. He, L\. Ren, Y\. Wang, B\. Peng, H\. Cheng, S\. Wang, P\. He, J\. Gao, Y\. J\. Lee,et al\.\(2026\)Latent recurrent transformer: architecture exploration, training strategies, and scaling behavior\.arXiv preprint arXiv:2605\.26797\.Cited by:[§2\.2](https://arxiv.org/html/2608.02870#S2.SS2.SSS0.Px2.p1.3),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px5.p1.2)\.
- D\. Hutchins, I\. Schlag, Y\. Wu, E\. Dyer, and B\. Neyshabur \(2022\)Block\-recurrent transformers\.InAdvances in Neural Information Processing Systems,Cited by:[1st item](https://arxiv.org/html/2608.02870#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2608.02870#S1.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px3.p1.1)\.
- D\. Hwang, W\. Wang, Z\. Huo, K\. C\. Sim, and P\. M\. Mengibar \(2024\)TransformerFAM: feedback attention is working memory\.arXiv preprint arXiv:2404\.09173\.External Links:[Link](https://arxiv.org/abs/2404.09173)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px3.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. Renard Lavaud, M\. Lachaux, P\. Stock, T\. Le Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. El Sayed \(2023\)Mistral 7b\.arXiv preprint arXiv:2310\.06825\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[2nd item](https://arxiv.org/html/2608.02870#S1.I1.i2.p1.1),[§1](https://arxiv.org/html/2608.02870#S1.p1.1)\.
- K\. Jordan, Y\. Jin, V\. Boza, Y\. Jiacheng, F\. Cesista, L\. Newhouse, and J\. Bernstein \(2024\)Muon: an optimizer for hidden layers in neural networks\.Note:[https://kellerjordan\.github\.io/posts/muon/](https://kellerjordan.github.io/posts/muon/)Cited by:[§4](https://arxiv.org/html/2608.02870#S4.SS0.SSS0.Px1.p1.10)\.
- A\. Karpathy \(2025\)Nanochat: the best ChatGPT that $100 can buy\.Note:[https://github\.com/karpathy/nanochat](https://github.com/karpathy/nanochat)GitHub repositoryCited by:[§2\.2](https://arxiv.org/html/2608.02870#S2.SS2.SSS0.Px1.p1.9),[§4](https://arxiv.org/html/2608.02870#S4.p1.1)\.
- A\. Katharopoulos, A\. Vyas, N\. Pappas, and F\. Fleuret \(2020\)Transformers are RNNs: fast autoregressive transformers with linear attention\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 5156–5165\.External Links:[Link](https://proceedings.mlr.press/v119/katharopoulos20a.html)Cited by:[1st item](https://arxiv.org/html/2608.02870#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2608.02870#S1.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px2.p1.1)\.
- A\. Kumar and P\. Isola \(2026\)Pretraining recurrent networks without recurrence\.arXiv preprint arXiv:2606\.06479\.External Links:[Link](https://arxiv.org/abs/2606.06479)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px5.p1.2)\.
- O\. Lieber, B\. Lenz, H\. Bata, G\. Cohen, J\. Osin, I\. Dalmedigos, E\. Safahi, S\. Meirom, Y\. Belinkov, S\. Shalev\-Shwartz,et al\.\(2024\)Jamba: a hybrid transformer\-mamba language model\.arXiv preprint arXiv:2403\.19887\.External Links:[Link](https://arxiv.org/abs/2403.19887)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px3.p1.1)\.
- Y\. H\. Lim, Q\. Zhu, J\. Selfridge, and M\. F\. Kasim \(2024\)Parallelizing non\-linear sequential models over the sequence length\.External Links:2309\.12252,[Link](https://arxiv.org/abs/2309.12252)Cited by:[3rd item](https://arxiv.org/html/2608.02870#S1.I1.i3.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px4.p1.1)\.
- B\. Liu, R\. Wang, L\. Wu, Y\. Feng, P\. Stone, and Q\. Liu \(2024a\)Longhorn: state space models are amortized online learners\.ArXivabs/2407\.14207\.External Links:[Link](https://api.semanticscholar.org/CorpusID:271310065)Cited by:[§1](https://arxiv.org/html/2608.02870#S1.p1.1)\.
- B\. Liu, R\. Wang, L\. Wu, Y\. Feng, P\. Stone, and Q\. Liu \(2024b\)Longhorn: state space models are amortized online learners\.External Links:2407\.14207,[Link](https://arxiv.org/abs/2407.14207)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px4.p1.1)\.
- A\. Lozhkov, L\. B\. Allal, L\. von Werra, and T\. Wolf \(2024\)FineWeb\-edu: the finest collection of educational content\.Note:[https://huggingface\.co/datasets/HuggingFaceFW/fineweb\-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)Cited by:[§4](https://arxiv.org/html/2608.02870#S4.SS0.SSS0.Px3.p1.1)\.
- A\. Mohtashami and M\. Jaggi \(2023\)Landmark attention: random\-access infinite context length for transformers\.arXiv preprint arXiv:2305\.16300\.External Links:[Link](https://arxiv.org/abs/2305.16300)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px1.p1.1)\.
- T\. Munkhdalai, M\. Faruqui, and S\. Gopal \(2024\)Leave no context behind: efficient infinite context transformers with infini\-attention\.External Links:2404\.07143,[Link](https://arxiv.org/abs/2404.07143)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px1.p1.1)\.
- D\. Paperno, G\. Kruszewski, A\. Lazaridou, Q\. N\. Pham, R\. Bernardi, S\. Pezzelle, M\. Baroni, G\. Boleda, and R\. Fernández \(2016\)The LAMBADA dataset: word prediction requiring a broad discourse context\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics,pp\. 1525–1534\.External Links:[Link](https://aclanthology.org/P16-1144/)Cited by:[§4](https://arxiv.org/html/2608.02870#S4.SS0.SSS0.Px3.p1.1)\.
- B\. Peng, E\. Alcaide, Q\. Anthony, A\. Albalak, S\. Arcadinho, H\. Cao, X\. Cheng, M\. Chung, M\. Grella,et al\.\(2023\)RWKV: reinventing RNNs for the transformer era\.arXiv preprint arXiv:2305\.13048\.External Links:[Link](https://arxiv.org/abs/2305.13048)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px2.p1.1)\.
- J\. Pfau, W\. Merrill, and S\. R\. Bowman \(2024\)Let’s think dot by dot: hidden computation in transformer language models\.arXiv preprint arXiv:2404\.15758\.External Links:[Link](https://arxiv.org/abs/2404.15758)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px5.p1.2)\.
- J\. Pilault, M\. Fathi, O\. Firat, C\. Pal, P\. Bacon, and R\. Goroshin \(2023\)Block\-state transformers\.Advances in Neural Information Processing Systems36,pp\. 7311–7329\.Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px3.p1.1)\.
- M\. Poli, S\. Massaroli, E\. Nguyen, D\. Y\. Fu, T\. Dao, S\. Baccus, Y\. Bengio, S\. Ermon, and C\. Ré \(2023\)Hyena hierarchy: towards larger convolutional language models\.InInternational Conference on Machine Learning,pp\. 28043–28078\.Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px2.p1.1)\.
- J\. W\. Rae, A\. Potapenko, S\. M\. Jayakumar, C\. Hillier, and T\. P\. Lillicrap \(2020\)Compressive transformers for long\-range sequence modelling\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SylKikSYDH)Cited by:[1st item](https://arxiv.org/html/2608.02870#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2608.02870#S1.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px1.p1.1)\.
- L\. Ren, Y\. Liu, Y\. Lu, Y\. Shen, C\. Liang, and W\. Chen \(2024\)Samba: simple hybrid state space models for efficient unlimited context language modeling\.External Links:2406\.07522,[Link](https://arxiv.org/abs/2406.07522)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px3.p1.1)\.
- K\. Sakaguchi, R\. L\. Bras, C\. Bhagavatula, and Y\. Choi \(2021\)WinoGrande: an adversarial winograd schema challenge at scale\.Proceedings of the AAAI Conference on Artificial Intelligence35\(10\),pp\. 8732–8740\.Cited by:[§4](https://arxiv.org/html/2608.02870#S4.SS0.SSS0.Px3.p1.1)\.
- M\. Sap, H\. Rashkin, D\. Chen, R\. Le Bras, and Y\. Choi \(2019\)Social IQA: commonsense reasoning about social interactions\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing,pp\. 4463–4473\.External Links:[Link](https://aclanthology.org/D19-1454/)Cited by:[§4](https://arxiv.org/html/2608.02870#S4.SS0.SSS0.Px3.p1.1)\.
- J\. T\. H\. Smith, A\. Warrington, and S\. W\. Linderman \(2022\)Simplified state space layers for sequence modeling\.arXiv preprint arXiv:2208\.04933\.External Links:[Link](https://arxiv.org/abs/2208.04933)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px2.p1.1)\.
- Y\. Sun, X\. Li, K\. Dalal, J\. Xu, A\. Vikram, G\. Zhang, Y\. Dubois, X\. Chen, X\. Wang, S\. Koyejo, T\. Hashimoto, and C\. Guestrin \(2024\)Learning to \(learn at test time\): RNNs with expressive hidden states\.External Links:2407\.04620,[Link](https://arxiv.org/abs/2407.04620)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px4.p1.1)\.
- Y\. Sun, L\. Dong, S\. Huang, S\. Ma, Y\. Xia, J\. Xue, J\. Wang, and F\. Wei \(2023\)Retentive network: a successor to transformer for large language models\.External Links:2307\.08621,[Link](https://arxiv.org/abs/2307.08621)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px3.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30\.External Links:[Link](https://arxiv.org/abs/1706.03762)Cited by:[3rd item](https://arxiv.org/html/2608.02870#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2608.02870#S1.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px1.p1.1)\.
- Y\. Wu, M\. N\. Rabe, D\. Hutchins, and C\. Szegedy \(2022\)Memorizing transformers\.arXiv preprint arXiv:2203\.08913\.External Links:[Link](https://arxiv.org/abs/2203.08913)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px1.p1.1)\.
- S\. Yang, J\. Kautz, and A\. Hatamizadeh \(2024\)Gated delta networks: improving mamba2 with delta rule\.External Links:2412\.06464,[Link](https://arxiv.org/abs/2412.06464)Cited by:[§1](https://arxiv.org/html/2608.02870#S1.p1.1),[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px2.p1.1)\.
- E\. Zelikman, G\. Harik, Y\. Shao, V\. Jayasiri, N\. Haber, and N\. D\. Goodman \(2024\)Quiet\-STaR: language models can teach themselves to think before speaking\.arXiv preprint arXiv:2403\.09629\.External Links:[Link](https://arxiv.org/abs/2403.09629)Cited by:[§3](https://arxiv.org/html/2608.02870#S3.SS0.SSS0.Px5.p1.2)\.
- R\. Zellers, A\. Holtzman, Y\. Bisk, A\. Farhadi, and Y\. Choi \(2019\)HellaSwag: can a machine really finish your sentence?\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4791–4800\.External Links:[Link](https://aclanthology.org/P19-1472/)Cited by:[§4](https://arxiv.org/html/2608.02870#S4.SS0.SSS0.Px3.p1.1)\.