An Input-Dependent Readout Phase for Resonant Sequence Models

# An Input-Dependent Readout Phasefor Resonant Sequence Models
Source: [https://arxiv.org/html/2610.12496](https://arxiv.org/html/2610.12496)
###### Abstract

Sequence models with a fixed\-size state avoid the quadratic cost of attention, but face a design choice\. An input\-dependent state transition gives content\-dependent memory at the cost of FFT\-based training, while a fixed transition keeps theO⁡\(T​log⁡T\)O\(T\\log T\)convolutional path but reads memory the same way for every input\. ResonatorLM, a recent model of the second kind, reads a bank of damped complex oscillators through a fixed learned phase per head\. We make that readout phase a function of the current token, which adds input dependence to the read only, keeps the convolution exact, and costs 0\.2% more parameters\. An analysis of the untrained bank shows that phase does encode elapsed time, but an ordinary token falls below the background within 2\-8 steps and no choice of damping extends linear reconstruction beyond about 16 tokens\. In 6M\-parameter character\-level models on WikiText\-2, the input\-dependent phase matches the original at the training length and removes its failure beyond it: trained on 256 tokens and tested on 1024, perplexity falls from 12\.81 to 3\.76\. We trace the failure to long\-memory heads that accumulate more history than training exposed them to\. The learned phase tracks the current character rather than distance and does not improve retrieval\.

###### Keywords:

Sequence modelling State\-space models Length generalisation Language modelling

## 1Introduction

Self\-attention\[[14](https://arxiv.org/html/2610.12496#bib.bib13)\]costs time quadratic in the context length, which has made sequence models with a fixed\-size recurrent state an active alternative\[[4](https://arxiv.org/html/2610.12496#bib.bib8),[6](https://arxiv.org/html/2610.12496#bib.bib9),[12](https://arxiv.org/html/2610.12496#bib.bib2),[5](https://arxiv.org/html/2610.12496#bib.bib10)\]\. These models share one decisive design choice: whether the state transition depends on the input\. Selective models such as Mamba and its successors\[[5](https://arxiv.org/html/2610.12496#bib.bib10),[9](https://arxiv.org/html/2610.12496#bib.bib3),[8](https://arxiv.org/html/2610.12496#bib.bib4),[7](https://arxiv.org/html/2610.12496#bib.bib5)\]make it input\-dependent and so decide per token what to store and what to forget\. The recurrence is then no longer a fixed convolution, and theO⁡\(T​log⁡T\)O\(T\\log T\)FFT training path is lost\. Structured state\-space models such as S4\[[6](https://arxiv.org/html/2610.12496#bib.bib9)\]keep the transition fixed\. Each layer is a convolution with a fixed kernel and trains with FFTs, but its memory is written and read identically for every input\.

We ask how much input dependence can be added to the fixed branch without leaving it, by making only the*read*of the state input\-dependent\. The question has a concrete form in resonant sequence models\. ResonatorLM\[[3](https://arxiv.org/html/2610.12496#bib.bib1)\]replaces attention with banks of damped complex oscillators\. Every past token enters the state at an angle proportional to its age, and the state is read by projecting onto one fixed, learned direction per head\. That direction selects which ages are read, and it never changes after training\. We let it depend on the current token\.

#### Contributions\.

1. 1\.An input\-dependent readout phase for resonant models that keeps the kernel input\-independent and the FFT path exact \(Sect\.[3](https://arxiv.org/html/2610.12496#S3)\), with a capacity identity showing that a fixed phase reads half of the stored state \(Proposition[1](https://arxiv.org/html/2610.12496#Thmproposition1)\)\.
2. 2\.A measurement of what the published oscillator bank can hold \(Sect\.[4](https://arxiv.org/html/2610.12496#S4)\): an ordinary token is undetectable after a few steps, and linear reconstruction stops at about 16 tokens whether the bank is damped, lengthened or undamped\.
3. 3\.Trained results \(Sect\.[5](https://arxiv.org/html/2610.12496#S5)\): the input\-dependent phase ties the original at the training length and removes its failure beyond it, a failure we trace causally to long\-memory heads \(Sect\.[5\.1](https://arxiv.org/html/2610.12496#S5.SS1)\)\. The learned phase is a content\-dependent read, not a pointer back in time \(Sect\.[5\.2](https://arxiv.org/html/2610.12496#S5.SS2)\)\.

## 2Background: ResonatorLM

A ResonatorLM mixer hasHHheads\. Headhhhas a dampingαh\>0\\alpha\_\{h\}\>0, a frequencyωh∈\(0,π\)\\omega\_\{h\}\\in\(0,\\pi\)and a phaseφh\\varphi\_\{h\}, all learned\. The inputxt∈ℝdx\_\{t\}\\in\\mathbb\{R\}^\{d\}is projected to a driveut,h∈ℝdhu\_\{t,h\}\\in\\mathbb\{R\}^\{d\_\{h\}\}and a gate\. Each head keeps a complex statest,h∈ℂdhs\_\{t,h\}\\in\\mathbb\{C\}^\{d\_\{h\}\}:

st,h=λhst−1,h\+ut,h,λh=e−αh\+i​ωh,sost,h=∑j≤te−αh​\(t−j\)ei​ωh​\(t−j\)uj,h\.s\_\{t,h\}=\\lambda\_\{h\}\\,s\_\{t\-1,h\}\+u\_\{t,h\},\\qquad\\lambda\_\{h\}=e^\{\-\\alpha\_\{h\}\+i\\omega\_\{h\}\},\\qquad\\text\{so\}\\qquad s\_\{t,h\}=\\sum\_\{j\\leq t\}e^\{\-\\alpha\_\{h\}\(t\-j\)\}\\,e^\{i\\omega\_\{h\}\(t\-j\)\}\\,u\_\{j,h\}\.\(1\)A token of ageτ=t−j\\tau=t\-jis stored with magnitudee−αh​τe^\{\-\\alpha\_\{h\}\\tau\}and angleωh​τ\\omega\_\{h\}\\tau\. The head outputs the real projection

yt,h=Re⁡\(e−i​φh​st,h\)=∑j≤tkh​\[t−j\]​uj,h,kh​\[τ\]=e−αh​τ​cos⁡\(ωh​τ−φh\),y\_\{t,h\}=\\operatorname\{Re\}\\\!\\left\(e^\{\-i\\varphi\_\{h\}\}s\_\{t,h\}\\right\)=\\sum\_\{j\\leq t\}k\_\{h\}\[t\-j\]\\,u\_\{j,h\},\\qquad k\_\{h\}\[\\tau\]=e^\{\-\\alpha\_\{h\}\\tau\}\\cos\(\\omega\_\{h\}\\tau\-\\varphi\_\{h\}\),\(2\)so a layer is a causal convolution with a fixed kernel, computed by FFT inO⁡\(T​log⁡T\)O\(T\\log T\)for training and by Eq\. \([1](https://arxiv.org/html/2610.12496#S2.E1)\) for decoding\. The output is gated, mixed across heads, and added to a short causal convolution\. Sinceφh\\varphi\_\{h\}is a constant, the readout keeps the component of the state along one fixed direction and discards the orthogonal one\. The gate acts after this projection and cannot recover what it discards\.111The original paper writes the kernel withcos⁡\(ω​t\+φ\)\\cos\(\\omega t\+\\varphi\)but a readout that unrolls tocos⁡\(ω​t−φ\)\\cos\(\\omega t\-\\varphi\)\. We use the latter throughout\.

## 3Method

###### Definition 1\(Input\-dependent readout phase\)

For headhhand positiontt, let

φt,h=φh\+π​tanh⁡\(wh⊤​x¯t\+bh\),yt,h=cos⁡φt,h​Re⁡st,h\+sin⁡φt,h​Im⁡st,h,\\varphi\_\{t,h\}=\\varphi\_\{h\}\+\\pi\\tanh\\\!\\left\(w\_\{h\}^\{\\top\}\\bar\{x\}\_\{t\}\+b\_\{h\}\\right\),\\qquad y\_\{t,h\}=\\cos\\varphi\_\{t,h\}\\,\\operatorname\{Re\}s\_\{t,h\}\+\\sin\\varphi\_\{t,h\}\\,\\operatorname\{Im\}s\_\{t,h\},\(3\)wherex¯t\\bar\{x\}\_\{t\}is the normalised mixer input and\(wh,bh\)\(w\_\{h\},b\_\{h\}\)are learned\.

The transitionλh\\lambda\_\{h\}is unchanged, so both state components are still causal convolutions with fixed kernels,

Re⁡st,h=∑τ≥0e−αh​τ​cos⁡\(ωh​τ\)​ut−τ,h,Im⁡st,h=∑τ≥0e−αh​τ​sin⁡\(ωh​τ\)​ut−τ,h,\\operatorname\{Re\}s\_\{t,h\}=\\sum\_\{\\tau\\geq 0\}e^\{\-\\alpha\_\{h\}\\tau\}\\cos\(\\omega\_\{h\}\\tau\)\\,u\_\{t\-\\tau,h\},\\qquad\\operatorname\{Im\}s\_\{t,h\}=\\sum\_\{\\tau\\geq 0\}e^\{\-\\alpha\_\{h\}\\tau\}\\sin\(\\omega\_\{h\}\\tau\)\\,u\_\{t\-\\tau,h\},\(4\)which share one forward FFT ofuu\. Equation \([3](https://arxiv.org/html/2610.12496#S3.E3)\) is then applied pointwise\. The FFT and recurrent paths agree to4\.4×10−164\.4\\times 10^\{\-16\}in float64\. The projection addsH⁡\(d\+1\)H\(d\+1\)parameters per layer, 12,336 in total \(\+0\.2%\+0\.2\\%\), and is zero\-initialised, so training starts exactly at the fixed\-phase model\.

The following identity quantifies what a fixed phase gives up\.

###### Proposition 1\(Capacity\)

Let the drive of one channel be white noise with unit variance, and letzt∈ℝmz\_\{t\}\\in\\mathbb\{R\}^\{m\}be linear features of the state with nonsingular covarianceΣ\\Sigma\. WriteR2​\(d\)R^\{2\}\(d\)for the fraction of the variance ofut−du\_\{t\-d\}that is linearly recoverable fromztz\_\{t\}\. Then∑d≥0R2​\(d\)=m\\sum\_\{d\\geq 0\}R^\{2\}\(d\)=m\. For the full state ofHHheads \(z=\[Re⁡s,Im⁡s\]z=\[\\operatorname\{Re\}s,\\operatorname\{Im\}s\]\) this is2​H2H; for any fixed real readout it isHH\.

###### Proof

Withgd=Cov⁡\(zt,ut−d\)g\_\{d\}=\\mathrm\{Cov\}\(z\_\{t\},u\_\{t\-d\}\),R2​\(d\)=gd⊤​Σ−1​gdR^\{2\}\(d\)=g\_\{d\}^\{\\top\}\\Sigma^\{\-1\}g\_\{d\}andΣ=∑dgd​gd⊤\\Sigma=\\sum\_\{d\}g\_\{d\}g\_\{d\}^\{\\top\}, so∑dR2​\(d\)=tr⁡\(Σ−1​∑dgd​gd⊤\)=tr⁡\(Im\)=m\\sum\_\{d\}R^\{2\}\(d\)=\\mathrm\{tr\}\\big\(\\Sigma^\{\-1\}\\sum\_\{d\}g\_\{d\}g\_\{d\}^\{\\top\}\\big\)=\\mathrm\{tr\}\(I\_\{m\}\)=m\. ∎

A fixed phase therefore leaves half of the recoverable information unread \(numerically8\.008\.00of16\.0016\.00for the published bank\)\. Choosing the phase per lag recovers essentially the full\-state value, but the total is spread thinly over lags:R2R^\{2\}falls below 0\.5 byd=5d=5and below 0\.1 beyondd=16d=16\.

## 4What the Oscillator Bank Can Hold

We first analyse the published bank without training:H=8H=8heads, half\-livesln⁡2/αh\\ln 2/\\alpha\_\{h\}log\-spaced over\[2,2048\]\[2,2048\], frequencies log\-spaced over\[π/1024,0\.99​π\]\[\\pi/1024,\\,0\.99\\pi\],dh=32d\_\{h\}=32, float64\.

#### Detectability\.

The bank is driven by white noise forT=4160T=4160steps, and one marked token of typical norm \(dh\\sqrt\{d\_\{h\}\}\) is addedddsteps back\. Reading at the angle that token occupies,φ=ωh​d\\varphi=\\omega\_\{h\}d, we measure

SNRh​\(d\)=𝔼​\|rh​\(s\)\|sd⁡\(rh​\(sbg\)\),rh​\(s\)=⟨Re⁡\(e−i​ωh​d​sh\),u^⟩,\\mathrm\{SNR\}\_\{h\}\(d\)=\\frac\{\\mathbb\{E\}\\,\\lvert r\_\{h\}\(s\)\\rvert\}\{\\mathrm\{sd\}\\,\\big\(r\_\{h\}\(s\_\{\\mathrm\{bg\}\}\)\\big\)\},\\qquad r\_\{h\}\(s\)=\\big\\langle\\operatorname\{Re\}\(e^\{\-i\\omega\_\{h\}d\}s\_\{h\}\),\\,\\hat\{u\}\\big\\rangle,\(5\)wheresbgs\_\{\\mathrm\{bg\}\}is the state without the token andu^\\hat\{u\}its direction\. With no token,SNR=2/π≈0\.80\\mathrm\{SNR\}=\\sqrt\{2/\\pi\}\\approx 0\.80exactly\. Figure[1](https://arxiv.org/html/2610.12496#S4.F1)shows that the token exceedsSNR=3\\mathrm\{SNR\}=3only ford≤2d\\leq 2, and heads with half\-life≥105\\geq 105remain near the null even ford=1d=1\. Their background grows from 1\.4 to 37\.3 in standard deviation as the half\-life lengthens: the damping that buys range also accumulates more background\.

![Refer to caption](https://arxiv.org/html/2610.12496v1/figures/fig_1b_snr_vs_distance.png)Figure 1:Detectability of one typical token against per\-step background, per head \(Eq\.[5](https://arxiv.org/html/2610.12496#S4.E5)\)\. Dashed lines markSNR=1\\mathrm\{SNR\}=1and33\.
#### Reconstruction from text\.

We drive the bank with WikiText\-2 characters\[[10](https://arxiv.org/html/2610.12496#bib.bib14)\]through a fixed random embedding, and train a linear decoder on\[Re⁡st,Im⁡st\]\[\\operatorname\{Re\}s\_\{t\},\\operatorname\{Im\}s\_\{t\}\]to predict the character att−dt\-d\. States are read att∈\[512,1024\)t\\in\[512,1024\), data are split by sequence, and the decoder is early\-stopped\. Five banks are compared: the published one, half\-lives up to6,9316\{,\}931\(the model’s floor onα\\alpha\), four heads nearly undamped, all heads undamped, and the six banks a trained model learned\. All damped banks give the same curve \(Fig\.[2](https://arxiv.org/html/2610.12496#S4.F2)\): above 98% tod=4d=4, 76% atd=8d=8, about 50% atd=12d=12, and chance byd=32d=32–6464\. The undamped bank is at chance even atd=0d=0, because no token dominates a sum in which every past token has equal weight\. The limit is crowding, not damping\.

![Refer to caption](https://arxiv.org/html/2610.12496v1/figures/fig_2b_reconstruction.png)Figure 2:Linear reconstruction of the characterddsteps back from the oscillator state\. Damped banks coincide; the undamped bank is unreadable\.Two further checks support this picture\. With no background, the eight heads jointly locate a token’s age exactly tod≥32,768d\\geq 32\{,\}768, sixteen times the longest period, so the time information is present\. With background, however, an ordinary token is located only tod≈8d\\approx 8\. Repeating both measurements with the bfloat16 roundings used in training changes the SNR by at most0\.09%0\.09\\%and reconstruction by at most 0\.65 points\. An input\-dependent phase can therefore choose*what*to read, but not recover what the state does not hold\.

## 5Trained Models

#### Setup\.

We follow the ResonatorLM protocol: WikiText\-2 raw at character level \(vocabulary 1013\), 6 layers,d=256d=256,H=8H=8,dh=32d\_\{h\}=32, 10,000 AdamW steps \(learning rate5×10−45\\times 10^\{\-4\}, cosine schedule, weight decay 0\.1\) with 8,192 tokens per step, and bfloat16\. The models compared are:A, the original \(fixedφh\\varphi\_\{h\}, initialised at 0; 6 seeds\);C, the proposal \(Eq\.[3](https://arxiv.org/html/2610.12496#S3.E3); 3 seeds\); two fixed\-phase controls \(3 seeds each\),Bwithφh\\varphi\_\{h\}initialised uniformly on\(−π,π\)\(\-\\pi,\\pi\)andEwith only the two longest\-memory heads initialised atπ/2\\pi/2; and matched transformers \(d=248d=248\) with rotary \(RoPE\)\[[13](https://arxiv.org/html/2610.12496#bib.bib12)\]or learned position embeddings \(6 seeds each\)\. A and C are trained atT∈\{256,512,1024\}T\\in\\\{256,512,1024\\\}and tested on the test split atTT,2​T2Tand4​T4T\. A run*collapses*ifPPL⁡\(4​T\)\>1\.5​PPL​\(T\)\\mathrm\{PPL\}\(4T\)\>1\.5\\,\\mathrm\{PPL\}\(T\)\.

#### At the training length\.

Table[1](https://arxiv.org/html/2610.12496#S5.T1)shows that C and A tie at every training length\. Both improve on the published ResonatorLM numbers, and at 256 the RoPE transformer is better than either, contrary to the original report\.

Table 1:Test perplexity and accuracy at the training length \(mean±\\pmsd over seeds\)\. Published values from\[[3](https://arxiv.org/html/2610.12496#bib.bib1)\]\.
#### Beyond the training length\.

Table[2](https://arxiv.org/html/2610.12496#S5.T2)and Fig\.[3](https://arxiv.org/html/2610.12496#S5.F3)give the main result\. C is lower than A in all six settings beyond the training length, with non\-overlapping seed ranges in each\. The effect is largest when training is short\. Trained at 256 and tested at 1024, A collapses in all 6 seeds \(perplexity 10\.3–14\.7, accuracy 45\.0%\) while C stays at 3\.70–3\.84 \(accuracy 63\.5%\)\. Trained at 512, A degrades by 40% at4​T4Tand C by 3%\. Trained at 1024, both stay within 3%\. The transformers degrade as well, most severely with learned positions\.

Table 2:Test perplexity when testing beyond the training lengthTT\(mean over seeds\)\. The tested length is shown in each block\. Ratio isPPL⁡\(4​T\)/PPL⁡\(T\)\\mathrm\{PPL\}\(4T\)/\\mathrm\{PPL\}\(T\); “collapsed” counts seeds with ratio\>1\.5\>1\.5\.![Refer to caption](https://arxiv.org/html/2610.12496v1/figures/fig_4_length_generalisation.png)Figure 3:Perplexity at1×1\\times,2×2\\timesand4×4\\timesthe training length, one line per seed\. A degrades beyond its training length, most when trained short; C does not\.
#### Controls\.

A fixed phase with a different initialisation does not reliably reproduce the effect \(Table[2](https://arxiv.org/html/2610.12496#S5.T2),T=256T=256\)\. B collapses in one of three seeds \(18\.79\) and avoids it in two \(4\.22, 3\.54\)\. E keeps its two longest heads nearπ/2\\pi/2throughout training \(\|cos⁡φ\|≈0\.2\\lvert\\cos\\varphi\\rvert\\approx 0\.2\) yet collapses in all three\. Across all runs trained at 256, 10 of 12 fixed\-phase runs collapse at 1024, against 0 of 3 for C\. Three seeds cannot show that input dependence is necessary, but it was the only variant that never collapsed\.

### 5\.1Why the Original Model Collapses

We evaluated the 15 models trained at 256 on the same 1024\-token sequences, measuring internal signals by position, and then modified the models at test time only\.

#### Where and what\.

For A, perplexity is 3\.40–3\.45 on positions 0–255 and 3\.52–3\.64 on 256–511, then 18\.4–41\.7 on 512–767 and 48\.8–76\.0 on 768–1023\. Among about 150 per\-layer and per\-head measurements, those that best separate collapsing from healthy runs all concern long\-memory heads \(learned half\-life above about 200\) in the upper layers\. For example, the RMS state of head 6 in layer 4 grows by at least3\.7×3\.7\\timesfrom the first to the last quarter of the sequence in every collapsing run, and by at most2\.0×2\.0\\timesin every healthy run \(means: A5\.5×5\.5\\times, C1\.7×1\.7\\times\)\. The upper layers thus receive long\-memory signals several times larger than any seen in training\.

#### Causal test\.

Table[3](https://arxiv.org/html/2610.12496#S5.T3)confirms the cause\. Truncating only the kernels of heads with half\-life above 256 to 256 lags removes the collapse \(A:12\.70→3\.4312\.70\\to 3\.43; E:7\.57→3\.417\.57\\to 3\.41\), and cannot change anything at 256\. Removing the constant component of the drive, whose accumulation in a slow head rotates with absolute position, repairs most of A’s collapse but none of E’s, so it is not the general cause\.

Table 3:Test\-time interventions on models trained at 256: perplexity at 256 / at 1024 \(mean over seeds 0–2\)\. Truncation cuts a head’s kernel to 256 lags\.
#### Consequence\.

The collapse has two remedies\. Truncation is free and on WikiText\-2 slightly better than C at 1024 \(3\.43 vs\. 3\.76\), but it discards all history older than the training length\. That costs nothing here, since Sect\.[4](https://arxiv.org/html/2610.12496#S4)shows that no linearly recoverable information survives that long\. The input\-dependent phase avoids the collapse with the full memory retained, because its long\-memory states drift far less\. Whether the retained memory helps requires a task that depends on older information, which we leave to future work\.

### 5\.2What the Learned Phase Does

Figure[4](https://arxiv.org/html/2610.12496#S5.F4)analyses the learned offsetδt,h=φt,h−φh\\delta\_\{t,h\}=\\varphi\_\{t,h\}\-\\varphi\_\{h\}of C on test data\.

- •*It varies\.*Its standard deviation is 0\.78–0\.98 rad in every layer, andtanh\\tanhsaturates on fewer than 3% of tokens\.
- •*It tracks content, not distance\.*In layers 1–5 the current character explains 31–58% of its variance, position in the word 17–33%, and distance to the previous occurrence of the same character 4–8%\. The implied lookback\(φt,hmod2​π\)/ωh\(\\varphi\_\{t,h\}\\bmod 2\\pi\)/\\omega\_\{h\}has rank correlation between−0\.05\-0\.05and0\.070\.07with that distance\.
- •*It is used\.*Fixing it atφh\\varphi\_\{h\}raises perplexity from 3\.49 to 5\.74–8\.54, and shuffling it across tokens to 7\.16–11\.15\.
- •*It does not retrieve more\.*A linear probe decoding the characterddsteps back from the final hidden state scores 77\.1/66\.7/45\.0/18\.6% for C against 78\.7/70\.0/50\.4/20\.2% for A atd=1,2,4,8d=1,2,4,8\.

![Refer to caption](https://arxiv.org/html/2610.12496v1/figures/fig_3_phi_distribution.png)Figure 4:The learned phase of C \(3 seeds\)\. \(a\) Offset distribution per head\. \(b\) Variance explained by the current character, position in word, and distance to the last same character\. \(c\) Perplexity with the phase intact, fixed, set to the best constant, or shuffled\.The phase therefore does not act as a pointer back in time\. By Eq\. \([3](https://arxiv.org/html/2610.12496#S3.E3)\) it chooses, per token, which mix ofRe⁡s\\operatorname\{Re\}sandIm⁡s\\operatorname\{Im\}sto read, and so reaches over a sequence the half of the state that Proposition[1](https://arxiv.org/html/2610.12496#Thmproposition1)shows a fixed phase leaves unread\. Causality is preserved \(maximum prefix change under a suffix perturbation2\.6×10−142\.6\\times 10^\{\-14\}in float64\), and measured throughput is0\.98×0\.98\\timesthat of A\.

## 6Related Work

#### Oscillatory and structured state spaces\.

LinOSS\[[12](https://arxiv.org/html/2610.12496#bib.bib2)\]studies oscillatory state\-space models and shows that undamped oscillators can be stable and expressive\. Section[4](https://arxiv.org/html/2610.12496#S4)shows that, under a linear readout of a small state, removing damping destroys rather than extends what can be read\. HiPPO and S4\[[4](https://arxiv.org/html/2610.12496#bib.bib8),[6](https://arxiv.org/html/2610.12496#bib.bib9)\]give principled alternatives to log\-spaced initialisation\.

#### Input\-dependent dynamics\.

Mamba\[[5](https://arxiv.org/html/2610.12496#bib.bib10)\], Mamba\-3\[[9](https://arxiv.org/html/2610.12496#bib.bib3)\], Kimi Linear\[[8](https://arxiv.org/html/2610.12496#bib.bib4)\]and Gated DeltaNet\-2\[[7](https://arxiv.org/html/2610.12496#bib.bib5)\]make the transition input\-dependent and give up the fixed convolution\. We place input dependence only in the read, which keeps the convolution but cannot create memory the write side does not keep\.

#### Recall and length extrapolation\.

Recall in efficient models is bounded by state size\[[1](https://arxiv.org/html/2610.12496#bib.bib6),[2](https://arxiv.org/html/2610.12496#bib.bib7)\]\. Length extrapolation in transformers depends on position encoding\[[11](https://arxiv.org/html/2610.12496#bib.bib11),[13](https://arxiv.org/html/2610.12496#bib.bib12)\]; our RoPE and learned\-position baselines show the same sensitivity\.

## 7Limitations and Conclusion

All results use 6M\-parameter character\-level models on WikiText\-2 with three seeds for the proposal\. We did not test the original paper’s transfer datasets or speed benchmarks, and on an associative\-recall task only one of three seeds of the proposal learned the task, so we draw no retrieval conclusion\. A free test\-time truncation also fixes the original’s collapse on this benchmark; the proposal’s advantage over it, retaining older history, is untested\.

Making the readout phase of ResonatorLM input\-dependent keeps the convolutional training path, adds 0\.2% parameters, matches the original at every training length, and removes its failure beyond the training length, which we trace to long\-memory heads accumulating more history than training exposed them to\. The benefit does not come from reaching further back: an ordinary token is lost in the oscillator state within a few steps whatever the damping, and the learned phase instead chooses, per character, which part of the state to read\.

## References

- \[1\]S\. Arora, S\. Eyuboglu, A\. Timalsina, I\. Johnson, M\. Poli, J\. Zou, A\. Rudra, and C\. Ré\(2023\)Zoology: measuring and improving recall in efficient language models\.Note:arXiv preprint arXiv:2312\.04927External Links:2312\.04927Cited by:[§6](https://arxiv.org/html/2610.12496#S6.SS0.SSS0.Px3.p1.1)\.
- \[2\]S\. Arora, S\. Eyuboglu, M\. Zhang, A\. Timalsina, S\. Alberti, D\. Zinsley, J\. Zou, A\. Rudra, and C\. Ré\(2024\)Simple linear attention language models balance the recall\-throughput tradeoff\.Note:arXiv preprint arXiv:2402\.18668External Links:2402\.18668Cited by:[§6](https://arxiv.org/html/2610.12496#S6.SS0.SSS0.Px3.p1.1)\.
- \[3\]A\. Chaudhury\(2026\)ResonatorLM: causal resonant field mixing for efficient long\-context language modeling\.Note:arXiv:2607\.05583v2External Links:2607\.05583Cited by:[§1](https://arxiv.org/html/2610.12496#S1.p2.1),[Table 1](https://arxiv.org/html/2610.12496#S5.T1)\.
- \[4\]A\. Gu, T\. Dao, S\. Ermon, A\. Rudra, and C\. Ré\(2020\)HiPPO: recurrent memory with optimal polynomial projections\.InAdvances in Neural Information Processing Systems,External Links:2008\.07669Cited by:[§1](https://arxiv.org/html/2610.12496#S1.p1.1),[§6](https://arxiv.org/html/2610.12496#S6.SS0.SSS0.Px1.p1.1)\.
- \[5\]A\. Gu and T\. Dao\(2023\)Mamba: linear\-time sequence modeling with selective state spaces\.Note:arXiv preprint arXiv:2312\.00752External Links:2312\.00752Cited by:[§1](https://arxiv.org/html/2610.12496#S1.p1.1),[§6](https://arxiv.org/html/2610.12496#S6.SS0.SSS0.Px2.p1.1)\.
- \[6\]A\. Gu, K\. Goel, and C\. Ré\(2022\)Efficiently modeling long sequences with structured state spaces\.InInternational Conference on Learning Representations,External Links:2111\.00396Cited by:[§1](https://arxiv.org/html/2610.12496#S1.p1.1),[§6](https://arxiv.org/html/2610.12496#S6.SS0.SSS0.Px1.p1.1)\.
- \[7\]A\. Hatamizadeh, Y\. Choi, and J\. Kautz\(2026\)Gated DeltaNet\-2: decoupling erase and write in linear attention\.Note:arXiv preprint arXiv:2605\.22791External Links:2605\.22791Cited by:[§1](https://arxiv.org/html/2610.12496#S1.p1.1),[§6](https://arxiv.org/html/2610.12496#S6.SS0.SSS0.Px2.p1.1)\.
- \[8\]Kimi Team Y\. Zhanget al\.\(2025\)Kimi Linear: an expressive, efficient attention architecture\.Note:arXiv preprint arXiv:2510\.26692External Links:2510\.26692Cited by:[§1](https://arxiv.org/html/2610.12496#S1.p1.1),[§6](https://arxiv.org/html/2610.12496#S6.SS0.SSS0.Px2.p1.1)\.
- \[9\]A\. Lahoti, K\. Y\. Li, B\. Chen, C\. Wang, A\. Bick, J\. Z\. Kolter, T\. Dao, and A\. Gu\(2026\)Mamba\-3: improved sequence modeling using state space principles\.Note:arXiv preprint arXiv:2603\.15569External Links:2603\.15569Cited by:[§1](https://arxiv.org/html/2610.12496#S1.p1.1),[§6](https://arxiv.org/html/2610.12496#S6.SS0.SSS0.Px2.p1.1)\.
- \[10\]S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\(2017\)Pointer sentinel mixture models\.InInternational Conference on Learning Representations,External Links:1609\.07843Cited by:[§4](https://arxiv.org/html/2610.12496#S4.SS0.SSS0.Px2.p1.1)\.
- \[11\]O\. Press, N\. A\. Smith, and M\. Lewis\(2022\)Train short, test long: attention with linear biases enables input length extrapolation\.InInternational Conference on Learning Representations,External Links:2108\.12409Cited by:[§6](https://arxiv.org/html/2610.12496#S6.SS0.SSS0.Px3.p1.1)\.
- \[12\]T\. K\. Rusch and D\. Rus\(2025\)Oscillatory state\-space models\.InInternational Conference on Learning Representations,External Links:2410\.03943Cited by:[§1](https://arxiv.org/html/2610.12496#S1.p1.1),[§6](https://arxiv.org/html/2610.12496#S6.SS0.SSS0.Px1.p1.1)\.
- \[13\]J\. Su, Y\. Lu, S\. Pan, A\. Murtadha, B\. Wen, and Y\. Liu\(2021\)RoFormer: enhanced transformer with rotary position embedding\.Note:arXiv preprint arXiv:2104\.09864External Links:2104\.09864Cited by:[§5](https://arxiv.org/html/2610.12496#S5.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2610.12496#S6.SS0.SSS0.Px3.p1.1)\.
- \[14\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2610.12496#S1.p1.1)\.