Comparing Transformers and Hybrid Models at the Token Level

# Comparing Transformers and Hybrid Models at the Token Level
Source: [https://arxiv.org/html/2606.20936](https://arxiv.org/html/2606.20936)
Yanhong Li Allen Institute for AI yanhongl@allenai\.org &William Merrill Allen Institute for AI willm@allenai\.org

###### Abstract

Hybrid language models that mix attention and recurrent layers have shown promise: theoretically, recurrent layers ameliorate the limitations of pure transformers on state tracking, and empirically, hybrids can outperform pure transformers in loss and downstream evaluations\(Waleffeet al\.,[2024](https://arxiv.org/html/2606.20936#bib.bib26); Merrillet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib11)\)\. Yet it remains unclear which data or capabilities drive these gains, and to what degree they reflect the theoretical advantages motivating hybrid models\. We address this question using the open weights from Olmo 3\(Olmoet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib10)\)and Olmo Hybrid\(Merrillet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib11)\): we compare the loss of a matched transformer and hybrid at the same target tokens under the same prefixes, stratifying the results by natural token tags, copy features, delimiter structure, and controlled synthetic probes\. The hybrid has lower loss on most tag families, but the gains are not uniform: they are largest for open\-class content words and smaller for many closed\-class function words\. Across prose, code, and markup, the hybrid’s loss advantage is larger on opening delimiters than on the corresponding closing delimiters, and nearly vanishes on repeatednn\-grams\. Synthetic probes show the same split: the hybrid is favored on pronoun\-memory and entity\-tracking tasks, whereas the transformer is favored on bracket\-matching tasks that require choosing closing delimiters\. These patterns suggest that the recurrent layers in hybrids improve predictions that leverage the semantic state of a document, whereas attention helps on tokens predictable bynn\-gram copying or syntactic bracket matching\. We conclude with proof\-of\-concept filtered evaluations showing how token\-level decompositions can sharpen pretraining diagnostics for hybrid architectures\.

## 1Introduction

Hybrid language models that mix attention with recurrent sequence layers have recently challenged the attention\-only transformer as the default architecture for large\-scale language modeling\. Theoretically, hybrid models are motivated by the complementary expressivity of attention and recurrent layers: attention allows retrieving information from past tokens, which is useful for copying and syntactic tasks like bracket matching, while recurrent layers offer advantages for constructing and updating latent state in ordered state\-tracking computations\(Merrillet al\.,[2024](https://arxiv.org/html/2606.20936#bib.bib21); Grazziet al\.,[2025](https://arxiv.org/html/2606.20936#bib.bib7)\)\. Theoretically, hybrid models inherit both of these strengths and, empirically, they can outperform transformers with the same pretraining budget in loss and downstream benchmarks\(Waleffeet al\.,[2024](https://arxiv.org/html/2606.20936#bib.bib26); Merrillet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib11)\)\. While these empirical results are impressive, it remains unclear what finegrained predictions are improved by adding hybrid layers to a model, or to what degree these relate to the theoretical advantages of hybrid models\. In this work, we therefore aim to understand the finegrained prediction differences between transformers and hybrid models*at the token level*, asking the following two questions:

1. 1\.*What kinds of tokens and prediction contexts do hybrid models predict better than transformers?*More concretely, which individual token occurrences account for the lower average loss of a hybrid model relative to a matched transformer?
2. 2\.*Do those token occurrences match the theoretical advantages of the component architectures?*More specifically, do hybrid\-favored tokens look like the expressivity advantages associated with recurrence, such as ordered state tracking, while transformer\-favored tokens look like the advantages associated with attention, such as recall from the visible prefix?

We study these questions by comparing two released 7B\-scale models from the same recipe family, Olmo 3 7B and Olmo Hybrid 7B\(Olmoet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib10); Merrillet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib11)\)\. For a target positioniiwith prefixx<ix\_\{<i\}and observed tokenxix\_\{i\}, we compute paired token losses

ℓiTr=−log⁡pTr​\(xi∣x<i\),ℓiHyb=−log⁡pHyb​\(xi∣x<i\),\\ell\_\{i\}^\{\\mathrm\{Tr\}\}=\-\\log p\_\{\\mathrm\{Tr\}\}\(x\_\{i\}\\mid x\_\{<i\}\),\\qquad\\ell\_\{i\}^\{\\mathrm\{Hyb\}\}=\-\\log p\_\{\\mathrm\{Hyb\}\}\(x\_\{i\}\\mid x\_\{<i\}\),and the paired gapΔi=ℓiTr−ℓiHyb\\Delta\_\{i\}=\\ell\_\{i\}^\{\\mathrm\{Tr\}\}\-\\ell\_\{i\}^\{\\mathrm\{Hyb\}\}\. ThusΔi\>0\\Delta\_\{i\}\>0means that the hybrid assigns higher probability, equivalently lower NLL, to the observed next token at the same prefix\. This paired quantity lets us move from the average question “which model has lower loss?” to the token\-level question “which prediction events produce the gain?”

Our first analysis is observational\. We computeΔi\\Delta\_\{i\}over prose, code, and markup, then align target tokens to surface tags: POS tags for prose and source\-level categories such as identifiers, strings, comments, text nodes, attributes, commands, brackets, and tags for structured text\. We report raw tag\-stratified means, which describe the absolute hybrid advantage on the actual corpus positions, and regression robustness checks, which ask whether the same patterns remain after controlling for difficulty, frequency, position, subword status, and local reuse\. The regression analysis is complementary to the raw summaries: it asks whether the tag\-level patterns persist after controlling for difficulty, frequency, position, subword status, and local reuse\.

The natural\-token results are non\-uniform\. The hybrid has lower loss on most token families, but the advantage is largest on open\-class content\-bearing categories\. In prose, content words have a larger raw gap than function words, and the aggregate content–function contrast remains after controls\. In structured domains, hybrid\-favored categories include identifiers, strings, comments, text nodes, attribute values, and commands\. By contrast, the hybrid advantage is smaller on closing delimiters and rigid formatting tokens, and it approaches zero on long repeatednn\-grams\. Thus, while the hybrid wins overall—especially on open\-class words—the transformer remains competitive, and sometimes favored, in contexts where attention is the important primitive: retrieving material already present in the visible prefix or satisfying an already\-open structural obligation\.

Our second analysis uses controlled synthetic probes to separate delayed information from the type of computation required at the target\. In a pronoun\-memory probe, the model sees people with roles and later must choose the pronoun for the person filling a queried role\. In an entity\-tracking probe, it sees entities bound to attributes and later must choose which entity has a queried attribute\. In a structural\-closure probe, it sees an opened bracket or tag region and later must predict the required closing token\. The first two probes require readout of a maintained role or entity–attribute binding and favor the hybrid\. The closure probe also involves delayed information, but the answer is already determined by an opener visible in the prefix, and it favors the transformer\. Thus the relevant distinction is not simply short versus long dependency, but whether the target is primarily a state\-conditioned choice, a visible\-prefix copy, or a closure decision\.

These results point to two different follow\-on uses of the token\-level analysis\. The first is for designing controlled state\-tracking benchmarks\. Existing state\-tracking tasks often ask a model to update a small, fixed set of symbols according to a sequence of instructions and then report the final symbol or value\. Such closed\-world tasks are useful because they isolate ordered update and readout, but once architectures solve the fixed\-slot setting, they become less informative for distinguishing future recurrent layers\. Our results suggest a more demanding, open\-world extension: discourse state tracking\. In these tasks, the input can introduce new people, objects, variables, or document regions over time, change their attributes or relations, and later require a context\-dependent prediction, such as which entity, value, or content word should come next\. This framing keeps the connection to state tracking while moving from fixed symbolic slots to the growing, relational state needed in natural text, code, and markup\. Controlled discourse\-state\-tracking tasks would therefore provide a more demanding testbed for developing linear\-RNN and hybrid sequence layers\.

The second use is for comparing architectures during pretraining experiments\. In 1B\-scale development runs comparing a Transformer, a Hybrid, and a Pure RNN, aggregate validation loss compresses distinct regimes into a single number\. Filtered token losses could be a useful way to compare and contrast architectures in this setting: losses on hybrid\-favored non\-copy tokens separate architectures more sharply, while a copy\-only filter exposes the complementary regime in which attention\-based models outperform the Pure RNN on visible\-prefix reuse\. Because these filters are computed from the same per\-token NLL as standard validation, they add little overhead while providing a more capability\-resolved view of training progress\. Reporting filtered token losses alongside aggregate validation loss can therefore show not only whether a design improves overall perplexity, but which predictive capabilities it improves or sacrifices\.

## 2Expressivity Background and Empirical Hypotheses

Expressivity theory gives a useful way to separate the computations that can be hidden inside next\-token prediction\. We organize the background by architectural primitive: attention, recurrence, and their combination in hybrid models\.

Transformer expressivity: copy/recall and structural matching\.Attention gives transformers a direct mechanism for selecting positions in the visible prefix\. Under standard formalizations, this lets transformers express recall\-style computations such asnn\-gram retrieval, and it also supports many bracket\-matching or structural\-matching problems by making the relevant earlier position or opener accessible\(Weisset al\.,[2021](https://arxiv.org/html/2606.20936#bib.bib32); Yaoet al\.,[2023](https://arxiv.org/html/2606.20936#bib.bib31)\)\. Thus attention should be especially useful when the next token can be recovered by reusing already\-visible material or by matching an explicit structural opener:

1. \(1\)John works for the National Hamburger Association of America\. The National Hamburger Association ofAmerica…\\ldots
2. \(2\)\( \[ \] \[ \{ \( \) \} \]\)\.

The first example asks for copy/recall from the visible prefix, while the second asks for structural matching against a visible opener\. In contrast, copy/recall is a known limitation of pure recurrent models with bounded state: copying or recalling arbitrary prefix information can require storing more information than a fixed\-size recurrent state can retain\(Aroraet al\.,[2024](https://arxiv.org/html/2606.20936#bib.bib23); Jelassiet al\.,[2024](https://arxiv.org/html/2606.20936#bib.bib9); Merrillet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib11)\)\.

RNN expressivity: ordered state tracking\.The complementary limitation of fixed\-depth transformers is ordered state composition\. Under standard fixed\-depth and log\-precision assumptions, transformer next\-token predictors are contained in low\-depth threshold\-circuit classes such as𝖳𝖢0\\mathsf\{TC\}^\{0\}; consequently, they cannot express general𝖭𝖢1\\mathsf\{NC\}^\{1\}\-complete ordered state\-composition problems unless𝖳𝖢0=𝖭𝖢1\\mathsf\{TC\}^\{0\}=\\mathsf\{NC\}^\{1\}\(Merrill and Sabharwal,[2023](https://arxiv.org/html/2606.20936#bib.bib3); Chiang,[2025](https://arxiv.org/html/2606.20936#bib.bib5)\)\. In contrast, modern linear RNNs with sufficiently expressive transition matrices, including DeltaNet/GDN variants with negative eigenvalues, can represent such state\-tracking computations\(Merrillet al\.,[2024](https://arxiv.org/html/2606.20936#bib.bib21); Grazziet al\.,[2025](https://arxiv.org/html/2606.20936#bib.bib7); Merrillet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib11)\)\. These results motivate the complementary hypothesis that predictions requiring ordered state updates should favor models with recurrent layers over pure transformers\. A simple program\-state example is:

1. \(3\)a,b,c = 1,2,3; a,c = c,a; assert a ==3\.

Hybrid expressivity: composing recall with state\.Hybrid models combine attention layers, which support recall from the visible prefix, with recurrent layers, which support ordered state updates\. This combination should allow them to handle contexts like[Items˜1](https://arxiv.org/html/2606.20936#S2.I1.i1),[2](https://arxiv.org/html/2606.20936#S2.I1.i2)and[3](https://arxiv.org/html/2606.20936#S2.I2.i3), where a pure RNN or a pure transformer is missing one of the relevant primitives\. Moreover, hybrids are not merely the union of two independent capabilities:Merrillet al\.\([2026](https://arxiv.org/html/2606.20936#bib.bib11)\)show that a GDN–attention hybrid can solve*state\-based recall*, where a model must track updates to a pointer and then use the resulting pointer to retrieve a value from the prefix, while neither a pure transformer nor a pure GDN model can express the full problem under the standard assumptions\. A code\-like example is:

1. \(4\)bits = \[0,1,0,0,…\\ldots\]; a,b,c = 3,1,2; a,c = c,a; assert bits\[a\] ==0\.

This expressivity picture gives the empirical hypotheses for the rest of the paper\. Next\-token prediction is not a single homogeneous computation: we treat each prefix as inducing a latent discourse/program state, and we view each target token as a query against, or an update to, some aspect of that state\. Some targets are recoverable from the visible prefix; others require ordered state construction and state\-conditioned readout; still others compose both\. We therefore expect attention\-heavy transformers to be competitive on visible\-prefix reuse and structural closure, recurrent layers to help on state\-conditioned predictions, and hybrids to be strongest when natural language, code, or markup requires both\. The remainder of the paper tests where these regimes appear in ordinary next\-token prediction across natural language, code, and markup\.

## 3Empirical Methodology

We ask which*individual next\-token predictions*account for the loss gap between a hybrid model and a matched transformer\. The basic unit of analysis is one target position\. For a packed token sequencex1:Lx\_\{1:L\}and target positionii, both models are evaluated on the same prefixx<ix\_\{<i\}and the same observed target tokenxix\_\{i\}\. Let

ℓiTr=−log⁡pTr​\(xi∣x<i\),ℓiHyb=−log⁡pHyb​\(xi∣x<i\),\\ell\_\{i\}^\{\\mathrm\{Tr\}\}=\-\\log p\_\{\\mathrm\{Tr\}\}\(x\_\{i\}\\mid x\_\{<i\}\),\\qquad\\ell\_\{i\}^\{\\mathrm\{Hyb\}\}=\-\\log p\_\{\\mathrm\{Hyb\}\}\(x\_\{i\}\\mid x\_\{<i\}\),whereTr\\mathrm\{Tr\}denotes the transformer andHyb\\mathrm\{Hyb\}denotes the hybrid model\. We define the paired token\-level loss gap111BecauseΔi\\Delta\_\{i\}is a log\-probability difference, a mean gap ofΔ¯\\bar\{\\Delta\}nats corresponds to a geometric\-mean probability ratio ofexp⁡\(Δ¯\)\\exp\(\\bar\{\\Delta\}\)in favor of the hybrid\. For example,0\.040\.04nats is about a4\.1%4\.1\\%probability ratio\. It is not a log\-odds ratio\.

Δi=ℓiTr−ℓiHyb=log⁡pHyb​\(xi∣x<i\)−log⁡pTr​\(xi∣x<i\)\.\\Delta\_\{i\}=\\ell\_\{i\}^\{\\mathrm\{Tr\}\}\-\\ell\_\{i\}^\{\\mathrm\{Hyb\}\}=\\log p\_\{\\mathrm\{Hyb\}\}\(x\_\{i\}\\mid x\_\{<i\}\)\-\\log p\_\{\\mathrm\{Tr\}\}\(x\_\{i\}\\mid x\_\{<i\}\)\.ThusΔi\>0\\Delta\_\{i\}\>0means that, at positionii, the hybrid assigns higher probability to the observed next token than the transformer\. All natural\-token analyses in this section aggregate these per\-position quantities\.

Models, domains and evaluation\.We compare two released 7B\-scale models from the same recipe family:Olmo 3 7BandOlmo Hybrid 7B\(Olmoet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib10); Merrillet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib11)\)\. This pair is closely matched in tokenizer, data mixture, and training recipe, making the per\-token gap primarily reflect the architectural difference \(the sequence mixer\)\. We evaluate on prose and structured text spanning natural language, code, and markup \(prose: PG\-19, News, Wikipedia, essay, textbooks and scientific papers; structured: Python, HTML,LaTeX\)\. Text is packed into length\-L=8192L\{=\}8192sequences and we compute next\-token NLL at every position\. Unless otherwise stated, all 7B token\-level and synthetic analyses use the final released checkpoint pair\. Additional preprocessing and checkpoint details are in Appendix[A](https://arxiv.org/html/2606.20936#A1)\.

Token tagging and alignment\.After computingΔi\\Delta\_\{i\}for every LM target position, we assign surface tags using the same two\-step procedure across domains\. First, we tag spans in the original source text\. Second, we align those source\-level tags to LM target tokens by character\-span overlap between the decoded LM token and the tagged source span\. Prose and structured domains differ in the source\-level tagger and tag inventory, and share the alignment procedure\. For prose, the source spans are words tagged with the Brown POS tagset\. For Python, HTML, andLaTeX, the source spans come from lightweight tokenizers or parsers, with categories such as identifiers, strings, comments, delimiters, tags, attributes, commands, and text nodes\. When an LM token overlaps multiple source\-level tags, we use multi\-tag attribution, so the sameΔi\\Delta\_\{i\}contributes to each overlapping tag’s summary\. Full tag taxonomies and alignment rules are in Appendix[A\.2](https://arxiv.org/html/2606.20936#A1.SS2)and Appendices[B\.1](https://arxiv.org/html/2606.20936#A2.SS1)–[B\.2](https://arxiv.org/html/2606.20936#A2.SS2)\. We also record each LM token’s word\-position type: whole word, prefix, middle subword, or suffix\.

Analysis I: Tag\-stratified raw summaries\.Raw here means unadjusted: we average the paired loss gap over the observed corpus positions assigned to a tag, without reweighting those positions or controlling for other variables\. This answers the descriptive question: among the actual corpus positions with a given tag, how much lower is the hybrid’s NLL than the transformer’s? For a tag or position setτ\\tau, letℐτ\\mathcal\{I\}\_\{\\tau\}be the set of target positions assigned that tag\. We report

Δ^​\(τ\)=1\|ℐτ\|​∑i∈ℐτΔi=𝔼​\[Δi∣i∈ℐτ\]\.\\widehat\{\\Delta\}\(\\tau\)=\\frac\{1\}\{\|\\mathcal\{I\}\_\{\\tau\}\|\}\\sum\_\{i\\in\\mathcal\{I\}\_\{\\tau\}\}\\Delta\_\{i\}\\;=\\;\\mathbb\{E\}\[\\Delta\_\{i\}\\mid i\\in\\mathcal\{I\}\_\{\\tau\}\]\.For example,τ\\taucan beNoun,Open Bracket, orHTML open tag\. If an LM token overlaps multiple source\-level tags, that position belongs to each correspondingℐτ\\mathcal\{I\}\_\{\\tau\}, so its token\-level gap contributes to each relevant raw summary\.

Analysis II: Regression robustness check\.Raw tag means provide the main descriptive summary: they show the hybrid–transformer gap on the actual positions belonging to each tag\. However, a raw tag mean can be confounded by other properties of those positions\. Tags are correlated with difficulty, frequency, position, subword status, and local reuse, and these covariates can affectΔi\\Delta\_\{i\}independently of tag membership\. For example, content and function categories differ substantially in token frequency\. We therefore fit linear regressions that control for these correlates:

Δi∼domaini\+tagi\+wposi\+relposi\+ℓ¯i\+ℓ¯i2\+∑k∈\{1,2,3,4\}copyk,i\+log⁡prevdisti\+log⁡freq​\(xi\)\.Δipaired NLL gap\-ℓTriℓHybi\(\>0= hybrid better\)domainprose source fixed effectstageither coarse POS family or aggregate word class, depending on the modelwpossubword position in word \(whole/prefix/middle/suffix\)relposrelative position in packed sequenceℓ¯imean NLL; difficulty proxy \(¯ℓi2allows curvature\)copykprefix\-reuse features for=k1,2,3,4;copy1,iis same\-token reuse \(the target token type appeared earlier in the prefix\), andcopyk,ifor≥k2indicates that the target completes a repeatedk\-gramprevdistdistance to previous occurrence of the same token type \(log\-scaled\)freqempirical target\-token type frequency in the pooled regression sample\\small\\vskip\-1\.0pt\\begin\{array\}\[\]\{@\{\}l@\{\\hspace\{1em\}\}l@\{\}\}\\begin\{aligned\} \\Delta\_\{i\}\\;\\sim\\;&\\textsc\{domain\}\_\{i\}\+\\textsc\{tag\}\_\{i\}\+\\textsc\{wpos\}\_\{i\}\\\\ &\+\\textsc\{relpos\}\_\{i\}\+\\bar\{\\ell\}\_\{i\}\+\\bar\{\\ell\}\_\{i\}^\{2\}\\\\ &\+\\sum\_\{k\\in\\\{1,2,3,4\\\}\}\\textsc\{copy\}\_\{k,i\}\\\\ &\+\\log\\textsc\{prevdist\}\_\{i\}\+\\log\\textsc\{freq\}\(x\_\{i\}\)\.\\end\{aligned\}\\hfil\\hskip 9\.24994pt&\{\\scriptsize\\begin\{tabular\}\[\]\{@\{\}l p\{28em\}@\{\}\}$\\Delta\_\{i\}$&paired NLL gap $\\ell^\{\\mathrm\{Tr\}\}\_\{i\}\-\\ell^\{\\mathrm\{Hyb\}\}\_\{i\}$ \($\>0$ = hybrid better\)\\\\ \{domain\}&prose source fixed effects\\\\ \{tag\}&either coarse POS family or aggregate word class, depending on the model\\\\ \{wpos\}&subword position in word \(whole/prefix/middle/suffix\)\\\\ \{relpos\}&relative position in packed sequence\\\\ $\\bar\{\\ell\}\_\{i\}$&mean NLL; difficulty proxy \($\\bar\{\\ell\}\_\{i\}^\{2\}$ allows curvature\)\\\\ \{copy\}$\{\}\_\{k\}$&prefix\-reuse features for $k=1,2,3,4$; $\\textsc\{copy\}\_\{1,i\}$ is same\-token reuse \(the target token type appeared earlier in the prefix\), and $\\textsc\{copy\}\_\{k,i\}$ for $k\\geq 2$ indicates that the target completes a repeated $k$\-gram\\\\ \{prevdist\}&distance to previous occurrence of the same token type \(log\-scaled\)\\\\ \{freq\}&empirical target\-token type frequency in the pooled regression sample\\\\ \\end\{tabular\}\}\\vskip\-1\.0pt\\end\{array\}\(1\)
We fit two instantiations of Equation[1](https://arxiv.org/html/2606.20936#S3.E1)\. The first is thecoarse\-tag model, wheretagis the full coarse Brown\-POS family; this model gives the detailed adjusted effects for nouns, verbs, auxiliaries, brackets, punctuation, and other coarse categories\. The second is theaggregate word\-class model, wheretagis replaced by the three\-way labelContent/Function/Other;222Contentincludes open\-class lexical categories such as nouns, main verbs, adjectives, and adverbs\.Functionincludes closed\-class grammatical categories such as determiners, prepositions, conjunctions, pronouns, auxiliaries, modals, wh\-categories, particles, and infinitivalTO\.Otherincludes punctuation, brackets, symbols, numerals, and remaining tags\.this model gives the aggregate rows in Figure[2](https://arxiv.org/html/2606.20936#S4.F2)\. We fit the aggregate model separately\. Both models include the same controls\.

The raw summaries and regression effects answer complementary questions: raw gaps show whether the hybrid is better or worse on a token family in absolute terms, while regression effects show whether that family is more or less hybrid\-favored than expected after controlling for confounds\.

Pronoun
memoryStructural
closureEntity
trackingLiam is the violinist\.
Naomi is the pilot\.
…ddfiller tokens …
the violinist reviewed the report, and<header\>
counter \+= 1
…ddfiller tokens …Julia carried the orange notebook\. Sofia carried the green folder\.
…ddfiller tokens … Q: Who carried the green folder?
\(A\) Sofia \(B\) Julia Answer:he</header\>Sofiavs\. sheNLL on closervs\. JuliaFigure 1:Synthetic probe examples\.We vary the distanceddbetween an antecedent/opener and a scored target token\.Analysis III: Controlled synthetic probes\.The natural\-token analyses are observational: they reveal where the gap appears in real text, but a surface tag does not directly specify the decision the model must make at the scored position\. We therefore add three controlled synthetic probe families \(Figure[1](https://arxiv.org/html/2606.20936#S3.F1)\):

- •Pronoun memory\. The prompt introduces two people with different genders and roles, inserts filler, then refers to one role and scores the matching pronoun, e\.g\.,*he*versus*she*\.
- •Entity tracking\. The prompt binds two entities to different attributes, inserts filler, then asks which entity has a queried attribute\.
- •Structural closure\. The prompt opens a bracketed or tagged region, inserts filler, then scores the required closing token\.

For each family, we keep the surface template and filler distribution fixed while varying the distanced∈\{32,64,128,256,512,1024\}d\\in\\\{32,64,128,256,512,1024\\\}between the relevant earlier cue and the scored target token\.

For the first two probes, we score them contrastively using accuracy𝟏​\[m\>0\]\\mathbf\{1\}\[m\>0\]and margin

m=log⁡p​\(y\+∣c\)−log⁡p​\(y−∣c\),m=\\log p\(y^\{\+\}\\mid c\)\-\\log p\(y^\{\-\}\\mid c\),wherey\+y^\{\+\}is the correct token,y−y^\{\-\}is a matched distractor, andccis the shared prefix\.Structural closureinstead tests visible structural closure, so we report NLL on the closing token,

ℓclose=−log⁡p​\(yclose∣c\)\.\\ell\_\{\\mathrm\{close\}\}=\-\\log p\(y\_\{\\mathrm\{close\}\}\\mid c\)\.

## 4Empirical Results

We organize the results into two parts\. Section[4\.1](https://arxiv.org/html/2606.20936#S4.SS1)analyzes natural tokens from prose, code, and markup; these analyses show where the hybrid–transformer gap appears in real data and distinguish absolute raw gaps from controlled regression effects\. Section[4\.2](https://arxiv.org/html/2606.20936#S4.SS2)uses controlled synthetic probes to test whether the same split appears when we directly manipulate the target type and antecedent distance\.

### 4\.1Natural\-token analysis

![Refer to caption](https://arxiv.org/html/2606.20936v1/x1.png)Figure 2:Raw and adjusted tag effects in prose\.Left panels show raw paired loss gaps, where positive values mean the hybrid has lower NLL on that token family\. Right panels show regression\-adjusted effects relative to the global mean from the regressions\. Top panels use the full coarse POS taxonomy; bottom panels use the separately fitted three\-way aggregate model \(content/function/other\)\.![Refer to caption](https://arxiv.org/html/2606.20936v1/x2.png)Figure 3:Tag vocabulary size is associated with the prose tag effects\.Each point is a coarse prose tag from Figure[2](https://arxiv.org/html/2606.20936#S4.F2), marked by aggregate word class\. The x\-axis islog⁡\|𝒱τ\|\\log\|\\mathcal\{V\}\_\{\\tau\}\|, where𝒱τ\\mathcal\{V\}\_\{\\tau\}is the set of distinct target\-token types observed with tagτ\\tau\. Left: raw paired loss gapΔ¯​\(τ\)\\bar\{\\Delta\}\(\\tau\)\. Right: regression\-adjusted tag effect relative to the global mean\.Hybrid advantage across token tags is largest for open\-class content words\.[Figure˜2](https://arxiv.org/html/2606.20936#S4.F2)shows that the hybrid is better on most prose tag families, but not uniformly so\. In the aggregate raw panel, content words have the largest hybrid advantage \(0\.03840\.0384nats\) versus function words \(0\.02380\.0238nats\), a difference of0\.01460\.0146nats \(about61%61\\%larger\)\. This ordering persists after controls \(difficulty, token/subword position, sequence position, local reuse, previous\-token distance, and token frequency\): content stays above the global mean, while function falls below it\. The regression gives a stricter test of the raw trend: after matching positions by difficulty, token frequency, subword status, sequence position, and local reuse, do the same tag\-level observations still point in the same direction? For the main pattern, they do\. The aggregate content–function contrast survives the controls, and many open\-class categories remain more hybrid\-favored than closed\-class or highly constrained categories\. The effects for individual function tags are more heterogeneous, but several function\-like categories \(e\.g\., auxiliaries, wh\-words,TO, pronouns\) still appear among the least hybrid\-favored\.

The aggregate contrast is also closely related to the open\-/closed\-class distinction: function words tend to come from small, closed inventories, whereas content words tend to be open class\. We therefore ask whether the tag\-level effects vary continuously with tag vocabulary size\.[Figure˜3](https://arxiv.org/html/2606.20936#S4.F3)plotslog⁡\|𝒱τ\|\\log\|\\mathcal\{V\}\_\{\\tau\}\|, the number of distinct target token types realized under tagτ\\tau, against both the raw and adjusted tag effects\. Open\-class tags \(large vocabularies\) tend to occupy the more hybrid\-favored region in both the raw and adjusted views\. After controls the slope is modest, which is expected because frequency, difficulty, and reuse explain part of the same variation; nevertheless, its positive direction agrees with the aggregate content–function result\. The weaker adjusted slope should not be read as eliminating the closed\-class pattern: several closed\-class/function\-like tags still have strongly negative adjusted effects, even though the overall vocabulary\-size relationship is less steep after controls\. A notable exception isExistential, which remains unusually hybrid\-favored despite a small choice set \(also under the controlled view\)\. A plausible explanation is that existential*there*often signals a discourse\-introducing construction \(“there is/are…\\ldots”\), so the decision to use it can depend on latent state updates even though the surface choice set is small\.

![Refer to caption](https://arxiv.org/html/2606.20936v1/x3.png)Figure 4:Brackets and repeatednn\-grams\.A:Open and close bracket raw gaps across domains; line segments show the open–close difference\.B:Raw paired loss gaps for repeated\-token events from repeated 1\-grams through 16\-grams\. Positive raw gaps mean lower hybrid NLL\.C:Implied repetition effects from an extension of Equation[1](https://arxiv.org/html/2606.20936#S3.E1)withcopy1–copy16\. Negative adjusted effects mean that repetition reduces the hybrid advantage relative to comparable non\-copy positions\.Opening vs\. closing brackets: the hybrid advantage is reduced on structural closure\.If the bracket pattern is about the predictive role of the target rather than the bracket character itself, then openers and closers should behave differently: openers begin a new region or scope, whereas closers satisfy an obligation already established by the visible prefix\.[Figure˜4](https://arxiv.org/html/2606.20936#S4.F4)A tests this prediction across domains\. Across all seven domains \(prose, Python, HTML,LaTeX\),*opening*brackets are consistently more hybrid\-favored than the corresponding*closing*brackets; the gap is largest in prose and remains visible in structured text\. This matters because opening and closing delimiters have similar surface form but different predictive roles\. Openers often initiate a new region/scope \(a state update\), whereas closers typically satisfy an already\-established structural obligation \(closure\)\. The cross\-domain consistency therefore supports an interpretation in terms of what computation is queried at the target, not the bracket character per se\.

Hybrid advantage nearly disappears on repeatednn\-grams\.[Figure˜4](https://arxiv.org/html/2606.20936#S4.F4)B–C isolates repeated spans, an observable proxy for prefix\-based copying in a weak sense rather than direct evidence of a copying mechanism\. We call a target a repeatednn\-gram event when the contiguousnn\-token sequence ending at the target has appeared earlier in the prefix\. For example, in “a b c d a bc”, the underlined target completes a repeated 3\-gram, “a b c”\.[Figure˜4](https://arxiv.org/html/2606.20936#S4.F4)B reports raw paired loss gaps on such targets forn=1,…,16n=1,\\ldots,16\. A repeated 1\-gram only means that the target token type appeared earlier in the prefix, so it captures ordinary token reuse rather than a clean repeated\-span continuation\. Asnngrows, the event becomes a cleaner repetition proxy: if a longer span appears twice, the target is increasingly determined by continuing material already visible in the prefix\. Consistent with this interpretation, the raw hybrid advantage shrinks rapidly withnnand approaches zero for long repeated spans\. In these settings, the repeated continuation in the visible prefix provides a strong prediction, so the hybrid no longer has a measurable absolute advantage\.

The regression view in[Figure˜4](https://arxiv.org/html/2606.20936#S4.F4)C makes the same point under controls\. We extend Equation[1](https://arxiv.org/html/2606.20936#S3.E1)by replacingcopy1–copy4withcopy1–copy16\. Because repeated\-nnevents nest \(a repeatednn\-gram also activates all shorter copy indicators\), we plot the*implied*repetition effect at lengthnnby summing active copy coefficients throughnn\. These implied effects are consistently transformer\-shifting \(negative\), with uncertainty growing for largennbecause long repeated spans are rare and the nested indicators are highly correlated\. Non\-repetition control coefficients are reported in Appendix[D](https://arxiv.org/html/2606.20936#A4)\.

### 4\.2Controlled synthetic analysis

![Refer to caption](https://arxiv.org/html/2606.20936v1/x4.png)Figure 5:Controlled probes at the final matched checkpoint\.Pronoun memory and entity tracking are scored by contrastive accuracy and log\-probability margin; structural closure is scored by NLL on the closing token\. Higher is better for accuracy and margin; lower is better for NLL\.[Figure˜5](https://arxiv.org/html/2606.20936#S4.F5)provides the controlled counterpart to the natural\-token split by fixing templates and varying the dependency distance and target type\. Pronoun memory favors the hybrid \(larger contrastive margins and typically higher accuracy\)\. Entity tracking shows a stronger separation: the hybrid stays above chance across distances, while the transformer dips below chance at intermediate distances and yields negative margins there\. Structural closure reverses sign: when the target is a closer whose opener is already present, the transformer attains lower NLL at every distance\. Thus delay alone is not the relevant variable: all three probes require information from earlier context, but only the state\-readout probes favor the hybrid\. This is consistent with the theoretical expectation that recurrent layers help most when prediction requires maintaining and querying latent state, whereas attention remains competitive or better when the target is a closure token determined by a visible opener\.

### 4\.3Summary of empirical findings

Across observational tagging and controlled probes, we find a consistent, non\-uniform token\-level structure in the hybrid–transformer gap: \(i\) the hybrid advantage is broad but especially pronounced on open\-class, content\-bearing choices; the effect is weaker and more heterogeneous for closed\-class/function categories; \(ii\) in contrast, the transformer matches or exceeds the hybrid on structural closure: closing delimiters are less hybrid\-favored than openers across domains, and the controlled closure probe favors the transformer; and \(iii\) finally, the hybrid advantage collapses on explicit visible\-prefix reuse \(repeatednn\-grams\), and repetition features shift the gap toward the transformer even after controls\.

## 5Interpretation: Hybrid Gains and Discourse\-State Tracking

We now give a preliminary theoretical account of the empirical split in §[4](https://arxiv.org/html/2606.20936#S4)\.

#### Setup\.

Fix a target positionttand prefixx<tx\_\{<t\}\. Letτ\\taudenote the target token’s coarse local class, such as a POS tag, delimiter class, or markup/code token type\. For any such class, define

𝒱τ=\{x∈𝒱:x​can realize class​τ\}\.\\mathcal\{V\}\_\{\\tau\}=\\\{x\\in\\mathcal\{V\}:x\\text\{ can realize class \}\\tau\\\}\.Since our empirical strata are assigned in the analysis, we treatτ\\tauas an oracle local\-slot variable\. Thus, when analyzing a fixed classτ\\tau, we use class\-conditional next\-token distributions; equivalently, the domain ofXtX\_\{t\}is restricted to𝒱τ\\mathcal\{V\}\_\{\\tau\}\. To keep the notation local to this fixed target and class, write

pτ⋆\(⋅\)=p⋆\(Xt=⋅∣X<t=x<t,Xt∈𝒱τ\)p^\{\\star\}\_\{\\tau\}\(\\cdot\)=p^\{\\star\}\\\!\\left\(X\_\{t\}=\\cdot\\mid X\_\{<t\}=x\_\{<t\},\\,X\_\{t\}\\in\\mathcal\{V\}\_\{\\tau\}\\right\)for the ideal full\-prefix next\-token distribution under the data\-generating process, conditional on the target class\. Letϕ\\phibe a feature map of the prefix\. A reduced predictor that can use only the information summarized byϕ​\(x<t\)\\phi\(x\_\{<t\}\), together with the same class restriction, predicts

pϕ,τ\(⋅\)=pϕ\(Xt=⋅∣ϕ\(x<t\),Xt∈𝒱τ\)\.p\_\{\\phi,\\tau\}\(\\cdot\)=p\_\{\\phi\}\\\!\\left\(X\_\{t\}=\\cdot\\mid\\phi\(x\_\{<t\}\),\\,X\_\{t\}\\in\\mathcal\{V\}\_\{\\tau\}\\right\)\.Going forward,pϕ,τp\_\{\\phi,\\tau\}denotes the best predictor under that feature restriction and class conditioning\. For an actual architecture,ϕ\\phican be viewed as the class of prefix features that the sequence mixer makes easy to express: attention can expose visible\-prefix retrieval and structural\-matching features, while recurrent layers can expose updated latent\-state features\. The reducible loss incurred by usingpϕ,τp\_\{\\phi,\\tau\}to approximatepτ⋆p^\{\\star\}\_\{\\tau\}is

ℰϕ,τ​\(t\)=H​\(pτ⋆,pϕ,τ\)−H​\(pτ⋆\)=DKL​\(pτ⋆∥pϕ,τ\),\\mathcal\{E\}\_\{\\phi,\\tau\}\(t\)=H\(p^\{\\star\}\_\{\\tau\},p\_\{\\phi,\\tau\}\)\-H\(p^\{\\star\}\_\{\\tau\}\)=D\_\{\\mathrm\{KL\}\}\\\!\\left\(p^\{\\star\}\_\{\\tau\}\\,\\\|\\,p\_\{\\phi,\\tau\}\\right\),where, for distributions supported on𝒱τ\\mathcal\{V\}\_\{\\tau\},

H​\(p\)=−∑x∈𝒱τp​\(x\)​log⁡p​\(x\),H​\(p,q\)=−∑x∈𝒱τp​\(x\)​log⁡q​\(x\)\.H\(p\)=\-\\sum\_\{x\\in\\mathcal\{V\}\_\{\\tau\}\}p\(x\)\\log p\(x\),\\qquad H\(p,q\)=\-\\sum\_\{x\\in\\mathcal\{V\}\_\{\\tau\}\}p\(x\)\\log q\(x\)\.A feature map is predictive enough for a token family when this class\-conditional KL divergence is small on that family\.

First: visible\-prefix sufficiency explains why the gap closes on recall and closure\.The common structure in recall and closure is simple: within the relevant local classτ\\tau, suppose a prefix featureϕ\\philets the reduced predictor determine the class\-conditional next\-token distribution\. Then, in the deterministic idealization,

pϕ,τ=pτ⋆and henceDKL​\(pτ⋆∥pϕ,τ\)=0\.p\_\{\\phi,\\tau\}=p^\{\\star\}\_\{\\tau\}\\qquad\\text\{and hence\}\\qquad D\_\{\\mathrm\{KL\}\}\(p^\{\\star\}\_\{\\tau\}\\\|p\_\{\\phi,\\tau\}\)=0\.In less deterministic natural text, the corresponding claim is that this class\-conditional KL is small\. Thus, once such a feature is available, a richer state representation has little additional loss to remove\.

In the repeated\-nn\-gram case,ϕ\\phiis the visible continuation of the longest repeated suffix: if the current suffix has occurred before and its continuation is unambiguous, then the target is predictable by continuing material already present in the prefix\. In the closure case,ϕ\\phiis the relevant structural matcher, such as the top of the active delimiter stack: if the target is a closing delimiter, the correct closer is predictable from that stack state\. Since both the matched transformer and the hybrid can express these features in the regimes measured by repeated continuations and bracket/tag closure, we expect little systematic difference between them there\. This explains why repeatednn\-grams and closing delimiters are transformer\-friendly or near\-neutral\.

Second: local class size bounds how much any richer feature map can help\.LetUτU\_\{\\tau\}be the uniform distribution on𝒱τ\\mathcal\{V\}\_\{\\tau\}, and letpclass,τp\_\{\\mathrm\{class\},\\tau\}be the best predictor that sees only the class restrictionXt∈𝒱τX\_\{t\}\\in\\mathcal\{V\}\_\{\\tau\}and no prefix features\. The following bound is the formal version of the POS\-vocabulary\-size effect in[Figure˜3](https://arxiv.org/html/2606.20936#S4.F3)\.

###### Proposition 1\.

Fix a target classτ\\tauand restrict the domain ofXtX\_\{t\}to𝒱τ\\mathcal\{V\}\_\{\\tau\}\. Then any richer feature mapϕ\\phi, evaluated through its best class\-conditional reduced predictor, satisfies

DKL​\(pτ⋆∥pϕ,τ\)≤log⁡\|𝒱τ\|\.D\_\{\\mathrm\{KL\}\}\(p^\{\\star\}\_\{\\tau\}\\\|p\_\{\\phi,\\tau\}\)\\leq\\log\|\\mathcal\{V\}\_\{\\tau\}\|\.

###### Proof\.

Becausepϕ,τp\_\{\\phi,\\tau\}is the best predictor under the feature restrictionϕ\\phiand the class restrictionXt∈𝒱τX\_\{t\}\\in\\mathcal\{V\}\_\{\\tau\}, it can always ignoreϕ​\(x<t\)\\phi\(x\_\{<t\}\)and emulate the best class\-only predictorpclass,τp\_\{\\mathrm\{class\},\\tau\}\. Therefore,

DKL​\(pτ⋆∥pϕ,τ\)\\displaystyle D\_\{\\mathrm\{KL\}\}\(p^\{\\star\}\_\{\\tau\}\\\|p\_\{\\phi,\\tau\}\)≤DKL​\(pτ⋆∥pclass,τ\)\\displaystyle\\leq D\_\{\\mathrm\{KL\}\}\(p^\{\\star\}\_\{\\tau\}\\\|p\_\{\\mathrm\{class\},\\tau\}\)≤DKL​\(pτ⋆∥Uτ\)\\displaystyle\\leq D\_\{\\mathrm\{KL\}\}\(p^\{\\star\}\_\{\\tau\}\\\|U\_\{\\tau\}\)=log⁡\|𝒱τ\|−H​\(pτ⋆\)≤log⁡\|𝒱τ\|\.∎\\displaystyle=\\log\|\\mathcal\{V\}\_\{\\tau\}\|\-H\(p^\{\\star\}\_\{\\tau\}\)\\leq\\log\|\\mathcal\{V\}\_\{\\tau\}\|\.\\qed

For small closed classes, knowing the local class leaves only a limited amount of possible extra information\. Thus, the additional benefit of information beyond the local class is bounded even when that information is useful\. For open classes, the bound is much looser: knowing that the next token is a noun, identifier, string, or text node can still leave substantial uncertainty about which token should fill the slot\. This prediction is consistent with our empirical finding that many function\-like categories have smaller and more heterogeneous hybrid gains, while larger open\-class categories leave more room for architectural differences\.

Finally: when recall, closure, and local class are not enough, the missing feature is semantic state\.We state this final case as a hypothesis\. For non\-copy open\-class choices, identifiers, attribute values, and controlled entity/role probes, the local class can specify the kind of slot being filled, while reuse and closure features still need not determine the filler\. We hypothesize that the semantic state of the discourse provides useful predictive cues in these contexts\.

More concretely, letδ\[t\]−1\\delta\_\{\[t\]\-1\}be a*discourse state*representing the longer\-lived discourse, program, or document state after the previous sentence/unit, where\[t\]\[t\]is the index of the sentence containing tokenttor the analogous local unit in code/markup\. In the context of code evaluation, the discourse state can be identified with the current variable assignments after a step of execution\. In linguistics, different theories of dynamic semantics provide different ways of instantiatingδj\\delta\_\{j\}to encode entities, events, relations, salience, variable bindings, scopes, or document regions\(Heim,[1982](https://arxiv.org/html/2606.20936#bib.bib14),[2002](https://arxiv.org/html/2606.20936#bib.bib16); Kamp and Reyle,[2013](https://arxiv.org/html/2606.20936#bib.bib15); Grosz and Sidner,[1986](https://arxiv.org/html/2606.20936#bib.bib17); Groszet al\.,[1995](https://arxiv.org/html/2606.20936#bib.bib18)\)\. At a high level of abstraction, we imagine that, for a sequence of sentencessent1,…,sentj\\mathrm\{sent\}\_\{1\},\\ldots,\\mathrm\{sent\}\_\{j\}, this discourse state is some structured object updated recurrently by each sentence:

δj=𝖴𝗉𝖽𝖺𝗍𝖾​\(δj−1,sentj\)\.\\delta\_\{j\}=\\mathsf\{Update\}\(\\delta\_\{j\-1\},\\mathrm\{sent\}\_\{j\}\)\.Thus, each sentence acts as an update operator over the discourse stateδj\\delta\_\{j\}\. While models may not explicitly store the discourse state as a structured object, we hypothesize it is useful for them to represent some potentially lossy part of it, sinceδj\\delta\_\{j\}carries information useful for predicting the next token that is not supplied by the local class, repeated\-prefix continuation, or local structural obligation alone\. For example, as shown in[Figure˜6](https://arxiv.org/html/2606.20936#S5.F6), in “John took the book and gave it to Mary\. Mary grabbed her,” the local class suggests a noun phrase, and neither copy nor delimiter state resolves the filler\. What makes*reading glasses*plausible is semantic state: Mary now has the book, so a reading\-related continuation becomes more likely\. This is the sense in which we use “state\-conditioned” or “semantically conditioned”: the local slot is partly specified, but the filler depends on accumulated context\. Thus, we hypothesize that discourse state tracking can provide useful additional information for next\-token prediction in open\-class contexts, and thus that models that can better represent discourse state updates may achieve lower loss in these contexts\.

Putting the three parts together, repeated continuations and closing delimiters are mostly explained by features that both architectures can access, while small closed classes leave limited room for any richer feature to help\. The largest hybrid gains should therefore appear on open\-class, non\-copy, state\-conditioned targets whose fillers depend on accumulated semantic, program, or document context\. This is the split seen in[Figures˜2](https://arxiv.org/html/2606.20936#S4.F2),[4](https://arxiv.org/html/2606.20936#S4.F4)and[5](https://arxiv.org/html/2606.20936#S4.F5)\.333One caveat is that our tags and copy features are noisy proxies for these latent components, so individual tokens may mix several regimes\. The argument should therefore be read as a token\-family\-level explanation of the observed directions\.

### 5\.1Discourse State Tracking Targets to Guide RNN Architectures

Much recent work on recurrent sequence layers uses theA5A\_\{5\}word problem as a standard benchmark for state tracking\(Merrillet al\.,[2024](https://arxiv.org/html/2606.20936#bib.bib21); Grazziet al\.,[2025](https://arxiv.org/html/2606.20936#bib.bib7); Penget al\.,[2025](https://arxiv.org/html/2606.20936#bib.bib28); Yanget al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib29); Merrillet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib11)\)\. In this task, the input specifies a sequence of permutations from the alternating groupA5A\_\{5\}; the model must compose these updates in order and then read out the resulting state, or a property of it\. In the notation above,A5A\_\{5\}is a closed\-world case whereδ\\deltais a fixed\-size finite state and the target is a deterministic readout from that state\. The right side of[Figure˜6](https://arxiv.org/html/2606.20936#S5.F6)gives a schematic example of this kind of closed\-world permutation\-update problem\.

A5A\_\{5\}has been useful because it cleanly isolates ordered update and readout over a fixed finite state space\. However, structured linear\-RNN variants can already solve this benchmark essentially perfectly\(Terzicet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib33)\), which makes it less informative for distinguishing future recurrent sequence layers\.

Moreover,A5A\_\{5\}does not capture certain properties of state tracking in discourse or code\. UnlikeA5A\_\{5\}, natural text and code can introduce new entities, events, variables, scopes, and situations over time, so the effective state can grow with input length\. The state is also relational: useful readouts may depend on who did what to whom, which object is salient, which scope is active, or which variable binding currently holds\(Kamp and Reyle,[2013](https://arxiv.org/html/2606.20936#bib.bib15)\)\. Thus, a broad takeaway from our analysis is that more realistic state tracking could be a good synthetic\-task target for future architectures\. Such tasks would preserve the clean update/readout structure ofA5A\_\{5\}while adding dynamic entity introduction and relational state, making them a more demanding testbed for future linear\-RNN and hybrid sequence layers\.

Discourse trackingJohn took the book and gave it to Mary\.Mary grabbed herpredict content phraseJohnbookMaryreadingglassesgivestohasmakes likelygrabsA5A\_\{5\}\-style state trackingx = \[a, b, c, d, e\]\# permutation \(1 2 3\)tmp=x\[0\]x\[0\]=x\[2\]x\[2\]=x\[1\]x\[1\]=tmpassert x\[2\] == bspecialcase

Figure 6:Discourse tracking as richer state tracking\.Left:predicting the masked content phrase requires tracking that the book has ended up with Mary and using that discourse state to make reading\-related objects likely\.Right:closed\-worldA5A\_\{5\}permutation updates apply ordered, deterministic state changes over a fixed set of slots, followed by a deterministic readout such as an assertion about one slot\.

## 6Application: Higher\-Signal Evaluations for Pretraining Hybrid Models

The decomposition in §[4](https://arxiv.org/html/2606.20936#S4)suggests that aggregate next\-token loss mixes several distinct computation regimes that respond differently to the choice of sequence mixer\. Because most tokens are easy for any reasonable architecture, aggregate loss is a relatively*low\-signal*target for architecture search: real but systematic gaps can be diluted by the easy majority\. We propose tracking*filtered*losses, computed from the same next\-token NLL, as a higher\-signal complement to aggregate validation\. The filters serve two roles\. First, they can amplify small architecture gaps by concentrating on the tokens where a capability matters\. Second, they can reveal different empirical behaviors across token families that a single aggregate number hides\. As a proof of concept, we show that*filtered*validation losses, computed from the same per\-token NLL as standard validation, can reveal capability differences between architectures that are nearly invisible in aggregate loss\.

Filters\.We evaluate three filters built from the token\-level features of §[3](https://arxiv.org/html/2606.20936#S3): \(i\)All tokens, the standard aggregate; \(ii\)Top\-10∩\\capNo\-Copy, restricted to the ten most hybrid\-favored open\-class POS families \([Figure˜2](https://arxiv.org/html/2606.20936#S4.F2)\) and excluding positions that complete a repeatednn\-gram forn≤4n\\leq 4; \(iii\)Copy\-5 only, positions that complete a repeated 5\-gram\. By construction, \(ii\) targets*state\-conditioned readout*while removing visible\-prefix retrieval, and \(iii\) isolates retrieval\.

Setup\.To check that filtered evals carry useful architectural signal during small\-scale pretraining, we evaluate checkpoints from three 1B\-parameter development training runs released byMerrillet al\.\([2026](https://arxiv.org/html/2606.20936#bib.bib11)\): aTransformer, aHybrid\(interleaved GDN/attention,3:13\{:\}1ratio\), and aPure RNN\(GDN, no attention\)\. The three are trained on a matched data mixture and budget under WSD scheduling; we score every annealed checkpoint and report token\-loss curves in[Figure˜7](https://arxiv.org/html/2606.20936#S6.F7)\.

State\-oriented filters amplify small architecture gaps\.Under aggregateAll tokensloss \([Figure˜7](https://arxiv.org/html/2606.20936#S6.F7), left\), the Transformer–Hybrid separation is small, with a maximum of roughly0\.060\.06nats and smaller gaps later in training\. UnderTop\-10∩\\capNo\-Copy\(middle\), the maximum Transformer–Hybrid separation is roughly0\.120\.12nats, about twice as large, and the ordering becomesHybrid<<Pure RNN<<Transformer\. This supports the intended use of the filter: removing copy positions and restricting to open\-class targets leaves a regime that is more sensitive to how well an architecture supports latent\-state construction and state\-conditioned readout\.

Copy filters expose what aggregate loss averages away\.In the aggregateAll Tokenscurve, the Transformer and Pure RNN appear roughly matched, especially at later checkpoints\. Taken alone, this comparison would suggest that the two architectures have similar validation behavior\. The filtered curves show that this apparent match is an average over different regimes\. TheTop\-10∩\\capNo\-Copyfilter separates the models on state\-oriented non\-copy targets, where the Pure RNN does better than the Transformer\. TheCopy\-5,onlyfilter reveals the compensating weakness: on positions that continue a repeated 5\-gram, where prediction mainly requires retrieving material from the visible prefix, the Pure RNN is consistently about0\.100\.10–0\.200\.20nats worse than the two attention\-based models\. Thus filtered loss makes visible a split that aggregate loss largely obscures: recurrence\-only models can look competitive on average while still lagging on visible\-prefix retrieval\.

A diagnostic for hybrid architecture design\.Aggregate loss reports whether an architecture is competitive overall; filtered evals diagnose*why*\. A Pure RNN that closes theTop\-10∩\\capNo\-Copygap but stays far behind attention\-based models onCopy\-5 onlyhas improved its state representation but not the retrieval problem; an architecture that does the reverse is borrowing attention’s strength without gaining on state\-tracking\. This decomposition is especially useful when iterating on*hybrid*architectures, where the design space spans the layer ratio, the choice of recurrent mixer, and the placement of attention: filtered evals make it possible to ask which of those knobs is buying which capability, rather than reading a single aggregate number that aggregates over both regimes\. Because both filters are computed from the same per\-token NLL as standard validation, they add negligible cost, and we suggest reporting them alongside aggregate loss during architecture sweeps\.

![Refer to caption](https://arxiv.org/html/2606.20936v1/x5.png)Figure 7:Filtered token losses surface architecture differences during 1B pretraining\.Token\-loss curves at WSD\-annealed checkpoints for a Transformer, a Hybrid, and a Pure RNN\.
## 7Conclusion

We compared a matched pair of models—a transformer model \(Olmo 3\) and a hybrid model \(Olmo Hybrid\)—at the token level to identify which next\-token predictions drive the average loss gap\. Three patterns recur: \(i\) the hybrid’s edge concentrates on open\-class content predictions; \(ii\) opening delimiters are hybrid\-favored while closing delimiters are transformer\-favored; and \(iii\) the hybrid advantage vanishes on visible\-prefix copying\. Overall, these findings seem to align with the expressivity benefits of hybrid models: attention provides direct visible\-prefix retrieval, while recurrent layers support ordered state update and state\-conditioned readout\. It therefore makes sense that the hybrid advantage concentrates on predictions whose answer depends on an evolving discourse, program, or document state, rather than on copying or bracket closure\.

Two directions follow\. First,A5A\_\{5\}\-style benchmarks capture only a closed\-world special case of the state tracking natural language requires; formal\-language tasks admitting dynamic entity introduction and relational structure, evaluated on linear\-RNN variants \(§[5\.1](https://arxiv.org/html/2606.20936#S5.SS1)\), would test whether architectures that better realize discourse\-style state tracking also improve on the token families driving the hybrid gap\. Second, the same token\-level profile can diagnose data\-mixture and post\-training choices: underperformance on a narrow family \(e\.g\., numeric literals, closing delimiters\) signals where targeted data might help, while the rest of the profile checks that gains do not erode state\-dependent predictions\. Our filtered\-evaluation results provide an encouraging proof of concept for this diagnostic use: token\-level decompositions can sharpen pretraining comparisons, expose capability tradeoffs that aggregate loss hides, and identify concrete token families for targeted intervention\. Future work can build on this proof of concept by exploring the practical implications of token\-level profiles for architecture search, data selection, and post\-training\.

## Acknowledgments and Disclosure of Funding

The authors thank Allyson Ettinger and Kyle Richardson for relevant feedback\.

## References

- Zoology: measuring and improving recall in efficient language models\.InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7\-11, 2024,External Links:[Link](https://openreview.net/forum?id=LY3ukUANko)Cited by:[§2](https://arxiv.org/html/2606.20936#S2.p2.2)\.
- S\. Bird, E\. Klein, and E\. Loper \(2009\)Natural language processing with python: analyzing text with the natural language toolkit\." O’Reilly Media, Inc\."\.Cited by:[§A\.2](https://arxiv.org/html/2606.20936#A1.SS2.SSS0.Px1.p1.1)\.
- D\. Chiang \(2025\)Transformers in uniform TC0\.Trans\. Mach\. Learn\. Res\.2025\.External Links:[Link](https://openreview.net/forum?id=ZA7D4nQuQF)Cited by:[§2](https://arxiv.org/html/2606.20936#S2.p3.3)\.
- W\. N\. Francis \(1979\)Brown corpus manual\.http://icame\. uib\. no/brown/bcm\. html\.Cited by:[§A\.2](https://arxiv.org/html/2606.20936#A1.SS2.SSS0.Px1.p1.1)\.
- R\. Grazzi, J\. Siems, A\. Zela, J\. K\. H\. Franke, F\. Hutter, and M\. Pontil \(2025\)Unlocking state\-tracking in linear rnns through negative eigenvalues\.InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025,External Links:[Link](https://openreview.net/forum?id=UvTo3tVBk2)Cited by:[§1](https://arxiv.org/html/2606.20936#S1.p1.1),[§2](https://arxiv.org/html/2606.20936#S2.p3.3),[§5\.1](https://arxiv.org/html/2606.20936#S5.SS1.p1.4)\.
- B\. J\. Grosz, A\. Joshi, and S\. Weinstein \(1995\)Centering: a framework for modeling the local coherence of discourse\.Computational linguistics21\(2\),pp\. 203–225\.Cited by:[§5](https://arxiv.org/html/2606.20936#S5.SS0.SSS0.Px1.p7.5)\.
- B\. J\. Grosz and C\. L\. Sidner \(1986\)Attention, intentions, and the structure of discourse\.Computational linguistics12\(3\),pp\. 175–204\.Cited by:[§5](https://arxiv.org/html/2606.20936#S5.SS0.SSS0.Px1.p7.5)\.
- I\. R\. Heim \(1982\)The semantics of definite and indefinite noun phrases\.University of Massachusetts Amherst\.Cited by:[§5](https://arxiv.org/html/2606.20936#S5.SS0.SSS0.Px1.p7.5)\.
- I\. Heim \(2002\)File change semantics and the familiarity theory of definiteness\.Formal semantics: The essential readings,pp\. 223–248\.Cited by:[§5](https://arxiv.org/html/2606.20936#S5.SS0.SSS0.Px1.p7.5)\.
- S\. Jelassi, D\. Brandfonbrener, S\. M\. Kakade, and E\. Malach \(2024\)Repeat after me: transformers are better than state space models at copying\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research,pp\. 21502–21521\.External Links:[Link](https://proceedings.mlr.press/v235/jelassi24a.html)Cited by:[§2](https://arxiv.org/html/2606.20936#S2.p2.2)\.
- H\. Kamp and U\. Reyle \(2013\)From discourse to logic: introduction to modeltheoretic semantics of natural language, formal logic and discourse representation theory\.Vol\.42,Springer Science & Business Media\.Cited by:[§5](https://arxiv.org/html/2606.20936#S5.SS0.SSS0.Px1.p7.5),[§5\.1](https://arxiv.org/html/2606.20936#S5.SS1.p3.3)\.
- W\. Merrill, Y\. Li, T\. Romero, A\. Svete, C\. Costello, P\. Dasigi, D\. Groeneveld, D\. Heineman, B\. Kuehl, N\. Lambert, C\. Li, K\. Lo, S\. Malik, D\. Matusz, B\. Minixhofer, J\. Morrison, L\. Soldaini, F\. Timbers, P\. Walsh, N\. A\. Smith, H\. Hajishirzi, and A\. Sabharwal \(2026\)Olmo hybrid: from theory to practice and back\.External Links:2604\.03444,[Link](https://arxiv.org/abs/2604.03444)Cited by:[Appendix F](https://arxiv.org/html/2606.20936#A6.p1.1),[Appendix G](https://arxiv.org/html/2606.20936#A7.p1.1),[§1](https://arxiv.org/html/2606.20936#S1.p1.1),[§1](https://arxiv.org/html/2606.20936#S1.p3.3),[§2](https://arxiv.org/html/2606.20936#S2.p2.2),[§2](https://arxiv.org/html/2606.20936#S2.p3.3),[§2](https://arxiv.org/html/2606.20936#S2.p4.1),[§3](https://arxiv.org/html/2606.20936#S3.p2.1),[§5\.1](https://arxiv.org/html/2606.20936#S5.SS1.p1.4),[§6](https://arxiv.org/html/2606.20936#S6.p3.1)\.
- W\. Merrill, J\. Petty, and A\. Sabharwal \(2024\)The illusion of state in state\-space models\.InForty\-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21\-27, 2024,R\. Salakhutdinov, Z\. Kolter, K\. A\. Heller, A\. Weller, N\. Oliver, J\. Scarlett, and F\. Berkenkamp \(Eds\.\),Proceedings of Machine Learning Research,pp\. 35492–35506\.External Links:[Link](https://proceedings.mlr.press/v235/merrill24a.html)Cited by:[§1](https://arxiv.org/html/2606.20936#S1.p1.1),[§2](https://arxiv.org/html/2606.20936#S2.p3.3),[§5\.1](https://arxiv.org/html/2606.20936#S5.SS1.p1.4)\.
- W\. Merrill and A\. Sabharwal \(2023\)The parallelism tradeoff: limitations of log\-precision transformers\.Trans\. Assoc\. Comput\. Linguistics11,pp\. 531–545\.External Links:[Link](https://doi.org/10.1162/tacl%5C_a%5C_00562),[Document](https://dx.doi.org/10.1162/TACL%5FA%5F00562)Cited by:[§2](https://arxiv.org/html/2606.20936#S2.p3.3)\.
- T\. Olmo, :, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison, J\. Morrison, J\. Poznanski, K\. Lo, L\. Soldaini, M\. Jordan, M\. Chen, M\. Noukhovitch, N\. Lambert, P\. Walsh, P\. Dasigi, R\. Berry, S\. Malik, S\. Shah, S\. Geng, S\. Arora, S\. Gupta, T\. Anderson, T\. Xiao, T\. Murray, T\. Romero, V\. Graf, A\. Asai, A\. Bhagia, A\. Wettig, A\. Liu, A\. Rangapur, C\. Anastasiades, C\. Huang, D\. Schwenk, H\. Trivedi, I\. Magnusson, J\. Lochner, J\. Liu, L\. J\. V\. Miranda, M\. Sap, M\. Morgan, M\. Schmitz, M\. Guerquin, M\. Wilson, R\. Huff, R\. L\. Bras, R\. Xin, R\. Shao, S\. Skjonsberg, S\. Z\. Shen, S\. S\. Li, T\. Wilde, V\. Pyatkin, W\. Merrill, Y\. Chang, Y\. Gu, Z\. Zeng, A\. Sabharwal, L\. Zettlemoyer, P\. W\. Koh, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi \(2026\)Olmo 3\.External Links:2512\.13961,[Link](https://arxiv.org/abs/2512.13961)Cited by:[Appendix G](https://arxiv.org/html/2606.20936#A7.p1.1),[§1](https://arxiv.org/html/2606.20936#S1.p3.3),[§3](https://arxiv.org/html/2606.20936#S3.p2.1)\.
- B\. Peng, R\. Zhang, D\. Goldstein, E\. Alcaide, X\. Du, H\. Hou, J\. Lin, J\. Liu, J\. Lu, W\. Merrill, G\. Song, K\. Tan, S\. Utpala, N\. Wilce, J\. S\. Wind, T\. Wu, D\. Wuttke, and C\. Zhou\-Zheng \(2025\)RWKV\-7 "goose" with expressive dynamic state evolution\.CoRRabs/2503\.14456\.External Links:[Link](https://doi.org/10.48550/arXiv.2503.14456),[Document](https://dx.doi.org/10.48550/ARXIV.2503.14456),2503\.14456Cited by:[§5\.1](https://arxiv.org/html/2606.20936#S5.SS1.p1.4)\.
- A\. Terzic, N\. Menet, M\. Hersche, T\. Hofmann, and A\. Rahimi \(2026\)Structured sparse transition matrices to enable state tracking in state\-space models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=RDbuSCWhad)Cited by:[§5\.1](https://arxiv.org/html/2606.20936#S5.SS1.p2.1)\.
- R\. Waleffe, W\. Byeon, D\. Riach, B\. Norick, V\. Korthikanti, T\. Dao, A\. Gu, A\. Hatamizadeh, S\. Singh, D\. Narayanan, G\. Kulshreshtha, V\. Singh, J\. Casper, J\. Kautz, M\. Shoeybi, and B\. Catanzaro \(2024\)An empirical study of mamba\-based language models\.CoRRabs/2406\.07887\.External Links:[Link](https://doi.org/10.48550/arXiv.2406.07887),[Document](https://dx.doi.org/10.48550/ARXIV.2406.07887),2406\.07887Cited by:[§1](https://arxiv.org/html/2606.20936#S1.p1.1)\.
- G\. Weiss, Y\. Goldberg, and E\. Yahav \(2021\)Thinking like transformers\.External Links:2106\.06981,[Link](https://arxiv.org/abs/2106.06981)Cited by:[§2](https://arxiv.org/html/2606.20936#S2.p2.1)\.
- S\. Yang, Y\. Shen, K\. Wen, S\. Tan, M\. Mishra, L\. Ren, R\. Panda, and Y\. Kim \(2026\)PaTH attention: position encoding via accumulating householder transformations\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=ZBlHEeSvKd)Cited by:[§5\.1](https://arxiv.org/html/2606.20936#S5.SS1.p1.4)\.
- S\. Yao, B\. Peng, C\. Papadimitriou, and K\. Narasimhan \(2023\)Self\-attention networks can process bounded hierarchical languages\.External Links:2105\.11115,[Link](https://arxiv.org/abs/2105.11115)Cited by:[§2](https://arxiv.org/html/2606.20936#S2.p2.1)\.

## Appendix AEmpirical methodology details

### A\.1Domains, packing, and pairing protocol

We evaluate both models on the same evaluation sequences and compute NLL at every position\. Text is packed into contiguous sequences of lengthT=8192T=8192\. Allcomparisons are*paired*at the level of a single next\-token decision:*same checkpoint, same prefix/context, same target token*\. For each positioniiwe computeΔi=ℓiT−ℓiH\\Delta\_\{i\}=\\ell^\{\\mathrm\{T\}\}\_\{i\}\-\\ell^\{\\mathrm\{H\}\}\_\{i\}\.

For the main text we focus on prose \(PG\-19, CC\-News, Wikipedia, ArXiv\) and three structured domains \(Python, HTML,LaTeX\)\. The pooled prose regression uses a slightly expanded prose suite at the final matched checkpoint \(including additional textbooks and scientific papers\), but uses the same packing and pairing protocol\.

### A\.2Token tagging and alignment

#### Prose POS tags and content/function mapping\.

We POS\-tag prose at the*word*level using the Brown tagset\[Francis,[1979](https://arxiv.org/html/2606.20936#bib.bib12), Birdet al\.,[2009](https://arxiv.org/html/2606.20936#bib.bib13)\], then map each Brown tag to a coarse family\. For the aggregate prose analysis, we define:

- •Content words:Noun,Verb,Adjective,Adverb,Interjection,Qualifier\.
- •Function words:Existential,Pronoun,Det/Article,Preposition,Conjunction,Aux BE,Aux HAVE,Aux DO,Modal,TO,Wh\-word\.
- •Other:punctuation, brackets, numerals, particles, and residual tags\.

#### Coarse Brown\-tag mapping\.

Table[1](https://arxiv.org/html/2606.20936#A1.T1)gives the exact mapping used to produce the coarse families in the main\-text prose figures\.

Table 1:Mapping from Brown tags to the coarse categories used in the prose analyses\.Coarse categoryBrown tags*Content words*NounNN,NN$,NNS,NNS$,NP,NP$,NPS,NPS$,NR,NR$,NRSVerbVB,VBD,VBG,VBN,VBZAdjectiveJJ,JJ$,JJR,JJS,JJTAdverbRB,RB$,RBR,RBT,RNInterjectionUHQualifierQL,QLP*Function words*PronounPP$,PP$$,PPL,PPLS,PPO,PPS,PPSS,PN,PN$Det/ArticleAT,DT,DT$,DTI,DTS,DTX,ABL,ABN,ABX,AP,AP$PrepositionINConjunctionCC,CSAux BEBE,BED,BEDZ,BEG,BEM,BEN,BER,BEZAux HAVEHV,HVD,HVG,HVN,HVZAux DODO,DOD,DOZModalMDTOTOExistentialEX*Other categories*Wh\-wordWDT,WP$,WPO,WPS,WQL,WRBNumeralCD,CD$,ODParticleRPPunctuation,,\.,:,’,’’,‘‘,\-\-,\*Open Bracket\(Close Bracket\)
#### Projecting word tags onto subword tokens\.

Because the LM uses subword tokenization, we assign each word\-level tag to every LM token whose decoded character span overlaps the tagged word\. Each LM token also receives a word\-position indicatorWhole/Prefix/Middle/Suffixto separate category effects from word\-onset effects\.

#### Structured domains and multi\-tag overlap\.

For Python we tokenize files using the standardtokenizemodule\. For HTML andLaTeXwe use lightweight parser/regex pipelines that separate text\-like content from structural tokens \(tags/delimiters/commands/whitespace\)\. We align source\-level tags to LM subword tokens by character\-span overlap\.

A single LM token can overlap*multiple*source tokens \(e\.g\.,\):spans both a close\-paren and a colon\)\. We therefore usemulti\-tag attribution: the LM\-token loss contributes to every overlapping source tag rather than forcing a single primary label\.

## Appendix BStructured\-domain tagsets and coarse breakdowns

This section supplies the structured\-domain tag taxonomies\.

### B\.1Python

#### Tagging and multi\-tag alignment\.

We tokenize source files with Python’stokenizemodule and map the resulting source\-level tags onto LM subword tokens by character\-span overlap\. Because one LM token can contain several Python tokens \(e\.g\.,\):\), the Python analysis uses multi\-tag attribution\.

Table 2:Python token taxonomy\.CoarseFine\-grained tagsExamplesIdentifieridentifier,upper\_camel,private,dunderx,MyClass,\_val,\_\_init\_\_Keywordkeyword,soft\_keyworddef,return,matchBuiltinbuiltinprint,len,ValueErrorOperatorarithmetic,assignment,comparison,bitwise\+,\+=,==,&Delimiterenclosure,delimiter,other\(,\),,,:,\.Stringstring,triple\_quoted,fstring,raw,bytes’hello’,"""doc""",f’\{x\}’Numberinteger,float,hex,octal,complex42,3\.14,0xFFCommentCOMMENT\# \.\.\.StructureNEWLINE,INDENT,WHITESPACEindentation and layout

### B\.2HTML

We tag HTML with a parser\-plus\-regex pipeline that distinguishes opening/closing markup, attributes, text nodes, comments, punctuation glyphs, and whitespace\.

Table 3:HTML token taxonomy\.CoarseFine\-grained tagsExamplesTagopen,close,self\_close,void,doctype<div\>,</div\>,<br/\>,<img\>Attributename,equals,value\_quoted,value\_unquotedclass,=,"main"Textcontent,entityfree\-form page textCommentcomment<\!\-\- \.\.\. \-\-\>Punctuationangle\_open,angle\_close,slash,quote<,\>,/,"Whitespacewsspaces and indentation
### B\.3LaTeX

We tagLaTeXwith a lightweight regex pipeline that separates commands, environments, math, tables, grouping symbols, text, comments, and newlines\.

Table 4:LaTeXtoken taxonomy\.CoarseFine\-grained tagsExamplesCommandcontrol,section,ref,cite,formatting, etc\.\\section,\\textbfEnvironmentbegin/end,begin/end\_math,begin/end\_table\\begin\{table\}Mathinline/display\_open/close,env\_name$,$$Tableampersand,hline,rule,multicolumn,linebreak&,\\\\Groupbrace/bracket\_open/close\{,\}Textcontent,spaceordinary words and spacesCommentcomment% \.\.\.Specialescaped,tilde,caret,underscoreescaped symbolsNewlinenewline,blank\_lineline\-level layout

## Appendix CControlled synthetic probes: templates and scoring

The main text reports results for three controlled probe families \(pronoun memory, entity tracking, structural closure\)\. This section records the minimal information needed to reproduce the probes\.

#### Distances and sampling\.

We evaluate antecedent distancesd∈\{32,64,128,256,512,1024\}d\\in\\\{32,64,128,256,512,1024\\\}\. For each family and distance we generate a fixed number of examples with a shared filler distribution across models\.

#### Scoring\.

Pronoun memory and entity tracking are scored*contrastively*at the target position using \(i\) accuracy𝟏​\[m\>0\]\\mathbf\{1\}\[m\>0\]and \(ii\) a log\-probability marginm=log⁡p​\(y\+∣c\)−log⁡p​\(y−∣c\)m=\\log p\(y^\{\+\}\\mid c\)\-\\log p\(y^\{\-\}\\mid c\), wherey\+y^\{\+\}is the correct target andy−y^\{\-\}is a matched distractor under the same prefixcc\. Structural closure is scored by the raw NLL on the closing token,ℓclose=−log⁡p​\(yclose∣c\)\\ell\_\{\\mathrm\{close\}\}=\-\\log p\(y\_\{\\mathrm\{close\}\}\\mid c\)\.

#### Prompt templates\.

We use simple, highly regular templates so that the only manipulated factor is antecedent distance\.

- •Pronoun memory:introduce two entities with different genders and roles; later, reference the role and force a pronoun choice \(e\.g\.,hevs\.she\)\.
- •Entity tracking:bind two same\-gender entities to distinct attributes; later, query the attribute and force a name choice\.
- •Structural closure:open a structure \(e\.g\., an HTML tag or bracketed span\), insert filler, and score the closing token\.

## Appendix DAdditional regression control diagnostics

The main text focuses on the dedicated repeated\-nn\-gram analysis in Figure[4](https://arxiv.org/html/2606.20936#S4.F4)\. Here we report the remaining control\-feature diagnostics from the original coarse\-tag regression\. Figure[8](https://arxiv.org/html/2606.20936#A4.F8)separates two quantities: raw paired gaps for concrete repeated\-token subsets, and fitted coefficients from the frequency\-adjusted regression in Equation[1](https://arxiv.org/html/2606.20936#S3.E1)\. The raw gaps ask whether the hybrid has lower NLL on a subset in absolute terms; the coefficients ask how each feature shifts the paired hybrid–transformer gap after controlling for domain, tag, word position, sequence position, difficulty, previous\-token distance, and target\-token frequency\.

The sign convention is the same as in the main text\. Positive raw gaps mean lower hybrid NLL, while positive coefficients increase the adjusted hybrid advantage\. The repeated\-token coefficients are negative or near zero, consistent with the conclusion that visible\-prefix reuse reduces the hybrid advantage once comparable tokens are matched by the regression\. The remaining controls are included as adjustment variables rather than central findings\.

![Refer to caption](https://arxiv.org/html/2606.20936v1/x6.png)Figure 8:Full control\-feature coefficient diagnostic\.Left: raw paired loss gaps for repeated 1–4\-gram events\. Right: frequency\-adjusted coefficients from the coarse\-tag regression, including difficulty, position, previous\-token distance, target\-token frequency, and repeated\-token controls\. Raw gaps show absolute hybrid advantage on repeated\-token subsets, whereas coefficients show adjusted shifts in the paired hybrid–transformer gap after the full regression specification\.
## Appendix ELimitations

Our analysis has several limitations\. First, our coarse POS, bracket, and copy tags are noisy proxies for the underlying computation a token requires\. A noun token can be a state\-conditioned readout in one context and a near\-copy in another; regression controls reduce but do not eliminate this confounding\. Second, the tag\-stratified and regression analyses are correlational\. The synthetic probes provide controlled manipulation, but each isolates a narrow construction and does not directly measure the contribution of individual recurrent layers\. Third, our interpretation in §[5](https://arxiv.org/html/2606.20936#S5)is a description of where the gap concentrates, not a mechanistic claim\. We do not localize the hybrid advantage to specific layers or heads, and dynamic\-semantics formalisms are used as a high\-level lens rather than as a probed representation\. Fourth, all evaluations are in English prose, Python, HTML, andLaTeX; we do not test other natural languages or programming languages\.

## Appendix FCompute resources

All token\-level analyses use forward passes only \(no training\) on the released Olmo 3 7B and Olmo Hybrid 7B checkpoints\. Scoring the full evaluation suite \(packed length\-8192 sequences across the four prose and three structured domains\) requires approximately 100 GPU\-hours on a single 8xH100 node\. The 1B filtered\-evaluation curves in §[6](https://arxiv.org/html/2606.20936#S6)re\-score released WSD\-annealed checkpoints fromMerrillet al\.\[[2026](https://arxiv.org/html/2606.20936#bib.bib11)\]and require approximately 20 additional GPU\-hours\.

## Appendix GLicenses for existing assets

We use the following released artifacts\.Models:Olmo 3 7B and Olmo Hybrid 7B\[Olmoet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib10), Merrillet al\.,[2026](https://arxiv.org/html/2606.20936#bib.bib11)\]are released under the Apache 2\.0 license; the 1B development checkpoints are released under the same terms\.Datasets:PG\-19 \(Apache 2\.0\); CC\-News \(CC\-BY\); Wikipedia \(CC BY\-SA 3\.0/4\.0 / GFDL\); ArXiv articles \(per individual paper licenses; we use only abstracts/text exposed via the public dump\)\.Tools:NLTK \(Apache 2\.0\) for the Brown POS tagset; the Pythontokenizemodule \(PSF license\)\. We use these assets in accordance with their licenses for non\-commercial research\.