What a Cross-Model Fixed-Point Census Can and Cannot Arbitrate About Repetition

# What a Cross-Model Fixed-Point Census Can and Cannot Arbitrate About Repetition
Source: [https://arxiv.org/html/2609.29507](https://arxiv.org/html/2609.29507)
###### Abstract

Two accounts of neural text degeneration coexist\. One locates the cause in the training data: repetition in the corpus produces repetition in the output, established by training models on repetition\-sorted data\. The other locates it in the trained network, in copying circuits and repetition features\. The accounts have never been arbitrated across a broad cohort of*pretrained*models, because the causal work necessarily trains its own\. We report an observational cross\-model measurement in a different currency: the fixed\-point structure of a model’s own short\-window argmax map, censused from9696random two\-token starts over1717off\-the\-shelf models — always unprompted, which a companion paper shows is a scope condition rather than a detail, since nine tokens of conditioning move this readout across most of its range — with the four\-way structural class stable across census seeds on1717of1717\. Three exhibits\. At fixed corpus \(The Pile\) and fixed scale, the class is*not determined*: across two size\-matched tiers,pythiais a funnel while RWKV, Mamba and a second transformer family are not, and both families hold their class across roughly an order of magnitude of scale\. Six of seven models in that ladder reach the*same*endpoint token, and the models that concentrate on it most strongly are among those that never stay there — so what varies is not where trajectories go but whether the destination self\-continues\. A corpus manipulation available off the shelf, the deduplicated Pythia suite, does not change the class\. And the corpus\-side*inflow*term proposed for this phenomenon does not select our endpoints once frequency is controlled, on English and on three further languages\. We are explicit about what this cannot do: it is observational and cannot refute a training intervention\. Funnels themselves are common in our cohort — eight of seventeen models across seven families and five corpora — so the limit is not that the phenomenon might be one model’s peculiarity; it is that within the single corpus where training data can be held fixed, only one of the available families funnels, so we cannot show from that subset that the split is corpus\-independent\.

## 1Introduction

### 1\.1Two accounts, and why they have not been arbitrated

Neural language models fall into repetitive loops under maximum\-a\-posteriori decoding, and there are two families of explanation for it\.

#### The data side\.

Repetition in the training text produces repetition in the output\.[Fu et al\. \(2021\)](https://arxiv.org/html/2609.29507#bib.bib4)derive this from corpus bigram structure, attributing degeneration to*high\-inflow*words — those whose probability sum over all preceding words is large — and state the thesis plainly: the repetition problem is caused by the language itself\. The empirical champion is[Li et al\. \(2023\)](https://arxiv.org/html/2609.29507#bib.bib6), who train GPT\-2 on repetition\-sorted shards of five datasets, report a strong correlation between repetition rates in training and generated text, and demonstrate causation by dropping out attention to repetitive words during training\. They further argue that the high\-inflow account, the likelihood\-objective account and the self\-reinforcement account all reduce to one factor: penalizing repetitions in data\.

#### The weights side\.

Repetition is a property of the trained network\. Copying circuits are the named mechanism, and the two sides have been connected:[Hernandez et al\. \(2022\)](https://arxiv.org/html/2609.29507#bib.bib5)show that repeated training data disproportionately damages induction heads, and[Aoyama and Wilcox \(2026\)](https://arxiv.org/html/2609.29507#bib.bib1)show that surface bigram repetition frequency governs whether induction heads form at all\.

#### The gap is the cohort, not the connection\.

We do not claim the accounts cannot see each other — they cite each other and cross repeatedly\. What is absent is a broad cohort of*independently pretrained*models measured on a common readout\. The causal work necessarily trains its own models, which is what makes it causal and also what makes it few\-model and single\-recipe\.[Li et al\. \(2023\)](https://arxiv.org/html/2609.29507#bib.bib6)say the remaining gap themselves: the model architecture and size may also contribute, but the two factors have not been quantitatively evaluated\.

### 1\.2What this paper is, stated before the results

This isobservational cross\-model evidence in a currency different from a training intervention\. It cannot refute[Li et al\. \(2023\)](https://arxiv.org/html/2609.29507#bib.bib6): they manipulated training data and measured the consequence, and we did not\. Our claim is narrower and of a different kind — that on a task\-free structural readout, holding corpus and scale fixed does not fix the outcome\.

We are equally explicit about the sharpest limit, and it goes here rather than in a limitations paragraph — but it is narrower than it first appears, and stating it loosely would misdescribe our own data\. Funnels are neither rare nor peculiar to one model:eight of our seventeen models are funnels, spanning seven families and five distinct training corpora, withϕ\\phifrom0\.3280\.328to1\.0001\.000\.What is scarce is not the phenomenon but a*second funnel family within one corpus*\. The only corpus for which we can hold training data fixed across families is The Pile, and of the families available there exactly one funnels\. So the split reported in §[4](https://arxiv.org/html/2609.29507#S4)is not in doubt, and neither is the existence of funnels across many unrelated models — what we cannot establish from that subset alone is that the split is*corpus\-independent*\.

## 2Setup

#### The readout, and what is already published\.

Iterate the model’s own two\-token conditional deterministically,xt\+1=arg⁡maxx⁡p⁡\(x∣xt−1,xt\)x\_\{t\+1\}=\\arg\\max\_\{x\}p\(x\\mid x\_\{t\-1\},x\_\{t\}\), from9696random two\-token starts, and census where trajectories land\. The readout is the*fixed\-point fraction*ϕ\\phi, the proportion of starts terminating in a token that maps to itself\.This map, and the contrast between a model with a dominant attracting fixed point and one without, are established in our own earlier work\([Vera Zúñiga, 2026b](https://arxiv.org/html/2609.29507#bib.bib10)\), which identifies the attracting fixed point of the argmax map and contrasts a model sending most starts to the newline token against one with no such point\. We claim neither here\. What is new is the scale of the census, the class taxonomy and its stability, and the corpus\-versus\-weights attribution the census makes possible\.

#### Scope: a short\-window readout, by nature and not by choice\.

A companion paper establishes that this readout*disappears*as the window widens: generalising the map so the state is the lastWWtokens, rawϕ\\phifalls to0\.0000\.000on four of six models byW=16W=16\([Vera Zúñiga, 2026a](https://arxiv.org/html/2609.29507#bib.bib9)\)\. Everything below is therefore a statement about how a model reads a*fragment*\. We put this in Setup rather than in Limits because it is a property of the construction rather than a shortfall of the evidence, and because a reader who takes ourϕ\\phifor a general property of the model would be taking it for something it is not\.

#### Scope: the raw domain, and only it\.

Every census reported here conditions on nothing: the two\-token starts are drawn at random and no prefix precedes them\. That is a second scope condition, and the same companion paper is what makes it load\-bearing rather than a technicality — nine tokens of conditioning moveϕ\\phiacross most of its range and change the four\-way class, while an instruction\-tuning intervention worth60\.560\.5IFEval points moves the class by zero\([Vera Zúñiga, 2026a](https://arxiv.org/html/2609.29507#bib.bib9)\)\. The classes below are therefore properties of*a model under this condition*, not of the model simpliciter, and a reader who carries one of our class labels to a prompted setting is carrying it somewhere it has been measured not to hold\. We state both scopes here, together, because they bound every number in the paper\.

#### Classes\.

Trajectories are classifiedfunnel\(one dominant attracting fixed point\),none\(cycles or wandering\),fragmented\(many small basins\) andborderline\. The*modal share*of a census is the fraction of the9696starts whose trajectory ends at the single most common endpoint token; it measures how concentrated the destinations are, independently of whether any of them is a fixed point\. Class is a deterministic function of the pair\(ϕ,modal share\)\(\\phi,\\text\{modal share\}\)under a rule fixed before the data was seen:

The gap matters for §[4](https://arxiv.org/html/2609.29507#S4), so we state it rather than leave it to be reconstructed:funnelandnoneare separated by an entire unoccupied band, and the models the paper contrasts sit far from both edges of it\. No boundary in this rule falls between the values being compared\. The funnel geometry — many states feeding one self\-continuing token — was derived theoretically by[Fu et al\. \(2021\)](https://arxiv.org/html/2609.29507#bib.bib4), whose inflow analysis explains why trajectories concentrate; the classes here are that geometry measured on a model’s own conditional rather than on corpus counts\.

#### Stability, which licenses everything else\.

Across1717off\-the\-shelf pretrained models censused at two independent seeds, the class is stable on1717of1717\. The modal endpoint token is stable on1515of1717; the two exceptions are reported and excluded from every comparison rather than assigned a class\.

#### Why this readout and not a generation statistic\.

Nothing is sampled and nothing is generated\. There is no decoding temperature, no prompt, and no text to score\. That makes the measurement cheap enough to run over many models, and it makes it a property of the conditional rather than of a generation procedure — but it also means our quantity is not therep​\-​n\\mathrm\{rep\}\\text\{\-\}nof the literature, and no result here transfers to that quantity without an argument we do not make\.

## 3Related work: an adjacent measurement, and what separates it

The dynamical\-systems framing of degeneration is not ours and we do not claim it\.[Du and Tanaka\-Ishii \(2025\)](https://arxiv.org/html/2609.29507#bib.bib3)describe degeneration as a collapse from a higher\-dimensional trajectory onto a lower\-dimensional attractor, borrowing explicitly from chaos theory, and measure a degeneration\-detecting dynamical property — the correlation dimension of next\-token log\-probability trajectories — across several independently trained families\. That is the closest published object to the present one, and the differences are worth stating precisely rather than leaving implicit\. Their state space is continuous log\-probability vectors seeded from real text; ours is a deterministic map over discrete token space seeded from random pairs\. Their quantity is a dimension estimated over a trajectory; ours is the fraction of starts reaching a token that maps to itself\. Neither measurement is a special case of the other, and the vocabulary they use predates and is independent of our use of it\.

Same\-corpus, size\-matched comparisons of pretrained models across architecture families are likewise an established design rather than a new one, and we adopt it:[Michaelov et al\. \(2024\)](https://arxiv.org/html/2609.29507#bib.bib7)compare Pythia, RWKV and Mamba checkpoints trained on The Pile at matched weight classes, and[Wang et al\. \(2025\)](https://arxiv.org/html/2609.29507#bib.bib11)compare Pythia against Mamba at matched size on induction circuits\. Our contribution on that axis is a readout, not a design\. Both report substantial cross\-architecture*similarity*, and[Michaelov et al\. \(2025\)](https://arxiv.org/html/2609.29507#bib.bib8)report consistency of behavioural phases across architecture, training data and scale\. §[4](https://arxiv.org/html/2609.29507#S4)is therefore a counterexample on a readout those studies do not use, and is presented as one\.

## 4E3: at fixed corpus and fixed scale, the class is not determined

tiermodelfamilyϕ\\phimodal sharemodal endpoint400Mpythia\-410mGPTNeoX0\.4580\.453`\\n`400Mrwkv\-4\-430m\-pileRWKV0\.0100\.521`time`400Mmamba\-370m\-hfMamba0\.0100\.823`first`150Mpythia\-160mGPTNeoX0\.4320\.432`\\n`150Mgpt\-neo\-125mGPTNeo0\.0520\.464`side`150Mrwkv\-4\-169m\-pileRWKV0\.0000\.474`\\n`150Mmamba\-130m\-hfMamba0\.0000\.573`The`Table 1:Two size\-matched tiers, every model trained on The Pile\. Within a tier the largest and smallest parameter counts are within1\.16×1\.16\\times\(400M\) and1\.35×1\.35\\times\(150M\) of each other, a bound fixed before the runs\. Both census seeds agree on the class for all seven\. Source: F178,results/size\_matched\_pile\.json\.Table[1](https://arxiv.org/html/2609.29507#S4.T1)holds the corpus fixed and the weight class fixed, and the class still varies\.pythiais a funnel in both tiers; RWKV, Mamba andgpt\-neoare not\.

#### It is not a scale effect, and that was the pre\-registered risk\.

An earlier version of this comparison usedgpt\-neo\-2\.7Bagainstpythia\-410m, which confounds family with a6\.6×6\.6\\timessize gap\. The decisive cell was registered in advance:gpt\-neo\-125magainstpythia\-160m,1\.28×1\.28\\timesapart on the same corpus, with the condition that if they landed in the same class the claim would be withdrawn\. They do not —ϕ=0\.052\\phi=0\.052against0\.4320\.432, both stable across seeds\.

#### We do not say “architecture”, and the reason is in the table\.

gpt\-neois a transformer, from the same laboratory, trained on the same corpus, and it sits with the recurrent models rather than withpythia\. The split is therefore not transformer\-versus\-recurrent, and naming it after architecture would be reading the table wrongly\. What the evidence supports is the weaker and more precise statement:at fixed corpus and fixed scale the class is not determined, and something on the weights side that we have not identified decides it\.

#### The domain is held fixed here too, and saying so is not a formality\.

Every cell in Table[1](https://arxiv.org/html/2609.29507#S4.T1)is a raw\-domain census, per Setup\. That third fixed condition deserves naming beside the other two, because unlike them it is known to be capable of moving the quantity: nine tokens of conditioning change this class in the companion measurement\([Vera Zúñiga, 2026a](https://arxiv.org/html/2609.29507#bib.bib9)\)\. So what E3 establishes is non\-determination*at one domain*\. Whether these same seven models separate the same way under a prefix is a different experiment, not a corollary of this one, and we do not run it\.

Table 2:Both families hold their class across scale on one corpus: Pythia over a14×14\\timesspan, GPT\-Neo over22×22\\times\. No model is class\-unstable across seeds\. Source: F179,results/family\_scale\_ladder\.json\.
#### The split is a stable family property, not two unrepresentative checkpoints\.

Table[2](https://arxiv.org/html/2609.29507#S4.T2)extends each family across scale\. Neither ladder breaks\.

#### The sharpest observation is in the endpoint column\.

Six of the seven models in Table[2](https://arxiv.org/html/2609.29507#S4.T2)reach the*same*endpoint token, the newline — every Pythia, and GPT\-Neo at13001300M and27002700M\. GPT\-Neo concentrates on it*more*strongly than some Pythias do: modal share0\.5360\.536and0\.5940\.594againstpythia\-160m’s0\.4320\.432\. Yet GPT\-Neo’sϕ\\phiis0\.0000\.000and0\.0360\.036while every Pythia exceeds0\.360\.36\. The same holds inside Table[1](https://arxiv.org/html/2609.29507#S4.T1)at matched size:rwkv\-4\-169m\-pileandpythia\-160mshare the newline endpoint at nearly the same concentration,0\.4740\.474against0\.4320\.432, withϕ\\phiof0\.0000\.000against0\.4320\.432\.

Where trajectories go is shared; whether the destination self\-continues is not\.The models that funnel hardest onto the newline are among those that never stay on it\.

#### The decomposition is arithmetic, not a reading of the table\.

Compareϕ\\phiagainst the modal share row by row in Table[2](https://arxiv.org/html/2609.29507#S4.T2)\. For every Pythia the two agree to within four census starts out of9696—0\.8020\.802against0\.7920\.792,0\.4320\.432against0\.4320\.432,0\.4580\.458against0\.4530\.453,0\.3650\.365against0\.4060\.406\. For every GPT\-Neo the share exceedsϕ\\phiby forty to fifty\-four starts\. In other wordsϕ\\phifactorises, to within the census resolution, as

ϕ≈\(mass arriving at the modal endpoint\)×\(whether that endpoint self\-continues\),\\phi\\;\\approx\\;\(\\text\{mass arriving at the modal endpoint\}\)\\times\(\\text\{whether that endpoint self\-continues\}\),with the second factor near11for one family and near00for the other while the first is comparable across both\. The corpus appears to fix the first term and something on the weights side the second\. We state this as a factorisation others can test rather than as an impression, and §[8](https://arxiv.org/html/2609.29507#S8)asks for exactly that test\.

## 5E1: an off\-the\-shelf corpus manipulation that does not change the class

The Pythia suite was released in duplicate, one copy trained on The Pile and one on a near\-deduplicated copy, explicitly so that deduplication could be studied\([Biderman et al\., 2023](https://arxiv.org/html/2609.29507#bib.bib2)\)\. That is the nearest thing to the data\-side manipulation available without training anything\.

Both members remainfunnel:ϕ=0\.458\\phi=0\.458forpythia\-410mand0\.4270\.427forpythia\-410m\-deduped, with the same modal endpoint token\.

#### This pair does not differ only in deduplication, and the caveat is not incidental\.

The deduplicated Pile is approximately207207B tokens while both suites are trained to approximately300300B\([Biderman et al\., 2023](https://arxiv.org/html/2609.29507#bib.bib2)\), so the deduplicated models run about1\.451\.45epochs and re\-encounter a large fraction of their corpus a second time\.In a paper about repetition this matters: the manipulation re\-introduces the very variable the data\-side account identifies as causal\. We therefore do not present this as a clean corpus manipulation\. It is a null on a readout with no prior expectation of invariance, from a pair whose difference is smaller than its label suggests\.

#### And a null here is weaker than it looks for a second reason\.

[Hernandez et al\. \(2022\)](https://arxiv.org/html/2609.29507#bib.bib5)show that damage from repeated data is strongly non\-monotonic — a double\-descent, concentrated in a particular range of repetition frequency\. If the Pile’s duplication does not fall in that range, the data\-side account predicts no effect either, and our null is consistent with both accounts rather than discriminating between them\. We report E1 because it is the manipulation the field would ask for, and we report what it does and does not establish\.

## 6E2: the corpus\-side inflow term does not select the endpoints, and this agrees with the data side

[Fu et al\. \(2021\)](https://arxiv.org/html/2609.29507#bib.bib4)attribute degeneration to high\-inflow words, where the inflow of a token is the probability sum over all its predecessors\. If the endpoints our maps reach are the tokens that account predicts, they should be high\-inflow ones\.

Thirteen of the seventeen models are readable for this comparison, and the four exclusions are of three different kinds\. Two \(LFM2\-2\.6B,starcoder2\-3b\) disagree with themselves across census seeds about which endpoint token is modal, so the quantity being predicted is not stable and they are dropped before anything is computed\. One \(bloom\-3b\) has an endpoint token that occurs*zero*times in an English corpus — a statement about the corpus, not about the theory, and §[6](https://arxiv.org/html/2609.29507#S6)returns to it\. One \(polyglot\-ko\-1\.3b\) had a tokenizer that would not load, which is an infrastructure failure and evidence about nothing\. We keep the three reasons apart because they license different inferences\.

On an uncontrolled reading the endpoints are high\-inflow:1212of the1313have an endpoint at or above the9090th inflow percentile, median99\.8799\.87\. That reading is an artefact of frequency\.[Fu et al\. \(2021\)](https://arxiv.org/html/2609.29507#bib.bib4)themselves distinguish the two, arguing that it is not the high\-frequency words but the high\-inflow words that lead to repetition, so we compare each endpoint against the5050tokens nearest it in log\-frequency\. Against that control the endpoint beats its matched peers in11of1313models, with a median matched percentile of32\.032\.0against5050by construction\.

The result is not an artefact of measuring every model on English\. Three models whose endpoints are not English were measured on their own languages at matched corpus size:6\.06\.0on Spanish,16\.016\.0on Korean and96\.096\.0on Japanese\. Two of the three were unmeasurable in English — their endpoint tokens occur zero times in an English corpus — and both land far below the null on the right corpus\.

#### This agrees with the data\-side account rather than opposing it\.

[Li et al\. \(2023\)](https://arxiv.org/html/2609.29507#bib.bib6)already subsume the inflow account into repetition\-in\-training\-data by a controlled experiment: merging only the repetitive high\-inflow pairs, a small fraction of training words, matches the full high\-inflow method, while merging random high\-inflow pairs of the same size does not\. Inflow was demoted by the data side itself\. Likewise[Michaelov et al\. \(2025\)](https://arxiv.org/html/2609.29507#bib.bib8)find that unigram frequency dominates a large share of word\-level behavioural variance across pretrained models\.E2 is therefore a consistency check, not a refutation: our maps funnel to*common*tokens rather than to high\-inflow ones, which is what both of those results predict\.

## 7Limits

#### The currency gap, first because it is the largest\.

[Li et al\. \(2023\)](https://arxiv.org/html/2609.29507#bib.bib6)manipulated training data and measured the consequence\. We measured pretrained models and compared them\. An observational contrast across models cannot refute a controlled intervention, and nothing here should be read as doing so\. Where our results bear on theirs at all, as in §[6](https://arxiv.org/html/2609.29507#S6), they agree\.

#### One funnel family*within the fixed\-corpus subset*\.

A reader may worry that §[4](https://arxiv.org/html/2609.29507#S4)rests on a peculiarity of one model suite\. Our own cohort answers that: funnels appear in seven families across five corpora \(§[1](https://arxiv.org/html/2609.29507#S1)\), so the phenomenon is common and is not a Pythia artefact\. The genuine limit is more specific\. Within The Pile — the one corpus where we can vary the family with training data held fixed — only Pythia funnels among the families available to us\. A second funnel family at fixed corpus would let us show the split is corpus\-independent rather than a fact about which families happen to be Pile\-trained; the obvious public candidate exceeded the memory available to us, and we did not obtain one\.That is what §[4](https://arxiv.org/html/2609.29507#S4)cannot currently show, and it is narrower than "the effect may not be real"\.

#### What “not determined” does and does not mean\.

§[4](https://arxiv.org/html/2609.29507#S4)holds corpus and parameter count fixed\. It does not hold fixed the training schedule, the optimiser, the data order, the tokenizer, or the many choices that differ between independently trained families\. A difference under those conditions means*not explained by corpus or scale*\. It does not identify what does explain it, and we make no attempt to\.

#### Readout scope\.

ϕ\\phiis a property of a deterministic two\-token map\. It is notrep​\-​n\\mathrm\{rep\}\\text\{\-\}non generated text, and the relationship between them is unestablished\. A model with no fixed points in this census may still degenerate under sampling, and we have not checked\.

#### Statistical refusals, recorded before the numbers\.

Nopp\-values are reported\. The cohort is1717models and the exhibits rest on two families; a significance test at thatnncould not fail informatively\. No rank correlations are computed between class and scale\. Where a comparison could not be made cleanly — two models whose modal endpoint is unstable across seeds, models whose endpoints do not occur in a given corpus — the models are named and excluded rather than assigned a value\.

## 8Conclusion

On a deterministic, task\-free readout of a model’s own conditional, holding the training corpus fixed and the parameter count fixed does not fix the fixed\-point structure\. Two families trained on the same corpus keep opposite classes across an order of magnitude of scale, and a corpus manipulation released expressly to study deduplication does not move the class either — though that pair is confounded by a second pass over much of its data, and we say so\. Meanwhile the corpus\-side term proposed for this phenomenon does not select the endpoints once frequency is controlled, in four languages, which is what the data\-side literature now predicts for itself\.

The observation we would most like tested by others is the decomposition in §[4](https://arxiv.org/html/2609.29507#S4): across families, scales and corpora, models agree remarkably about*which*token trajectories arrive at and disagree about whether that token maps to itself\. Six of seven models in one ladder land on the newline; the ones that concentrate on it most strongly are the ones that never stay\. If that survives a cohort with more than one funnel family in it, it is a fact about language models that neither account currently explains\.

## References

- Aoyama and Wilcox \[2026\]Tatsuya Aoyama and Ethan Gotlieb Wilcox\.Predicting the emergence of induction heads in language model pretraining, 2026\.
- Biderman et al\. \[2023\]Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal\.Pythia: A suite for analyzing large language models across training and scaling\.In*International Conference on Machine Learning \(ICML\)*, 2023\.
- Du and Tanaka\-Ishii \[2025\]Xin Du and Kumiko Tanaka\-Ishii\.Correlation dimension of auto\-regressive large language models\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.
- Fu et al\. \[2021\]Zihao Fu, Wai Lam, Anthony Man\-Cho So, and Bei Shi\.A theoretical analysis of the repetition problem in text generation\.In*Proceedings of the AAAI Conference on Artificial Intelligence \(AAAI\)*, 2021\.
- Hernandez et al\. \[2022\]Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El\-Showk, Nelson Elhage, Zac Hatfield\-Dodds, Tom Henighan, Tristan Hume, Scott Johnston, Ben Mann, Chris Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish\.Scaling laws and interpretability of learning from repeated data, 2022\.
- Li et al\. \[2023\]Huayang Li, Tian Lan, Zihao Fu, Deng Cai, Lemao Liu, Nigel Collier, Taro Watanabe, and Yixuan Su\.Repetition in repetition out: Towards understanding neural text degeneration from the data perspective\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.
- Michaelov et al\. \[2024\]James A\. Michaelov, Catherine Arnett, and Benjamin K\. Bergen\.Revenge of the fallen? recurrent models match transformers at predicting human language comprehension metrics\.In*Conference on Language Modeling \(COLM\)*, 2024\.
- Michaelov et al\. \[2025\]James A\. Michaelov, Roger P\. Levy, and Benjamin K\. Bergen\.Language model behavioral phases are consistent across architecture, training data, and scale\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2025\.
- Vera Zúñiga \[2026a\]Nicolás Vera Zúñiga\.Prompt–model interaction reaches the fixed points: a deterministic, task\-free structural readout — and the factorizations of it that failed, 2026a\.
- Vera Zúñiga \[2026b\]Nicolás Vera Zúñiga\.What iterated self\-feeding probes of language models measure, and a test that separates the construction from the model, 2026b\.
- Wang et al\. \[2025\]Junxuan Wang, Xuyang Ge, Wentao Shu, Leyi Tang, Yunhua Zhou, Tianxiang Sun, and Xipeng Qiu\.Towards universality: Studying mechanistic similarity across language model architectures\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.