Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning

# Pure Exemplar Models Mimic Abstraction-First Learning
Source: [https://arxiv.org/html/2608.00821](https://arxiv.org/html/2608.00821)
## Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction\-First Learning

Zachary Nicholas Houghton University of Oregon Vail Systems, Inc\. znh@uoregon\.edu&Vsevolod Kapatsinski University of Oregon

###### Abstract

Whether idiosyncratic, item\-specific knowledge is learned before abstract class\-level generalizations, or vice versa, is a central question in language learning, with exemplar and abstraction\-based theories making opposite predictions\. Recent methods have claimed to show that, at least for large language models, abstract knowledge is learned first\. We show that these methods fall short: pure memorizer models with no abstract representations can appear, by the same criteria, to learn either item\-specific or class\-level knowledge first, depending on their sensitivity to individual observations, with the transition point governed by the distributional properties of the input\. We further argue that the distinction between item\-specific and abstract knowledge may be ill\-defined for distributed representations, as a word’s class\-level properties may not be separable from its item\-specific properties\.

Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction\-First Learning

Zachary Nicholas HoughtonUniversity of OregonVail Systems, Inc\.znh@uoregon\.eduVsevolod KapatsinskiUniversity of Oregon

## 1Introduction

Language learning draws on both abstract representations and item\-specific knowledge\. A learner must learn the idiosyncrasies for verbs like*give*and*run*and represent them as distinct lexical items, while also discovering that*give*and*sell*share class\-level distributional preferences that set them apart from motion verbs like*go*and*walk*\. How these two levels of knowledge interact, and crucially which emerges first during learning, is a central question in cognitive science and linguistics\.

A substantial body of work argues that abstract representations play a central role in linguistic knowledge, enabling generalizations that extend productively to novel items beyond what has been directly observed\(Berko,[1958](https://arxiv.org/html/2608.00821#bib.bib1); Albright and Hayes,[2003](https://arxiv.org/html/2608.00821#bib.bib2); Tomasello,[2005](https://arxiv.org/html/2608.00821#bib.bib3); Goldberg,[2003](https://arxiv.org/html/2608.00821#bib.bib5); Kapatsinski,[2018a](https://arxiv.org/html/2608.00821#bib.bib14),[2014](https://arxiv.org/html/2608.00821#bib.bib46),[2012](https://arxiv.org/html/2608.00821#bib.bib47); Morgan and Levy,[2016a](https://arxiv.org/html/2608.00821#bib.bib4),[2024](https://arxiv.org/html/2608.00821#bib.bib11); Houghton,[2025](https://arxiv.org/html/2608.00821#bib.bib12); Milinet al\.,[2023](https://arxiv.org/html/2608.00821#bib.bib39); Harmon and Kapatsinski,[2017](https://arxiv.org/html/2608.00821#bib.bib42)\)\.Berko \([1958](https://arxiv.org/html/2608.00821#bib.bib1)\)showed that children could generate the plural of*wug*\(*wugs*\) without any prior experience with the word, demonstrating that morphological knowledge must be at least partly abstract and productive\. In the domain of phonology,Albright and Hayes \([2003](https://arxiv.org/html/2608.00821#bib.bib2)\)showed that an abstraction\-based model out\-performs other models in the learning of English plural allomorphy\. Indeed evidence for abstractions has been pursued across every subfield in linguistics with each domain offering distinct empirical windows into how abstract representations are structured and learned\(Tomasello,[2005](https://arxiv.org/html/2608.00821#bib.bib3); Goldberg,[2003](https://arxiv.org/html/2608.00821#bib.bib5); Kapatsinski,[2018a](https://arxiv.org/html/2608.00821#bib.bib14); Morgan and Levy,[2016a](https://arxiv.org/html/2608.00821#bib.bib4),[2024](https://arxiv.org/html/2608.00821#bib.bib11); Houghton,[2025](https://arxiv.org/html/2608.00821#bib.bib12); Houghtonet al\.,[2025](https://arxiv.org/html/2608.00821#bib.bib13); Pluthet al\.,[2026](https://arxiv.org/html/2608.00821#bib.bib17)\)\.

Exemplar theories offer a competing account of how such knowledge is represented, particularly in the categorization literature\(Nosofsky,[1986](https://arxiv.org/html/2608.00821#bib.bib7); Nosofsky and Kruschke,[1992](https://arxiv.org/html/2608.00821#bib.bib6); Ambridge,[2020b](https://arxiv.org/html/2608.00821#bib.bib8); Bybee and Hopper,[2001](https://arxiv.org/html/2608.00821#bib.bib20); Bybee,[2001](https://arxiv.org/html/2608.00821#bib.bib21); Stemberger and MacWhinney,[1986](https://arxiv.org/html/2608.00821#bib.bib18),[2004](https://arxiv.org/html/2608.00821#bib.bib19); Bybee,[1999](https://arxiv.org/html/2608.00821#bib.bib27),[2002](https://arxiv.org/html/2608.00821#bib.bib28),[2003](https://arxiv.org/html/2608.00821#bib.bib29); Bybee and Scheibman,[1999](https://arxiv.org/html/2608.00821#bib.bib30); Pierrehumbert,[2001](https://arxiv.org/html/2608.00821#bib.bib33),[2002](https://arxiv.org/html/2608.00821#bib.bib34); McMurrayet al\.,[2002](https://arxiv.org/html/2608.00821#bib.bib32); Ráczet al\.,[2020](https://arxiv.org/html/2608.00821#bib.bib36); Goldrick and Cole,[2023](https://arxiv.org/html/2608.00821#bib.bib38); Gradoville,[2023](https://arxiv.org/html/2608.00821#bib.bib40)\)\. Rather than positing abstract representations, exemplar models argue that categories are collections of stored instances, and that category membership is determined by similarity to previously encountered exemplars\. This framework naturally accounts for the frequency effects on gradient category membership: a robin is judged more bird\-like than a chicken not because it better fits an abstract representation, but because it is more similar to the bulk of stored exemplars\. Apparent generalizations emerge from the density structure of the exemplar cloud, with no explicit abstraction required\. In linguistics, usage\-based approaches have developed this intuition extensively\(Bybee and Hopper,[2001](https://arxiv.org/html/2608.00821#bib.bib20); Bybee,[2001](https://arxiv.org/html/2608.00821#bib.bib21)\): token frequency strengthens the representation of specific forms, explaining why high\-frequency irregulars like*went*and*broke*resist regularization, while type frequency of a pattern licenses analogical extension to novel items without requiring a stored abstract rule\.Stemberger and MacWhinney \([1986](https://arxiv.org/html/2608.00821#bib.bib18)\)provided early psycholinguistic evidence for frequency\-sensitive storage\(see also Stemberger and MacWhinney,[2004](https://arxiv.org/html/2608.00821#bib.bib19)\), showing that high\-frequency regular forms are retrieved holistically rather than generated compositionally\.Ambridge \([2020b](https://arxiv.org/html/2608.00821#bib.bib8)\)extends this reasoning to argument structure, arguing that even apparently abstract grammatical patterns are best explained as emergent generalizations over stored exemplars\.

Between these extremes lie hybrid accounts in which abstract representations exist but are themselves built from specific instances\(e\.g\., Ambridge,[2020a](https://arxiv.org/html/2608.00821#bib.bib9); Goldberg,[2003](https://arxiv.org/html/2608.00821#bib.bib5); Kapatsinski,[2018a](https://arxiv.org/html/2608.00821#bib.bib14); Stemberger and MacWhinney,[1986](https://arxiv.org/html/2608.00821#bib.bib18); Morgan and Levy,[2016a](https://arxiv.org/html/2608.00821#bib.bib4),[2015](https://arxiv.org/html/2608.00821#bib.bib45),[b](https://arxiv.org/html/2608.00821#bib.bib48); Liu and Morgan,[2020](https://arxiv.org/html/2608.00821#bib.bib49); Houghton,[2025](https://arxiv.org/html/2608.00821#bib.bib12); Houghtonet al\.,[2026](https://arxiv.org/html/2608.00821#bib.bib43); Houghton and Morgan,[2023](https://arxiv.org/html/2608.00821#bib.bib44)\)\. The connectionist model ofRumelhart and McClelland \([1986](https://arxiv.org/html/2608.00821#bib.bib10)\)was foundational here, and not simply because it reproduced the behavioral data: the deeper significance was that its connection weights jointly encoded both item\-specific frequency knowledge and the class\-level regularity of the past\-tense rule in the same distributed representation, with no principled boundary between what is stored as an exemplar and what is computed via abstract knowledge\. This reframed the debate: in a distributed representational system, class\-level abstractions and item\-specific exemplar knowledge are not two separate mechanisms but two levels of description of the same learned weight structure\.Kapatsinski \([2018b](https://arxiv.org/html/2608.00821#bib.bib15)\)further developed this perspective theoretically, arguing that frequency\-sensitive learning is inherently hybrid: because connection strengths are shaped by the full history of individual exposures, the learned representation simultaneously reflects item\-specific token frequency and class\-level type frequency without requiring either to be computed separately;Kapatsinski \([2021](https://arxiv.org/html/2608.00821#bib.bib25)\)provides a formal instantiation of this in phonological learning, where hierarchical inference over experience\-tagged exemplars allocates credit simultaneously to word\-specific and phoneme\-class\-level representations\.

The convergence on a hybrid account shifts the central question: rather than asking whether abstract or exemplar\-based representations exist, the more pressing issue is which emerges first during learning\. This ordering question has direct consequences for theories of language learning\. If class\-level abstract representations emerge first, early generalization can scaffold subsequent item\-specific learning\. If item\-specific exemplars accumulate first, abstractions emerge bottom\-up through the gradual organization of individual instances\.Tomasello \([2005](https://arxiv.org/html/2608.00821#bib.bib3)\)anticipated this question at the developmental level, arguing that children’s early constructions are item\-specific and that productive abstraction only emerges gradually throughout learning\. The two accounts thus make opposite predictions about the temporal ordering of class\-level versus item\-specific knowledge, predictions that are in principle detectable from learning trajectories\.

These questions have gained new empirical traction with the availability of large language models, whose learning trajectories can be observed directly over the course of training, making them a productive testing ground for theoretical predictions about the ordering of class\-level and item\-specific learning\.

One such study exploring this wasJian and Manning \([2026](https://arxiv.org/html/2608.00821#bib.bib26)\), who investigated this directly by analyzing learning trajectories in GPT\-2 small over the course of pretraining\. Their study tracked next\-token prediction distributions conditioned on argument structure constructions across four verb classes: to\-Dative verbs \(e\.g\.,*give*,*sell*\), Motion verbs \(e\.g\.,*go*,*walk*\), Reciprocal verbs \(e\.g\.,*speak*,*meet*\), and spray\-load verbs \(e\.g\.,*spray*,*load*\)\. Using Jensen\-Shannon divergence \(DJSD\_\{\\text\{JS\}\}\) to calculate how different probability distributions were for different verbs, they operationalized two types of learning onset\. Between\-class onset is declared when a verb’s probability distribution over next\-token predictions is more similar to same\-class verbs than to verbs from the opposite class, assessed via a one\-tailed Mann\-Whitney U\-test\. Within\-class onset is declared when verbs of the same class begin to diverge from one another as item\-level idiosyncrasies are learned, measured as the mean within\-classDJSD\_\{\\text\{JS\}\}\. Their key finding was that for to\-Dative and Motion verbs, between\-class differentiation emerged approximately 50 training steps before within\-class divergence, consistent with the abstraction\-first account\.

To rule out the possibility that the abstraction\-first pattern was an artifact of the training corpus rather than the model,Jian and Manning \([2026](https://arxiv.org/html/2608.00821#bib.bib26)\)implemented a count\-based distributional baseline using add\-kksmoothing applied to co\-occurrence vectors from the same corpus\. This baseline showed the opposite pattern: item\-specific divergence emerged before class distinctions became apparent, matching the exemplar\-first prediction\.Jian and Manning \([2026](https://arxiv.org/html/2608.00821#bib.bib26)\)interpret this contrast as evidence that GPT\-2 has an abstraction\-first learning bias that the count\-based baseline lacks\. This interpretation, however, rests on the assumption that the onset ordering reliably reflects learning strategy rather than incidental properties of the learner, such as how sensitive the learner is to individual observations\. Whether the ordering is in fact diagnostic of an abstraction\-first bias, or whether it is a consequence of learner sensitivity, remains an open question\.

### 1\.1Present Study

We investigate this question through controlled simulations using the same verb classes, distributional metrics, and onset criteria asJian and Manning \([2026](https://arxiv.org/html/2608.00821#bib.bib26)\), making our results directly comparable to theirs\. We use two idealized learner models that share an identical data\-generation process but differ in how they estimate verb distributions from observations\. Both are*pure memorizers*: each verb’s distribution is estimated entirely from its own observations, with no information flow between verbs, no class\-level representations, and no capacity to generalize\.

The first model, theZero\-Sensitivity Learner\(ZSL\), provides a noiseless analytical baseline in which each new observation shifts the learner’s estimate smoothly toward the true distribution, unperturbed by the idiosyncratic identity of individual tokens\. The second model, theVariable\-Sensitivity Learner\(VSL\), is a Dirichlet\-Multinomial learner that formalizes the add\-kksmoothing ofJian and Manning \([2026](https://arxiv.org/html/2608.00821#bib.bib26)\)’s count\-based baseline within a Bayesian framework, with sensitivity to individual observations controlled by a single parameterkk\. At lowkk, early observations dominate and idiosyncratic tokens drive the trajectory; at highkk, the estimate is heavily smoothed toward uniformity, suppressing item\-level noise\. We use these exemplar models to explore whether observation\-level sensitivity can lead to the appearance of abstraction\-first learning\.

Our key contributions are as follows:

- •We demonstrate that the ordering of between\-class onset \(obo\_\{b\}\) and within\-class onset \(owo\_\{w\}\) is not a fixed property of abstract\-first or exemplar\-first learning, but depends on the interaction between learner sensitivity \(kk\) and the structure of the input distribution\.
- •We introduce the Zero\-Sensitivity Learner \(an exemplar model\) as a noiseless baseline, showing thatob<owo\_\{b\}<o\_\{w\}holds universally when observation\-level noise is removed\.
- •We show that the Variable\-Sensitivity Learner \(also a pure exemplar model\) replicates the exemplar\-first ordering \(ow<obo\_\{w\}<o\_\{b\}\) of the count\-based baseline at lowkkand transitions to the abstraction\-first ordering \(ob<owo\_\{b\}<o\_\{w\}\) at highkk, demonstrating that the onset ordering tracks learner sensitivity to individual observations rather than abstraction\-first or exemplar\-first learning\.

## 2Simulations

We formalize our central claim through two simulations that share an identical data\-generation process but differ in how the learner estimates verb distributions from observations\. Both models arepure memorizers\(i\.e\., exemplar models\): each verb’s distribution is estimated solely from its own observations, completely independently of other verbs, with no information flow between verbs and no class\-level representations\. Both models assume that the learner begins from a near\-uniform state: before sufficient evidence has accumulated, all verbs are nearly indistinguishable\. We confirm this assumption empirically for a transformer language model in Appendix[A](https://arxiv.org/html/2608.00821#A1)\.111All code and analyses in this paper can be found here:[https://github\.com/znhoughton/exemplar\-abstraction\-sims](https://github.com/znhoughton/exemplar-abstraction-sims)\.

### 2\.1Data Generation

The following data\-generation procedure is identical across both models; what differs between them is exclusively the learner model: how each learner constructs its estimateP^v\\hat\{P\}\_\{v\}from the available evidence, described separately for each model below\.

#### 2\.1\.1Verb Classes and Vocabulary

FollowingJian and Manning \([2026](https://arxiv.org/html/2608.00821#bib.bib26)\), we use two verb classes of 35 and 36 items respectively\. The specific verb types are irrelevant to the simulations; only the class sizes matter\. The vocabulary is simplified toV=1000V=1000tokens for computational tractability; the qualitative results are robust to vocabulary size\.

#### 2\.1\.2True Distribution Construction

Each verbvvhasnpref=50n\_\{\\text\{pref\}\}=50preferred tokens drawn from the vocabulary which represent the tokens that receive a non\-zero probability after encountering verbvv\. Two proportion parameters, class\_overlap∈\[0,1\]\\in\[0,1\]and item\_overlap∈\[0,1\]\\in\[0,1\], control how much token sharing occurs across and within classes\. Preferred tokens are partitioned into three types:

- •Cross tokens\(ncross=⌊class\_overlap×npref⌋n\_\{\\text\{cross\}\}=\\lfloor\\text\{class\\\_overlap\}\\times n\_\{\\text\{pref\}\}\\rfloor\): shared by all verbs in both classes, representing distributional properties common regardless of verb class\.
- •Within\-class tokens\(nwithin=⌊\(item\_overlap−class\_overlap\)×npref⌋n\_\{\\text\{within\}\}=\\lfloor\(\\text\{item\\\_overlap\}\-\\text\{class\\\_overlap\}\)\\times n\_\{\\text\{pref\}\}\\rfloor\): shared among verbs of the same class only, encoding class\-level argument preferences\.
- •Idiosyncratic tokens\(nidio=npref−ncross−nwithinn\_\{\\text\{idio\}\}=n\_\{\\text\{pref\}\}\-n\_\{\\text\{cross\}\}\-n\_\{\\text\{within\}\}\): unique to each verb,222Given the vocabulary size, idiosyncratic tokens cannot always be fully unique across all 71 verbs: when\(NA\+NB\)×nidio\(N\_\{A\}\+N\_\{B\}\)\\times n\_\{\\text\{idio\}\}exceeds the size of the available token pool, some reuse is unavoidable by construction\. Token assignment is designed to spread this reuse as evenly as possible, so that no idiosyncratic token is ever shared by more than two verbs\.representing item\-specific collocational preferences\.

Preferred token weights are drawn from a log\-normal distributionLogNormal​\(log⁡μ−σ22,σ\)\\text\{LogNormal\}\(\\log\\mu\-\\frac\{\\sigma^\{2\}\}\{2\},\\,\\sigma\), parameterized so that𝔼​\[weight\]=μ\\mathbb\{E\}\[\\text\{weight\}\]=\\muregardless ofσ\\sigma\.333For a standardLogNormal​\(μ,σ\)\\text\{LogNormal\}\(\\mu,\\sigma\)distribution, the expectation iseμ\+σ22e^\{\\mu\+\\frac\{\\sigma^\{2\}\}\{2\}\}rather thaneμe^\{\\mu\}due to Jensen’s Inequality\. By shifting the first parameter tolog⁡μ−σ22\\log\\mu\-\\frac\{\\sigma^\{2\}\}\{2\}, the\+σ22\+\\frac\{\\sigma^\{2\}\}\{2\}term cancels out, forcing the final expectation to be exactlyμ\\mu\.Non\-preferred tokens receive a uniform background weight of 1\.0\. Each verb’s true distributionPv​\(t\)P\_\{v\}\(t\)\(the probability of vocabulary itemt∈\{1,…,V\}t\\in\\\{1,\\ldots,V\\\}\) is obtained by normalizing these raw weights:

Pv​\(t\)=wt∑t′wt′\{P\_\{v\}\(t\)=\\frac\{w\_\{t\}\}\{\\sum\_\{t^\{\\prime\}\}w\_\{t^\{\\prime\}\}\}\}\(1\)

#### 2\.1\.3Onset Criteria

Both criteria matchJian and Manning \([2026](https://arxiv.org/html/2608.00821#bib.bib26)\)’s Experiments 1 and 2\. Pairwise distances between verb distributions are measured using the Jensen\-Shannon divergence:

DJS​\(P,Q\)=H​\(P\+Q2\)−12​\[H​\(P\)\+H​\(Q\)\]\{D\_\{\\text\{JS\}\}\(P,Q\)=H\\\!\\left\(\\tfrac\{P\+Q\}\{2\}\\right\)\-\\tfrac\{1\}\{2\}\\bigl\[H\(P\)\+H\(Q\)\\bigr\]\}\(2\)
whereH​\(⋅\)H\(\\cdot\)denotes Shannon entropy\.DJSD\_\{\\text\{JS\}\}ranges from 0 \(identical distributions\) tolog⁡2≈0\.693\\log 2\\approx 0\.693\(maximally dissimilar distributions\)\.

Between\-class onset \(obo\_\{b\}\) is declared at the first step at which≥10%\\geq 10\\%of verbs individually pass a one\-tailed Mann\-Whitney U\-test \(p<0\.001p<0\.001\) comparing that verb’sDJSD\_\{\\text\{JS\}\}distances to out\-of\-class verbs against its distances to same\-class verbs\.

Within\-class onset \(owo\_\{w\}\) is declared at the first step at which the mean within\-classDJSD\_\{\\text\{JS\}\},D¯within\\bar\{D\}\_\{\\text\{within\}\}, exceeds 0\.01 for three consecutive steps:

D¯within=12\[1\(NA2\)∑i<j∈ADJS\(i,j\)\+1\(NB2\)∑i<j∈BDJS\(i,j\)\]\{\\begin\{split\}\\bar\{D\}\_\{\\text\{within\}\}&=\\frac\{1\}\{2\}\\left\[\\frac\{1\}\{\\binom\{N\_\{A\}\}\{2\}\}\\sum\_\{i<j\\,\\in\\,A\}D\_\{\\text\{JS\}\}\(i,j\)\\right\.\\\\ &\\left\.\\qquad\+\\frac\{1\}\{\\binom\{N\_\{B\}\}\{2\}\}\\sum\_\{i<j\\,\\in\\,B\}D\_\{\\text\{JS\}\}\(i,j\)\\right\]\\end\{split\}\}\(3\)
whereAAandBBdenote the to\-Dative and Motion verb classes \(NA=35N\_\{A\}=35andNB=36N\_\{B\}=36verbs respectively\), and the sums are over all unique within\-class verb pairs\.

The two criteria are fundamentally asymmetric:obo\_\{b\}is a*relative*rank test that fires whenever between\-class distances consistently exceed within\-class distances, regardless of absolute magnitude\.owo\_\{w\}is an*absolute*threshold test that requiresDJSD\_\{\\text\{JS\}\}to reach a specific magnitude\. As we show below, this asymmetry is the key mechanism underlying theob<owo\_\{b\}<o\_\{w\}ordering\.

All models share the same structural parameters \(μ∈\{10,30,60,100\}\\mu\\in\\\{10,30,60,100\\\};σ∈\{0\.5,1\.0,1\.5\}\\sigma\\in\\\{0\.5,1\.0,1\.5\\\}; item\_overlap∈\{0\.5,0\.6,0\.7\}\\in\\\{0\.5,0\.6,0\.7\\\}; class\_overlap∈\{0\.2,0\.3,0\.4\}\\in\\\{0\.2,0\.3,0\.4\\\}\), subject to the constraint class\_overlap<<item\_overlap, yielding 108 valid structural combinations \(full details in Appendix[B](https://arxiv.org/html/2608.00821#A2)\)\. Each model additionally varies its own sensitivity parameters, which are described in the respective sections below\. All conditions are replicated 50 times per combination\. The models share identical data\-generation procedures and onset criteria; they differ only in how the learner estimatesP^v\\hat\{P\}\_\{v\}from available evidence\.

### 2\.2Zero\-Sensitivity Learner

#### 2\.2\.1Learner Model

The Zero\-Sensitivity Learner represents the theoretical limit of a maximally conservative learner: one whose estimates are entirely unaffected by any individual token observation\. Whereas a realistic distributional learner is shaped by whichever specific tokens happen to appear earliest in a verb’s contexts \(which may be idiosyncratic and unrepresentative\), the Zero\-Sensitivity Learner’s trajectory is noiseless and monotone, moving smoothly from complete ignorance toward full knowledge of each verb’s distribution\. It is an analytical baseline that isolates the effect of learning progress from observation\-level noise\.

Formally, the learner’s estimate of verbvv’s distribution at learning stageα∈\[0,1\]\\alpha\\in\[0,1\]is a linear interpolation between the uniform distribution andPvP\_\{v\}:

P^v​\(α\)=\(1−α\)⋅Uniform\+α⋅Pv\{\\hat\{P\}\_\{v\}\(\\alpha\)=\(1\-\\alpha\)\\cdot\\text\{Uniform\}\+\\alpha\\cdot P\_\{v\}\}\(4\)
Each component has a precise interpretation\.PvP\_\{v\}is verbvv’strue distribution: the stable distribution that verb would converge to given unlimited data\. Critically, each verb has its ownPvP\_\{v\}: dative verbs like*give*and*sell*have systematically different true distributions from motion verbs like*go*and*run*\(reflecting different argument preferences\), and individual verbs within the same class have modestly distinct distributions from one another \(reflecting idiosyncratic collocational preferences\)\. TheUniformterm \(1/V1/Vfor each of theVVvocabulary items\) represents the learner’s starting state: a learner with no distributional knowledge assigns equal probability to every token regardless of which verb appears in context\.

The parameterα\\alphais the central quantity of this model\. Mathematically, it is the weight placed on the true distribution relative to the uniform baseline, linearly interpolating between the two\. Conceptually,α\\alpharepresents the stage of learning: atα=0\\alpha=0, the learner’s estimated distributions are entirely uniform, reflecting no learned knowledge of verb\-specific structure; atα=1\\alpha=1, the learner has fully converged to each verb’s true distributionPvP\_\{v\}\. Intermediate values ofα\\alphacorrespond to intermediate stages of convergence\.

Because the trajectory is deterministic, the only signals driving divergence between verb distributions are the structural properties encoded in thePvP\_\{v\}themselves: the class\-level structure \(within\-class shared tokens\) creates systematic between\-class divergence detectable byobo\_\{b\}, while idiosyncratic verb\-specific tokens create within\-class divergence that drivesowo\_\{w\}\.

We evaluate the71×7171\\times 71pairwiseDJSD\_\{\\text\{JS\}\}matrix and check both onset criteria at 25 log\-spaced values ofα\\alphafrom 0\.001 to 1\.0, tracing the full learning trajectory\. Log spacing concentrates evaluation points nearα=0\\alpha=0, where onset behavior first emerges\. Each parameter combination is run for 50 random seeds, yielding108×50=5,400108\\times 50=5\{,\}400runs total\.

#### 2\.2\.2ZSL Results

The Zero\-Sensitivity Learner producesob<owo\_\{b\}<o\_\{w\}in 85% of seeds overall,ow<obo\_\{w\}<o\_\{b\}in 12%, and neither onset detected in 3%\. Figure[1](https://arxiv.org/html/2608.00821#S2.F1)shows the fraction of seeds yieldingob<owo\_\{b\}<o\_\{w\}across the\(σ,gap\)\(\\sigma,\\text\{gap\}\)parameter space, averaged overμ∈\{10,30,60,100\}\\mu\\in\\\{10,30,60,100\\\}\.

![Refer to caption](https://arxiv.org/html/2608.00821v1/x1.png)Figure 1:Zero\-Sensitivity Learner: fraction of seeds withob<owo\_\{b\}<o\_\{w\}across the parameter space, averaged overμ∈\{10,30,60,100\}\\mu\\in\\\{10,30,60,100\\\}\. Blue =ob<owo\_\{b\}<o\_\{w\}dominates; red =ow<obo\_\{w\}<o\_\{b\}dominates; yellow≈0\.5\\approx 0\.5marks the transition\. The complement of the displayed fraction is split betweenow<obo\_\{w\}<o\_\{b\}and neither\-detected seeds\.Theob<owo\_\{b\}<o\_\{w\}result is universal forσ=0\.5\\sigma=0\.5across all parameter combinations\. Forσ=1\.0\\sigma=1\.0, the single exception is the weakest possible class signal \(gap=0\.1=0\.1\), whereow<obo\_\{w\}<o\_\{b\}occurs in 28% of seeds on average; all other combinations produceob<owo\_\{b\}<o\_\{w\}\. Forσ=1\.5\\sigma=1\.5,ob<owo\_\{b\}<o\_\{w\}breaks down at small overlap gaps, where high variance between verbs creates sufficient idiosyncratic noise to driveowo\_\{w\}before any class structure accumulates\. As the between\-class gap increases,ob<owo\_\{b\}<o\_\{w\}recovers to 95% at gap=0\.5=0\.5\.

The exceptions are concentrated in the parameter region with the weakest class signal: small gap, highσ\\sigma, and lowμ\\mu\(see Appendix[C](https://arxiv.org/html/2608.00821#A3)for the full per\-μ\\mubreakdown\)\. This is the expected behavior: theobo\_\{b\}criterion requires a detectable class signal, and neither ordering can be reliably observed when the class structure is too weak to distinguish from noise\. In the next section, we explore what might be driving this effect\.

### 2\.3Variable\-Sensitivity Learner

#### 2\.3\.1Learner Model

The Variable\-Sensitivity Learner accumulates distributional evidence incrementally, constructing its estimate of each verb’s distribution from the tokens observed in that verb’s contexts\. Unlike the Zero\-Sensitivity Learner, whose trajectory is noiseless and deterministic, the Variable\-Sensitivity Learner’s estimates are shaped by whichever specific tokens appear earliest, including idiosyncratic tokens that may not reflect the verb’s stable class\-level distribution\. How strongly those early, potentially unrepresentative observations shape the learner’s estimate is governed entirely by the smoothing parameterkk\.

Formally, the model is aDirichlet\-Multinomialconjugate Bayesian learner\. Before any observations, the learner holds apriorover verbvv’s token distribution: a symmetric Dirichlet,Dir​\(k,k,…,k\)\\text\{Dir\}\(k,k,\\ldots,k\), encoding equal prior belief across allVVtoken types\. This is the multinomial analogue of a Beta\(α1,α2\)\(\\alpha\_\{1\},\\alpha\_\{2\}\)prior: just as Beta\(k,k\)\(k,k\)encodeskkpseudo\-successes andkkpseudo\-failures before any data,Dir​\(k,…,k\)\\text\{Dir\}\(k,\\ldots,k\)encodeskkpseudo\-observations of each of theVVtoken types before any verb contexts are seen\. Largerkkmeans a stronger prior: more pseudo\-evidence the real data must overcome before the estimated distribution departs from uniform\.

Rather than sweepingα\\alphadirectly, the Variable\-Sensitivity Learner tracksnobsn\_\{\\text\{obs\}\}: the number of tokens the learner has encountered for each verb\. Thelikelihoodis Multinomial: each token is drawn independently fromPvP\_\{v\}\. Because the Dirichlet is the conjugate prior to the Multinomial, theposteriorafternobsn\_\{\\text\{obs\}\}observations, wherectc\_\{t\}denotes the count of vocabulary itemtt, is simply:

Dir\(k\+c1,…,k\+cV\)\{\\begin\{split\}&\\text\{Dir\}\\\!\\bigl\(k\+c\_\{1\},\\;\\ldots,\\\\ &\\qquad k\+c\_\{V\}\\bigr\)\\end\{split\}\}\(5\)
Because the mean of a Dirichlet\(α1,…,αV\)\(\\alpha\_\{1\},\\ldots,\\alpha\_\{V\}\)distribution for tokenttisαt/∑jαj\\alpha\_\{t\}/\\sum\_\{j\}\\alpha\_\{j\}, theposterior meanfor tokenttfollows directly: substitutingαt=k\+count​\(t∣v,nobs\)\\alpha\_\{t\}=k\+\\text\{count\}\(t\\mid v,n\_\{\\text\{obs\}\}\)and∑jαj=k​V\+nobs\\sum\_\{j\}\\alpha\_\{j\}=kV\+n\_\{\\text\{obs\}\}gives:

P^v​\(t\)=count​\(t∣v,nobs\)\+knobs\+k⋅V\{\\hat\{P\}\_\{v\}\(t\)=\\frac\{\\text\{count\}\(t\\mid v,\\,n\_\{\\text\{obs\}\}\)\+k\}\{n\_\{\\text\{obs\}\}\+k\\cdot V\}\}\(6\)
This is the posterior mean of the Dirichlet\-Multinomial, derived as a direct consequence of Bayesian updating\. The numerator addskkpseudo\-observations to the raw count; the denominator incorporates both thenobsn\_\{\\text\{obs\}\}real observations and thek⋅Vk\\cdot Vtotal pseudo\-observations from the prior\.

The parameterkkis therefore theconcentrationof the Dirichlet prior, measuring how strongly the learner holds on to its initial uniform belief\. Conceptually,kkcontrols howconservative\(or equivalently, howsensitive\) the learner is to individual observations\. Ahigh\-kklearner is conservative: the large pseudo\-count massk⋅Vk\\cdot Vin the denominator means each new observation shifts the estimate by only a small amount, requiring many consistent observations before the estimated distribution departs meaningfully from uniform\. This suppresses idiosyncratic early\-observation noise and allows stable, class\-consistent patterns to accumulate gradually\. Alow\-kklearner is highly sensitive: with few pseudo\-counts, each observation has a large effect on the estimate, and the learner’s distribution for a verb quickly reflects whichever tokens happened to appear first, regardless of whether those tokens are class\-typical or idiosyncratic to that verb\.

Ask→∞k\\to\\infty, stochastic deviations around the posterior mean vanish and the Variable\-Sensitivity Learner converges to the deterministic trajectory of the Zero\-Sensitivity Learner \(see Appendix[D](https://arxiv.org/html/2608.00821#A4)for the derivation\): the expected value ofP^v\\hat\{P\}\_\{v\}at anynobsn\_\{\\text\{obs\}\}is exactly the Zero\-Sensitivity Learner’s interpolation withα=nobs/\(nobs\+k​V\)\\alpha=n\_\{\\text\{obs\}\}/\(n\_\{\\text\{obs\}\}\+kV\), and stochastic deviations around that expectation vanish askkgrows\.

In concrete terms, the learner accumulates tokens for all verbs in parallel\. At each observation, every verbvvreceives one token: a vocabulary item drawn fromPvP\_\{v\}, which increments that item’s running count forvvby 1\. Equation[6](https://arxiv.org/html/2608.00821#S2.E6)can be evaluated at any point to read off the current estimateP^v\\hat\{P\}\_\{v\}: each token type’s count inflated bykkpseudo\-observations, divided bynobs\+k​Vn\_\{\\text\{obs\}\}\+kV, wherenobsn\_\{\\text\{obs\}\}is the number of tokens accumulated for that verb\. We simulate the learner across 5,000 tokens per verb and examine the learner’s state at 20 log\-spaced values ofnobsn\_\{\\text\{obs\}\}from 1 to 5,000; because the same sequence underlies all snapshots for a given seed, the tokens atnobs=100n\_\{\\text\{obs\}\}=100are a strict superset of those atnobs=10n\_\{\\text\{obs\}\}=10\. Every verb encounters the same number of tokens at every checkpoint, so verb frequency plays no role; the only source of variability across verbs and seeds is which specific tokens happen to appear\.444The assumption of equal observations per verb is an idealization\. InJian and Manning \([2026](https://arxiv.org/html/2608.00821#bib.bib26)\)’s corpus baseline, verbs differ substantially in frequency, so more frequent verbs accumulate observations faster\. We tested a Zipfian extension \(Model 3\) in which the individual observations were sampled from a Zipfian distribution whereby each verb’s observation count is proportional to its frequency rank\. The results replicate the qualitative pattern from the equal\-frequency VSL model; this analysis is presented in Appendix[E](https://arxiv.org/html/2608.00821#A5)\.

#### 2\.3\.2VSL Results

Overall results bykkare shown in Table[1](https://arxiv.org/html/2608.00821#S2.T1)\. Fork≤0\.1k\\leq 0\.1\(effective pseudo\-count totalk⋅V≤100k\\cdot V\\leq 100\),ow<obo\_\{w\}<o\_\{b\}holds in approximately 99% of seeds regardless of structural parameters\. The transition fromow<obo\_\{w\}<o\_\{b\}toob<owo\_\{b\}<o\_\{w\}begins atk=0\.5k=0\.5\(k⋅V=500k\\cdot V=500pseudo\-counts\), where both orderings occur with roughly equal frequency and many seeds produce a tie \(ob=owo\_\{b\}=o\_\{w\}at the samenobsn\_\{\\text\{obs\}\}grid point\)\. Byk=1\.0k=1\.0\(k⋅V=1,000k\\cdot V=1\{,\}000pseudo\-counts\), the prior is strong enough to fully suppress idiosyncratic early\-observation noise:ob<owo\_\{b\}<o\_\{w\}predominates across the majority of structural parameter combinations\.

Table 1:Variable\-Sensitivity Learner overall means bykk\. ‘Tie’ indicates both onsets detected at the samenobsn\_\{\\text\{obs\}\}grid point\.Figure[2](https://arxiv.org/html/2608.00821#S2.F2)illustrates why: it shows theobo\_\{b\}andowo\_\{w\}criterion trajectories overnobsn\_\{\\text\{obs\}\}for three values ofkkstraddling the transition, making the mechanism directly visible\. At lowkk,owo\_\{w\}spikes immediately at the first observations whileobo\_\{b\}accumulates slowly; at highkk,owo\_\{w\}is suppressed andobo\_\{b\}detects the class\-level asymmetry first\.

![Refer to caption](https://arxiv.org/html/2608.00821v1/k_trajectories.png)Figure 2:Variable\-Sensitivity Learner:obo\_\{b\}andowo\_\{w\}criteria over observations per verb \(nobsn\_\{\\text\{obs\}\}\) for three values ofkkstraddling the transition\. Blue solid:obo\_\{b\}criterion \(fraction of verbs passing the Mann\-Whitney test, left axis\)\. Red dashed:owo\_\{w\}criterion \(mean within\-classDJSD\_\{\\text\{JS\}\}, right axis\)\. Dotted lines mark the onset thresholds \(CLASS\_FRAC=0\.10=0\.10; DJS\_THRESH=0\.01=0\.01\)\. Parameters:μ=100\\mu=100,σ=0\.5\\sigma=0\.5, item\_overlap=0\.7=0\.7, class\_overlap=0\.2=0\.2; 20 seeds; 95% CI ribbons\.The breakdown across the full\(σ,gap\)\(\\sigma,\\text\{gap\}\)parameter space is shown in Figure[3](https://arxiv.org/html/2608.00821#S2.F3)fork∈\{0\.01,0\.1,0\.5,1\.0\}k\\in\\\{0\.01,0\.1,0\.5,1\.0\\\}\(results atk=0\.001k=0\.001are virtually identical tok=0\.01k=0\.01and are omitted from the figure; full numerical results for allkkvalues in Appendix[F](https://arxiv.org/html/2608.00821#A6)\)\. Atk=0\.01k=0\.01,ow<obo\_\{w\}<o\_\{b\}is universal across nearly all parameter combinations, with a small number of seeds failing to detect either onset only in the weak\-signal corner \(small gap, highσ\\sigma, lowμ\\mu\)\. Atk=0\.1k=0\.1, the pattern is qualitatively unchanged:ow<obo\_\{w\}<o\_\{b\}remains near\-universal, indicating thatk⋅V=100k\\cdot V=100pseudo\-counts are still insufficient to suppress idiosyncratic noise\. The transition emerges atk=0\.5k=0\.5andob<owo\_\{b\}<o\_\{w\}dominates byk=1\.0k=1\.0\.

![Refer to caption](https://arxiv.org/html/2608.00821v1/x2.png)Figure 3:Variable\-Sensitivity Learner: fraction of seeds withob<owo\_\{b\}<o\_\{w\}bykk,σ\\sigma, and overlap gap, averaged overμ∈\{10,30,60,100\}\\mu\\in\\\{10,30,60,100\\\}\. Blue =ob<owo\_\{b\}<o\_\{w\}dominates; red =ow<obo\_\{w\}<o\_\{b\}dominates\. Atk≤0\.1k\\leq 0\.1virtually all seeds produceow<obo\_\{w\}<o\_\{b\}; the transition emerges atk=0\.5k=0\.5andob<owo\_\{b\}<o\_\{w\}dominates byk=1\.0k=1\.0\.Theow<obo\_\{w\}<o\_\{b\}result is completely invariant to theDJSD\_\{\\text\{JS\}\}threshold at allkkvalues \(see Appendix[G](https://arxiv.org/html/2608.00821#A7)\)\. At lowkk, within\-classDJSD\_\{\\text\{JS\}\}spikes immediately to values exceeding any reasonable threshold in the tested range, soowo\_\{w\}fires at the first observation regardless of threshold choice\. Theob<owo\_\{b\}<o\_\{w\}result in the high\-kkregime is similarly robust across all tested values of CLASS\_FRAC andDJSD\_\{\\text\{JS\}\}threshold\. The qualitative pattern also holds when verbs accumulate counts at rates proportional to their frequency rather than equally, relaxing what may be an unrealistic assumption \(see Appendix[E](https://arxiv.org/html/2608.00821#A5)\)\.

Taken together, both learners demonstrate that theob<owo\_\{b\}<o\_\{w\}ordering is a predictable consequence of low sensitivity to individual observations, not of an abstraction\-first inductive bias\. Both are pure memorizers: each verb’s distribution is estimated entirely from its own smoothed token counts, with no information flow between verbs and no class\-level representations\. Despite this, both produceob<owo\_\{b\}<o\_\{w\}whenever sensitivity is sufficiently low: at the deterministic limit \(Zero\-Sensitivity Learner\) or with sufficient add\-kksmoothing \(Variable\-Sensitivity Learner, highkk\)\. The ordering of onsets is therefore diagnostic of observation sensitivity, not of abstraction\. Conversely, high sensitivity \(lowkk\) producesow<obo\_\{w\}<o\_\{b\}, which is the signature of a learner shaped by idiosyncratic early observations before consistent class\-level patterns accumulate\.

## 3Discussion and Conclusion

The present study demonstrates that the abstraction\-first ordering arises in pure memorizers with no class\-level representations, no information flow between verbs, and no inductive bias toward generalization, provided that sensitivity to individual observations is sufficiently low\. The onset ordering is therefore not diagnostic of abstraction\-first learning; it reflects how sensitive the learner is to individual observations\. These results do not preclude the possibility that transformer models are abstraction\-first learners, but they show that the evidenceJian and Manning \([2026](https://arxiv.org/html/2608.00821#bib.bib26)\)provide is insufficient to establish this: a high\-kkexemplar memorizer produces the same apparent onset pattern\. There is independent reason to think transformers may behave like high\-kklearners:Kapatsinski \([2026](https://arxiv.org/html/2608.00821#bib.bib16)\)shows that transformers perform adaptive partial pooling, weighting individual observations against class\-level averages in a frequency\-sensitive way\. This suppresses idiosyncratic noise in the same way highkkdoes, providing a mechanistic basis for why GPT\-2 might show apparent abstraction\-first ordering\. To be clear, this pooling should not itself be taken as evidence of abstraction: pooling over instances can be a natural consequence of how distributed representations are structured, reflecting the geometry of the representational space rather than the presence of discrete class\-level structures separable from item\-specific knowledge\. Theob<owo\_\{b\}<o\_\{w\}pattern could arise from this distributed geometry without any explicit encoding of class membership\.

A deeper challenge concerns whether the exemplar\-versus\-abstraction dichotomy is well\-defined for distributed representations in the first place\. Theobo\_\{b\}andowo\_\{w\}onset criteria presuppose that between\-class differentiation and within\-class divergence are separable learning events, with one marking the emergence of abstract class knowledge and the other the accumulation of item\-specific exemplars\. But for a learner whose representations are purely distributional, these may not be separable\. Learning the distributional properties of*give*, which argument types and constructions it favors, necessarily encodes its class membership, because what makes*give*’s distribution distinctive partly consists of the features it shares with*sell*and that set it apart from*go*\. The class\-level signal is not a separate layer of knowledge sitting above the item\-specific knowledge; it is embedded within it\. The onset criteria treat between\-class and within\-class divergence as tracking two separable learning events, but for distributed representations there may be no clean seam between the two, and interpreting them as indexing qualitatively distinct kinds of knowledge assumes a conceptual distinction that the representations themselves may not support\.

In summary, the ordering of between\-class and within\-class onset is a function of observation sensitivity, not of learning strategy\. A pure memorizer with no class\-level representations produces the abstraction\-first pattern whenever sensitivity is sufficiently low, which means thatJian and Manning \([2026](https://arxiv.org/html/2608.00821#bib.bib26)\)’s methodology cannot distinguish abstraction\-first learning from high\-kkexemplar learning\. More broadly, the exemplar\-versus\-abstraction distinction may itself be ill\-defined for distributed representations: a word’s class\-level properties are embedded within its item\-specific distribution, making the two difficult to dissociate from distributional evidence alone\. Whether transformer models are abstraction\-first learners in any theoretically meaningful sense remains an open question, but answering it will require methodology that goes beyond the ordering of onset criteria\.

## 4Limitations

The present simulations use synthetic verb distributions rather than real corpus frequencies\. This is a deliberate design choice that allows precise control over the overlap structure, but it means the specific parameter values that produce each ordering do not map directly to naturalistic language\. This concern is mitigated by the breadth of the parameter sweep: the core results hold across wide ranges ofμ\\mu,σ\\sigma, item overlap, and class overlap, suggesting they are not artifacts of any particular distributional assumption\. Similarly, we use two verb classes rather than the four analyzed byJian and Manning \([2026](https://arxiv.org/html/2608.00821#bib.bib26)\); a multi\-class scenario introduces additional between\-class pairings that our model does not capture\. Given that the onset ordering is driven by the asymmetry between relative and absolute criteria rather than by the number of classes, we do not expect this to change the qualitative conclusions, but it remains to be verified\.

## References

- Rules vs\. analogy in english past tenses: a computational/experimental study\.Cognition90\(2\),pp\. 119–161\.External Links:[Document](https://dx.doi.org/10.1016/S0010-0277%2803%2900146-X)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1)\.
- B\. Ambridge \(2020a\)Abstractions made of exemplars or ‘you’re all right, and i’ve changed my mind’: response to commentators\.First Language40\(5\-6\),pp\. 640–659\.External Links:[Document](https://dx.doi.org/10.1177/0142723720949723)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- B\. Ambridge \(2020b\)Against stored abstractions: a radical exemplar model of language acquisition\.First Language40\(5\-6\),pp\. 509–559\.External Links:[Document](https://dx.doi.org/10.1177/0142723719869731)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- J\. Berko \(1958\)The child’s learning of english morphology\.*WORD*14\(2\-3\),pp\. 150–177\.External Links:[Document](https://dx.doi.org/10.1080/00437956.1958.11659661),[Link](http://www.tandfonline.com/doi/full/10.1080/00437956.1958.11659661)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1)\.
- J\. Bybee and P\. Hopper \(2001\)Introduction to frequency and the emergence of linguistic structure\.InFrequency and the Emergence of Linguistic Structure,J\. L\. Bybee and P\. J\. Hopper \(Eds\.\),pp\. 1–26\.Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- J\. Bybee and J\. Scheibman \(1999\)The effect of usage on degrees of constituency: the reduction of don’t in english\.Linguistics37\(4\),pp\. 575–596\.External Links:[Document](https://dx.doi.org/10.1515/ling.37.4.575)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- J\. Bybee \(1999\)Usage\-based phonology\.InFunctionalism and Formalism in Linguistics, Volume I: General Papers,M\. Darnell, E\. Moravcsik, F\. Newmeyer, M\. Noonan, and K\. Wheatley \(Eds\.\),pp\. 211–242\.Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- J\. Bybee \(2001\)Phonology and language use\.Cambridge Studies in Linguistics,Cambridge University Press\.Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- J\. Bybee \(2002\)Word frequency and context of use in the lexical diffusion of phonetically conditioned sound change\.Language Variation and Change14\(3\),pp\. 261–290\.External Links:[Document](https://dx.doi.org/10.1017/S0954394502143018)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- J\. Bybee \(2003\)Mechanisms of change in grammaticization: the role of frequency\.InThe Handbook of Historical Linguistics,B\. D\. Joseph and R\. D\. Janda \(Eds\.\),pp\. 602–623\.External Links:[Document](https://dx.doi.org/10.1002/9780470756393.ch19)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- A\. E\. Goldberg \(2003\)Constructions: a new theoretical approach to language\.Trends in Cognitive Sciences7\(5\),pp\. 219–224\.External Links:[Document](https://dx.doi.org/10.1016/S1364-6613%2803%2900080-9)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1),[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- M\. Goldrick and J\. Cole \(2023\)Advancement of phonetics in the 21st century: Exemplar models of speech production\.Journal of Phonetics99,pp\. 101254\.External Links:[Document](https://dx.doi.org/10.1016/j.wocn.2023.101254)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- M\. Gradoville \(2023\)The future of exemplar theory\.InThe Handbook of Usage\-Based Linguistics,M\. Díaz\-Campos and S\. Balasch \(Eds\.\),pp\. 527–544\.External Links:[Document](https://dx.doi.org/10.1002/9781119839859.ch29)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- Z\. Harmon and V\. Kapatsinski \(2017\)Putting old tools to novel uses: the role of form accessibility in semantic extension\.Cognitive Psychology98,pp\. 22–44\.External Links:[Document](https://dx.doi.org/10.1016/j.cogpsych.2017.08.002),[Link](http://dx.doi.org/10.1016/j.cogpsych.2017.08.002)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1)\.
- Z\. N\. Houghton and E\. Morgan \(2023\)Does predictability drive the holistic storage of compound nouns?\.InProceedings of the 45th Annual Meeting of the Cognitive Science Society,External Links:[Link](https://escholarship.org/uc/item/7kz7w10b)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- Z\. N\. Houghton, K\. Sagae, and E\. Morgan \(2025\)The role of abstract representations and observed preferences in the ordering of binomials in large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),Vienna, Austria,pp\. 695–702\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-short.55)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1)\.
- Z\. N\. Houghton, Y\. Zhou, D\. Pluth, and V\. K\. Gurbani \(2026\)The holistic storage of verb\+up phrases in text\-based and audio\-based language models\.External Links:2606\.13993,[Link](https://arxiv.org/abs/2606.13993)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- Z\. N\. Houghton \(2025\)Multi\-word representations in minds and models: investigating the storage of multi\-word phrases in humans and large language models\.Ph\.D\. Thesis,University of California, Davis\.External Links:[Link](https://search.proquest.com/openview/59e91c3252682d11c048247ba152f037/1?pq-origsite=gscholar&cbl=18750&diss=y)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1),[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- J\. Jian and C\. D\. Manning \(2026\)Humans and transformer LMs: abstraction drives language learning\.InProceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Rabat, Morocco,pp\. 752–765\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.eacl-long.32)Cited by:[§1\.1](https://arxiv.org/html/2608.00821#S1.SS1.p1.1),[§1\.1](https://arxiv.org/html/2608.00821#S1.SS1.p2.4),[§1](https://arxiv.org/html/2608.00821#S1.p7.2),[§1](https://arxiv.org/html/2608.00821#S1.p8.1),[§2\.1\.1](https://arxiv.org/html/2608.00821#S2.SS1.SSS1.p1.1),[§2\.1\.3](https://arxiv.org/html/2608.00821#S2.SS1.SSS3.p1.1),[§3](https://arxiv.org/html/2608.00821#S3.p1.4),[§3](https://arxiv.org/html/2608.00821#S3.p3.1),[§4](https://arxiv.org/html/2608.00821#S4.p1.2),[footnote 4](https://arxiv.org/html/2608.00821#footnote4)\.
- V\. Kapatsinski \(2012\)What statistics do learners track? rules, constraints or schemas in \(artificial\) grammar learning\.InFrequency effects in language: Learning and processing,S\. Th\. Gries and D\. Divjak \(Eds\.\),pp\. 53–82\.Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1)\.
- V\. Kapatsinski \(2014\)What is grammar like? a usage\-based constructionist perspective\.Linguistic Issues in Language Technology11\(1\),pp\. 1–41\.External Links:[Link](https://aclanthology.org/2014.lilt-11.2/)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1)\.
- V\. Kapatsinski \(2018a\)Changing minds changing tools: from learning theory to language acquisition to language change\.MIT Press\.External Links:[Link](https://books.google.com/books?hl=en&lr=&id=YZxjDwAAQBAJ&oi=fnd&pg=PR5&dq=kapatsinski+changing+minds&ots=9bGhgkCaY0&sig=MHfWF9cbhbtMmx33a0FYSM6AMAs)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1),[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- V\. Kapatsinski \(2018b\)Words versus rules \(storage versus online production/processing\) in morphology\.InOxford Research Encyclopedia of Linguistics,M\. Aronoff \(Ed\.\),External Links:[Document](https://dx.doi.org/10.1093/acrefore/9780199384655.013.598)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- V\. Kapatsinski \(2021\)Hierarchical inference in sound change: words, sounds, and frequency of use\.Frontiers in Psychology12,pp\. 652664\.External Links:[Document](https://dx.doi.org/10.3389/fpsyg.2021.652664)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- V\. Kapatsinski \(2026\)Transformers perform adaptive partial pooling\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2602.03980)Cited by:[§3](https://arxiv.org/html/2608.00821#S3.p1.4)\.
- Z\. Liu and E\. Morgan \(2020\)Frequency\-dependent regularization in constituent ordering preferences\.InProceedings of the 42nd Annual Meeting of the Cognitive Science Society,pp\. 2990–2996\.External Links:[Link](https://cognitivesciencesociety.org/cogsci20/papers/0751/0751.pdf)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- B\. McMurray, M\. K\. Tanenhaus, and R\. N\. Aslin \(2002\)Gradient effects of within\-category phonetic variation on lexical access\.Cognition86\(2\),pp\. B33–B42\.External Links:[Document](https://dx.doi.org/10.1016/S0010-0277%2802%2900157-9)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- P\. Milin, B\. V\. Tucker, and D\. Divjak \(2023\)A learning perspective on the emergence of abstractions: the curious case of phone\(me\)s\.Language and Cognition15\(4\),pp\. 740–762\.External Links:[Document](https://dx.doi.org/10.1017/langcog.2023.11)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1)\.
- E\. Morgan and R\. Levy \(2015\)Modeling idiosyncratic preferences: how generative knowledge and expression frequency jointly determine language structure\.InProceedings of the 37th Annual Meeting of the Cognitive Science Society,pp\. 1649–1654\.External Links:[Link](https://escholarship.org/uc/item/7rm1r6jh)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- E\. Morgan and R\. Levy \(2016a\)Abstract knowledge versus direct experience in processing of binomial expressions\.Cognition157,pp\. 384–402\.External Links:[Document](https://dx.doi.org/10.1016/j.cognition.2016.09.011),[Link](http://dx.doi.org/10.1016/j.cognition.2016.09.011)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1),[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- E\. Morgan and R\. Levy \(2016b\)Frequency\-dependent regularization in iterated learning\.InThe Evolution of Language: Proceedings of the 11th International Conference \(EVOLANG11\),Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- E\. Morgan and R\. Levy \(2024\)Productive knowledge and item\-specific knowledge trade off as a function of frequency in multiword expression processing\.Language100\(4\),pp\. e195–e224\.External Links:[Link](https://muse.jhu.edu/pub/24/article/947046)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1)\.
- R\. M\. Nosofsky and J\. K\. Kruschke \(1992\)Investigations of an exemplar\-based connectionist model of category learning\.Psychology of Learning and Motivation28,pp\. 207–250\.External Links:[Document](https://dx.doi.org/10.1016/S0079-7421%2808%2960491-0)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- R\. M\. Nosofsky \(1986\)Attention, similarity, and the identification\-categorization relationship\.Journal of Experimental Psychology: General115\(1\),pp\. 39–61\.External Links:[Document](https://dx.doi.org/10.1037/0096-3445.115.1.39)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- J\. B\. Pierrehumbert \(2001\)Exemplar dynamics: Word frequency, lenition and contrast\.InFrequency and the Emergence of Linguistic Structure,J\. L\. Bybee and P\. J\. Hopper \(Eds\.\),pp\. 137–158\.External Links:[Document](https://dx.doi.org/10.1075/tsl.45.08pie)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- J\. B\. Pierrehumbert \(2002\)Word\-specific phonetics\.InLaboratory Phonology 7,C\. Gussenhoven and N\. Warner \(Eds\.\),pp\. 101–140\.External Links:[Document](https://dx.doi.org/10.1515/9783110197105.1.101)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- D\. Pluth, Z\. N\. Houghton, Y\. Zhou, and V\. K\. Gurbani \(2026\)Mechanistic interpretability of asr models using sparse autoencoders\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2605.12225)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1)\.
- P\. Rácz, C\. Beckner, J\. B\. Hay, and J\. B\. Pierrehumbert \(2020\)Morphological convergence as on\-Line lexical analogy\.Language96\(4\),pp\. 735–770\.External Links:[Document](https://dx.doi.org/10.1353/lan.2020.0061)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- D\. E\. Rumelhart and J\. L\. McClelland \(1986\)On learning the past tenses of english verbs\.InParallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol\. 2,J\. L\. McClelland and D\. E\. Rumelhart \(Eds\.\),pp\. 216–271\.Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- J\. P\. Stemberger and B\. MacWhinney \(1986\)Frequency and the lexical storage of regularly inflected forms\.Memory & Cognition14\(1\),pp\. 17–26\.External Links:[Document](https://dx.doi.org/10.3758/BF03209225)Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1),[§1](https://arxiv.org/html/2608.00821#S1.p4.1)\.
- J\. P\. Stemberger and B\. MacWhinney \(2004\)Are inflected forms stored in the lexicon?\.InMorphology: Critical Concepts in Linguistics,F\. Katamba \(Ed\.\),Vol\.6,pp\. 107–122\.Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p3.1)\.
- M\. Tomasello \(2005\)Constructing a language: a usage\-based theory of language acquisition\.Harvard University Press,Cambridge, MA\.Cited by:[§1](https://arxiv.org/html/2608.00821#S1.p2.1),[§1](https://arxiv.org/html/2608.00821#S1.p5.1)\.

## Appendix AEmpirical Confirmation of Near\-Uniform Starting State

Both simulations assume that the learner begins from a state where all verb distributions are near\-uniform: the model initially assigns essentially equal probability to all tokens regardless of which verb appears in the context\. This is theα=0\\alpha=0state in the Zero\-Sensitivity Learner and thenobs≈0n\_\{\\text\{obs\}\}\\approx 0state in the Variable\-Sensitivity Learner\.

To confirm this assumption for an actual transformer language model, we computed next\-token distributions for all 71 J&M verbs at training checkpoint 1 ofznhoughton/opt\-babylm\-125m\-1ep\-dense\-seed964\(OPT\-125M trained on 100M words of BabyLM data\)\. We used the unified template"They \{verb\} to the"for both verb classes\. Using a single template is essential: class\-specific templates would introduce between\-classDJSD\_\{\\text\{JS\}\}through template differences rather than verb identity\.

Table 2:Next\-token distribution statistics at checkpoint 1 of OPT\-125M \(BabyLM\)\. All 71 verbs are at 98\.3% of maximum entropy at the first checkpoint\.At checkpoint 1, all 71 verbs are at 98\.3% of maximum entropy\. Within\-class and between\-classDJSD\_\{\\text\{JS\}\}are virtually identical \(ratio=1\.002\\text\{ratio\}=1\.002\) and both fall well below theowo\_\{w\}threshold of 0\.01\. There is no class\-level signal at step 1; the model treats all 71 verbs as effectively equivalent\. This confirms the near\-uniform starting condition assumed by the simulations\.

We note two limitations\. First, this confirms the starting condition for OPT\-125M trained on BabyLM data, not for GPT\-2 on OpenWebText as in J&M\. However, the near\-uniform starting condition is a general property of randomly initialized transformers trained with standard autoregressive objectives: before sufficient gradient updates, all token predictions default to the marginal distribution of the training data, independent of verb identity\. Second, the OPT\-125M vocabulary has 8,192 tokens rather than the GPT\-2 vocabulary of 50,257, so the absolute entropy values are not directly comparable, but the 98\.3% saturation result is vocabulary\-size independent\.

## Appendix BParameter Grid

Table 3:Parameter grid\. All combinations with class\_overlap<<item\_overlap are valid \(108 structural combinations\)\.Constraint: class\_overlap<<item\_overlap \(108 valid structural combinations\)\. The Zero\-Sensitivity Learner is run over the 108 structural combinations only \(108×50=5,400108\\times 50=5\{,\}400runs\)\. The Variable\-Sensitivity Learner additionally sweepskk, giving108×5×50=27,000108\\times 5\\times 50=27\{,\}000total runs; results atk=0\.001k=0\.001are excluded from main\-text figures as they are virtually identical tok=0\.01k=0\.01\.

## Appendix CDetailed Results byμ\\mu

The following tables show the per\-μ\\mubreakdown for the Zero\-Sensitivity Learner \(the results most sensitive to structural parameters\)\. Each cell is averaged over the \(item\_overlap, class\_overlap\) pairs that yield the given overlap gap\.σ=0\.5\\sigma=0\.5is omitted:ob<ow=1\.00o\_\{b\}<o\_\{w\}=1\.00for all 20 combinations atσ=0\.5\\sigma=0\.5regardless ofμ\\mu\.

Table 4:Zero\-Sensitivity Learner,σ=1\.0\\sigma=1\.0: results by overlap gap andμ\\mu\. Format:ob<owo\_\{b\}<o\_\{w\}/ow<obo\_\{w\}<o\_\{b\}/ neither\.Table 5:Zero\-Sensitivity Learner,σ=1\.5\\sigma=1\.5: results by overlap gap andμ\\mu\. Format:ob<owo\_\{b\}<o\_\{w\}/ow<obo\_\{w\}<o\_\{b\}/ neither\.Theμ\\mueffect is most pronounced atσ=1\.5\\sigma=1\.5with small overlap gaps: higherμ\\muconsistently shifts results towardob<owo\_\{b\}<o\_\{w\}by making preferred tokens more dominant over background noise, strengthening the detectable class signal\. Atσ=1\.0\\sigma=1\.0, only the weakest class signal \(gap=0\.1=0\.1, lowμ\\mu\) deviates from universalob<owo\_\{b\}<o\_\{w\}\.

For the Variable\-Sensitivity Learner atk=0\.5k=0\.5\(the transitional regime whereμ\\mueffects are most relevant\), the sameμ\\mu\-driven pattern emerges: in general, higherμ\\muand larger gap shift results towardob<owo\_\{b\}<o\_\{w\}, consistent with the interpretation thatμ\\muamplifies the class signal relative to idiosyncratic noise \(see Appendix[D](https://arxiv.org/html/2608.00821#A4)for the formal ZSL\-VSL correspondence\)\. Full per\-μ\\muresults fork=0\.5k=0\.5are shown below \(σ=0\.5\\sigma=0\.5omitted:ob<ow≈1\.00o\_\{b\}<o\_\{w\}\\approx 1\.00throughout\)\.

Table 6:Variable\-Sensitivity Learner,k=0\.5k=0\.5: results byσ\\sigma, overlap gap, andμ\\mu\. Format:ob<owo\_\{b\}<o\_\{w\}/ow<obo\_\{w\}<o\_\{b\}/ neither\.
## Appendix DThe Zero\-Sensitivity Learner as thek→∞k\\to\\inftyLimit of the Variable\-Sensitivity Learner

In the Variable\-Sensitivity Learner, the smoothed empirical distribution at corpus sizenobsn\_\{\\text\{obs\}\}is:

P^v​\(t\)=count​\(t∣v,nobs\)\+knobs\+k⋅V\\hat\{P\}\_\{v\}\(t\)=\\frac\{\\text\{count\}\(t\\mid v,\\,n\_\{\\text\{obs\}\}\)\+k\}\{n\_\{\\text\{obs\}\}\+k\\cdot V\}
Taking the expectation over random token draws fromPvP\_\{v\}, and noting that𝔼​\[count​\(t∣v,nobs\)\]=nobs⋅Pv​\(t\)\\mathbb\{E\}\[\\text\{count\}\(t\\mid v,n\_\{\\text\{obs\}\}\)\]=n\_\{\\text\{obs\}\}\\cdot P\_\{v\}\(t\):

𝔼​\[P^v​\(t\)\]=nobs⋅Pv​\(t\)\+knobs\+k⋅V=nobsnobs\+k​V⋅Pv​\(t\)\+k​Vnobs\+k​V⋅1V\\mathbb\{E\}\[\\hat\{P\}\_\{v\}\(t\)\]=\\frac\{n\_\{\\text\{obs\}\}\\cdot P\_\{v\}\(t\)\+k\}\{n\_\{\\text\{obs\}\}\+k\\cdot V\}=\\frac\{n\_\{\\text\{obs\}\}\}\{n\_\{\\text\{obs\}\}\+kV\}\\cdot P\_\{v\}\(t\)\+\\frac\{kV\}\{n\_\{\\text\{obs\}\}\+kV\}\\cdot\\frac\{1\}\{V\}
Definingα​\(nobs\)=nobs/\(nobs\+k​V\)\\alpha\(n\_\{\\text\{obs\}\}\)=n\_\{\\text\{obs\}\}/\(n\_\{\\text\{obs\}\}\+kV\), this is exactly:

𝔼​\[P^v​\(nobs\)\]=α⋅Pv\+\(1−α\)⋅Uniform\\mathbb\{E\}\[\\hat\{P\}\_\{v\}\(n\_\{\\text\{obs\}\}\)\]=\\alpha\\cdot P\_\{v\}\+\(1\-\\alpha\)\\cdot\\text\{Uniform\}
which is Equation[4](https://arxiv.org/html/2608.00821#S2.E4), the Zero\-Sensitivity Learner’s interpolation formula, with learning progressα\\alpha\. The variance ofP^v\\hat\{P\}\_\{v\}around this expectation shrinks askkgrows, vanishing entirely ask→∞k\\to\\infty: the Variable\-Sensitivity Learner converges to the deterministic Zero\-Sensitivity Learner trajectory at eachα\\alphastep\.

This derivation has two important consequences\. First, it establishes that the Zero\-Sensitivity Learner and the Variable\-Sensitivity Learner are not categorically different learners: they are the same model at different levels of sensitivity, withkkas the governing parameter\. “Monotone trajectory” is not a mechanistically distinct property of the Zero\-Sensitivity Learner; it describes the expected behavior of Equation[6](https://arxiv.org/html/2608.00821#S2.E6)in the limit of largekk\. Second, it means that theα\\alphaaxis in the Zero\-Sensitivity Learner and thenobsn\_\{\\text\{obs\}\}axis in the Variable\-Sensitivity Learner are directly related: for a givenkk,α=nobs/\(nobs\+k​V\)\\alpha=n\_\{\\text\{obs\}\}/\(n\_\{\\text\{obs\}\}\+kV\), so the same learning stage corresponds to different corpus sizes depending onkk\. A learner withk=0\.5k=0\.5atnobs=500n\_\{\\text\{obs\}\}=500has exactlyα=0\.5\\alpha=0\.5, the same learning progress as the Zero\-Sensitivity Learner at that point in its trajectory\.

## Appendix EZipfian Verb Frequency Extension \(Model 3\)

The Variable\-Sensitivity Learner assumes all verbs are observed equally often at every checkpoint\. Model 3 relaxes this by allocating observations across verbs according to a Zipfian distribution: a total budget ofNtotalN\_\{\\text\{total\}\}tokens is drawn, and at each draw a verb is sampled with probability proportional to1/rs1/r^\{s\}\(whererris its randomly assigned frequency rank andssis the Zipf exponent\), followed by sampling a single token from that verb’s true distributionPvP\_\{v\}\. Checkpoints are defined on theNtotalN\_\{\\text\{total\}\}axis; at each checkpoint, different verbs have accumulated different numbers of observations depending on their rank\. The mean observations per verb at the maximum checkpoint equals 5,000, matching Model 2’s maximumnobsn\_\{\\text\{obs\}\}\.

The qualitative pattern from Model 2 is preserved: askkincreases, the fraction of seeds showingob<owo\_\{b\}<o\_\{w\}rises and the fraction showingow<obo\_\{w\}<o\_\{b\}falls\. Atk=0\.01k=0\.01, virtually all detected seeds showow<obo\_\{w\}<o\_\{b\}\(94%\), while atk=5k=5,ob<owo\_\{b\}<o\_\{w\}is the dominant outcome \(51% vs\. 25%\)\. The key difference from Model 2 is that the crossover is shifted to higherkk: in the equal\-frequency learner,k=0\.5k=0\.5already reaches the transition point, whereas in the Zipfian model it falls aroundk=3k=3–55\. This shift arises because rare verbs accumulate few observations regardless of total input size, leaving their item\-specific distributions heavily smoothed toward the background untilkkis large enough to overcome this\. Within the Zipfian model, steeper frequency skew \(s=1\.5s=1\.5\) further delays theobo\_\{b\}onset relative to shallower skew \(s=0\.5s=0\.5\), because rare verbs require disproportionately more total tokens to reveal item\-specific structure\. Figure[4](https://arxiv.org/html/2608.00821#A5.F4)shows this transition across the full\(σ,gap\)\(\\sigma,\\text\{gap\}\)parameter space, averaged overμ\\muandss; the full grid results broken down byss\(50 seeds per combination\) are shown in TableLABEL:tbl\-model3\-full\.

![Refer to caption](https://arxiv.org/html/2608.00821v1/x3.png)Figure 4:Zipfian Variable\-Sensitivity Learner: fraction of seeds withob<owo\_\{b\}<o\_\{w\}bykk,σ\\sigma, and overlap gap, averaged overμ∈\{10,30,60,100\}\\mu\\in\\\{10,30,60,100\\\}and Zipf exponents∈\{0\.5,1\.0,1\.5\}s\\in\\\{0\.5,1\.0,1\.5\\\}\. Blue =ob<owo\_\{b\}<o\_\{w\}dominates; red =ow<obo\_\{w\}<o\_\{b\}dominates\. Compare to the equal\-frequency Variable\-Sensitivity Learner \(Figure[3](https://arxiv.org/html/2608.00821#S2.F3)\): the transition shifts to substantially higherkk\.Table 7:Model 3 \(Zipfian\): fraction of seeds bykk,ss,σ\\sigma, and overlap gap, averaged overμ\\mu\. Format:ob<owo\_\{b\}<o\_\{w\}/ow<obo\_\{w\}<o\_\{b\}\.kkssσ\\sigmagap = 0\.1gap = 0\.2gap = 0\.3gap = 0\.4gap = 0\.5k=0\.001k=0\.001s=0\.5s=0\.50\.50\.00 / 1\.000\.00 / 1\.000\.00 / 1\.000\.00 / 1\.000\.00 / 1\.00k=0\.001k=0\.001s=0\.5s=0\.51\.00\.00 / 0\.830\.00 / 1\.000\.00 / 1\.000\.00 / 1\.000\.00 / 1\.00k=0\.001k=0\.001s=0\.5s=0\.51\.50\.00 / 0\.710\.00 / 0\.850\.00 / 1\.000\.00 / 1\.000\.00 / 1\.00k=0\.001k=0\.001s=1s=10\.50\.00 / 0\.780\.00 / 1\.000\.00 / 1\.000\.00 / 0\.990\.00 / 1\.00k=0\.001k=0\.001s=1s=11\.00\.00 / 0\.760\.00 / 0\.860\.00 / 1\.000\.00 / 0\.990\.00 / 1\.00k=0\.001k=0\.001s=1s=11\.50\.00 / 0\.610\.00 / 0\.780\.00 / 0\.860\.00 / 0\.980\.00 / 1\.00k=0\.001k=0\.001s=1\.5s=1\.50\.50\.00 / 0\.760\.00 / 0\.870\.00 / 1\.000\.00 / 1\.000\.00 / 1\.00k=0\.001k=0\.001s=1\.5s=1\.51\.00\.00 / 0\.710\.00 / 0\.810\.00 / 0\.960\.00 / 1\.000\.00 / 1\.00k=0\.001k=0\.001s=1\.5s=1\.51\.50\.00 / 0\.520\.00 / 0\.760\.00 / 0\.800\.00 / 0\.870\.00 / 0\.99k=0\.01k=0\.01s=0\.5s=0\.50\.50\.00 / 1\.000\.00 / 1\.000\.00 / 1\.000\.00 / 1\.000\.00 / 1\.00k=0\.01k=0\.01s=0\.5s=0\.51\.00\.00 / 0\.850\.00 / 1\.000\.00 / 1\.000\.00 / 1\.000\.00 / 1\.00k=0\.01k=0\.01s=0\.5s=0\.51\.50\.00 / 0\.710\.00 / 0\.910\.00 / 1\.000\.00 / 1\.000\.00 / 1\.00k=0\.01k=0\.01s=1s=10\.50\.00 / 0\.820\.00 / 1\.000\.00 / 1\.000\.00 / 0\.990\.00 / 1\.00k=0\.01k=0\.01s=1s=11\.00\.00 / 0\.770\.00 / 0\.960\.00 / 1\.000\.00 / 0\.990\.00 / 1\.00k=0\.01k=0\.01s=1s=11\.50\.00 / 0\.600\.00 / 0\.800\.00 / 0\.920\.00 / 0\.990\.00 / 1\.00k=0\.01k=0\.01s=1\.5s=1\.50\.50\.00 / 0\.760\.00 / 0\.920\.00 / 1\.000\.00 / 1\.000\.00 / 1\.00k=0\.01k=0\.01s=1\.5s=1\.51\.00\.00 / 0\.720\.00 / 0\.860\.00 / 0\.990\.00 / 1\.000\.00 / 1\.00k=0\.01k=0\.01s=1\.5s=1\.51\.50\.00 / 0\.530\.00 / 0\.780\.00 / 0\.830\.00 / 0\.940\.00 / 0\.99k=0\.1k=0\.1s=0\.5s=0\.50\.50\.00 / 1\.000\.00 / 1\.000\.00 / 1\.000\.00 / 0\.990\.00 / 1\.00k=0\.1k=0\.1s=0\.5s=0\.51\.00\.00 / 0\.900\.00 / 1\.000\.00 / 1\.000\.00 / 1\.000\.00 / 1\.00k=0\.1k=0\.1s=0\.5s=0\.51\.50\.00 / 0\.710\.00 / 0\.920\.00 / 1\.000\.00 / 1\.000\.00 / 1\.00k=0\.1k=0\.1s=1s=10\.50\.00 / 0\.950\.00 / 1\.000\.00 / 1\.000\.01 / 0\.990\.02 / 0\.98k=0\.1k=0\.1s=1s=11\.00\.00 / 0\.810\.00 / 1\.000\.00 / 1\.000\.01 / 0\.990\.02 / 0\.98k=0\.1k=0\.1s=1s=11\.50\.00 / 0\.640\.00 / 0\.830\.00 / 0\.980\.01 / 0\.990\.02 / 0\.98k=0\.1k=0\.1s=1\.5s=1\.50\.50\.02 / 0\.820\.00 / 1\.000\.00 / 1\.000\.00 / 1\.000\.00 / 1\.00k=0\.1k=0\.1s=1\.5s=1\.51\.00\.02 / 0\.760\.00 / 0\.970\.00 / 1\.000\.00 / 1\.000\.00 / 1\.00k=0\.1k=0\.1s=1\.5s=1\.51\.50\.02 / 0\.510\.00 / 0\.800\.00 / 0\.880\.00 / 1\.000\.00 / 1\.00k=0\.5k=0\.5s=0\.5s=0\.50\.50\.02 / 0\.980\.02 / 0\.970\.03 / 0\.920\.04 / 0\.730\.12 / 0\.49k=0\.5k=0\.5s=0\.5s=0\.51\.00\.02 / 0\.970\.02 / 0\.980\.01 / 0\.960\.01 / 0\.920\.06 / 0\.71k=0\.5k=0\.5s=0\.5s=0\.51\.50\.02 / 0\.600\.01 / 0\.860\.01 / 0\.990\.00 / 0\.970\.04 / 0\.91k=0\.5k=0\.5s=1s=10\.50\.00 / 1\.000\.00 / 1\.000\.00 / 0\.990\.03 / 0\.950\.04 / 0\.84k=0\.5k=0\.5s=1s=11\.00\.00 / 0\.940\.00 / 1\.000\.01 / 0\.990\.03 / 0\.970\.04 / 0\.93k=0\.5k=0\.5s=1s=11\.50\.00 / 0\.470\.00 / 0\.850\.00 / 0\.990\.03 / 0\.970\.04 / 0\.95k=0\.5k=0\.5s=1\.5s=1\.50\.50\.02 / 0\.950\.00 / 1\.000\.00 / 1\.000\.00 / 1\.000\.02 / 0\.98k=0\.5k=0\.5s=1\.5s=1\.51\.00\.02 / 0\.550\.00 / 0\.990\.00 / 1\.000\.00 / 1\.000\.02 / 0\.98k=0\.5k=0\.5s=1\.5s=1\.51\.50\.02 / 0\.080\.00 / 0\.600\.00 / 0\.830\.00 / 0\.970\.01 / 0\.98k=1k=1s=0\.5s=0\.50\.50\.03 / 0\.940\.12 / 0\.520\.54 / 0\.260\.73 / 0\.230\.76 / 0\.20k=1k=1s=0\.5s=0\.51\.00\.02 / 0\.960\.03 / 0\.930\.08 / 0\.700\.28 / 0\.410\.54 / 0\.24k=1k=1s=0\.5s=0\.51\.50\.02 / 0\.490\.01 / 0\.820\.01 / 0\.970\.04 / 0\.900\.15 / 0\.66k=1k=1s=1s=10\.50\.00 / 1\.000\.01 / 0\.940\.07 / 0\.570\.32 / 0\.340\.61 / 0\.28k=1k=1s=1s=11\.00\.00 / 0\.920\.01 / 0\.980\.02 / 0\.940\.06 / 0\.710\.23 / 0\.47k=1k=1s=1s=11\.50\.00 / 0\.180\.00 / 0\.810\.00 / 0\.960\.04 / 0\.950\.05 / 0\.90k=1k=1s=1\.5s=1\.50\.50\.02 / 0\.800\.00 / 0\.990\.00 / 0\.980\.01 / 0\.880\.05 / 0\.59k=1k=1s=1\.5s=1\.51\.00\.02 / 0\.310\.00 / 0\.820\.00 / 0\.980\.00 / 0\.990\.02 / 0\.90k=1k=1s=1\.5s=1\.51\.50\.02 / 0\.060\.00 / 0\.350\.00 / 0\.710\.00 / 0\.790\.01 / 0\.88k=3k=3s=0\.5s=0\.50\.50\.20 / 0\.530\.85 / 0\.011\.00 / 0\.001\.00 / 0\.001\.00 / 0\.00k=3k=3s=0\.5s=0\.51\.00\.06 / 0\.800\.25 / 0\.540\.72 / 0\.190\.84 / 0\.040\.95 / 0\.00k=3k=3s=0\.5s=0\.51\.50\.04 / 0\.200\.04 / 0\.760\.05 / 0\.820\.19 / 0\.620\.50 / 0\.29k=3k=3s=1s=10\.50\.04 / 0\.890\.72 / 0\.170\.90 / 0\.010\.99 / 0\.001\.00 / 0\.00k=3k=3s=1s=11\.00\.02 / 0\.360\.06 / 0\.790\.47 / 0\.330\.72 / 0\.100\.88 / 0\.01k=3k=3s=1s=11\.50\.00 / 0\.060\.01 / 0\.430\.02 / 0\.760\.04 / 0\.800\.20 / 0\.61k=3k=3s=1\.5s=1\.50\.50\.02 / 0\.780\.05 / 0\.700\.66 / 0\.060\.93 / 0\.001\.00 / 0\.00k=3k=3s=1\.5s=1\.51\.00\.02 / 0\.120\.01 / 0\.760\.07 / 0\.630\.39 / 0\.320\.67 / 0\.16k=3k=3s=1\.5s=1\.51\.50\.02 / 0\.010\.01 / 0\.110\.01 / 0\.570\.02 / 0\.720\.05 / 0\.67k=5k=5s=0\.5s=0\.50\.50\.48 / 0\.280\.99 / 0\.001\.00 / 0\.001\.00 / 0\.001\.00 / 0\.00k=5k=5s=0\.5s=0\.51\.00\.06 / 0\.660\.36 / 0\.350\.87 / 0\.031\.00 / 0\.001\.00 / 0\.00k=5k=5s=0\.5s=0\.51\.50\.04 / 0\.110\.04 / 0\.680\.06 / 0\.720\.28 / 0\.480\.63 / 0\.22k=5k=5s=1s=10\.50\.10 / 0\.720\.97 / 0\.001\.00 / 0\.001\.00 / 0\.001\.00 / 0\.00k=5k=5s=1s=11\.00\.02 / 0\.200\.13 / 0\.570\.72 / 0\.170\.91 / 0\.010\.99 / 0\.00k=5k=5s=1s=11\.50\.00 / 0\.030\.01 / 0\.240\.02 / 0\.660\.11 / 0\.600\.35 / 0\.40k=5k=5s=1\.5s=1\.50\.50\.02 / 0\.760\.36 / 0\.230\.96 / 0\.001\.00 / 0\.001\.00 / 0\.00k=5k=5s=1\.5s=1\.51\.00\.02 / 0\.060\.02 / 0\.720\.37 / 0\.270\.79 / 0\.050\.95 / 0\.00k=5k=5s=1\.5s=1\.51\.50\.02 / 0\.000\.01 / 0\.070\.01 / 0\.510\.04 / 0\.610\.17 / 0\.42k=10k=10s=0\.5s=0\.50\.50\.68 / 0\.000\.75 / 0\.000\.75 / 0\.000\.75 / 0\.000\.75 / 0\.00k=10k=10s=0\.5s=0\.51\.00\.07 / 0\.360\.64 / 0\.160\.98 / 0\.001\.00 / 0\.001\.00 / 0\.00k=10k=10s=0\.5s=0\.51\.50\.03 / 0\.050\.03 / 0\.430\.08 / 0\.610\.37 / 0\.310\.72 / 0\.14k=10k=10s=1s=10\.50\.27 / 0\.220\.75 / 0\.000\.75 / 0\.000\.75 / 0\.000\.75 / 0\.00k=10k=10s=1s=11\.00\.03 / 0\.070\.32 / 0\.300\.89 / 0\.031\.00 / 0\.001\.00 / 0\.00k=10k=10s=1s=11\.50\.00 / 0\.020\.01 / 0\.080\.03 / 0\.530\.17 / 0\.470\.52 / 0\.21k=10k=10s=1\.5s=1\.50\.50\.03 / 0\.280\.68 / 0\.000\.75 / 0\.000\.75 / 0\.000\.75 / 0\.00k=10k=10s=1\.5s=1\.51\.00\.01 / 0\.030\.07 / 0\.540\.66 / 0\.030\.75 / 0\.000\.76 / 0\.00k=10k=10s=1\.5s=1\.51\.50\.02 / 0\.000\.01 / 0\.020\.02 / 0\.360\.11 / 0\.480\.46 / 0\.20Table 7:Model 3 \(Zipfian\): fraction of seeds bykk,ss,σ\\sigma, and overlap gap, averaged overμ\\mu\. Format:ob<owo\_\{b\}<o\_\{w\}/ow<obo\_\{w\}<o\_\{b\}\.\(continued\)
## Appendix FFull Numerical Results

Raw numerical results underlying Figure[1](https://arxiv.org/html/2608.00821#S2.F1)and Figure[3](https://arxiv.org/html/2608.00821#S2.F3)\. Values show the fraction of seeds with each outcome, averaged overμ∈\{10,30,60,100\}\\mu\\in\\\{10,30,60,100\\\}\.

Zero\-Sensitivity Learner\(Figure[1](https://arxiv.org/html/2608.00821#S2.F1)\)\. Format:ob<owo\_\{b\}<o\_\{w\}/ow<obo\_\{w\}<o\_\{b\}/ neither\.

Table 8:Zero\-Sensitivity Learner: fraction of seeds byσ\\sigmaand overlap gap\.Variable\-Sensitivity Learner\(Figure[3](https://arxiv.org/html/2608.00821#S2.F3)\)\. Format:ob<owo\_\{b\}<o\_\{w\}/ow<obo\_\{w\}<o\_\{b\}/ tie / neither\.

Table 9:Variable\-Sensitivity Learner: fraction of seeds bykk,σ\\sigma, and overlap gap \(allkkvalues includingk=0\.001k=0\.001\)\.
## Appendix GThreshold Robustness

Theobo\_\{b\}criterion depends on two thresholds: CLASS\_FRAC \(the minimum fraction of verbs that must pass the Mann\-Whitney test\) and the Mann\-Whitney significance cutoffp<0\.001p<0\.001\. Theowo\_\{w\}criterion depends on DJS\_THRESH \(the absolute within\-classDJSD\_\{\\text\{JS\}\}threshold\) and SUSTAIN \(the required number of consecutive steps above threshold, fixed at 3 throughout\)\. Baseline values are CLASS\_FRAC=0\.10=0\.10and DJS\_THRESH=0\.01=0\.01\.

We note that these thresholds are not directly taken from Jian and Manning \(2026\)’s Experiment 1: J&M identify onsets visually from plotted trajectories for GPT\-2 and do not specify a formal CLASS\_FRAC forobo\_\{b\}or a formal DJS\_THRESH forowo\_\{w\}in that experiment\. The DJS\_THRESH value is borrowed from their Experiment 2 breakpoint criterion \(sustainedDJSD\_\{\\text\{JS\}\}at least 0\.01 above a per\-verb baseline\), applied here to mean within\-classDJSD\_\{\\text\{JS\}\}across verb pairs\. A direct calibration check against J&M’s GPT\-2 trajectories is not possible: their intermediate training checkpoints are not publicly available\.

To assess robustness, we precomputed the fullDJSD\_\{\\text\{JS\}\}\-matrix trajectory \(50 seeds each\) for two representative parameter combinations: one that producesob<owo\_\{b\}<o\_\{w\}\(Zero\-Sensitivity Learner,μ=100\\mu=100,σ=0\.5\\sigma=0\.5, item\_overlap=0\.7=0\.7, class\_overlap=0\.2=0\.2\) and one that producesow<obo\_\{w\}<o\_\{b\}\(Variable\-Sensitivity Learner,k=0\.001k=0\.001, same structural parameters\)\. We then applied onset detection across a grid of CLASS\_FRAC∈\{0\.01,0\.05,0\.10,0\.20,0\.30\}\\in\\\{0\.01,0\.05,0\.10,0\.20,0\.30\\\}and DJS\_THRESH∈\{0\.001,0\.005,0\.010,0\.020,0\.050\}\\in\\\{0\.001,0\.005,0\.010,0\.020,0\.050\\\}without recomputing the DJS matrices\.

Table 10:Case 1 \(Zero\-Sensitivity Learner,ob<owo\_\{b\}<o\_\{w\}\): fraction of seeds withob<owo\_\{b\}<o\_\{w\}across all threshold combinations\.ow<ob=0\.00o\_\{w\}<o\_\{b\}=0\.00and neither=0\.00=0\.00throughout\.Table 11:Case 2 \(Variable\-Sensitivity Learner,k=0\.001k=0\.001,ow<obo\_\{w\}<o\_\{b\}\): fraction of seeds withow<obo\_\{w\}<o\_\{b\}, which holds in 100% of seeds across every threshold combination tested;ob<ow=0\.00o\_\{b\}<o\_\{w\}=0\.00throughout\. Results are invariant to both CLASS\_FRAC and DJS\_THRESH because within\-classDJSD\_\{\\text\{JS\}\}spikes to≈0\.5\\approx 0\.5within the first few observations, trivially exceeding any tested DJS\_THRESH, whileobo\_\{b\}requires several observations to accumulate even a single passing verb at any tested CLASS\_FRAC\.Both orderings are perfectly robust to threshold choice:ob<owo\_\{b\}<o\_\{w\}holds in 100% of seeds across all 25 combinations in Case 1, andow<obo\_\{w\}<o\_\{b\}holds in 100% of seeds across all 25 combinations in Case 2; the reverse ordering never occurs in either case, including at our baseline CLASS\_FRAC=0\.10=0\.10\. The results are invariant to DJS\_THRESH because at lowkk, within\-classDJSD\_\{\\text\{JS\}\}immediately far exceeds any reasonable threshold, and invariant to CLASS\_FRAC because between\-class structure takes measurably longer to accumulate regardless of how permissive the threshold is set\.