追踪语言习得中的形态、语义与分化
摘要
该论文利用 Pythia 和 OLMo 等开源模型的中间检查点，系统研究了大语言模型在训练过程中形态结构、嵌入漂移、重复主导与共现效应的演化规律，发现模型在语义嵌入稳定之前就已习得形态结构，且训练数据规模有助于发散识别，而共现和重复则有负面影响。
arXiv:2610.19041v1 Announce Type: new
Abstract: Understanding the dynamics of language acquisition in Large Language Models (LLMs) is crucial for uncovering how statistical representations evolve into structured linguistic capabilities. However, most existing studies focus primarily on fully trained models, leaving the developmental trajectory across intermediate training stages largely underexplored. In this work, we leverage intermediate checkpoints from open-source models (e.g., Pythia and OLMo) under teacher-forcing evaluation to systematically investigate language acquisition across training steps and model scales. We focus on four core characteristics: morphological structure acquisition, embedding drift, repetition dominance, and co-occurrence effects. Our findings reveal that models acquire morphological structures well before semantic embeddings stabilize. Furthermore, we show that training dataset size has a positive impact on divergence recognition, while co-occurrence during pre-training and repetition during inference have a negative impact on divergence recognition and associative link generation.
查看缓存全文
缓存时间: 2026/10/15 06:48
# Tracing Morphology, Semantics, and Divergence in Language Acquisition
Source: [https://arxiv.org/html/2610.19041](https://arxiv.org/html/2610.19041)
Haoshuo FuLiang FengNing YanMasood S\. MortazaviAffiliation:Futurewei Technologies, USAAffiliation:\{fshirin, hfu, lfeng, yan\.ningyan, masood\.mortazavi\}@futurewei\.com
###### Abstract
Understanding the dynamics of language acquisition in Large Language Models \(LLMs\) is crucial for uncovering how statistical representations evolve into structured linguistic capabilities\. However, most existing studies focus primarily on fully trained models, leaving the developmental trajectory across intermediate training stages largely underexplored\. In this work, we leverage intermediate checkpoints from open\-source models \(e\.g\., Pythia and OLMo\) under teacher\-forcing evaluation to systematically investigate language acquisition across training steps and model scales\. We focus on four core characteristics: morphological structure acquisition, embedding drift, repetition dominance, and co\-occurrence effects\. Our findings reveal that models acquire morphological structures well before semantic embeddings stabilize\. Furthermore, we show that training dataset size has a positive impact on divergence recognition, while co\-occurrence during pre\-training and repetition during inference have a negative impact on divergence recognition and associative link generation\.
## 1Introduction
Linguists and cognitive scientists have developed multiple explanations and theories for “Plato’s Problem”—how human beings come to acquire and then creatively use a native language \(and other cognitive faculties\) having been exposed to minimal experience at an early age\[[34](https://arxiv.org/html/2610.19041#bib.bib27),[8](https://arxiv.org/html/2610.19041#bib.bib28),[26](https://arxiv.org/html/2610.19041#bib.bib29),[12](https://arxiv.org/html/2610.19041#bib.bib30)\]\. Language acquisition includes learning morphological structures and semantic representations\[[13](https://arxiv.org/html/2610.19041#bib.bib31),[37](https://arxiv.org/html/2610.19041#bib.bib15),[17](https://arxiv.org/html/2610.19041#bib.bib13),[20](https://arxiv.org/html/2610.19041#bib.bib32),[19](https://arxiv.org/html/2610.19041#bib.bib33),[33](https://arxiv.org/html/2610.19041#bib.bib34),[23](https://arxiv.org/html/2610.19041#bib.bib35),[28](https://arxiv.org/html/2610.19041#bib.bib36)\]\. Statistical induction of morphological inflections helps LLMs handle syntactic requirements, referential agreements, complex vocabularies, meaning generalization \(to rare or invented words\), potential translation or grammar errors, and semantic similarity of words\[[29](https://arxiv.org/html/2610.19041#bib.bib14),[27](https://arxiv.org/html/2610.19041#bib.bib38)\]\.
Unlike humans, who acquire linguistic skills from real\-life experience, large language models \(LLMs\) learn to generate text through statistical\-learning \(e\.g\., next\-token prediction\) on massive text corpora\. However, the evolution of linguistic capabilities across time \(training steps\) and scale \(model size\) remains underexplored\. To investigate the structure of language acquisition by LLMs, we study and analyze four underlying characteristics:\(i\)morphological inflection\[[9](https://arxiv.org/html/2610.19041#bib.bib12)\];\(ii\)embedding drift\[[38](https://arxiv.org/html/2610.19041#bib.bib19),[22](https://arxiv.org/html/2610.19041#bib.bib20)\];\(iii\)recall and repetition\[[2](https://arxiv.org/html/2610.19041#bib.bib21)\]; and\(iv\)co\-occurrence\[[18](https://arxiv.org/html/2610.19041#bib.bib23)\]\.
We explore how these characteristics influence two linguistic creativity tests–Remote Associates Test \(RAT\)\[[6](https://arxiv.org/html/2610.19041#bib.bib26)\]and a variation on Divergent Association Task \(DAT\)\[[25](https://arxiv.org/html/2610.19041#bib.bib24)\]\. Our modified DAT measures the competence to recognize semantic divergence\. RAT measures the competence to recall a single linking word that associates with each member of a set of diverging ones\. Although there may be disagreements about how to define or measure linguistic and other forms of creativity\[[15](https://arxiv.org/html/2610.19041#bib.bib39)\], we investigate related and prerequisite competencies and how they evolve during language acquisition:
- •We evaluate the co\-evolution of morphological structures and embedding drift for Pythia suite\[[5](https://arxiv.org/html/2610.19041#bib.bib2)\]and OLMo3\[[32](https://arxiv.org/html/2610.19041#bib.bib3)\]111To save space, similar results for GPT\-2\[[31](https://arxiv.org/html/2610.19041#bib.bib5),[30](https://arxiv.org/html/2610.19041#bib.bib6)\]and OLMo2\[[24](https://arxiv.org/html/2610.19041#bib.bib4)\]have been placed in App\.[D\.1](https://arxiv.org/html/2610.19041#A4.SS1)\-[D\.5](https://arxiv.org/html/2610.19041#A4.SS5)\.\. To our knowledge, theevolutionof morphological competence across time and scale has not been studied earlier\.
- •We show that, in the open\-source language models we have studied, the onset and maturity of morphological competence occur before the onset and maturity of semantic competence\.
- •We investigate the evolution of linguistic creativity during training based on RAT \(for link generation\) and our modified version of DAT \(for divergence recognition\)\. To date, no previous studies have explored the evolution of creativity measures or how the acquisition of other linguistic competencies or characteristics might impact them\.
- •We also discover that although larger volumes of language\-acquisition data can positively impact divergence recognition, repetition and co\-occurrence have a negative impact on both of our creativity measures\.
## 2Related Work
### 2\.1Morphology Acquisition
Morphemes are the smallest semantic units in any natural language\. Their inflections produce colors of added meaning in sentential contexts\.[Saeed et al\. \[29\]](https://arxiv.org/html/2610.19041#bib.bib14)investigated morphological induction competence in LLMs with rich morphologically structures focusing on tense, gender, number, and case\.[Ismayilzada et al\. \[17\]](https://arxiv.org/html/2610.19041#bib.bib13)showed that LLMs struggle with these abilities \(more so in agglutinative languages\) because they rely heavily on memorized lexical associations\. The authors in\[[3](https://arxiv.org/html/2610.19041#bib.bib16)\]and\[[10](https://arxiv.org/html/2610.19041#bib.bib17)\]showed Byte Pair Encoding \(BPE\)\[[21](https://arxiv.org/html/2610.19041#bib.bib11)\]disregards meaningful morpheme boundaries resulting in suboptimal and inconsistent segmentation, ambiguity, and disrupted semantic coherence \(more so in morphologically rich languages\)\. BPE is a fundamental tokenizer, widely used in different foundation models, such as Pythia\[[5](https://arxiv.org/html/2610.19041#bib.bib2)\]and OLMo\[[32](https://arxiv.org/html/2610.19041#bib.bib3),[24](https://arxiv.org/html/2610.19041#bib.bib4)\]\.
### 2\.2Semantic Acquisition
[Peng et al\. \[27\]](https://arxiv.org/html/2610.19041#bib.bib38)investigate LLMs’ vector embeddings for semantic similarity computation and analysis, observing that stronger LLMs display better performance in semantic similarity tasks\.[Xu et al\. \[38\]](https://arxiv.org/html/2610.19041#bib.bib19)observed three distinct stages in semantic feature evolution:\(i\)Initialization and Warmup Phase—behaving as random noise with activations lacking clear semantic structure;\(ii\)Emergent Phase—experiencing “geometric convergence” where similar data points cluster together into distinct regions, and gaining recognizable semantic meaning \(e\.g\., separating verbs from nouns\); and\(iii\)Convergent Phase—entering a consistent period wherein semantic meaning does not change rapidly and the internal representation reaches its mature structure\.
### 2\.3Repetition and Co\-occurrence
[Armeni et al\. \[2\]](https://arxiv.org/html/2610.19041#bib.bib21)underscored the effect of repetition and short\-term memory \(STM\) across time and scale\. Employing verbatim in\-context retrieval as a measure of artificial STM, they demonstrated that transformers achieve STM early in their training, and that STM correlates positively with an array of LLM performance metrics\. Token co\-occurrence reinforces the link between two tokens based on their contextual co\-occurrences\. Observing that larger LLMs are more capable of learning more complex token patterns than smaller ones,[Yan et al\. \[39\]](https://arxiv.org/html/2610.19041#bib.bib22)explored token reinforcement as models seek to maximize next\-token likelihoods\.
## 3Model Selection
We present our findings for LLMs with publicly available language\-training checkpoints222Due to space limitations, evaluations for other models, i\.e\., OLMo2 and GPT\-2, can be found in Appendix[D](https://arxiv.org/html/2610.19041#A4), and extra information about the models’ architecture and training checkpoints are available in Appendices[A](https://arxiv.org/html/2610.19041#A1)and[B](https://arxiv.org/html/2610.19041#A2)\.:\(i\)Pythia\[[5](https://arxiv.org/html/2610.19041#bib.bib2)\]is a suite of decoder\-only language models ranging from 14M to 12B parameters, including 154 intermediate checkpoints\. The models were trained on the Pile dataset\[[14](https://arxiv.org/html/2610.19041#bib.bib8)\], a 300B\-token English\-only corpus containing texts from 22 sources\. We evaluate \{70M, 160M, 410M, 1B, 6\.9B, 12B\} model sizes at 27 training checkpoints333To save space, we provide results for 70M and 12B\. Results for other scales are available in Appendix\.\. All checkpoints were accessed through the HuggingFace Transformers library\[[36](https://arxiv.org/html/2610.19041#bib.bib10)\]; and\(ii\)Open Language Model \(OLMo\)–version 3 \(OLMo3\)\[[32](https://arxiv.org/html/2610.19041#bib.bib3)\]comes in two sizes \(7B and 32B\) and four core variants \(Base, Instruct, Think, and RL Zero\)\. We evaluate OLMO3\-7B\-Base
trained on the 5\.9T\-token Dolma\-3 dataset \(mix of web text and academic data\) through three stages—language acquisition, reasoning acquisition, and long\-context extensions\.
## 4Tracing Morphological Preferences
Among common lexical categories\[[1](https://arxiv.org/html/2610.19041#bib.bib18)\]—nouns, verbs, adjectives, adverbs, determiners, auxiliary verbs, prepositions, conjunctions, pronouns, and interjections—we focus on three:\(i\)Verbs: tense inflections and modal auxiliaries;\(ii\)Nouns: derivational affixes, singular and plural forms; and\(iii\)Pronouns: possessive and gender agreements\.For each of these categories, we design a number of unit tests444Designing a comprehensive test for LLMs to learning the morphological structures is beyond the scope of this paper\. The simple test here covers the defined lexical categories\. Also, our teacher\-forcing setup cannot be employed to investigate morphological learning profiles of certain lexical categories, such as personal pronouns, as they start the incipit prompts\.\.
\(a\)Pythia–Without closing token\.
\(b\)Pythia–With closing token "\."\.
Figure 1:Without a closing token, target probabilities shift due to valid alternate continuations after the presumably unfavorable inflection\.Each case includes anincipitprompt followed by atarget–an inflection of a morpheme followed by one or moreclosingtokens\. Each incipit prompt is provided to the LLM, and the generation probability of the target is computed via teacher forcing\[[35](https://arxiv.org/html/2610.19041#bib.bib1)\]\. The closing tokens would ensure the proper exclusion \(or attenuation\) of alternate continuations \(see Fig\.[1](https://arxiv.org/html/2610.19041#S4.F1)\)\. Using simple grammatical rules, we construct many \(favorable,unfavorable\) target pairs—for both of which we track changes in generation probabilities over training steps, expecting the favorable target generation probability to surpass unfavorable one\. For example, for the incipit prompt “I love”, we may have favorable and unfavorable targets “books\.” and “book\.”\. Whenever targetttis broken intoNtN\_\{t\}tokens, the probability oftt,PtP\_\{t\}, is the product of its token probabilitiespip\_\{i\}conditioned on the corresponding prior token sequencexix\_\{i\}that starts with the incipitxx,log\(Pt\|x\)=∑i=1Ntlog\(pi\|xi\)log\(P\_\{t\}\|x\)=\\sum\_\{i=1\}^\{N\_\{t\}\}\\log\(p\_\{i\}\|x\_\{i\}\)\. As language acquisition \(training\) progresses, we expect increasingly higher probabilities for favorable target morphological inflections compared to unfavorable ones\.
### 4\.1Morphological Structures per Lexical Category
Figure[2](https://arxiv.org/html/2610.19041#S4.F2)shows the acquisition trajectories for one representative case in each of the selected lexical categories\. The blue and red graphs show how generation probabilities for favorable and unfavorable targets evolve, respectively\.
\(a\)Does hesee you?
\(b\)Smiles indicatehappiness\.
\(c\)This book ismine\.
Figure 2:Morphological inflections across time for Pythia\-70M \(top\), Pythia\-12B \(middle\), and OLMo3\-Stage1 \(bottom\)\. The horizontal axis represents the number and percentage of total training tokens elapsed in the training steps; the vertical axis shows the log scale of target words’ probabilities\.For Pythia, the blue curves exhibit a “first turning point” that marks a transition from an initially steady state to onset of maturation—before or at about 100M tokens for both Pythia\-12B and Pythia\-70M\. With rare exceptions, the probability of favorable targets settle at a higher level than unfavorable targets\. We observe that the onset of learning morphological structures—starting at the “first turning point”– usually by or before 100M training tokens precedes the onset of semantic maturation of the models \(See Sec\.[5](https://arxiv.org/html/2610.19041#S5)\)\. Morphological maturation occurs in both Pythia and OLMo3 models between 1B and 100B tokens, after a “second turning point”\.
\(a\)Pythia—Favorable Targets
\(b\)Pythia—Unfavorable Targets
\(c\)OLMo3
Figure 3:Morphological inflections in Pythia suite and OLMo3 w\.r\.t\. all unit\-test cases\. Shadows represent the standard deviation of results\.
### 4\.2Three Phases in Acquiring Morphological Competence
While the examples discussed in Sec\.[4\.1](https://arxiv.org/html/2610.19041#S4.SS1)illustrate the details of morphological acquisition, Fig\.[3](https://arxiv.org/html/2610.19041#S4.F3)provides a comprehensive overview aggregated across all unit\-test cases \(the full list is available in Table[D\.1](https://arxiv.org/html/2610.19041#A4.T1)\) for various model scales over time\. As shown in Figs\.[3\(a\)](https://arxiv.org/html/2610.19041#S4.F3.sf1)and[3\(b\)](https://arxiv.org/html/2610.19041#S4.F3.sf2), Pythia models exhibit a characteristic three\-phase trajectory:Initialization:a steady state, from the earliest steps to the first turning point, which occurs before 100M tokens;Accelerated Learning:an increasing trend between the first turning point and second turning point, which occurs at 1B to 10B tokens; andMaturity Plateau:a relatively stabilized state after the second turning point\. In Fig\.[3\(c\)](https://arxiv.org/html/2610.19041#S4.F3.sf3), OLMo3 appears to follow the same three\-phase behavior\. Unfortunately, OLMo3 has not released early phase checkpoints near the first turning point, which represents the onset of morphological acquisitions\. At settlement, after the second turning point, both Pythia and OLMo3 prefer to predict favorable morphological forms by a factor of roughly 30 to 35\.
## 5Tracing Semantic Stabilization
We further explore how embedding drift characterizes semantic acquisition of models over time\.[Xu et al\. \[38\]](https://arxiv.org/html/2610.19041#bib.bib19)measure the progression of cosine similarity of decoder vectors among intermediate checkpoints and their final state, observing a drift\. We measure the cosine similarity of target words’ embeddings at any intermediate checkpoints with respect to the final checkpoints’ embedding per incipit prompt per model scale\.
Figure 4:Semantic embedding stability for Pythia\-70M \(top\), Pythia\-12B \(middle\), and OLMo3\-Stage1 \(bottom\): left \(Verbs\), middle \(Nouns\), and right \(Pronouns\)\.Figure[4](https://arxiv.org/html/2610.19041#S5.F4)shows the embedding drifts for Pythia\-70M \(top row\), Pythia\-12B \(middle row\), and OLMo3 \(bottom row\) w\.r\.t\. different lexical categories: Verbs \(blue\), Nouns \(green\), and Pronouns \(red\)555The results for other Pythia scales, GPT\-2, and OLMo2 are available in App\.[D\.2](https://arxiv.org/html/2610.19041#A4.SS2)\.in left, middle, and right, respectively\. The figure adopts a word\-centric view, illustrating how the similarity of a specific word with its final state evolves across all training checkpoints\. This visualization shows that the similarity initially remains relatively stable, then experiences a sudden increase during training, and gradually transitions into a plateau as it approaches the end\. These trends align with the three phases we previously identified in Sec\.[4](https://arxiv.org/html/2610.19041#S4): initialization \(a steady phase of high noise and no established semantics in the model’s internal activations\), onset of accelerated learning \(after the first turning point with embeddings moving to their semantic settlement\), and maturity plateau \(a steady phase where embeddings settling along their final directions\)\. The three\-phase structure is consistent with phases in\[[38](https://arxiv.org/html/2610.19041#bib.bib19)\]\.
Based on the two top rows in Fig\.[4](https://arxiv.org/html/2610.19041#S5.F4), Pythia’s onset of semantic acquisition occurs well after 1B training tokens\. Furthermore, comparing Figs\.[3](https://arxiv.org/html/2610.19041#S4.F3)and[4](https://arxiv.org/html/2610.19041#S5.F4), we observe that the models experience the onset of morphology acquisitionpriorto the onset of semantic acquisition\. Finally, while Pythia experiences a plateau in morphology acquisition between 1B to 10B training tokens, it experiences its semantic\-maturity plateaus muchlater—after 100B training tokens\. OLMo3 shows similar trends with the onset of morphology plateau at 10B to 100B and semantic plateau at 2T training tokens\. We conclude that the onset and maturity of morphological competence occur before the onset and maturity of semantic competence\.
## 6Repetition and Co\-occurrence
LLMs rely on the statistics of token patterns in their attention window and perform well on such patterns—e\.g\., on few\-shot exemplars in sequential blocks\[[37](https://arxiv.org/html/2610.19041#bib.bib15)\]or on retrieval from recently listed sets\[[2](https://arxiv.org/html/2610.19041#bib.bib21)\]\. Following the latter, and to investigate whether semantic divergence in a list is recognized by LLMs, we devised a variation on DAT\. DAT was originally designed to measure linguistic creativity\[[25](https://arxiv.org/html/2610.19041#bib.bib24),[4](https://arxiv.org/html/2610.19041#bib.bib25)\]\. DAT asks subjects to producennsemantically divergent words and computes a divergence score based on the average mutual cosine similarities in the produced set\. Since models can hardly adhere to prompts in their initial training checkpoints and rarely generate appropriate target words, we modify the DAT task to recall\-based DAT \(rDAT\), which emphasizes divergence recognition and recall\. As indicated in the first row of Table[1](https://arxiv.org/html/2610.19041#S6.T1), we prompt the model with a set of 10 words and expectNtwN\_\{tw\}most divergent words to be selected\. We setNtw=3N\_\{tw\}=3, referring to targets as “triplets”, and adopt the DAT score to measure divergence in selected triplets\.
Table 1:Prompt formats for creativity tests, rDAT and RAT\.The total number of distinct triplets built for a 10\-word list is equal to\(103\)=120\\binom\{10\}\{3\}=120\. We sort the triplet in descending order of their DAT scores\. For each triplet, we apply teacher forcing\[[35](https://arxiv.org/html/2610.19041#bib.bib1)\]—to compute the joint probability of target token sequence conditioned on the incipit token sequence\. If the tokenizer breaks a wordww\(in the target triplet\) intoNwN\_\{w\}tokens, we compute the corresponding geometric mean probabilities asGMw=exp\(∑i=1Nwlog\(pi\)/Nw\)GM\_\{w\}=\\exp\{\(\\sum\_\{i=1\}^\{N\_\{w\}\}\\log\(p\_\{i\}\)/N\_\{w\}\}\), where andpip\_\{i\}is the conditional probability ofii\-th token inww\. The probability for each triplet arrangement is the joint conditional probability of the word\-sequence in the triplet, i\.e\.,Pj=∏k=13GMjk,j=1,…,120P\_\{j\}=\\prod\_\{k=1\}^\{3\}GM\_\{jk\},j=1,\\ldots,120\. We compute the overall probability of each triplet as the average probability of all its3\!3\!permutations\. As basis for our study, we select 16 mid\-range 10\-word lists666We will refer to such a 10\-word list as “list”\. Results using 32 \(instead of 16\) such lists can be found in App\.[D\.4](https://arxiv.org/html/2610.19041#A4.SS4)\., from the dataset released by\[[4](https://arxiv.org/html/2610.19041#bib.bib25)\]\. Our selected lists have DAT\-scores ranging from 60 to 70\.
\(a\)Pythia\-70M
\(b\)Pythia\-12B
\(c\)OLMo3
Figure 5:Our rDAT results across time for Pythia\-70M, Pythia\-12B, and OLMo3with a reference 10\-word list “male, female, cat, dog, chalk, cheese, fish, bird, animal, vegetable”\.Figure[5](https://arxiv.org/html/2610.19041#S6.F5)shows the results for an example rDAT test with the list “male, female, cat, dog, chalk, cheese, fish, bird, animal, vegetable”\. Among its sub\-triplets, “female, chalk, vegetable” \(colored red and labeled “main”\) has the highest DAT score\. In addition, we exclude “main” and color five triplets whose generation probability at the last checkpoint is greater than other triplets in the test, labeling them as “1st” to “5th”\. Figure[5](https://arxiv.org/html/2610.19041#S6.F5), leads to three observations\. First, the general rising production probabilities for all 120 triplets confirm the evolving STM, repetition, and recall skills\[[2](https://arxiv.org/html/2610.19041#bib.bib21)\]\. Second, all three models express accumulated probabilities in triplets with greatest chances of co\-occurrence in the training corpus\. Third, the increasing main triplet’s production probability in OLMo3 clearly expresses a growing prompt and divergence recognition capability as it consumes roughly 19 folds more training tokens than either Pythia\-70M or Pythia\-12B—both of which fails to show this increase in divergence recognition\. Larger volumes of training tokens lead to better prompt and divergence recognition\.
## 7Semantic Divergence and Phrasal Association Links
As another procedural test of creativity, we use RAT\.[Bowden and Jung\-Beeman \[6\]](https://arxiv.org/html/2610.19041#bib.bib26)use a version of RAT which consists of 144 questions\. Subjects are asked to find a single word that connects three diverging stimulus words\. The prompt format is shown in the second row of Table[1](https://arxiv.org/html/2610.19041#S6.T1)\. Based on the resolution rate, we define three categories: Easy, Medium, and Hard, answered by over 60%, 20% to 60%, and under 20% of participants, respectively\. Three sample RAT cases are Easy: “rocking, wheel, high”→\\rightarrow“chair”; Medium: “print, berry, bird”→\\rightarrow“blue”; Hard: and “way, ground, weather”→\\rightarrow‘‘fair’’777Extra evaluations are available in App\.[D\.5](https://arxiv.org/html/2610.19041#A4.SS5)\.\. For each incipit, we measure all 144 target words’ probabilities\. Overall, we discover that morphological and semantic learning shapes and occurs earlier than prompt and divergence recognition \(using rDAT\) as well as associative link generation \(using RAT\)\.
### 7\.1Divergence Recognition
For each rDAT list, we sort the set of its sub\-tripletsLLbased on their DAT scores and choose the top\-10 subset888The results for top\-1 and top\-5 candidate triplets also show the same trends \(see App\.[D\.4](https://arxiv.org/html/2610.19041#A4.SS4)\)\.LtL\_\{t\}of triplets, and compute its probability sum at the intermediate checkpointcc,St,c=∑i=1\|Lt\|Pi,cS\_\{t,c\}=\\sum\_\{i=1\}^\{\|L\_\{t\}\|\}P\_\{i,c\}, \(forc=1,…,Cc=1,\\ldots,C\)\. We divide this by the probability sum of all\(103\)\\binom\{10\}\{3\}triplets either at the first \(SDR=∑i=1\|L\|Pi,1S\_\{DR\}=\\sum\_\{i=1\}^\{\|L\|\}P\_\{i,1\}\) or at the last \(SMR=∑i=1\|L\|Pi,CS\_\{MR\}=\\sum\_\{i=1\}^\{\|L\|\}P\_\{i,C\}\) checkpoint, and refer to these as Development Ratio \(DR\) and Maturity Ratio \(MR\), respectively\.
\(a\)Pythia—Development Ratio
\(b\)Pythia—Maturity Ratio
\(c\)OLMo3\-Stage1
Figure 6:Performance of Pythia and OLMo3: DR and MR for top\-10 triplets of 16 lists in rDAT test\.The results in Fig\.[6](https://arxiv.org/html/2610.19041#S7.F6)indicate an approximate value just below10−110^\{\-1\}for both initial DR and final MR\. This implies that both in the beginning, when the entropy is high, and in the end, when the models have matured, the models assign roughly equal probabilities to all triplets so that the DR \(at the start\) and MR \(at the end\) are almost equal to the ratio of the number of most divergent triplets to the total 120 possible triplets extracted from the ten words,10/12010/120\. Almost identical initial DR and final MR values indicate that, handling rDAT, Pythia and OLMo3 tend to repeat elements of a list more than they tend to recognize divergence in triplets of elements\. This could be a due weakness in rDAT in comparison to RAT, which does not rely on repetition\. Note that rDAT shifts the blame for low divergence recognition to the paucity of training data—see Sec\.[6](https://arxiv.org/html/2610.19041#S6)\. The results in Fig\.[6](https://arxiv.org/html/2610.19041#S7.F6)underscore a balanced rise in all triplet probabilities, following the semantic plateauing in Fig\.[4](https://arxiv.org/html/2610.19041#S5.F4)\. Analyzing the ratios in Figs\.[6\(a\)](https://arxiv.org/html/2610.19041#S7.F6.sf1)and[6\(b\)](https://arxiv.org/html/2610.19041#S7.F6.sf2), we note that divergent triplets’ probability increases not only during training but also to varying degree across scales: Larger models reinforce the production probability more than smaller ones\.
### 7\.2Association Recognition
Figure[7](https://arxiv.org/html/2610.19041#S7.F7)shows Pythia\-70M \(top row\), Pythia\-12B \(middle row\), and OLMo3 \(bottom row\) behaviors w\.r\.t\. Easy, Medium, and Hard \(RAT\) cases, at left, middle, and right columns, respectively\. The correct answer is labeled “main” and colored red\. We also show two targets with the highest probabilities at the last checkpoint\. The results show the poor performance of the models in solving RAT\-related puzzles\. The linked words in the RAT test consist of two words \(e\.g\., blue\+\+berry→\\rightarrowblueberry in Fig\.[7\(b\)](https://arxiv.org/html/2610.19041#S7.F7.sf2)\)\. This requires models to understand the structural chemistry of how words fuse to form new words\. However, both Pythia and OLMo3 use BPE tokenizer \(a sub\-word algorithm to split text into smaller chunks\) and default to surface\-level thematic associations \(fruit→\\rightarrowapple→\\rightarrowred\)\.
Figure[7](https://arxiv.org/html/2610.19041#S7.F7)shows that the models do not necessarily look for a single word that structurally attaches to all three words\. Rather, they seem to prioritize words that frequently co\-occur near them in training data\. If models cannot find the associating word, they default to a word that has weak but broad links to the divergent triplet\. For example, “rock” has the highest probability among all targets in the Easy example case \(Fig\.[7\(a\)](https://arxiv.org/html/2610.19041#S7.F7.sf1)\)\. We see “rocking” as the first word in the incipit prompt\. This implies that repetition significantly affects the RAT score of the models, too\. On the other hand, Fig\.[7\(c\)](https://arxiv.org/html/2610.19041#S7.F7.sf3)emphasizes the impact of co\-occurrence where the tight relationship between “ground”, “whether”, “air”, “sun”, “snow”, “sky”, and “green” \(all belong to the semantic category of nature, environment, and atmospheric phenomena\) directs models to prefer these words over the target word “fair”\.
\(a\)“rocking, wheel, high”→\\rightarrow“chair”
\(b\)“print, berry, bird”→\\rightarrow“blue”
\(c\)“way, ground, weather”→\\rightarrow“fair”
Figure 7:RAT results across time for Pythia\-70M \(top\), Pythia\-12B \(middle\), and OLMo3 \(bottom\): \(a\) Easy; \(b\) Medium; and \(c\) Hard cases\.
## 8Conclusion
Our experiments, using checkpoints of models in language training \(Pythia and OLMo3\), show that LLMs acquire morphological and syntactic competence before acquiring stable semantic embeddings\. Using two operational measures for creative use of language \(rDAT and RAT\), we observe that, in the absence of large enough training data\-sets, models barely acquire divergent or associative linguistic competence\. Instead, repetition and co\-occurrence dominate their behavior\. LLM language training progresses through three stages:Initialization\(embeddings experience high noise and lack any clear semantic structure\),Accelerated Learning\(embeddings move towards semantic stability\), andMaturity Plateau\(embeddings converge to their stable form\)\. Semantic divergence recognition and associative link generation follow the same pattern\. Co\-occurrence in training corpus and repetition in inference negatively impact these competencies\.
Future possible research directions include:\(i\)expanding our morphological dataset;\(ii\)analyzing the impact of model initialization;\(iii\)investigating the impact of training dataset—for example, Pythia can also be trained on a larger training dataset, e\.g\., Dolma\-3;\(iv\)analyzing the behavior of Instruct and Think training999As a preliminary finding, OLMo2\-7B\-Instruct training has no impact on its morphological preferences\. See App\.[D](https://arxiv.org/html/2610.19041#A4)\.;\(v\)increasing the number of the checkpoints of models such as OLMo3 during language training;\(vi\)devising new creativity measures that are possible to evaluate during language acquisition\.
## References
- \[1\]C\. Anderson, B\. Bjorkman, D\. Denis, J\. Doner, M\. Grant, N\. Sanders, and A\. Taniguchi\(2022\)Essentials of linguistics\.2nd edition,eCampusOntario\.External Links:ISBN 9781927565506,[Link](https://ecampusontario.pressbooks.pub/essentialsoflinguistics2/)Cited by:[§4](https://arxiv.org/html/2610.19041#S4.p1.1)\.
- \[2\]K\. Armeni, M\. Pranjić, and S\. Pollak\(2024\)Transformer verbatim in\-context retrieval across time and scale\.InProceedings of the 28th Conference on Computational Natural Language Learning,L\. Barak and M\. Alikhani \(Eds\.\),Miami, FL, USA,pp\. 56–68\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.conll-1.6)Cited by:[item \(iii\)](https://arxiv.org/html/2610.19041#S1.I1.i3.2),[§2\.3](https://arxiv.org/html/2610.19041#S2.SS3.p1.1),[§6](https://arxiv.org/html/2610.19041#S6.p1.1),[§6](https://arxiv.org/html/2610.19041#S6.p3.1)\.
- \[3\]E\. Asgari, Y\. E\. Kheir, and M\. A\. S\. Javaheri\(2025\)MorphBPE: a morpho\-aware tokenizer bridging linguistic complexity for efficient llm training across morphologies\.External Links:2502\.00894,[Link](https://arxiv.org/abs/2502.00894)Cited by:[§2\.1](https://arxiv.org/html/2610.19041#S2.SS1.p1.1)\.
- \[4\]A\. Bellemare\-Pepin, F\. Lespinasse, P\. Thölke, Y\. Harel, K\. Mathewson, J\. A\. Olson, Y\. Bengio, and K\. Jerbi\(2026\)Divergent creativity in humans and large language models\.Scientific Reports16\(1\),pp\. 25157\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-25157-3),[Link](https://www.nature.com/articles/s41598-025-25157-3)Cited by:[§6](https://arxiv.org/html/2610.19041#S6.p1.1),[§6](https://arxiv.org/html/2610.19041#S6.p2.1)\.
- \[5\]S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff,et al\.\(2023\)Pythia: a suite for analyzing large language models across training and scaling\.InInternational Conference on Machine Learning,pp\. 2397–2430\.Cited by:[Appendix A](https://arxiv.org/html/2610.19041#A1.p4.1),[1st item](https://arxiv.org/html/2610.19041#S1.I2.i1.p1.1),[§2\.1](https://arxiv.org/html/2610.19041#S2.SS1.p1.1),[item \(i\)](https://arxiv.org/html/2610.19041#S3.I1.i1.2)\.
- \[6\]E\. M\. Bowden and M\. Jung\-Beeman\(2003\)Normative data for 144 compound remote associate problems\.Behavior Research Methods, Instruments, & Computers35\(4\),pp\. 634–639\.External Links:[Document](https://dx.doi.org/10.3758/bf03195543)Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p3.1),[§7](https://arxiv.org/html/2610.19041#S7.p1.1)\.
- \[7\]L\. Carlucci and J\. Case\(2013\)On the necessity of u\-shaped learning\.Topics in Cognitive Science5\(1\),pp\. 56–88\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1111/tops.12002)Cited by:[§D\.1](https://arxiv.org/html/2610.19041#A4.SS1.p3.1)\.
- \[8\]N\. Chomsky\(1986\)Knowledge of language: its nature, origin, and use\.Convergence,Praeger,New York\.Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1)\.
- \[9\]D\. Držík and J\. Kapusta\(2026\)The importance of morphology\-aware subword tokenization for nlp tasks in slovak language modeling\.Expert Systems with Applications312,pp\. 131492\.External Links:ISSN 0957\-4174,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.eswa.2026.131492)Cited by:[item \(i\)](https://arxiv.org/html/2610.19041#S1.I1.i1.2)\.
- \[10\]D\. Držík and J\. Kapusta\(2026\)The importance of morphology\-aware subword tokenization for nlp tasks in slovak language modeling\.Expert Systems with Applications312,pp\. 131492\.External Links:ISSN 0957\-4174,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.eswa.2026.131492)Cited by:[§2\.1](https://arxiv.org/html/2610.19041#S2.SS1.p1.1)\.
- \[11\]Elliot Bolton and David Hall and Michihiro Yasunaga and Tony Lee and Chris Manning and Percy Liang\(2021\)BioMedLM\.Stanford Center for Research on Foundation Models \(CRFM\)\.External Links:[Link](https://crfm.stanford.edu/2022/12/15/biomedlm.html)Cited by:[Appendix A](https://arxiv.org/html/2610.19041#A1.p4.1)\.
- \[12\]N\. Evans and S\. C\. Levinson\(2009\)The myth of language universals: language diversity and its importance for cognitive science\.Behavioral and Brain Sciences32\(5\),pp\. 429–448\.External Links:[Document](https://dx.doi.org/10.1017/S0140525X0999094X)Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1)\.
- \[13\]J\. A\. Fodor and Z\. W\. Pylyshyn\(1988\)Connectionism and cognitive architecture: a critical analysis\.Cognition28\(1\),pp\. 3–71\.External Links:ISSN 0010\-0277,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/0010-0277%2888%2990031-5)Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1)\.
- \[14\]L\. Gao, S\. Biderman, S\. Black, L\. Golding, T\. Hoppe, C\. Foster, J\. Phang, H\. He, A\. Thite, N\. Nabeshima, S\. Presser, and C\. Leahy\(2020\)The pile: an 800gb dataset of diverse text for language modeling\.External Links:2101\.00027,[Link](https://arxiv.org/abs/2101.00027)Cited by:[item \(i\)](https://arxiv.org/html/2610.19041#S3.I1.i1.2)\.
- \[15\]B\. Ghiselin \(Ed\.\)\(1985\)The creative process: reflections on the invention in the arts and sciences\.University of California Press,Berkeley, CA\.External Links:ISBN 978\-0520054530,[Link](https://www.amazon.com/Creative-Process-Reflections-Invention-Sciences/dp/0520054539)Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p3.1)\.
- \[16\]A\. Gokaslan and V\. Cohen\(2019\)OpenWebText corpus\.Note:[http://Skylion007\.github\.io/OpenWebTextCorpus](http://skylion007.github.io/OpenWebTextCorpus)Cited by:[Appendix A](https://arxiv.org/html/2610.19041#A1.p2.1),[Appendix A](https://arxiv.org/html/2610.19041#A1.p4.1)\.
- \[17\]M\. Ismayilzada, D\. Circi, J\. Sälevä, H\. Sirin, A\. Köksal, B\. Dhingra, A\. Bosselut, D\. Ataman, and L\. van der Plas\(2025\)Evaluating morphological compositional generalization in large language models\.External Links:2410\.12656,[Link](https://arxiv.org/abs/2410.12656)Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.19041#S2.SS1.p1.1)\.
- \[18\]C\. Jiang, M\. Zhang, Y\. Cao, J\. Ye, X\. Fan, S\. Dou, Z\. Xi, J\. Sun, Y\. Dong, Y\. Shen, J\. Tong, B\. Fan, T\. Gui, Q\. Zhang, and X\. Huang\(2026\)Beyond scaling: measuring and predicting the upper bound of knowledge retention in language model pre\-training\.External Links:2502\.04066,[Link](https://arxiv.org/abs/2502.04066)Cited by:[item \(iv\)](https://arxiv.org/html/2610.19041#S1.I1.i4.2)\.
- \[19\]C\. R\. Johnson, M\. R\. L\. Petruck, C\. F\. Baker, M\. Ellsworth, J\. Ruppenhofer, and C\. J\. Fillmore\(2003\)FrameNet: theory and practice\.Technical ReportTechnical ReportVersion 1\.1,International Computer Science Institute,Berkeley, CA\.Note:Printed December 29, 2003Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1)\.
- \[20\]J\. Katz and J\. Fodor\(1963\)The structure of a semantic theory\.Language39,pp\. 170–210\.Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1)\.
- \[21\]L\. Kozma and J\. Voderholzer\(2024\)Theoretical analysis of byte\-pair encoding\.External Links:2411\.08671,[Link](https://arxiv.org/abs/2411.08671)Cited by:[§2\.1](https://arxiv.org/html/2610.19041#S2.SS1.p1.1)\.
- \[22\]A\. Lopardo, A\. Harish, C\. Arnett, and A\. Gupta\(2026\)Weight tying biases token embeddings towards the output space\.External Links:2603\.26663,[Link](https://arxiv.org/abs/2603.26663)Cited by:[item \(ii\)](https://arxiv.org/html/2610.19041#S1.I1.i2.2)\.
- \[23\]T\. Mikolov, K\. Chen, G\. Corrado, and J\. Dean\(2013\)Efficient estimation of word representations in vector space\.arXiv preprint arXiv:1301\.3781\.Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1)\.
- \[24\]T\. OLMo, P\. Walsh, L\. Soldaini, D\. Groeneveld, K\. Lo, S\. Arora, A\. Bhagia, Y\. Gu, S\. Huang, M\. Jordan, N\. Lambert, D\. Schwenk, O\. Tafjord, T\. Anderson, D\. Atkinson, F\. Brahman, C\. Clark, P\. Dasigi, N\. Dziri, A\. Ettinger, M\. Guerquin, D\. Heineman, H\. Ivison, P\. W\. Koh, J\. Liu, S\. Malik, W\. Merrill, L\. J\. V\. Miranda, J\. Morrison, T\. Murray, C\. Nam, J\. Poznanski, V\. Pyatkin, A\. Rangapur, M\. Schmitz, S\. Skjonsberg, D\. Wadden, C\. Wilhelm, M\. Wilson, L\. Zettlemoyer, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi\(2025\)2 olmo 2 furious\.External Links:2501\.00656,[Link](https://arxiv.org/abs/2501.00656)Cited by:[Appendix A](https://arxiv.org/html/2610.19041#A1.p1.1),[Appendix A](https://arxiv.org/html/2610.19041#A1.p3.1),[Appendix A](https://arxiv.org/html/2610.19041#A1.p4.1),[Appendix D](https://arxiv.org/html/2610.19041#A4.p1.1),[§2\.1](https://arxiv.org/html/2610.19041#S2.SS1.p1.1),[footnote 1](https://arxiv.org/html/2610.19041#footnote1)\.
- \[25\]J\. A\. Olson, J\. Nahas, D\. Chmoulevitch, S\. J\. Cropper, and M\. E\. Webb\(2021\)Naming unrelated words predicts creativity\.Proceedings of the National Academy of Sciences118\(25\),pp\. e2022340118\.External Links:[Document](https://dx.doi.org/10.1073/pnas.2022340118)Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p3.1),[§6](https://arxiv.org/html/2610.19041#S6.p1.1)\.
- \[26\]W\. O’Grady\(2005\)How children learn language\.Cambridge Approaches to Linguistics,Cambridge University Press,Cambridge\.External Links:ISBN 9780521531924Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1)\.
- \[27\]X\. Peng, H\. Jiang, J\. Chen, M\. Liu, and X\. Chen\(2025\)Leveraging large language models and embedding representations for enhanced word similarity computation\.Scientific Reports16\(1\),pp\. 1494\.External Links:[Document](https://dx.doi.org/10.1038/s41598-025-31102-1),[Link](https://www.nature.com/articles/s41598-025-31102-1)Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.19041#S2.SS2.p1.1)\.
- \[28\]J\. Pennington, R\. Socher, and C\. D\. Manning\(2014\)GloVe: global vectors for word representation\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 1532–1543\.External Links:[Link](https://aclanthology.org/D14-1162/)Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1)\.
- \[29\]M\. J\. Saeed, T\. Vehvilainen, E\. Fedoseev, S\. Caliskan, and T\. Vodolazova\(2025\)IMPACT: inflectional morphology probes across complex typologies\.External Links:2506\.23929,[Link](https://arxiv.org/abs/2506.23929)Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.19041#S2.SS1.p1.1)\.
- \[30\]Stanford Center for Research on Foundation Models \(CRFM\)\(2021\)Mistral: large scale language modeling made easy\.The Board of Trustees of The Leland Stanford Junior University\.External Links:[Link](https://nlp.stanford.edu/mistral/getting_started/download.html)Cited by:[Appendix D](https://arxiv.org/html/2610.19041#A4.p1.1),[footnote 1](https://arxiv.org/html/2610.19041#footnote1)\.
- \[31\]Stanford Center for Research on Foundation Models \(CRFM\)\(2021\)Mistral: large scale language modeling made easy\.The Board of Trustees of The Leland Stanford Junior University\.External Links:[Link](https://github.com/stanford-crfm/mistral)Cited by:[Appendix A](https://arxiv.org/html/2610.19041#A1.p1.1),[Appendix A](https://arxiv.org/html/2610.19041#A1.p2.1),[Appendix A](https://arxiv.org/html/2610.19041#A1.p4.1),[Appendix D](https://arxiv.org/html/2610.19041#A4.p1.1),[footnote 1](https://arxiv.org/html/2610.19041#footnote1)\.
- \[32\]Team Olmo, :, A\. Ettinger, A\. Bertsch, B\. Kuehl, D\. Graham, D\. Heineman, D\. Groeneveld, F\. Brahman, F\. Timbers, H\. Ivison, J\. Morrison, J\. Poznanski, K\. Lo, L\. Soldaini, M\. Jordan, M\. Chen, M\. Noukhovitch, N\. Lambert, P\. Walsh, P\. Dasigi, R\. Berry, S\. Malik, S\. Shah, S\. Geng, S\. Arora, S\. Gupta, T\. Anderson, T\. Xiao, T\. Murray, T\. Romero, V\. Graf, A\. Asai, A\. Bhagia, A\. Wettig, A\. Liu, A\. Rangapur, C\. Anastasiades, C\. Huang, D\. Schwenk, H\. Trivedi, I\. Magnusson, J\. Lochner, J\. Liu, L\. J\. V\. Miranda, M\. Sap, M\. Morgan, M\. Schmitz, M\. Guerquin, M\. Wilson, R\. Huff, R\. L\. Bras, R\. Xin, R\. Shao, S\. Skjonsberg, S\. Z\. Shen, S\. S\. Li, T\. Wilde, V\. Pyatkin, W\. Merrill, Y\. Chang, Y\. Gu, Z\. Zeng, A\. Sabharwal, L\. Zettlemoyer, P\. W\. Koh, A\. Farhadi, N\. A\. Smith, and H\. Hajishirzi\(2026\)Olmo 3\.External Links:2512\.13961,[Link](https://arxiv.org/abs/2512.13961)Cited by:[Appendix A](https://arxiv.org/html/2610.19041#A1.p4.1),[1st item](https://arxiv.org/html/2610.19041#S1.I2.i1.p1.1),[§2\.1](https://arxiv.org/html/2610.19041#S2.SS1.p1.1),[item \(ii\)](https://arxiv.org/html/2610.19041#S3.I1.i2.2)\.
- \[33\]P\. D\. Turney and P\. Pantel\(2010\)From frequency to meaning: vector space models of semantics\.Journal of Artificial Intelligence Research37,pp\. 141–188\.External Links:[Link](https://jair.org/index.php/jair/article/view/10640)Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1)\.
- \[34\]L\. S\. Vygotsky\(1962\)Thought and language\.The MIT Press,Cambridge, MA\.Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1)\.
- \[35\]R\. J\. Williams and D\. Zipser\(1989\)A learning algorithm for continually running fully recurrent neural networks\.Neural Computation1\(2\),pp\. 270–280\.External Links:[Document](https://dx.doi.org/10.1162/neco.1989.1.2.270)Cited by:[§4](https://arxiv.org/html/2610.19041#S4.p2.1),[§6](https://arxiv.org/html/2610.19041#S6.p2.1)\.
- \[36\]T\. Wolf, L\. Debut, V\. Sanh, J\. Chaumond, C\. Delangue, A\. Moi, P\. Cistac, T\. Rault, R\. Louf, M\. Funtowicz, J\. Davison, S\. Shleifer, P\. von Platen, C\. Ma, Y\. Jernite, J\. Plu, C\. Xu, T\. Le Scao, S\. Gugger, M\. Drame, Q\. Lhoest, and A\. Rush\(2020\)Transformers: state\-of\-the\-art natural language processing\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,Q\. Liu and D\. Schlangen \(Eds\.\),Online,pp\. 38–45\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-demos.6)Cited by:[Appendix A](https://arxiv.org/html/2610.19041#A1.p2.1),[item \(i\)](https://arxiv.org/html/2610.19041#S3.I1.i1.2)\.
- \[37\]Y\. Xiong and S\. Wu\(2025\)Do large language models learn like humans: interleaved and spaced practice in morphological learning\.Acta Psychologica260,pp\. 105518\.External Links:ISSN 0001\-6918,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.actpsy.2025.105518)Cited by:[§1](https://arxiv.org/html/2610.19041#S1.p1.1),[§6](https://arxiv.org/html/2610.19041#S6.p1.1)\.
- \[38\]Y\. Xu, Y\. Wang, H\. Huang, and H\. Wang\(2025\)Tracking the feature dynamics in llm training: a mechanistic study\.External Links:2412\.17626,[Link](https://arxiv.org/abs/2412.17626)Cited by:[item \(ii\)](https://arxiv.org/html/2610.19041#S1.I1.i2.2),[§2\.2](https://arxiv.org/html/2610.19041#S2.SS2.p1.1),[§5](https://arxiv.org/html/2610.19041#S5.p1.1),[§5](https://arxiv.org/html/2610.19041#S5.p2.1)\.
- \[39\]J\. Yan, J\. Xu, C\. Song, C\. Wu, Y\. Li, and Y\. Zhang\(2024\)Understanding in\-context learning from repetitions\.External Links:2310\.00297,[Link](https://arxiv.org/abs/2310.00297)Cited by:[§2\.3](https://arxiv.org/html/2610.19041#S2.SS3.p1.1)\.
## Appendix AArchitectural Differences
In Sec\.[3](https://arxiv.org/html/2610.19041#S3), we reviewed Pythia suite and OLMo3\. Here, we introduce GPT\-2\[[31](https://arxiv.org/html/2610.19041#bib.bib5)\]and OLMo2\[[24](https://arxiv.org/html/2610.19041#bib.bib4)\]\.
GPT\-2:Built upon the Hugging Face Transformers library\[[36](https://arxiv.org/html/2610.19041#bib.bib10)\], the Stanford CRFM team trained five GPT\-2 Medium \(345M\-355M\) and five GPT\-2 Small \(124M\) models on the OpenWebText Corpus\[[16](https://arxiv.org/html/2610.19041#bib.bib9)\]over 209B tokens\. Each model corresponds to a unique seed\. Only intermediate checkpoints are available for small models: Alias, Battlestar, Caprica, Darkmatter, and Expanse\[[31](https://arxiv.org/html/2610.19041#bib.bib5)\]\.
OLMo2:Prior to OLMo3, Allen Institute for AI developed OLMo2, including core model sizes of 1B, 7B, 13B, and 32B parameters\. Each scale features raw base pre\-trained models alongside post\-trained instruct/chat variations adapted using Tülu alignment recipes\[[24](https://arxiv.org/html/2610.19041#bib.bib4)\]\. Here, we employ OLMo2\-7B\-Base and Instruct, and for simplicity, we call them OLMo2 and OLMo2\-Instruct, respectively\.
Table[A\.1](https://arxiv.org/html/2610.19041#A1.T1)indicates the architecture of the under\-study LLMs, Pythia\[[5](https://arxiv.org/html/2610.19041#bib.bib2)\], GPT\-2\[[11](https://arxiv.org/html/2610.19041#bib.bib7)\], OLMo2\[[24](https://arxiv.org/html/2610.19041#bib.bib4)\], and OLMo3\[[32](https://arxiv.org/html/2610.19041#bib.bib3)\]\. Unlike Pythia and GPT\-2, which use classic pre\-layer normalization, OLMo3 maintains the post\-normalization structure inherited from OLMo2, with applying an additional RMSNorm layer to queries and keys before computing the attention\. This prohibits attention logits to overgrow and hence enforces superior training stability at scale\. Moreover, compared to Pythia and GPT\-2 that are trained over 300B\-token\[[5](https://arxiv.org/html/2610.19041#bib.bib2)\]and 9B\-token\[[16](https://arxiv.org/html/2610.19041#bib.bib9)\]datasets respectively, OLMo2 and OLMo3 are trained on the 5\.9T\-token\[[32](https://arxiv.org/html/2610.19041#bib.bib3),[24](https://arxiv.org/html/2610.19041#bib.bib4)\]dataset in their pretraining stage\. For more information, the readers are referred to the corresponding references\[[5](https://arxiv.org/html/2610.19041#bib.bib2),[32](https://arxiv.org/html/2610.19041#bib.bib3),[31](https://arxiv.org/html/2610.19041#bib.bib5),[24](https://arxiv.org/html/2610.19041#bib.bib4)\]\.
Table A\.1:Architectures of under\-study LLMs
## Appendix BMain Checkpoints
Pythia\.We conduct experiments for the following checkpoints: \{0, 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1000, 3000, 13000, 23000, 33000, 43000, 53000, 63000, 73000, 83000, 93000, 103000, 113000, 123000, 133000, 143000\}\.
GPT\-2\.The following training checkpoints are used for GPT\-2 small evaluations: \{0, 10, 50, 100, 150, 200, 500, 1000, 1500, 2000, 5000, 10000, 12000, 15000, 20000, 50000, 100000, 120000, 150000, 200000, 250000, 300000, 350000, 360000, 370000, 380000, 390000, 400000\}\.
OLMo2\.We choose the following checkpoints among all available checkpoints for OLMo2\-7B\-Base: \{1000, 2000, 4000, 8000, 16000, 32000, 64000, 128000, 256000, 512000, 600000, 700000, 800000, 900000, 910000, 920000, 928000, 928646, ’main’\}\. We use all released checkpoints for OLMo2\-7B\-Instruct: \{60, 120, 180, 240, 300, 360, ’main’\}\.
OLMo3\.The following checkpoints are used per stage:
- •Stage 1: \{0, 1000, 2000, 4000, 8000, 16000, 32000, 64000, 128000, 256000, 512000, 1024000, 1100000, 1200000, 1300000, 1400000, 1410000, 1411000, 1412000, 1413000, 1413814\}\.
- •Stage 2: \{1000, 2000, 4000, 8000, 16000, 32000, 40000, 42000, 44000, 46000, 47000, 47600\}\.
- •Stage 3: \{1000, 2000, 4000, 8000, 10000, 11000, 11900, 11921, ’main’\}\.
## Appendix CComputational Resources
Our computational framework utilizes NVIDIA H100 NVL GPUs, each featuring 94 GB of VRAM \(from a total pool of 16 available nodes\)\. Experiments are conducted on either a single\-GPU setup for smaller configurations or a two\-GPU data\-parallel configuration for larger models\. Evaluations are performed in couple of hours for all morphology, RAT, and rDAT tests\.
## Appendix DExtending Results
In Sec\.[3](https://arxiv.org/html/2610.19041#S3), we reviewed Pythia suite and OLMo3\. Here, we first introduce GPT\-2\[[31](https://arxiv.org/html/2610.19041#bib.bib5),[30](https://arxiv.org/html/2610.19041#bib.bib6)\]and OLMo2\[[24](https://arxiv.org/html/2610.19041#bib.bib4)\]\. We then present the extended evaluation results within each relevant section for Pythia suite \(160M, 410M, 1B, 6\.9B\), GPT\-2, OLMo2, and OLMo2\-Instruct\.
Table D\.1:List of unit\-test cases in various lexical categories for some common morphological inflections\. Here, ␣ shows a space character\.\(a\)GPT\-2—Favorable Targets
\(b\)GPT\-2—Unfavorable Targets
\(c\)OLMo2
\(d\)OLMo2\-Instruct
Figure D\.1:Morphological inflections in GPT\-2 and OLMo2variants w\.r\.t\. all unit\-test cases\.
\(a\)Does hesee you?
\(b\)Smiles indicatehappiness\.
\(c\)This book ismine\.
Figure D\.2:Morphological structures across time for Pythia suite: 160M, 410M, 1B, and 6\.9B—first to fourth rows, respectively\.
\(a\)Does hesee you?
\(b\)Smiles indicatehappiness\.
\(c\)This book ismine\.
Figure D\.3:Morphological structures across time for OLMo2 \(top\) and OLMo2\-Instruct \(bottom\)\.### D\.1Tracing Morphological Preferences
Table[D\.1](https://arxiv.org/html/2610.19041#A4.T1)summarizes all unit\-test cases for morphological structures\. In Fig\.[D\.1](https://arxiv.org/html/2610.19041#A4.F1), we see the three\-phase behavior for GPT\-2 seeds, where Initialization occurs before 100M tokens; then, the model falls into the Accelerated Learning phase, which continues up to 10B tokens; after that, the model is in the Maturity Plateau phase where the learning is almost converged to its final state\. Similar to OLMo3 \(see Fig\.[3\(c\)](https://arxiv.org/html/2610.19041#S4.F3.sf3)\), OLMo2 follows a two\-phase behavior \(a combination of Initialization and Accelerated Learning phases followed by the Maturity Plateau\)\. This is while Fig\.[1\(d\)](https://arxiv.org/html/2610.19041#A4.F1.sf4)states that post\-training \(instruction\) does not play an important role in learning process\.
We further consider unit\-test cases mentioned in Fig\.[2](https://arxiv.org/html/2610.19041#S4.F2)\. Figures[D\.2](https://arxiv.org/html/2610.19041#A4.F2)\-[D\.4](https://arxiv.org/html/2610.19041#A4.F4)show the performance of the Pythia suite, the GPT\-2 seeds, and the OLMo2 suite for the cases of morphological structures mentioned above, respectively\.
Similar to Pythia\-70M, Pythia\-160M results in higher probabilities for unfavorable targets for cases in Figs\.[2\(a\)](https://arxiv.org/html/2610.19041#A4.F2.sf1)and[2\(b\)](https://arxiv.org/html/2610.19041#A4.F2.sf2)\. As expected, this trend changes for larger models \(410M, 1B, and 6\.9B\), even the gap between favorable and unfavorable targets increases following enlarging the scales; however, Fig\.[2\(b\)](https://arxiv.org/html/2610.19041#A4.F2.sf2)shows that Pythia\-6\.9B is not successful in prioritizing the favorable target, “happiness”, over “happy”\. Comparing Pythia\-6\.9B and Pythia\-12B \(see Fig\.[2](https://arxiv.org/html/2610.19041#S4.F2)\), the results highlight the architectural mastery of Pythia\-12B that leads to a more rigid internal grasp of English syntax than Pythia\-6\.9B\. Furthermore, the results underscore a fascinating aspect of LLM training:emergence, scale, and training trajectories are non\-linear\. This phenomenon is known asU\-shaped learning\[[7](https://arxiv.org/html/2610.19041#bib.bib37)\]in neural networks: good performance followed by bad performance followed by good performance once again\. This is translated to: the smaller models \(410M and 1B\) perform better than Pythia\-6\.9B\. However, the performance is recovered by Pythia\-12B\. Smaller models heavily weight literal pattern matching found in dictionary definitions or formal encyclopedia text within the training data, whereas medium\-sized models \(Pythia\-6\.9B\) tend to absorb broader web data, colloquial expressions, and association heuristics\. As a result, the massive co\-occurrence of “smiles” and “happy” across standard web text might distract or mislead the middle\-sized models\. When a model is large enough \(Pythia\-12B\), it reconciles these forces, by synthesizing both the formal grammatical rule and the raw data correlation to results in the correct answer\. Figure[2\(c\)](https://arxiv.org/html/2610.19041#A4.F2.sf3)includes the results for the incipit “This book is”\. Following the assessments mentioned in Sec\.[4\.1](https://arxiv.org/html/2610.19041#S4.SS1), that emphasizes the impact of frequency on LLMs performance, all models prioritize the highly repeated combination “is my” over rare phrase “is mine” in their training data; thus, they fail to learn the morphological structure\.
Figures[D\.3](https://arxiv.org/html/2610.19041#A4.F3)and[D\.4](https://arxiv.org/html/2610.19041#A4.F4)illustrate the corresponding results for different GPT\-2 and OLMo2 variations, respectively\. While all GPT\-2 variations are successful in learning morphological structures of Nouns \(Fig\.[4\(b\)](https://arxiv.org/html/2610.19041#A4.F4.sf2)\), some \(Battlestar, Caprica, and Darkmatter\) fail the test \(Fig\.[4\(a\)](https://arxiv.org/html/2610.19041#A4.F4.sf1)\)\. OLMo2 variations are also successful in Verbs and Nouns cases, except OLMo2\-Instruct that is misled in the Verbs case\. Similar to Pythia suite, all GPT\-2 and OLMo2 variations underperform the Pronouns case \(Fig\.[4\(c\)](https://arxiv.org/html/2610.19041#A4.F4.sf3)\)\. Indeed, OLMo2 has been trained on 4\.05\-T tokens of raw web text, books, academic papers, and code, whereas OLMo2\-Instruct is derived from OLMo2 through several post\-training stages using the Tülu\-3 recipe\. Unlike OLMo2that sees the third\-person verbs frequently in its training data, OLMo2\-Instruct is a chat\-based model where rarely feature isolated narrative fragments starting with the third\-person or a third person pronoun is followed by auxiliary verbs\.
Training data shape the morphological structure\.Figure[D\.5](https://arxiv.org/html/2610.19041#A4.F5)indicates two correct morphological structures, “He sees you\.” and “Does he see you?”\. Small Pythia models \(70M and 160M\) fail to learn the correct structure for the first case\. This failure can happen because “Does he see you?” appears more often in the training data than “He sees you\.”\.
\(a\)Does hesee you?
\(b\)Smiles indicatehappiness\.
\(c\)This book ismine\.
Figure D\.4:Morphological structures across time for GPT\-2 variations: Alias, Battlestar, Caprica, Darkmatter, and Expanse—first to fifth rows, respectively\.\(a\)Hesees you?
\(b\)Does hesee you?
Figure D\.5:Training corpus strongly affects morphological structure\.
Figure D\.6:Semantic embedding stability for Pythia suite: 160M, 410M, 1B, and 6\.9B—first to fourth rows, respectively; left \(Verbs\), middle \(Nouns\), and right \(Pronouns\)\.
Figure D\.7:Semantic embedding stability for OLMo2 and OLMo2\-Instruct: left \(Verbs\), middle \(Nouns\), and right \(Pronouns\)\.
Figure D\.8:Semantic embedding stability for GPT\-2 variations: Alias, Battlestar, Caprica, Darkmatter, and Expanse—first to fifth rows, respectively; left \(Verbs\), middle \(Nouns\), and right \(Pronouns\)\.Figure D\.9:Rank evolution of of words in The Pile dataset over checkpoints\.
### D\.2Tracing Semantic Stabilization
Figures[D\.6](https://arxiv.org/html/2610.19041#A4.F6)\-[D\.8](https://arxiv.org/html/2610.19041#A4.F8)represent embedding drifts for Pythia suit, GPT\-2 variations, and OLMo2 variations, respectively\. As expected, all models follow their three\-phase behavioral mentioned in Sec\.[D\.1](https://arxiv.org/html/2610.19041#A4.SS1)\. Again, OLMo2\-Instruct behavior is fixed across time, stating that post\-training lacks a significant impact on semantic learning\.
### D\.3Tracing Frequency of Words
Figure[D\.9](https://arxiv.org/html/2610.19041#A4.F9)shows the evolution of the frequency \(rank\) of words in the Pile data set over training of Pythia scales, where a lower rank indicates higher popularity of words\. Comparing these results with the semantic learning shown in Figs\.[4](https://arxiv.org/html/2610.19041#S5.F4)and[D\.6](https://arxiv.org/html/2610.19041#A4.F6)indicates that the models stabilize the frequency of the words once the semantic learning plateaus\.
### D\.4Divergence Recognition
In Sec\.[7\.1](https://arxiv.org/html/2610.19041#S7.SS1), we assessed the performance of Pythia suit and OLMo3 for the rDAT test with 16 lists, where we measured Development Ratio \(DR\) and Maturity Ratio \(MR\) for top\-10 triplets\. Here, repeat our experiments for top\-kktriplets, wherek∈\{1,5\}k\\in\\\{1,5\\\}\. Also, we extend our evaluation to top\-10 triplets of 32 lists for all Pythia suit, GPT\-2 variations\. OLMo2 variations, and OLMo3\.
\(a\)Pythia—Development Ratio
\(b\)Pythia—Maturity Ratio
\(c\)OLMo3
\(d\)Pythia—Development Ratio
\(e\)Pythia—Maturity Ratio
\(f\)OLMo3
Figure D\.10:Performance of Pythia suite and OLMo3: baseline and maturity likelihoods for 16 lists in rDAT test—top row \(a\-c\), top\-1 triplets; bottom row \(d\-f\), top\-5 triplets\.\(a\)Pythia—Development Ratio
\(b\)Pythia—Maturity Ratio
\(c\)GPT\-2—Development Ratio
\(d\)GPT\-2—Maturity Ratio
\(e\)OLMo2
\(f\)OLMo2\-Instruct
\(g\)OLMo3
Figure D\.11:Performance of Pythia suite, GPT\-2 variations, OLMo2 variations, and OLMo3: baseline and maturity likelihoods for top\-10 triplets of 32 lists in rDAT test\.The top and bottom rows in Fig\.[D\.10](https://arxiv.org/html/2610.19041#A4.F10)show the results for top\-1 and top\-5 triplets, where all results follow the same trend as top\-10 triplet’s in Sec\.[7\.1](https://arxiv.org/html/2610.19041#S7.SS1)\. Also, the results are shown to be proportional tokk; morekk, the probability sum of the top tripletskkincreases \(a higher DR\), and the triplets converge to their final values \(a higher MR\)\. Moreover, as discussed in Sec\.[7\.1](https://arxiv.org/html/2610.19041#S7.SS1), the initial DR and the final MR values are comparable tok/\(103\)k/\\binom\{10\}\{3\}\. Figure[D\.11](https://arxiv.org/html/2610.19041#A4.F11)shows the corresponding results for 32 lists \(including 16 lists in Fig\.[D\.10](https://arxiv.org/html/2610.19041#A4.F10)\)\. As expected, all models follow the three\-phase behavior defined in Sec\.[5](https://arxiv.org/html/2610.19041#S5); models’ performance improves over time and scale, except OLMo2\-Instruct whose evolution does not vary significantly; and both the initial DR and final MR values are comparable to10/\(103\)10/\\binom\{10\}\{3\}\.
### D\.5Association Recognition
Following experiments in Sec\.[7\.2](https://arxiv.org/html/2610.19041#S7.SS2), we evaluate models’ performance for the RAT test w\.r\.t\. the correct answer \(labeled “main” and colored “red”\) and the two targets with the highest probabilities at the last checkpoint in Figs\.[D\.12](https://arxiv.org/html/2610.19041#A4.F12)\-[D\.14](https://arxiv.org/html/2610.19041#A4.F14)for Pythia suite, GPT\-2 seeds, an OLMo2 suite, respectively\. Compared to Fig\.[7](https://arxiv.org/html/2610.19041#S7.F7), OLMo2 variations can do better than other models for Easy and Medium cases\. According to Fig\.[14\(a\)](https://arxiv.org/html/2610.19041#A4.F14.sf1), OLMo2 increases the probability of the correct answer, “chair”, over training time\. However, “rock” owns the highest probability among all target words, emphasizing the impact of repetition on the model’s performance\. Also, Fig\.[14\(b\)](https://arxiv.org/html/2610.19041#A4.F14.sf2)indicates that OLMo2 increases the probability of “blue” over “tree” most of the times, particularly at the last checkpoint\. On the other hand, according to Fig\.[14\(a\)](https://arxiv.org/html/2610.19041#A4.F14.sf1), OLMo2\-Instruct gives a higher probability to “chair”, showing the influence of instruction on the model’s performance\. However, our experiments on the other unit\-test cases reveal that this behavior is not consistent withallunit\-test cases in RAT, yet formostcases; not only Easy cases but also Medium ones\.
\(a\)“rocking, wheel, high”→\\rightarrow“chair”
\(b\)“print, berry, bird”→\\rightarrow“blue”
\(c\)“way, ground, weather”→\\rightarrow“fair”
Figure D\.12:RAT results across time for Pythia suit: 160M, 410M, 1B, and 6\.9B—first to fourth rows, respectively; \(a\) Easy; \(b\) Medium; and \(c\) Hard cases\.
\(a\)“rocking, wheel, high”→\\rightarrow“chair”
\(b\)“print, berry, bird”→\\rightarrow“blue”
\(c\)“way, ground, weather”→\\rightarrow“fair”
Figure D\.13:RAT results across time for GPT\-2 seeds: Alias, Battlestar, Caprica, Darkmatter, and Expanse—first to fifth rows, respectively; \(a\) Easy; \(b\) Medium; and \(c\) Hard cases\.
\(a\)“rocking, wheel, high”→\\rightarrow“chair”
\(b\)“print, berry, bird”→\\rightarrow“blue”
\(c\)“way, ground, weather”→\\rightarrow“fair”
Figure D\.14:RAT results across time for OLMo2 \(top row\) and OLMo2\-Instruct \(bottom row\): \(a\) Easy; \(b\) Medium; and \(c\) Hard cases\.相似文章
从复制到组合：语言模型如何发展出上下文学习能力
本文提出一种受控诊断任务，区分语言模型在上下文学习中的“复制示例”与“组合应用规则”两种行为，发现提示控制、答案偏好、无候选构造和迁移能力在训练中沿不同时间线发展，且合成定义文档训练能提升组合式响应。
语言模型学习什么以及何时学习？隐性课程假设
本文提出隐性课程假设，证明语言模型预训练遵循一个结构化的、组合性的课程，其中能力跨架构一致涌现，并可从内部表示预测。作者通过设计涵盖检索、形态学、共指消解、推理和数学的任务进行验证，发现四个模型族中涌现顺序高度一致（ρ=0.81）。
语言、语言模型与我们正在讨论的内容
本文将语言模型视为语言产物，使用意大利模型来探究其训练过程，并质疑其对语言的表征。文章主张在自然语言处理中区分技术产品和研究工具。
基于智能体的形态交替模式演化模型
本文介绍了使用多智能体模拟语言中形态交替模式（如'go/went'）的出现，并利用AI历史语言学家（基于大语言模型）评估演化形态相对于真实语言的合理性。
仅在混淆处可检测：经验证的重复计数揭示语言模型中的成员证据
本文通过使用OLMo-2和Pythia开放预训练语料库中的精确重复计数来研究语言模型中的成员推理，揭示了典型的重复水平显示出最小的暴露痕迹，并且明显的成员信号通常与句子知名度等因素混淆。