Language Acquisition Device in Large Language Models

# Language Acquisition Device in Large Language Models
Source: [https://arxiv.org/html/2605.16758](https://arxiv.org/html/2605.16758)
Masato Mita Taiga Someya Ryo Yoshida Yohei Oseki The University of Tokyo \{mita, tsomeya, yoshiryo0617, oseki\}@g\.ecc\.u\-tokyo\.ac\.jp

###### Abstract

Large Language Models \(LLMs\) remain substantially less data\-efficient than humans\. Pre\-pretraining \(PPT\) on synthetic languages has been proposed to close this gap, with prior work emphasizing highly expressive formal languages such askk\-Shuffle Dyck\. Inspired by the*Language Acquisition Device \(LAD\)*hypothesis, which posits that innate constraints preemptively restrict the learner’s hypothesis space to natural\-language\-like structure, we propose*LAD\-inspired PPT*: pre\-pretraining onMP\-Struct, a formal language whose strings encode hierarchical composition, feature\-based dependencies, and long\-distance displacement viaMerge,Agree, andMove\. A brief 500\-step PPT withMP\-Structmatches strong formal\-language baselines in token efficiency while additionally imparting a human\-like resistance to structurally implausible languages \(e\.g\.,Reverse\)\. Analyzing simplified variants, we find thatMP\-Struct Coreoutperformskk\-Shuffle Dyck despite not being definable in C\-RASP \(a formal bound on transformer expressivity\), challenging the prior hypothesis that effective PPT languages must be both hierarchically expressive and circuit\-theoretically learnable\. We show that*functional landmarks*, which reduce dependency resolution ambiguity, are a key driver, suggesting that effective PPT design depends not only on expressivity but also on the accessibility of dependency resolution\.

Language Acquisition Device in Large Language Models

Masato Mita Taiga Someya Ryo Yoshida Yohei OsekiThe University of Tokyo\{mita, tsomeya, yoshiryo0617, oseki\}@g\.ecc\.u\-tokyo\.ac\.jp

## 1Introduction

Large language models \(LLMs\) exhibit general linguistic abilities comparable to those of humans; however, their efficiency in language acquisition remains far inferior\. While humans acquire language from limited text, LLMs typically require orders of magnitude more data to achieve strong performanceWarstadtet al\.\([2023](https://arxiv.org/html/2605.16758#bib.bib108)\)\. This gap suggests that current LLMs rely on learning from an overly permissive hypothesis spaceYunet al\.\([2020](https://arxiv.org/html/2605.16758#bib.bib9)\), leaving substantial room for improving learning efficiency through better inductive biases\.

Recent work byHuet al\.\([2025](https://arxiv.org/html/2605.16758#bib.bib47)\)explores*pre\-pretraining \(PPT\)*, where models are first trained on synthetic sequences to acquire useful structural biases before standard pretraining\. They show that highly expressive formal languages, such askk\-Shuffle Dyck, can improve token efficiency by exercising the model’s ability to process hierarchical dependencies\. They further interpret these results through the*expressivity hypothesis*, which posits that a formal language conferring a helpful inductive bias should be hierarchically structured in the sense of the*Chomsky hierarchyChomsky \([1959](https://arxiv.org/html/2605.16758#bib.bib1)\)*\(either context\-free or context\-sensitive\) and definable in C\-RASPYang and Chiang \([2024](https://arxiv.org/html/2605.16758#bib.bib24)\), a formal measure of*circuit complexity*for transformers\.

However, such formal languages prioritize abstract structural complexity and often lack key properties characteristic of natural language\. This raises the question:beyond Chomsky\-hierarchy complexity and circuit complexity, do natural\-language\-like properties, such as dependencies conditioned on fixed hierarchical structure, also contribute to effective PPT?In this work, we approach this question from a complementary perspective\. Taking inspiration from the*Language Acquisition Device \(LAD\)*hypothesisChomsky \([1965](https://arxiv.org/html/2605.16758#bib.bib103)\), which suggests that innate structural constraints can restrict the hypothesis space and favor natural\-language\-like structure, we ask whether incorporating such constraints into synthetic sequences can further improve learning efficiency\. Guided by this idea, we designMP\-Struct, a synthetic generator taking cues from the*Minimalist Program \(MP\)*, which produces sequences where dependencies are embedded within a fixed hierarchical structure rather than freely interleaved\.

We evaluate LAD\-inspired PPT onPythia\-1BBidermanet al\.\([2023](https://arxiv.org/html/2605.16758#bib.bib43)\)\. A brief 500\-step PPT phase withMP\-Structconsistently improves token efficiency over training from scratch and achieves performance comparable to strong baselines based on formal languages such askk\-Shuffle Dyck\. Moreover, the resulting models exhibit a directionally asymmetric inductive bias: compared tokk\-Shuffle Dyck,MP\-Structshows greater resistance to directionally reversed sequences, consistent with the LAD\-inspired design goal of favoring natural\-language\-like directional structure\.

To better understand these effects, we analyze simplified variants of the generator\. Notably,MP\-Struct Core, an idealized abstraction of our generator, achieves higher efficiency thankk\-Shuffle Dyck despite not being definable in C\-RASP, a formal lower bound on the expressivity of future\-masked soft attention transformers\. This observation is not fully explained by the expressivity hypothesis, which predicts that effective PPT languages should be hierarchically structured and definable in C\-RASP\. Our analysis instead points to a complementary factor: the*accessibility*of dependency retrieval\. While structures that encode dependencies purely through bracketed hierarchy \(e\.g\.,kk\-Shuffle Dyck\) define valid dependencies, they may leave multiple plausible antecedents, potentially leading to higher retrieval ambiguity\. In contrast,MP\-StructandMP\-Struct Coreintroduce explicit structural cues—*functional landmarks*—that can make dependencies easier to locate, thereby reducing the effective search cost for attention\-based models\. We hypothesize that these differences in accessibility contribute to the observed efficiency gains\. More broadly, this suggests that effective PPT design may depend not only on formal expressivity, but also on how structural information is organized to support efficient dependency retrieval\.

## 2Related Work

### 2\.1LAD/UG and Minimalism

A longstanding challenge in language acquisition is explaining how children converge on rich grammatical competence from comparatively limited and noisy input \(often termed thepoverty of the stimulus\)Chomsky \([1965](https://arxiv.org/html/2605.16758#bib.bib103)\); Clark and Lappin \([2011](https://arxiv.org/html/2605.16758#bib.bib86)\)\. The LAD hypothesis addresses this gap by proposing that learners are endowed with Universal Grammar \(UG\), a species\-specific set of constraints that sharply restricts the hypothesis space of possible grammars\. From this perspective, UG serves as a strong innate inductive bias, filtering out “non\-human\-like” hypotheses*a priori*\.

In contemporary generative grammar, the*Minimalist Program*\(MP\) refines this LAD/UG model by seeking to minimize the computational machinery of language to a small set of operationsChomsky \([1995](https://arxiv.org/html/2605.16758#bib.bib2),[2000](https://arxiv.org/html/2605.16758#bib.bib3),[2001](https://arxiv.org/html/2605.16758#bib.bib4)\)\. A central component of this framework isMerge, a combinatory operation that builds hierarchical structure\. In addition, operations such asMove\(displacement\) andAgree\(feature valuation\) are commonly assumed to play roles in dependency formation and morphosyntactic licensingChomsky \([2000](https://arxiv.org/html/2605.16758#bib.bib3),[2001](https://arxiv.org/html/2605.16758#bib.bib4)\)\.

### 2\.2Pre\-pretraining on Synthetic Structures

Algorithm 1Data Generation Procedure \(MP\-Struct\)Notation:ℒ\\mathcal\{L\}: lexicon,V/N/DV/N/D: lexical categories,v​PvP: verb phrase,T/CT/C: functional heads,tt: trace,u/i​N​u​mu/iNum: number features,w​h∈\{\+,−\}wh\\in\\\{\+,\-\\\}\.

1:Input:lexicon

ℒ\\mathcal\{L\}, parameters

θ\\theta
2:Output:token sequence

SS
3:Step 1: Base Structure viaMerge

4:Sample lexical items

V,D1,D2,N1,N2∼ℒV,D\_\{1\},D\_\{2\},N\_\{1\},N\_\{2\}\\sim\\mathcal\{L\}
5:

D​Ps​u​b​j=Merge​\(D1,N1\)DP\_\{subj\}=\\textsc\{Merge\}\(D\_\{1\},N\_\{1\}\),

D​Po​b​j=Merge​\(D2,N2\)DP\_\{obj\}=\\textsc\{Merge\}\(D\_\{2\},N\_\{2\}\)
6:

V′=Merge​\(V,D​Po​b​j\)V^\{\\prime\}=\\textsc\{Merge\}\(V,DP\_\{obj\}\);

v​P=Merge​\(D​Ps​u​b​j,V′\)vP=\\textsc\{Merge\}\(DP\_\{subj\},V^\{\\prime\}\)
7:// Yields a hierarchical phrase structure with subject and object

8:Step 2: Functional Structure andAgree

9:Assign number feature

i​N​u​m∈\{s​g,p​l\}iNum\\in\\\{sg,pl\\\}to

D​Ps​u​b​jDP\_\{subj\}
10:Create

T​\[u​N​u​m\]T\[uNum\]andMergewith

v​PvP
11:Set

u​N​u​m←i​N​u​muNum\\leftarrow iNumvia

Agree​\(T,D​Ps​u​b​j\)\\textsc\{Agree\}\(T,DP\_\{subj\}\)
12:Form

T​P=\[T​P​D​Ps​u​b​j​T​\[u​N​u​m\]​v​P\]TP=\[TP\\ DP\_\{subj\}\\ T\[uNum\]\\ vP\]
13:// Yields a subject–verb agreement dependency encoded in the structure

14:Step 3:Move\(Dependency Encoding\)

15:Copy

D​Ps​u​b​jDP\_\{subj\}to clause\-initial position

16:Replace its original position with a trace:

ts​u​b​jt\_\{subj\}
17:Form

T​P=\[T​P​D​Ps​u​b​j​T​\[v​P​ts​u​b​j​\[V′​V​D​Po​b​j\]\]\]TP=\[TP\\ DP\_\{subj\}\\ T\\ \[vP\\ t\_\{subj\}\\ \[V^\{\\prime\}\\ V\\ DP\_\{obj\}\]\]\]
18:Sample

w​h∈\{\+,−\}wh\\in\\\{\+,\-\\\}
19:Merge

C​\[w​h\]C\[wh\]with

T​PTP
20:if

w​h=\+wh=\+then

21:Select a

D​PDPin

T​PTPand mark it as

D​Pw​hDP\_\{wh\}
22:Copy

D​Pw​hDP\_\{wh\}to clause\-initial position

23:Replace its original position with a trace

tw​ht\_\{wh\}
24:Form

C​P=\[C​P​D​Pw​h​C​\[T​P​…​tw​h​…\]\]CP=\[CP\\ DP\_\{wh\}\\ C\\ \[TP\\ \.\.\.\\ t\_\{wh\}\\ \.\.\.\]\]
25:end if

26:// Yields a long\-distance dependency between a moved element and its trace

27:Step 4: Linearization

28:Traverse the tree in pre\-order

29:Output brackets, nonterminal labels, features, and traces

30:Remove lexical terminals \(

V,N,DV,N,D\)

→S\\rightarrow S
31:// Yields a token sequence encoding structural relations without lexical content

Pre\-pretraining \(PPT\) on synthetic structures has emerged as a new learning paradigm for language models\. Existing studies have reported that training models via next\-token prediction on data possessing hierarchical structures—such as MIDI music, programming languages, or specific formal languages—can impart useful inductive biases, thereby improving the efficiency of subsequent natural language learningPapadimitriou and Jurafsky \([2020](https://arxiv.org/html/2605.16758#bib.bib21)\); Ri and Tsuruoka \([2022](https://arxiv.org/html/2605.16758#bib.bib25)\); Papadimitriou and Jurafsky \([2023](https://arxiv.org/html/2605.16758#bib.bib26)\)\.

More recently,Huet al\.\([2025](https://arxiv.org/html/2605.16758#bib.bib47)\)advanced this paradigm by introducing the*Expressivity Hypothesis*, which posits that a formal language conferring a helpful inductive bias should be hierarchically structured—either context\-free or context\-sensitive—and definable in C\-RASPYang and Chiang \([2024](https://arxiv.org/html/2605.16758#bib.bib24)\)\. However, while ’s approach successfully exercises the model’s generic computational capacity, it primarily focuses on abstract structural expressivity rather than properties characteristic of natural language\. For instance,kk\-Shuffle Dyck defines dependencies purely through bracket\-matching rules, allowing flexible crossing patterns but lacking the asymmetric, head\-driven organization typically observed in natural language\. This raises the question of whether incorporating more natural\-language\-like structural properties into synthetic sequences could lead to more effective inductive biases\.

In this work, we take the LAD/UG perspective as motivation: rather than deriving a formal grammar, we operationalize the structural properties that these operations are assumed to encode—hierarchical composition, feature\-based dependencies, and long\-distance displacement—as explicit sequence\-level patterns for use in a PPT setting\.

## 3Methods

The goal of LAD\-inspired PPT is to inject an inductive bias that proactively restricts the model’s hypothesis space to natural\-language\-like structure before standard pretraining begins\. To this end, we proposeMP\-Struct, a data generator designed to produce*serialized structural representations*—token sequences that make explicit the hierarchical and dependency structure assumed to underlie natural language\. The generator is not intended to model natural language itself, nor does it derive sequences from a formal grammar\. Instead, it operationalizes three structural properties—hierarchical composition \(Merge\), feature\-based agreement \(Agree\), and long\-distance displacement \(Move\)—as abstract sequence\-level patterns, stripped of all lexical content\.

MP\-Structgenerates data according to Algorithm[1](https://arxiv.org/html/2605.16758#alg1)in the following steps\.

#### Step 1: Base Structure viaMerge

We sample lexical items from a lexicon and construct av​PvPbottom\-up using theMergeoperation\. This procedure yields a recursive hierarchical structure—rather than a flat sequence—which forms the structural backbone of the generated data\.

#### Step 2: Functional Structure andAgree

We introduce a functional headTTwith an uninterpretable number feature \(u​N​u​muNum\), and assign an interpretable number feature \(i​N​u​miNum\) to the subjectD​PDP\. The value ofu​N​u​muNumis then determined via agreement with the subject\. This step encodes feature\-based dependencies within the hierarchical structure\.

#### Step 3:Move\(Dependency Encoding\)

We encode long\-distance dependencies by copying elements to higher structural positions and replacing their original occurrences with traces\. Specifically, the subjectD​PDPis copied to a higher position, forming a dependency between the copied element and its trace\. We optionally introduce a complementizerCCwith a binaryw​hwhfeature; whenw​h=\+wh=\+, aD​PDPis selected, copied to a higher position, and linked to its original position via a trace\. These operations result in structured dependencies spanning multiple hierarchical levels\.

#### Step 4: Linearization

We traverse the derived tree and output a sequence consisting of structural brackets, nonterminal labels, features, and traces, while omitting lexical items\. This design isolates structural information from lexical content: by removing lexical tokens, the model cannot rely on surface co\-occurrence patterns and is instead encouraged to process hierarchical structure and dependency relations directly\. In the context of pre\-pretraining, this is intended to encourage the model to acquire linguistically motivated inductive biases independently of lexical semantics, which may transfer to subsequent natural language pretraining\.

## 4Experiments

We test whether introducing linguistically motivated inductive biases through PPT can improve the efficiency of subsequent natural language learning\.

### 4\.1Experimental Setup

#### Model and training protocol\.

We follow the blockwise learning paradigm ofHuet al\.\([2025](https://arxiv.org/html/2605.16758#bib.bib47)\)and usePythia\-1BBidermanet al\.\([2023](https://arxiv.org/html/2605.16758#bib.bib43)\)as the base model\. Each run consists of \(i\) an optional PPT phase and \(ii\) a natural\-language pretraining phase\. For pretraining \(PT\), we train onC4Raffelet al\.\([2019](https://arxiv.org/html/2605.16758#bib.bib19)\)for25,00025\{,\}000optimization steps\. For all PPT conditions, we fix the budget for the synthetic PPT phase to500500steps, and subsequently we transfer the parameters to initialize the standard PT\. All experiments are conducted with three random seeds, and we report the mean over seeds\. The details of the training hyperparameters are provided in Appendix[A](https://arxiv.org/html/2605.16758#A1)\.

#### PPT corpora\.

For PPT, we pretrain on either \(i\) unstructured synthetic sequences \(Random\), \(ii\) formal languages with explicit recursion and/or crossing dependencies \(1\-Dyck,kk\-Shuffle Dyck\), or \(iii\) our LAD\-inspired structural representations \(MP\-Struct\)\. All synthetic sequences are tokenized with thePythia\-1Btokenizer\. Detailed generation hyperparameters are provided in Appendix[B](https://arxiv.org/html/2605.16758#A2)\.

### 4\.2Baselines

We compareMP\-Structwith the following baselines to isolate specific sources of efficiency gains:

- •Non\-PPT:Standard pretraining from random initialization\. This serves as the baseline to quantify the absolute benefit of introducing any PPT phase\.
- •Random:An unstructured PPT control trained on i\.i\.d\. uniformly sampled tokens\. This verifies that gains are not merely due to additional gradient updates or data exposure, but specifically stem fromstructuralinductive bias\.
- •1\-Dyck:A minimal recursion baseline representing pure context\-free structure \(definable in C\-RASP\)\. This tests the sufficiency of pure recursive nesting without the complexity of crossing dependencies\.
- •kk\-Shuffle Dyck:The current state\-of\-the\-art formal biasHuet al\.\([2025](https://arxiv.org/html/2605.16758#bib.bib47)\)\. We specifically adopt the configuration withk=64k=64, following the base configuration ofHuet al\.\([2025](https://arxiv.org/html/2605.16758#bib.bib47)\)\. It is a context\-sensitive language yet remains definable in C\-RASP, theoretically necessitating both Stack\- and Queue\-like memory operations\.

![Refer to caption](https://arxiv.org/html/2605.16758v1/x1.png)Figure 1:Comparison of C4 validation loss at 25,000 steps across different pre\-pretraining conditions\. The left section comparesMP\-Structagainst baselines, while the right section \(separated by the dashed line\) presents an ablation study removing specific linguistic components \(Merge,Move,Agree\)\.Table 1:BLiMP, MRS, and Efficiency Gain \(Efficiency\)\.†indicates a significant difference from Non\-PPT at the 5% level\. “–” indicates that the condition did not improve upon Non\-PPT \(i\.e\., yielded negative MRS and Efficiency values\), and is therefore excluded from the efficiency comparison\.
### 4\.3Evaluation Metrics

Following prior work on PPTHuet al\.\([2025](https://arxiv.org/html/2605.16758#bib.bib47)\), we evaluate whether improvements in learning efficiency are achieved without degrading the quality of the acquired grammar\.

- •Learning Efficiency:We use two metrics fromHuet al\.\([2025](https://arxiv.org/html/2605.16758#bib.bib47)\)\. Lety1y\_\{1\}be the number of PT steps for the Non\-PPT baseline,xxthe number of PPT steps, andy2y\_\{2\}the number of PT steps at which the PPT model first matches the loss of Non\-PPT aty1y\_\{1\}\.Marginal Rate of Substitution \(MRS\)measures how many PT steps are saved per PPT step: MRS=y1−y2x\\text\{MRS\}=\\frac\{y\_\{1\}\-y\_\{2\}\}\{x\}\(1\)Efficiency Gainmeasures the reduction in total training steps required to reach matched performance: Efficiency Gain=1−y2\+xy1\\text\{Efficiency Gain\}=1\-\\frac\{y\_\{2\}\+x\}\{y\_\{1\}\}\(2\)A concrete calculation example is provided in Appendix[D](https://arxiv.org/html/2605.16758#A4)\.
- •Grammatical Generalization:We use BLiMPWarstadtet al\.\([2020](https://arxiv.org/html/2605.16758#bib.bib85)\), which evaluates English grammar using minimal pairs\.

### 4\.4Results

Figure[1](https://arxiv.org/html/2605.16758#S4.F1)presents the C4 training loss after 25,000 steps, and Table[1](https://arxiv.org/html/2605.16758#S4.T1)summarizes the grammatical generalization and the learning efficiency\.

#### 1\. Learning Efficiency Gains fromMP\-Struct\.

MP\-Structoutperforms both the Non\-PPT baseline and the unstructured Random control in terms of best loss \(Figure[1](https://arxiv.org/html/2605.16758#S4.F1), left\)\. Quantitatively,MP\-Structachieves an MRS of 15\.3 on average relative to Non\-PPT, corresponding to an average efficiency gain of 29% \(up to 35%\)\. In contrast, the 1\-Dyck baseline does not yield consistent improvements, suggesting that recursive structure alone may not be sufficient to improve learning efficiency\.

#### 2\. Synergy of Linguistic Operations\.

To isolate the drivers of this efficiency gain, we conducted an ablation study \(Figure[1](https://arxiv.org/html/2605.16758#S4.F1), right\)\. The results reveal that removing any single component of the generator—Merge,Agree, orMove—results in a worse final loss compared to the fullMP\-Structmodel\. This confirms that the efficiency gain is not driven by any isolated feature but by thesynergistic interactionof hierarchical phrase structure \(Merge\) and functional dependencies \(Agree/Move\), validating the theoretical design of Algorithm[1](https://arxiv.org/html/2605.16758#alg1)\.

#### 3\. Comparable Performance tokk\-Shuffle Dyck\.

MP\-Structachieves an average efficiency gain of 29%, comparable to the strong baselinekk\-Shuffle Dyck \(29%\)\. Whilekk\-Shuffle Dyck shows marginally higher BLiMP scores,MP\-Structyields a BLiMP score comparable to the Non\-PPT baseline \(0\.7550\.755vs\.0\.7580\.758\)\. This suggests that the induced inductive bias primarily facilitates learning efficiency rather than improving final grammatical generalization\. These results indicate that linguistically motivated inductive biases can serve as an alternative to high\-expressivity baselines for improving token efficiency, and motivate further analysis of the factors underlying these gains \(§[6](https://arxiv.org/html/2605.16758#S6)\)\.

## 5Analysis I: Quality of Inductive Bias

![Refer to caption](https://arxiv.org/html/2605.16758v1/x2.png)Figure 2:Robustness against semantic perturbation\(Δsens=ℒJW−ℒNL\\Delta\_\{\\text\{sens\}\}=\\mathcal\{L\}\_\{\\text\{JW\}\}\-\\mathcal\{L\}\_\{\\text\{NL\}\}\)\. This metric quantifies the performance gap between semantic\-free Jabberwocky inputs and natural language, where lower values indicate less reliance on lexical co\-occurrence\.![Refer to caption](https://arxiv.org/html/2605.16758v1/x3.png)\(a\)Shuffle
![Refer to caption](https://arxiv.org/html/2605.16758v1/x4.png)\(b\)Reverse
![Refer to caption](https://arxiv.org/html/2605.16758v1/x5.png)\(c\)Hop

Figure 3:Structural selectivity\(Δsel=ℒImp−ℒNL\\Delta\_\{\\text\{sel\}\}=\\mathcal\{L\}\_\{\\text\{Imp\}\}\-\\mathcal\{L\}\_\{\\text\{NL\}\}\) across three impossible language conditions\. Positive values indicate a human\-like preference for natural linguistic constraints over impossible distortions\.Having established the efficiency ofMP\-Struct, we now investigate the*nature*of the acquired biases\.

### 5\.1Validation of Structural Robustness

To examine whether the observed efficiency gains are associated with improved structural processing, we perform a*Jabberwocky \(JW\)*meaning attenuation analysisCarroll \([1871](https://arxiv.org/html/2605.16758#bib.bib17)\)\. This analysis is intended to probe the extent to which models rely on structural information independently of lexical semantics\. We construct JW variants of both the training \(C4\) and evaluation \(WikitextMerityet al\.\([2016](https://arxiv.org/html/2605.16758#bib.bib18)\)\) corpora by randomly replacing content words while preserving function words, punctuation, and word order, conditioned on fine\-grained POS tags to preserve morphology\.111We provide the details of data construction in Appendix[E](https://arxiv.org/html/2605.16758#A5)

The engineering significance of this analysis lies in quantifying the*disentanglement*of syntactic processing from semantic correlation\. Current LLMs often rely on lexical co\-occurrence statistics to minimize lossDziriet al\.\([2023](https://arxiv.org/html/2605.16758#bib.bib8)\); Berglundet al\.\([2024](https://arxiv.org/html/2605.16758#bib.bib7)\)\. However, a robust language learner is expected to maintain predictive performance even when semantic cues are reduced, relying instead on structural regularitiesGulordavaet al\.\([2018](https://arxiv.org/html/2605.16758#bib.bib6)\)\.

To assess this, we train separate models under two conditions: natural language \(NL\) and its Jabberwocky \(JW\) counterpart, and evaluate each model on the corresponding data \(i\.e\., NL→\\rightarrowNL and JW→\\rightarrowJW\)\. We then compare their losses to quantify sensitivity to semantic information\.

We define sensitivity asΔsens=ℒJW−ℒNL\\Delta\_\{\\text\{sens\}\}=\\mathcal\{L\}\_\{\\text\{JW\}\}\-\\mathcal\{L\}\_\{\\text\{NL\}\}, whereℒNL\\mathcal\{L\}\_\{\\text\{NL\}\}andℒJW\\mathcal\{L\}\_\{\\text\{JW\}\}denote the losses obtained under the NL→\\rightarrowNL and JW→\\rightarrowJW conditions, respectively\. A smallerΔsens\\Delta\_\{\\text\{sens\}\}indicates that performance degrades less when semantic information is attenuated, which may suggest greater reliance on structural information\.

#### Results\.

Figure[2](https://arxiv.org/html/2605.16758#S5.F2)shows that while Non\-PPT relies heavily on semantic co\-occurrence,MP\-Structachieves a lowerΔsens\\Delta\_\{\\text\{sens\}\}than the strong baselinekk\-Shuffle Dyck\.

Whilekk\-Shuffle Dyck theoretically requires memory operations capable of tracking long\-distance dependencies, its symbol types lack explicit cues that differentiate their structural roles, potentially leaving multiple plausible antecedents and leading to higher dependency retrieval ambiguity\. In contrast,MP\-Structintroduces distinct structural markers \(e\.g\., functional categories such asTTandCC\), which may help reduce ambiguity in identifying relevant dependencies\.

One possible interpretation is that such explicit cues make it easier for the model to rely on structural information when semantic content is attenuated\. Accordingly, the improved robustness ofMP\-Structunder meaning attenuation may reflect a greater reliance on structural regularities, rather than lexical co\-occurrence alone\. To further investigate this hypothesis, we analyze the factors underlying these efficiency gains in §[6](https://arxiv.org/html/2605.16758#S6)\.

### 5\.2Resistance to Impossible Languages

We next examine whether the initialization induced byMP\-Structtends to align with constraints characteristic of human language \(UG\) by probing learning behavior on*impossible languages*\. Following the protocol ofKalliniet al\.\([2024](https://arxiv.org/html/2605.16758#bib.bib20)\), we construct synthetically perturbed corpora by applying deterministic transformations to the NL training data\. Specifically, we adopt the following implementations from their framework to violate specific linguistic universals while retaining statistical regularities:222We provide the details of data construction in Appendix[F](https://arxiv.org/html/2605.16758#A6)\.

- •Shuffle: We useDeterministicShufflewith a window size ofs=21s=21\. This operation permutes tokens deterministically within a fixed local window, destroying localnn\-gram statistics and syntactic constituency while preserving the global bag\-of\-words distribution\.
- •Reverse: We useFullReverse, which reverses the token order of the entire sequence\. While computationally deterministic \(requiring a stack\), this transformation violates the incremental, left\-to\-right processing constraint fundamental to human language performance\.
- •Hop: We useWordHop\(specifically with a 4\-word shift\)\. This transformation introduces a dependency based on linear counting: a functional marker is placed at a fixed linear distance \(four words\) after its associated verb\. This mimics “impossible” grammatical rules that rely on counting word positions rather than structural configurations\.

We quantify the model’s preference for natural constraints using*structural selectivity*:Δsel=LImp−LNL\\Delta\_\{\\text\{sel\}\}=L\_\{\\text\{Imp\}\}\-L\_\{\\text\{NL\}\}\.*A larger \(positive\)Δ*sel*\\Delta\_\{\\text\{sel\}\}*indicates that the model finds natural language significantly easier to learn than impossible languages, implying a bias toward human\-like structures\.

#### Results and Discussion\.

Figure[3](https://arxiv.org/html/2605.16758#S5.F3)presents the structural selectivity scores \(Δsel\\Delta\_\{\\text\{sel\}\}\) at 25,000 pretraining steps across seeds\. In theShufflecondition \(Fig\.[3](https://arxiv.org/html/2605.16758#S5.F3)a\), all models exhibit consistently positiveΔsel\\Delta\_\{\\text\{sel\}\}, indicating that a preference for local structural coherence is broadly shared regardless of PPT condition\. In theHopcondition \(Fig\.[3](https://arxiv.org/html/2605.16758#S5.F3)c\),Δsel\\Delta\_\{\\text\{sel\}\}values are negative across conditions, reflecting the Transformer’s inherent capacity to capture long\-distance dependencies\. These two conditions do not clearly differentiate the PPT conditions from each other or from Non\-PPT\.

The most informative divergence appears in theReversecondition \(Fig\.[3](https://arxiv.org/html/2605.16758#S5.F3)b\)\. Here,kk\-Shuffle Dyck yieldsΔsel≈0\\Delta\_\{\\text\{sel\}\}\\approx 0, suggesting that it incentivizes the acquisition of generic processing strategies capable of handling sequences in any direction—an excessive flexibility that lacks linguistic structural constraints\. In contrast,MP\-Structmaintains a clearly positiveΔsel\\Delta\_\{\\text\{sel\}\}, indicating resistance to reversed sequences\. This resistance is not incidental: the sequences generated byMP\-Structencode a fundamentally directional structure, in which displacement consistently targets structurally higher positions to the left, imposing a directional asymmetry on the linearized sequence\. Reversing the input string directly violates these directional biases, making reversed sequences genuinely harder to process for a model that has internalized them\. We interpret this as evidence thatMP\-Structinstills a directionally asymmetric inductive bias, consistent with the LAD\-inspired design goal of favoring natural\-language\-like structure\.

## 6Analysis II: Drivers of Efficiency

Table 2:Decomposition of Abstract Generative Conditions\.We contrastGenerickk\-SDwithMP\-Struct Core, which introduces diverse functional heads as explicit landmarks\. Both conditions share the same set of dependency types; they differ in whether dependencies are randomly interleaved \(Generickk\-SD\) or organized within a fixed hierarchical topology with landmark tokens adjacent to each dependency site \(MP\-Struct Core\)\. Colors:Hierarchy \[ \],Dependency \( \), andHeads\.Algorithm 2Data Generation Procedure \(MP\-Struct Core\)Notation:\[⋅\]\[\\,\\cdot\\,\]/\(⋅\)\(\\,\\cdot\\,\): structural/dependency bracket pair,T/CT/C: functional heads,tt: trace,AGR∈\{AGRA,AGRB\}\\mathrm\{AGR\}\\in\\\{\\mathrm\{AGR\}\_\{A\},\\mathrm\{AGR\}\_\{B\}\\\}: agreement dependency \(subject–verb number agreement\),SEL\\mathrm\{SEL\}: selectional dependency \(head selects its complement, e\.g\.,DDselectsNN\),MOVE\\mathrm\{MOVE\}: movement dependency,HXH\_\{X\}: head token for layerX∈\{C​P,T​P,V​P\}X\\in\\\{CP,TP,VP\\\},w​h∈\{\+,−\}wh\\in\\\{\+,\-\\\}\.

1:Input:sequence length

LL
2:Output:token sequence

SS
3:Step 1: Base Structure viaMerge

4:Sample

w​h∈\{\+,−\}wh\\in\\\{\+,\-\\\}and

AGR∈\{AGRA,AGRB\}\\mathrm\{AGR\}\\in\\\{\\mathrm\{AGR\}\_\{A\},\\mathrm\{AGR\}\_\{B\}\\\}
5:Construct

v​PvPcontaining head

HV​PH\_\{VP\}, a subject slot \(trace

ttif

w​h=\+wh\{=\}\+, else empty\), and one object linked via

SEL\\mathrm\{SEL\}dependency

6:Shuffle

v​PvP\-internal elements to vary surface order

7:// Yields an abstract verb phrase with a selectional dependency between head and object

8:Step 2: Functional Structure andAgree

9:Assign

AGR\\mathrm\{AGR\}feature to

D​Ps​u​b​jDP\_\{subj\}; create

T​\[HT​P,AGR\]T\[H\_\{TP\},\\,\\mathrm\{AGR\}\]andMergewith

v​PvP
10:if

w​h=\+wh=\+thenleave Spec\-TP emptyelseplace

D​Ps​u​b​jDP\_\{subj\}at Spec\-TPend if

11:Form

T​P=\[T​P​Spec​\-​TP​T​v​P\]TP=\[TP\\ \\mathrm\{Spec\\text\{\-\}TP\}\\ T\\ vP\]
12:// Yields a subject–verb agreement dependency marked by landmarkHT​PH\_\{TP\}

13:Step 3:Move\(Dependency Encoding\)

14:if

w​h=\+wh=\+then

15:Copy

D​Ps​u​b​jDP\_\{subj\}to Spec\-CP; leave trace

ts​u​b​jt\_\{subj\}at its

v​PvP\-internal position

16:Attach licensor

MOVE\\mathrm\{MOVE\}to

CC; form

C​P=\[C​P​D​Ps​u​b​j​C​\[HC​P,MOVE\]​T​P\]CP=\[CP\\ DP\_\{subj\}\\ C\[H\_\{CP\},\\,\\mathrm\{MOVE\}\]\\ TP\]
17:else

18:Form

C​P=\[C​P​\(empty\)​C​\[HC​P\]​T​P\]CP=\[CP\\ \\text\{\(empty\)\}\\ C\[H\_\{CP\}\]\\ TP\]
19:end if

20:// Yields a long\-distance movement dependency anchored by landmarkHC​PH\_\{CP\}

21:Step 4: Linearization

22:Traverse the tree in pre\-order

23:Output brackets, dependency markers, and traces

→S\\rightarrow S
24:Repeat Steps 1–4 and concatenate until

\|S\|≥L\|S\|\\geq L
25:// Yields a token sequence where each dependency site is marked by an unambiguous landmark

![Refer to caption](https://arxiv.org/html/2605.16758v1/x6.png)Figure 4:Comparison of C4 loss at 25,000 steps across abstract conditions\.The results in §[5\.1](https://arxiv.org/html/2605.16758#S5.SS1)suggest that the observed efficiency gains may be related to differences in how structural information is represented, particularly the availability of explicit cues for dependency resolution\. This raises a more specific question:are these gains driven solely by structural expressivity \(i\.e\., C\-RASP definability\), or by how dependencies are organized and made accessible to the model?

### 6\.1Decomposition and Abstract Conditions

To better isolate the factors contributing to learning efficiency, we construct a controlled experimental setting in which the set of structural operations is held constant across conditions\. Specifically, both conditions share the same primitive operations asMP\-Struct: one type of recursive structure and four types of functional dependencies \(see Appendix[G](https://arxiv.org/html/2605.16758#A7)for details\)\. The only factor varied is how these components are organized, allowing us to attribute any difference in efficiency directly to the organization of dependencies rather than to their number or type\. Within this controlled setting, we analyze how the*organization*of dependencies affects learning, focusing on what we term*dependency identification ambiguity*\.

Dependency identification ambiguity refers to the extent to which a dependency endpoint \(e\.g\.,\)1\) provides sufficient cues to uniquely identify its corresponding start point \(e\.g\.,\(1\)\. When such cues are limited, multiple candidate antecedents remain plausible, leading to high ambiguity\. Conversely, when structural markers are present near relevant positions, the set of candidates can be sharply constrained, resulting in lower ambiguity\.

![Refer to caption](https://arxiv.org/html/2605.16758v1/x7.png)Figure 5:The trajectory of the training loss\.Based on this perspective, we define two contrasting conditions:

- •1\. Generickk\-SD:A condition where the same primitive components are randomly interleaved\. As illustrated in Table[2](https://arxiv.org/html/2605.16758#S6.T2), multiple dependency start points of the same type \(e\.g\., two instances of\(1\) may appear in an unstructured manner, providing weak cues for identifying which one corresponds to a given endpoint \(e\.g\.,\)1\)\. As a result, multiple candidates remain plausible, leading tohigh dependency identification ambiguity\.
- •2\.MP\-Struct Core:A distillation ofMP\-Structinto a pure abstract formal language, designed to preserve its two key structural properties while eliminating lexical content \(see Algorithm[2](https://arxiv.org/html/2605.16758#alg2)for the full generation procedure333We provide supplementary detail on the mapping fromMP\-Structto its abstract counterpart in Appendix[H](https://arxiv.org/html/2605.16758#A8)\.\): - –Fixed Topology\.The derivation strictly follows the hierarchical pathC​P→T​P→v​PCP\\to TP\\to vP, enforcing the same recursive subordination asMP\-Struct\. Unlike Generickk\-SD, where dependencies are randomly interleaved, every dependency inMP\-Struct Coreis generated within its designated structural domain\. - –Abstract Landmarks\.The functional heads ofMP\-Struct\(CC,TT,vv\) are replaced by distinct abstract tokensHC​PH\_\{CP\},HT​PH\_\{TP\},HV​PH\_\{VP\}, which are systematically placed adjacent to their associated dependency brackets\. This ensures that each dependency site is marked by an unambiguous, consistent cue, mirroring the role of functional heads without introducing lexical noise\. Together, these properties result inlow dependency identification ambiguity: the head tokens serve as explicit landmarks that sharply localize the relevant search space for each dependency\.

### 6\.2Analysis of Efficiency Factors

Figure[4](https://arxiv.org/html/2605.16758#S6.F4)summarizes the results of our abstraction study\.444The trajectory of the LM loss is provided in Figure[5](https://arxiv.org/html/2605.16758#S6.F5)\.Among the abstract conditions,MP\-Struct Coreoutperforms Generickk\-SD in terms of token efficiency\. Quantitatively,MP\-Struct Coreachieves an MRS of 16\.2 and an average Efficiency Gain of 31% \(up to 37%\) as shown in Table[1](https://arxiv.org/html/2605.16758#S4.T1)\. These values are higher than those ofkk\-Shuffle Dyck \(MRS: 15\.6, Efficiency: 29%\)\. These results suggest that differences in how structural components are organized may have a substantial impact on learning efficiency, even when the underlying set of operations—one type of recursive structure and four types of functional dependencies—is held constant across conditions\.

In line with the design described in §[6\.1](https://arxiv.org/html/2605.16758#S6.SS1), the two conditions differ primarily in the degree of dependency identification ambiguity\. Generickk\-SD exhibits higher ambiguity due to the lack of explicit cues for identifying dependency relations, whereasMP\-Struct Coreprovides more localized structural markers that help constrain the set of plausible antecedents\. From this perspective, reduced dependency identification ambiguity may contribute to more efficient learning, consistent with the hypothesis that structural accessibility—beyond expressivity alone—plays a role in effective PPT design\.

### 6\.3Theoretical Implication

A key implication of this analysis concerns the relationship to the expressivity hypothesisHuet al\.\([2025](https://arxiv.org/html/2605.16758#bib.bib47)\)\. This hypothesis posits that a formal language conferring a helpful inductive bias should be hierarchically structured and definable in C\-RASPYang and Chiang \([2024](https://arxiv.org/html/2605.16758#bib.bib24)\), the latter serving as a formal lower bound on what future\-masked soft attention transformers can express\.

However,MP\-Struct Coreis not definable in C\-RASP\. The generator enforces a strict adjacency constraint: functional head tokens \(e\.g\.,HCH\_\{C\}\) are systematically placed immediately before their associated dependency brackets, requiring a predicate that jointly references both the head position and the dependency position\. C\-RASP, being a restriction of FO\(M\) that permits only single\-index predicates per quantifierYang and Chiang \([2024](https://arxiv.org/html/2605.16758#bib.bib24)\), cannot express such a two\-position constraint\. WhileHuet al\.\([2025](https://arxiv.org/html/2605.16758#bib.bib47)\)suggest that C\-RASP\-definability is a desirable property for effective PPT languages, this is framed as a tendency rather than a strict requirement—their results show only that C\-RASP\-definable languages*generally*achieve equal or better performance, not that non\-C\-RASP\-definable languages necessarily fail\. Consistent with this,MP\-Struct Coreachieves higher efficiency thankk\-Shuffle Dyck despite not being C\-RASP\-definable\.

Taken together, these results suggest that C\-RASP\-definability is neither necessary nor sufficient for effective PPT, and that the organization of dependencies—specifically, the availability of explicit structural cues that reduce retrieval ambiguity—is a key complementary factor\.

## 7Conclusion

We proposed*LAD\-inspired PPT*, a pre\-pretraining framework in whichMP\-Struct—a formal language taking cues from the Minimalist Program—injects linguistically motivated inductive biases before standard pretraining\. Our results show that such biases improve learning efficiency comparably to strong formal baselines, while additionally instilling a directionally asymmetric inductive bias, as evidenced by greater resistance to directionally reversed sequences compared tokk\-Shuffle Dyck\. Through controlled analyses, we find that the expressivity hypothesis alone does not fully account for these gains:MP\-Struct Coreoutperformskk\-Shuffle Dyck despite not being definable in C\-RASP\. Instead, our analysis points to*functional landmarks*—explicit structural cues that reduce dependency identification ambiguity—as a key complementary factor, suggesting that effective PPT design depends not only on formal expressivity but also on how structural information is organized to support efficient dependency retrieval\.

## Limitations

While our results provide compelling evidence for the efficacy of linguistically motivated PPT, several limitations remain\.

#### Scale and Architecture\.

Our experiments were conducted primarily on the Pythia\-1B modelBidermanet al\.\([2023](https://arxiv.org/html/2605.16758#bib.bib43)\)\. While we observed consistent trends across seed runs and smaller scales, it remains to be verified whether the efficiency gains ofMP\-Structscale linearly to significantly larger models \(e\.g\., 7B or 70B parameters\) or alternative architectures \(e\.g\., State\-Space Models\)\.

#### Monolingual Evaluation\.

We evaluated grammatical generalization using BLiMPWarstadtet al\.\([2020](https://arxiv.org/html/2605.16758#bib.bib85)\), which is limited to English\. AlthoughMP\-Structis designed based on Universal Grammar principles \(e\.g\.,MergeandMove\) assumed to be language\-universal, our current validation does not explicitly confirm improved acquisition efficiency for typologically distinct languages \(e\.g\., head\-final languages like Japanese or morphologically rich languages\)\.

#### Operationalization of Dependency Identification Ambiguity\.

While we use dependency identification ambiguity as an explanatory construct, it currently lacks a formal, corpus\-independent definition that would allow quantitative comparison across arbitrary languages\. For instance, it remains unclear whetherkk\-Shuffle Dyck, C4, or other corpora exhibit higher or lower ambiguity than the conditions studied in §[6](https://arxiv.org/html/2605.16758#S6), limiting the generalizability of our claims\. Furthermore, the comparison between Generickk\-SD andMP\-Struct Coremay not isolate ambiguity as cleanly as intended: introducing landmark tokens not only reduces retrieval ambiguity but also increases vocabulary size, which may alter other properties of the language such as entropy\. Disentangling these confounds—for example, by controlling for unigram entropy or vocabulary size—remains an important direction for future work\.

## Acknowledgments

We thank the anonymous reviewers for their helpful comments and suggestions\. This work was supported by JSPS KAKENHI Grant Number JP24H00087, Grant\-in\-Aid for JSPS Fellows JP24KJ0800, JST BOOST Grant Number JPMJBY24B2, JST CREST Grant Number JPMJCR2565, and JST PRESTO Grant Number JPMJPR21C2\.

## References

- L\. Berglund, M\. Tong, M\. Kaufmann, M\. Balesni, A\. Stickland, T\. Korbak, and O\. Evans \(2024\)The reversal curse: llms trained on "a is b" fail to learn "b is a"\.InInternational Conference on Representation Learning,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 18623–18642\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/5178b2f2d7c44aa390c0777dc77b3f0c-Paper-Conference.pdf)Cited by:[§5\.1](https://arxiv.org/html/2605.16758#S5.SS1.p2.1)\.
- S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff,et al\.\(2023\)Pythia: a suite for analyzing large language models across training and scaling\.InInternational Conference on Machine Learning,pp\. 2397–2430\.Cited by:[§1](https://arxiv.org/html/2605.16758#S1.p4.2),[§4\.1](https://arxiv.org/html/2605.16758#S4.SS1.SSS0.Px1.p1.2),[Scale and Architecture\.](https://arxiv.org/html/2605.16758#Sx1.SS0.SSS0.Px1.p1.1)\.
- L\. Carroll \(1871\)Through the looking\-glass, and what alice found there\.Macmillan\.Cited by:[§5\.1](https://arxiv.org/html/2605.16758#S5.SS1.p1.1)\.
- N\. Chomsky \(1995\)The minimalist program\.Current studies in linguistics series,MIT Press\.External Links:ISBN 9780262531283,LCCN 95004654,[Link](https://books.google.co.jp/books?id=vtPQiYCNpjgC)Cited by:[§2\.1](https://arxiv.org/html/2605.16758#S2.SS1.p2.1)\.
- N\. Chomsky \(1959\)On Certain Formal Properties of Grammars\.Information and Control2\(2\),pp\. 137–167\.External Links:ISSN 0019\-9958,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/S0019-9958%2859%2990362-6),[Link](https://www.sciencedirect.com/science/article/pii/S0019995859903626)Cited by:[§1](https://arxiv.org/html/2605.16758#S1.p2.1.3)\.
- N\. Chomsky \(1965\)Aspects of the theory of syntax\.The MIT Press,Cambridge\.External Links:[Link](http://www.amazon.com/Aspects-Theory-Syntax-Noam-Chomsky/dp/0262530074)Cited by:[§1](https://arxiv.org/html/2605.16758#S1.p3.1),[§2\.1](https://arxiv.org/html/2605.16758#S2.SS1.p1.1)\.
- N\. Chomsky \(2000\)Minimalist inquiries: the framework\.InStep by Step: Essays on Minimalist Syntax in Honor of Howard Lasnik,R\. Martin, D\. Michaels, and J\. Uriagereka \(Eds\.\),pp\. 89–155\.Cited by:[§2\.1](https://arxiv.org/html/2605.16758#S2.SS1.p2.1)\.
- N\. Chomsky \(2001\)Derivation by phase\.InKen Hale: A Life in Language,External Links:ISBN 9780262316125,[Document](https://dx.doi.org/10.7551/mitpress/4056.003.0004),[Link](https://doi.org/10.7551/mitpress/4056.003.0004),https://direct\.mit\.edu/book/chapter\-pdf/2308182/9780262316125\_caa\.pdfCited by:[§2\.1](https://arxiv.org/html/2605.16758#S2.SS1.p2.1)\.
- A\. Clark and S\. Lappin \(2011\)Linguistic nativism and the poverty of the stimulus\.Wiley\-Blackwell\.Cited by:[§2\.1](https://arxiv.org/html/2605.16758#S2.SS1.p1.1)\.
- N\. Dziri, X\. Lu, M\. Sclar, X\. L\. Li, L\. Jiang, B\. Y\. Lin, P\. West, C\. Bhagavatula, R\. Le Bras, J\. D\. Hwang, S\. Sanyal, S\. Welleck, X\. Ren, A\. Ettinger, Z\. Harchaoui, and Y\. Choi \(2023\)Faith and fate: limits of transformers on compositionality\.InProceedings of the 37th International Conference on Neural Information Processing Systems,NIPS ’23,Red Hook, NY, USA\.Cited by:[§5\.1](https://arxiv.org/html/2605.16758#S5.SS1.p2.1)\.
- K\. Gulordava, P\. Bojanowski, E\. Grave, T\. Linzen, and M\. Baroni \(2018\)Colorless green recurrent networks dream hierarchically\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long Papers\),M\. Walker, H\. Ji, and A\. Stent \(Eds\.\),New Orleans, Louisiana,pp\. 1195–1205\.External Links:[Link](https://aclanthology.org/N18-1108/),[Document](https://dx.doi.org/10.18653/v1/N18-1108)Cited by:[§5\.1](https://arxiv.org/html/2605.16758#S5.SS1.p2.1)\.
- M\. Y\. Hu, J\. Petty, C\. Shi, W\. Merrill, and T\. Linzen \(2025\)Between circuits and Chomsky: pre\-pretraining on formal languages imparts linguistic biases\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 9691–9709\.External Links:[Link](https://aclanthology.org/2025.acl-long.478/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.478),ISBN 979\-8\-89176\-251\-0Cited by:[Table 3](https://arxiv.org/html/2605.16758#A1.T3),[Appendix A](https://arxiv.org/html/2605.16758#A1.p1.1),[§1](https://arxiv.org/html/2605.16758#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.16758#S2.SS2.p2.1),[4th item](https://arxiv.org/html/2605.16758#S4.I1.i4.p1.2),[1st item](https://arxiv.org/html/2605.16758#S4.I2.i1.p1.4),[§4\.1](https://arxiv.org/html/2605.16758#S4.SS1.SSS0.Px1.p1.2),[§4\.3](https://arxiv.org/html/2605.16758#S4.SS3.p1.1),[§6\.3](https://arxiv.org/html/2605.16758#S6.SS3.p1.1),[§6\.3](https://arxiv.org/html/2605.16758#S6.SS3.p2.2)\.
- J\. Kallini, I\. Papadimitriou, R\. Futrell, K\. Mahowald, and C\. Potts \(2024\)Mission: impossible language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 14691–14714\.External Links:[Link](https://aclanthology.org/2024.acl-long.787/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.787)Cited by:[Appendix F](https://arxiv.org/html/2605.16758#A6.p1.1),[§5\.2](https://arxiv.org/html/2605.16758#S5.SS2.p1.1)\.
- S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher \(2016\)Pointer sentinel mixture models\.External Links:1609\.07843Cited by:[§5\.1](https://arxiv.org/html/2605.16758#S5.SS1.p1.1)\.
- I\. Papadimitriou and D\. Jurafsky \(2020\)Learning Music Helps You Read: Using transfer to study linguistic structure in language models\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),B\. Webber, T\. Cohn, Y\. He, and Y\. Liu \(Eds\.\),Online,pp\. 6829–6839\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.554/),[Document](https://dx.doi.org/10.18653/v1/2020.emnlp-main.554)Cited by:[§2\.2](https://arxiv.org/html/2605.16758#S2.SS2.p1.1)\.
- I\. Papadimitriou and D\. Jurafsky \(2023\)Injecting structural hints: using language models to study inductive biases in language learning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 8402–8413\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.563/),[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.563)Cited by:[§2\.2](https://arxiv.org/html/2605.16758#S2.SS2.p1.1)\.
- C\. Raffel, N\. M\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2019\)Exploring the Limits of Transfer Learning with a Unified Text\-to\-Text Transformer\.Journal of Machine Learning Research21,pp\. 140:1–140:67\.External Links:[Link](https://jmlr.org/papers/volume21/20-074/20-074.pdf)Cited by:[§4\.1](https://arxiv.org/html/2605.16758#S4.SS1.SSS0.Px1.p1.2)\.
- R\. Ri and Y\. Tsuruoka \(2022\)Pretraining with artificial language: studying transferable knowledge in language models\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 7302–7315\.External Links:[Link](https://aclanthology.org/2022.acl-long.504/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.504)Cited by:[§2\.2](https://arxiv.org/html/2605.16758#S2.SS2.p1.1)\.
- A\. Warstadt, A\. Mueller, L\. Choshen, E\. Wilcox, C\. Zhuang, J\. Ciro, R\. Mosquera, B\. Paranjabe, A\. Williams, T\. Linzen, and R\. Cotterell \(2023\)Findings of the BabyLM challenge: sample\-efficient pretraining on developmentally plausible corpora\.InProceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning,pp\. 1–34\.External Links:[Link](https://aclanthology.org/2023.conll-babylm.1),[Document](https://dx.doi.org/10.18653/v1/2023.conll-babylm.1)Cited by:[§1](https://arxiv.org/html/2605.16758#S1.p1.1)\.
- A\. Warstadt, A\. Parrish, H\. Liu, A\. Mohananey, W\. Peng, S\. Wang, and S\. R\. Bowman \(2020\)BLiMP: the benchmark of linguistic minimal pairs for English\.Transactions of the Association for Computational Linguistics8,pp\. 377–392\.External Links:[Link](https://aclanthology.org/2020.tacl-1.25/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00321)Cited by:[2nd item](https://arxiv.org/html/2605.16758#S4.I2.i2.p1.1),[Monolingual Evaluation\.](https://arxiv.org/html/2605.16758#Sx1.SS0.SSS0.Px2.p1.1)\.
- A\. Yang and D\. Chiang \(2024\)Counting like transformers: compiling temporal counting logic into softmax transformers\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=FmhPg4UJ9K)Cited by:[§1](https://arxiv.org/html/2605.16758#S1.p2.1),[§2\.2](https://arxiv.org/html/2605.16758#S2.SS2.p2.1),[§6\.3](https://arxiv.org/html/2605.16758#S6.SS3.p1.1),[§6\.3](https://arxiv.org/html/2605.16758#S6.SS3.p2.2)\.
- C\. Yun, S\. Bhojanapalli, A\. S\. Rawat, S\. J\. Reddi, and S\. Kumar \(2020\)Are transformers universal approximators of sequence\-to\-sequence functions?\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2605.16758#S1.p1.1)\.

## Appendix ATraining Configurations

Table[3](https://arxiv.org/html/2605.16758#A1.T3)shows the shared training hyperparameters for PPT and PT, followingHuet al\.\([2025](https://arxiv.org/html/2605.16758#bib.bib47)\)\. For the experiment, a single NVIDIA RTX 6000 Ada \(48GB\) GPU was used, and the training time for each run was approximately 20 hours\.

Table 3:Shared training hyperparameters for PPT and PT \(identical across all models\), followingHuet al\.\([2025](https://arxiv.org/html/2605.16758#bib.bib47)\)except for max sequence length\.
## Appendix BHyperparameters forMP\-Struct

Table[4](https://arxiv.org/html/2605.16758#A2.T4)shows the hyperparameters forMP\-Struct\.

Table 4:MP\-Structcorpus generation hyperparameters\.
## Appendix CExamples used in pre\-pretraining

Table[5](https://arxiv.org/html/2605.16758#A3.T5)shows the examples used in pre\-pretraining\.

Table 5:Examples used in pre\-pretraining\.
## Appendix DCalculation Examples of Learning Efficiency Metrics

We present an actual calculation example using the results from one trial \(Seed=0\) in our experiment\. When the reference point is set toy1=25,000y\_\{1\}=25\{,\}000, the loss of the baseline \(Non\-PPT\) was approximately3\.6333\.633\. The proposed method \(MP\-Struct\) first reached this loss aty2≈15,755y\_\{2\}\\approx 15\{,\}755steps\. Since the number of formal language training steps isx=500x=500, the metrics are calculated as follows:

MRS=25,000−15,755500=9,245500=18\.49\\begin\{split\}\\text\{MRS\}&=\\frac\{25\{,\}000\-15\{,\}755\}\{500\}\\\\ &=\\frac\{9\{,\}245\}\{500\}=18\.49\\end\{split\}\(3\)
Efficiency Gain=1−15,755\+50025,000=1−0\.65=0\.35\\begin\{split\}\\text\{Efficiency Gain\}&=1\-\\frac\{15\{,\}755\+500\}\{25\{,\}000\}\\\\ &=1\-0\.65=0\.35\\end\{split\}\(4\)

## Appendix EJabberwocky Dataset Construction

To evaluate the structural robustness of the models, we constructed a Jabberwocky \(JW\) variant of the C4 dataset\. This process aims to eliminate semantic correlations from lexical co\-occurrence while strictly preserving the syntactic structure and morphological consistency of the original text\.

#### Implementation Details

We implemented the generation pipeline using thespaCylibrary with theen\_core\_web\_smmodel\. The transformation process operates as follows:

- •Fine\-grained POS Tagging:We first tokenize the input text and assign fine\-grained Part\-of\-Speech \(POS\) tags\. Unlike coarse tags \(e\.g\.,NOUN\), fine\-grained tags \(e\.g\.,NNSfor plural nouns,VBDfor past tense verbs\) allow us to distinguish morphological forms strictly\.
- •Content Word Identification:We identify content words defined as tokens belonging to the set of coarse categories:\{NOUN, VERB, ADJ, ADV\}\. Function words \(e\.g\., determiners, prepositions\) and punctuation are preserved to maintain the grammatical structures\.
- •Tag\-wise Shuffling \(Document Level\):To preserve morphological agreement \(e\.g\., subject\-verb number agreement\), we strictly shuffle words within the same fine\-grained tag category\. Specifically, we group all content words within a processing batch by their fine\-grained tags and shuffle these buckets randomly\.
- •Lexical Replacement with Casing Constraints:Each content word in the original sequence is replaced by a different word drawn from the corresponding shuffled tag bucket\. Crucially, we apply casing constraints: if the original token was capitalized \(e\.g\., sentence initial\), the replaced token is capitalized to maintain sentence boundaries\.

By using fine\-grained tags rather than coarse categories, our method ensures that, for instance, a singular noun is always replaced by another singular noun, and a past\-tense verb by another past\-tense verb\. This guarantees that the resulting sequences preserve syntactic well\-formedness despite the removal of semantic information\.

## Appendix FImpossible Language Datasets Construction

To evaluate whether the model’s inductive bias aligns with human\-like linguistic constraints, we constructed three “Impossible Language” datasets\. We adapted the perturbation logic from the official implementation ofKalliniet al\.\([2024](https://arxiv.org/html/2605.16758#bib.bib20)\)555[https://github\.com/jkallini/mission\-impossible\-language\-models/](https://github.com/jkallini/mission-impossible-language-models/)and integrated it into our preprocessing pipeline\.

While the Jabberwocky dataset operates at the document level to maintain vocabulary pools, the following transformations were applied at thesentence levelafter segmentation usingspaCy\. We generated the datasets using the following configurations:

- •SHUFFLE: Generated with the argumentshuffle\_deterministic21\. This transformation permutes tokens deterministically within a fixed local window of sizes=21s=21\. By destroying local word order \(n\-grams\) while preserving global bag\-of\-words statistics, this condition tests the model’s reliance on local syntactic constituency\.
- •REVERSE: Generated with the argumentreverse\_full\. This operation reverses the token order of the entire sentence string \(w1,w2,…,wn→wn,…,w2,w1w\_\{1\},w\_\{2\},\\dots,w\_\{n\}\\to w\_\{n\},\\dots,w\_\{2\},w\_\{1\}\)\. While computationally deterministic \(requiring a stack\), this transformation violates the incremental, left\-to\-right processing constraint fundamental to human language performance\.
- •HOP: Generated with the argumenthop\_words4\. This transformation introduces a dependency based on linear counting rather than structural configuration\. Specifically, a functional marker is placed at a fixed linear distance ofk=4k=4words after its associated verb\. This mimics “impossible” grammatical rules that rely on counting word positions in the linear string, violating the structure\-dependence principle of Universal Grammar\.

All transformations were applied to the same subset of the C4 training data as the other conditions, ensuring comparable data volume and lexical coverage\.

## Appendix GComplexity Calibration of Abstract Conditions

In §[6](https://arxiv.org/html/2605.16758#S6), we parameterize our abstract formal languages by two values:kstructk\_\{\\text\{struct\}\}, the number of bracket types used for hierarchical structure, andkdepk\_\{\\text\{dep\}\}, the number of distinct dependency types\. We setkstruct=1k\_\{\\text\{struct\}\}=1\(corresponding to 1\-Dyck\) andkdep=4k\_\{\\text\{dep\}\}=4\(corresponding to 4\-Shuffle Dyck\)\. This choice is not arbitrary but is derived from an analysis of the dependency types inherent in the fullMP\-Structgenerator\.MP\-Structgenerates sequences based on five distinct structural operations:

1. 1\.Structure \(Merge\):The fundamental recursive skeleton formed by brackets \(e\.g\.,\[…\]\[\\dots\]\)\. This corresponds to the1\-Dyckcomponent\.
2. 2\.Dependencies \(Move/Agree/Select\):Within this skeleton, four distinct types of long\-distance dependencies are established\. These correspond to the4\-Shuffle Dyckcomponent \(k=4k=4\): - •Type 1: Agreement \(Plural\)\.The dependency betweenTp​lT\_\{pl\}andD​Pp​lDP\_\{pl\}\. - •Type 2: Agreement \(Singular\)\.The dependency betweenTs​gT\_\{sg\}andD​Ps​gDP\_\{sg\}\. - •Type 3: Movement\.The dependency between a functional head \(e\.g\.,CC\) and a trace \(T​RTR\) formed by Wh\-movement\. - •Type 4: Selection\.The local dependency where a determiner \(DD\) selects a noun \(NN\)\.

By settingkstruct=1k\_\{\\text\{struct\}\}=1andkdep=4k\_\{\\text\{dep\}\}=4, we ensure that bothGenerickk\-SDandMP\-Struct Corepossess the same "vocabulary size" of dependency types as the original model, allowing us to isolate the effect of topological arrangement and landmarks without confounding factors related to task complexity\.

## Appendix HMP\-Struct Core: Design Details

The design ofMP\-Struct Coreand its motivation are described in §[6\.1](https://arxiv.org/html/2605.16758#S6.SS1)\. Here we provide supplementary detail on the mapping fromMP\-Structto its abstract counterpart\.

#### Fixed Topology

The derivation inMP\-Structfollows a recursive path \(C​P→T​P→v​PCP\\to TP\\to vP\)\.MP\-Struct Corestrictly enforces this same hierarchical subordination, ensuring that every dependency is generated within its designated structural domain\.

#### Abstract Landmarks

The functional heads ofMP\-Structare replaced by distinct abstract tokens with the following correspondence:

- •Complementizer \(CC\)→HC​P\\to H\_\{CP\}:Marks the clause boundary and movement landing site\.
- •Tense \(TT\)→HT​P\\to H\_\{TP\}:Marks the inflectional domain and agreement trigger\.
- •Verb \(vv\)→HV​P\\to H\_\{VP\}:Marks the thematic domain \(argument structure\)\.

Argument structure is fixed to a transitive frame, ensuring thatHC​PH\_\{CP\},HT​PH\_\{TP\}, andHV​PH\_\{VP\}serve as unambiguous, consistent landmarks across all generated sequences\.