Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling

# Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling
Source: [https://arxiv.org/html/2609.09691](https://arxiv.org/html/2609.09691)
Hongtao MuTianyu ZhouHansen LiuTao JiAffiliation:College of Foreign Languages and Literature, Fudan UniversityAffiliation:[tsfan24@m\.fudan\.edu\.cn](mailto:[email protected])[taoji@fudan\.edu\.cn](mailto:[email protected])

###### Abstract

When training data are limited, increasing parameter count is not the only way to improve language\-model performance\. A small parameter set, when repeatedly applied, can also deliver comparable performance\. We study Looped GPT\-BERT in the BabyLM 2026 Strict\-small setting, combining GPT\-BERT’s masked next\-token and causal language\-modeling objectives with depth\-wise parameter sharing\. We train on a preprocessed 7\.48M\-word English corpus and compare objective ratios, non\-looped and looped architectures, and loop counts\. Our final4×124\\times 12model uses four physical layers for twelve recurrent traversals and contains 12\.18M parameters\. The BabyLM 2026 leaderboard reports an Overall Average of 35\.42 and an NLP Average of 48\.48\. Compared with public BabyLM 10M Strict\-small GPT\-2 and GPT\-BERT baselines, it achieves comparable performance on selected linguistic and downstream metrics, including BLiMP and GLUE, with fewer parameters\. The loop ablations show that additional recurrent computation can improve training and preserve strong performance on selected linguistic tasks, whereas poorer performance on other tasks may reveal an inherent limitation of the looped design: using only a few physical layers restricts the model’s representational space\.

\\faIcon\{envelope\}\\faIcon\{envelope\}footnotetext:Corresponding author\.## 1Introduction

Language\-model performance has commonly improved through jointly scaling model size, data, and training computation\. Scaling\-law studies describe predictable power\-law relations between loss, parameter count, data, and compute\([Kaplan et al\., 2020](https://arxiv.org/html/2609.09691#bib.bib2)\), while compute\-optimal analyses emphasize that parameter and token budgets should be allocated jointly\([Hoffmann et al\., 2022](https://arxiv.org/html/2609.09691#bib.bib3)\)\. These results largely concern much larger autoregressive models, where model capacity and per\-example computation grow together\. Under constrained data and parameter budgets, however, they need not: repeatedly applying the same parameters can create a deeper computational path without proportionally increasing the number of learned weights\. We therefore ask whether recurrent computation induced by shared parameters can compensate for reduced physical depth and parameter capacity in low\-resource language\-model pretraining\.

The BabyLM Challenge provides a controlled setting for this question by restricting pretraining text to a developmentally plausible scale and evaluating linguistic, downstream, and human\-like behavior\([Warstadt et al\., 2023](https://arxiv.org/html/2609.09691#bib.bib4)\)\. We work in the BabyLM 2026 Strict\-small track on a cleaned 7\.48M\-word English corpus\. Our starting point is GPT\-BERT\([Charpentier and Samuel, 2024](https://arxiv.org/html/2609.09691#bib.bib6)\), which trains masked next\-token prediction \(MNTP\) and causal language modeling \(CLM\) in one Transformer parameter stack\. It consequently supports both bidirectional masked prediction and autoregressive scoring or generation\. This hybrid objective gives a further dimension to study: when the backbone is looped, does the relative amount of masked and causal supervision change the capabilities the architecture acquires?

We introduce depth\-wise recurrent parameter sharing on top of GPT\-BERT\. Universal Transformers repeatedly share transformations along depth\([Dehghani et al\., 2019](https://arxiv.org/html/2609.09691#bib.bib7)\), and ALBERT shows that cross\-layer sharing can substantially reduce encoder parameters\([Lan et al\., 2020](https://arxiv.org/html/2609.09691#bib.bib8)\)\. Most directly, Looped Transformers inject a fixed input and the prior loop state into a shared backbone, increasing effective computational depth through iteration\([Yang et al\., 2024](https://arxiv.org/html/2609.09691#bib.bib9)\)\. Their strongest evidence, however, comes from in\-context algorithm\-learning tasks such as linear functions, decision trees, and neural\-network regression\. Natural\-language pretraining must simultaneously acquire lexical knowledge, syntactic regularities, world knowledge, and discourse state; whether those computations can be supported by repeated shared layers remains an open empirical question\.

We address three questions: how the BERT:GPT training ratio changes the training objective and downstream abilities; whether effective depth formed by repeatedly applying a small number of physical layers can retain language ability with substantially fewer parameters; and whether training and downstream tasks continue to benefit as the number of loops increases\. We summarize our contributions as follows:

- •We introduce Looped GPT\-BERT, combining GPT\-BERT’s hybrid MNTP/CLM objective with depth\-wise recurrent parameter sharing\.
- •We study how the BERT:GPT objective ratio affects training and downstream behavior in the looped architecture\.
- •We compare a 12\-layer non\-looped model with four physical layers applied 1/3/6/12 times, showing how recurrent computation closes the gap from fewer learned layers and where additional loops yield diminishing returns\.
- •We evaluate the final model in the BabyLM Challenge, where its 12\.18M parameters achieve comparable performance to public Strict\-small baselines on linguistic and downstream tasks\.

Table 1:Representative corpus\-cleaning operations\. Examples are shortened for presentation\.
## 2Related Work

#### Data\-Efficient LM Pretraining

BabyLM shifts attention from performance under web\-scale training to learning efficiency under a controlled data budget\. The first challenge showed that architecture, pretraining objective, preprocessing, and curriculum can all substantially affect small\-model performance, with no single approach dominating every downstream task\([Warstadt et al\., 2023](https://arxiv.org/html/2609.09691#bib.bib4);[Hu et al\., 2024](https://arxiv.org/html/2609.09691#bib.bib17)\)\. Subsequent work further found that the effect of data composition depends on model scale: a genre mixture suitable for one model size need not be optimal for another\([Yam and Paek, 2024](https://arxiv.org/html/2609.09691#bib.bib5)\)\. Other BabyLM studies explore variation sets, child\-inspired data and vocabulary choices, explicit linguistic information, and self\-distillation as complementary routes to sample\-efficient pretraining\([Haga et al\., 2024](https://arxiv.org/html/2609.09691#bib.bib18);[Ghanizadeh and Dousti, 2024](https://arxiv.org/html/2609.09691#bib.bib20);[Edman et al\., 2024](https://arxiv.org/html/2609.09691#bib.bib21);[Nair et al\., 2024](https://arxiv.org/html/2609.09691#bib.bib19)\)\. These approaches primarily improve sample efficiency through data selection, ordering, or objectives\. Complementarily, we study parameter efficiency: keeping the training corpus and model width fixed, we replace part of the independently parameterized depth with repeated computation through shared physical layers\.

#### Hybrid Causal and Masked Modeling

CLM predicts the next token from left context, whereas conventional MLM predicts selected tokens at their own positions from bidirectional context\. The two objectives therefore assign different meanings to the same output position in a decoder\-style model\. GPT\-BERT resolves this mismatch with MNTP\([Charpentier and Samuel, 2024](https://arxiv.org/html/2609.09691#bib.bib6)\): when tokenxk\+1x\_\{k\+1\}is masked, supervision is shifted left so that the hidden state at positionkkpredicts the originalxk\+1x\_\{k\+1\}\. Consequently, causal and masked examples share a single next\-token vocabulary projection and loss interface\. GPT\-mode rows retain their original tokens and use a lower\-triangular attention mask; MNTP\-mode rows corrupt selected input tokens, permit bidirectional attention, and compute loss only at the corresponding shifted next\-token positions\. Each training row is assigned one mode, rather than receiving both losses\. AntLM similarly combines causal and masked language\-modeling objectives in a BabyLM setting, alternating between them during training\([Yu et al\., 2024](https://arxiv.org/html/2609.09691#bib.bib16)\)\.

#### Recurrent Depth and Parameter Sharing

Universal Transformers update representations recurrently along depth while retaining parallel computation over sequence positions\([Dehghani et al\., 2019](https://arxiv.org/html/2609.09691#bib.bib7)\); ALBERT uses cross\-layer parameter sharing primarily for parameter efficiency and shows that the sharing strategy affects downstream behavior\([Lan et al\., 2020](https://arxiv.org/html/2609.09691#bib.bib8)\)\. Looped Transformers use the recurrencezt\+1=f⁡\(x\+zt\)z\_\{t\+1\}=f\(x\+z\_\{t\}\), wherexxis a fixed input embedding,ztz\_\{t\}is the loop state, andffis a shared Transformer backbone\([Yang et al\., 2024](https://arxiv.org/html/2609.09691#bib.bib9)\)\. This makes effective computational depth grow with loop count while the parameter count is governed mainly by the number of physical layers\.[Kohli et al\. \(2026\)](https://arxiv.org/html/2609.09691#bib.bib10)provide a more direct natural\-language reference: recurrent\-depth Transformers can improve systematic generalization and depth extrapolation in implicit multi\-hop reasoning, but excessive loops can also reduce prediction quality through overthinking\. Neither line of work establishes that shared loops are uniformly suitable for lexical, syntactic, world\-knowledge, and discourse\-state learning\. We therefore compare equal\-application and equal\-parameter configurations across linguistic and state\-tracking tasks\.

## 3Method

### 3\.1Corpus Preparation and Tokenization

We construct our training corpus by selecting and cleaning six source corpora: BNC Spoken, CHILDES, Project Gutenberg, OpenSubtitles, Simple English Wikipedia, and Switchboard\. The resultingnew\_data1corpus is produced by our rule\-based normalization and filtering pipeline, which removes redundancy, short or low\-quality content, case and structural noise, and normalizes dialogue formatting\. The preprocessing code and source files are available in our[preprocessing repository](https://github.com/JT-Ushio/babylm26-nlp-spring)\.Table[1](https://arxiv.org/html/2609.09691#S1.T1)summarizes the operations and representative input–output examples\. After cleaning, the six selected corpora yield 7,482,189 whitespace\-delimited words\. We use this complete post\-cleaning corpus directly for pretraining rather than adding or padding data to reach the 10M\-word upper bound\. Their source\-wise counts are reported inTable[2](https://arxiv.org/html/2609.09691#S3.T2)\.

We use two byte\-pair encoding \(BPE\) tokenizers\. The official 16k GPT\-BERT tokenizer is used only for early non\-looped ratio experiments, while the 8k tokenizer trained on thenew\_data1corpus is used for the main and loop experiments\. BPE provides a subword representation that can handle words beyond a fixed vocabulary\([Sennrich et al\., 2016](https://arxiv.org/html/2609.09691#bib.bib1)\)\. We choose the 8k vocabulary as a parameter\-allocation trade\-off for this small\-data setting: with hidden size 384, its tied embedding table contains8,192×384≈3\.158\{,\}192\\times 384\\approx 3\.15M parameters\. In our corpus, reducing the vocabulary from 16k to 8k increases the measured fertility from 1\.4381 to 1\.4787 tokens per whitespace\-delimited word, a 2\.82% increase\. This indicates a modest change in word\-level context coverage for the fixed 128\-token input, while avoiding a larger embedding allocation\.

Table 2:Word counts after preprocessing\.
### 3\.2Hybrid GPT\-BERT Objective

Letx=\(x1,…,xT\)x=\(x\_\{1\},\\ldots,x\_\{T\}\)be a token sequence andθ\\thetathe shared model parameters\. For GPT\-mode rows, a causal attention mask is used and every non\-padding next token is supervised:

ℒGPT=−∑t=1T−1logpθ\(xt\+1∣x≤t\)\.\\mathcal\{L\}\_\{\\mathrm\{GPT\}\}=\-\\sum\_\{t=1\}^\{T\-1\}\\log p\_\{\\theta\}\(x\_\{t\+1\}\\mid x\_\{\\leq t\}\)\.\(1\)
For BERT\-mode rows, predictable positions are sampled with a masking probability that decreases linearly from 0\.3 to 0\.15\. Of the selected tokens, 80% are replaced by the mask token, 10% by a random token, and 10% are left unchanged\. Bidirectional attention is permitted\. The target is shifted to the preceding hidden\-state position so that it remains aligned with the decoder\-style next\-token head\. Ifx~\\tilde\{x\}denotes the corrupted input, the objective is

ℒBERT=−∑t∈Mlogpθ\(xt∣x~∖t\)\.\\mathcal\{L\}\_\{\\mathrm\{BERT\}\}=\-\\sum\_\{t\\in M\}\\log p\_\{\\theta\}\(x\_\{t\}\\mid\\tilde\{x\}\_\{\\setminus t\}\)\.\(2\)Only shifted positions corresponding to selected masks contribute to this loss; all other labels are set to the ignore index\. Thus, unmasked BERT positions never enter the loss\. A batch\-level ratio ofrB:rGr\_\{\\mathrm\{B\}\}:r\_\{\\mathrm\{G\}\}assigns complete rows to the two modes, and the optimized objective averages valid supervised tokens with azz\-loss regularizer weighted by10−410^\{\-4\}\. Our final setting uses1:31\{:\}3, so one quarter of rows use MNTP and three quarters use CLM\.

### 3\.3Looped Transformer Backbone

The non\-looped baseline contains twelve independent Transformer layers\. The looped backbone instead contains four independently parameterized physical layers, each with attention and feed\-forward sublayers\. Each loop adds the static token representationxxto the preceding loop stateztz\_\{t\}and applies the same four\-layer stackFθF\_\{\\theta\}:

z0\\displaystyle z\_\{0\}=0,\\displaystyle=0,\(3\)zt\+1\\displaystyle z\_\{t\+1\}=Fθ\(x\+zt\),t=0,…,L−1\.\\displaystyle=F\_\{\\theta\}\(x\+z\_\{t\}\),\\quad t=0,\\ldots,L\-1\.Thus,zt\+1z\_\{t\+1\}is the final representation produced by the current traversal and is simply carried into the next pass; it is not an additional network operation\. Within physical layerii, attention and feed\-forward transformations are applied sequentially:

ut\(i\)\\displaystyle u\_\{t\}^\{\(i\)\}=DWAi,A⁡\(ht\(i\)\+Attni⁡\(ht\(i\)\)\),\\displaystyle=\\operatorname\{DWA\}\_\{i,A\}\\\!\\left\(h\_\{t\}^\{\(i\)\}\+\\operatorname\{Attn\}\_\{i\}\(h\_\{t\}^\{\(i\)\}\)\\right\),\(4\)ht\(i\+1\)\\displaystyle h\_\{t\}^\{\(i\+1\)\}=DWAi,F⁡\(ut\(i\)\+FFNi⁡\(ut\(i\)\)\)\.\\displaystyle=\\operatorname\{DWA\}\_\{i,F\}\\\!\\left\(u\_\{t\}^\{\(i\)\}\+\\operatorname\{FFN\}\_\{i\}\(u\_\{t\}^\{\(i\)\}\)\\right\)\.GPT\-BERT’s dynamic weighted accumulation \(DWA\) is a learnable short\-range residual mixer, not an additional recurrent state\. We restrict DWA to representation fusion within each physical layer and reinitialize it on every loop; information across loops is carried exclusively byztz\_\{t\}\. A4×L4\\times Lmodel therefore has four sets of Transformer parameters but performs4​L4Llayer applications\. The4×34\\times 3,4×64\\times 6, and4×124\\times 12variants perform 12, 24, and 48 layer applications, respectively, while keeping the same parameter count\. This “effective depth” counts layer applications only: repeatedly applying shared parameters is not equivalent to a non\-shared Transformer with the same number of layers\.

### 3\.4Training Configuration

Table[3](https://arxiv.org/html/2609.09691#S3.T3)lists the main configuration\. Beyond the structural hyperparameters shown there, optimization uses LAMB\([You et al\., 2020](https://arxiv.org/html/2609.09691#bib.bib15)\), a peak learning rate of 0\.0141, a minimum learning rate of 0\.00141, and cosine decay for 2,600 steps followed by a constant minimum rate\. We select the peak learning rate and decay horizon through the controlled studies in Section[5](https://arxiv.org/html/2609.09691#S5)\. All pretraining and fine\-tuning runs use random seed 42\.

Table 3:Main pretraining configuration\.

## 4Experiments

### 4\.1Evaluation Protocol

We follow the BabyLM 2026 Strict\-small evaluation pipeline\. The final checkpoint receives the full evaluation: the zero\-shot suite includes BLiMP and BLiMP Supplement minimal\-pair syntax judgments\([Warstadt et al\., 2020](https://arxiv.org/html/2609.09691#bib.bib11)\), EWoK world knowledge\([Ivanova et al\., 2024](https://arxiv.org/html/2609.09691#bib.bib12)\), Entity Tracking, COMPS conceptual properties, GlobalPIQA commonsense reasoning, Reading, and Age of Acquisition \(AoA\)\. Fine\-tuning covers BoolQ, MNLI, MRPC, MultiRC, QQP, RTE, and WSC, the GLUE/SuperGLUE\-style natural\-language\-understanding subset used by the challenge\([Wang et al\., 2019](https://arxiv.org/html/2609.09691#bib.bib14)\)\. We report the leaderboard aggregates, where NLP Average pools NLP tasks and Human\-like Average includes Reading, AoA, and related measures\. We use the official causal backend for all reported GPT\-BERT results\.

Beyond the final*main*revision, we retain intermediate checkpoints at one\-million\-word intervals from*chck\_1M*through*chck\_10M*, and at ten\-million\-word intervals from*chck\_20M*through*chck\_70M*\. We run the official fast evaluation on each of these checkpoints\. The ten\-epoch training run contains approximately 74\.8M word presentations, so*chck\_70M*is the last complete ten\-million\-word milestone\. The final model and all intermediate revisions are released in one public Hugging Face repository\.

### 4\.2BabyLM 2026 Leaderboard Results

Table 4:BabyLM 2026 Strict\-small leaderboard comparison on aggregate and selected task\-level metrics\. “Context” denotes the maximum training sequence length, “Sup\.” denotes BLiMP Supplement, “Entity” denotes Entity Tracking, “GPIQA” denotes GlobalPIQA, and “GLUE” denotes the official \(Super\)GLUE aggregate\. Dashes indicate results not reported for the corresponding public entry\. Aggregate scores for entries with unreported tasks are not directly comparable\.Table[4](https://arxiv.org/html/2609.09691#S4.T4)reports the final4×124\\times 12Looped GPT\-BERT alongside two publicly released BabyLM Strict\-small references: the official GPT\-2 baseline and the GPT\-BERT causal\-focus baseline\. The leaderboard snapshot was accessed on July 21, 2026\.111[https://huggingface\.co/spaces/BabyLM\-community/BabyLM\-Leaderboard\-2026](https://huggingface.co/spaces/BabyLM-community/BabyLM-Leaderboard-2026)

With 12\.18M parameters, the final model is substantially smaller than both public references\. It reaches 71\.19 on BLiMP, compared with 65\.23 for GPT\-2 and 71\.66 for GPT\-BERT causal\-focus, while its GLUE aggregate is 62\.55, compared with 63\.80 and 65\.13\. This pattern illustrates the trade\-off studied here: at the cost of additional computation, repeated application of shared layers can retain strong performance on selected tasks with a smaller independently parameterized model, with gains varying across tasks\.

### 4\.3Effect of the BERT:GPT Ratio

We examine the BERT:GPT ratio under two tokenizers\. The official 16k tokenizer is used for early non\-looped ratio exploration, whereas the 8k tokenizer trained on thenew\_data1corpus is used for all main and loop experiments\. The15:115\{:\}1configuration is BERT\-heavy,1:11\{:\}1is balanced, and1:31\{:\}3is GPT\-heavy\. The two settings test whether the ratio trend is consistent; they are not used to compare tokenizer quality\.Tables[5](https://arxiv.org/html/2609.09691#S4.T5)and[6](https://arxiv.org/html/2609.09691#S4.T6)report the principal 8k comparison, including training endpoints and available zero\-shot results\.

Table 5:Non\-looped training results across objective ratios with the custom 8k tokenizer\.Table 6:Non\-looped zero\-shot results across objective ratios with the custom 8k tokenizer\.For context,Table[7](https://arxiv.org/html/2609.09691#S4.T7)reports the corresponding early experiments with the official 16k tokenizer\. Increasing the proportion of GPT rows lowers the final mixed training loss in both tokenizer settings, with1:31\{:\}3giving the strongest endpoint\.

Table 7:Non\-looped training results across objective ratios with the official 16k tokenizer\.The1:31\{:\}3ratio improves BLiMP, BLiMP Supplement, and COMPS under the 8k tokenizer, but reduces Entity Tracking from 38\.66 to 22\.87\. This pattern suggests that objective mixing changes the distribution of learned capabilities rather than merely the aggregate loss\. More CLM supervision repeatedly trains left\-to\-right next\-token prediction and is therefore closely aligned with sequential generation, local dependencies, and causal syntactic scoring\. More MNTP supervision requires reconstructing a target from both sides of its context and may better support integrating multiple positions and relations, which is useful for tracking entities and their states\. We therefore use1:31\{:\}3as the main configuration for subsequent loop experiments targeting generative and syntactic ability\.

### 4\.4Effect of Looped Depth

We first examine training loss and token accuracy under the1:31\{:\}3objective\.Figure[1](https://arxiv.org/html/2609.09691#S4.F1)plots how these two measures change over training for the 12\-layer non\-looped model and for four\-layer models with11,33,66, or1212applications per forward pass\. The4×34\\times 3and 12\-layer non\-looped models both execute twelve layer applications per forward pass, while the looped model uses only four independently parameterized layers\.

Figure 1:Training loss and token accuracy for the 8k\-tokenizer,1:31\{:\}3objective across loop depths\. The looped models use four physical layers; the non\-looped model uses twelve independently parameterized layers\.The4×14\\times 1control is visibly less stable at the beginning of training: it maintains a higher loss and lower token accuracy than the models with repeated layer applications\. Increasing the loop count lowers the training loss and generally raises token accuracy, bringing the looped models closer to the 12\-layer non\-looped trajectory\. The largest improvement occurs when moving from4×14\\times 1to4×34\\times 3; the changes from4×34\\times 3to4×64\\times 6and from4×64\\times 6to4×124\\times 12are smaller\. Thus, repeated computation recovers much of the training behavior of the non\-looped model, while additional loops provide diminishing returns\.

Table 8:Zero\-shot comparison across objective ratios and loop counts\.We next turn to the zero\-shot task results inTable[8](https://arxiv.org/html/2609.09691#S4.T8)\. The table compares the 12\-layer non\-looped model with four physical layers applied11,33,66, or1212times, using BERT:GPT ratios of1:31\{:\}3in the upper block and1:11\{:\}1in the lower block\. Under1:31\{:\}3, increasing the loop count from4×14\\times 1to4×34\\times 3substantially improves BLiMP, from 65\.08 to 71\.21, slightly exceeding the non\-looped score of 70\.54\. Increasing the depth beyond4×34\\times 3does not produce a consistent additional gain: BLiMP is 70\.49 for4×64\\times 6and 71\.19 for4×124\\times 12\. Under1:11\{:\}1,4×64\\times 6gives the highest BLiMP score \(69\.38\), compared with 63\.77 for the non\-looped model and 62\.28–64\.45 for the other looped settings\. Overall, the looped models preserve or improve several syntactic and reading\-related scores despite using fewer independently parameterized layers, while the effect of recurrent depth varies across tasks\.

Unlike the syntactic measures above, Entity Tracking is more sensitive to parameter sharing\. Overall, the looped variants are weaker than the corresponding non\-looped model on this task, with one exception: under1:31\{:\}3,4×64\\times 6reaches 28\.66, compared with 22\.87 for the non\-looped model\. Even in this setting, the score falls to 15\.78 with4×124\\times 12, so the additional computation from six to twelve applications does not preserve the4×64\\times 6improvement\. Under1:11\{:\}1, all looped variants remain below the non\-looped score of 38\.66, with scores between 13\.10 and 17\.52\. One possible explanation is that entity tracking requires distinguishing and updating multiple entities, their states, and state changes\([Kim and Schuster, 2023](https://arxiv.org/html/2609.09691#bib.bib13)\)\. Reusing four attention/FFN parameter sets rather than twelve independent sets may leave less representational space for the layer\-specific transformations needed to maintain multiple entities and relations\. This contrast suggests that recurrent depth can compensate for reduced parameterization on some linguistic tasks, but cannot uniformly replace the representational flexibility of independently parameterized layers\. Considering both downstream performance and inference cost,4×64\\times 6is a practical intermediate configuration, but the table also shows that the preferred loop depth depends on the objective ratio and task\.

### 4\.5Inference Cost

We measure inference cost for the 12\-layer non\-looped model and the4×34\\times 3,4×64\\times 6, and4×124\\times 12looped models with the same batch size of 32, sequence length of 128, 20 warmup steps, and 100 timed forward passes on one RTX 3090\. As shown inTable[9](https://arxiv.org/html/2609.09691#S4.T9),4×34\\times 3is better than the 12\-layer non\-looped model on every measured systems metric: it has lower latency, higher token throughput, and lower peak memory use\. Increasing the loop count from4×34\\times 3to4×64\\times 6and4×124\\times 12roughly doubles the latency at each step, while throughput decreases and memory use increases\. Considering both downstream performance and inference speed,4×64\\times 6provides a practical compromise: it is slower than4×34\\times 3but performs better on some downstream tasks, while remaining substantially faster than4×124\\times 12\.

Table 9:Inference speed measured with batch size 32, sequence length 128, 20 warmup steps, and 100 timed forward passes on one RTX 3090\.

## 5Optimization Ablations

### 5\.1Selecting the Peak Learning Rate

We first select a peak learning rate that lowers the training objective quickly without producing clear instability\. Before full training, we run four\-epoch non\-looped sweeps at 0\.0075, 0\.01, 0\.0141, and 0\.02 with all other settings fixed\. AsTable[10](https://arxiv.org/html/2609.09691#S5.T10)shows, the first three settings decrease stably, whereas 0\.02 degrades after approximately step 600 and produces gradient\-norm spikes\. Although 0\.01 has the highest final token accuracy, 0\.0141 achieves the lowest final loss \(3\.6847\) and is selected as the peak learning rate for the main experiments\. This sweep provides a preliminary learning\-rate choice based on the non\-looped model and does not fully optimize the rate for looped models\. We examine the effect of decay speed on training stability separately below\.

Table 10:Four\-epoch peak learning\-rate sweep\.
### 5\.2Learning\-Rate Decay and Stability

We select two candidate decay lengths from the training budget: 4,080 steps spans the full optimizer\-step budget of the run, whereas 2,600 steps reaches the minimum earlier and remains there for the rest of training\. With total training steps fixed,*lr\_schedule\_steps*determines how quickly cosine decay reaches its minimum: a smaller value decays earlier and remains at the minimum longer, while a larger value retains a higher learning rate later in training\. We compare these decay rates in4×64\\times 6and4×124\\times 12models\. The main 2,600\-step schedule reaches 10% of the peak learning rate and then holds 0\.00141; the slower 4,080\-step schedule maintains a higher learning rate into the later stages\. Because shared physical layers are repeatedly applied across loops, the effect of one parameter update can be propagated through multiple layer applications\. A high early learning rate may therefore amplify update oscillations, while earlier decay may help shared parameters enter a useful region more smoothly\.

Figure 2:Training loss and token accuracy for the learning\-rate decay ablation\. Solid and dashed lines denote 2,600\-step and 4,080\-step decay, respectively; colors distinguish4×64\\times 6and4×124\\times 12\.Figure[2](https://arxiv.org/html/2609.09691#S5.F2)shows that the 2,600\-step schedule has its clearest advantage in early and middle training: loss declines faster and token accuracy rises sooner\. Earlier decay, however, can reduce the opportunity to explore alternative parameter regions and leave the model near a suboptimal solution\. The 4,080\-step schedule gradually catches up later, consistent with its higher late\-stage learning rate preserving additional exploration\. Nevertheless, the 2,600\-step curves are smoother overall and show no clear final\-metric disadvantage\. We therefore choose 2,600 steps as a practical compromise between stability and late\-stage exploration under the current compute budget and single\-seed setting, not as a schedule that is uniformly superior at every point in training\.

### 5\.3Qualitative Generation Examples

The causal interface also supports open\-ended completion\.Table[11](https://arxiv.org/html/2609.09691#S5.T11)presents selected completions from the final1:31\{:\}3,4×124\\times 12model using temperature 0\.9, top\-kk40, top\-pp0\.95, and seed 42\. The examples illustrate locally coherent continuation and appropriate termination in short responses; they are qualitative examples rather than a controlled generation evaluation\.

Table 11:Selected sampled completions from the final1:31\{:\}3,4×124\\times 12model\.

## 6Conclusion

We combine GPT\-BERT’s hybrid CLM/MNTP training with depth\-wise recurrent parameter sharing to study a “fewer parameters, more computation” language\-model design under BabyLM Strict\-small constraints\. Reusing four physical layers for twelve loops yields a 12\.18M\-parameter model whose training objective is close to that of a 29\.9M non\-looped model\. The final model reaches 71\.19 on BLiMP and 62\.55 on GLUE, compared with 71\.66 and 65\.13 for the official GPT\-BERT causal\-focus reference, despite its smaller parameter budget\. On the reported task\-level metrics, looped models preserve or improve selected syntactic abilities, while Entity Tracking becomes weaker as a result of parameter sharing\. The ablations further show that gains from additional loops saturate quickly and differ across tasks\. Looped GPT\-BERT is therefore an architecture with clear advantages and limitations: under constrained data and parameter budgets, it can support syntactic understanding and generation, but its objective ratio and recurrent depth require further adjustment\.

## Limitations

- •Data and tokenizer\.The main experiments use one cleaned corpus and one 8k tokenizer\. We do not independently ablate raw data, rule\-based cleaning, alternative cleaning procedures, and tokenization\.
- •Training variance\.Each model is trained with a single random seed, so sub\-percentage\-point differences may fall within training variance\.
- •Compute matching\.The4×124\\times 12model performs substantially more layer applications and computation than the 12\-layer non\-looped baseline\. This is a parameter\-efficiency study, not a FLOP\-matched comparison\.
- •Objective mixing\.Changing theBert:Gptratio also changes supervision density and attention visibility; mixed training losses are therefore not perfectly homogeneous across ratios\.
- •Capability boundary\.The final model is weak on AoA and Entity Tracking, indicating limited cognitive\-similarity and state\-tracking ability\.
- •Scope\.Our conclusions are limited to approximately 12M parameters, a 7\.48M\-word corpus, ten epochs, and a maximum sequence length of 128; they should not be directly extrapolated to larger models or longer contexts\.

## Ethics Statement

The model uses English text supplied by, or derived from, the BabyLM Challenge and introduces no newly collected personal data\. The sources include books, subtitles, encyclopedic text, and dialogue, so they may retain social biases or inappropriate content from the original material; automated cleaning cannot guarantee their removal\. Cleaning may also remove dialectal, conversational, or minority\-language patterns and introduce stylistic preferences through normalization\. This small model is intended for research and should not be deployed directly in high\-stakes applications\.

## Reproducibility Statement

The Hugging Face repository contains the final model and sixteen intermediate checkpoints\. The*main*revision stores the final model;*chck\_1M*–*chck\_10M*are one\-million\-word checkpoints, and*chck\_20M*–*chck\_70M*are ten\-million\-word checkpoints\. It also includes*configuration\_gpt\_bert\.py*,*modeling\_gpt\_bert\.py*, and tokenizer files, allowing the model to be loaded with*trust\_remote\_code=True*\.222[https://huggingface\.co/Survivor613/BabyLM2026\-Strict\-Small\-looped\_GPT\-BERT](https://huggingface.co/Survivor613/BabyLM2026-Strict-Small-looped_GPT-BERT)Training configurations and scripts are available in the accompanying code repository,333[https://github\.com/Survivor613/BabyLM\-2026](https://github.com/Survivor613/BabyLM-2026)and the data\-cleaning workflow is documented separately\.444[https://github\.com/JT\-Ushio/babylm26\-nlp\-spring](https://github.com/JT-Ushio/babylm26-nlp-spring)

## Acknowledgments

The authors thank the reviewers for their helpful comments and suggestions, as well as the BabyLM organizers for maintaining the datasets, evaluation pipeline, and leaderboard\. This work was partially funded by the National Natural Science Foundation of China \(No\. 62506079\)\.

## References

- Charpentier and Samuel \(2024\)L\. G\. G\. Charpentier and D\. SamuelGPT or BERT: why not both?\.InThe 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning,Miami, FL, USA,pp\. 262–283\.External Links:[Link](https://aclanthology.org/2024.conll-babylm.24/)Cited by:[§1](https://arxiv.org/html/2609.09691#S1.p2.1),[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px2.p1.1)\.
- Dehghaniet al\.\(2019\)M\. Dehghani, S\. Gouws, O\. Vinyals, J\. Uszkoreit, and L\. KaiserUniversal transformers\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=HyzdRiR9Y7)Cited by:[§1](https://arxiv.org/html/2609.09691#S1.p3.1),[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px3.p1.1)\.
- Edmanet al\.\(2024\)L\. Edman, L\. Bylinina, F\. Ghorbanpour, and A\. FraserAre BabyLMs second language learners?\.InThe 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning,Miami, FL, USA,pp\. 166–173\.External Links:[Link](https://aclanthology.org/2024.conll-babylm.14/)Cited by:[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px1.p1.1)\.
- Ghanizadeh and Dousti \(2024\)M\. A\. Ghanizadeh and M\. J\. DoustiTowards data\-efficient language models: a child\-inspired approach to language learning\.InThe 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning,Miami, FL, USA,pp\. 22–27\.External Links:[Link](https://aclanthology.org/2024.conll-babylm.2/)Cited by:[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px1.p1.1)\.
- Hagaet al\.\(2024\)A\. Haga, A\. Fukatsu, M\. Oba, A\. Bisazza, and Y\. OsekiBabyLM challenge: exploring the effect of variation sets on language model training efficiency\.InThe 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning,Miami, FL, USA,pp\. 252–261\.External Links:[Link](https://aclanthology.org/2024.conll-babylm.23/)Cited by:[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px1.p1.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark,et al\.Training compute\-optimal large language models\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 30016–30030\.Cited by:[§1](https://arxiv.org/html/2609.09691#S1.p1.1)\.
- Huet al\.\(2024\)M\. Y\. Hu, A\. Mueller, C\. Ross, A\. Williams, T\. Linzen, C\. Zhuang, L\. Choshen, R\. Cotterell, A\. Warstadt, and E\. G\. WilcoxFindings of the second BabyLM challenge: sample\-efficient pretraining on developmentally plausible corpora\.InThe 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning,Miami, FL, USA,pp\. 1–21\.External Links:[Link](https://aclanthology.org/2024.conll-babylm.1/)Cited by:[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px1.p1.1)\.
- Ivanovaet al\.\(2024\)A\. A\. Ivanova, A\. Sathe, B\. Lipkin, U\. Kumar, S\. Radkani, T\. H\. Clark, C\. Kauf, J\. Hu, R\. T\. Pramod, G\. Grand,et al\.Elements of world knowledge \(EWoK\): a cognition\-inspired framework for evaluating basic world knowledge in language models\.arXiv preprint arXiv:2405\.09605\.External Links:[Link](https://arxiv.org/abs/2405.09605)Cited by:[§4\.1](https://arxiv.org/html/2609.09691#S4.SS1.p1.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. AmodeiScaling laws for neural language models\.arXiv preprint arXiv:2001\.08361\.External Links:[Link](https://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2609.09691#S1.p1.1)\.
- Kim and Schuster \(2023\)N\. Kim and S\. SchusterEntity tracking in language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Toronto, Canada,pp\. 3835–3855\.External Links:[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.213),[Link](https://aclanthology.org/2023.acl-long.213/)Cited by:[§4\.4](https://arxiv.org/html/2609.09691#S4.SS4.p4.1)\.
- Kohliet al\.\(2026\)H\. Kohli, S\. Parthasarathy, H\. Sun, and Y\. YaoLoop, think, & generalize: implicit reasoning in recurrent\-depth transformers\.arXiv preprint arXiv:2604\.07822\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2604.07822),[Link](https://arxiv.org/abs/2604.07822)Cited by:[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px3.p1.1)\.
- Lanet al\.\(2020\)Z\. Lan, M\. Chen, S\. Goodman, K\. Gimpel, P\. Sharma, and R\. SoricutALBERT: a lite BERT for self\-supervised learning of language representations\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=H1eA7AEtvS)Cited by:[§1](https://arxiv.org/html/2609.09691#S1.p3.1),[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px3.p1.1)\.
- Nairet al\.\(2024\)A\. Nair, A\. Hancharova, M\. Kumar, and A\. GharaeeBabyLM challenge: experimenting with self\-distillation and reverse\-distillation for language model pre\-training on constrained datasets\.InThe 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning,Miami, FL, USA,pp\. 28–36\.External Links:[Link](https://aclanthology.org/2024.conll-babylm.3/)Cited by:[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px1.p1.1)\.
- Sennrichet al\.\(2016\)R\. Sennrich, B\. Haddow, and A\. BirchNeural machine translation of rare words with subword units\.InProceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1715–1725\.External Links:[Document](https://dx.doi.org/10.18653/v1/P16-1162),[Link](https://aclanthology.org/P16-1162/)Cited by:[§3\.1](https://arxiv.org/html/2609.09691#S3.SS1.p2.1)\.
- Wanget al\.\(2019\)A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. R\. BowmanGLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rJ4km2R5t7)Cited by:[§4\.1](https://arxiv.org/html/2609.09691#S4.SS1.p1.1)\.
- Warstadtet al\.\(2023\)A\. Warstadt, A\. Mueller, L\. Choshen, E\. Wilcox, C\. Zhuang, J\. Ciro, R\. Mosquera, B\. Paranjape, A\. Williams, T\. Linzen, and R\. CotterellFindings of the BabyLM challenge: sample\-efficient pretraining on developmentally plausible corpora\.InProceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning,Singapore,pp\. 1–34\.External Links:[Link](https://aclanthology.org/2023.conll-babylm.1/)Cited by:[§1](https://arxiv.org/html/2609.09691#S1.p2.1),[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px1.p1.1)\.
- Warstadtet al\.\(2020\)A\. Warstadt, A\. Parrish, H\. Liu, A\. Mohananey, W\. Peng, S\. Wang, and S\. R\. BowmanBLiMP: the benchmark of linguistic minimal pairs for English\.Transactions of the Association for Computational Linguistics8,pp\. 377–392\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00321),[Link](https://aclanthology.org/2020.tacl-1.25/)Cited by:[§4\.1](https://arxiv.org/html/2609.09691#S4.SS1.p1.1)\.
- Yam and Paek \(2024\)H\. M\. Yam and N\. PaekWhat should baby models read? exploring sample\-efficient data composition on model performance\.InThe 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning,Miami, FL, USA,pp\. 284–291\.External Links:[Link](https://aclanthology.org/2024.conll-babylm.25/)Cited by:[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2024\)L\. Yang, K\. Lee, R\. Nowak, and D\. PapailiopoulosLooped transformers are better at learning learning algorithms\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=XpVoUnPuYV)Cited by:[§1](https://arxiv.org/html/2609.09691#S1.p3.1),[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px3.p1.1)\.
- Youet al\.\(2020\)Y\. You, J\. Li, J\. Hseu, X\. Song, J\. Demmel, and C\. HsiehLarge batch optimization for deep learning: training BERT in 76 minutes\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Syx4wnEtvH)Cited by:[§3\.4](https://arxiv.org/html/2609.09691#S3.SS4.p1.1)\.
- Yuet al\.\(2024\)X\. Yu, B\. Guo, S\. Luo, J\. Wang, T\. Ji, and Y\. WuAntLM: bridging causal and masked language models\.InThe 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning,Miami, FL, USA,pp\. 324–331\.External Links:[Link](https://aclanthology.org/2024.conll-babylm.29/)Cited by:[§2](https://arxiv.org/html/2609.09691#S2.SS0.SSS0.Px2.p1.1)\.