当最弱语言不断变化：面向多语言 BabyLM 的 ARMOR-DP

# When the Weakest Language Keeps Moving: ARMOR-DP for Multilingual BabyLM
Source: [https://arxiv.org/html/2610.33583](https://arxiv.org/html/2610.33583)
###### Abstract

Multilingual BabyLM systems usually fix language sampling proportions even though the weakest language can change during training\. We present ARMOR\-DP, a modular framework that diagnoses imbalance arising from representation, exposure, capacity, and objective pressure, and uses per\-language validation loss to update sampling\. Controlled experiments show that equal character budgets create a severe representation artifact for Chinese; byte\-level encoding removes much of it, while static Chinese protection can subsequently make English the weakest language\. The dynamic controller tracks these transitions and achieves a better score–balance tradeoff than fixed protection\. On official evaluation, curriculum\-dynamic checkpoints outperform a matched static Chinese\-protective baseline while using fewer Chinese epochs; scaling capacity yields the strongest internal ARMOR\-DP system\. These results support a simple conclusion: multilingual weakness is a training\-time state rather than a fixed property of a language\. ARMOR\-DP therefore separates diagnosis from intervention, allowing protection to follow the current bottleneck instead of a predetermined language label\.

\\faIcon\{handshake\}\\faIcon\{handshake\}footnotetext:Equal contribution\.\\faIcon\{envelope\}\\faIcon\{envelope\}footnotetext:Corresponding author\.## 1Introduction

The BabyLM challenge tasks a model with acquiring language from a developmentally plausible data budget, and the 2026 BabyBabelLM track extends this premise to the multilingual setting\([BabyLM Organizers, 2026b](https://arxiv.org/html/2610.33583#bib.bib1);[BabyLM Organizers, 2026a](https://arxiv.org/html/2610.33583#bib.bib2)\)\. Naturalistic linguistic input is not a fixed curriculum: properties of child\-directed speech change over development and can track children’s emerging linguistic abilities\([Vosoughi and Roy, 2012](https://arxiv.org/html/2610.33583#bib.bib3)\)\. We use this responsiveness as a motivating analogy, not as a claim that caregivers optimize explicit language proportions\.

![Refer to caption](https://arxiv.org/html/2610.33583v1/figures/weakness_moves_overview.png)

Figure 1:Central hypothesis of ARMOR\-DP\. Panel A shows that equal character budgets do not create equal representation burden across scripts; Panel B shows that the bottleneck can move after protection is added\. The takeaway is that weakest\-language protection should be measured during training rather than fixed to one language in advance\.In contrast, standard multilingual pretraining relies on static sampling temperatures or predetermined corpus proportions, which are agnostic to the model’s evolving proficiency\. Under BabyLM’s stringent capacity and data constraints, languages compete for a shared parameter budget, and a fixed ratio can underserve whichever language currently lags\. A responsive artificial learner should therefore reallocate exposure online\. Doing so requires first diagnosing why a language is weak at a particular moment—a question that cannot be answered by exposure alone\.

Our central hypothesis is that a language can beweakfor at least four different reasons\. It can be weak because its script is poorly represented by the tokenizer \(representation\), because it receives insufficient effectiveexposure, because the shared model has insufficient language\-privatecapacity, or because the objective allocates gradient pressure in a way that amplifies already frequent units \(objective\)\. Therefore, we proposeARMOR\-DP\(ArmorforDevelopmentalPressure\), a modular diagnostic and intervention framework\.

Our contributions are threefold\. First, we show that the identity of the weakest language moves as representation, exposure, and capacity interventions take effect\. Second, ARMOR\-DP separates four possible sources of weakness and turns per\-language validation loss into an online sampling signal\. Third, our official evaluations show that curriculum\-dynamic training improves on a matched static Chinese\-protective system while remaining within the challenge budget\.

## 2ARMOR\-DP Framework

ARMOR\-DP is a modular diagnostic and intervention toolkit\. It separates four sources of multilingual weakness, each paired with a corresponding probe or intervention\. Components can be enabled or omitted according to empirical evidence rather than treated as a mandatory stack\.

#### Representation armor\.

We use UTF\-8 byte units as a script\-fair representation\. It removes a key comparison artifact, in which Chinese characters are treated as sparse atomic symbols while Latin\-script languages reuse a small alphabet\. This choice is based on tokenization\-free and byte\-level models\([Xue et al\., 2022](https://arxiv.org/html/2610.33583#bib.bib14);[Clark et al\., 2022](https://arxiv.org/html/2610.33583#bib.bib7);[Tay et al\., 2022](https://arxiv.org/html/2610.33583#bib.bib13)\)\.

#### Exposure armor\.

We compare balanced sampling with ZH\-protective schedules\. Our primary static baseline uses English/Dutch/Chinese ratios of 25/25/50\. We also test a curriculum that starts balanced and switches to 25/25/50 halfway through training\. This is related to temperature\-based multilingual sampling and curriculum learning\([Conneau et al\., 2020](https://arxiv.org/html/2610.33583#bib.bib8);[Bengio et al\., 2009](https://arxiv.org/html/2610.33583#bib.bib6)\)\.

#### Capacity armor\.

We test two lightweight capacity mechanisms: language adapters and a language\-conditioned routing gate over attention heads\. We employ the routing gate as an interpretable capacity intervention for multilingual adaptation\. Rather than introducing a new routing mechanism, our goal is to analyze how capacity allocation interacts with language imbalance within the ARMOR\-DP framework\([Houlsby et al\., 2019](https://arxiv.org/html/2610.33583#bib.bib10);[Pfeiffer et al\., 2020](https://arxiv.org/html/2610.33583#bib.bib12);[Fedus et al\., 2022](https://arxiv.org/html/2610.33583#bib.bib9)\)\.

#### Objective armor\.

We test both a naive rare\-token weighted objective and a pressure\-aware objective that combines current excess loss, script pressure, and exposure\. The rare\-token diagnostic multiplies token\-level cross\-entropy by a weight derived only from that byte type’s frequency in the diagnostic corpus: less frequent types receive larger weights, clipped to the range reported in Appendix[10](https://arxiv.org/html/2610.33583#A2.T10)\. It is included as a simple frequency\-only analogue of loss reweighting\([Lin et al\., 2017](https://arxiv.org/html/2610.33583#bib.bib11)\), not as part of the final checkpoint\. In our diagnostics it over\-protects Chinese after exposure has already helped it, whereas pressure\-aware weighting reduces this failure\.

#### Dynamic protection controller\.

The dynamic controller is responsible for adapting the sampling distribution as multilingual learning progresses\. Instead of assuming that the weakest language remains fixed throughout training, it continuously estimates the current learning pressure of each language and reallocates training probability accordingly\. At each update steptt, the controller measures languageℓ\\ell’s relative weakness by its validation loss deviation from the multilingual mean\. Formally, the pressure score is defined as:

ρℓt=β​log⁡LℓtL¯t\+λs​log⁡sℓ−λe​pℓt,\\rho\_\{\\ell\}^\{t\}=\\beta\\log\\frac\{L\_\{\\ell\}^\{t\}\}\{\\bar\{L\}^\{t\}\}\+\\lambda\_\{s\}\\log s\_\{\\ell\}\-\\lambda\_\{e\}p\_\{\\ell\}^\{t\},The first term amplifies the language\-specific loss gap; we setβ=2\.0\\beta=2\.0to make a moderate 10% loss difference translate into a meaningful sampling shift, while avoiding overreaction to early\-training noise\. The second term,λs​log⁡sℓ\\lambda\_\{s\}\\log s\_\{\\ell\}withsℓs\_\{\\ell\}as bytes\-per\-character, is an optional script\-complexity prior—we disable it for all final checkpoints \(λs=0\\lambda\_\{s\}=0\) because our ablations show that this static bias over\-corrects Chinese in later training and widens the balance gap\. The third term,−λe​pℓt\-\\lambda\_\{e\}p\_\{\\ell\}^\{t\}withλe=0\.5\\lambda\_\{e\}=0\.5, acts as a negative feedback that gently penalizes languages already receiving high sampling probability, preventing the controller from chasing loss fluctuations too aggressively\.

We then convertρℓt\\rho\_\{\\ell\}^\{t\}into a target distributionqℓtq\_\{\\ell\}^\{t\}via softmax with temperatureτ\\tau, and smooth the update with momentumη\\eta:

qℓt=exp⁡\(τ​ρℓt\)∑kexp⁡\(τ​ρkt\),p~ℓt\+1=\(1−η\)​pℓt\+η​qℓt\.q\_\{\\ell\}^\{t\}=\\frac\{\\exp\(\\tau\\rho\_\{\\ell\}^\{t\}\)\}\{\\sum\_\{k\}\\exp\(\\tau\\rho\_\{k\}^\{t\}\)\},\\qquad\\tilde\{p\}\_\{\\ell\}^\{t\+1\}=\(1\-\\eta\)p\_\{\\ell\}^\{t\}\+\\eta q\_\{\\ell\}^\{t\}\.We setτ=2\.0\\tau=2\.0to produce moderate distributions \(avoiding hard one\-hot assignments\) andη=0\.5\\eta=0\.5to balance responsiveness to new signals against stability across updates\. Finally, we clipp~t\+1\\tilde\{p\}^\{t\+1\}to\[pmin,pmax\]=\[0\.18,0\.50\]\[p\_\{\\min\},p\_\{\\max\}\]=\[0\.18,0\.50\]and renormalize\. The lower bound ensures that no language drops below 18% sampling probability \(roughly half of uniform 33%\), preventing starvation; the upper bound caps any single language at 50%, enforcing a balancing constraint that aligns with the 10\-epoch BabyLM budget and prevents dominance by one language\. The curriculum\-dynamic schedule used by submission\-facing runs is detailed in Algorithm[B](https://arxiv.org/html/2610.33583#A2.SS0.SSS0.Px1)\.

## 3Experimental Setup

We first conduct diagnostic experiments to isolate the four challenges identified by ARMOR\-DP before training submission\-facing checkpoints\. These probes use the official gated BabyBabelLM dataset rows available through the Hugging Face dataset viewer and construct equal\-length 128\-character windows with an equal number of samples for each language\. Unless otherwise stated, the focused diagnostic probe uses 180 windows per language, sequence length 64, a 2\-layer decoder\-only Transformer with hidden size 56 and four attention heads, and is trained for 240 steps over five random seeds\. These deliberately short runs isolate mechanisms; they are not used as estimates of full\-scale language competence\. All benchmark claims come from the 1M\- and 100M\-budget checkpoints described below\. We report the mean validation loss across languages together with the balance gap

Δ=maxℓ⁡Lℓ−minℓ⁡Lℓ,\\Delta=\\max\_\{\\ell\}L\_\{\\ell\}\-\\min\_\{\\ell\}L\_\{\\ell\},which measures cross\-linguistic imbalance\. These diagnostic experiments are intended to identify the dominant source of multilingual weakness and guide component selection rather than serve as the final benchmark evaluation\.

Based on the diagnostic analysis, we train submission\-facing checkpoints by instantiating the three most effective ARMOR\-DP components, namely byte\-level representation, exposure control, and the dynamic sampling controller\. The objective layer is retained as an optional diagnostic module and is evaluated separately through ablation rather than included in the final submission\. For byte\-level representation, all Unicode text is encoded using standard UTF\-8 byte sequences, where each byte is treated as an individual token without language\-specific preprocessing or segmentation\. Byte values are mapped to training ids by adding an offset of two to reserve ids 0 and 1 for padding and EOS tokens\. This mapping is shared throughout training and Hugging Face export, ensuring complete consistency between training and official evaluation\.

For the 100M frontier, we train curriculum\-dynamic checkpoints with a context length of 512 at model sizes d64, d256, d512, and d768, together with a matched d512 static ZH50 baseline\. Training metadata records the actual number of non\-padding predicted tokens, wall\-clock throughput, per\-language consumed epochs, and the maximum consumed epoch, confirming that every reported competition candidate remains below the official ten\-epoch limit\.

We evaluate all checkpoints using the official BabyLM public zero\-shot and fine\-tuning pipelines\. Before full evaluation, interface compatibility is verified using approximately 100k predicted tokens, 300 official rows per language, training sequence length 128, and exported context length 512\. The released equal\-trilingual GPT\-2 baseline from the official evaluation README is included as a public horizontal reference\([BabyLM Organizers, 2026c](https://arxiv.org/html/2610.33583#bib.bib4)\)\. Public\-suite results are used for model selection because leaderboard\-only tasks become available only after submission\. The selected d768 checkpoint is finally submitted to the official leaderboard, and the returned multilingual score is reported separately\.

Table[1](https://arxiv.org/html/2610.33583#S3.T1)maps the checkpoint names used throughout the results to their active components\.

Table 1:Component map for the checkpoint families reported in Sections 4–7\. The d256, d512, and d768 final systems are capacity\-scaled instances of curriculum\-dynamic\.
## 4Results

### 4\.1Balanced input is not balanced pressure

Figure[1](https://arxiv.org/html/2610.33583#S1.F1)compresses the first failure mode behind ARMOR\-DP: a balanced multilingual budget is not automatically a balanced learning pressure\. With the same 23,040\-character budget per language, Chinese contains 66,228 UTF\-8 bytes and 1,535 unique characters, while English and Dutch each have about 23,000 bytes and fewer than 60 unique characters\.

This pressure is visible in Table[2](https://arxiv.org/html/2610.33583#S4.T2)\. A character\-level balanced baseline collapses on Chinese, with a mean loss of 3\.066 and a balance gap of 1\.858\. Switching only the representation to byte units reduces mean loss to 2\.362 and gap to 0\.185\. This is the clearest evidence for the first ARMOR\-DP layer\.

Table 2:Controlled ARMOR\-DP probe on official BabyBabelLM rows\. Representation armor removes the Chinese collapse; exposure and capacity improve mean loss but can transfer worst\-language status to English\.![Refer to caption](https://arxiv.org/html/2610.33583v1/figures/armor_tradeoff.png)

Figure 2:End\-of\-run score/balance tradeoff across diagnostic systems\. The axes are final mean validation loss and final language gap \(not training steps\); lower is better on both axes\. Byte representation removes the first collapse, while ZH\-protective adapters improve mean loss but widen the language gap, making English the weakest language\.
### 4\.2Protection moves the bottleneck

Exposure and capacity armor improve the average score but change which language needs protection\. Figure[2](https://arxiv.org/html/2610.33583#S4.F2)makes this tradeoff visible without requiring the reader to scan every loss column in Table[2](https://arxiv.org/html/2610.33583#S4.T2)\. Byte \+ ZH50 adapters obtains the lowest mean loss in the controlled probe, but the worst language is English in all seeds\. This transfer is not a failure of the framework; it is the phenomenon the framework should explain\. A multilingual BabyLM system should not protect a fixed language\. It should protect the language currently under the highest developmental pressure\.

### 4\.3Dynamic control turns movement into signal

We therefore ran a dynamic probe\. Every 24 steps, the controller evaluates per\-language losses and updates sampling ratios with capped adaptive protection\. The pure dynamic adapter system keeps the balance gap close to balanced training while slightly lowering mean loss\. A script\-prior variant pushes more mass to Chinese and achieves lower mean loss than pure dynamic control, but at a larger gap\. Static ZH50 remains the lowest mean\-loss configuration in this small run, but has the largest gap among the adapter systems\.

The logged update points also provide a direct trace of weakest\-language movement\. In the objective probe, the weakest language often begins as Dutch, moves to Chinese during early byte\-level learning, and then transfers to English after Chinese\-protective exposure has taken effect\. Across three seeds, the adaptive exposure controller shows two NL→\\rightarrowZH transitions and three ZH→\\rightarrowEN transitions\. This supports the main claim that “weakest language” is a training\-time state, not a fixed label\.

Table 3:Dynamic protection probe over three seeds\. Dynamic loss control is most useful as a balance\-preserving controller; static ZH50 still optimizes mean loss at the cost of English\.![Refer to caption](https://arxiv.org/html/2610.33583v1/figures/dynamic_trace.png)

Figure 3:How the dynamic controller changes exposure\. Each curve is the sampling probability for one language, and the x\-axis is the logged controller\-update index from a dynamic probe \(unlike the end\-of\-run axes in Figure[2](https://arxiv.org/html/2610.33583#S4.F2)\)\. The controller first raises Chinese exposure, then returns probability mass to English and Dutch as Chinese becomes easier, showing why fixed ZH protection can become stale\.![Refer to caption](https://arxiv.org/html/2610.33583v1/figures/weakest_transition.png)

Figure 4:Weakest\-language identity over training\. The highlighted language at each step is the current highest\-loss language\. The bottleneck moves from Dutch to Chinese and then to English, supporting the claim that weakness is a training\-time state rather than a fixed language label\.
### 4\.4Objectives must not repeat exposure

Before comparing objective variants, we distinguish genuine pressure weighting from a reduction artifact: averaging losses by language already equalizes languages even when explicit pressure weights are weak\.

Table[4](https://arxiv.org/html/2610.33583#S4.T4)compares objective variants under byte\-level representation and language adapters\. Naive rare\-token weighting is the least attractive objective in this confirmation run: it lowers Chinese loss but widens the final gap by making English much worse\. Pressure\-aware weighting is more controlled\. Under a fixed ZH50 exposure schedule, it reduces the gap from 0\.348 to 0\.232 with only a modest mean\-loss cost\. However, it still does not beat the balance\-preserving adaptive exposure baseline\. We therefore treat objective armor as necessary but not solved: the evidence favors a pressure\-aware objective, while the final system should tune its strength so it does not duplicate exposure protection\.

We later found that objective design has one additional pitfall\. A language\-level pressure loss can improve balance simply by changing the reduction unit from token means to language means, even when pressure weights are weak\. To separate the objective layer from this hidden equalization effect, we added a token\-level pressure reduction with warmup, end decay, and a capped maximum weight of 1\.4\. With strength 0\.20, this scheduled objective reduces the local 1M validation gap from 0\.253 to 0\.240 while keeping mean loss essentially unchanged \(2\.264 to 2\.262\)\. On the public zero\-shot suite, it reduces the curriculum\-dynamic language gap from 0\.0359 to 0\.0330, but lowers overall accuracy from 0\.4634 to 0\.4612\. We therefore keep curriculum\-dynamic as the scorer and treat scheduled token pressure as evidence for the objective layer rather than as the final checkpoint\.

Table 4:Objective\-layer probe over three seeds\. Pressure\-aware weighting is substantially less damaging than rare\-token weighting and reduces the over\-protection gap under ZH50, but adaptive exposure remains the better balance\-preserving controller\.
### 4\.5Public evaluation separates score and diagnosis

Table[5](https://arxiv.org/html/2610.33583#S4.T5)reports a full pass through the official multilingual zero\-shot task groups for the original five 1M\-token checkpoints plus the best scheduled token\-pressure variant\. These are not interface smoke tests: each checkpoint is loaded through the HuggingFace causal\-LM path and evaluated with the official English, Dutch, and Chineselm\-evaltask definitions\. Hidden leaderboard tasks are not available locally, so the table should be read as public\-suite evidence\.

The official score changes the model\-selection story\. Static ZH50 remains the best local language\-modeling\-loss checkpoint, but it is not the strongest zero\-shot system\. Curriculum\-dynamic protection obtains the best mean score by a small margin over balanced byte adapters, and dynamic\+pressure obtains the smallest language gap and variance while sacrificing mean accuracy\. This is the desired ARMOR\-DP separation: the competition\-facing scorer, the balance\-preserving controller, and the diagnostic pressure objective need not be the same checkpoint\.

Table 5:Full public official zero\-shot evaluation for 1M\-token checkpoints\. Overall is the mean over English, Dutch, and Chinese task\-group means\. Gap and variance are computed over language means, so lower is more balanced\.We evaluate all checkpoints using the official multilingual fine\-tuning pipeline over 22 public task\-language pairs\. Since fine\-tuning losses are not directly comparable across task\-specific classifiers, Table[6](https://arxiv.org/html/2610.33583#S4.T6)reports the official accuracy metric\. Curriculum\-dynamic achieves the highest fine\-tuning mean and the best average of zero\-shot and fine\-tuning performance, primarily driven by improved Chinese accuracy, which increases from 0\.210 under balanced adapters and 0\.207 under static ZH50 to 0\.244\. Pressure\-aware variants instead prioritize cross\-lingual balance, with curriculum\-dynamic\+pressure achieving the smallest fine\-tuning gap while shifting the weakest language from Chinese to Dutch\.

Table 6:Full public official fine\-tuning evaluation\. Fine is the mean over language\-level task means\. Combined is the unweighted average of the zero\-shot overall score in Table[5](https://arxiv.org/html/2610.33583#S4.T5)and the fine\-tuning mean here\. Gap and variance are computed over fine\-tuning language means\.
### 4\.6Focused 1M grid

The next local experiment scales the most relevant systems from the 100k\-token interface probe to 1M predicted tokens, using 600 cached official rows per language\. Table[7](https://arxiv.org/html/2610.33583#S4.T7)shows the 1M local validation losses and the actual consumed epochs logged by the training sampler\. All runs remain below the 10\-epoch competition limit\. Static ZH50 obtains the lowest mean loss but does so by driving Chinese loss far below English and Dutch, making English the weakest language and widening the gap to 0\.389\. Curriculum\-dynamic protection gives up 0\.015 mean loss while reducing the gap to 0\.253\. Balanced byte adapters are also highly competitive, which reinforces the representation claim: script\-fair representation is a prerequisite before exposure control can be interpreted\.

Although dynamic\+pressure and curriculum\-dynamic\+pressure improve cross\-lingual balance, they incur a reduction in the overall mean score compared with curriculum\-dynamic alone\. We therefore retain pressure\-aware weighting as an optional fourth\-layer mechanism for diagnostic analysis rather than including it in the final submission, suggesting that its effectiveness depends on appropriate strength scheduling\.

Table 7:Focused 1M\-token grid on cached official rows\. The epoch column is the maximum actual consumed epoch over English, Dutch, and Chinese, recorded in each checkpoint’s training metadata\.
### 4\.7Capacity scaling makes dynamic protection competitive

The official public suite requires an interface that can tolerate long zero\-shot contexts\. We therefore closed a 512\-context line after the 1M public evaluation\. A first budget curve with the tiny d64 model showed that simply training longer was not enough: the best 100M d64 curriculum\-dynamic run reached only 0\.333 combined\. A substantial improvement came from capacity armor\. Increasing the same curriculum\-dynamic system to d256, d512, and d768 raises the public combined score to 0\.371, 0\.381, and 0\.387 respectively\.

This changes the ARMOR\-DP submission decision\. The 100M d768 checkpoint is the strongest ARMOR\-DP public scorer and also has a competitive public language gap, while the d512 checkpoint provides the clearest matched\-capacity comparison against static ZH50\. A matched d512 static ZH50 baseline falls to 0\.359 combined, and its local language\-modeling gap is much larger: 0\.495 versus 0\.079 for d512 curriculum\-dynamic\. Static ZH50 also consumes 9\.78 Chinese epochs, close to the 10\-epoch competition limit, whereas curriculum\-dynamic remains below 6\.61 epochs for every language\. Thus ARMOR\-DP’s dynamic exposure is no longer only diagnostic at the final scale; it is also the stronger public\-suite system under matched capacity\. The released equal\-trilingual GPT\-2 baseline still sets a higher public combined reference \(0\.410 versus 0\.387\), mainly through zero\-shot accuracy, while d768 is comparable on fine\-tuning\. This makes the current award\-facing claim sharper: ARMOR\-DP explains and improves the dynamic\-protection frontier, but it should not be described as a finished leaderboard winner before hidden evaluation\. The organizer GPT\-2 uses a conventional subword tokenizer, whereas ARMOR\-DP uses bytes; a token therefore does not represent the same amount of linguistic content in the two systems\. We include GPT\-2 only as an external score reference, not as a controlled tokenizer comparison\. Figure[5](https://arxiv.org/html/2610.33583#S4.F5)turns this model\-selection result into the frontier view used for the final submission decision: moving from d64 to larger curriculum\-dynamic models improves score without returning to the large local gap of static ZH50\.

Table 8:Public official capacity frontier\. “Non\-pad tok\.“ counts actual predicted non\-padding tokens; all ARMOR\-DP rows remain under the 10\-epoch competition cap\. The d512 curr\.\-dyn\. and d512 static rows are matched\-capacity systems\. The GPT\-2 row is the released equal\-trilingual baseline from the official evaluation README; its subword tokens are not content\-equivalent to ARMOR\-DP’s byte tokens\.#### Final leaderboard submission\.

We submit the d768 curriculum\-dynamic checkpoint as the final challenge\-facing ARMOR\-DP system\. The official leaderboard accepts the submission as challenge eligible and returns a multilingual average of 38\.32\. The language averages are 35\.34 for English, 40\.96 for Dutch, and 38\.65 for Chinese\. This result should be read as a final submission check rather than a new model\-selection signal: all system choices above were made from public official evaluations, and the leaderboard result confirms the submitted checkpoint while preserving the paper’s main claim as diagnostic rather than leaderboard dominance\.

![Refer to caption](https://arxiv.org/html/2610.33583v1/figures/scaling_frontier.png)

Figure 5:Internal ARMOR\-DP public score/balance frontier for the 100M line\. The released GPT\-2 equal\-trilingual baseline is higher \(0\.4097 combined\) and is reported in Table[8](https://arxiv.org/html/2610.33583#S4.T8), but is not drawn because it is an external organizer baseline rather than an ARMOR\-DP variant\. Within the internal frontier, capacity scaling turns curriculum\-dynamic protection into the best\-performing ARMOR\-DP scorer, while the matched static ZH50 baseline sits lower despite a small public gap\.

## 5Related Work

ARMOR\-DP builds on three lines of work\. First, BabyLM studies sample\-efficient pretraining under developmentally plausible data budgets\([BabyLM Organizers, 2026b](https://arxiv.org/html/2610.33583#bib.bib1);[BabyLM Organizers, 2026a](https://arxiv.org/html/2610.33583#bib.bib2);[Warstadt and others, 2025](https://arxiv.org/html/2610.33583#bib.bib5)\)\. Our setting adds the multilingual constraint that a fixed token or character budget can interact with script and morphology\. Second, token\-free and byte\-level models show that character and byte units can reduce tokenizer\-specific assumptions\([Xue et al\., 2022](https://arxiv.org/html/2610.33583#bib.bib14);[Clark et al\., 2022](https://arxiv.org/html/2610.33583#bib.bib7);[Tay et al\., 2022](https://arxiv.org/html/2610.33583#bib.bib13)\); here we use bytes as a diagnostic representation, not as a claim that bytes are universally optimal\. Third, multilingual sampling, adapters, curriculum learning, sparse routing, and loss reweighting provide tools for allocating exposure, capacity, and gradient pressure\([Conneau et al\., 2020](https://arxiv.org/html/2610.33583#bib.bib8);[Houlsby et al\., 2019](https://arxiv.org/html/2610.33583#bib.bib10);[Pfeiffer et al\., 2020](https://arxiv.org/html/2610.33583#bib.bib12);[Bengio et al\., 2009](https://arxiv.org/html/2610.33583#bib.bib6);[Fedus et al\., 2022](https://arxiv.org/html/2610.33583#bib.bib9);[Lin et al\., 2017](https://arxiv.org/html/2610.33583#bib.bib11)\)\. ARMOR\-DP’s contribution is to combine these tools around the moving identity of the weakest language\.

## 6Discussion

Our experiments show that multilingual imbalance is an evolving training state: the dominant weakness shifts from representation to exposure and then capacity, so static language protection becomes progressively less effective\. ARMOR\-DP separates these factors and their interventions, allowing adaptation to follow the current learning state rather than a predetermined schedule\.

For the BabyLM challenge, byte\-level representation, exposure control, and the dynamic controller consistently improve multilingual performance and form our final system\. The objective layer is optional: pressure\-aware weighting reduces cross\-lingual gaps but modestly lowers the mean score, suggesting that its strength should be scheduled rather than applied unconditionally\. This modular separation also makes interactions and future combinations easier to analyze\.

## 7Submission Choice

The final submission is the curriculum\-dynamic d768 checkpoint, which has the best overall public performance while maintaining competitive cross\-lingual balance\. We retain the matched d512 checkpoint as the principal diagnostic control against static ZH50\.

## 8Conclusion

ARMOR\-DP reframes multilingual BabyLM as protection of the language currently under greatest developmental pressure\. Across diagnostics, 1M probes, and capacity scaling, representation establishes a script\-fair foundation, exposure and capacity improve performance, and dynamic control tracks bottleneck transitions that fixed protection misses\. By separating diagnosis from modular, empirically instantiable interventions, ARMOR\-DP offers a flexible basis for future multilingual training\.

## Limitations

The experiments use small diagnostic models and cached subsets of official dataset rows\. The public official suite is complete for the checkpoints reported here, and we include the released GPT\-2 equal\-trilingual public baseline for orientation\. Leaderboard\-only tasks were not available during model selection; they were scored only after the final d768 checkpoint was submitted\. We therefore treat the official leaderboard average as a post\-selection submission result, not as a tuning signal\. The 100M\-budget runs use a 100M maximum\-token training budget, but actual predicted non\-padding tokens are lower because many cached examples are shorter than 512 tokens\. The 100M capacity frontier is single\-seed because of compute cost; diagnostic claims are supported by multi\-seed small probes, but large\-scale score variance remains unmeasured\. The controller is evaluated on only three languages, and a larger language set may produce noisier pressure signals and require different update bounds\. We also report one controller configuration rather than a full sensitivity sweep\. The script\-complexity prior was the most brittle component: it over\-corrected Chinese later in training and was disabled in final runs\. A scheduled prior that decays after early training remains an open direction\.

## References

- BabyLM Organizers \(2026a\)BabyLM OrganizersBabyBabelLM: multilingual developmentally plausible language modeling\.Note:[https://babylm\.github\.io/babybabellm/](https://babylm.github.io/babybabellm/)Accessed 2026\-05\-06Cited by:[§1](https://arxiv.org/html/2610.33583#S1.p1.1),[§5](https://arxiv.org/html/2610.33583#S5.p1.1)\.
- BabyLM Organizers \(2026b\)BabyLM OrganizersBabyLM challenge guidelines\.Note:[https://babylm\.github\.io/guidelines\.html](https://babylm.github.io/guidelines.html)Accessed 2026\-05\-06Cited by:[§1](https://arxiv.org/html/2610.33583#S1.p1.1),[§5](https://arxiv.org/html/2610.33583#S5.p1.1)\.
- BabyLM Organizers \(2026c\)BabyLM OrganizersBabyLM evaluation pipeline\.Note:[https://github\.com/babylm\-org/babylm\-eval](https://github.com/babylm-org/babylm-eval)Accessed 2026\-05\-10Cited by:[§3](https://arxiv.org/html/2610.33583#S3.p4.1)\.
- Bengioet al\.\(2009\)Y\. Bengio, J\. Louradour, R\. Collobert, and J\. WestonCurriculum learning\.InProceedings of the 26th International Conference on Machine Learning,pp\. 41–48\.Cited by:[§2](https://arxiv.org/html/2610.33583#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2610.33583#S5.p1.1)\.
- Clarket al\.\(2022\)J\. H\. Clark, D\. Garrette, I\. Turc, and J\. WietingCANINE: pre\-training an efficient tokenization\-free encoder for language representation\.Transactions of the Association for Computational Linguistics10,pp\. 73–91\.Cited by:[§2](https://arxiv.org/html/2610.33583#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.33583#S5.p1.1)\.
- Conneauet al\.\(2020\)A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. StoyanovUnsupervised cross\-lingual representation learning at scale\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 8440–8451\.Cited by:[§2](https://arxiv.org/html/2610.33583#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2610.33583#S5.p1.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.Cited by:[§2](https://arxiv.org/html/2610.33583#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2610.33583#S5.p1.1)\.
- Houlsbyet al\.\(2019\)N\. Houlsby, A\. Giurgiu, S\. Jastrzebski, B\. Morrone, Q\. de Laroussilhe, A\. Gesmundo, M\. Attariyan, and S\. GellyParameter\-efficient transfer learning for NLP\.InProceedings of the 36th International Conference on Machine Learning,pp\. 2790–2799\.Cited by:[§2](https://arxiv.org/html/2610.33583#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2610.33583#S5.p1.1)\.
- Linet al\.\(2017\)T\. Lin, P\. Goyal, R\. Girshick, K\. He, and P\. DollárFocal loss for dense object detection\.InProceedings of the IEEE International Conference on Computer Vision,pp\. 2980–2988\.Cited by:[§2](https://arxiv.org/html/2610.33583#S2.SS0.SSS0.Px4.p1.1),[§5](https://arxiv.org/html/2610.33583#S5.p1.1)\.
- Pfeifferet al\.\(2020\)J\. Pfeiffer, I\. Vulić, I\. Gurevych, and S\. RuderMAD\-X: an adapter\-based framework for multi\-task cross\-lingual transfer\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing,pp\. 7654–7673\.Cited by:[§2](https://arxiv.org/html/2610.33583#S2.SS0.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2610.33583#S5.p1.1)\.
- Tayet al\.\(2022\)Y\. Tay, V\. Q\. Tran, S\. Ruder, J\. Gupta, H\. W\. Chung, D\. Bahri, Z\. Qin, S\. Baumgartner, C\. Yu, and D\. MetzlerCharformer: fast character transformers via gradient\-based subword tokenization\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2610.33583#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.33583#S5.p1.1)\.
- Vosoughi and Roy \(2012\)S\. Vosoughi and D\. K\. RoyA longitudinal study of prosodic exaggeration in child\-directed speech\.InProceedings of Speech Prosody 2012,pp\. 194–197\.External Links:[Document](https://dx.doi.org/10.21437/SpeechProsody.2012-50)Cited by:[§1](https://arxiv.org/html/2610.33583#S1.p1.1)\.
- Warstadtet al\.\(2025\)A\. Warstadtet al\.The third BabyLM challenge: sample\-efficient pretraining on developmentally plausible corpora\.InProceedings of the BabyLM Challenge,Cited by:[§5](https://arxiv.org/html/2610.33583#S5.p1.1)\.
- Xueet al\.\(2022\)L\. Xue, A\. Barua, N\. Constant, R\. Al\-Rfou, S\. Narang, M\. Kale, A\. Roberts, and C\. RaffelByT5: towards a token\-free future with pre\-trained byte\-to\-byte models\.Transactions of the Association for Computational Linguistics10,pp\. 291–306\.Cited by:[§2](https://arxiv.org/html/2610.33583#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.33583#S5.p1.1)\.

## Appendix AReproducibility Details

The main paper keeps the eight\-page submission concise\. This appendix records the implementation details that are most likely to matter for reproducing or auditing the ARMOR\-DP claim\.

#### Data and tokenization\.

All controlled diagnostics use English, Dutch, and Chinese rows from the BabyBabelLM public data interface\. We form equal\-length windows per language for the diagnostic probes so that representation and exposure interventions can be compared under matched surface budgets\. Submission\-facing checkpoints use a UTF\-8 byte tokenizer with the same id mapping in training and HuggingFace export\. Padding labels are masked from the language\-modeling loss, and all reported token counts are predicted non\-padding tokens rather than padded sequence slots\.

#### Evaluation\.

The public official suite is run through the HuggingFace causal\-LM loading path for zero\-shot tasks and through the official multilingual fine\-tuning pipeline for supervised tasks\. Hidden leaderboard tasks are not locally available\. The fine\-tuning pass contains 22 public task\-language pairs for the 100M capacity frontier, with no missing tasks\.

#### Epoch accounting\.

The training loop records consumed epochs per language by counting the actual examples sampled for that language\. This is stricter than reporting optimizer steps alone: a static ZH50 mixer can keep the global token budget modest while nearly exhausting the Chinese\-language epoch cap\. All competition\-facing rows reported in the main paper and in the ledger below remain below the 10\-epoch limit\.

Table 9:Training\-accounting ledger\. Epochs are actual consumed epochs for English, Dutch, and Chinese, not estimates from optimizer steps\. “Wall min” is reported when available from the remote training metadata\.

## Appendix BHyperparameters and Controller Settings

Table 10:Core hyperparameters\. All competition\-facing runs setmax\_epochs=10; the actual consumed\-epoch ledger is in Table[9](https://arxiv.org/html/2610.33583#A1.T9)\.
#### Controller pseudocode\.

Algorithm 1: Curriculum\-dynamic exposure controller\.Initializep=\(1/3,1/3,1/3\)p=\(1/3,1/3,1/3\); train balanced untilT/2T/2; setp=\(0\.25,0\.25,0\.50\)p=\(0\.25,0\.25,0\.50\); afterT/2T/2, everyKKsteps, evaluateLℓtL\_\{\\ell\}^\{t\}, computeρℓt\\rho\_\{\\ell\}^\{t\}andqℓtq\_\{\\ell\}^\{t\}withβ=2\.0,η=0\.5,τ=2\.0,λe=0\.5,λs=0\\beta=2\.0,\\eta=0\.5,\\tau=2\.0,\\lambda\_\{e\}=0\.5,\\lambda\_\{s\}=0, then clip to\[0\.18,0\.50\]\[0\.18,0\.50\], renormalize, and continue\.

#### Controller sensitivity\.

We did not run an exhaustive sensitivity sweep\. The clearest sensitivity was to the script prior: enablingλs\>0\\lambda\_\{s\}\>0could help Chinese early but over\-corrected it later, so all final checkpoints useλs=0\\lambda\_\{s\}=0\. A scheduled, decaying script prior may preserve the early benefit without the late bias\. The behavior of the bounds and update rule with substantially more than three languages is left for future work\.

## Appendix CAdditional Public Evaluation Details

Table[11](https://arxiv.org/html/2610.33583#A3.T11)adds the 5M 512\-context public suite that is summarized only implicitly in the main paper\. At 5M, curriculum\-dynamic already beats static ZH50 on combined score, mostly because static ZH50 loses more fine\-tuning accuracy\. At 100M, the same direction becomes the submission\-facing line\.

Table 11:Additional public official combined results\. Combined is the unweighted average of zero\-shot and fine\-tuning means\.
Table[12](https://arxiv.org/html/2610.33583#A3.T12)expands the public fine\-tuning tasks for the final 100M d768 curriculum\-dynamic scorer\. The table is not used to tune claims in the main paper; it is included to make clear which public task groups underlie the reported fine\-tuning mean\.

Table 12:Public fine\-tuning task accuracies for the 100M d768 curriculum\-dynamic checkpoint\. Dashes indicate language\-task pairs not present in the local public suite\.

## Appendix DNegative Results and Deferred Directions

We preserve the failed or deferred branches because they help distinguish the framework from a post\-hoc winning recipe\. Static ZH50 is retained as the matched baseline that makes the dynamic comparison meaningful, rather than discarded after the final system is chosen\.

#### Simple gates were not enough\.

The original gate direction was too weak when treated as a generic attention modification\. A language\-conditioned routing gate became more interpretable as capacity armor and obtained a reasonable diagnostic tradeoff, but it did not beat adapters as the main scorer\. We therefore use adapters in the final competition\-facing line and keep routing gates as a capacity diagnostic\.

#### Rare\-token weighting over\-protects Chinese\.

Rare\-token weighting lowers Chinese loss after Chinese\-protective exposure has already helped, but it widens the balance gap by making English the weakest language\. In the objective probe, ZH50\+rare reaches a gap of 0\.481, compared with 0\.348 for ZH50 and 0\.118 for adaptive exposure\. We treat this as a negative result for frequency\-only objectives\.

#### Pressure\-aware weighting is useful but not final\.

Pressure\-aware weighting aligns better with ARMOR\-DP than rare\-token weighting and gives the smallest public zero\-shot gap in the 1M suite and the smallest public fine\-tuning gap with curriculum\-dynamic exposure\. Because it lowers mean public score in the current tuning range, the final scorer omits it; pressure objectives remain a fourth\-layer research direction\.