A Declarative-Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models

# A Declarative–Procedural Perspective on Expert Routing in Bilingual Mixture-of-Experts Language Models
Source: [https://arxiv.org/html/2608.15102](https://arxiv.org/html/2608.15102)
RaghulAffiliation:Sri Sivasubramaniya Nadar College of Engineering, Chennai, IndiaDurairaj ThenmozhiAffiliation:Shiv Nadar University Chennai, Indiaamrit2410182@ssn\.edu\.in, raghul2510435@ssn\.edu\.inthenmozhid@snuchennai\.edu\.in

###### Abstract

We investigate whether Mixture\-of\-Experts \(MoE\) language models develop linguistically structured expert routing during bilingual language acquisition\. Inspired by the Declarative–Procedural framework, we analyze lexical, grammatical, and syntactic processing in a decoder\-only English–German MoE Transformer trained under sequential language exposure\. We construct a probe\-based validation set and extract token\-level routing distributions to quantify category\-dependent specialisation using mutual information, routing entropy, and Jensen–Shannon distance\. The curriculum\-trained model exhibits a peak mutual information of 0\.1148 at layer 5, indicating category\-dependent differences in routing distributions across linguistic categories\. Surprisingly, a no\-curriculum baseline trained on mixed English–German data shows stronger aggregate specialisation, reaching a peak mutual information of 0\.2599 at the same layer\. These results suggest that interpretable linguistic organization emerges within MoE routing patterns even without sequential language exposure\. A replication at a second training seed shows that the no\-curriculum condition’s specialisation concentrates on a single language whose identity is seed\-dependent, whereas the curriculum consistently yields a stable, language\-balanced routing profile; rather than uniformly increasing specialisation, staged bilingual exposure reduces single\-language dominance\. The official Github repository:[Github link](https://github.com/Amrit828/DP-Theory-MOE-Interpretability-Research)

## 1Introduction

Mixture\-of\-Experts \(MoE\) language models route each token across a subset of expert networks instead of sending every token through the same feed\-forward network[14](https://arxiv.org/html/2608.15102#bib.bib12);[8](https://arxiv.org/html/2608.15102#bib.bib6)\. This makes MoE models useful not only for scaling, but also for studying how neural language models choose experts internally\. If different kinds of linguistic tokens are routed to different experts, the routing mechanism may exhibit an organized form of expert distribution inside the model\.

In this work, we examine whether bilingual MoE language models develop routing patterns that correspond to meaningful linguistic categories\. We focus on three categories motivated by the Declarative–Procedural view of language: lexical knowledge, grammatical processing, and syntactic structure[17](https://arxiv.org/html/2608.15102#bib.bib14);[18](https://arxiv.org/html/2608.15102#bib.bib15);[19](https://arxiv.org/html/2608.15102#bib.bib16)\. In this paper, a probe is a token chosen from a sentence to be examined\. Lexical probes are tokens that test word knowledge, such as irregular forms\. Grammatical probes are tokens that test rule\-based morphology and agreement\. Syntactic probes are tokens that participate in clause structure and sentence organization\. Instead of examining only the model’s final predictions, we directly analyse the expert\-routing distributions associated with these probe tokens\.

We train a sparse bilingual transformer using a staged English–German curriculum\. The training curriculum starts with English and gradually transitions to German, approximating structured second\-language exposure and curriculum learning in bilingual settings[1](https://arxiv.org/html/2608.15102#bib.bib1);[12](https://arxiv.org/html/2608.15102#bib.bib10);[22](https://arxiv.org/html/2608.15102#bib.bib18)\. We compare this structured curriculum to an unstructured no\-curriculum setting in which English and German are introduced without gradual cumulative exposure\. This lets us examine whether MoE routing becomes organized by language and whether staged bilingual exposure affects how that organization is distributed across languages\.

To measure the development of an organized routing schema, we use a held\-out validation set with lexical, grammatical, and syntactic probes in both English and German\. We examine the router’s probability distribution over experts at each MoE layer for each probe token\. We then measure the relationship between expert routing and linguistic category using mutual information, entropy, Jensen–Shannon divergence, and sentence\-level permutation testing[13](https://arxiv.org/html/2608.15102#bib.bib11);[10](https://arxiv.org/html/2608.15102#bib.bib7);[3](https://arxiv.org/html/2608.15102#bib.bib2)\. This complements recent work that studies behavioral declarative and procedural knowledge in datasets and large language models[9](https://arxiv.org/html/2608.15102#bib.bib8)by moving the analysis to the level of internal routing\.

Our results show that MoE routing distributions contain measurable information about linguistic category membership: the category of a probe token can be partially inferred from its expert probability distribution because routing and category exhibit non\-trivial mutual information\. Lexical, grammatical, and syntactic probes are routed differently\. The clearest specialisation appears in intermediate MoE layers, whereas later layers route more diffusely and show reduced mutual information\. We also find that curriculum affects how specialisation is balanced between languages\. Without a curriculum, routing specialisation concentrates heavily on one language, and this concentration is seed\-dependent; the staged L1–L2 curriculum instead yields a language\-balanced routing organization that is stable across seeds\.

In general, this study shows that bilingual MoE routers develop a language\-sensitive quantifiable expert\-routing structure that reflects linguistically significant differences\. These findings suggest that expert routing provides a useful mechanistic window into how bilingual language models allocate computation across different types of language processing\.

The central motivation of this paper is to move beyond asking whether a bilingual MoE model performs well on a diagnostic task and instead ask how its internal routing mechanism organizes linguistic computation\. Our explicit contribution is a routing\-level analysis framework for testing whether expert allocation is sensitive to lexical, grammatical, and syntactic probe categories under staged bilingual exposure\. By comparing a forward English–German curriculum with a no\-curriculum baseline, we isolate how training order changes the distribution of routing specialisation across languages\. This makes the study a mechanistic investigation of expert routing in bilingual MoE models, rather than a general benchmark of language\-model accuracy\.

## 2Related Work

This section places the study in the context of four connected areas: the Declarative–Procedural view of language, the linguistic analysis of transformer models, expert routing in MoE architectures, and curriculum learning for bilingual exposure\. Together, these areas lead to the main question of the paper: does expert routing in a bilingual MoE model become sensitive to lexical, grammatical, and syntactic differences? Also, does staged language exposure change that routing structure?

##### Declarative–Procedural theory\.

The Declarative–Procedural framework divides language knowledge into two main parts[17](https://arxiv.org/html/2608.15102#bib.bib14);[18](https://arxiv.org/html/2608.15102#bib.bib15);[19](https://arxiv.org/html/2608.15102#bib.bib16)\. Declarative knowledge relates to stored word knowledge, such as vocabulary and irregular forms\. Procedural knowledge relates to rule\-based grammar and compositional structure\. In this paper, we use this framework only as a linguistic lens\. We do not claim that MoE routers are the same as human memory systems\.

##### Declarative and procedural knowledge in LLMs\.

Recent studies have looked at declarative and procedural knowledge in language models mainly through model outputs and task performance[9](https://arxiv.org/html/2608.15102#bib.bib8)\. Our work explores this question within the model\. Instead of just asking if the model provides the right answer, we investigate whether a sparse bilingual model allocates computation differently for varying types of probe tokens\.

##### Transformer linguistic structure\.

Previous studies indicate that transformer layers do not all encode the same type of linguistic information[20](https://arxiv.org/html/2608.15102#bib.bib17);[16](https://arxiv.org/html/2608.15102#bib.bib13);[2](https://arxiv.org/html/2608.15102#bib.bib3);[5](https://arxiv.org/html/2608.15102#bib.bib5)\. Intermediate layers often have clearer syntactic and semantic structures than very early or very late layers\. This motivates our layer\-wise analysis of MoE routing\. We specifically question whether middle routed layers show stronger category\-sensitive expert allocation\.

##### Mixture\-of\-Experts routing\.

MoE models use routers to direct tokens to a subset of experts[14](https://arxiv.org/html/2608.15102#bib.bib12);[8](https://arxiv.org/html/2608.15102#bib.bib6)\. Earlier work has shown that experts can specialize based on token statistics, domains, routing design, and balancing constraints[4](https://arxiv.org/html/2608.15102#bib.bib4);[7](https://arxiv.org/html/2608.15102#bib.bib19);[15](https://arxiv.org/html/2608.15102#bib.bib20)\. Our research stands out by focusing on bilingual linguistic probe categories and exploring how curriculum structure impacts the distribution of routing specialization across English and German\.

##### Curriculum learning and bilingual exposure\.

Curriculum learning investigates how the order of training examples influences learning[1](https://arxiv.org/html/2608.15102#bib.bib1);[12](https://arxiv.org/html/2608.15102#bib.bib10);[22](https://arxiv.org/html/2608.15102#bib.bib18)\. In bilingual training, staged exposure is useful because it allows us to compare two scenarios: one where the model sees English first and German later, and another where both languages are mixed from the start\. This comparison helps us assess whether the order of language exposure affects how routing specialization develops across languages\.

## 3Routing Analysis Framework

The model contains MoE blocks at the routed layersL=\{1,3,5,7\}L=\\\{1,3,5,7\\\}, each withN=8N=8experts\. LetEEdenote the expert\-routing random variable, a categorical variable whose values are the eight experts, and letCCdenote the linguistic category random variable, taking the three \(not binary\) valuesC=\{lexical,grammatical,syntactic\}C=\\\{\\textit\{lexical\},\\textit\{grammatical\},\\textit\{syntactic\}\\\}\. For a tokenttat routed layerll, the router produces a probability distribution over the eight experts, denotedr\(l\)​\(t\)r^\{\(l\)\}\(t\); throughout the paper this is the full post\-softmax router distribution, taken*before*top\-kkmasking, not the sparse renormalised vector used in the forward computation\.

##### Probing mechanism\.

Probe tokens are annotated in each sentence using a parser \(details in Methodology and Appendix[A](https://arxiv.org/html/2608.15102#A1)\)\. The routing vector for the probe token is extracted from the router during inference and used to compute the routing statistics below, independently for each routed layer\.

We testH0:P⁡\(E\|C\)=P⁡\(E\)H\_\{0\}:P\(E\|C\)=P\(E\)\(routing is independent of category\) againstH1:P⁡\(E\|C\)≠P⁡\(E\)H\_\{1\}:P\(E\|C\)\\neq P\(E\), equivalentlyH0:M​I​\(E,C\)=0H\_\{0\}:MI\(E;C\)=0versusH1:M​I​\(E,C\)\>0H\_\{1\}:MI\(E;C\)\>0\. This concerns the aggregate routing distribution, not individual tokens: the cumulative expert allocation of several thousand category probes can still match the global allocation, soH0H\_\{0\}is an empirically realisable regime rather than a strawman – a random\-routing control produces it almost exactly \(MI≈0\.002\\approx 0\.002; Appendix[G](https://arxiv.org/html/2608.15102#A7)\), and the load\-balancing objective used during training explicitly pushes the router toward category\-independent expert usage\.

#### 3\.0\.1Statistical Testing Framework

The primary statistical tool used is mutual information between expert\-routing distributions and linguistic category\. Higher mutual information indicates a stronger association between the router’s expert probability mass and linguistic category\.

The marginal routing probability assigned to experteeis computed as

p⁡\(e\)=∑τ∈Tr\(l\)​\(τ\)​\[e\]∑e′∑τ∈Tr\(l\)​\(τ\)​\[e′\]p\(e\)=\\frac\{\\sum\_\{\\tau\\in T\}r^\{\(l\)\}\(\\tau\)\[e\]\}\{\\sum\_\{e^\{\\prime\}\}\\sum\_\{\\tau\\in T\}r^\{\(l\)\}\(\\tau\)\[e^\{\\prime\}\]\}\(1\)
and the joint probability between experteeand categoryccis computed as

p⁡\(e,c\)=∑τ∈Tcr\(l\)​\(τ\)​\[e\]∑c′∑e′∑τ∈Tc′r\(l\)​\(τ\)​\[e′\]p\(e,c\)=\\frac\{\\sum\_\{\\tau\\in T\_\{c\}\}r^\{\(l\)\}\(\\tau\)\[e\]\}\{\\sum\_\{c^\{\\prime\}\}\\sum\_\{e^\{\\prime\}\}\\sum\_\{\\tau\\in T\_\{c^\{\\prime\}\}\}r^\{\(l\)\}\(\\tau\)\[e^\{\\prime\}\]\}\(2\)
whereTTdenotes the complete set of probe tokens andTcT\_\{c\}denotes the subset belonging to categorycc\.

Mutual information is then computed as

M​I​\(E,C\)=∑e∑cp⁡\(e,c\)​log⁡\(p⁡\(e,c\)p⁡\(e\)​p​\(c\)\)MI\(E;C\)=\\sum\_\{e\}\\sum\_\{c\}p\(e,c\)\\log\\left\(\\frac\{p\(e,c\)\}\{p\(e\)p\(c\)\}\\right\)\(3\)
wherep⁡\(e\)p\(e\)is the marginal routing probability assigned to experteeandp⁡\(c\)p\(c\)is the marginal probability of the routed token belonging to categorycc\. Robustness is assessed with a sentence\-level permutation test atp<0\.01p<0\.01\.

#### 3\.0\.2Entropy Analysis

Entropy is the second statistical tool, measuring how distributed routing is across experts: higher entropy indicates a more diffuse expert\-usage distribution, lower entropy indicates concentration on a smaller subset of experts\. Shannon entropy is given by

H\(E\)=−∑ep\(e\)lnp\(e\)H\(E\)=\-\\sum\_\{e\}p\(e\)\\ln p\(e\)\(4\)wherep⁡\(e\)p\(e\)is the marginal distribution of experteebeing chosen; when computed for a specific category,p⁡\(e\)p\(e\)is estimated only from that category’s probe tokens, so category\-wise entropy measures how concentrated or diffuse expert usage is per category\.

Jensen–Shannon Divergence \(JSD\) measures how routing differs between linguistic categories, given by

JSD\(P\(E\|Ci\)\|\|P\(E\|Cj\)\)\\displaystyle JSD\\\!\\left\(P\(E\|C\_\{i\}\)\\;\|\|\\;P\(E\|C\_\{j\}\)\\right\)\(5\)=12DK​L\(P\(E\|Ci\)\|\|M\)\\displaystyle=\\frac\{1\}\{2\}D\_\{KL\}\\\!\\left\(P\(E\|C\_\{i\}\)\\;\|\|\\;M\\right\)\+12DK​L\(P\(E\|Cj\)\|\|M\)\\displaystyle\+\\frac\{1\}\{2\}D\_\{KL\}\\\!\\left\(P\(E\|C\_\{j\}\)\\;\|\|\\;M\\right\)
where

M=12​\(P⁡\(E\|Ci\)\+P⁡\(E\|Cj\)\)\.M=\\frac\{1\}\{2\}\\left\(P\(E\|C\_\{i\}\)\+P\(E\|C\_\{j\}\)\\right\)\.
We report its square root, the*Jensen–Shannon distance*dJ​S=J​S​Dd\_\{JS\}=\\sqrt\{JSD\}, throughout the paper: unlike the divergence itself, this is a true metric[6](https://arxiv.org/html/2608.15102#bib.bib22), making the three pairwise category separations directly comparable\.

## 4Methodology

### 4\.1Dataset Construction and Probe Annotation

The dataset is constructed from the FineWeb\-Edu corpus[11](https://arxiv.org/html/2608.15102#bib.bib9)for the English part and the German portion of mC4[21](https://arxiv.org/html/2608.15102#bib.bib21)for the German part\. The pipeline uses the Stanza module to categorise words in each sentence as lexical, grammatical, and syntactic\. The taxonomy for this classification is given in Appendix[A](https://arxiv.org/html/2608.15102#A1)\.

One probe token is annotated for each sentence\-category pair\. A sentence may contain probe tokens belonging to multiple linguistic categories and can therefore appear in more than one category\-specific dataset\. In such cases, the sentence is duplicated across the relevant categories, with each category using its corresponding probe token\.

The following table provides the size of the English and German datasets:

Table 1:Dataset size by language and linguistic category\.Note:Lexical probe extraction was terminated before reaching the target dataset size because of computational constraints, resulting in smaller lexical subsets for both languages\. This does not affect the reported analyses, which are conducted on a fixed held\-out validation set\.

### 4\.2Sequential L1–L2 Curriculum

The curriculum is designed to approximate staged second\-language acquisition\. Here, the first language is English \(EN\) and the second language is German \(DE\)\. The first 1–8 epochs consist of an EN\-only curriculum\. From epochs 9–16, the curriculum slowly transitions from EN\-only to bilingual \(EN \+ DE\)\. The choice of language was based on the differences in grammatical structure and the linguistic category taxonomy \(more details in Appendix[A](https://arxiv.org/html/2608.15102#A1)\)\.

λs=0\.20\+0\.05​\(s−9\),9≤s≤14\\lambda\_\{s\}=0\.20\+0\.05\(s\-9\),\\qquad 9\\leq s\\leq 14\(6\)
whereλs\\lambda\_\{s\}denotes the proportion of German samples presented during epochss\.

Within each epoch, the amount of lexical, grammatical, and syntactic probe tokens is equal and follows a repetition \+ new exposure system\. This system can be given by

C​u​rs\\displaystyle Cur\_\{s\}=0\.6⋅C​u​rs−1\\displaystyle=0\.6\\cdot Cur\_\{s\-1\}\(7\)\+0\.4⋅\(NeworCurs′wheres′<s−1\)\\displaystyle\+0\.4\\cdot\(New\\;\\text\{or\}\\;Cur\_\{s^\{\\prime\}\}\\text\{ where \}s^\{\\prime\}<s\-1\)
for2≤s≤162\\leq s\\leq 16\.

C​u​r1=1⋅N​e​wCur\_\{1\}=1\\cdot New\(8\)
whereC​u​rsCur\_\{s\}is the curriculum of epochssandN​e​wNewis the set of samples that are not part of any previous curriculum\.

### 4\.3Diagnostic Held\-out Validation

The validation set consists of 12,000 samples \(6,000 per language and 2,000 per category per language\)\. The validation set is a minimal\-pairs dataset consisting of a pair of sentences,S\+S^\{\+\}andS−S^\{\-\}, whereS\+S^\{\+\}is a linguistically valid and acceptable sentence andS−S^\{\-\}is a linguistically invalid sentence\.

TheS\+S^\{\+\}set was generated synthetically using OpenAI’s GPT\-5\.1 model in order to obtain controlled, diverse, and task\-specific examples\. The generation was guided by rigorous prompting \(prompts provided in Appendix[E](https://arxiv.org/html/2608.15102#A5)\), with valid probe tokens injected into the prompts\. Automatic post\-generation validation is applied to ensure that generated sentences follow the correct category subtype, are not duplicates, and contain the required metadata\. TheS−S^\{\-\}sentence is then deterministically obtained by modifying the probe token to invalidate the sentence in a linguistically meaningful way \(rules provided in Appendix[E](https://arxiv.org/html/2608.15102#A5)\)\.

### 4\.4Experimental Setup

A custom 8\-layer MoE language model is used, with a feed\-forward dimension of 2048, an embedding size of 512, and 8 attention heads per layer\. The model uses the multilingual mBERT tokenizer with a vocabulary size of 119,547\. The total parameter count of the model is 86\.4 million\.

A relatively small model size is chosen to facilitate controlled experimentation and clearer analysis of expert routing behaviour\. The MoE layers are placed in alternate transformer layers \(1, 3, 5, and 7\)\. The primary training setup uses the proposed sequential L1–L2 curriculum, with an additional No\-Curriculum setting used as an ablation baseline\.

The no\-curriculum baseline uses a fixed 80:20 English\-German mixture at every epoch\. This keeps the overall bilingual composition broadly similar to the forward curriculum, although the aggregate exposure is not exactly identical: the forward curriculum yields 81\.56% English and 18\.44% German across the full schedule, whereas the no\-curriculum condition remains fixed at 80% English and 20% German\.

Both conditions receive the same total number of training samples\. The difference between them is the order in which the languages are introduced\. In the forward curriculum, the model is trained on English first and German is introduced gradually\. In the no\-curriculum condition, English and German are mixed from the start using the same overall English\-German proportions\.

The primary runs use top\-kkrouting withk=3k=3and a router load\-balancing coefficient of 0\.01\. Additional hyperparameters are provided in Appendix[C](https://arxiv.org/html/2608.15102#A3)\.

## 5Results and Analysis

All the results, mathematical analyses, and metrics reported below are obtained through inference on the forward\-curriculum model and the no\-curriculum ablation model over the held\-out validation set\. Unless otherwise stated, all MI values reported in Sections 5\.1–5\.5 are computed on the pooled bilingual validation set combining English and German probes\. Language\-specific analyses are introduced separately in Section 5\.6\.

### 5\.1Behavioural Performance

Table[2](https://arxiv.org/html/2608.15102#S5.T2)reports validation performance for the forward\-curriculum and no\-curriculum conditions\. The two setups achieve broadly comparable behavioural outcomes on the held\-out diagnostic validation set\. The Accuracy is computed as token\-level next\-token accuracy on the grammatical sentence strings only, averaged over all non\-padding tokens; it is not a pairwise sentence\-choice accuracy betweenS\+S^\{\+\}andS−S^\{\-\}\. The forward\-curriculum model attains lower perplexity \(111\.19\) than the no\-curriculum model \(117\.01\), whereas the no\-curriculum condition achieves marginally higher accuracy \(29\.65% versus 28\.99%\)\.

Because the two training conditions achieve similar behavioural performance, the routing patterns reported in the following sections are less likely to reflect simple differences in overall model competence\.

The relatively low absolute scores reflect the challenging evaluation setup\. The purpose of the diagnostic set is to expose routing behaviour under controlled linguistic contrasts rather than to serve as a benchmark of language\-model capability\. These scores are not the main outcome of the paper\. They just indicate that both models can handle the diagnostic examples\. The primary analysis focuses on the router\. We examine whether different token types are sent to different experts during validation\.

Table 2:Validation performance on the held\-out diagnostic probe set\.
### 5\.2Emergence of Category\-Dependent Routing

The primary analysis investigates whether routing specialisation occurs across different linguistic categories\. Mutual Information \(MI\) is computed from the inference routing logs to quantify this phenomenon\. The MI values vary across layers, indicating that the degree of specialisation differs throughout the network\. Layer 5 shows the highest mutual information value \(0\.1148\), suggesting the strongest category\-dependent routing behaviour\. In contrast, the final routed layer \(layer 7\) shows an MI value substantially lower than the other routed layers\.

![Refer to caption](https://arxiv.org/html/2608.15102v1/mi.png)Figure 1:Layer\-wise mutual information between expert\-routing distributions and linguistic categories\.To determine whether the obtained MI values \(routing\-category associations\) are statistically significant, a permutation test was conducted using 1,000 sentence\-level permutations\. Across all routed layers, the observed MI values were significantly higher than the corresponding layer\-wise null distributions, with statistical significance ofp<0\.001p<0\.001for all layers\. This result makes a shuffled\-label explanation unlikely and supports the presence of category\-dependent routing behaviour\.

For context, Appendix[G](https://arxiv.org/html/2608.15102#A7)shows a frozen \(untrained\) router reaches MI 0\.0274 and random routing reaches≈\\approx0\.002 at layer 5: most of the trained model’s 0\.1148 requires learned routing, but the frozen baseline is not negligible \(roughly a quarter of the effect\) and should not be attributed entirely to router learning\.

Table 3:Permutation test results for routing\-category mutual information\. All routed layers exhibit statistically significant routing\-category associations\. All reported p\-values correspond to the minimum attainable empirical value under 1000 permutations, indicating that no permuted sample exceeded the observed MI\.
### 5\.3Expert Allocation Patterns

The secondary analysis supporting the expert specialisation claim is performed through category\-wise and layer\-wise expert allocation patterns\. This analysis also explains how experts have been utilized across different linguistic categories\. Although the probability mass is distributed across different experts for each category, certain experts consistently receive higher routing probabilities for specific categories\. This indicates the presence of preferential expert allocation rather than uniform routing\.

![Refer to caption](https://arxiv.org/html/2608.15102v1/category_heatmap.png)Figure 2:Category\-conditioned expert allocation patterns\.We can also observe that there is no collapse to a single expert for any particular linguistic category\. Instead, each category exhibits a distribution of expert preferences, where some experts are selected more frequently than others\. For example, grammatical probes show stronger preferences towards Experts E1 and E3, while syntactic probes exhibit higher utilisation of Experts E3–E5\. This shows that category\-dependent specialisation emerges through differences in routing preference rather than strict expert exclusivity\.

Figure[3](https://arxiv.org/html/2608.15102#S5.F3)presents the layer\-wise expert usage distribution\. It is interesting to note that the different routed layers exhibit different expert usage distributions, with different subsets of experts dominating at different stages of the processing\. This observation is consistent with the layer\-wise MI analysis, suggesting that the routing distribution differs across different parts of the model\.

![Refer to caption](https://arxiv.org/html/2608.15102v1/expert_usage.png)Figure 3:Layer\-wise expert utilisation across routed layers\.While the heatmap and expert allocation reveal how the experts are allocated layer\-wise as well as category\-wise, they do not indicate how diverse or concentrated the routing distribution is\. To examine this aspect, we analyse routing entropy across the routed layers\.

### 5\.4Routing Entropy Analysis

The entropy values vary across different layers and among the different categories as well\. Grammatical probes exhibit the lowest entropy consistently in all the layers \(with the strongest effect in layer 3\), which suggests that the expert utilisation distribution is more concentrated with a smaller subset of experts consistently utilized for this category\.

In contrast, the syntactic probes exhibit the highest entropy values across different layers \(or equal to lexical in layer 1\)\. This indicates that expert usage is spread across more experts for this particular category across all layers\. Together, these observations suggest that different linguistic categories rely on distinct routing strategies, with grammatical processing exhibiting more concentrated expert utilisation and syntactic processing exhibiting more distributed expert usage\.

![Refer to caption](https://arxiv.org/html/2608.15102v1/entropy.png)Figure 4:Routing entropy across routed layers and linguistic categories\. Lower entropy indicates more concentrated expert utilisation\.
### 5\.5Category Separation Analysis

While entropy explains how concentrated or distributed the expert utilisation is, it does not explain how the routing distribution differs between different linguistic categories\. To examine this aspect, we compute the pairwise Jensen–Shannon distance \(Section[3](https://arxiv.org/html/2608.15102#S3)\) between category\-wise routing distributions\.

Table 4:Pairwise Jensen–Shannon distance between category\-conditioned routing distributions\.Table[4](https://arxiv.org/html/2608.15102#S5.T4)shows the pairwise Jensen–Shannon distances between grammatical, lexical and syntactic routing distributions\. All category pairs exhibit non\-zero separation, suggesting that the router assigns different expert utilisation distributions to different linguistic categories\. We can observe that the routing distributions are farthest apart between lexical and syntactic processing, whereas grammatical and lexical processing are closest\. These observations are consistent with the mutual information and expert allocation analyses, providing additional evidence for category\-dependent routing\.

### 5\.6Effect of Curriculum on Specialisation

In the no\-curriculum setting, the mutual information peaks at 0\.2599 in layer 5 and reaches 0\.1945 in layer 1\. Similar to the forward curriculum setting, the strongest routing\-category association is observed in the intermediate routed layers\. These observations indicate that routing specialisation emerges under both training conditions, suggesting that curriculum learning is not strictly necessary for category\-dependent routing to develop\.

Table 5:Pooled bilingual routing\-category MI under the no\-curriculum condition\.The following table shows language\-wise MI values computed separately for English and German\.

Table 6:Language\-wise routing\-category mutual information for the forward curriculum and no\-curriculum settings\.To verify that the observed category\-sensitive routing is not reducible to language identity alone, we additionally compute conditional mutual informationI⁡\(R;C∣L\)I\(R;C\\mid L\), whereLLdenotes language identity\. This analysis measures whether routing still contains information about lexical, grammatical, and syntactic category membership after conditioning on whether the probe is English or German\. As reported in Appendix Table[12](https://arxiv.org/html/2608.15102#A7.T12), conditional mutual information remains non\-zero across routed layers in both training conditions, indicating that the routing\-category relationship is not explained solely by English\-German separation\.

Table[6](https://arxiv.org/html/2608.15102#S5.T6)shows the language\-wise mutual information for each routed layer under both the forward curriculum and no\-curriculum settings\. The language\-wise MI values are drastically different for English\. In particular, at layer 5, the MI value increases from 0\.1025 in the forward curriculum setting to 0\.6457 in the no\-curriculum setting, representing an increase of approximately 6\.3 times\. A similar trend can be observed across all routed layers\.

In contrast, the difference between the category\-dependent routing effect of the two setups is much less visible in German, where MI remains around 0\.1 at layers 1 and 5 for both setups \(0\.1444 in the forward curriculum setting and 0\.1116 in the no\-curriculum setting at layer 5 for German language\)\. A similar effect is observed in the other layers, where the differences remain comparatively small\. A paired bootstrap over the identical 12,000 validation records \(2,000 resamples\) finds this German layer\-5 difference statistically reliable \(Δ​MI=\+0\.0328\\Delta\\mathrm\{MI\}=\+0\.0328, 95% CI\[\+0\.0211,\+0\.0446\]\[\+0\.0211,\+0\.0446\],p=0\.001p=0\.001\), while English MI is significantly higher without the curriculum at every layer\.

##### Seed replication\.

Because the results above come from a single run per condition, we replicated both conditions at a second seed \(full values in Appendix[G](https://arxiv.org/html/2608.15102#A7), Table[10](https://arxiv.org/html/2608.15102#A7.T10)\)\. The structural findings reproduce \(middle\-layer concentration, final\-layer collapse, higher pooled MI without the curriculum at every layer\), but the per\-language composition does not: at the second seed the no\-curriculum model concentrates on German rather than English \(DE layer\-5 MI 0\.213 vs\. EN 0\.078\), the reverse of the seed\-42 pattern, so*which*language dominates is seed\-dependent and the per\-language contrasts above should not be generalised across runs\. What is robust across both seeds is the asymmetry itself and its remedy: the no\-curriculum condition is dominated by a single language \(\|\|EN−\-DE\|\|MI at layer 5: 0\.53 and 0\.13\), whereas the forward curriculum yields a balanced profile \(0\.04 and 0\.08\) and a more stable pooled MI \(0\.115 and 0\.089, versus 0\.260 and 0\.113 without the curriculum\)\.

Curriculum therefore does not uniformly increase specialisation or consistently favour either language: uncurriculated training acts as an initialization lottery handing specialisation to one language, and staged L1–L2 exposure converts this into a stable, language\-balanced profile – a reduction in single\-language dominance rather than a language\-specific improvement\.

## 6Discussion

Routing distributions provide significant insight into linguistic category membership, and specialisation appears in both training conditions, so curriculum learning is not necessary for category\-sensitive routing to emerge; the open question is how curriculum changes its distribution across languages and layers\. The strongest routing\-category relationship occurs in intermediate layers, particularly layer 5, consistent with prior work suggesting intermediate transformer layers carry clearer linguistic structure, while the final routed layer shows lower mutual information and higher entropy, implying routing becomes more evenly spread across experts rather than category\-separated\.

The comparison between conditions is where pooled statistics can mislead: combining English and German into one score makes the no\-curriculum model look stronger, but per\-language analysis shows this pooled signal is a single\-language phenomenon whose beneficiary is decided by the random seed, whereas the forward curriculum yields a language\-balanced profile that is stable across seeds\. Sequential L1–L2 exposure therefore does not simply boost or suppress specialisation; its seed\-robust effect is to make the cross\-lingual organization of specialisation predictable, i\.e\., to reduce single\-language dominance rather than to redistribute specialisation toward a particular language\.

Interpreting the behavioural accuracy values requires considering the study design: the model is intentionally small and the validation set uses controlled minimal\-pair contrasts, so the aim is not high benchmark accuracy but comparable internal routing patterns across probe families; the behavioural results mainly indicate both models are sufficiently functional for meaningful routing analysis\.

## 7Conclusion

Bilingual MoE routing develops measurable category\-dependent organization across lexical, grammatical, and syntactic probe families\. The strongest routing specialisation is observed in intermediate routed layers and is robust on held\-out validation data and across two training seeds\. Curriculum learning does not uniformly increase specialisation; its seed\-robust effect is to reduce single\-language dominance rather than to favour a specific language\. Without a curriculum, routing specialisation concentrates on a single language whose identity varies with the seed; staged L1–L2 exposure converts this initialization lottery into a stable, language\-balanced bilingual specialisation profile\.

## Limitations

This study uses a single EN–DE language pair and a single primary sparse architecture\. While the observed routing patterns are consistent across multiple analyses, including mutual information, entropy, Jensen–Shannon distance, permutation testing, and conditional mutual information controls, the extent to which the findings generalize to other language families, larger\-scale MoE architectures, or different curriculum schedules remains an open question\. The main conditions are replicated at two training seeds; per\-language MI magnitudes vary across seeds, and we accordingly claim only the seed\-stable structure \(middle\-layer concentration, final\-layer collapse, and the curriculum’s language\-balancing effect\), not per\-language magnitudes\.

The diagnostic validation set is generated through a controlled LLM\-assisted pipeline and subsequently validated using parser\-based checks and deterministic probe construction rules\. Although these procedures improve consistency and coverage, the evaluation set does not undergo manual human validation\. Future work could incorporate expert human review and naturally occurring linguistic examples to further verify that the observed routing patterns generalize beyond synthetic diagnostic probes\. The validation set is dominated by single\-piece probes under the mBERT tokenizer, but full subword\-aggregation robustness was not recomputed for the final archived routing logs used in this draft

The present study uses distinct source corpora for English and German\. Although language\-conditioned analyses remain significant, future work should examine matched\-domain multilingual corpora\.

A reverse curriculum is not included because the forward curriculum is explicitly designed around a fixed cumulative English\-German exposure ratio\. Simply reversing the schedule would alter not only the temporal order of language presentation but also the total exposure received by each language across training\. Consequently, a naive DE→EN reversal would confound sequencing effects with differences in cumulative language exposure\. A fair reverse\-curriculum comparison would therefore require a separately constructed schedule that preserves overall language proportions while reversing the order of introduction\.

Several potential confounds were explicitly examined\. Routing\-category associations remained stable across alternative subword aggregation strategies and remained non\-zero after conditioning on language identity and coarse lexical frequency controls\. Nevertheless, the present study does not fully disentangle all possible interactions between linguistic category, token identity, lexical frequency, morphological complexity, and part\-of\-speech information\. As a result, the reported routing effects should be interpreted as category\-sensitive routing behaviour rather than evidence of perfectly isolated category\-specific mechanisms\.

A stricter frequency\-balanced lexical control was explored but was not included in the final analysis\. The curriculum\-generated lexical distributions were highly skewed, making it difficult to construct well\-matched irregular and regular subsets while maintaining sufficient probe coverage and sample diversity\.

We do not yet report extensive architectural sweeps, multiple language pairs, or broad\-scale hyperparameter sensitivity analyses\. In addition, expert intervention experiments \(e\.g\., expert masking or routing interventions\) are not included, limiting our ability to make strong causal claims about the functional role of individual experts\.

Finally, the work focuses specifically on sparse MoE architectures because expert\-routing distributions constitute the primary object of analysis\. Dense transformers do not expose an explicit routing mechanism, making direct comparisons of routing specialisation impossible\. While dense baselines remain useful for behavioural benchmarking, they cannot provide the routing\-level signals studied in this work\.

While the present study focuses on identifying category\-dependent routing behaviour, future work could perform expert masking or routing interventions to determine whether the identified expert preferences play a causal role in linguistic processing\.

## References

- Bengioet al\.\(2009\)Y\. Bengio, J\. Louradour, R\. Collobert, and J\. WestonCurriculum learning\.InProceedings of the 26th International Conference on Machine Learning,pp\. 41–48\.External Links:[Link](https://icml.cc/2009/papers/119.pdf)Cited by:[§1](https://arxiv.org/html/2608.15102#S1.p3.1),[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px5.p1.1)\.
- Clarket al\.\(2019\)K\. Clark, U\. Khandelwal, O\. Levy, and C\. D\. ManningWhat does BERT look at? an analysis of BERT’s attention\.Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,pp\. 276–286\.External Links:[Document](https://dx.doi.org/10.18653/v1/W19-4828),[Link](https://aclanthology.org/W19-4828/)Cited by:[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px3.p1.1)\.
- Cover and Thomas \(2006\)T\. M\. Cover and J\. A\. ThomasElements of information theory\.2 edition,Wiley\.Cited by:[§1](https://arxiv.org/html/2608.15102#S1.p4.1)\.
- Daiet al\.\(2024\)D\. Dai, C\. Deng, C\. Zhao, R\. X\. Xu, H\. Gao, D\. Chen, J\. Li, W\. Zeng, X\. Yu, Y\. Wu, Z\. Xie, Y\. K\. Li, P\. Huang, F\. Luo, C\. Ruan, Z\. Sui, and W\. LiangDeepSeekMoE: towards ultimate expert specialization in mixture\-of\-experts language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1280–1297\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.70),[Link](https://aclanthology.org/2024.acl-long.70/)Cited by:[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px4.p1.1)\.
- Elhageet al\.\(2021\)N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly, N\. DasSarma, D\. Drain, D\. Ganguli, Z\. Hatfield\-Dodds, D\. Hernandez, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, T\. Brown, J\. Clark, J\. Kaplan, S\. McCandlish, and C\. OlahA mathematical framework for transformer circuits\.Transformer Circuits Thread\.External Links:[Link](https://transformer-circuits.pub/2021/framework/)Cited by:[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px3.p1.1)\.
- Endres and Schindelin \(2003\)D\. M\. Endres and J\. E\. SchindelinA new metric for probability distributions\.IEEE Transactions on Information Theory49\(7\),pp\. 1858–1860\.Cited by:[§3\.0\.2](https://arxiv.org/html/2608.15102#S3.SS0.SSS2.p5.1)\.
- Falkeet al\.\(2026\)T\. Falke, N\. Anastassacos, S\. Tan, C\. R\. Meas, C\. S\. Prakash, N\. Sekhar, M\. S\. Bari, K\. Kompella, and G\. F\. ElsayedMoE routing testbed: studying expert specialization and routing behavior at small scale\.External Links:2604\.07030,[Link](https://arxiv.org/abs/2604.07030)Cited by:[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px4.p1.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research23\(120\),pp\. 1–39\.External Links:[Link](https://jmlr.org/papers/v23/21-0998.html)Cited by:[§1](https://arxiv.org/html/2608.15102#S1.p1.1),[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px4.p1.1)\.
- Liet al\.\(2024\)Z\. Li, H\. Lin, Y\. Lu, H\. Xiang, X\. Han, and L\. SunMeta\-cognitive analysis: evaluating declarative and procedural knowledge in datasets and large language models\.InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\),pp\. 11222–11228\.External Links:[Link](https://aclanthology.org/2024.lrec-coling-main.978/)Cited by:[§1](https://arxiv.org/html/2608.15102#S1.p4.1),[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px2.p1.1)\.
- Lin \(1991\)J\. LinDivergence measures based on the shannon entropy\.IEEE Transactions on Information Theory37\(1\),pp\. 145–151\.External Links:[Document](https://dx.doi.org/10.1109/18.61115)Cited by:[§1](https://arxiv.org/html/2608.15102#S1.p4.1)\.
- Penedoet al\.\(2024\)G\. Penedo, H\. Kydlíček, L\. Ben Allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. von Werra, and T\. WolfThe fineweb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=n6SCkn2QaG)Cited by:[§4\.1](https://arxiv.org/html/2608.15102#S4.SS1.p1.1)\.
- Platanioset al\.\(2019\)E\. A\. Platanios, O\. Stretcu, G\. Neubig, B\. Póczos, and T\. MitchellCompetence\-based curriculum learning for neural machine translation\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1,pp\. 1162–1172\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1119),[Link](https://aclanthology.org/N19-1119/)Cited by:[§1](https://arxiv.org/html/2608.15102#S1.p3.1),[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px5.p1.1)\.
- Shannon \(1948\)C\. E\. ShannonA mathematical theory of communication\.Bell System Technical Journal27\(3\),pp\. 379–423\.External Links:[Document](https://dx.doi.org/10.1002/j.1538-7305.1948.tb01338.x)Cited by:[§1](https://arxiv.org/html/2608.15102#S1.p4.1)\.
- Shazeeret al\.\(2017\)N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. DeanOutrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InProceedings of the International Conference on Learning Representations,External Links:[Link](https://research.google/pubs/pub45929/)Cited by:[§1](https://arxiv.org/html/2608.15102#S1.p1.1),[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px4.p1.1)\.
- Sunet al\.\(2026\)H\. Sun, Y\. Liu, Y\. Wu, and L\. SunExpert threshold routing for autoregressive language modeling with dynamic computation allocation and load balancing\.External Links:2603\.11535,[Link](https://arxiv.org/abs/2603.11535)Cited by:[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px4.p1.1)\.
- Tenneyet al\.\(2019\)I\. Tenney, D\. Das, and E\. PavlickBERT rediscovers the classical NLP pipeline\.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,pp\. 4593–4601\.External Links:[Document](https://dx.doi.org/10.18653/v1/P19-1452),[Link](https://aclanthology.org/P19-1452/)Cited by:[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px3.p1.1)\.
- Ullman \(2001a\)M\. T\. UllmanA neurocognitive perspective on language: the declarative/procedural model\.Nature Reviews Neuroscience2\(10\),pp\. 717–726\.External Links:[Document](https://dx.doi.org/10.1038/35094573)Cited by:[§1](https://arxiv.org/html/2608.15102#S1.p2.1),[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px1.p1.1)\.
- Ullman \(2001b\)M\. T\. UllmanThe neural basis of lexicon and grammar in first and second language: the declarative/procedural model\.Bilingualism: Language and Cognition4\(2\),pp\. 105–122\.External Links:[Document](https://dx.doi.org/10.1017/S1366728901000220)Cited by:[§1](https://arxiv.org/html/2608.15102#S1.p2.1),[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px1.p1.1)\.
- Ullman \(2020\)M\. T\. UllmanThe declarative/procedural model\.InTheories in Second Language Acquisition: An Introduction,B\. VanPatten, J\. Williams, G\. D\. Keating, and S\. Wulff \(Eds\.\),pp\. 128–161\.External Links:[Document](https://dx.doi.org/10.4324/9780429503986-7)Cited by:[§1](https://arxiv.org/html/2608.15102#S1.p2.1),[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px1.p1.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems 30,pp\. 5998–6008\.Cited by:[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px3.p1.1)\.
- Xueet al\.\(2021\)L\. Xue, N\. Constant, A\. Roberts, M\. Kale, R\. Al\-Rfou, A\. Siddhant, A\. Barua, and C\. RaffelMT5: a massively multilingual pre\-trained text\-to\-text transformer\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,Online,pp\. 483–498\.External Links:[Link](https://aclanthology.org/2021.naacl-main.41/),[Document](https://dx.doi.org/10.18653/v1/2021.naacl-main.41)Cited by:[§4\.1](https://arxiv.org/html/2608.15102#S4.SS1.p1.1)\.
- Zhanget al\.\(2019\)X\. Zhang, P\. Shapiro, G\. Kumar, P\. McNamee, M\. Carpuat, and K\. DuhCurriculum learning for domain adaptation in neural machine translation\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1,pp\. 1903–1915\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1189),[Link](https://aclanthology.org/N19-1189/)Cited by:[§1](https://arxiv.org/html/2608.15102#S1.p3.1),[§2](https://arxiv.org/html/2608.15102#S2.SS0.SSS0.Px5.p1.1)\.

## Appendix AProbe Token Taxonomy

### English lexical probes

A tokenτ\\tauis classified as English lexical if either condition holds:

1. 1\.Irregular past\-tense verb\.upos​\(τ\)=VERB\\textsc\{upos\}\(\\tau\)=\\texttt\{VERB\},Tense=Past∈feats​\(τ\)\\texttt\{Tense=Past\}\\in\\textsc\{feats\}\(\\tau\),VerbForm=Fin∈feats​\(τ\)\\texttt\{VerbForm=Fin\}\\in\\textsc\{feats\}\(\\tau\), andlemma​\(τ\)∈𝒱irrEN\\textsc\{lemma\}\(\\tau\)\\in\\mathcal\{V\}^\{\\text\{EN\}\}\_\{\\text\{irr\}\}\.
2. 2\.Irregular plural noun\.upos​\(τ\)=NOUN\\textsc\{upos\}\(\\tau\)=\\texttt\{NOUN\}andtext​\(τ\)∈𝒩irrEN\\textsc\{text\}\(\\tau\)\\in\\mathcal\{N\}^\{\\text\{EN\}\}\_\{\\text\{irr\}\}\.

### German lexical probes

German lexical probes are operationalized as item\-specific or lexically exceptional forms:

1. 1\.Strong past\-tense verbs\.upos​\(τ\)=VERB\\textsc\{upos\}\(\\tau\)=\\texttt\{VERB\},Tense=Past∈feats​\(τ\)\\texttt\{Tense=Past\}\\in\\textsc\{feats\}\(\\tau\),VerbForm=Fin∈feats​\(τ\)\\texttt\{VerbForm=Fin\}\\in\\textsc\{feats\}\(\\tau\), andlemma​\(τ\)∈𝒱irrDE\\textsc\{lemma\}\(\\tau\)\\in\\mathcal\{V\}^\{\\text\{DE\}\}\_\{\\text\{irr\}\}\.
2. 2\.Suppletive or highly irregular finite paradigm forms\.text​\(τ\)∈𝒮DE\\textsc\{text\}\(\\tau\)\\in\\mathcal\{S\}\_\{\\text\{DE\}\}andupos​\(τ\)∈\{VERB,AUX\}\\textsc\{upos\}\(\\tau\)\\in\\\{\\texttt\{VERB\},\\texttt\{AUX\}\\\}\.
3. 3\.Irregular plural nouns\.upos​\(τ\)=NOUN\\textsc\{upos\}\(\\tau\)=\\texttt\{NOUN\},Number=Plur∈feats​\(τ\)\\texttt\{Number=Plur\}\\in\\textsc\{feats\}\(\\tau\), andtext​\(τ\)∈𝒩irrDE\\textsc\{text\}\(\\tau\)\\in\\mathcal\{N\}^\{\\text\{DE\}\}\_\{\\text\{irr\}\}\.
4. 4\.Gender\-marked determiners\.upos​\(τ\)=DET\\textsc\{upos\}\(\\tau\)=\\texttt\{DET\}with nominative singular definite or indefinite gender\-marked forms\. These probes are grouped with lexical probes because they depend on lexically specified noun\-gender distinctions, even though they also involve morphosyntactic marking\.

### English grammatical probes

1. 1\.Regular past\-tense verb\.upos​\(τ\)=VERB\\textsc\{upos\}\(\\tau\)=\\texttt\{VERB\},Tense=Past∈feats​\(τ\)\\texttt\{Tense=Past\}\\in\\textsc\{feats\}\(\\tau\),VerbForm=Fin∈feats​\(τ\)\\texttt\{VerbForm=Fin\}\\in\\textsc\{feats\}\(\\tau\), andlemma​\(τ\)∉𝒱irrEN\\textsc\{lemma\}\(\\tau\)\\notin\\mathcal\{V\}^\{\\text\{EN\}\}\_\{\\text\{irr\}\}\.
2. 2\.Auxiliary agreement\.upos​\(τ\)=AUX\\textsc\{upos\}\(\\tau\)=\\texttt\{AUX\}with number\-marked auxiliary agreement\.

### German grammatical probes

1. 1\.Weak past\-tense verb\.upos​\(τ\)=VERB\\textsc\{upos\}\(\\tau\)=\\texttt\{VERB\},Tense=Past∈feats​\(τ\)\\texttt\{Tense=Past\}\\in\\textsc\{feats\}\(\\tau\),VerbForm=Fin∈feats​\(τ\)\\texttt\{VerbForm=Fin\}\\in\\textsc\{feats\}\(\\tau\),lemma​\(τ\)∉𝒱irrDE\\textsc\{lemma\}\(\\tau\)\\notin\\mathcal\{V\}^\{\\text\{DE\}\}\_\{\\text\{irr\}\}, and the form is not part of the suppletive set\.
2. 2\.Auxiliary agreement\.upos​\(τ\)=AUX\\textsc\{upos\}\(\\tau\)=\\texttt\{AUX\}with number\-marked auxiliary agreement\.

### Syntactic probes

1. 1\.upos​\(τ\)=SCONJ\\textsc\{upos\}\(\\tau\)=\\texttt\{SCONJ\}, or
2. 2\.deprel​\(τ\)∈\{advcl,ccomp,xcomp,acl,csubj\}\\textsc\{deprel\}\(\\tau\)\\in\\\{\\texttt\{advcl\},\\texttt\{ccomp\},\\texttt\{xcomp\},\\texttt\{acl\},\\texttt\{csubj\}\\\}\.

These labels should be interpreted as operational probe families rather than perfectly discrete theoretical classes\. Their purpose is to test whether MoE routing is sensitive to broad linguistically motivated contrasts under a consistent parser\-based annotation scheme\.

## Appendix BFull Curriculum Schedule

### Category sampling

Within every epoch, lexical, grammatical, and syntactic sentences are sampled uniformly and shuffled randomly\.

### Forward curriculum

Table 7:Forward curriculum L2 exposure schedule\.

## Appendix CHyperparameter Details

### Computing Infrastructure and Budget

All training, validation, and routing\-analysis experiments were conducted on NVIDIA T4 GPUs and NVIDIA RTX A6000 GPUs\. The primary bilingual MoE model used in the study contains 86\.4M parameters\. Each training condition was run for 16 epochs with 300,000 sentence instances per epoch, corresponding to 4\.8M sentence instances per condition\. Validation, routing extraction, and downstream statistical analyses were performed from saved checkpoints and held\-out routing logs on the same hardware class\. Depending on hardware availability, experiments were conducted on either NVIDIA T4 or NVIDIA RTX A6000 GPUs\. Training time therefore varied across runs, ranging from approximately 14–28 GPU\-hours per condition

## Appendix DAdditional Routing Analysis Details

### Permutation testing

Category labels are shuffled at the sentence level for 1,000 permutations per layer\. Empiricalpp\-values are computed as the fraction of permutations withI⁡\(R,C\)≥IobservedI\(R;C\)\\geq I\_\{\\text\{observed\}\}\. This tests whether the observed routing\-category dependence is larger than would be expected under category\-independent routing\.

## Appendix EOpenAI Generation Details, Validation Parsing, and Prompt Examples

The held\-out validation sentences were generated with the OpenAI Responses API using the Python OpenAI client\. In the generation script, the default model wasGPT\-5\.1, configurable through theOPENAI\_MODELenvironment variable\. Requests were made withmax\_output\_tokens=420,reasoning\.effort="none",store=false, request timeout40s, andmax\_retries=0\. The API call also included a fixed instruction string:“Follow the output format exactly\. Return only the requested lines and nothing else\.”

Generated candidates were then automatically parsed and validated with Stanza using the processorstokenize,pos,lemma, anddepparse\. For each generatedS\+S^\{\+\}sentence, the pipeline checked that the marked probe token occurred exactly once, that the output matched the required subtype inventory, and that the parsed probe token satisfied the subtype\-specific constraints used in dataset construction \(e\.g\., finite past\-tense verb, plural noun, auxiliary, or subordinating conjunction, depending on the subtype\)\. Additional filters removed malformed outputs, duplicates, and subtype violations before acceptance into the held\-out set\.

For the final validation pair construction, the ungrammatical sentenceS−S^\{\-\}was obtained deterministically from the validatedS\+S^\{\+\}sentence by editing only the probe token or deleting the probe in the syntactic conjunction cases\. The transformation rule depended on subtype\. For example, English irregular past forms were replaced with regularized forms, English regular past\-tense verbs were replaced with their lemma/base form, auxiliary agreement probes were swapped to mismatching number forms, German strong verbs were weak\-regularized, German suppletive forms were replaced with present/agreement\-incompatible alternatives, and subordinating conjunction probes were removed to create a syntactically degraded variant\. This yielded minimal\-pair contrasts in whichS\+S^\{\+\}remained parser\-validated andS−S^\{\-\}differed by a controlled probe\-level manipulation\.

Below we provide two illustrative prompt examples adapted directly from the generation templates used in the pipeline\.

##### Prompt Example 1: English irregular verb probe\.

> Write exactly 10 EN sentences\. Subtype: irregular\_verb Preferred lemmas: become, draw, awake, speak, throw, choose, write, drive, sing Avoid recent probes: none Rules: \- Output exactly 10 lines, no more and no fewer\. \- One sentence per line\. \- 6–12 words per sentence\. \- Prefer a different listed lemma on each line\. \- Mark target as \[PROBE: word\]\. \- Prefer the correct irregular simple\-past form of one listed lemma\. \- Probe must be the main verb\. \- Use simple past only\. \- No auxiliaries or participles with the probe\. \- Use the real irregular past form, not the base form\. \- Examples: become→\\rightarrowbecame, awake→\\rightarrowawoke, draw→\\rightarrowdrew\. \- Prefer the listed lemmas strongly\. \- If one listed lemma feels awkward, use another listed lemma\. \- No numbering, bullets, quotes, parentheses, or extra text\. \- Bare sentence text only\.

##### Prompt Example 2: German subordinating conjunction probe\.

> Write exactly 10 DE sentences\. Subtype: sconj Preferred subordinating conjunctions: weil, obwohl, bevor, nachdem, falls, sobald, damit, dass Avoid recent probes: none Rules: \- Output exactly 10 lines, no more and no fewer\. \- One sentence per line\. \- 6–12 words per sentence\. \- Prefer a different listed conjunction on each line\. \- Mark target as \[PROBE: word\]\. \- Prefer one listed subordinating conjunction exactly as shown\. \- Use the probe as a true subordinating conjunction with UPOS=SCONJ\. \- Build a clear subordinate\-clause construction\. \- Keep the sentence fully grammatical as written\. \- Avoid coordinators, adverbs, or discourse markers\. \- No numbering, bullets, quotes, parentheses, or extra text\. \- Bare sentence text only\.

### Parsing and Validation Tools

Parser\-based probe annotation and validation were performed with Stanza\. We used the Stanza pipelines for English and German with the processorstokenize,pos,lemma, anddepparse\. These pipeline outputs were used to identify probe tokens, verify subtype constraints, and validate generated held\-out examples\.

## Appendix FDataset Diversity Statistics

To address concerns regarding dataset health and template collapse, we provide diversity statistics for the diagnostic set\. All reported probe sets maintain 100% sentence\-pair uniqueness\.

Table 8:Illustrative diagnostic probe diversity statistics\.
## Appendix GAblation Controls

The appendix currently includes four additional controls beyond the main forward and no\-curriculum MoE comparisons: random routing, frozen routing, top\-kk, and load\-balancing \(α=0\.005\\alpha=0\.005\) ablations under the forward curriculum\. These controls are intended to distinguish curriculum effects from generic sparse\-routing effects and to test how sensitive the observed specialisation is to router flexibility and routing sparsity\.

Table 9:Routing\-control and sparsity ablations evaluated on the held\-out validation set \(MI@L5\)\. Random routing collapses routing\-category dependence toward near\-null values, indicating that specialisation does not arise from architectural sparsity alone\. Freezing router parameters preserves weaker but non\-zero specialisation\. Lower load balancing increases routing\-category dependence, while the routing\-sparsity ablations show that top\-k=2k=2yields a similar but slightly lower pooled MI than the main model, whereas top\-k=4k=4produces substantially stronger category\-conditioned routing, especially for English probes\.The random\-routing condition replaces learned router assignments with uniformly sampled expert selection while preserving the underlying expert parameters\. As expected, routing\-category mutual information collapses to near\-zero values, indicating that the observed specialisation patterns require adaptive routing rather than arising from architectural sparsity alone\.

The frozen\-routing condition initializes routing normally but prevents subsequent router optimization during training\. Although specialisation is substantially weaker than in the fully trainable model, non\-trivial routing\-category dependence remains, suggesting that expert differentiation can emerge through changes in token representations interacting with fixed routing boundaries\.

Load\-balancing strength additionally exerts a strong influence on specialisation structure\. Reducing the auxiliary load\-balancing coefficient toα=0\.005\\alpha=0\.005markedly increases routing\-category dependence, indicating that weaker balancing constraints permit stronger expert partitioning\. However, this increase coincides with increasingly skewed expert utilisation, suggesting a trade\-off between specialisation strength and balanced expert participation\.

Routing sparsity also affects specialisation, but not in the initially expected direction\. Reducing expert selection from top\-k=3k=3to top\-k=2k=2leaves pooled routing\-category dependence at a similar level, with a slight decrease overall\. In contrast, increasing routing breadth to top\-k=4k=4yields a substantially stronger routing\-category association, especially for English probes\. In the present runs, broader routing therefore coincides with stronger category\-conditioned separation rather than weaker specialisation\.

### G\.1Seed Replication Detail

Table[10](https://arxiv.org/html/2608.15102#A7.T10)gives the full per\-seed, per\-language breakdown underlying the seed\-replication discussion in Section 5\.6\. The pooled structure \(no\-curriculum\>\>forward at layer 5\) and the curriculum’s smaller cross\-language asymmetry reproduce across both seeds; the identity of the language that dominates under the no\-curriculum condition does not\.

Table 10:Layer\-5 routing\-category MI at both training seeds\.
### G\.2Frequency\-Binned Lexical Analysis

To examine whether lexical routing specialisation is driven primarily by token frequency, English lexical probes were grouped into frequency bins using curriculum\-weighted lemma frequencies\. Mutual information was then recomputed using only lexical irregular\-versus\-regular contrasts within each frequency bin\.

Table 11:English lexical irregular\-versus\-regular routing mutual information at layer 5 after binning probe lemmas by curriculum\-weighted frequency\.Mutual information remains non\-zero across all frequency bins in both training conditions\. Notably, the strongest routing\-category association is observed for rare lexical items rather than the most frequent items\. This suggests that the lexical routing effect cannot be explained solely by the highest\-frequency lexical forms\.

Table 12:Conditional mutual information between routing and linguistic category given language,I⁡\(R;C∣L\)I\(R;C\\mid L\), across routed layers\. In both the forward\-curriculum and no\-curriculum conditions, conditional MI remains clearly non\-zero and permutation\-significant at all routed layers \(p=0\.000999p=0\.000999\), indicating that category\-sensitive routing is not reducible to language separation alone\.We additionally examined the evolution of routing specialisation during training using epoch\-wise routing logs\. These analyses are provided here as supplementary evidence\.

### G\.3Routing Specialisation Dynamics During Training

To analyse routing specialisation throughout training, mutual information and entropy were computed from the routing logs at checkpoints from each training epoch\. Figure[5](https://arxiv.org/html/2608.15102#A7.F5)shows that category\-dependent routing emerges rapidly during the early stages of training and stabilises over the following epochs\.

![Refer to caption](https://arxiv.org/html/2608.15102v1/overall_mi_over_epochs.png)Figure 5:Evolution of routing\-category mutual information across training epochs for the routed layers\.The relative ordering of the routed layers remains largely consistent throughout training, with layer 5 exhibiting the highest mutual information across all epochs\. The figure also shows a decline in category\-dependent routing after epoch 8\. Following this transition, the mutual information decreases across all routed layers before stabilising again during the later stages of training\.

![Refer to caption](https://arxiv.org/html/2608.15102v1/overall_entropy_over_epochs.png)Figure 6:Evolution of routing entropy across training epochs for the routed layers\.Figure[6](https://arxiv.org/html/2608.15102#A7.F6)presents the routing entropy across the routed layers throughout training\. A complementary trend is observed between mutual information and entropy\. As mutual information decreases after epoch 8, routing entropy increases across all routed layers during the same period\. This indicates that expert utilisation becomes progressively more distributed while category\-dependent routing behaviour remains present\.