面向专家专业化与路由的博弈论数据分配

arXiv cs.CL 论文

摘要

该论文提出了 InflEns,这是一种 LoRA 专家混合模型,其训练权重和推理路由均源自固定提示特征空间中的同一个常和选址博弈,无需单独训练门控网络。在七项安全基准上的评估表明,该方法的表现与学习得到的门控网络相当,能在约 90% 的提示上正确选择最佳专家,还能将有害行为追溯到单个专家并加以修正,将 HarmBench 上的攻击成功率从 0.51 降至 0.02。

arXiv:2611.25167v1 Announce Type: cross Abstract: Can the allocation used to train experts also predict which expert will perform best on a prompt? Such a connection would make the intended division of labor inspectable before training and allow the allocation itself to guide inference. We introduce InflEns, a mixture of LoRA experts whose training weights and inference routes come from the same location game in a fixed prompt-trait space. Players choose positions to maximize their expected shares of the prompt distribution, without using expert losses. We utilize the game's kernel-general theoretical criterion for when co-located positions become locally unstable to separation. We evaluate InflEns on a corpus assembled from seven safety-related benchmarks, which demonstrates that the highest-share expert has the lowest held-out response negative log-likelihood (NLL) on approximately 90% of prompts across four kernels under a sharpened simplex representation, compared with approximately 15% after permuting allocations across prompts. We demonstrate that the method performs comparably to published learned gates without having to train an explicit gate on the experts. Because the router is fixed, we can trace harmful behavior to a single expert and retrain only that expert, which lowers HarmBench attack success from 0.51 to 0.02 while keeping instruction following. The game also gives us several controls over how the experts specialize, including the kernel choice and its parameters, the trait-space geometry, and the number of experts. Importantly, InflEns is a general framework that can be extended to any encoder and data distribution.
查看原文
查看缓存全文

缓存时间: 2026/11/20 09:07

# Game-Theoretic Data Allocation for Expert Specialization and Routing
Source: [https://arxiv.org/html/2611.25167](https://arxiv.org/html/2611.25167)
Denis LimAffiliation:Independent ResearcherSoroush VosoughiAffiliation:Dartmouth College

###### Abstract

Can the allocation used to train experts also predict which expert will perform best on a prompt? Such a connection would make the intended division of labor inspectable before training and allow the allocation itself to guide inference\. We introduceInflEns, a mixture of LoRA experts whose training weights and inference routes come from the same location game in a fixed prompt\-trait space\. Players choose positions to maximize their expected shares of the prompt distribution, without using expert losses\. We utilize the game’s kernel\-general theoretical criterion for when co\-located positions become locally unstable to separation\. We evaluateInflEnson a corpus assembled from seven safety\-related benchmarks, which demonstrates that the highest\-share expert has the lowest held\-out response negative log\-likelihood \(NLL\) on approximately 90% of prompts across four kernels under a sharpened simplex representation, compared with approximately 15% after permuting allocations across prompts\. We demonstrate that the method performs comparably to published learned gates without having to train an explicit gate on the experts\. Because the router is fixed, we can trace harmful behavior to a single expert and retrain only that expert, which lowers HarmBench attack success from 0\.51 to 0\.02 while keeping instruction following\. The game also gives us several controls over how the experts specialize, including the kernel choice and its parameters, the trait\-space geometry, and the number of experts\. Importantly,InflEnsis a general framework that can be extended to any encoder and data distribution\.

## 1Introduction

In many mixture\-of\-experts systems, expert specialization develops as the gate and experts are optimized together\([Shazeer et al\., 2017](https://arxiv.org/html/2611.25167#bib.bib50)\)\. We ask whether this division of labor can be specified through a training allocation that also predicts where each expert will be useful\. An explicit allocation would let a designer inspect which experts receive training signal from particular kinds of prompts, such as privacy\-sensitive requests or attempted jailbreaks, before training the adapters\. It could also supply a rule for selecting among the trained experts after training has been conducted\.

We introduceInflEns, a method for training experts in a trait space using game dynamics, and we demonstrateInflEns’s applicability with a mixture of LoRA experts\([Hu et al\., 2022](https://arxiv.org/html/2611.25167#bib.bib18)\)\. To start,InflEnsneeds to have a fixed representation that maps data, prompts in this case, to trait vectors, which in turn builds a resource space that players can compete over\. With the resource representation in hand, players choose positions in the resource trait space, and an influence kernel centered on the players’ positions determines the share of each data point’s \(here, each prompt’s or prompt response pair’s\) learning signal that a player receives relative to its competitors\. Each player aims to maximize its expected share of the training signal\. Note that this differs from methods trained on loss, because the players are not explicitly trained to minimize their loss, only to maximize how much training signal they receive\. Because the shares of each data point’s signal sum to one, the game is constant\-sum, so no data point’s signal goes to waste and one player’s gain comes at another’s expense\. In our example setting with prompts as data and LoRA adapters as players, the players’ shares weight their associated adapters’ training losses and determine their inference allocation\. Before we start training we solve the position game numerically, so that players start near equilibrium, but we show other starts as well\. We then allow positions to adapt through a gradient ascent algorithm with game\-utility gradients as the guiding gradient; this allows our system to move toward stable equilibria as more of the data distribution is revealed, and organizes the experts into predictable positions within their trait space\. Thus,InflEnsresults in the predictable routing of training signal to experts and leads to predictable emergence of experts and ultimately requires no training of a gate on losses\.

![Refer to caption](https://arxiv.org/html/2611.25167v1/fig_schema.png)Figure 1:InflEnsmakes the allocation of training data explicit through a location game, the influencer’s game, which requires a resource distribution for players to compete over\.aWe construct the resource distribution by encoding text data and projecting it onto trait axes \(Appendix[D](https://arxiv.org/html/2611.25167#A4)\), giving a density over the trait space\.bEach player holds a position in their trait space \(the same as the resource space in this work\), and an influence kernel centered on that position determines how much of each data point’s training signal the player receives\.cDuring training, each sampled batch is routed to the experts in proportion to their influence shares, the experts train on their weighted signal, and the positions are then updated by the game gradient before the next batch is sampled\.dAfter training, the experts have settled into positions in the trait space, and a new prompt is routed by encoding it and selecting the expert with the largest share, or a top\-kkmixture, without a learned gate\.Existing clustering\-based methods also use data geometry to organize expert training and inference\([Gururangan et al\., 2023](https://arxiv.org/html/2611.25167#bib.bib15);[Gou et al\., 2026](https://arxiv.org/html/2611.25167#bib.bib14)\)\. A distinguishing factor ofInflEnsis that positions follow from individual competitive payoffs derived from the influencer’s game framework\([Lovett & Fu, 2026](https://arxiv.org/html/2611.25167#bib.bib36)\)\. InInflEnswe utilize the stability properties of the influencer’s game, which provides a framework for analyzing and predicting agents’ behavior\. The influencer’s game provides a symmetric stability criterion that applies across log\-concave influence kernels and gives an explicit boundary determined by player count, kernel geometry, and prompt\-trait covariance\. The influencer’s game’s boundary provides us a reference scale for kernel reach \(the curvature of the kernel\) and identifies when co\-location becomes unstable and gives way to separation of players\. We then test whether the induced allocation predicts the relative response loss of the trained experts\.

In the setting of this paper,InflEnsis evaluated against other training methods on a corpus of prompts assembled from seven safety\-related benchmarks, from which we build a fixed, benchmark\-informed trait representation\. This provides a resource distribution on which players apply the influencer’s game’s dynamics\. For the prinicple experiment the trait map and resource distribution use the full prompt population, while test examples are excluded from adapter optimization and gate fitting\. The trait\-space representation and learend axises of the data corpus can vary, and we deomonstrate that the method is not limited to the specific trait\-space representation\.

We also run separate experiments to test whetherInflEnsimproves predictive performance\. In the principal seven\-adapter configuration, token\-level probability mixing improves NLL by 0\.105 nats over a pooled adapter, and with the trained experts held fixed, game\-weighted mixing outperforms uniform weighting, showing that the prompt\-dependent allocation contributes to the gain\. Across four data splits, the game allocation also achieves lower expected NLL than validation\-trained linear gates when sampling one expert, while completed\-sequence probability mixtures perform similarly\.

The game gives us a way to study expert organization through choices about training data, competitive positions, and influence, which makes the division of training data both inspectable and testable through its predictions about expert performance\.

### 1\.1Related work

#### Sparse mixtures and routing\.

Pre\-existing adaptive mixtures and sparse Transformer MoEs conventionally learn an input\-dependent gate jointly with their experts\([Jacobs et al\., 1991](https://arxiv.org/html/2611.25167#bib.bib20);[Shazeer et al\., 2017](https://arxiv.org/html/2611.25167#bib.bib50);[Fedus et al\., 2022](https://arxiv.org/html/2611.25167#bib.bib12)\)\. Alternative assignment schemes show that such joint learning is not always necessary: token hashing and stochastic expert selection can rival learned routers in the settings studied, whereas StableMoE learns a router in an initial stage and then freezes it to limit assignment drift\([Roller et al\., 2021](https://arxiv.org/html/2611.25167#bib.bib47);[Zuo et al\., 2022](https://arxiv.org/html/2611.25167#bib.bib68);[Dai et al\., 2022](https://arxiv.org/html/2611.25167#bib.bib11)\)\. Capacity balancing introduces a separate trade\-off; for instance, BASE guarantees equal per\-batch expert loads through constrained assignment\([Lewis et al\., 2021](https://arxiv.org/html/2611.25167#bib.bib31)\), potentially forcing certain tokens to be routed to sub\-optimal experts simply to satisfy the strict uniform distribution constraints\. Conversely, computing load\-balancing losses within each micro\-batch can spread domain\-specific sequences across experts and inhibit domain specialization\([Qiu et al\., 2025](https://arxiv.org/html/2611.25167#bib.bib46)\)\. Studies of existing trained MoE routers have found that what an expert specializes in depends on the method; for instance, OpenMoE routes mostly by token identity\([Xue et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib58)\), while Mixtral’s experts show little topical separation but Mixtral groups tokens by syntax and by position in the sequence\([Jiang et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib24)\), and OLMoE shows some specialization by domain and vocabulary\([Muennighoff et al\., 2025](https://arxiv.org/html/2611.25167#bib.bib39)\)\. A recent preprint further relates expert\-use similarity to hidden\-state geometry\([Wang et al\., 2026](https://arxiv.org/html/2611.25167#bib.bib54)\)\.InflEnsinstead defines prompt allocation through the influencer’s game in a trait space; thus, a key defining feature ofInflEnsis that its allocations depend on game\-derived positions rather than adapter performance\. A benefit ofInflEns’s game structure is that the initial allocation and the local stability of co\-located positions can be analyzed before fine\-tuning, and the experts’ coordinates determine their subsequent routes rather than a gate learned on their losses\.

#### Expert libraries, composition, and post\-hoc routing\.

Modular systems are another line of work, and they mainly differ in when specialization is decided and how the experts are recombined afterward\. BTM trains full\-model branches on domains defined by metadata and then combines the branches’ predictions using a domain posterior derived from their likelihoods\([Li et al\., 2022](https://arxiv.org/html/2611.25167#bib.bib33)\)\. c\-BTM builds on this by replacing the metadata with clusters of the training corpus, and the cluster centers are then reused at inference to weight each expert\([Gururangan et al\., 2023](https://arxiv.org/html/2611.25167#bib.bib15)\)\. BTX takes a different route, first training domain experts and then converting them into an MoE, where a token\-level gate is learned on top of them\([Sukhbaatar et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib52)\)\. At the scale of adapters, MoCLE follows a similar idea to c\-BTM, clustering instruction embeddings and activating LoRA experts based on how similar an input is to each cluster\([Gou et al\., 2026](https://arxiv.org/html/2611.25167#bib.bib14)\)\. A separate group of methods starts only after the adapters have been trained\. AdapterSoup averages the weights of a selected set of adapters\([Chronopoulou et al\., 2023](https://arxiv.org/html/2611.25167#bib.bib8)\), MoLE and PHATGOOSE learn a gate over the trained adapters\([Wu et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib57);[Muqeeth et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib40)\), Arrow builds routing prototypes directly from the adapter weights so that no further training is needed\([Ostapenko et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib45)\), and LoraRetriever learns which adapters to retrieve for a given input\([Zhao et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib67)\)\. At the level of whole models, RouteLLM learns from preference data which model should answer a prompt\([Ong et al\., 2025](https://arxiv.org/html/2611.25167#bib.bib44)\), and recent work compares routing, merging, and ensembling of parameter\-efficient experts head to head\([Lotfi et al\., 2026](https://arxiv.org/html/2611.25167#bib.bib35)\)\. Of these, c\-BTM and MoCLE are the closest toInflEns, since in both the geometry of the clusters shapes what each expert trains on and how it is used at inference\. The distinguishing factor ofInflEnsis that the positions organizing the experts come from the influencer’s game rather than from clustering, and because these positions are solved before any adapter is trained, their shares can route prompts without needing a gate fitted on the experts’ losses\.

#### Game\-theoretic and competitive mixtures\.

Game theory has also been applied to multi\-expert learning, and prior work mainly differs in who the players are and what they control\. In Nash\-MTL the players are tasks, which bargain over a shared parameter update\([Navon et al\., 2022](https://arxiv.org/html/2611.25167#bib.bib41)\), while mean\-field and congestion formulations instead treat tokens as the players, trading off routing quality against the load placed on each expert\([Sen, 2026](https://arxiv.org/html/2611.25167#bib.bib49);[Mouzouni, 2026](https://arxiv.org/html/2611.25167#bib.bib38)\)\. Strategic\-predictor models take yet another view, letting learners compete for the data points they can serve accurately\([Ben\-Porat & Tennenholtz, 2019](https://arxiv.org/html/2611.25167#bib.bib3)\)\. Within MoEs specifically, Mixture of Bidders routes batches through VCG auctions\([Vyas, 2025](https://arxiv.org/html/2611.25167#bib.bib53)\), NAMEx uses bargaining to set the weights for merging experts\([Nguyen et al\., 2026](https://arxiv.org/html/2611.25167#bib.bib42)\), and a gate\-free vision MoE uses each expert’s marginal contribution to decide how much it participates\([Agarap et al\., 2026](https://arxiv.org/html/2611.25167#bib.bib1)\)\. CompeteSMoE trains a router to reproduce the assignments of whichever experts respond most strongly\([Nguyen et al\., 2025](https://arxiv.org/html/2611.25167#bib.bib43)\), and a related model of softmax routing finds a pitchfork bifurcation, but in the load placed on each expert rather than in the experts’ positions\([Kiselev, 2026](https://arxiv.org/html/2611.25167#bib.bib26)\)\. InInflEnsthe players are the experts themselves, and each one chooses a position in a fixed trait space in order to earn a share of the training signal\. We solve the influencer’s game numerically to give the players their starting positions and use its local stability analysis to predict when those positions separate, and the resulting shares then weight adapter training and route inference\.

## 2The Influencer’s Game

We use the influencer’s game of[Lovett & Fu \(2026\)](https://arxiv.org/html/2611.25167#bib.bib36)\. Let a data point’s frozen trait vector beb∈𝔹b\\in\\mathbb\{B\}, drawn from a resource distributionBB\. Playeri∈\{1,…,N\}i\\in\\\{1,\\ldots,N\\\}chooses positionxi∈𝔹x\_\{i\}\\in\\mathbb\{B\}\. Then for a positive influence kernelfκ​\(x,b,σ\)f\_\{\\kappa\}\(x,b;\\sigma\)with parameterσ\\sigma, a player’s share and expected payoff are

Gi​\(𝐱,b\)=fκ​\(xi,b,σ\)∑j=1Nfκ​\(xj,b,σ\),ui​\(𝐱\)=𝔼B​\[Gi​\(𝐱,b\)\]\.G\_\{i\}\(\\mathbf\{x\},b\)=\\frac\{f\_\{\\kappa\}\(x\_\{i\},b;\\sigma\)\}\{\\sum\_\{j=1\}^\{N\}f\_\{\\kappa\}\(x\_\{j\},b;\\sigma\)\},\\qquad u\_\{i\}\(\\mathbf\{x\}\)=\\mathbb\{E\}\_\{B\}\\\!\\left\[G\_\{i\}\(\\mathbf\{x\},b\)\\right\]\.\(1\)
For every resource point∑iGi=1\\sum\_\{i\}G\_\{i\}=1, hence∑iui=1\\sum\_\{i\}u\_\{i\}=1at every configuration, indicating the game is constant\-sum\. Importantly, being constant\-sum implies that one player’s gain comes at the expense of the other player\(s\)\. The constant\-sum nature means that specialists will be competing for the same resource signal and are therefore encouraged to explore new niches when a niche is crowded and their influence kernel is not sufficiently wide\. Importantly, the utility of the player is based on their training signal share, and their accuracy is an emergent property of the training on that signal\. In turn, this means that a player’s received training signal is tractable through the history of the game, allowing us to predict specialists without explicitly testing their accuracy during training\.

### 2\.1Equilibrium and instability

We start by assuming that players are at a symmetric profilex1=⋯=xN=x∘x\_\{1\}=\\cdots=x\_\{N\}=x^\{\\circ\}, meaning they share a position in the trait space\. We let the first derivative of a player’s log\-influence at the trait points⁡\(b\)s\(b\)bes⁡\(b\)=∇x​log​fκ​\(x∘,b,σ\)s\(b\)=\\nabla\_\{x\}\\log f\_\{\\kappa\}\(x^\{\\circ\},b;\\sigma\)\. Then an interior symmetric stationary point obeys𝔼B​\[s​\(b\)\]=0\\mathbb\{E\}\_\{B\}\\\!\\left\[s\(b\)\\right\]=0\. Define restoring log\-curvature and score scatter by

K=−𝔼B​\[∇x2​log⁡fκ​\(x∘,b,σ\)\],C=𝔼B​\[s⁡\(b\)​s​\(b\)⊤\]\.K=\-\\mathbb\{E\}\_\{B\}\\\!\\left\[\\nabla\_\{x\}^\{2\}\\log f\_\{\\kappa\}\(x^\{\\circ\},b;\\sigma\)\\right\],\\qquad C=\\mathbb\{E\}\_\{B\}\\\!\\left\[s\(b\)s\(b\)^\{\\top\}\\right\]\.\(2\)WhenKKis positive definite on the relevant tangent space, the antisymmetric, player\-separating mode has instability depth

0pt=N−2N−1λmax\(K−1/2CK−1/2\)\.0pt=\\frac\{N\-2\}\{N\-1\}\\,\\lambda\_\{\\max\}\\\!\\left\(K^\{\-1/2\}CK^\{\-1/2\}\\right\)\.\(3\)For0​p​t<10pt<1the symmetric profile is locally stable, while0​p​t\>10pt\>1gives a player\-separating linear instability\. Here0​p​t=10pt=1is the stability boundary and, under a transversal parameter crossing, the first local bifurcation from the symmetric equlibirium\. Importantly, the log\-concavity of a kernel indicates there is always a kernel parameter regime where the symmetric equilibrium is stable and unique\. Additionally, for two effective players, the prefactor vanishes, so there is no separating\-mode bifurcation and the symmetric equilibrium is always locally stable\. Appendix[B](https://arxiv.org/html/2611.25167#A2)derives these claims from Equation[1](https://arxiv.org/html/2611.25167#S2.E1)\.

We study Gaussian, hyperbolic, and exponential\-power kernels whose parameters determine the kernel’s spread from its mode \(standard deviation, etc\.\), as well as kernels whose parameters determine the kernel’s shape, including the simplex\-native Dirichlet kernel and a coordinate\-wise product\-Beta kernel\. We denote a kernel’s reach byσ\\sigma, where Gaussian log influence scales asσ−2\\sigma^\{\-2\}, hyperbolic asσ−1\\sigma^\{\-1\}, and exponential\-power asσ−β\\sigma^\{\-\\beta\}; the shape kernels place inverse reach inside their density parameters and normalizers\. To simplify comparisons across kernel types, we compare kernels by their depth in Equation[3](https://arxiv.org/html/2611.25167#S2.E3)and by their allocation sharpness \(Section[4](https://arxiv.org/html/2611.25167#S4)\); Appendix[G](https://arxiv.org/html/2611.25167#A7)gives every implemented closed form\.

## 3Methods

The game router needs to have a trait assigned to every data point so that experts have a space to compete within for training signal\. Beyond the requirement of a trait space, other properties of the data distribution, such as dimensionality or labels, do not limit the applicability ofInflEns\. Naturally, many empirical data sets have high\-dimensional trait spaces when using raw embeddings, such as the 4096\-dimensional embeddings of Qwen3\-Embedding\-8B\([Zhang et al\., 2025](https://arxiv.org/html/2611.25167#bib.bib65)\), which likely leads to very sparse resource distributions with limited data sets\. Resource sparsity in

![Refer to caption](https://arxiv.org/html/2611.25167v1/fig_research_trait_slices.png)

![Refer to caption](https://arxiv.org/html/2611.25167v1/x1.png)

![Refer to caption](https://arxiv.org/html/2611.25167v1/x2.png)

Figure 2:Trait Space example; held\-out prompt density in three conditional simplex views atT=0\.35T=0\.35\. Grey is the combined density; colored contours mark benchmark origin\. Panels \(g\)–\(i\) are the bottom row of Figure[7](https://arxiv.org/html/2611.25167#A4.F7), which also gives the cube andT=1T=1views\.itself does not limit the effectiveness ofInflEnsbut can make it more difficult to demonstrate the nuances of its impacts\. To address this, we transform the 4096\-dimensional embeddings of Qwen3\-Embedding\-8B to seven axes defined by safety\-relevant data sets; Appendix[F](https://arxiv.org/html/2611.25167#A6)replaces them with axes learned without labels\. We generate a mixed dataset from seven benchmarks on distinct safety\-relevant properties: harm, hallucination, jailbreak, privacy, over\-refusal, policy violation, and prompt injection \(Appendix[A](https://arxiv.org/html/2611.25167#A1)\)\. Each safety\-relevant property then defines an axis label that we use for data points\. The loss and the reported NLL cover the full chat text, and four of the seven sources contribute prompts only \(47% of training records\), so on those the models are trained and scored on the prompts themselves; about 10% of training targets are unsafe responses and 9% hallucinated answers \(Table[3](https://arxiv.org/html/2611.25167#A1.T3)\)\.

To generate the trait space, we use a frozen, 4\-bit quantized Qwen3\-Embedding\-8B encoder throughout\. For each axis we fit a shrinkage Fisher discriminant direction in the embedding space, using the benchmark’s own labels \(for example, harmful versus harmless responses\) blended with a direction that separates that benchmark’s prompts from the rest\. Each new direction is orthogonalized against the directions already fitted, and each coordinate is residualized against the earlier coordinates, so that overlapping benchmarks do not collapse onto a single axis\. The resulting scores are mapped to\[0,1\]\[0,1\]by a per\-axis empirical CDF, so each coordinate is a percentile within the corpus and the seven marginals are uniform over the corpus by construction\. The simplex variant applies a softmax at temperatureT=0\.35T=0\.35to these coordinates\. Both the cube and the simplex are shown in Figure[7](https://arxiv.org/html/2611.25167#A4.F7)in Appendix[D](https://arxiv.org/html/2611.25167#A4), where the construction is given in full\. The 28,355 prompts of the combined corpus, mapped into this space, form the resource distribution that the experts compete over for training signal\.

The principal experiment places fourteen clones of the base model at seven co\-located pairs of positions and trains one rank\-16 LoRA adapter per pair\. The pairing of the clones is motivated by two game properties: a symmetric two\-player game with positive log\-concave influence kernels has no player\-separating instability \(Section[2\.1](https://arxiv.org/html/2611.25167#S2.SS1)\), and duplicating every solved location into an equal co\-located pair leaves the effective allocation exactly unchanged \(Appendix[B](https://arxiv.org/html/2611.25167#A2)\)\. In effect, this implies that the pairs behave as one agent when updating their positions, with the added benefit of increased diversity at a lower depth than individual initialization\. The pair is therefore the routing and adapter\-training unit: its members’ shares sum to the allocation a single agent would receive at that position in the seven\-player game, but at a much lower depth\. We keep the even split between members only in the position dynamics, where each clone follows its own utility gradient in the fourteen\-player game\. Equal co\-located clones receive identical updates, so pairing is an invariant of the update rather than an explicit copy operation \(Appendix[C](https://arxiv.org/html/2611.25167#A3)\)\. Solving the eight\-player position game directly shows the same invariance, as players started in four co\-located pairs remain paired at every reach below0\.7​σ∗0\.7\\sigma^\{\*\}\(Figure[6](https://arxiv.org/html/2611.25167#A3.F6)\)\.

In each experiment, the pipeline completes 12 rounds, consisting of the following steps: it projects a training batch to traits, computes pair shares by Equation[1](https://arxiv.org/html/2611.25167#S2.E1), and assigns each prompt to its top\-kkpairs, with retained shares renormalized as loss weights\. The densek=Nk=Narm retains allNNpairs, whereask=1k=1is a hard allocation\. In the dynamic\-position arm, after allocation, each clone takes a simultaneous projected ascent step on its game payoff, using the score of the chosen influence kernel over the whole round batch\. A fixed\-position control disables every position\-update path while leaving assignment and adapter training otherwise unchanged\. Appendix[I](https://arxiv.org/html/2611.25167#A9)separately ablates fan\-out, sampling, and loss weighting, showing why the main experiments retain the full share\-weighted allocation\.

At evaluation, final pair positions yieldGp​\(b\)G\_\{p\}\(b\)without fitting a gate to held\-out expert losses\. We distinguish expected NLL∑pGp​\(b\)​ℓp​\(b\)\\sum\_\{p\}G\_\{p\}\(b\)\\ell\_\{p\}\(b\)\(the loss of sampling one pair\), argmax commitment, and the NLL of mixing pair probabilities over a completed sequence, which can exploit probability averaging that a single\-expert decision cannot\. In each experiment, we use a hindsight best\-expert oracle to supply an upper bound on recoverable routing headroom\. To show how a single model compares, a pooled comparator trains on the corresponding union of routed batches, matching the training signal the seven experts receive in aggregate\. Therefore, an expert and pooled adapter each receive the same 1,248 optimizer steps\.

The 28,355 records \(Table[2](https://arxiv.org/html/2611.25167#A1.T2)\) are partitioned independently within each source using a persisted seeded manifest: 19,848 train, 2,836 validation, and 5,671 test records \(a 70/10/20 split\)\. Every arm reuses those indices\.We fit the trait axes, quantile normalization, and resource KDE to the full 28,355\-record population before any adapters are trained; this resource KDE is only used for calculating the theoretical equilibrium of the distributions\. “Test” therefore means held out from adapter optimization, gate fitting, and expert\-loss feedback, but not from resource\-map estimation\. Refitting all three on the training partition only lowers the routed single\-expert gain by 0\.0091 nats on three splits without changing expert quality \(Appendix[E](https://arxiv.org/html/2611.25167#A5)\)\. The principal arm starts from Qwen2\.5\-1\.5B\-Instruct\([Yang et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib61)\), trains rank\-16 LoRA adapters for twelve rounds, uses soft routing with weights equal to the proportion of influence over data \(Appendix[I](https://arxiv.org/html/2611.25167#A9)\), and scores a 5,671\-prompt held\-out pool\. Appendix[O](https://arxiv.org/html/2611.25167#A15)ablates the base model’s family and size\.

## 4Results

### 4\.1Game Controls: Equilibrium Positioning, Influence Kernel, and Resource Geometry

Figure[3](https://arxiv.org/html/2611.25167#S4.F3)a showsInflEnstrained on theT=0\.35T=0\.35Dirichlet game with two to nine pairs, depth matched to the seven\-pair arm, on three data splits under the pinned adaptive, pinned fixed and equilibrium adaptive starts compared in Section[4\.2](https://arxiv.org/html/2611.25167#S4.SS2)\. Gain peaks at 0\.125 nats at six pairs under all three, and the starts agree within their split spread from four pairs on\. At this reach, however, the exact equilibrium lies only 0\.005–0\.020 from the unsolved start \(neighbouring pairs are 0\.10–0\.24 apart\), so the comparison cannot show whether a distant equilibrium start helps \(Figure[12](https://arxiv.org/html/2611.25167#A8.F12)\)\.

Figure 3:The starting configuration changes little once the game is depth matched on the sharpened simplex, whereas induced allocation sharpness matters more than kernel identity\. \(a\) Expected routing gain is pooled\-generalist NLL minus the share\-weighted NLL of sampling one pair \(higher is better\), for theT=0\.35T=0\.35Dirichlet game at two to nine pairs under three starts; bands are one standard deviation across three data splits\. \(b\) Matching bifurcation depth leaves a large gain spread across four kernels \(dark red\); matching prompt\-level sharpness narrows it \(orange\), and sharpening the simplex projection toT=0\.35T=0\.35\(dispersed\) raises and nearly equalizes the gains \(blue\)\. Hollow diamonds are cube controls\.In Figure[3](https://arxiv.org/html/2611.25167#S4.F3)b, four kernels and three trait\-space geometries show that trait\-space geometry shifts how much kernel identity matters\. Matched on bifurcation depth on the simplex atT=1T=1\(dense\), expected gain spans 0\.0407 nats across the four kernels and tracks their sharpness almost exactly, with the sharpest kernel best and the most diffuse worst\. Matching sharpness by shifting kernel parameters collapses that spread to 0\.0055 nats\. Meanwhile, sharpening the simplex projection toT=0\.35T=0\.35\(dispersed\) lifts every kernel to roughly 0\.115 nats with a spread of 0\.0046, effectively eliminating any difference between the kernels when depth matched\. Table[1](https://arxiv.org/html/2611.25167#S4.T1)shows the same collapse in how well the allocation predicts expert quality: top\-1 agreement spans 82\.6–87\.3% across kernels atT=1T=1and 89\.2–90\.8% atT=0\.35T=0\.35, against a shuffled\-allocation null of about 15%\. This indicates that kernel choice and resource distribution impact the predictive power of the game router\.

Importantly, in Table[1](https://arxiv.org/html/2611.25167#S4.T1)the game router ranks the best expert first about six times as often as chance, indicating that specialization follows fromInflEns’s allocation\. The simplex geometries also show higher router–oracle agreement and mixture capture than the cube, but a bidirectional frozen\-expert control shows that each expert family prefers the geometry in which it trained, so experts trained in one geometry do not transfer to another \(Appendix[D](https://arxiv.org/html/2611.25167#A4)\)\.

Table 1:How well the game router predicts expert quality, per influence kernel and simplex temperatureTT, which sets how concentrated the resource distribution is\. Each cell is the mean±\\pmsample standard deviation over three training seeds of the seven\-pair run; the seeds share one data split\. Higher is better for every measure\. Top 1, top 2 and top 3 are the percentages of held\-out prompts whose best expert receives the largest share, one of the two largest, or one of the three largest; each null shuffles allocations across prompts\. Spearman is the mean within\-prompt rank correlation between allocation and expert quality\.
### 4\.2Kernel Parameters

Kernel parameters control how broadly an expert competes for training signal and can determine the depth of the influence kernel past its bifurcation threshold\. To test this effect onInflEns, in Figure[4](https://arxiv.org/html/2611.25167#S4.F4)we limit to seven pairs, use the first local bifurcationσ∗\\sigma^\{\*\}as a reference scale for a spherical multivariate Gaussian influence kernel’s reach parameterσ\\sigma, and then sweep the reach and measure the resulting routing response\. We compare three conditions: an equilibrium adaptive start, which begins at the solved empirical\-game equilibrium and tends to stay near it; a pinned adaptive start, which begins every reach on a data split from the same unsolved configuration; and a pinned fixed condition, identical but with positions never updated \(Appendix[H](https://arxiv.org/html/2611.25167#A8)\)\.

Initialization acts on the shape of the reach response and leaves its level unchanged\. Mean per\-split roughness \(excluding the degenerate 0\.70 endpoint\) is 0\.483 for the equilibrium adaptive start and 1\.266 for the pinned start over three data splits each, and the per\-split ranges do not overlap \(0\.43–0\.53 against 1\.07–1\.45\)\.

Both starts peak at reach 0\.40 or 0\.45, so a shared start fixes where the response peaks, but the equilibrium adaptive start is 2\.6 times smoother and makes the choice of which configuration to pin less arbitrary\. Mean gain does not separate the two adaptive starts, at 0\.0764 nats for the pinned and 0\.0747 nats for the equilibrium adaptive start, and the paired difference,−0\.0018±0\.0082\-0\.0018\\pm 0\.0082nats across splits, is small next to its own spread \(Figure[4](https://arxiv.org/html/2611.25167#S4.F4)\)\.

Figure 4:The equilibrium adaptive start stabilizes the reach response without changing mean gain; freezing pinned positions costs gain without adding regularity\. Each point is a within\-data\-seed contrast against its pooled generalist; bands are sample standard deviations across three splits per condition\.The pinned start runs had not settled by round 11: they closed 43% of their distance to the equilibrium adaptive runs on average, and continuing the position update past training carries only 15 of the 27 non\-degenerate reach–split cells to the same fixed point \(Appendix[H](https://arxiv.org/html/2611.25167#A8)\), indicating there is some choice of equilibria that needs to be made\.

### 4\.3Routing evidence and controls

With the seven\-pair expert family fixed, we first try to determine how much of the gain comes from how the experts’ outputs are combined\. Under the standard token\-level estimator, the routed mixture gains 0\.1049 nats over pooling all of the data to one generalist model\. Additionally, the completed\-sequence estimator we use for the same\-expert nulls gives a slightly more optimistic 0\.1093–0\.1095 nats\. On that same estimator, mixing the experts uniformly still retains a 0\.0661\-nat gain, whereas shuffling the expert identities loses 0\.0671 nats and uniform random hard routing loses 0\.4144, which shows that the game’s allocation carries information about which expert suits which prompt \(Appendix[I](https://arxiv.org/html/2611.25167#A9), Figure[13](https://arxiv.org/html/2611.25167#A9.F13)\)\. In these nulls we only change the combiner, and the same appendix shows the impact of changing the way the adapters are trained\. Throughout, we report the dense top\-k=Nk=Ncase as our default, whereNNis the number of pairs and every pair contributes to the mixture, but other methods are shown in Appendix[I](https://arxiv.org/html/2611.25167#A9)\.

To compareInflEnsto a learned gate, we first freeze all adapters and fit linear prompt\-conditioned gates on the validation expert losses\. Across four data splits, the game allocation beats both the full\-NLL soft\-target gate and a best\-expert classification gate by 0\.071–0\.090 nats when a single expert is committed to per prompt\. Under full probability mixing, the sign divides two\-to\-two across splits and the largest difference is only 0\.0017 nats, which is at the measured run\-to\-run noise floor\. These results support that a gate\-free allocation is sufficient for hard routing to experts\.

We also compareInflEnswith routers trained end to end\. We implement five published gates \(dense softmax, noisy top\-kk, Switch, Mixtral and DeepSeek\-V3\) and two distance gates \(Gaussian and X\-MoE cosine\)\. Each gate is trained jointly with seven LoRA experts on the game’s batches, using the sameT=0\.35T=0\.35trait coordinates as input \(Figure[5](https://arxiv.org/html/2611.25167#S4.F5)\)\. We train the five published gates with the router learning rate raised to10−210^\{\-2\}, because at the experts’ learning rate a prompt\-level router barely moves in 1,248 steps \(Appendix[L](https://arxiv.org/html/2611.25167#A12)\)\. With that correction, the dense softmax gate \(1\.8864\) is within 0\.0035 nats of the game’s token\-level mixture over all pairs \(1\.8899\), and DeepSeek\-V3 and Mixtral, which route each prompt to two experts, trail the game’s top\-2 mixture \(1\.8901\) by 0\.003 and 0\.005 nats, all within run\-to\-run noise\. The difference between the methods lies in the experts themselves: the best single component of a jointly trained mixture gains only 0\.05–0\.09 nats over pooling, against 0\.143 for the game’s best expert, so the game’s experts can be served on their own while the gates’ components only work when blended\.

The game’s allocation is not the only way to route its own experts\. When the frozen linear soft\-target gate routes the game’s experts, it scores 1\.8999 under the same token\-level estimator, 0\.0101 nats behind the game’s shares\. When the published gates are instead fitted directly on the game’s frozen experts, the distance gates route them better than the game’s shares, with X\-MoE cosine scoring 1\.8782 and the Gaussian gate 1\.8795 \(0\.0116 and 0\.0104 nats ahead\), while dense softmax and DeepSeek\-V3 match the game and noisy top\-kk, Switch and Mixtral trail it \(Appendix[L](https://arxiv.org/html/2611.25167#A12)\)\. The game’s allocation is therefore not the best possible router for its own experts, but it is a good one, and a distinguishing factor is that it is fixed before training rather than learned on the experts’ losses\.

Finally, we test whether the game’s allocation still produces experts when it is used as the gate of a single jointly trained mixture\. Fixing the gate to the game’s shares costs the same as training a learned gate and still yields experts: its components alone reach 0\.125 nats over pooling at the oracle, against 0\.05–0\.09 for the learned gates, while the blended network scores 1\.9110 \(Appendix[L](https://arxiv.org/html/2611.25167#A12)\)\. We can also blend the game’s separately trained experts into one network under the same gate, which needs no further training and scores 1\.8932, within 0\.0033 nats of mixing their outputs, so the game’s mixture does not require seven forward passes at inference\. The game also needs no router tuning, although it uses seven times the training compute for the same number of updates per adapter\.

Figure 5:Published gating rules against the game on theT=0\.35T=0\.35simplex \(seed\-0 split, one run per arm\)\. \(a\) Routed\-mixture NLL as gain over the pooled generalist \(higher is better\), token level throughout\. Filled markers are gates trained jointly at router learning rate10−210^\{\-2\}, hollow markers at the experts’ rate, and diamonds are gates fitted post hoc on the game’s frozen experts; the top row fixes the gate to the game’s shares, and the dotted line is the hindsight\-best expert\. \(b\) The per\-prompt best single component of each mixture against the game’s best single expert\. Runs at the experts’ router learning rate are compared in Appendix[L](https://arxiv.org/html/2611.25167#A12)\.
### 4\.4Additional experiments

Experimenting with Qwen2\.5 model sizes of 1\.5B, 3B, 7B, 14B, and 32B shows that each size improves over its own pooled comparator while oracle\-headroom capture declines, and the allocation effects also transfer to two other model families, Gemma and Llama \(Appendix[O](https://arxiv.org/html/2611.25167#A15)\)\.

Additionally,InflEnscompares favorably within a ten\-family baseline audit, including a frozen gate, same\-expert nulls, balanced c\-BTM, BASE\-style assignment, MoLoRA, KnOTS, and a pooled\-seed deep ensemble \(Appendix[P](https://arxiv.org/html/2611.25167#A16)\)\. In the most directly comparable of these, on the same seed\-0 split and estimator, the game mixture nearly doubles c\-BTM’s gain over pooling \(NLL 1\.9057, \+0\.1095 vs\. 1\.9564, \+0\.0588\), while balanced c\-BTM degrades to 1\.9647\.

We also experimented with learned axises instead of assigned lables\. With label\-free axes built fromkk\-means clusters of the training prompts,InflEnskeeps about 83% of its gain \(0\.090±0\.0020\.090\\pm 0\.002nats over three splits\), although a learned dense gate reaches a lower NLL there without producing experts that work alone \(Appendix[F](https://arxiv.org/html/2611.25167#A6)\)\.

To compare with soft clustering, we give softkk\-means the same trait space, resource information, dense share\-weighted training, expert count and budget as the game, with positions frozen\. Thekk\-means positions are interchangeable with the game’s equilibrium positions \(paired difference0\.0006±0\.00070\.0006\\pm 0\.0007nats over three splits\), so the game reaches clustering\-quality positions from a competitive objective\. Softkk\-means’ own allocation trails the game’s by 0\.010 nats when one expert is sampled and by 0\.002 under probability mixing, and most of that gap comes from the Dirichlet kernel \(Appendix[Q](https://arxiv.org/html/2611.25167#A17)\)\. Thus kernel choice and parameterization giveInflEnsa competitive advantage over soft clustering, and the game may possibly be a generalization of softkk\-means to arbitrary influence kernels and trait\-space geometries\.

Out of distribution, we test GSM8K, which no training benchmark resembles, and seven unseen datasets of the trained kinds\. Fine\-tuning on this corpus costs every model much of the base model’s math ability, and the game keeps the most, answering about twice as many problems as the pooled adapter \(accuracy 0\.41 against 0\.21 over three splits\)\. On the unseen datasets both mixtures beat pooling on full text with a dense gate slightly ahead, while on response tokens alone no mixture beats the pooled adapter \(Appendix[S](https://arxiv.org/html/2611.25167#A19)\)\.

On safety benchmarks \(Appendix[R](https://arxiv.org/html/2611.25167#A18)\) the game outperformed the pooled adapter, but the base model far outperformed all trained models because the corpus trains on unsafe responses and bare harmful prompts; retraining on safe responses returns every method to the base model’s attack\-success level, where the game keeps the most instruction following\. Because the router is fixed, the harm can also be traced to an expert before anything is retrained: 94% of the HarmBench prompts route to one pair, and retraining only that pair on safe responses lowers attack success from 0\.51 to 0\.02 while IFEval changes by−0\.011\-0\.011, within noise; the same data trained into another expert changes nothing\.

## 5Limitations and Discussion

The headline claim ofInflEnsis that the influencer’s game makes specialization of experts an explicit process rather than an empirically derived one\. Through experiments, we have seen that even without training a gate on experts’ losses,InflEnspredicted the best expert 90% of the time as its top choice and the best expert 96% of the time in its top\-2 choices when routing data\. This has many consequences for interpretability, since we may isolate how data is routed and to which experts\. For safety, this means that we may target experts with specific safety data, or if the MoE experts perform poorly on a specific data point in the trait space, identify which expert is explicitly responsible and retrain it alone \(Section[4\.4](https://arxiv.org/html/2611.25167#S4.SS4)\)\.

Importantly, there are many levers that researchers can pull in order to shape the behavior of the MoE through the game structure, including the number of experts, the data distribution representation, and kernel choice and its parameters\. Perhaps more useful are the equilibria given by the influencer’s game component, which provide targets for models adapting to an unknown data distribution or possibly a changing one\. If the data distribution is known, then the experts’ equilibrium positions can be calculated before training time and used as stable initializations for expert specialization\. There are also limitations\.InflEnsrelies on symmetric and log\-concave kernels, which implies that experts cannot have multiple central points of expertise and that their influence cannot decay asymmetrically\. While the first can be addressed by a larger number of experts, there is a real cost to training more adapters\. The second limitation is more restrictive because the theory fundamentally relies on symmetric decay of influence, but the need for asymmetric kernels remains uncertain\. The game also routes per prompt, since it needs one fixed trait vector per data point; over conventional sparse\-upcycled experts its shares come within 0\.006 nats of a learned dense router, but no token\-level router we trained matched the pooled adapter \(Appendix[M](https://arxiv.org/html/2611.25167#A13)\)\.

Perhaps the most promising future direction is online adaptation to new data streams\. Because there is no trained gate, there is no learned router to collapse\. The game defines a position payoff for any observed resource distribution, so an updated prompt distribution implies new position incentives and a new candidate equilibrium\. In the present fixed\-distribution experiments, the position dynamics already correct an initial position mismatch \(Section[4\.2](https://arxiv.org/html/2611.25167#S4.SS2)\), although this does not test distribution shift\. A direct test would change the prompt distribution over time, compare predicted and observed position movement, and continue adapter training under the revised allocations\.

## 6Conclusion

InflEnstreats expert coordination as a location game over a trait space of a data distribution\. Its equilibrium supplies a starting position before any adapter is trained, and its allocation is a falsifiable prediction about held\-out expert quality rather than a fitted gate\. The components ofInflEnsare not limited to the language model setting or by choices such as the encoder and data type, and may contribute to other areas of work\. The open question is how this method will apply to online settings where the data distribution is not fixed\.

## 7AI Use Disclosure

Generative AI tools, including OpenAI Codex, assisted with literature discovery and citation checking, manuscript organization and language editing, and software implementation and debugging\. The authors reviewed all retained prose and code, checked citation\-dependent claims against primary sources, tested AI\-assisted implementation, and verified reported numerical claims against saved experimental artifacts\. AI\-generated output was not treated as empirical evidence or credited with authorship; the authors take responsibility for the paper and all associated artifacts\.

## 8Ethics Statement

This work develops a routing and training\-allocation method and conducts no human\-subject experiments or new collection of personal data; the experiments use existing public benchmarks whose provenance is documented in Appendix[A](https://arxiv.org/html/2611.25167#A1)\. The method may make expert behavior easier to audit, but it does not guarantee safety or fairness, and specialization can concentrate failures as well as useful capabilities\. Appendix[R](https://arxiv.org/html/2611.25167#A18)reports the observed safety limitations, including that fine\-tuning substantially degraded alignment relative to the base model\.

## 9Reproducibility Statement

The game assumptions and stability analysis are stated in Section[2](https://arxiv.org/html/2611.25167#S2)and proved in Appendix[B](https://arxiv.org/html/2611.25167#A2); the data sources, retained counts, shared splits, model and adapter settings, routing procedure, and evaluation estimators are documented in Section[4](https://arxiv.org/html/2611.25167#S4)and Appendices[A](https://arxiv.org/html/2611.25167#A1)–[I](https://arxiv.org/html/2611.25167#A9)\. Appendices[O](https://arxiv.org/html/2611.25167#A15),[P](https://arxiv.org/html/2611.25167#A16), and[R](https://arxiv.org/html/2611.25167#A18)specify the ablation, baseline, and behavioral protocols and their qualifications\. The accompanying code uses versioned YAML configurations, records the resolved configuration and run history, and exposes a single pipeline entry point; Appendix[T](https://arxiv.org/html/2611.25167#A20)gives the command and identifies the result artifacts underlying the reported analyses\. Access to the code is available here:[https://anonymous\.4open\.science/r/infl\_ens\-2516/README\.md](https://anonymous.4open.science/r/infl_ens-2516/README.md)\.

## References

- Agarap et al\. \(2026\)Abien Fred Agarap, Inigo Miguel Benavides, and Sara Ann Venturina\.Nash bargaining for gate\-free mixture\-of\-experts\.In*NExT\-Game Workshop at the International Conference on Machine Learning*, 2026\.URL[https://openreview\.net/pdf/566cfde3e6550eeda0f5e759166e06374388b217\.pdf](https://openreview.net/pdf/566cfde3e6550eeda0f5e759166e06374388b217.pdf)\.Workshop paper\.
- ai4Privacy \(2023\)ai4Privacy\.PII\-masking\-200k: A dataset of synthetic documents with labelled personally identifiable information\.Hugging Face dataset, 2023\.URL[https://huggingface\.co/datasets/ai4privacy/pii\-masking\-200k](https://huggingface.co/datasets/ai4privacy/pii-masking-200k)\.
- Ben\-Porat & Tennenholtz \(2019\)Omer Ben\-Porat and Moshe Tennenholtz\.Regression equilibrium\.In*Proceedings of the 2019 ACM Conference on Economics and Computation*, pp\. 173–191, 2019\.doi:10\.1145/3328526\.3329560\.URL[https://arxiv\.org/abs/1905\.02576](https://arxiv.org/abs/1905.02576)\.
- Bianchi et al\. \(2024\)Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou\.Safety\-tuned LLaMAs: Lessons from improving the safety of large language models that follow instructions\.In*International Conference on Learning Representations*, 2024\.arXiv:2309\.07875\.
- Brahman et al\. \(2024\)Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, Yulia Tsvetkov, Noah A\. Smith, Yejin Choi, and Hannaneh Hajishirzi\.The art of saying no: Contextual noncompliance in language models\.In*Advances in Neural Information Processing Systems*, volume 37, 2024\.arXiv:2407\.12043\.
- Chao et al\. \(2024\)Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J\. Pappas, Florian Tramèr, Hamed Hassani, and Eric Wong\.JailbreakBench: An open robustness benchmark for jailbreaking large language models\.In*Advances in Neural Information Processing Systems Datasets and Benchmarks Track*, 2024\.arXiv:2404\.01318\.
- Chen et al\. \(2025\)Sizhe Chen, Arman Zharmagambetov, David Wagner, and Chuan Guo\.Meta SecAlign: A secure foundation LLM against prompt injection attacks\.*arXiv preprint arXiv:2507\.02735*, 2025\.
- Chronopoulou et al\. \(2023\)Alexandra Chronopoulou, Matthew E\. Peters, Alexander Fraser, and Jesse Dodge\.AdapterSoup: Weight averaging to improve generalization of pretrained language models\.In*Findings of the Association for Computational Linguistics: EACL 2023*, pp\. 2054–2063\. Association for Computational Linguistics, 2023\.doi:10\.18653/v1/2023\.findings\-eacl\.153\.URL[https://aclanthology\.org/2023\.findings\-eacl\.153/](https://aclanthology.org/2023.findings-eacl.153/)\.
- Cobbe et al\. \(2021\)Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman\.Training verifiers to solve math word problems\.*arXiv preprint arXiv:2110\.14168*, 2021\.
- Cui et al\. \(2025\)Justin Cui, Wei\-Lin Chiang, Ion Stoica, and Cho\-Jui Hsieh\.OR\-Bench: An over\-refusal benchmark for large language models\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267, pp\. 11515–11542, 2025\.URL[https://proceedings\.mlr\.press/v267/cui25a\.html](https://proceedings.mlr.press/v267/cui25a.html)\.
- Dai et al\. \(2022\)Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei\.StableMoE: Stable routing strategy for mixture of experts\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics*, pp\. 7085–7095, Dublin, Ireland, 2022\. Association for Computational Linguistics\.doi:10\.18653/v1/2022\.acl\-long\.489\.URL[https://aclanthology\.org/2022\.acl\-long\.489/](https://aclanthology.org/2022.acl-long.489/)\.
- Fedus et al\. \(2022\)William Fedus, Barret Zoph, and Noam Shazeer\.Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity\.*Journal of Machine Learning Research*, 23\(120\):1–39, 2022\.
- Ghosh et al\. \(2025\)Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, and Christopher Parisien\.AEGIS2\.0: A diverse AI safety dataset and risks taxonomy for alignment of LLM guardrails\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pp\. 5992–6026, 2025\.URL[https://aclanthology\.org/2025\.naacl\-long\.306/](https://aclanthology.org/2025.naacl-long.306/)\.
- Gou et al\. \(2026\)Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Zhenguo Li, Dit\-Yan Yeung, James T\. Kwok, and Yu Zhang\.Mixture of cluster\-conditional LoRA experts for vision\-language instruction tuning\.*IEEE Transactions on Image Processing*, 35:3881–3892, 2026\.doi:10\.1109/TIP\.2026\.3678087\.arXiv:2312\.12379\.
- Gururangan et al\. \(2023\)Suchin Gururangan, Margaret Li, Mike Lewis, Weijia Shi, Tim Althoff, Noah A\. Smith, and Luke Zettlemoyer\.Scaling expert language models with unsupervised domain discovery\.*arXiv preprint arXiv:2303\.14177*, 2023\.URL[https://arxiv\.org/abs/2303\.14177](https://arxiv.org/abs/2303.14177)\.
- Hammoud et al\. \(2024\)Hasan Abed Al Kader Hammoud, Umberto Michieli, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem, and Mete Ozay\.Model merging and safety alignment: One bad model spoils the bunch\.In*Findings of the Association for Computational Linguistics: EMNLP 2024*, 2024\.arXiv:2406\.14563\.
- Hsu et al\. \(2024\)Chia\-Yi Hsu, Yu\-Lin Tsai, Chih\-Hsun Lin, Pin\-Yu Chen, Chia\-Mu Yu, and Chun\-Ying Huang\.Safe LoRA: The silver lining of reducing safety risks when fine\-tuning large language models\.In*Advances in Neural Information Processing Systems*, volume 37, 2024\.arXiv:2405\.16833\.
- Hu et al\. \(2022\)Edward J\. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen\-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen\.LoRA: Low\-rank adaptation of large language models\.In*International Conference on Learning Representations*, 2022\.
- Inan et al\. \(2023\)Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa\.Llama Guard: LLM\-based input\-output safeguard for human\-AI conversations\.*arXiv preprint arXiv:2312\.06674*, 2023\.
- Jacobs et al\. \(1991\)Robert A\. Jacobs, Michael I\. Jordan, Steven J\. Nowlan, and Geoffrey E\. Hinton\.Adaptive mixtures of local experts\.*Neural Computation*, 3:79–87, 1991\.URL[https://www\.cs\.toronto\.edu/~hinton/absps/jjnh91\.pdf](https://www.cs.toronto.edu/~hinton/absps/jjnh91.pdf)\.
- Jajoo \(2026\)Sanskar Jajoo\.Neuralchemy prompt injection threat matrix\.Hugging Face dataset, binary configuration, 2026\.URL[https://huggingface\.co/datasets/neuralchemy/prompt\-injection\-Threat\-Matrix](https://huggingface.co/datasets/neuralchemy/prompt-injection-Threat-Matrix)\.
- Ji et al\. \(2023\)Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou Wang, and Yaodong Yang\.BeaverTails: Towards improved safety alignment of LLM via a human\-preference dataset\.In*Advances in Neural Information Processing Systems Datasets and Benchmarks Track*, 2023\.arXiv:2307\.04657\.
- Ji et al\. \(2024\)Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang\.PKU\-SafeRLHF: Towards multi\-level safety alignment for LLMs with human preference\.*arXiv preprint arXiv:2406\.15513*, 2024\.
- Jiang et al\. \(2024\)Albert Q\. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al\.Mixtral of experts\.*arXiv preprint arXiv:2401\.04088*, 2024\.URL[https://arxiv\.org/abs/2401\.04088](https://arxiv.org/abs/2401.04088)\.
- Kim et al\. \(2026\)Jaehan Kim, Minkyoo Song, Seungwon Shin, and Sooel Son\.SafeMoE: Safe fine\-tuning for MoE LLMs by aligning harmful input routing\.In*International Conference on Learning Representations*, 2026\.arXiv:2509\.22745\.
- Kiselev \(2026\)O\. M\. Kiselev\.A minimal bifurcation model of load imbalance in a softmax mixture\-of\-experts router\.*arXiv preprint arXiv:2605\.29121*, 2026\.
- Kohli et al\. \(2025\)Harsh Kohli, Helian Feng, Lenon Minorics, Bhoomit Vasani, Xin He, and Ali Kebarighotbi\.EWoRA: Expert weighted low\-rank adaptation for heterogeneous data\.In*Findings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics*, pp\. 1729–1737, 2025\.doi:10\.18653/v1/2025\.findings\-ijcnlp\.108\.URL[https://aclanthology\.org/2025\.findings\-ijcnlp\.108/](https://aclanthology.org/2025.findings-ijcnlp.108/)\.
- Komatsuzaki et al\. \(2023\)Aran Komatsuzaki, Joan Puigcerver, James Lee\-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby\.Sparse upcycling: Training mixture\-of\-experts from dense checkpoints\.In*International Conference on Learning Representations*, 2023\.arXiv:2212\.05055\.
- Lakshminarayanan et al\. \(2017\)Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell\.Simple and scalable predictive uncertainty estimation using deep ensembles\.In*Advances in Neural Information Processing Systems*, volume 30, 2017\.URL[https://proceedings\.neurips\.cc/paper/2017/hash/9ef2ed4b7fd2c810847ffa5fa85bce38\-Abstract\.html](https://proceedings.neurips.cc/paper/2017/hash/9ef2ed4b7fd2c810847ffa5fa85bce38-Abstract.html)\.
- Lambert et al\. \(2024\)Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V\. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al\.Tülu 3: Pushing frontiers in open language model post\-training\.*arXiv preprint arXiv:2411\.15124*, 2024\.
- Lewis et al\. \(2021\)Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer\.BASE layers: Simplifying training of large, sparse models\.In*Proceedings of the 38th International Conference on Machine Learning*, pp\. 6265–6274, 2021\.URL[https://proceedings\.mlr\.press/v139/lewis21a\.html](https://proceedings.mlr.press/v139/lewis21a.html)\.
- Li et al\. \(2023\)Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian\-Yun Nie, and Ji\-Rong Wen\.HaluEval: A large\-scale hallucination evaluation benchmark for large language models\.In*Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp\. 6449–6464, 2023\.doi:10\.18653/v1/2023\.emnlp\-main\.397\.URL[https://aclanthology\.org/2023\.emnlp\-main\.397/](https://aclanthology.org/2023.emnlp-main.397/)\.
- Li et al\. \(2022\)Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A\. Smith, and Luke Zettlemoyer\.Branch\-train\-merge: Embarrassingly parallel training of expert language models\.*arXiv preprint arXiv:2208\.03306*, 2022\.URL[https://arxiv\.org/abs/2208\.03306](https://arxiv.org/abs/2208.03306)\.
- Lin et al\. \(2022\)Stephanie Lin, Jacob Hilton, and Owain Evans\.TruthfulQA: Measuring how models mimic human falsehoods\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 3214–3252, 2022\.
- Lotfi et al\. \(2026\)Sanae Lotfi, Lucas Caccia, Alessandro Sordoni, Jordan T\. Ash, and Miroslav Dudík\.Trade\-offs in ensembling, merging and routing among parameter\-efficient experts\.*Transactions on Machine Learning Research*, 2026\.arXiv:2603\.03535\.
- Lovett & Fu \(2026\)Mark Lovett and Feng Fu\.Learning dynamics of the influencer’s game in resource landscapes\.*arXiv preprint arXiv:2611\.17148*, 2026\.
- Ma et al\. \(2024\)Yufei Ma, Zihan Liang, Huangyu Dai, Ben Chen, Dehong Gao, Zhuoran Ran, Zihan Wang, Linbo Jin, Wen Jiang, Guannan Zhang, Xiaoyan Cai, and Libin Yang\.MoDULA: Mixture of domain\-specific and universal LoRA for multi\-task learning\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp\. 2758–2770, 2024\.doi:10\.18653/v1/2024\.emnlp\-main\.161\.URL[https://aclanthology\.org/2024\.emnlp\-main\.161/](https://aclanthology.org/2024.emnlp-main.161/)\.
- Mouzouni \(2026\)Charafeddine Mouzouni\.Three phases of expert routing: How load balance evolves during mixture\-of\-experts training\.*arXiv preprint arXiv:2604\.04230*, 2026\.
- Muennighoff et al\. \(2025\)Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al\.OLMoE: Open mixture\-of\-experts language models\.In*International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=xXTkbTBmqq](https://openreview.net/forum?id=xXTkbTBmqq)\.arXiv:2409\.02060\.
- Muqeeth et al\. \(2024\)Mohammed Muqeeth, Haokun Liu, Yufan Liu, and Colin Raffel\.Learning to route among specialized experts for zero\-shot generalization\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pp\. 36829–36846\. PMLR, 2024\.URL[https://proceedings\.mlr\.press/v235/muqeeth24a\.html](https://proceedings.mlr.press/v235/muqeeth24a.html)\.arXiv:2402\.05859\.
- Navon et al\. \(2022\)Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya\.Multi\-task learning as a bargaining game\.In*Proceedings of the 39th International Conference on Machine Learning*, pp\. 16428–16446, 2022\.URL[https://proceedings\.mlr\.press/v162/navon22a\.html](https://proceedings.mlr.press/v162/navon22a.html)\.
- Nguyen et al\. \(2026\)Dung Viet Nguyen, Anh Thi Nguyen, Minh Hoang Nguyen, Luc Nguyen, Shiqi Jiang, Ethan Fetaya, Duy Linh Tran, Gal Chechik, and Tan Nguyen\.Expert merging in sparse mixture of experts with nash bargaining\.In*International Conference on Learning Representations*, 2026\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2026/hash/ef9f16cd0cb8d7797ea35c0ee7964cdf\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2026/hash/ef9f16cd0cb8d7797ea35c0ee7964cdf-Abstract-Conference.html)\.
- Nguyen et al\. \(2025\)Nam V\. Nguyen, Huy Nguyen, Quang Pham, Van Nguyen, Savitha Ramasamy, and Nhat Ho\.CompeteSMoE: Statistically guaranteed mixture of experts training via competition\.*arXiv preprint arXiv:2505\.13380*, 2025\.URL[https://arxiv\.org/abs/2505\.13380](https://arxiv.org/abs/2505.13380)\.
- Ong et al\. \(2025\)Isaac Ong, Amjad Almahairi, Vincent Wu, Wei\-Lin Chiang, Tianhao Wu, Joseph E\. Gonzalez, M\. Waleed Kadous, and Ion Stoica\.RouteLLM: Learning to route LLMs from preference data\.In*International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=8sSqNntaMr](https://openreview.net/forum?id=8sSqNntaMr)\.arXiv:2406\.18665\.
- Ostapenko et al\. \(2024\)Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti, Laurent Charlin, Nicolas Le Roux, Lucas Caccia, and Alessandro Sordoni\.Towards modular LLMs by building and reusing a library of LoRAs\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pp\. 38885–38904, 2024\.URL[https://proceedings\.mlr\.press/v235/ostapenko24a\.html](https://proceedings.mlr.press/v235/ostapenko24a.html)\.
- Qiu et al\. \(2025\)Zihan Qiu, Zeyu Huang, Bo Zheng, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, and Junyang Lin\.Demons in the detail: On implementing load balancing loss for training specialized mixture\-of\-expert models\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics*, pp\. 5005–5018, Vienna, Austria, 2025\. Association for Computational Linguistics\.doi:10\.18653/v1/2025\.acl\-long\.249\.URL[https://aclanthology\.org/2025\.acl\-long\.249/](https://aclanthology.org/2025.acl-long.249/)\.arXiv:2501\.11873\.
- Roller et al\. \(2021\)Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston\.Hash layers for large sparse models\.In*Advances in Neural Information Processing Systems*, volume 34, 2021\.arXiv:2106\.04426\.
- Röttger et al\. \(2024\)Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy\.XSTest: A test suite for identifying exaggerated safety behaviours in large language models\.In*Proceedings of NAACL\-HLT*, 2024\.arXiv:2308\.01263\.
- Sen \(2026\)Nevroz Sen\.Bridging game theory and transformer routing: Mean field equilibria for mixture of experts\.In*NExT\-Game Workshop at ICML*, 2026\.URL[https://openreview\.net/pdf/e1a5acd239f250576298ae0057d2bf278b212da1\.pdf](https://openreview.net/pdf/e1a5acd239f250576298ae0057d2bf278b212da1.pdf)\.
- Shazeer et al\. \(2017\)Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean\.Outrageously large neural networks: The sparsely\-gated mixture\-of\-experts layer\.In*International Conference on Learning Representations*, 2017\.
- Stoica et al\. \(2025\)George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman\.Model merging with SVD to tie the KnOTS\.In*International Conference on Learning Representations*, 2025\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2025/hash/0d4f8a5109c5083b5307fcd0bddae7a7\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2025/hash/0d4f8a5109c5083b5307fcd0bddae7a7-Abstract-Conference.html)\.
- Sukhbaatar et al\. \(2024\)Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen\-tau Yih, Jason Weston, and Xian Li\.Branch\-train\-MiX: Mixing expert LLMs into a mixture\-of\-experts LLM\.In*Conference on Language Modeling*, 2024\.URL[https://openreview\.net/forum?id=nqLAuMOF6n](https://openreview.net/forum?id=nqLAuMOF6n)\.arXiv:2403\.07816\.
- Vyas \(2025\)Dev Vyas\.MoB: Mixture of bidders\.*arXiv preprint arXiv:2512\.10969*, 2025\.URL[https://arxiv\.org/abs/2512\.10969](https://arxiv.org/abs/2512.10969)\.
- Wang et al\. \(2026\)Xi Wang, Soufiane Hayou, and Eric Nalisnick\.The myth of expert specialization in MoEs: Why routing reflects geometry, not necessarily domain expertise\.*arXiv preprint arXiv:2604\.09780*, 2026\.
- Wang et al\. \(2022\)Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao\.AdaMix: Mixture\-of\-adaptations for parameter\-efficient model tuning\.In*Proceedings of EMNLP*, pp\. 5744–5760, 2022\.URL[https://aclanthology\.org/2022\.emnlp\-main\.388/](https://aclanthology.org/2022.emnlp-main.388/)\.
- Wang et al\. \(2024\)Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin\.Do\-not\-answer: Evaluating safeguards in LLMs\.In*Findings of the Association for Computational Linguistics: EACL 2024*, pp\. 896–911, 2024\.doi:10\.18653/v1/2024\.findings\-eacl\.61\.URL[https://aclanthology\.org/2024\.findings\-eacl\.61/](https://aclanthology.org/2024.findings-eacl.61/)\.
- Wu et al\. \(2024\)Xun Wu, Shaohan Huang, and Furu Wei\.Mixture of LoRA experts\.In*International Conference on Learning Representations*, 2024\.arXiv:2404\.13628\.
- Xue et al\. \(2024\)Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You\.OpenMoE: An early effort on open mixture\-of\-experts language models\.In*Proceedings of the 41st International Conference on Machine Learning*, 2024\.arXiv:2402\.01739\.
- Yadav et al\. \(2023\)Prateek Yadav, Derek Tam, Leshem Choshen, Colin A\. Raffel, and Mohit Bansal\.TIES\-merging: Resolving interference when merging models\.In*Advances in Neural Information Processing Systems*, volume 36, 2023\.URL[https://papers\.nips\.cc/paper/2023/hash/1644c9af28ab7916874f6fd6228a9bcf\-Abstract\-Conference\.html](https://papers.nips.cc/paper/2023/hash/1644c9af28ab7916874f6fd6228a9bcf-Abstract-Conference.html)\.
- Yahma \(2023\)Yahma\.Alpaca\-cleaned\.Hugging Face dataset; cleaned version of Stanford Alpaca, 2023\.URL[https://huggingface\.co/datasets/yahma/alpaca\-cleaned](https://huggingface.co/datasets/yahma/alpaca-cleaned)\.
- Yang et al\. \(2024\)An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al\.Qwen2\.5 technical report\.*arXiv preprint arXiv:2412\.15115*, 2024\.
- Yi et al\. \(2025\)Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu\.Benchmarking and defending against indirect prompt injection attacks on large language models\.In*Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, 2025\.doi:10\.1145/3690624\.3709179\.arXiv:2312\.14197\.
- Yu et al\. \(2024\)Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li\.Language models are super mario: Absorbing abilities from homologous models as a free lunch\.In*Proceedings of the 41st International Conference on Machine Learning*, pp\. 57755–57775, 2024\.URL[https://proceedings\.mlr\.press/v235/yu24p\.html](https://proceedings.mlr.press/v235/yu24p.html)\.
- Zadouri et al\. \(2024\)Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermiş, Acyr Locatelli, and Sara Hooker\.Pushing mixture of experts to the limit: Extremely parameter efficient MoE for instruction tuning\.In*International Conference on Learning Representations*, 2024\.URL[https://proceedings\.iclr\.cc/paper\_files/paper/2024/hash/6d00071564ec447466fc4577743cf1b3\-Abstract\-Conference\.html](https://proceedings.iclr.cc/paper_files/paper/2024/hash/6d00071564ec447466fc4577743cf1b3-Abstract-Conference.html)\.
- Zhang et al\. \(2025\)Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou\.Qwen3 embedding: Advancing text embedding and reranking through foundation models\.*arXiv preprint arXiv:2506\.05176*, 2025\.
- Zhao et al\. \(2025\)Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin, et al\.Qwen3Guard technical report\.*arXiv preprint arXiv:2510\.14276*, 2025\.
- Zhao et al\. \(2024\)Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu\.LoraRetriever: Input\-aware LoRA retrieval and composition for mixed tasks in the wild\.In*Findings of the Association for Computational Linguistics: ACL 2024*, pp\. 4447–4462, Bangkok, Thailand, 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.findings\-acl\.263\.URL[https://aclanthology\.org/2024\.findings\-acl\.263/](https://aclanthology.org/2024.findings-acl.263/)\.
- Zuo et al\. \(2022\)Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao\.Taming sparsely activated transformer with stochastic experts\.In*International Conference on Learning Representations*, 2022\.arXiv:2110\.04260\.

## Appendix AData provenance

Table[2](https://arxiv.org/html/2611.25167#A1.T2)lists the seven public datasets that make up the resource population, one per trait axis, with the retained sample count and the source annotation used to estimate each direction\. The signals label the prompt, not our models: the over\-refusal signal, for instance, distinguishes OR\-Bench’s benign\-but\-seemingly\-toxic prompts from its toxic controls, not whether one of our models refused\.

Table 2:Data used to construct the resource landscape\. Counts are the records retained before the shared split\. “Signal” is the annotation used to learn the corresponding trait direction\.#### What the adapters are trained to produce\.

The fine\-tuning loss covers every non\-padding token of the chat text, prompt included, and the corpus keeps both labels of its three response\-bearing sources\. Table[3](https://arxiv.org/html/2611.25167#A1.T3)lists the resulting targets in the seed\-0 training partition\. About 10% of the 19,848 training records carry an unsafe response as their target and 9% a hallucinated answer, and 47% are prompts with no response, on which the model is trained to reproduce the prompt\. The test partition has the same composition: 2,671 of its 5,671 records come from the four prompt\-only sources and contribute prompt tokens alone, so every NLL in this paper scores full chat text\. Every arm, the pooled adapter included, trains and is scored on this corpus\. Appendix[R](https://arxiv.org/html/2611.25167#A18)traces the behavioral degradation after fine\-tuning to these targets and retrains without them\.

Table 3:Training targets by source, seed\-0 training partition \(19,848 records\)\.

## Appendix BGame\-theoretic derivations

The stability condition of Section[2\.1](https://arxiv.org/html/2611.25167#S2.SS1)and the pairing result used in Section[3](https://arxiv.org/html/2611.25167#S3)are due to[Lovett & Fu \(2026\)](https://arxiv.org/html/2611.25167#bib.bib36); this appendix repeats their derivations in the notation of this paper\.

#### Regularity and symmetric stationarity\.

Let allN≥2N\\geq 2players use the same positive kernelfκ​\(⋅,b,σ\)∈C2f\_\{\\kappa\}\(\\cdot,b;\\sigma\)\\in C^\{2\}on an open neighborhood of their strategy domain, and suppose differentiation can pass through the expectation overBB\. The formulas below are Euclidean; for simplex\-valued positions, all gradients, Hessians, and definiteness statements are restricted to the simplex tangent space\.

Writesi​\(b\)=∇xi​log​fκ​\(xi,b,σ\)s\_\{i\}\(b\)=\\nabla\_\{x\_\{i\}\}\\log f\_\{\\kappa\}\(x\_\{i\},b;\\sigma\)andHi​\(b\)=∇xi2​log⁡fκ​\(xi,b,σ\)H\_\{i\}\(b\)=\\nabla\_\{x\_\{i\}\}^\{2\}\\log f\_\{\\kappa\}\(x\_\{i\},b;\\sigma\)\. Differentiating the normalized share gives

∇xiGi=Gi\(1−Gi\)si,∇xjGi=−GiGjsj\(j≠i\),\\nabla\_\{x\_\{i\}\}G\_\{i\}=G\_\{i\}\(1\-G\_\{i\}\)s\_\{i\},\\qquad\\nabla\_\{x\_\{j\}\}G\_\{i\}=\-G\_\{i\}G\_\{j\}s\_\{j\}\\quad\(j\\neq i\),\(4\)and hence the positional gradient field is

Fi​\(𝐱\):=∇xiui​\(𝐱\)=𝔼B​\[Gi​\(1−Gi\)​si\]\.F\_\{i\}\(\\mathbf\{x\}\):=\\nabla\_\{x\_\{i\}\}u\_\{i\}\(\\mathbf\{x\}\)=\\mathbb\{E\}\_\{B\}\\\!\\left\[G\_\{i\}\(1\-G\_\{i\}\)s\_\{i\}\\right\]\.\(5\)At a symmetric profilex1=⋯=xN=x∘x\_\{1\}=\\cdots=x\_\{N\}=x^\{\\circ\}, every share is1/N1/N, so

∇xiui=N−1N2​𝔼B​\[s⁡\(b\)\]\.\\nabla\_\{x\_\{i\}\}u\_\{i\}=\\frac\{N\-1\}\{N^\{2\}\}\\mathbb\{E\}\_\{B\}\\\!\\left\[s\(b\)\\right\]\.\(6\)Thus an interior symmetric profile is stationary exactly when𝔼B​\[s​\(b\)\]=0\\mathbb\{E\}\_\{B\}\\\!\\left\[s\(b\)\\right\]=0\. Equivalently, it is a stationary point of

Φ⁡\(x\)=𝔼B​\[log⁡fκ​\(x,b,σ\)\]\.\\Phi\(x\)=\\mathbb\{E\}\_\{B\}\\\!\\left\[\\log f\_\{\\kappa\}\(x,b;\\sigma\)\\right\]\.\(7\)If the strategy domain is compact,Φ\\Phiis continuous, and its maximizer is interior, such a point exists; ifΦ\\Phiis strictly concave, it is unique\. On a boundary, Equation[6](https://arxiv.org/html/2611.25167#A2.E6)is replaced by the projected first\-order condition\. These conditions do not certify a global Nash equilibrium or a separated solution, which require best\-response or numerical checks\.

#### Jacobian and player\-separating modes\.

Atx∘x^\{\\circ\}, let

H¯=𝔼B​\[H⁡\(b\)\],C=𝔼B​\[s⁡\(b\)​s​\(b\)⊤\],K=−H¯\.\\bar\{H\}=\\mathbb\{E\}\_\{B\}\\\!\\left\[H\(b\)\\right\],\\qquad C=\\mathbb\{E\}\_\{B\}\\\!\\left\[s\(b\)s\(b\)^\{\\top\}\\right\],\\qquad K=\-\\bar\{H\}\.\(8\)At a stationary pointCCis the score covariance\. Differentiating Equation[5](https://arxiv.org/html/2611.25167#A2.E5)once more and substitutingGi=1/NG\_\{i\}=1/Ngives identical diagonal and off\-diagonal blocks,

A:=∂Fi∂xi⊤\\displaystyle A:=\\frac\{\\partial F\_\{i\}\}\{\\partial x\_\{i\}^\{\\top\}\}=N−1N2​H¯\+\(N−1\)​\(N−2\)N3​C,\\displaystyle=\\frac\{N\-1\}\{N^\{2\}\}\\bar\{H\}\+\\frac\{\(N\-1\)\(N\-2\)\}\{N^\{3\}\}C,Δ:=∂Fi∂xj⊤\\displaystyle\\Delta:=\\frac\{\\partial F\_\{i\}\}\{\\partial x\_\{j\}^\{\\top\}\}=−N−2N3​C,j≠i\.\\displaystyle=\-\\frac\{N\-2\}\{N^\{3\}\}C,\\qquad j\\neq i\.\(9\)Consequently, the fullN​L×N​LNL\\times NLJacobian hasAAon its diagonal andΔ\\Deltaoff its diagonal\. Equivalently,

J=IN⊗A\+\(𝟏𝟏⊤−IN\)⊗Δ\.J=I\_\{N\}\\otimes A\+\(\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\-I\_\{N\}\)\\otimes\\Delta\.The player\-index space splits into the collective directionspan⁡\{𝟏\}\\operatorname\{span\}\\\{\\mathbf\{1\}\\\}and its orthogonal, sum\-zero subspace\. The correspondingL×LL\\times Lblocks are

J\+\\displaystyle J\_\{\+\}=A\+\(N−1\)​Δ=−N−1N2​K,\\displaystyle=A\+\(N\-1\)\\Delta=\-\\frac\{N\-1\}\{N^\{2\}\}K,J−\\displaystyle J\_\{\-\}=A−Δ=N−1N2​\[−K\+N−2N−1​C\],\\displaystyle=A\-\\Delta=\\frac\{N\-1\}\{N^\{2\}\}\\left\[\-K\+\\frac\{N\-2\}\{N\-1\}C\\right\],\(10\)whereJ\+J\_\{\+\}appears once andJ−J\_\{\-\}appears with multiplicityN−1N\-1\. To verify the decomposition, choose an orthogonal matrixPPwhose first row is𝟏⊤/N\\mathbf\{1\}^\{\\top\}/\\sqrt\{N\}\. ThenP​𝟏𝟏⊤​P⊤=diag⁡\(N,0,…,0\)P\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}P^\{\\top\}=\\operatorname\{diag\}\(N,0,\\ldots,0\), and conjugatingJJbyP⊗ILP\\otimes I\_\{L\}yieldsdiag⁡\(J\+,J−,…,J−\)\\operatorname\{diag\}\(J\_\{\+\},J\_\{\-\},\\ldots,J\_\{\-\}\)\.

AssumeK≻0K\\succ 0on the relevant space\. The collective block is then negative definite\. Congruence of the separating block with

−I\+N−2N−1K−1/2CK−1/2\-I\+\\frac\{N\-2\}\{N\-1\}K^\{\-1/2\}CK^\{\-1/2\}shows thatJ−≺0J\_\{\-\}\\prec 0exactly when the depth in Equation[3](https://arxiv.org/html/2611.25167#S2.E3)is below one\. Thus0​p​t<10pt<1gives local asymptotic stability underx˙i=Fi​\(𝐱\)\\dot\{x\}\_\{i\}=F\_\{i\}\(\\mathbf\{x\}\),0​p​t\>10pt\>1gives a separating direction with positive linear growth, and0​p​t=10pt=1is the zero\-eigenvalue boundary\. This proves the stability claim in Section[2\.1](https://arxiv.org/html/2611.25167#S2.SS1)\. In one dimension the same condition is

−𝔼B​\[H⁡\(b\)\]\>N−2N−1​𝔼B​\[s​\(b\)2\]\.\-\\mathbb\{E\}\_\{B\}\\\!\\left\[H\(b\)\\right\]\>\\frac\{N\-2\}\{N\-1\}\\mathbb\{E\}\_\{B\}\\\!\\left\[s\(b\)^\{2\}\\right\]\.\(11\)

#### Two\-player and Gaussian consequences\.

WhenN=2N=2, the score\-scatter terms andΔ\\Deltavanish:J\+=J−=−K/4J\_\{\+\}=J\_\{\-\}=\-K/4\. Hence any interior symmetric stationary point withK≻0K\\succ 0is locally asymptotically stable\.

For the elliptical Gaussian kernel, letQ≻0Q\\succ 0and

log⁡fκ​\(x,b,σ\)=−\(b−x\)⊤​Q​\(b−x\)2​σ2\+const\.\\log f\_\{\\kappa\}\(x,b;\\sigma\)=\-\\frac\{\(b\-x\)^\{\\top\}Q\(b\-x\)\}\{2\\sigma^\{2\}\}\+\\mathrm\{const\}\.Thens=Q⁡\(b−x\)/σ2s=Q\(b\-x\)/\\sigma^\{2\}andH=−Q/σ2H=\-Q/\\sigma^\{2\}\. Equation[6](https://arxiv.org/html/2611.25167#A2.E6)givesx∘=𝔼B​\[b\]x^\{\\circ\}=\\mathbb\{E\}\_\{B\}\\\!\\left\[b\\right\]for an unconstrained interior solution\. WritingΣB=CovB⁡\(b\)\\Sigma\_\{B\}=\\operatorname\{Cov\}\_\{B\}\(b\),

K=Qσ2,C=Q​ΣB​Qσ4,K=\\frac\{Q\}\{\\sigma^\{2\}\},\\qquad C=\\frac\{Q\\Sigma\_\{B\}Q\}\{\\sigma^\{4\}\},so the stability boundary has the closed form

σ∗=\[N−2N−1​λmax​\(Q1/2​ΣB​Q1/2\)\]1/2\.\\sigma^\{\*\}=\\left\[\\frac\{N\-2\}\{N\-1\}\\lambda\_\{\\max\}\\\!\\left\(Q^\{1/2\}\\Sigma\_\{B\}Q^\{1/2\}\\right\)\\right\]^\{1/2\}\.\(12\)The symmetric Gaussian profile is stable forσ\>σ∗\\sigma\>\\sigma^\{\*\}and separating\-unstable forσ<σ∗\\sigma<\\sigma^\{\*\}\. WithQ=IQ=Ithis becomesσ∗=N−2N−1​λmax​\(ΣB\)\\sigma^\{\*\}=\\sqrt\{\\frac\{N\-2\}\{N\-1\}\\lambda\_\{\\max\}\(\\Sigma\_\{B\}\)\}; in one dimension it reduces toσ∗=N−2N−1​σB\\sigma^\{\*\}=\\sqrt\{\\frac\{N\-2\}\{N\-1\}\}\\,\\sigma\_\{B\}\.

#### Exact reduction for co\-located groups\.

Suppose each effective playerppis represented byrrequal\-size clones with the same position and kernel valuefp​\(b\)f\_\{p\}\(b\)\. Each clone receivesfp​\(b\)/\(r​∑qfq​\(b\)\)f\_\{p\}\(b\)/\(r\\sum\_\{q\}f\_\{q\}\(b\)\), and summing its group gives

∑a=1rGp,a​\(b\)=r​fp​\(b\)r​∑qfq​\(b\)=fp​\(b\)∑qfq​\(b\)\.\\sum\_\{a=1\}^\{r\}G\_\{p,a\}\(b\)=\\frac\{rf\_\{p\}\(b\)\}\{r\\sum\_\{q\}f\_\{q\}\(b\)\}=\\frac\{f\_\{p\}\(b\)\}\{\\sum\_\{q\}f\_\{q\}\(b\)\}\.\(13\)Therefore equal\-size groups that remain co\-located have exactly the shares and summed utilities of the reduced game with one player per group, so takingr=2r=2duplicates each solved location without perturbing the effective allocation\. With the two\-player stability result above, this motivates initializing the experts in pairs\. The reduction is an identity on the co\-located subspace; two\-player stability does not prove that two unconstrained members embedded in a larger game would remain together\. The reduction also fails when group sizes differ\. Appendix[C](https://arxiv.org/html/2611.25167#A3)shows why the implemented update preserves equal co\-location and distinguishes the clone\-level gradient from a jointly deviating reduced\-game player\.

## Appendix CGeneral\-kernel game\-gradient updates

The position update uses the payoff in Equation[1](https://arxiv.org/html/2611.25167#S2.E1), not language\-model loss\. Letsi​\(b\)=∇xi​log​fκ​\(xi,b,σ\)s\_\{i\}\(b\)=\\nabla\_\{x\_\{i\}\}\\log f\_\{\\kappa\}\(x\_\{i\},b;\\sigma\)\. The exact population vector field derived in Appendix[B](https://arxiv.org/html/2611.25167#A2)is

∇xiui​\(𝐱\)=𝔼B​\[Gi​\(1−Gi\)​si​\(b\)\]\.\\nabla\_\{x\_\{i\}\}u\_\{i\}\(\\mathbf\{x\}\)=\\mathbb\{E\}\_\{B\}\\\!\\left\[G\_\{i\}\(1\-G\_\{i\}\)s\_\{i\}\(b\)\\right\]\.\(14\)At roundtt, a batchb1,…,bMb\_\{1\},\\ldots,b\_\{M\}replaces the resource expectation, giving the stochastic field

g^i\(t\)=1M​∑m=1MGi​\(𝐱\(t\),bm\)​\[1−Gi​\(𝐱\(t\),bm\)\]​si​\(bm\)\.\\widehat\{g\}\_\{i\}^\{\(t\)\}=\\frac\{1\}\{M\}\\sum\_\{m=1\}^\{M\}G\_\{i\}\(\\mathbf\{x\}^\{\(t\)\},b\_\{m\}\)\\bigl\[1\-G\_\{i\}\(\\mathbf\{x\}^\{\(t\)\},b\_\{m\}\)\\bigr\]s\_\{i\}\(b\_\{m\}\)\.\(15\)For batches sampled fromBB, this is an unbiased estimator of Equation[14](https://arxiv.org/html/2611.25167#A3.E14)\.

#### General update\.

The method applies the simultaneous projected step

𝐱\(t\+1\)=Π𝔹​\(𝐱\(t\)\+η¯t​PT​𝔹​g^\(t\)\)\.\\mathbf\{x\}^\{\(t\+1\)\}=\\Pi\_\{\\mathbb\{B\}\}\\\!\\left\(\\mathbf\{x\}^\{\(t\)\}\+\\bar\{\\eta\}\_\{t\}P\_\{T\\mathbb\{B\}\}\\widehat\{g\}^\{\(t\)\}\\right\)\.\(16\)Each influence family supplies its own scoresi​\(b\)s\_\{i\}\(b\)using the closed forms in Appendix[G](https://arxiv.org/html/2611.25167#A7); no centroid assumption is required\.PT​𝔹P\_\{T\\mathbb\{B\}\}removes components normal to the domain \(notably the all\-ones direction on the simplex\), andΠ𝔹\\Pi\_\{\\mathbb\{B\}\}restores feasibility\. A single population\-wide effective step sizeη¯t\\bar\{\\eta\}\_\{t\}is reduced when necessary so that the largest player movement respects the step cap\. Before the final feasibility projection, this scalar preserves the joint game\-gradient direction and the relative magnitudes of the players’ updates\. The kernel\-family experiments use this rule with the observed round batch\.

Because the batch defines the empirical resource measure in Equation[15](https://arxiv.org/html/2611.25167#A3.E15), a changing prompt distribution changes the vector field in a predictable way\. Repeated projected steps can therefore track a moving resource landscape without learning a new router objective\.

#### Gaussian centroid as a special case\.

For an isotropic Gaussian kernel,si​\(bm\)=\(bm−xi\)/σ2s\_\{i\}\(b\_\{m\}\)=\(b\_\{m\}\-x\_\{i\}\)/\\sigma^\{2\}\. Define

wi​m=Gi​\(𝐱,bm\)​\[1−Gi​\(𝐱,bm\)\],Wi=∑m=1Mwi​m,x^i=∑mwi​m​bmWi\.w\_\{im\}=G\_\{i\}\(\\mathbf\{x\},b\_\{m\}\)\\bigl\[1\-G\_\{i\}\(\\mathbf\{x\},b\_\{m\}\)\\bigr\],\\quad W\_\{i\}=\\sum\_\{m=1\}^\{M\}w\_\{im\},\\quad\\widehat\{x\}\_\{i\}=\\frac\{\\sum\_\{m\}w\_\{im\}b\_\{m\}\}\{W\_\{i\}\}\.\(17\)Then

g^i=WiM​σ2​\(x^i−xi\)\.\\widehat\{g\}\_\{i\}=\\frac\{W\_\{i\}\}\{M\\sigma^\{2\}\}\(\\widehat\{x\}\_\{i\}\-x\_\{i\}\)\.\(18\)The reported principal Gaussian arms evaluate this direction through the blended centroid step

xi\(t\+1\)=\(1−β\)​xi\(t\)\+β​x^i=xi\(t\)\+ηi\(t\)​g^i\(t\),ηi\(t\)=β​M​σ2Wi\>0,x\_\{i\}^\{\(t\+1\)\}=\(1\-\\beta\)x\_\{i\}^\{\(t\)\}\+\\beta\\widehat\{x\}\_\{i\}=x\_\{i\}^\{\(t\)\}\+\\eta\_\{i\}^\{\(t\)\}\\widehat\{g\}\_\{i\}^\{\(t\)\},\\qquad\\eta\_\{i\}^\{\(t\)\}=\\frac\{\\beta M\\sigma^\{2\}\}\{W\_\{i\}\}\>0,\(19\)withβ=0\.5\\beta=0\.5\. This specialization is exactly parallel to the empirical Gaussian utility gradient and has the same stationary points wheneverWi\>0W\_\{i\}\>0\. Its step size is an agent\- and batch\-adaptive positive rescaling rather than the population\-wide learning rate in Equation[16](https://arxiv.org/html/2611.25167#A3.E16), so the two implementations do not have identical step magnitudes\.

The direct general\-kernel rule evaluates Equation[15](https://arxiv.org/html/2611.25167#A3.E15)for every clone and every prompt in the round batch\. Likewise, the dense Gaussian specialization evaluateswi​mw\_\{im\}on the whole batch\. Neither rule truncates its position mass to the top\-kktraining assignments or renormalizes it across retained pairs\.

The reported hard\-routing Gaussian arm uses the corresponding stochastic centroid estimator\. IfZi​mZ\_\{im\}indicates that promptmmwas sampled for cloneii, then

𝔼⁡\[Zi​m​\(1−Gi\)∣bm\]=Gi​\(1−Gi\)\.\\mathbb\{E\}\\\!\\left\[Z\_\{im\}\(1\-G\_\{i\}\)\\mid b\_\{m\}\\right\]=G\_\{i\}\(1\-G\_\{i\}\)\.\(20\)The selected prompts are therefore weighted by\(1−Gi\)\(1\-G\_\{i\}\): their unnormalized centroid mass estimates the same theoretical coefficient, but the finite\-batch ratio is noisier than evaluating the full batch\. Updated positions affect routing beginning with the next round\.

#### Why pairs remain co\-located\.

For two equal clonesaaanda′a^\{\\prime\}at the same position,Ga​\(bm\)=Ga′​\(bm\)G\_\{a\}\(b\_\{m\}\)=G\_\{a^\{\\prime\}\}\(b\_\{m\}\)for every prompt\. Their rows ofwi​mw\_\{im\}, their centroid targets, and therefore their independently computed updates are identical\. Exact co\-location is an invariant of the dense theory\-matched dynamics; the implementation does not average a pair and write one forced position back to both members\.

IfGpG\_\{p\}denotes the summed share of a two\-clone pair, then each member hasGi=Gp/2G\_\{i\}=G\_\{p\}/2and follows the coefficientGi​\(1−Gi\)G\_\{i\}\(1\-G\_\{i\}\)\. A hypothetical player that moved the two clones jointly to maximize their summed utility would instead have coefficientGp​\(1−Gp\)G\_\{p\}\(1\-G\_\{p\}\)\. Equation[13](https://arxiv.org/html/2611.25167#A2.E13)therefore gives an exact reduction of allocations and summed utilities on the co\-located subspace, but not an identity between these two gradient fields\. The implemented update matches each clone’s own utility gradient in the fourteen\-player game, while routing and adapter training operate on seven aggregated pairs\.

#### Group structure of the position game\.

Figure[6](https://arxiv.org/html/2611.25167#A3.F6)solves the position game alone, with no adapter training, for eight individual players under an isotropic Gaussian kernel on the empirical measure of all 28,355 prompts, for whichσ∗=0\.356\\sigma^\{\*\}=0\.356\. Each of 41 reaches from2​σ∗2\\sigma^\{\*\}to0\.05​σ∗0\.05\\sigma^\{\*\}is solved independently from the same seeded start with the blended centroid step of Equation[19](https://arxiv.org/html/2611.25167#A3.E19), for up to 8,000 steps and without perturbation, on three seeds\. Players withinmax⁡\(0\.075,0\.35​σ\)\\max\(0\.075,\\,0\.35\\sigma\)of one another are counted as one group\. From free starts, each player at a random prompt, the eight players stay together down toσ∗\\sigma^\{\*\}, separate into five and three just below it and fragment into pairs and single players by0\.6​σ∗0\.6\\sigma^\{\*\}; below about0\.3​σ∗0\.3\\sigma^\{\*\}many solves do not settle within the budget and the seeds disagree\. From four co\-located pairs, the players pass through6\+26\+2and4\+2\+24\+2\+2and settle into four pairs from about0\.7​σ∗0\.7\\sigma^\{\*\}to0\.05​σ∗0\.05\\sigma^\{\*\}; every one of these solves converges and no pair separates, as the invariance above requires\.

Figure 6:Co\-located groups of eight players in the Gaussian position game against the reach, with no adapter training\. Each branch is a group of co\-located players, colored and labeled by its member count, and forks where the group splits; the reach shrinks downward\. Left, every player starts at a random prompt; right, the players start as four co\-located pairs\. Both trees use the same 41 reaches and three seeds per reach, and each row draws the most common partition\. Dashed branches did not settle within 8,000 steps; the right margin gives seed agreement and the number of seeds that cycled or did not settle\.

## Appendix DTrait\-space geometry

The cube retains seven independently calibrated coordinate levels, whereas the simplex encodes their relative composition\. For calibrated scoresc∈\[0,1\]7c\\in\[0,1\]^\{7\}, the latter uses

bj​\(T\)=exp⁡\(cj/T\)∑ℓ=17exp⁡\(cℓ/T\),T\>0\.b\_\{j\}\(T\)=\\frac\{\\exp\(c\_\{j\}/T\)\}\{\\sum\_\{\\ell=1\}^\{7\}\\exp\(c\_\{\\ell\}/T\)\},\\qquad T\>0\.\(21\)Figure[7](https://arxiv.org/html/2611.25167#A4.F7)shows the held\-out resource density in these spaces\. Each ternary view conditionally renormalizes its three named components; the other four components of the full seven\-axis representation are not assumed to be zero\. AtT=1T=1the bounded logits confine each displayed ternary component to approximately\[0\.155,0\.576\]\[0\.155,0\.576\], crowding the prompt cloud into the interior\. AtT=0\.35T=0\.35that possible range expands to approximately\[0\.028,0\.897\]\[0\.028,0\.897\], and the measured support spreads toward the corners\.

![Refer to caption](https://arxiv.org/html/2611.25167v1/x3.png)Figure 7:The trait projection controls how much geometric separation the kernel can exploit\. Columns show three benchmark groupings among 5,671 held\-out routing prompts: two\-coordinate cube projections \(top\), followed by conditional three\-component simplex views atT=1T=1\(middle\) andT=0\.35T=0\.35\(bottom\)\. Grey is the combined prompt density; colored contours and markers denote benchmark origin, with each class normalized to its own density mode and therefore not encoding prevalence\. Each triangle renormalizes only its three named components; the other four coordinates are not assumed to be zero\. TheT=1T=1transform compresses prompts near the simplex center, whereasT=0\.35T=0\.35spreads them toward distinct vertices\. These are resource densities, not expert territories, and demonstrate why denser geometry requires a more peaked influence kernel to obtain selective routing\.This geometry affects how peaked an influence kernel must be to distinguish competitors\. For the reported Gaussian arms with identity shape matrix, the allocation log\-odds obey

log⁡Gi​\(b\)Gj​\(b\)=∥b−xj∥2−∥b−xi∥22​σ2\.\\log\\frac\{G\_\{i\}\(b\)\}\{G\_\{j\}\(b\)\}=\\frac\{\\lVert b\-x\_\{j\}\\rVert^\{2\}\-\\lVert b\-x\_\{i\}\\rVert^\{2\}\}\{2\\sigma^\{2\}\}\.\(22\)When resources are concentrated where rival positions have similar affinity, a broad kernel makes these odds small and spreads weight across experts\. Narrowing effective reach, increasing kernel tempering, or using a less compressive trait projection can produce a more discriminative allocation\. This can lower expected single\-expert NLL when diffuse routing gives weight to weaker experts, as the sharpness\-matchedT=1T=1comparison and theT=0\.35T=0\.35end\-to\-end arms in Figure[3](https://arxiv.org/html/2611.25167#S4.F3)suggest\. A compact cloud already close to one expert can route sharply, and extra peaking need not improve probability mixing or preserve adequate training mass for every expert\.

With a fixed trait encoder and expert family, reachσ\\sigmaor a tempering exponent can be validated as router controls while monitoring allocation entropy, expert loads, and the intended inference NLL\. Trait temperatureTTis a separate geometry lever, but changing it also changes the position game; positions must be re\-solved or recalibrated, and the resulting router tested against the same frozen experts\. The reportedT=0\.35T=0\.35arms instead retrained experts under the new geometry, so they do not establish a free post\-training gain\.

An end\-to\-end simplex experiment shows higher router–oracle agreement \(about 0\.87 versus 0\.71\) and mixture capture \(91\.5–92\.6% versus 82\.0–84\.9%\) than its cube counterpart\. A bidirectional frozen\-expert cross\-routing control qualifies that comparison\. Simplex\-trained experts score 1\.9383 NLL under their own geometry versus 2\.0739 under cube routing, while cube\-trained experts score 1\.9465 under cube versus 2\.0244 under simplex, so each family prefers the geometry in which it trained\. Absolute reach and a density\-estimation routine also differ between the two arms, and one reverse solve did not converge\.

Appendix[G](https://arxiv.org/html/2611.25167#A7)gives the kernel closed forms and the distinct ways their parameters control reach and sharpness\.

## Appendix EAn inductive trait space

Every other trait space in this paper is fitted on all 28,355 records \(Section[3](https://arxiv.org/html/2611.25167#S3)\): the axis directions on all benchmark labels, and the quantile maps and resource KDE on all prompts, so the test prompts help shape the space that routes them\. We refit all three on the training partition only \(trait\_space\.fit\_partition: train\) and rerun the seven\-pairT=0\.35T=0\.35Dirichlet arm with nothing else changed, on three data splits; reach is held at its transductive value, and the start is re\-solved inside the train\-only space so no position carries test information\. The train\-only axes differ noticeably from the originals \(\|cos\|\|\\cos\|0\.86–0\.97\) while the resource distribution moves little \(total variation 2\.2–2\.4%\)\. Table[4](https://arxiv.org/html/2611.25167#A5.T4)shows the result: the share\-weighted single expert loses−0\.0091±0\.0027\-0\.0091\\pm 0\.0027nats, with the same sign on every split, about 8% of the gain; the mixture loses−0\.0042\-0\.0042and top\-1 agreement \-1\.6 points, while the hindsight\-best expert does not move \(0\.0000±0\.00090\.0000\\pm 0\.0009\)\. The transductive fit therefore inflates routing, not expert quality, and routed levels elsewhere in the paper should be read as about 0\.007–0\.012 nats optimistic; comparisons between arms that share one trait space give both sides the same advantage\.

Table 4:Trait space fitted on all records \(all\) or on the training partition only \(train\), seven\-pairT=0\.35T=0\.35Dirichlet arm, test partition\. Gains in nats \(higher is better\); top\-1 in percent\.
## Appendix FLabel\-free trait axes

The seven axes of Section[3](https://arxiv.org/html/2611.25167#S3)are fitted on benchmark annotations\. Here we replace them with seven directions of the same prompt embeddings that use no labels and are fitted on the training partition only\. Everything downstream is unchanged: the quantile maps, theT=0\.35T=0\.35simplex, the resource distribution, one expert pair per axis, and a reach re\-solved to the principal arm’s instability depth\. Each arm is retrained on three data splits and scored against its own split’s pooled adapter\.

#### Cluster\-derived axes\.

The game places experts on the modes of the resource distribution, so our main label\-free space is built to find modes\. The training prompts’ embeddings are reduced to their top 50 principal components and grouped bykk\-means withK=7K=7clusters and 10 restarts\. One shrinkage Fisher direction then separates each cluster from the remaining prompts, orthogonalized against the directions already found\. This is the labelled construction of Section[3](https://arxiv.org/html/2611.25167#S3)with benchmark identity replaced by cluster identity, and we refer to the result as pseudo\-Fisher axes\. The clusters recover most of the benchmark structure without seeing it: each is mostly one benchmark, with mean purity 0\.82\.

#### PCA and ICA axes\.

Two simpler spaces serve as comparisons: the top seven principal components of the training prompts’ embeddings, and the seven most non\-Gaussian of 20 whitened independent components\.

Figure 8:The game on labelled and label\-free trait axes, seven pairs,T=0\.35T=0\.35\.aGain over each split’s pooled adapter; markers are means over three data splits and bands one sample standard deviation\. One sampled expert is the share\-weighted loss of sampling one pair, and the hindsight\-best expert is the oracle\. The dense gate is a dense softmax gate trained end to end on the same axes with one expert per axis; it was not trained on the train\-only labelled axes\. The hollow marker is the token\-level mixture on the labelled axes, which exists for the seed\-0 split only\. The dense gate’s lead over the token\-level mixture is0\.034±0\.0060\.034\\pm 0\.006nats on pseudo\-Fisher axes,0\.072±0\.0040\.072\\pm 0\.004on PCA and0\.059±0\.0050\.059\\pm 0\.005on ICA, and 0\.012 on the labelled axes of the seed\-0 split\.bFraction of test prompts whose best expert receives the largest share, against a shuffled null of 0\.15\.
#### Routing gain\.

Figure[8](https://arxiv.org/html/2611.25167#A6.F8)gives the result\. On pseudo\-Fisher axes the game keeps about 83% of its train\-only labelled gain and identifies each prompt’s best expert 81% of the time\. PCA keeps about 70%, and the pseudo\-Fisher game improves on it on every split and estimator \(paired,0\.016±0\.0020\.016\\pm 0\.002nats expected and0\.017±0\.0020\.017\\pm 0\.002token level\)\. PCA models the data as one ellipsoid and returns directions of largest overall variance, which run through several clusters at once, so the prompts spread over the interior of the simplex and the game has fewer distinct modes to compete over \(Figure[9](https://arxiv.org/html/2611.25167#A6.F9)\)\. ICA gives almost no routing gain\. The axes therefore need not be hand\-assigned, but they must carry task\-relevant, multimodal variation\. Ten clusters are worse than seven for the game on the seed\-0 split \(expected gain 0\.0581\), where one pair receives 0\.2% of the load and the extra axes force a three times smaller reach\. Six pairs on the seven pseudo\-Fisher axes change the seed\-0 expected gain from 0\.0908 to 0\.0867\. Over three training seeds on the seed\-0 split, the pseudo\-Fisher game’s token\-level mixture scores1\.9070±0\.00111\.9070\\pm 0\.0011\.

![Refer to caption](https://arxiv.org/html/2611.25167v1/trait_simplex_slices_spaces.png)Figure 9:Seed\-0 held\-out prompts on the three benchmark triplets of Figure[2](https://arxiv.org/html/2611.25167#S3.F2), as conditional views of three simplex components atT=0\.35T=0\.35\. Grey is the smoothed density of the triplet’s prompts; colored contours and centroid markers give the benchmark of origin\. Rows are labelled Fisher axes \(a–c\), label\-free PCA axes \(d–f\) and label\-free pseudo\-Fisher axes \(g–i\), the last two fitted on the training partition only\. In the label\-free rows each corner is the axis that best separates that benchmark, with its AUC\. On PCA axes privacy and over\-refusal share one region \(e\) and injection overlaps jailbreak \(f\); on pseudo\-Fisher axes they return to their own corners\.
#### What the label\-free axes encode\.

Each PCA axis tracks one benchmark, with the same assignment on all three splits\. Benchmark\-versus\-rest AUC is 0\.95 for over\-refusal on the first component, 0\.92 for prompt injection, 0\.89 for privacy, 0\.84 for policy violation, 0\.84 for harm, and 0\.78 and 0\.71 for hallucination on the last two; only jailbreak has no axis of its own\. The PCA subspace is nonetheless nearly orthogonal to the labelled Fisher subspace \(principal angles 76–90∘\), so different directions of the 4096\-dimensional embedding carry overlapping task information\. On pseudo\-Fisher axes privacy, over\-refusal and policy violation separate with AUC 0\.99, 0\.97 and 0\.87, and the one impure cluster is harm \(purity 0\.75\)\. Expert purity follows the axes: six of seven labelled experts win at least 85% of their prompts from one benchmark \(Table[24](https://arxiv.org/html/2611.25167#A18.T24)\), against two of seven on PCA axes and none on ICA axes\.

#### Learned gates on the same axes\.

A dense softmax gate trained end to end on each label\-free space, at router learning rate10−210^\{\-2\}and a1×161\\times 16micro\-batch, reaches a lower NLL than the game \(dashed line in Figure[8](https://arxiv.org/html/2611.25167#A6.F8)a\)\. Its lead is0\.072±0\.0040\.072\\pm 0\.004nats on PCA axes and halves to0\.034±0\.0060\.034\\pm 0\.006on pseudo\-Fisher axes, where the gate itself does0\.022±0\.0060\.022\\pm 0\.006worse than on PCA\. Over three training seeds on the seed\-0 split the pseudo\-Fisher dense gate scores1\.8740±0\.00541\.8740\\pm 0\.0054, a paired gap to the game of0\.033±0\.0050\.033\\pm 0\.005, so the game is about five times more seed\-stable\. As on labelled axes \(Appendix[L](https://arxiv.org/html/2611.25167#A12)\), the gate’s components do not specialize: on PCA axes its hindsight\-best component \(2\.119\) is worse than its blended network, and every component alone is worse than the pooled adapter\. The other gates trained end to end on the pseudo\-Fisher axes on the seed\-0 split score 1\.8775 \(Gaussian\), 1\.8966 \(DeepSeek\-V3\), 1\.9167 \(Mixtral top\-2\) and 1\.9753 \(Switch top\-1\), against 1\.8692 for the dense gate and 1\.9074 for the game’s token\-level mixture, and their components also work only when blended\. c\-BTM loses its whole gain on PCA axes \(2\.0196 against the pooled 2\.0154\), so the game’s lead over it widens from 0\.058 nats on labelled axes to 0\.079\.

#### Gates fitted on the game’s label\-free experts\.

Table[5](https://arxiv.org/html/2611.25167#A6.T5)repeats the post hoc comparison of Appendix[L](https://arxiv.org/html/2611.25167#A12)on the label\-free spaces: the published gates are fitted on validation losses over the game’s frozen experts and scored on the test pool\. The distance gates again route the game’s experts slightly better than the game’s own shares, by 0\.005 nats on pseudo\-Fisher axes and 0\.008 on PCA as a sequence mixture\. Mixed per token on the pseudo\-Fisher experts, the X\-MoE and Gaussian gates score 1\.8949 and 1\.8950 against 1\.9071 for the game’s shares\. Better routing over the game’s experts therefore closes 0\.012 of the dense gate’s 0\.038 seed\-0 lead on these axes, and the remainder lies in what the jointly trained network learns\.

Table 5:Gates fitted post hoc on the game’s frozen label\-free experts, seed\-0 split, test NLL \(lower is better; pooled adapter 2\.0154\)\. Fitted on the 2,836 validation records, means over three gate seeds\. Top\-1 is agreement with the hindsight\-best expert\.

## Appendix GInfluence kernels and design rationale

All kernel families used by the game specify a positive influencefκ​\(x,b\)f\_\{\\kappa\}\(x,b\); Equation[1](https://arxiv.org/html/2611.25167#S2.E1)normalizes their values across players\. The solver uses the position scores=∇x​log​fκs=\\nabla\_\{x\}\\log f\_\{\\kappa\}and log\-HessianH=∇x2​log⁡fκH=\\nabla\_\{x\}^\{2\}\\log f\_\{\\kappa\}to evaluate the game gradient and the local instability criterion\. Belowσ\>0\\sigma\>0is competitive reach, and all derivatives are with respect toxx\.

For the three distance families, letM≻0M\\succ 0be a fixed shape matrix,Q=M−1Q=M^\{\-1\},v=b−xv=b\-x,q=Q​vq=Qv,d2=v⊤​Q​vd^\{2\}=v^\{\\top\}Qv, andu=δ2\+d2u=\\delta^\{2\}\+d^\{2\}with smoothing radiusδ\>0\\delta\>0\. Their implemented log\-influences are

log⁡fG​\(x,b\)\\displaystyle\\log f\_\{\\mathrm\{G\}\}\(x,b\)=−d22​σ2,\\displaystyle=\-\\frac\{d^\{2\}\}\{2\\sigma^\{2\}\},Gaussian,\\displaystyle\\text\{Gaussian\},\(23\)log⁡fH​\(x,b\)\\displaystyle\\log f\_\{\\mathrm\{H\}\}\(x,b\)=−uσ,\\displaystyle=\-\\frac\{\\sqrt\{u\}\}\{\\sigma\},hyperbolic,\\displaystyle\\text\{hyperbolic\},\(24\)log⁡fEP,β​\(x,b\)\\displaystyle\\log f\_\{\\mathrm\{EP\},\\beta\}\(x,b\)=−uβ/2β​σβ,\\displaystyle=\-\\frac\{u^\{\\beta/2\}\}\{\\beta\\sigma^\{\\beta\}\},exponential power,β≥1\.\\displaystyle\\text\{exponential power\},\\quad\\beta\\geq 1\.\(25\)The Gaussian is the conventional smooth, constant\-curvature reference; its omitted Gaussian\-density normalizer is independent of position and cancels fromGiG\_\{i\}\. The smoothed hyperbolic family gives a different radial tail and finite curvature whenb=xb=x\. Exponential power varies tail shape within one family, so it tests whether allocation sharpness, rather than a nominal kernel label, accounts for routing differences\. Atβ=1\\beta=1it is exactly hyperbolic; atβ=2\\beta=2it differs from Gaussian only by the position\-independent term−δ2/\(2σ2\)\-\\delta^\{2\}/\(2\\sigma^\{2\}\), so allocation, score, and Hessian coincide\. The constraintβ≥1\\beta\\geq 1preserves the log\-concavity used in the local analysis\. In the reported distance\-kernel armsM=IM=I; the usual smoothing radius isδ=0\.1\\delta=0\.1, with a smaller\-radius hyperbolic control analyzed separately\.

The closed\-form derivatives of these three log\-influences are

sG\\displaystyle s\_\{\\mathrm\{G\}\}=qσ2,\\displaystyle=\\frac\{q\}\{\\sigma^\{2\}\},HG\\displaystyle H\_\{\\mathrm\{G\}\}=−Qσ2,\\displaystyle=\-\\frac\{Q\}\{\\sigma^\{2\}\},\(26\)sH\\displaystyle s\_\{\\mathrm\{H\}\}=qσ​u,\\displaystyle=\\frac\{q\}\{\\sigma\\sqrt\{u\}\},HH\\displaystyle H\_\{\\mathrm\{H\}\}=−Qσ​u\+q​q⊤σ​u3/2,\\displaystyle=\-\\frac\{Q\}\{\\sigma\\sqrt\{u\}\}\+\\frac\{qq^\{\\top\}\}\{\\sigma u^\{3/2\}\},\(27\)sEP,β\\displaystyle s\_\{\\mathrm\{EP\},\\beta\}=uβ/2−1​qσβ,\\displaystyle=\\frac\{u^\{\\beta/2\-1\}q\}\{\\sigma^\{\\beta\}\},HEP,β\\displaystyle H\_\{\\mathrm\{EP\},\\beta\}=−uβ/2−1​Q\+\(β−2\)​uβ/2−2​q​q⊤σβ\.\\displaystyle=\-\\frac\{u^\{\\beta/2\-1\}Q\+\(\\beta\-2\)u^\{\\beta/2\-2\}qq^\{\\top\}\}\{\\sigma^\{\\beta\}\}\.\(28\)
The*Dirichlet*family treats a prompt as a composition on the open simplex:b∈Δ∘L−1b\\in\\Delta\_\{\\circ\}^\{L\-1\}andx∈ΔL−1x\\in\\Delta^\{L\-1\}\. Withαℓ=1\+xℓ/σ\\alpha\_\{\\ell\}=1\+x\_\{\\ell\}/\\sigmaandA=∑ℓαℓ=L\+1/σA=\\sum\_\{\\ell\}\\alpha\_\{\\ell\}=L\+1/\\sigmaon the simplex,

fD​\(x,b\)=Γ⁡\(A\)∏ℓ=1LΓ⁡\(αℓ\)​∏ℓ=1Lbℓαℓ−1\.f\_\{\\mathrm\{D\}\}\(x,b\)=\\frac\{\\Gamma\(A\)\}\{\\prod\_\{\\ell=1\}^\{L\}\\Gamma\(\\alpha\_\{\\ell\}\)\}\\prod\_\{\\ell=1\}^\{L\}b\_\{\\ell\}^\{\\alpha\_\{\\ell\}\-1\}\.\(29\)The mode is parameterized byxx\. It tests a simplex\-native, non\-Euclidean influence law against the radial distance kernels on the same prompt geometry\. Unlike the Gaussian prefactor, the Gamma normalizer depends onxxand must remain in the allocation\. Writingψ\\psiandψ1\\psi\_\{1\}for the digamma and trigamma functions, the ambient derivatives implemented before tangent projection are

sD,ℓ=log⁡bℓ−ψ⁡\(αℓ\)\+ψ⁡\(A\)σ,HD=ψ1​\(A\)​𝟏𝟏⊤−diag⁡\(ψ1​\(αℓ\)\)σ2\.s\_\{\\mathrm\{D\},\\ell\}=\\frac\{\\log b\_\{\\ell\}\-\\psi\(\\alpha\_\{\\ell\}\)\+\\psi\(A\)\}\{\\sigma\},\\qquad H\_\{\\mathrm\{D\}\}=\\frac\{\\psi\_\{1\}\(A\)\\mathbf\{1\}\\mathbf\{1\}^\{\\top\}\-\\operatorname\{diag\}\(\\psi\_\{1\}\(\\alpha\_\{\\ell\}\)\)\}\{\\sigma^\{2\}\}\.\(30\)
The*product\-Beta*family instead treats coordinates as separately bounded traits\. Forx∈\[0,1\]Lx\\in\[0,1\]^\{L\},b∈\(0,1\)Lb\\in\(0,1\)^\{L\}, letaℓ=1\+xℓ/σa\_\{\\ell\}=1\+x\_\{\\ell\}/\\sigmaandcℓ=1\+\(1−xℓ\)/σc\_\{\\ell\}=1\+\(1\-x\_\{\\ell\}\)/\\sigma\. Then

fPB​\(x,b\)=∏ℓ=1LΓ⁡\(aℓ\+cℓ\)Γ⁡\(aℓ\)​Γ​\(cℓ\)​bℓaℓ−1​\(1−bℓ\)cℓ−1\.f\_\{\\mathrm\{PB\}\}\(x,b\)=\\prod\_\{\\ell=1\}^\{L\}\\frac\{\\Gamma\(a\_\{\\ell\}\+c\_\{\\ell\}\)\}\{\\Gamma\(a\_\{\\ell\}\)\\Gamma\(c\_\{\\ell\}\)\}b\_\{\\ell\}^\{a\_\{\\ell\}\-1\}\(1\-b\_\{\\ell\}\)^\{c\_\{\\ell\}\-1\}\.\(31\)Each factor has modexℓx\_\{\\ell\}, with its position\-dependent normalizer retained\. It is the bounded, coordinatewise contrast to the coupled Dirichlet composition\. Its native density is on the cube; simplex experiments evaluate the same positive function on simplex\-embedded coordinates, which does not make it a Dirichlet or a normalized simplex density\. Its implemented derivatives are

sPB,ℓ=log⁡bℓ−log⁡\(1−bℓ\)−ψ⁡\(aℓ\)\+ψ⁡\(cℓ\)σ,HPB=−diag⁡\(ψ1​\(aℓ\)\+ψ1​\(cℓ\)\)σ2\.s\_\{\\mathrm\{PB\},\\ell\}=\\frac\{\\log b\_\{\\ell\}\-\\log\(1\-b\_\{\\ell\}\)\-\\psi\(a\_\{\\ell\}\)\+\\psi\(c\_\{\\ell\}\)\}\{\\sigma\},\\qquad H\_\{\\mathrm\{PB\}\}=\-\\frac\{\\operatorname\{diag\}\(\\psi\_\{1\}\(a\_\{\\ell\}\)\+\\psi\_\{1\}\(c\_\{\\ell\}\)\)\}\{\\sigma^\{2\}\}\.\(32\)
The implementation also permits an optional*tempering*wrapper for any base family:

fκ,τ​\(x,b\)=fκ​\(x,b\)τ,sκ,τ=τ​sκ,Hκ,τ=τ​Hκ,τ\>0\.f\_\{\\kappa,\\tau\}\(x,b\)=f\_\{\\kappa\}\(x,b\)^\{\\tau\},\\qquad s\_\{\\kappa,\\tau\}=\\tau s\_\{\\kappa\},\\qquad H\_\{\\kappa,\\tau\}=\\tau H\_\{\\kappa\},\\qquad\\tau\>0\.\(33\)It provides a separate allocation\-sharpness control\. For exponential power it is equivalent to reachσ/τ1/β\\sigma/\\tau^\{1/\\beta\}, but for Dirichlet and product\-Beta the Gamma normalizers preclude that simple rescaling\. This kernel exponentτ\\tauis distinct from the trait\-projection softmax temperatureTTin Figure[7](https://arxiv.org/html/2611.25167#A4.F7)\. Across families we solve reach at a matched instability depth and then measure prompt\-level allocation sharpness\.

Appendix Figure[10](https://arxiv.org/html/2611.25167#A7.F10)expands the calibration result in Figure[3](https://arxiv.org/html/2611.25167#S4.F3)\(right\)\. The main figure reports expected routing gain in nats; this diagnostic instead measures the fraction of available hindsight\-oracle headroom captured\. For each prompt letH\(G\(b\)\)=−∑iGi\(b\)logGi\(b\)H\(G\(b\)\)=\-\\sum\_\{i\}G\_\{i\}\(b\)\\log G\_\{i\}\(b\)and defineKeff=exp⁡\(𝔼B​\[H⁡\(G⁡\(b\)\)\]\)K\_\{\\mathrm\{eff\}\}=\\exp\\\!\\left\(\\mathbb\{E\}\_\{B\}\\\!\\left\[H\(G\(b\)\)\\right\]\\right\); lowerKeffK\_\{\\mathrm\{eff\}\}means a sharper allocation, not more balanced expert loads\. On six nondegenerate distance kernels atT=1T=1, the descriptive fit is

capture^\(%\)=73\.15/Keff\+12\.07,\\widehat\{\\mathrm\{capture\}\}\(\\%\)=73\.15/K\_\{\\mathrm\{eff\}\}\+12\.07,\(34\)withr=0\.9995r=0\.9995\. Two shape kernels were held out of that fit, and theT=0\.35T=0\.35points test a changed trait projection\. Theβ=4\\beta=4endpoint has collapsed positions\.

Figure 10:Oracle\-headroom capture is governed primarily by the sharpness of the induced allocation\. \(a\) The horizontal axis is inverse effective expert count,1/Keff1/K\_\{\\mathrm\{eff\}\}, computed from prompt\-level allocation entropy; larger values mean more decisive routing\. Captured headroom lies almost on a one\-dimensional trend for six nondegenerateT=1T=1distance kernels \(dashed fit,r=0\.9995r=0\.9995\)\. Hollow shape\-kernel points andT=0\.35T=0\.35triangles were not used to fit the line but follow the same relationship\. Theβ=4\\beta=4cross is a collapsed\-position boundary and is excluded\. \(b\) Sharpening the trait projection fromT=1T=1toT=0\.35T=0\.35raises all five families to approximately 81% capture; labels give the percentage\-point increases\. The disappearance of the kernel ranking supports allocation sharpness, rather than family name or matched local depth alone, as the operative deployment quantity\.
## Appendix HInitialization and position controls

The naive adaptive, equilibrium adaptive and pinned starts describe where the pairs begin, and positions adapt during training under all three; fixed describes what is held constant afterward\. The naive adaptive start used in the pair\-count arms came from an approximate lattice\-resource solve whose density estimator divided each cell by a nearest\-prompt kernel value, flattening the resource measure\. These positions are not purely random, but they are not a reliable equilibrium of the measured prompt distribution\. The main text calls this the unsolved start; appendix figures and tables keep the label naive adaptive\. The equilibrium adaptive start instead numerically solves the position game against the empirical prompt measure before adapter training\.

Figure[12](https://arxiv.org/html/2611.25167#A8.F12)compares these positions at two, five, and seven pairs; every plotted training trajectory begins from the naive adaptive start and ends at its round\-11 position\. The number of dominant represented trait axes grows from two to four to six, so additional pairs expand trait coverage rather than merely copy a single location\. The direct NLL contrast between starts is in Figure[3](https://arxiv.org/html/2611.25167#S4.F3)\.

The pinned start begins every reach on a data split from that split’sσ=0\.50\\sigma=0\.50unsolved configuration\. It does not disable position updates, so positions adapt from it as they do from the equilibrium adaptive start\. Figure[11](https://arxiv.org/html/2611.25167#A8.F11)tracks the matched distance between pinned and equilibrium adaptive runs at the same reach\. Neither kind of run is at a fixed point after 11 rounds, because the update uses a sampled batch and a constant blend of 0\.5, but the equilibrium adaptive runs sit 0\.012–0\.019 from the fixed point of their full\-batch update, while the pinned runs sit 0\.038–0\.155 from theirs and would need 79–413 further rounds to reach it\. Continuing the update without training, which is exact because it never reads an adapter, carries 15 of the 27 non\-degenerate pinned runs to the same fixed point as their equilibrium adaptive counterpart\. The other 12, all at reaches of 0\.50 or below and all but one on data splits 1 and 2, settle 0\.16–0\.18 away at a different fixed point\. By round 11 the pinned runs close 78% of their initial distance on data split 0 but only 30% and 21% on splits 1 and 2\. Section[4\.1](https://arxiv.org/html/2611.25167#S4.SS1)reports the roughness and mean\-gain comparison\.

Figure 11:Pinned runs approach the equilibrium adaptive runs on one data split and stall on another\. Top rows: the seven pairs at reach 0\.50 on data splits 0 and 2, matched by nearest round\-11 position\. Grey dashed polygons are the pinned start, orange polygons the pinned run at round 11 and blue polygons the equilibrium adaptive run at round 11; the numbers under each radar give that pair’s pinned\-to\-equilibrium distance at round 0 and at round 11\. Bottom: matched distance per round, median \(line\) and range \(band\) over reaches 0\.05–0\.60 on each split\. Dots at right are the distance between the two runs’ fixed points when the position update is continued past round 11 without training\.Figure 12:Additional pairs expand coverage of the seven\-dimensional resource landscape\. Rows contain every pair in the two\-, five\-, and seven\-pair naive adaptive start arms; each radar radius is one trait coordinate\. Grey dashed polygons are the naive adaptive starts these runs used, solid colored polygons are the round\-11 positions, and red dash–dot polygons are counterfactual empirical\-resource equilibria rather than training starts\. Bold colored labels mark each endpoint’s dominant trait\. The number of represented dominant traits grows from two to four to six, showing that extra capacity produces new positional roles rather than merely duplicating one expert\.
## Appendix IRouting\-rule ablations

The same game allocation can be exposed to adapter training in materially different ways\. We separate three choices: how many of the seven pairs receive each prompt, whether a retained pair is selected deterministically or sampled, and whether its loss is scaled by its allocation share or given unit weight\. These choices do not change the location\-game solution, but they determine how much of its graded allocation signal reaches the adapters\. Figure[14](https://arxiv.org/html/2611.25167#A9.F14)compares five end\-to\-end arms on the same data seed 0 split and against the same pooled generalist\. Thus this is a training\-and\-routing ablation, not a post\-hoc recombination of one fixed expert family\.

#### Same\-expert nulls\.

Before changing how the adapters are trained, Figure[13](https://arxiv.org/html/2611.25167#A9.F13)holds the seven trained pairs fixed and changes only the combiner, using the completed\-sequence estimator quoted in Section[4\.3](https://arxiv.org/html/2611.25167#S4.SS3)\.

Figure 13:The game allocation carries expert\-specific routing information\. Holding all seven trained pairs fixed, the game combiner improves over the pooled model, whereas shuffling position–expert identities destroys the gain and uniform random commitment is substantially worse\. Orange bars mix sequence probabilities; dark bars report sampling/commitment semantics, and the oracle is a hindsight ceiling \(higher gain is better\)\.
#### Routing rules as training distributions\.

Every rule acts at the pair level on the pair sharesGp​\(b\)G\_\{p\}\(b\), which sum to one over theP=7P=7pairs\. For each promptbba rule chooses a support𝒮⁡\(b\)\\mathcal\{S\}\(b\)of pairs that train on it and a loss weightwp​\(b\)≥0w\_\{p\}\(b\)\\geq 0for each, withwp​\(b\)=0w\_\{p\}\(b\)=0whenp∉𝒮⁡\(b\)p\\notin\\mathcal\{S\}\(b\)\. The trainer then fits pairpp’s adapter to the weight\-normalized mean of its per\-example response\-token cross\-entropyℓ⁡\(θp,b\)\\ell\(\\theta\_\{p\};b\)over the round batch,

L^p​\(θp\)=∑m=1Mwp​\(bm\)​ℓ​\(θp,bm\)∑m=1Mwp​\(bm\)\.\\hat\{L\}\_\{p\}\(\\theta\_\{p\}\)=\\frac\{\\sum\_\{m=1\}^\{M\}w\_\{p\}\(b\_\{m\}\)\\,\\ell\(\\theta\_\{p\};b\_\{m\}\)\}\{\\sum\_\{m=1\}^\{M\}w\_\{p\}\(b\_\{m\}\)\}\.\(35\)As the batch grows this approaches the expected loss under the pair\-specific training distribution

Bp​\(b\)=w¯p​\(b\)​B​\(b\)𝔼B​\[w¯p​\(b\)\],w¯p​\(b\)=𝔼⁡\[wp​\(b\)∣b\],B\_\{p\}\(b\)=\\frac\{\\bar\{w\}\_\{p\}\(b\)\\,B\(b\)\}\{\\mathbb\{E\}\_\{B\}\\\!\\left\[\\bar\{w\}\_\{p\}\(b\)\\right\]\},\\qquad\\bar\{w\}\_\{p\}\(b\)=\\mathbb\{E\}\\bigl\[w\_\{p\}\(b\)\\mid b\\bigr\],\(36\)where the inner expectation is over any randomness in the rule\. In expectation, then, a routing rule reaches training only throughw¯p\\bar\{w\}\_\{p\}, and two rules with the samew¯p\\bar\{w\}\_\{p\}differ only in the variance of the weights they realize\. Table[6](https://arxiv.org/html/2611.25167#A9.T6)givesw¯p\\bar\{w\}\_\{p\}for each arm\.

#### Selection\.

The top\-kkrules differ in how the support is chosen\. Withgp​\(b\)g\_\{p\}\(b\)independent standard Gumbel noise andτ∈\{0,1\}\\tau\\in\\\{0,1\\\},

𝒮k​\(b\)=topk⁡\{log⁡Gp​\(b\)\+τ​gp​\(b\)\}p=1P\.\\mathcal\{S\}\_\{k\}\(b\)=\\operatorname\{top\}\_\{k\}\\bigl\\\{\\log G\_\{p\}\(b\)\+\\tau\\,g\_\{p\}\(b\)\\bigr\\\}\_\{p=1\}^\{P\}\.\(37\)Atτ=0\\tau=0this is the deterministic setTk​\(b\)T\_\{k\}\(b\)of thekklargest shares\. Atτ=1\\tau=1it drawskkdistinct pairs by successive sampling without replacement in proportion toG⁡\(b\)G\(b\), and fork=1k=1it reduces to a single categorical drawp^​\(b\)∼G​\(b\)\\hat\{p\}\(b\)\\sim G\(b\), the Gumbel\-max trick\. Under sampling, pairppis included with probabilityπp\(k\)\(b\)=Pr\[p∈𝒮k\(b\)\]\\pi^\{\(k\)\}\_\{p\}\(b\)=\\Pr\[p\\in\\mathcal\{S\}\_\{k\}\(b\)\], which satisfiesGp​\(b\)≤πp\(k\)​\(b\)≤1G\_\{p\}\(b\)\\leq\\pi^\{\(k\)\}\_\{p\}\(b\)\\leq 1, because a pair drawn first is always included, and∑pπp\(k\)​\(b\)=k\\sum\_\{p\}\\pi^\{\(k\)\}\_\{p\}\(b\)=k\.

Table 6:Expected loss weightw¯p​\(b\)\\bar\{w\}\_\{p\}\(b\)that each routing rule gives pairppon promptbb, which by Equation[36](https://arxiv.org/html/2611.25167#A9.E36)fixes the pair’s training distribution in expectation\.T3​\(b\)T\_\{3\}\(b\)is the set of the three largest shares,Z3​\(b\)=∑r∈T3​\(b\)Gr​\(b\)Z\_\{3\}\(b\)=\\sum\_\{r\\in T\_\{3\}\(b\)\}G\_\{r\}\(b\), andπp\(3\)​\(b\)\\pi^\{\(3\)\}\_\{p\}\(b\)is the probability that pairppis among three pairs drawn without replacement in proportion toG⁡\(b\)G\(b\)\.
#### Soft and hard\.

A rule is*hard*when the pairs that train on a prompt are drawn at random from the allocation and each drawn pair trains at unit weight, and*soft*when both the support and the weights are fixed functions of the allocation\. The two are linked by expectation: replacing a hard rule’s random inclusion indicator by its expectation gives a soft rule with the samew¯p\\bar\{w\}\_\{p\}, and hence the same training distribution in Equation[36](https://arxiv.org/html/2611.25167#A9.E36)\. The one exact pair in this ablation is hard top\-1 and dense soft routing\. Under hard top\-1,wp\(b\)=𝟏\[p^\(b\)=p\]w\_\{p\}\(b\)=\\mathbf\{1\}\[\\hat\{p\}\(b\)=p\], so

w¯p\(b\)=Pr\[p^\(b\)=p\]=Gp\(b\),\\bar\{w\}\_\{p\}\(b\)=\\Pr\\bigl\[\\hat\{p\}\(b\)=p\\bigr\]=G\_\{p\}\(b\),\(38\)which is exactly the dense share\-weighted weight\. Both rules therefore train pairpptowardBp∝Gp​BB\_\{p\}\\propto G\_\{p\}\\,B, whose total mass𝔼B​\[Gp​\(b\)\]\\mathbb\{E\}\_\{B\}\\\!\\left\[G\_\{p\}\(b\)\\right\]is the pair’s own game payoffupu\_\{p\}from Equation[1](https://arxiv.org/html/2611.25167#S2.E1): the share a pair wins in the game is the share of the training data it receives\. What differs is the realization\. The hard weight on each prompt is a Bernoulli draw with varianceGp​\(1−Gp\)G\_\{p\}\(1\-G\_\{p\}\), so a hard pair trains at full weight on a random subset of about𝔼B​\[Gp\]​M\\mathbb\{E\}\_\{B\}\\\!\\left\[G\_\{p\}\\right\]Mprompts per round, roughly a seventh of the batch when loads are balanced, while a soft pair trains on allMMprompts at fixed fractional weight\. Because the two rules share an expected training distribution, the 0\.085\-nat gap between them in Figure[14](https://arxiv.org/html/2611.25167#A9.F14)comes from how that distribution is realized, not from what it is\.

The three top\-3 rules sit between these extremes, and none is the soft counterpart of another\. Deterministic unit weighting,𝟏\[p∈T3\(b\)\]\\mathbf\{1\}\[p\\in T\_\{3\}\(b\)\], keeps only membership: a pair gets zero weight on a prompt as soon as its share falls to fourth place, and full weight inside the top three however unequal the three shares are\. Share weighting keeps the grading inside the support but inflates every retained weight by1/Z3​\(b\)≥11/Z\_\{3\}\(b\)\\geq 1\. Sampled top\-3 trains towardπp\(3\)\\pi^\{\(3\)\}\_\{p\}, which is continuous in the shares and bounded below byGpG\_\{p\}; its soft counterpart would weight byπp\(3\)\\pi^\{\(3\)\}\_\{p\}, which neither deterministic top\-3 rule does\.

Figure 14:Dense, share\-weighted use of the game allocation is necessary for the strongest routing result on data seed 0\. The left panel compares each arm’s expected single\-expert NLL \(orange square\) with its hindsight\-oracle NLL \(blue circle\) and the common pooled generalist \(vertical dashed line; lower is better\)\. Printed margins are router minus pooled, so negative values are improvements\. Moving from dense all\-seven routing to top\-three support, then removing share weights or sampling the three retained pairs, progressively weakens both learned experts and routing; hard top\-1 is worse than pooling\. The right panel shows why oracle agreement alone is misleading: hard top\-1 agrees with the best expert as often as the dense arm but has the worst NLL\.Dense share\-weighted routing gives the lowest single\-expert expected NLL, improving over the pooled model by 0\.0693 nats\. Retaining only the top three shares reduces the gain to 0\.0415 nats\. At fixed top\-three support, replacing share weights with unit weights reduces it again to 0\.0315 nats, and drawing the three pairs in proportion to their shares instead of taking the three largest gives 0\.0282 nats\. Hard top\-1 is the only arm worse than pooling, by 0\.0156 nats\. The oracle endpoints worsen in the same order, indicating that sparsification changes the experts learned during training rather than only degrading the final selector\. Dense routing is consequently a substantive part of the method: it preserves the continuous competitive signal through expert learning, at the cost of training every pair on every prompt\.

Router–oracle agreement is not sufficient to choose among these designs\. Hard top\-1 has the highest agreement \(0\.706\) yet the worst NLL, while dense share\-weighted routing has nearly identical agreement \(0\.705\) and the best NLL\. Agreement records only which pair wins; it ignores the competence of that pair and the loss incurred on disagreements\. The game therefore affords both a ranking and a calibrated allocation over experts, and the latter is lost when its shares are collapsed too early to a hard decision\.

## Appendix JPair count and fifteen experts

Figure[3](https://arxiv.org/html/2611.25167#S4.F3)a shows two to nine pairs of theT=0\.35T=0\.35Dirichlet game; the sweep runs to fifteen, depth matched to the seven\-pair arm at every count, on three splits \(Table[7](https://arxiv.org/html/2611.25167#A10.T7)\)\. Single\-expert gain peaks at six pairs under every start and then falls slowly, while the hindsight\-best expert keeps improving to fifteen; the gap opens because top\-1 agreement falls from 95% at six pairs to 58% at fifteen, although the best expert stays among the top three shares for at least 93% of prompts\. The two adaptive starts are indistinguishable at every count; at this reach the exact equilibrium lies only 0\.005–0\.020 from the unsolved start\. From nine pairs on, the pinned fixed start beats pinned adaptive on all three splits at every count, by 0\.001–0\.007 nats with a 3–6 point higher top\-1, which suggests that with many experts the position updates blur the allocation the experts specialize to\. Every pinned fixed run was checked to hold its positions byte\-identical to the start in all twelve rounds\.

Table 7:Pair\-count sweep,T=0\.35T=0\.35Dirichlet, mean±\\pmSD over three splits\. PA, PF and EA are the pinned adaptive, pinned fixed and equilibrium adaptive starts; gains are for the share\-weighted single expert \(nats, higher is better\)\. Top\-1 and top\-3 are agreement with the per\-prompt best expert \(percent\); the last column is the oracle gain under EA\. Bold marks the best value in each column; the shaded row is the best on single\-expert gain\.#### Fifteen experts across kernels\.

Repeating the fourT=0\.35T=0\.35kernel arms of Table[1](https://arxiv.org/html/2611.25167#S4.T1)with fifteen pairs, depth matched per kernel, lowers top\-3 chance from3/73/7to3/153/15\(Table[8](https://arxiv.org/html/2611.25167#A10.T8)\)\. More experts raise the ceiling \(oracle 1\.859–1\.861 against 1\.873–1\.876\) and lower what routing reaches \(expected NLL 1\.916–1\.922 against 1\.899–1\.903\), so the gap between them more than doubles\. Top\-1 agreement falls to 0\.59–0\.60 while top\-3 stays at 0\.93–0\.94, against excess\-over\-chance of\+0\.74\+0\.74, and the four kernels remain within 0\.010 of each other on every measure\. On the Dirichlet arm, starting at the exact equilibrium changes nothing measurable, doubling the depth or the learning rate hurts, and a second training pass improves the experts \(oracle 1\.806\) without improving routing \(top\-1 0\.576\)\.

Table 8:Seven against fifteen pairs atT=0\.35T=0\.35, one run each on the seed\-0 split \(NLL, lower is better; agreement, higher is better\)\. Dirichlet variants change one setting of the fifteen\-pair arm\. Bold marks the best value in each column; the shaded row is the best on single\-expert NLL\.

## Appendix KPer\-benchmark experts

A label\-defined baseline trains seven experts, one per benchmark, each on its own benchmark’s prompts from every batch the game saw, and routes test prompts by benchmark centroid in the trait space, by a classifier fitted on validation, by the true benchmark label, or in hindsight \(Table[9](https://arxiv.org/html/2611.25167#A11.T9)\)\. Even with true labels, which route each benchmark’s prompts to that benchmark’s expert, it trails the game’s single routed pair by 0\.032 nats, and its hindsight best expert also loses to the game’s routed choice; the shortfall is in the experts, which are weakest on do\-not\-answer and prompt injection, where content is shared with other benchmarks\. The comparison favors the game in compute, since a per\-benchmark expert trains only on its own benchmark’s share of each batch\.

Table 9:Per\-benchmark experts against the game, one expert per prompt, seed\-0 test pool\. Gain over the pooled generalist \(2\.0154\), higher is better\.
## Appendix LPublished gating rules

Figure[5](https://arxiv.org/html/2611.25167#S4.F5)trains five published gates and two distance gates jointly with seven LoRA experts\. Every arm uses the game’s twelve batches, the seed\-0 split, the same seven\-dimensionalT=0\.35T=0\.35trait coordinates as router input, rank\-16 LoRA components, a2×82\\times 8micro\-batch on one A100, and 1,248 optimizer steps\. Gates act per prompt, not per token, so MoLE’s per\-layer gating, Switch’s capacity factor and DeepSeek\-V3’s sequence\-wise balance term have no direct analogue and are not reproduced; each gate’s load\-balancing term is computed over the effective batch at every optimizer step\.

#### Router learning rate\.

Trained as specified, with the router at the experts’ learning rate of2×10−42\\times 10^\{\-4\}, every gate trails the game \(Table[10](https://arxiv.org/html/2611.25167#A12.T10)\)\. In 1,248 steps a prompt\-level router barely moves from its initialization\. Switch therefore scales its single expert’s output by a gate probability near uniform, noisy top\-kkkeeps its initial noise and does worse than the pooled model, and Mixtral without a balance term collapses onto about four experts\. Raising the router learning rate to10−210^\{\-2\}, fifty times the experts’ rate, lets each router sharpen within the same budget; the value was set once after the first round and not searched, and no game setting was tuned\. The two distance gates were run only at the specified rate\.

Table 10:Published gates trained jointly with their experts,T=0\.35T=0\.35, seed\-0 test pool \(NLL; lower is better\)\. “As specified” trains the router at the experts’ learning rate; “router10−210^\{\-2\}” raises it\. The last column is the per\-prompt best single component of the run shown in Figure[5](https://arxiv.org/html/2611.25167#S4.F5)\. Pooled generalist: 2\.0154\. Game: 1\.8899 \(token\-level mixture over all pairs\), 1\.8727 \(best single expert\)\.
#### Gates on the game’s experts\.

The published gates are also fitted post hoc on the game’s seven frozenT=0\.35T=0\.35experts, on validation expert losses with hyperparameters chosen by cross\-validation \(paper/gating\_methods\.py\), refit for three seeds and scored token by token in the same forward passes as the game’s own shares \(paper/posthoc\_gate\_weights\.py,paper/token\_ensemble\_eval\.py \-\-extra\-weights\)\. The refits reproduce the stored completed\-sequence results exactly\. Table[11](https://arxiv.org/html/2611.25167#A12.T11)gives the token\-level mixtures; these are the diamonds of Figure[5](https://arxiv.org/html/2611.25167#S4.F5)a\. The distance gates beat the game’s shares by about 0\.01 nats, which the completed\-sequence estimator had understated as 0\.006; Mixtral without a balance term collapses onto a few experts on every seed\.

Table 11:Published gates fitted on the game’s frozen experts, token\-level mixture NLL on the seed\-0 test pool \(lower is better\), mean and standard deviation over three seeds\. Game’s own shares: 1\.8899; they are a fixed function of the frozen experts and are not refit, so they have no seed spread\.
#### The game’s shares as a fixed gate\.

The game’s training cost comes from training each pair as a standalone adapter\. To test whether its allocation still yields specialists at a jointly trained gate’s cost, we train one seven\-component LoRA mixture of the same form as Figure[5](https://arxiv.org/html/2611.25167#S4.F5)’s gates, but with the gate fixed to the game’s shares: each prompt’s weights are theT=0\.35T=0\.35Dirichlet game’s allocation, computed from that game run’s pair positions at the start of each round, with nothing learned \(gate\_kind: game\)\. It trains on the game’s twelve batches in 49 minutes on one A100 with 1\.36 M training tokens, the jointly trained gates’ budget and a seventh of the game’s\. As one blended network it scores 1\.9110, a gain of 0\.104 nats over pooling, against 1\.8864 for the dense softmax gate at router learning rate10−210^\{\-2\}\. Its components are specialists: on the prompts the game assigns to a component, that component alone is the best of the seven 90% of the time, as for the game’s separately trained experts, whereas a jointly trained dense softmax gate’s own component is best on 1\.7% of prompts\. Table[12](https://arxiv.org/html/2611.25167#A12.T12)scores these components token by token, each run alone through a forced one\-hot gate, under the game’s shares and under gates fitted on their validation losses\. Mixed per token by the game’s shares they score 1\.9085, better than the blended network that trained them; fitted distance gates improve on the game’s shares by about 0\.013 nats \(1\.8951 and 1\.8962\), the same headroom as on the separate experts\. The separate experts are better by 0\.017–0\.019 nats at every routing level, including the oracle, so the sevenfold training compute buys stronger experts rather than more routable ones\. The game’s separately trained experts can also be blended into one network under the same fixed gate, by copying each pair’s LoRA factors into the matching component, with no further training\. The blended network scores 1\.8932, within 0\.0033 nats of mixing the experts’ outputs, so serving the game’s mixture does not require seven forward passes\. This is one run on the seed\-0 split\.

Table 12:Token\-level mixture NLL on the seed\-0 test pool \(lower is better\) for components trained in one blended mixture gated by the game’s shares \(1×\\timestraining compute\) and for the game’s separately trained experts \(7×\\times\)\. Fitted gates are means over three seeds \(standard deviation at most 0\.001\)\. The last column is the difference; negative favors the blended training\.
#### Linear gates on the game’s experts\.

The frozen linear gates of Section[4\.3](https://arxiv.org/html/2611.25167#S4.SS3)are fitted on validation expert losses with every expert frozen, then scored token by token on the game’s sevenT=0\.35T=0\.35experts in the same forward passes as the game’s own shares \(paper/token\_ensemble\_eval\.py \-\-extra\-weights\)\. Under that token\-level estimator the soft\-target gate scores 1\.8999 and the best\-expert classifier gate 1\.8934, against 1\.8899 for the game\. The completed\-sequence estimator narrows the soft\-target gate’s gap to 0\.0041 nats \(1\.8909 against 1\.8868\), because it leans on the best\-fitting expert and so depends less on the weights\.

## Appendix MConventional mixture of experts

The game’s allocation can also gate a conventional mixture of experts\. By sparse upcycling\([Komatsuzaki et al\., 2023](https://arxiv.org/html/2611.25167#bib.bib28)\), the feed\-forward network of decoder layers 20, 22, 24 and 26 of Qwen2\.5\-1\.5B\-Instruct is copied into seven trainable feed\-forward experts per layer, mixed at the block output\. A prompt\-level gate on theT=0\.35T=0\.35trait coordinates weights the experts, and every arm trains on the game’s batches for twelve rounds at effective batch 16, with expert learning rate2×10−52\\times 10^\{\-5\}and gate learning rate10−210^\{\-2\}\. The game arm fixes the gate to the game’s shares, and the other arms learn theirs\. Each arm is retrained on three data splits\.

Table 13:Gates over seven sparse\-upcycled feed\-forward experts, test NLL \(lower is better\) and gain over each split’s pooled LoRA adapter, mean±\\pmsample standard deviation over three data splits\. Oracle is the hindsight\-best expert alone\.Table[13](https://arxiv.org/html/2611.25167#A13.T13)gives the result\. Paired over the three splits, the game’s shares trail the learned dense gate by0\.0059±0\.00420\.0059\\pm 0\.0042nats and lead the Mixtral and Switch gates by 0\.016 and 0\.050 on every split\. The game’s hindsight\-best expert is0\.033±0\.0070\.033\\pm 0\.007nats better than the dense gate’s, and the game is the only arm whose oracle is below its own blended NLL, so its feed\-forward experts are the most specialized\. All gains in this setting are about a tenth of those with LoRA experts on every layer, so the result concerns the allocation as a gate for conventional experts; upcycling itself adds little on this corpus\.

#### Token\-level routing\.

The game needs one fixed trait vector per data point, so it routes per prompt\. To test whether its allocation is useful to a token\-level router, each upcycled layer is given its own router over its input hidden state\. A dense router and a Switch router learned from scratch are compared with a dense router whose logits carry the game’s per\-prompt log\-shares as a bias, either fixed for the run or annealed to zero by round 6\. Table[14](https://arxiv.org/html/2611.25167#A13.T14)shows the seed\-0 split\. Every token\-level router ends behind the pooled adapter within this budget of 1,248 steps, while prompt\-level routing of the same experts stays ahead of it\. The game’s shares as a fixed prior halve the token router’s deficit, and the router that only starts from them keeps a quarter of that benefit\. Without the bias a prompt’s tokens spread over 5\.8–6\.4 effective experts and the routers’ per\-prompt choice has little relation to the game’s assignment \(normalized mutual information 0\.20–0\.30\)\. With it they concentrate on 2\.6–2\.9 experts and the per\-prompt choice agrees with the game’s for 88% of prompts under the fixed prior and 76% after the prior has been annealed away\.

Table 14:Prompt\-level and token\-level routers over the seven upcycled feed\-forward experts, seed\-0 split, one run per arm\. Gain is over the pooled LoRA adapter \(2\.0154\)\.RouterLevelBlended NLLGain over pooledInflEnsshares, fixedprompt2\.0070\+0\.008\+0\.008Dense softmax, learnedprompt2\.0059\+0\.010\+0\.010Switch top\-1, learnedprompt2\.0593−0\.044\-0\.044Dense softmax withInflEnslog\-shares as a fixed priortoken2\.065−0\.049\-0\.049Dense softmax, prior annealed to zero by round 6token2\.091−0\.075\-0\.075Dense softmax, learned from scratchtoken2\.116−0\.100\-0\.100Switch top\-1, learned from scratchtoken2\.162−0\.147\-0\.147

## Appendix NTraining cost

Every adapter in every arm receives 1,248 optimizer updates; what differs is the base\-model compute behind them \(Table[16](https://arxiv.org/html/2611.25167#A14.T16)\)\. A jointly trained gate computes all seven component updates inside one forward pass, while the game trains each pair as a standalone adapter with its own pass, so it costs seven times the training compute for the same number of updates per adapter\. A pooled adapter trained for the game’s full budget, seven epochs per round, overfits each round’s 1,654 prompts \(test NLL 3\.39\) and is not a usable compute\-matched comparator\. The jointly trained mixtures fail the same way \(Table[15](https://arxiv.org/html/2611.25167#A14.T15)\)\. At seven epochs per round the dense and DeepSeek\-V3 gates and the game\-share gate each process the game’s 9\.50 M training tokens, and each ends about one nat worse than pooling\. The game\-share mixture reaches a training loss of 0\.69, so the loss is memorization of the repeated batches rather than a failure to train\. Repeating data is therefore not a compute match for the game, whose seven experts each see every batch once\.

Table 15:Test NLL of the single blended model at one and seven epochs per round on the seed\-0T=0\.35T=0\.35split \(lower is better\)\. Seven epochs gives each arm the game’s training tokens by repeating each round’s batch\. Gates use gate learning rate10−210^\{\-2\}\.Table 16:Training tokens summed over adapters and rounds, wall\-clock on one A100, and forward passes per test prompt, for theT=0\.35T=0\.35arms\.
## Appendix OModel size and family ablations

These checks ask whether the same game\-derived coordination mechanism remains useful when the base language model changes\. They are not tests of the positional bifurcation threshold, which depends on the prompt\-resource geometry and game configuration rather than model parameter count alone\. Figure[15](https://arxiv.org/html/2611.25167#A15.F15)collects two distinct experiments\.

The Qwen2\.5 size ladder in panel \(a\) uses five pairs, an equilibrium adaptive start, and the same trait geometry across 1\.5B, 3B, 7B, 14B, and 32B models\. Each rung is compared with its own pooled generalist\. Absolute held\-out NLL falls with model size for both pooled and routed systems, while sequence\-mixture oracle\-headroom capture declines from 87\.4% to 84\.2%\. Routing improves on the pooled comparator at every rung, but the available oracle gap narrows\. There is one run per rung, the 7B run needed 48 non\-finite\-gradient guard updates, and the 32B run used an off\-protocol microbatch configuration\.

Panel \(b\) uses the existing seven\-pair Qwen2\.5\-1\.5B, Llama\-3\.2\-1B, and Gemma\-3\-1B arms\. Their learned mixtures each improve over the respective model’s pooled control, so the allocation is not specific to one base\-model family\. The plotted values are within\-model NLL differences, since raw per\-token NLLs cannot be ranked across different tokenizers\. Model family, tokenizer, and parameter count vary together, and these arms do not share the five\-pair equilibrium adaptive start protocol of panel \(a\), so this is limited portability evidence rather than a ranking of architectures\.

\(a\) Qwen2\.5 size ladder

\(b\) Model\-family comparison

Figure 15:The game\-derived allocation transfers across model scale and family, but these ablations are portability checks rather than architecture rankings\. \(a\) Across five Qwen2\.5 sizes, the upper plot compares pooled\-generalist, routed\-argmax, game\-weighted sequence\-mixture, and hindsight\-oracle NLL \(lower is better\); the shaded region is the available oracle envelope\. The lower plot shows that the mixture captures 84–87% of that envelope at every size while argmax gain declines\. The 7B guard intervention and off\-protocol 32B microbatch are annotated, and each size has one run\. \(b\) For Qwen, Llama, and Gemma, every quantity is an NLL gain over that family’s own pooled adapter: the game\-weighted mixture improves all three and remains below the oracle ceiling, while the shaded Jensen gap separates probability mixing from sampling one expert\. Raw NLL is not compared across tokenizers, and panels \(a\) and \(b\) use different pair\-count and initialization protocols\.
## Appendix PBaseline families and deep ensembles

Table[17](https://arxiv.org/html/2611.25167#A16.T17)condenses a ten\-family implementation audit\. Every family has a held\-out scored artifact; “measured” does not mean an exact reproduction of its reference implementation\. The comparisons answer different questions: routing nulls hold experts fixed, assignment controls retrain them, and the jointly trained adapter mixtures are single models, so we do not sort all their NLLs into one leaderboard\. The same\-expert nulls and validation\-trained frozen linear gate are controls defined for this study, not reproductions of named methods; the other rows cite the work motivating each baseline\.

Table 17:Status and interpretive role of the ten baseline families\. All are measured, and the qualifications are part of each result\. The shaded first row isInflEns, the reference each family is compared against\.FamilyComparison madeSalient qualificationInflEns\(ours\)Seven game\-positioned rank\-16 LoRA pairs; the solved allocation weights training and routes inference, with no fitted gate\.Seed\-0 split: completed\-sequence mixture NLL 1\.9059 and hard commitment 1\.9298, against 2\.0152 pooled\.Same\-expert nullsUniform mixture, shuffled expert identities, uniform random hard choice, game allocation, and hindsight oracle on fixed expert losses\.Separates expert diversity from the prompt–expert mapping; one\-split offline analysis\.Frozen gateValidation\-fitted linear soft\-target and best\-expert gates on frozen experts, replicated on four splits\.Stronger game result under single\-expert commitment; full probability mixing is unresolved\.BASE\-style assignmentProportional versus quota\-balanced assignment\([Lewis et al\., 2021](https://arxiv.org/html/2611.25167#bib.bib31)\)crossed with fixed versus strategically updated positions\.All four cells start at the same equilibrium; entropic OT is not implemented\.Balanced c\-BTMBalanced versus unconstrained train\-only clusters\([Gururangan et al\., 2023](https://arxiv.org/html/2611.25167#bib.bib15)\); dense, top\-3, and hard centroid inference\.Fixed\-partition clustering, not the online soft\-k\-means position control\.EWoRA\-styleSeven jointly trained rank\-16 components\([Kohli et al\., 2025](https://arxiv.org/html/2611.25167#bib.bib27)\)with a dense trait\-conditioned gate versus a pooled rank\-112 adapter\.Mechanism\-level implementation; smaller micro\-batch at matched effective batch size\.AdaMixIndependent versus shared down\-projections\([Wang et al\., 2022](https://arxiv.org/html/2611.25167#bib.bib55)\)with stochastic routes, consistency, and post\-training merge held fixed\.Tests shared structure at this training budget, not longer published schedules\.MoLoRAHidden\-state token/layer gates\([Zadouri et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib64)\), dense and top\-3, versus corresponding prompt\-level trait gates\.Routers start at zero; auxiliary load loss is averaged globally across layers\.MoDULA\-ResUniversal adapter folded into the base, then residual experts and a staged router\([Ma et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib37)\)\.Genuine residual construction; improvement is relative to its own stage\-1 pooled model\.KnOTS / TIES / DAREValidated weight\-space merges\([Stoica et al\., 2025](https://arxiv.org/html/2611.25167#bib.bib51);[Yadav et al\., 2023](https://arxiv.org/html/2611.25167#bib.bib59);[Yu et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib63)\)from label\-trained and game\-trained experts, refactored at serving ranks 16 and 112\.Forty\-two candidates per expert source; even the best game\-expert merge loses to pooling\.Pooled\-seed ensembleUniform probability mixture\([Lakshminarayanan et al\., 2017](https://arxiv.org/html/2611.25167#bib.bib29)\)of four pooled rank\-16 adapters trained on the same manifest\.Distinct training seeds, but cumulative LoRA means initialization varies only at round 0\.Table 17:Baseline families \(continued\)\.The*fixed\-expert*controls separate two sources of value\. At seven pairs, the game has a\+0\.1095\+0\.1095\-nat completed\-sequence mixture gain over pooling; uniform mixing of the same experts retains\+0\.0661\+0\.0661, while shuffling expert identities gives−0\.0671\-0\.0671and uniform random hard routing gives−0\.4144\-0\.4144\. The experts are useful without the game router, but the learned identity\-to\-trait correspondence adds0\.04340\.0434nats beyond a uniform mixture\. A frozen linear gate trained from full validation expert\-NLL vectors is a more demanding alternative\. Across four splits the game wins all four expected\-NLL \(single\-expert commitment\) comparisons by0\.0710\.071–0\.0900\.090nats; under completed\-sequence probability mixing the sign divides two\-to\-two, with at most0\.00170\.0017nats separation\. This supports the game’s allocation as a competitive hard router, not a general accuracy advantage over learned soft gates\.

The*assignment*controls narrow the game\-theoretic claim\. In the BASE\-style2×22\\times 2of Table[18](https://arxiv.org/html/2611.25167#A16.T18), which does not test BASE’s original efficiency setting, quota balancing costs about0\.0870\.087nats of expected NLL regardless of whether positions continue to move, whereas fixing positions at the solved equilibrium costs at most0\.00030\.0003nats in expected NLL at this operating point\.

Table 18:Seven\-pair factorial control; lower NLL is better\. Every cell starts from the same solved equilibrium and shares the reach, split, and pooled comparator\. The shaded row is theInflEnsconfiguration\.The independent c\-BTM comparison \(Table[19](https://arxiv.org/html/2611.25167#A16.T19)\) separates balance from sparse serving: imposing cluster quotas degrades all three inference modes and reduces the oracle envelope, while top\-3 centroid inference is nearly indistinguishable from dense inference within either partition\. These are single\-split results; the0\.00040\.0004–0\.00070\.0007top\-3 gaps are below the measured game\-arm run\-to\-run standard deviation of0\.00170\.0017nats\.

Table 19:c\-BTM partition and inference fan\-out on the seed\-0 split\. Entries are completed\-sequence mixture NLL \(lower is better\); the pooled rank\-16 adapter scores 2\.0152\. Oracle envelope is pooled NLL minus hindsight expert NLL\. The shadedInflEnsrow applies the same three fan\-outs to the game allocation over its seven trained pairs\.The*single\-model architecture*baselines instead test whether other ways to spend adapter parameters reproduce specialization without separately trained, game\-allocated experts\. Their end\-to\-end model NLLs belong in Table[20](https://arxiv.org/html/2611.25167#A16.T20), not beside a post\-hoc sequence\-mixture NLL: a jointly trained network is evaluated in one forward pass, whereas the latter mixes likelihoods from seven separately evaluated experts\. EWoRA\-style dense gating reaches 1\.9712, within0\.00250\.0025nats of the rank\-112 pooled adapter’s 1\.9737 at matched stored LoRA rank\. AdaMix’s shared down\-projection improves over independent projections by0\.03870\.0387nats, but both remain worse than the pooled rank\-16 control, and the shared arm’s hindsight expert envelope is negative\.

Table 20:Single\-model adapter baselines, seed\-0 split, end\-to\-end model NLL\. The two pooled rows are capacity references, not routing methods, and none of these numbers is directly comparable with the game’s post\-hoc completed\-sequence mixture\. The shadedInflEnsrow therefore reports hard commitment, which also evaluates one adapter per prompt; its seven rank\-16 pairs store the same LoRA rank as pooled rank\-112\.The gate\-resolution contrast is also a gate\-input contrast: MoLoRA’s hidden\-state token/layer gates score 2\.0029 \(dense\) and 1\.9955 \(top\-3\), against 1\.9712 and 1\.9231 for the corresponding prompt\-trait gates\. Token routing is worse by0\.03170\.0317and0\.07240\.0724nats in these cells\. MoDULA\-Res scores 1\.9909, a0\.02430\.0243\-nat improvement over its stage\-1 pooled adapter, and uniquely among these new architectures has a positive hindsight expert envelope, though it still trails pooled rank\-112\. Weight\-space merging asks whether trained experts can be compressed into one adapter\. None of 42 candidates from each of two expert sources matches pooling\. The best game\-expert KnOTS–TIES merge scores 2\.1239,0\.1090\.109nats worse than the pooled rank\-16 model, and even its hindsight oracle scores 2\.0758\. These merge operators did not preserve the experts’ conditional complementarity\.

*Independent training variance*does not explain all of the routing gain, yet it can add to it\. Table[21](https://arxiv.org/html/2611.25167#A16.T21)holds data and architecture fixed while adding pooled\-seed members\. Four members recover only\+0\.0305\+0\.0305nats over one pooled adapter, versus\+0\.1095\+0\.1095for the seven\-pair game mixture on the same split and estimator\. The four pooled adapters have only a0\.00210\.0021\-nat individual\-NLL spread, and cumulative training weakens their independence relative to four from\-scratch runs\.

Table 21:Pooled\-seed deep ensemble versus the game router, seed\-0 split\. All non\-pooled entries here use a completed\-sequence probability mixture rather than the token\-level serving estimator; gains are relative to one pooled rank\-16 adapter, which scores 2\.0152\.Five independently seeded games offer a separate extension of the mechanism\. Each game routes over seven experts; the ensemble averages the resulting 35 component probabilities per token\. It scores 1\.8600 against 1\.9103 for one re\-scored game and 2\.0152 for pooling \(Table[22](https://arxiv.org/html/2611.25167#A16.T22)\)\. Game allocation and deep ensembling are complementary: the additional five\-game gain is0\.05030\.0503nats on the standard token\-level estimator\. The completed\-sequence estimator is optimistic by0\.00440\.0044nats for seven components and0\.01400\.0140for 35\. The five\-game result requires 35 component evaluations per prompt under the measured implementation, so the gain is not compute\-neutral\.

Across these baselines, the game’s distinctive affordance is not a universal NLL win: a resource distribution, kernel, and solved positions specify an inspectable competitive allocation before the adapters are trained\.

Table 22:Game deep ensemble on the seed\-0 split\. Token\-level NLL mixes each observed token’s conditional probabilities and is the serving estimator; completed\-sequence NLL mixes whole\-sequence likelihoods and is shown only for sensitivity\.
## Appendix QSoft clustering with matched training

The c\-BTM baseline of Appendix[P](https://arxiv.org/html/2611.25167#A16)trains each expert on a hard cluster, so it differs fromInflEnsin training weights as well as in positions\. Here every arm shares the game’s trait space, its resource information, dense share\-weighted training, seven expert pairs, batches and budget, and positions are frozen in every arm\. The arms differ only in where the positions come from and in the kernel that turns them into shares\. Hardkk\-means centroids are scored under the game’s Dirichlet kernel and reach\. Softkk\-means, fitted by EM as an equal\-weight isotropic Gaussian mixture, uses its own responsibilities, which are Gaussian\-kernel shares at the fitted bandwidth \(σ=0\.0604\\sigma=0\.0604\); a second softkk\-means arm keeps those centroids and sets the Gaussian reach to the game’s instability depth \(σ=0\.0475\\sigma=0\.0475\)\. Each arm is retrained on three data splits\.

Figure 16:Clustering positions against the game’s equilibrium positions with matched dense training, seven pairs,T=0\.35T=0\.35\. Markers are means over three data splits, bars one sample standard deviation and small dots the individual splits\.aGain over the pooled adapter under three routing rules, and top\-1 agreement with the hindsight\-best expert\.bPaired differences on the same splits in the same four columns; right of zero favors the first\-named arm\. The fitted\-bandwidth arm spreads each prompt over 1\.22 effective experts, against 1\.12–1\.13 for the other three\. For reference, the adaptive game reaches0\.1183±0\.00290\.1183\\pm 0\.0029expected\.The positions are interchangeable \(Figure[16](https://arxiv.org/html/2611.25167#A17.F16)a\):kk\-means centroids under the game’s kernel match the game’s equilibrium positions within0\.0006±0\.00070\.0006\\pm 0\.0007nats, so the competitive objective reaches clustering\-quality positions without a clustering step\. The difference is in the allocation rule\. Softkk\-means with its own responsibilities trails the game by0\.0100±0\.00060\.0100\\pm 0\.0006nats when one expert is sampled and by 2\.6 points of top\-1 agreement, with the same sign on every split \(Figure[16](https://arxiv.org/html/2611.25167#A17.F16)b\)\. Most of that comes from the simplex\-native Dirichlet kernel, which leads a Gaussian kernel at equal sharpness by 0\.006–0\.009 nats under every routing rule\. The reach set by the stability analysis helps sampled routing by 0\.0035 nats and costs 0\.0025 under argmax routing and mixing\. Under full probability mixing the game’s total lead shrinks to 0\.0024 nats\.

## Appendix RSafety benchmarks

Safety is not part of the location game’s payoff or a prerequisite for its allocation rule\. The external evaluation asks a separate question: does game\-derived routing change behavior after the adapters have been trained? We use the official HarmBench attack\-success and StrongREJECT jailbreak graders\. ConfAIde probes contextual privacy reasoning; IFEval measures strict instruction following, an adjacent capability check rather than a safety benchmark\. The behavioral configurations also run XSTest\([Röttger et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib48)\), TruthfulQA\([Lin et al\., 2022](https://arxiv.org/html/2611.25167#bib.bib34)\), and BIPIA\([Yi et al\., 2025](https://arxiv.org/html/2611.25167#bib.bib62)\)under lexical proxy graders; those results are direction\-only and are not reported here\.

Figure[17](https://arxiv.org/html/2611.25167#A18.F17)separates three contrasts across three independently trained data splits\. Relative to each split’s pooled adapter, game\-routed experts have lower HarmBench attack success and lower StrongREJECT harmfulness under both starts and both decoding protocols: all 24 signed safety comparisons favor routing, as do the 12 IFEval comparisons\. Neither probability mixing over experts nor the equilibrium adaptive start has a consistent behavioral advantage over argmax routing and the naive adaptive start\. There is one run per split, StrongREJECT has only 60 cases, and HarmBench’s retained case count differs slightly across splits, so the bars describe spread across splits and are no confidence intervals\.

Every fine\-tuned model is far less safe than the base model\. Averaged over the three splits, the pooled adapter is worse than the base model by0\.589±0\.0520\.589\\pm 0\.052on HarmBench,0\.505±0\.0480\.505\\pm 0\.048on StrongREJECT and0\.160±0\.0290\.160\\pm 0\.029on IFEval, and no routed arm recovers the base level\. Routing therefore contains part of the damage relative to pooling without removing it, consistent with reports that fine\-tuning drifts the routing of harmful inputs in MoE models\([Kim et al\., 2026](https://arxiv.org/html/2611.25167#bib.bib25)\)and that merging experts erodes alignment\([Hammoud et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib16)\)\. Projecting adapter updates onto an alignment subspace\([Hsu et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib17)\)or adding an input–output guard\([Inan et al\., 2023](https://arxiv.org/html/2611.25167#bib.bib19)\)is orthogonal to allocation and was not applied; the retraining below corrects the training targets instead, in the spirit of mixing safety demonstrations into the data\([Bianchi et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib4)\)\. ConfAIde’s conditional error is computed only for parseable ratings and its response coverage varies sharply between arms, so it cannot support a privacy\-improvement claim\.

Figure 17:Behavioral contrasts on the original corpus over three independently trained data splits, official graders\. Markers are means, bars one sample standard deviation and small dots the individual splits\. Differences are signed so that right of zero favors the first\-named arm: HarmBench and StrongREJECT reverse their native direction, IFEval does not\. ConfAIde is omitted because its conditional error changes with response coverage\.#### Cause of the degradation\.

The degradation follows from the training corpus\. Table[3](https://arxiv.org/html/2611.25167#A1.T3)shows that 1,999 training targets are unsafe BeaverTails responses and 1,747 are hallucinated HaluEval answers, and that four sources contribute bare prompts, harmful requests and injection attempts among them, which the model is trained to reproduce because the loss covers prompt tokens\. Every arm trains on this corpus, the pooled adapter included, so the contrasts of Figure[17](https://arxiv.org/html/2611.25167#A18.F17)compare like with like, and the behavioral levels measure this corpus more than fine\-tuning in general\. We test the cause with two retrainings on the seed\-0 split, each graded with the official HarmBench, StrongREJECT and IFEval graders in one run per comparison \(Figure[18](https://arxiv.org/html/2611.25167#A18.F18)\)\.

#### Removing the unsafe targets\.

The first retraining removes the unsafe and hallucinated targets, leaving 16,102 training records; validation, test, trait space, reach and start are unchanged\. HarmBench attack success falls for every method: from 0\.508 to 0\.183 for the game \(paired difference−0\.325\-0\.325, bootstrap 95% interval\[−0\.398,−0\.251\]\[\-0\.398,\-0\.251\]\), from 0\.450 to 0\.288 for a dense softmax gate at the2×82\\times 8micro\-batch of the original run \(−0\.162\-0\.162\[−0\.236,−0\.089\]\[\-0\.236,\-0\.089\]\) and from 0\.639 to 0\.435 for the pooled adapter \(−0\.204\-0\.204\[−0\.283,−0\.131\]\[\-0\.283,\-0\.131\]\)\. StrongREJECT moves the same way \(0\.455 to 0\.147, 0\.474 to 0\.305 and 0\.570 to 0\.415\)\. Recovery is partial, since the base model scores 0\.052 on HarmBench\. The order of the methods on the cleaned corpus is not stable across runs\. Over three training seeds the game’s HarmBench rate is0\.194±0\.1000\.194\\pm 0\.100and that of a dense gate at a1×161\\times 16micro\-batch0\.077±0\.0450\.077\\pm 0\.045; the dense gate is below the game at every seed and on a second data split \(0\.129 against 0\.227\), while on StrongREJECT the two trade places\. The seed\-to\-seed spread is as large as the gaps between methods, so we claim no safety ordering on this corpus\. Instruction following is more consistent\. The game has the highest IFEval score in every run \(0\.296±0\.0100\.296\\pm 0\.010over the three seeds against0\.283±0\.0060\.283\\pm 0\.006for the pooled adapter and0\.266±0\.0170\.266\\pm 0\.017for the dense gate; 0\.311 against 0\.270 and 0\.264 on the second split\), a margin of0\.030±0\.0130\.030\\pm 0\.013over the dense gate\.

#### Safe responses with a prompt\-masked loss\.

The second retraining also removes the bare prompts as targets\. Every record without a response, and every unsafe BeaverTails record, receives a generated response that the Qwen3Guard\-Gen\-4B guard\([Zhao et al\., 2025](https://arxiv.org/html/2611.25167#bib.bib66)\)labels safe: refusals or helpful answers from Llama\-3\.1\-Tülu\-3\-8B\([Lambert et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib30)\)for the harm, policy\-violation, over\-refusal and jailbreak prompts, and answers that engage with the content and ignore the embedded instruction from Meta\-SecAlign\-8B\([Chen et al\., 2025](https://arxiv.org/html/2611.25167#bib.bib7)\)for the prompt injections\. Hallucinated targets stay removed, the 120 training records without an accepted response are dropped, and the loss covers response tokens only\. In total 11,227 of 17,981 training targets are replaced\. Under this recipe the degradation disappears for every method: HarmBench attack success is at most 0\.021 for every arm and training seed, and StrongREJECT at most 0\.040, below the base model’s 0\.062\. The cost is more refusal of benign prompts, in direction under a lexical proxy grader, and lower instruction following\. IFEval over three training seeds is0\.274±0\.0180\.274\\pm 0\.018for the game,0\.233±0\.0340\.233\\pm 0\.034for the dense gate and0\.169±0\.0300\.169\\pm 0\.030for the pooled adapter, against 0\.405 for the base model\. The game is ahead at every seed, by0\.105±0\.0190\.105\\pm 0\.019over the pooled adapter, so with every method at the safety floor the game keeps the most instruction following\. A control separates the two changes: masking the prompt out of the pooled adapter’s loss on the cleaned corpus, with no generated responses, raises its IFEval from 0\.281 to 0\.316 and lowers its HarmBench rate from 0\.435 to 0\.340, so the loss of instruction following comes from the generated responses, which trade helpfulness for refusal\. With the pseudo\-Fisher axes of Appendix[F](https://arxiv.org/html/2611.25167#A6)the result is the same on the seed\-0 split: IFEval 0\.294 for the game against 0\.259 for its dense gate, with HarmBench at 0\.011 and 0\.000\.

Figure 18:Behavior across the three training corpora, seed\-0 split, official graders \(HarmBenchn=191n=191, StrongREJECTn=60n=60, IFEvaln=541n=541\)\. On the two corrected corpora each marker is the mean over three training seeds, the band is one sample standard deviation and the small dots are the individual seeds\. The original corpus has one run per method \(hollow markers\), and the dashed line is the base model\. The dense gate is trained at a1×161\\times 16micro\-batch on every corpus\.
### R\.1Interpretability across data splits

Table[23](https://arxiv.org/html/2611.25167#A18.T23)reports the interpretability measurements of Section[4\.3](https://arxiv.org/html/2611.25167#S4.SS3)for the principal seven\-pair arm on each of three independently trained data splits\. The last row is a measured control that gives every prompt the allocation of a random other prompt\. Its Spearman correlation is 0\.12 and not zero, because shuffled shares still track expert quality through each expert’s average share, so the allocation’s excess over the control is about 0\.48\. Its span is negative because sharp shares placed on the wrong prompts score far worse than a uniform mixture\.

Table 23:Game shares predict the specialization learned by the adapters, although held\-out expert losses were never used to fit the router\. Top 1 and top 3 are the fractions of held\-out prompts whose best expert appears among the largest one or three shares; a uniform draw gives1/71/7and3/73/7\. The span column is the fraction of the uniform\-to\-oracle NLL gap the allocation recovers\. The permuted row pairs each prompt with the allocation of a random other prompt \(200 permutations, mean of the three splits; standard deviation over permutations 0\.005, 0\.007, 0\.005 and 3 points\)\.#### Experts in plain language\.

Because the allocation is a fixed rule on the trait space, each expert can be described from the test prompts on which it receives the largest share\. Table[24](https://arxiv.org/html/2611.25167#A18.T24)does this for the seven\-pairT=0\.35T=0\.35arm on the seed\-0 split\. Six of the seven pairs take at least 85% of their prompts from one benchmark, and the seventh collects harmful requests from three sources\. Each pair beats the pooled adapter on the prompts it wins\.

Table 24:The experts of the seven\-pairT=0\.35T=0\.35arm described from the test prompts they win \(seed\-0 split\)\. Gain is the pair’s NLL improvement over the pooled adapter on those prompts, in nats\.

### R\.2Tracing harm to one expert and editing it

Because the allocation is fixed, harmful behavior can be traced through the router before any model is changed\. We group the per\-case official scores of the seed\-0 original\-corpus game by the expert that receives the largest share of each prompt\. The HarmBench score of 0\.508 divides into 0\.487 from prompts routed to pair 5, the mixed harmful\-intent expert of Table[24](https://arxiv.org/html/2611.25167#A18.T24), and 0\.021 from pair 4\. The StrongREJECT score of 0\.455 divides into 0\.359 from pair 5, 0\.059 from pair 6 and 0\.037 from pair 4\. The router sends 94% of the HarmBench prompts and 63% of the StrongREJECT prompts to pair 5, and no HarmBench prompt to pair 6, the BeaverTails harm expert\. The trace therefore predicts that making pair 5 safe leaves 0\.021 on HarmBench and 0\.096 on StrongREJECT\.

We test the prediction by editing single experts with the router frozen\. The positions, kernel and shares are unchanged, so every prompt keeps its route\. The edited pair continues training for one epoch on the 1,656 training records the allocation assigns to it, with the safe responses and prompt\-masked loss of the retraining above, and the other six adapters stay identical to the original\.

Table 25:Single\-expert edits of the seed\-0 original\-corpus game with the router frozen, official graders \(HarmBenchn=191n=191, StrongREJECTn=60n=60, IFEvaln=541n=541\), one training run per row\. Lower is better for HarmBench and StrongREJECT, higher for IFEval\. The harm\-source edit replaces only the targets of the harm, policy\-violation and jailbreak records\. The wrong\-expert control trains pair 0, the privacy expert, on pair 5’s records with the same recipe\. The dense gate’s IFEval was generated on a different GPU type from the other rows\.ModelHarmBenchStrongREJECTIFEvalInflEns, original corpus0\.5080\.4550\.309Pair 5 edited, one epoch0\.0160\.0990\.298Pair 5 edited, three epochs0\.0160\.1030\.299Pair 5 edited, harm\-source targets only0\.0310\.1210\.305Pair 6 edited0\.5080\.4000\.309Wrong\-expert control \(pair 0 on pair 5’s records\)0\.5030\.4570\.324InflEns, every expert retrained on safe responses0\.0000\.0170\.253Dense gate, original corpus0\.4290\.4620\.294Pooled adapter, original corpus0\.6390\.5700\.214Table[25](https://arxiv.org/html/2611.25167#A18.T25)gives the result\. Editing pair 5 alone takes HarmBench from 0\.508 to 0\.016 \(paired difference−0\.492\-0\.492, bootstrap 95% interval\[−0\.565,−0\.419\]\[\-0\.565,\-0\.419\]\) and StrongREJECT from 0\.455 to 0\.099 \(−0\.356\-0\.356\[−0\.449,−0\.266\]\[\-0\.449,\-0\.266\]\), against the predicted 0\.021 and 0\.096\. Repeating the trace on the edited model puts pair 5’s contribution at 0\.000 on HarmBench and 0\.004 on StrongREJECT\. Instruction following is kept: IFEval changes by−0\.011\-0\.011\[−0\.033,\+0\.011\]\[\-0\.033,\+0\.011\], where retraining every expert costs0\.0550\.055\[0\.017,0\.094\]\[0\.017,0\.094\], and the edited game is within 0\.016 of the fully retrained one on HarmBench\. Three epochs give the same scores as one, and replacing only the harm\-source targets comes within 0\.016 on HarmBench and 0\.022 on StrongREJECT of the full edit\.

An edit acts mainly through the prompts routed to the edited expert, weighted by its share\. Editing pair 6 leaves HarmBench unchanged on every case and lowers StrongREJECT by 0\.055\[0\.009,0\.112\]\[0\.009,0\.112\], as pair 6’s traced contribution falls from 0\.059 to 0\.010\. The wrong\-expert control is a consistency check on this account: with the routing frozen, the HarmBench prompts still reach the unedited pair 5, and training pair 5’s records into the privacy expert moves HarmBench by−0\.005\-0\.005\[−0\.016,0\.000\]\[\-0\.016,0\.000\]and StrongREJECT by\+0\.002\+0\.002\[0\.000,\+0\.005\]\[0\.000,\+0\.005\]\. A jointly trained dense gate offers no such handle, since no single component of it serves these prompts; on the same corpus it scores 0\.429 on HarmBench,0\.4140\.414\[0\.340,0\.487\]\[0\.340,0\.487\]above the edited game\.

#### Reset, more experts and shorter edits\.

Table[26](https://arxiv.org/html/2611.25167#A18.T26)repeats the trace\-then\-edit cycle on further arms of the same split, with each prediction computed from the pre\-edit trace before the arm was graded\. Resetting pair 5 to the base model, with no retraining at all, gives 0\.063 on HarmBench and 0\.128 on StrongREJECT against predictions of 0\.068 and 0\.139, and an IFEval of 0\.314 against a predicted 0\.315\. The reset is worse than the one\-epoch edit by 0\.047\[0\.021,0\.079\]\[0\.021,0\.079\]on HarmBench and 0\.029\[0\.002,0\.065\]\[0\.002,0\.065\]on StrongREJECT, so the edit does more than remove what pair 5 had learned\. The trace leaves 0\.059 of the StrongREJECT score with pair 6 and 0\.037 with pair 4\. Editing pairs 5 and 6 lowers StrongREJECT to 0\.050, against a prediction of about 0\.04 and−0\.049\-0\.049\[−0\.100,−0\.008\]\[\-0\.100,\-0\.008\]from the pair\-5 edit, and adding pair 4 reaches the scores of the fully retrained game on both suites\. The edit also needs little training\. A quarter epoch already gives 0\.016 on HarmBench and 0\.107 on StrongREJECT \(\+0\.008\+0\.008\[−0\.007,\+0\.032\]\[\-0\.007,\+0\.032\]against one epoch\), half an epoch gives 0\.016 and 0\.096, and the re\-traced StrongREJECT contribution of pair 5 falls from 0\.359 to 0\.011, 0\.001 and 0\.004 at a quarter, a half and one epoch\. The IFEval scores of the quarter\-epoch, two\-expert and three\-expert edits are within two points of the original, the size of the one\-epoch edit’s interval, so we see no instruction\-following cost\. The XSTest proxy shows the price of editing more experts: refusal of safe prompts rises from 0\.068 with one edited expert to 0\.309 with two, 0\.326 with three and 0\.369 with all seven retrained\.

Table 26:Further edits of the seed\-0 original\-corpus game with the router frozen, all graded on one host, one training run per row\. Recipe as in Table[25](https://arxiv.org/html/2611.25167#A18.T25)\. Lower is better for HarmBench, StrongREJECT and both XSTest columns, higher for IFEval\. XSTest is a lexical refusal proxy and gives direction and rough size only: “safe refused” is refusal of safe prompts and “unsafe complied” is compliance with unsafe ones\.
#### Retraining the whole corpus on the same targets\.

The alternative to an edit is to retrain every model on a corpus whose harm sources carry safe responses, with the rest of the corpus and the unmasked loss unchanged \(Table[27](https://arxiv.org/html/2611.25167#A18.T27)\)\. All three retrained models reach 0\.01 to 0\.02 on HarmBench\. Against the original model graded on the same host, the game changes by−0\.487\-0\.487\[−0\.560,−0\.414\]\[\-0\.560,\-0\.414\]on HarmBench and−0\.421\-0\.421\(standard error 0\.050\) on StrongREJECT, the dense gate by−0\.408\-0\.408\(0\.037\) and−0\.476\-0\.476\(0\.048\), and the pooled adapter by−0\.613\-0\.613\(0\.035\) and−0\.553\-0\.553\(0\.047\)\. Instruction following is kept by the game \(−0\.017\-0\.017\[−0\.052,\+0\.018\]\[\-0\.052,\+0\.018\]\) and by the dense gate \(−0\.028\-0\.028\[−0\.067,\+0\.009\]\[\-0\.067,\+0\.009\]\), and the routed in\-distribution gain over the pooled adapter is unchanged \(\+0\.125\+0\.125against\+0\.117\+0\.117for the game,\+0\.139\+0\.139against\+0\.137\+0\.137for the dense gate\)\. On HarmBench the retrained game is within−0\.010\-0\.010\[−0\.042,\+0\.021\]\[\-0\.042,\+0\.021\]of the one\-expert edit that uses the same harm\-source targets\. The cost is over\-refusal: every retrained model refuses a third or more of the safe XSTest prompts, against 0\.047 for that one\-expert edit\.

Table 27:Whole\-corpus retraining with safe responses substituted for the three harm sources only, seed\-0 split, one training run per row\. Columns as in Table[26](https://arxiv.org/html/2611.25167#A18.T26)\. Cells marked†were graded on a second host; each original\-corpus value is the one graded on the same host as the retrained cell above it, so some differ from Table[25](https://arxiv.org/html/2611.25167#A18.T25)by up to 0\.03\.
#### A blind replicate on a second split\.

We repeated the study on a second data split with the predictions written down before any edit was trained \(Table[28](https://arxiv.org/html/2611.25167#A18.T28)\)\. No safe\-response corpus exists for this split, so the edits use the harm\-source recipe\. The unedited game scores 0\.598 on HarmBench and 0\.491 on StrongREJECT\. Editing pair 5 for one epoch lowers them to 0\.046 and 0\.138, and editing pairs 4, 5 and 6 to 0\.046 and 0\.039\. The wrong\-expert control does not move, as predicted\. The edits fall short of their predictions, which were 0\.010 and 0\.112 for the one\-epoch edit and 0\.000 on both suites for the three\-expert edit, and every edit arm misses the tolerance of 0\.02 set in advance: the edited experts keep 5 to 8% of their harm\. Three epochs do not close the gap \(−0\.005\-0\.005\[−0\.031,\+0\.021\]\[\-0\.031,\+0\.021\]and−0\.020\-0\.020\[−0\.061,\+0\.010\]\[\-0\.061,\+0\.010\]against one epoch\)\. On the XSTest proxy the three\-expert edit raises refusal of safe prompts from 0\.008 to 0\.388 and lowers compliance with unsafe prompts from 0\.965 to 0\.090\. The edit therefore replicates in direction and size on a second split and the control stays unmoved, while the trace overstates how completely an edited expert is cleaned\.

Table 28:Blind replicate of the expert edit on a second data split, harm\-source recipe, all graded on one host, one training run per row \(lower is better\)\. Changes are paired differences from the unedited game of that split \(HarmBench 0\.598, StrongREJECT 0\.491\) with bootstrap 95% intervals\.ModelHarmBenchChangeStrongREJECTChangePair 5 edited, one epoch0\.046−0\.552\-0\.552\[−0\.624,−0\.474\]\[\-0\.624,\-0\.474\]0\.138−0\.354\-0\.354\[−0\.449,−0\.255\]\[\-0\.449,\-0\.255\]Pair 5 edited, three epochs0\.041−0\.557\-0\.557\[−0\.629,−0\.485\]\[\-0\.629,\-0\.485\]0\.118−0\.373\-0\.373\[−0\.465,−0\.282\]\[\-0\.465,\-0\.282\]Pairs 4, 5 and 6 edited0\.046−0\.552\-0\.552\[−0\.624,−0\.474\]\[\-0\.624,\-0\.474\]0\.039−0\.453\-0\.453\[−0\.541,−0\.361\]\[\-0\.541,\-0\.361\]Wrong\-expert control \(pair 0 on pair 5’s records\)0\.608\+0\.010\+0\.010\[−0\.031,\+0\.052\]\[\-0\.031,\+0\.052\]0\.521\+0\.030\+0\.030\[−0\.006,\+0\.075\]\[\-0\.006,\+0\.075\]Pooled adapter, unedited0\.582−0\.015\-0\.015\[−0\.093,\+0\.062\]\[\-0\.093,\+0\.062\]0\.519\+0\.028\+0\.028\[−0\.055,\+0\.110\]\[\-0\.055,\+0\.110\]Dense gate, unedited0\.294−0\.304\-0\.304\[−0\.387,−0\.222\]\[\-0\.387,\-0\.222\]0\.394−0\.097\-0\.097\[−0\.181,−0\.014\]\[\-0\.181,\-0\.014\]

## Appendix SOut\-of\-distribution evaluation

The trained models are scored on data that no arm saw, with every prompt routed through the unchanged trait space\. Unless stated otherwise the models are the seven\-pairT=0\.35T=0\.35game, a dense softmax gate and the pooled adapter of the seed\-0 split\.

#### Far from the training distribution\.

None of the seven training benchmarks is mathematical\. We score all 1,319 GSM8K test problems\([Cobbe et al\., 2021](https://arxiv.org/html/2611.25167#bib.bib9)\)two ways: the NLL of the reference solution with the question as context, and answer accuracy from greedy decoding of 512 new tokens with an exact match on the final number\. The three training corpora are the original one and the two corrected ones of Appendix[R](https://arxiv.org/html/2611.25167#A18)\. Table[29](https://arxiv.org/html/2611.25167#A19.T29)shows that fine\-tuning on this corpus costs every model most of the base model’s ability, and that the game keeps the most\. On the original corpus its mixture has the lowest solution NLL of the fine\-tuned models, and its routed expert answers more than twice as many problems as the pooled adapter\. With safe responses the pooled adapter has the lowest solution NLL while the game stays the most accurate\. The game’s hindsight\-best expert, chosen per problem with the reference solution, has a solution NLL of 0\.669 on the original corpus, 0\.676 with unsafe targets removed and 0\.661 with safe responses\. It is an upper bound on routing with no decoding counterpart, and it nearly matches the base model, so some experts keep their mathematical ability, and the router, which has never seen such prompts, sends 55% of the problems to one expert that is good but not the best\.

Table 29:GSM8K, 1,319 test problems, seed\-0 split\. Solution NLL in nats per token \(lower is better\) and answer accuracy \(higher is better\) for models trained on the original corpus, with unsafe targets removed, and with safe responses and a prompt\-masked loss\. The base model scores 0\.668 and 0\.706\.The accuracy result holds over three data splits of the original corpus\. With the train\-only trait space of Appendix[E](https://arxiv.org/html/2611.25167#A5)on every split and the dense gate at a1×161\\times 16micro\-batch, the game’s mixture answers0\.414±0\.0560\.414\\pm 0\.056of the problems against0\.207±0\.0310\.207\\pm 0\.031for the pooled adapter and0\.222±0\.0200\.222\\pm 0\.020for the dense gate, a paired lead of0\.207±0\.0330\.207\\pm 0\.033over the pooled adapter and0\.192±0\.0440\.192\\pm 0\.044over the dense gate\. The generations explain the gap\. The pooled adapter and the dense gate answer in one short line \(median 20 and 23 new tokens against 283 for the base model\), whereas the game’s experts keep more of the worked solution \(median 70–73 tokens\)\. The game also inherits one failure from the corpus: on every split the pair trained on the privacy data receives 118 to 176 name\-heavy problems and rewrites them with the entities masked instead of solving them\. On pseudo\-Fisher axes the game’s mixture degrades least on solution NLL on all three splits, better than the pooled adapter by0\.043±0\.0130\.043\\pm 0\.013nats and than its dense gate by0\.032±0\.0140\.032\\pm 0\.014, and on the seed\-0 split it answers 0\.305 of the problems against 0\.215 for its dense gate\.

#### New datasets of the trained kinds\.

For each training category we take a public dataset that was never used in training and remove prompts that duplicate a training prompt exactly: PKU\-SafeRLHF\([Ji et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib23)\), TruthfulQA, a jailbreak\-classification set \(jackhhao/jailbreak\-classification\), PII\-Masking\-300k from the publisher of the privacy training data, XSTest,deepset/prompt\-injectionsand CoCoNot\([Brahman et al\., 2024](https://arxiv.org/html/2611.25167#bib.bib5)\)\. Table[30](https://arxiv.org/html/2611.25167#A19.T30)scores them with the full\-chat\-text NLL used throughout\. Both mixtures beat the pooled adapter on average, the game’s by 0\.045 nats and the dense gate’s by 0\.062, so the dense gate is ahead of the game by 0\.018\. Routing does not follow a dataset’s nominal category\. Except for the privacy set, only 3–26% of a new dataset’s prompts reach the expert that serves its category in distribution, and on XSTest, the injection set and CoCoNot the experts the router chooses instead are better than the category’s own expert, so the new prompts resemble other regions of the trait space\. A nearest\-centroid check agrees: test prompts land with their own benchmark 92% of the time on labelled axes, and new datasets land with their nominal category 26% of the time, with 25–29% on the label\-free spaces of Appendix[F](https://arxiv.org/html/2611.25167#A6)\. Each category is learned from one source, and a new dataset of the same kind is often closer to a different source\.

Table 30:Seven unseen datasets of the trained kinds, original\-corpus models of the seed\-0 split, full\-chat\-text NLL \(lower is better\)\. The last column is the share of prompts routed to the expert that serves the dataset’s category, with the in\-distribution share in parentheses\.The full\-text metric includes the prompt, which the fine\-tuned models learn to model\. On the response tokens of the three datasets that have reference responses \(PKU\-SafeRLHF, TruthfulQA and PII\-Masking\-300k, 2,772 records\), no mixture beats the pooled adapter on average on the original corpus: the pooled adapter scores 1\.331, the dense gate 1\.334 and the game’s mixture 1\.342, against 1\.543 for the base model\. The game’s hindsight\-best expert scores 1\.219, so the experts carry variety on these responses that neither router reaches on new data\. On pseudo\-Fisher axes over three data splits, both mixtures beat the pooled adapter on full text and the dense gate leads the game by0\.018±0\.0120\.018\\pm 0\.012nats, the labelled\-axes picture again\.

#### Why the game loses on new harm data\.

PKU\-SafeRLHF is the one new dataset on which the game trails the pooled adapter in Table[30](https://arxiv.org/html/2611.25167#A19.T30), and the per\-record scores locate the loss in the routing\. On PKU response tokens the harm expert alone matches the pooled adapter \(1\.8916 against 1\.8920\) and is the best expert for 72% of the records, but only 26% of them are routed to it, against 84% of the BeaverTails test prompts\. The game’s mixture therefore scores 1\.9421, while its hindsight\-best expert scores 1\.8820\. The label\-free pseudo\-Fisher axes route 50% of PKU to the harm expert and lose less \(1\.9219\)\. Two changes to the labelled trait space, each with the game and its pooled adapter retrained, do not move the PKU prompts \(Table[31](https://arxiv.org/html/2611.25167#A19.T31)\)\. The first rebuilds the harm category from two sources, 2,500 BeaverTails and 2,500 Aegis 2\.0 records\([Ghosh et al\., 2025](https://arxiv.org/html/2611.25167#bib.bib13)\)\. The two\-source harm expert now beats its own pooled adapter on PKU responses, but the routed share does not rise and the mixture still trails\. The second fits every labelled axis on prompt embeddings, where the harm and hallucination axes are otherwise fitted on responses\. It keeps the in\-distribution gain and again leaves the routed share unchanged or lower\. In every variant the PKU prompts sit at a mean harm coordinate of 0\.21–0\.22, against 0\.33 for the BeaverTails test prompts\. The failure on a new harm source therefore lies in where its prompts sit in the encoder’s embedding, and neither the text the axes are fitted on nor the number of sources per category changes it\.

Table 31:PKU\-SafeRLHF under changes to the labelled trait space, seed\-0 split, train\-only axes, response\-token NLL \(lower is better\) on the 942 PKU prompts absent from every compared corpus\. Routed is the share of PKU prompts whose largest share goes to the harm expert\. The last column is the game’s expected gain over its pooled adapter on the in\-distribution test pool, full chat text\.

## Appendix TReproduction

#### Environment and entry point\.

The implementation is a Python package requiring Python 3\.10 or later; dependency constraints and optional training, plotting, testing, and behavioral\-evaluation extras are specified inpyproject\.toml\. All experiments use the same YAML\-driven pipeline rather than experiment\-specific scripts\. The principal run can be validated and executed with:

```
python -m pip install -e \
  ".[ml,configs,figures,test,behavioral]"
python -m pytest
python -m infl_ens.pipeline \
  --config configs/experiments/seven_axis_theory_eq.yaml \
  --dry-run
python -m infl_ens.pipeline \
  --config configs/experiments/seven_axis_theory_eq.yaml
```

The dry run resolves every layered arm configuration and reports its output directory and trait\-space fingerprint without loading a model\. Experiment files for the parameter sweeps, model ablations, baselines, and behavioral evaluation are colocated underconfigs/experiments/; each declares its arms, stage order, evaluation partition, and figure builders\. Configurations that define a smoke block can additionally be checked with\-\-smokebefore a full GPU run\.

#### Data identity, seeds, and cached geometry\.

The shared split manifest stores stable record identifiers and the within\-benchmark train/validation/test assignment, so controlled arms consume identical examples rather than merely matching split sizes\. Training, splitting, routing, and generation seeds are fields of the resolved configuration\. Each completed run writes the fully flattenedresolved\_config\.yaml; this file, rather than the compositional arm fragment, is the authoritative record of model, LoRA, game, and evaluation settings\. The principal box\-coordinate trait space uses Qwen3\-Embedding\-8B\-AWQ\-INT4 and has configuration fingerprint3b42c68a8dd334c5; all corresponding arms validate this identity before loading the shared cache\.

#### Artifact contract\.

Every training run writeshistory\.jsonwith its round states andresolved\_config\.yamlwith its effective configuration; runs using the shared manifest also writedata\_split\.json\. Final routing summaries are reported in round\-11routing\_ensemble\_diagnostics\.json\. The underlyingmerge\_nll\.npy,g\_merge\.npy, andtoken\_counts\.npyarrays are sufficient to recompute the reported commitment, expected\-NLL, and probability\-mixture estimators without reloading model checkpoints\. Behavioral generation and score artifacts are schema\-versioned and record ordered\-input, checkpoint, trait\-coordinate, and content digests; the reported behavioral tables use these artifacts rather than reparsing model logs\. Experiment\-levelstage\_status\.jsonrecords stage completion and timing, while figure generation reads the saved artifacts without rerunning training\.

#### Numerical reproducibility caveat\.

The four\-bit AWQ trait encoder is not bitwise invariant across GPU architectures\. In a 206\-prompt cross\-hardware check spanning L40S, A5500, and H200 devices, recomputing coordinates changed the hard\-routing argmax for 2\.4–3\.4% of prompts\. Exact reproduction therefore requires reuse of the original trait\-space and behavioral\-coordinate caches, not merely the same encoder model identifier\. Behavioral coordinate caches are keyed by both the suite digest and trait\-space fingerprint, and their content hash is included in generation provenance; rebuilding coordinates on different hardware should be treated as a robustness replication rather than a bitwise reproduction\.

相似文章

通过有限专家库实现通信高效的专家路由

arXiv cs.LG

本文提出了一种针对稀疏混合专家(MoE)模型中通信高效专家路由的信息论框架,将门控机制视为随机信道,并推导实用的互信息估计器以分析有限专家库上的准确率-速率权衡。

在博弈论中,通才有时胜过专才

MIT News — Artificial Intelligence

麻省理工学院研究人员合著的一篇论文表明,通用策略梯度算法在不完全信息博弈中可以胜过专门的博弈论算法,挑战了该领域长期以来的假设。