Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data

# Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data
Source: [https://arxiv.org/html/2609.18842](https://arxiv.org/html/2609.18842)
Ross M\. ClarkeAffiliation:Boltzbit LimitedYichuan ZhangAffiliation:Boltzbit LimitedJosé Miguel Hernández\-LobatoAffiliation:University of CambridgeAffiliation:Boltzbit Limited

###### Abstract

The scaling laws hold that a language model grows more capable with more parameters and more training data, and Mixture\-of\-Experts \(MoE\) architectures have ridden these laws to remarkable results, activating only a fraction of an enormous stored parameter bank for each token\. That success is built on static pretraining data\. A deployed model faces a different world, where much of the data that would make it more useful is not in its training set but in the live interaction it is currently handling, such as the facts a user supplies or the corrections they give\. A conventional model cannot learn from this data, because its weights are frozen after training\. Instead, the knowledge and behaviour supplied at run time are placed in the prompt, by retrieval or instruction, and re\-read on every request only to be discarded once the request ends\. We ask how an architecture could learn from live interaction by writing it into its weights\. Taking inspiration from MoE, we propose theInfinite\-Parameter LLM\. A compact hypernetwork turns the data given at run time into a low\-rank modulation of a shared base network, so the feed\-forward weights are generated from live data rather than stored in a fixed bank\. Where prior weight generators read the context once and freeze, we carry a Bayesian belief over the generator’s latent code and update it online, so the effective weight is re\-derived from that evolving belief as the session proceeds rather than fixed after one read\. The stored footprint stays fixed, yet the weights the model can compile are effectively infinite\. For the knowledge and behaviour supplied at run time, carrying them in the weights rather than the prompt is amortized in compute, frees the context window, persists across turns, and can generalise better than in\-context use\. We specify an evaluation protocol that tests exactly this against in\-context learning and retrieval\.

## 1Introduction

For half a decade, the reliable way to a more capable language model has been to train a bigger one on more data, and the scaling laws make the dependence on data precise\. Capability rises predictably with the amount of training data, alongside parameters and compute\([Kaplan et al\., 2020](https://arxiv.org/html/2609.18842#bib.bib41);[Hoffmann et al\., 2022](https://arxiv.org/html/2609.18842#bib.bib34)\); data is a first\-class input to capability\. Two facts about*where*the data comes from now shape the problem\. The first is that the supply of static pretraining text is finite\. Current models are on track to exhaust the stock of public human\-generated text between roughly 2026 and 2032\([Villalobos et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib84)\), so the easy gains from simply pretraining on more of it are running down\. The second, and the one we build on, is that data has not stopped growing so much as changed form\. Deployed models, and increasingly the agents built on them, generate an enormous and fast\-growing stream of*interaction*data, from the questions users ask and the documents they bring to the corrections they give and the outcomes an agent observes\. This data is produced at inference, from real use, and it is exactly the data a model most needs to become useful to*this*user on*this*task\. If data is what buys capability, this is where the next of it will come from\.

The trouble is that today’s models cannot learn from this data in the loop\. A deployed model is frozen, and the interaction that just happened changes nothing about its weights\. The workarounds all keep the data*outside*the weights\. One family puts it in theprompt, where retrieval, long context, few\-shot examples, and system prompts carry the knowledge a model draws on and the behaviour it should follow in the context window, re\-read token by token on every request and discarded when the request ends\. The other family builds*around*the frozen model with agent harnesses, tool orchestration, and external memories, engineering scaffolding that manages data without ever changing the network\. Both avoid the harder question\. If the valuable new data is generated by use, the model should be able to*learn*from it, which means its weights must change\.

This is the question we pursue, and it concerns the*architecture and its weight\-update rule*, not the harness around it\. To learn from live interaction, a model needs weights that can take on new knowledge and behaviour cheaply, at inference, and hold onto what matters\. We take our inspiration for such an architecture from the Mixture\-of\-Experts models already at the frontier\.

A Mixture\-of\-Experts model stores a large bank of expert sub\-networks and routes each token through only a few of them; DeepSeek\-V3 holds 671B parameters yet activates 37B per token\([DeepSeek\-AI, 2024](https://arxiv.org/html/2609.18842#bib.bib20)\), and models such as Mixtral\([Jiang and others, 2024](https://arxiv.org/html/2609.18842#bib.bib39)\), Qwen3\([Qwen Team, 2025](https://arxiv.org/html/2609.18842#bib.bib70)\), Kimi K2, and Llama\-4 span a similar range\. Seen through the right lens, such a model is less a collection of separate experts than a single network whose weights vary with the input\. For a tokenxx, an MoE layer applies the effective weightWeff​\(x\)=∑igi​\(x\)​WiW\_\{\\text\{eff\}\}\(x\)=\\sum\_\{i\}g\_\{i\}\(x\)\\,W\_\{i\}, a combination of stored experts selected by an input\-dependent gate, which is precisely the conditional\-computation construction of[Bengio et al\. \(2013\)](https://arxiv.org/html/2609.18842#bib.bib10)and[Shazeer et al\. \(2017\)](https://arxiv.org/html/2609.18842#bib.bib77)\. A MoE is, in this sense, a dense feed\-forward network endowed with*dynamic*, per\-token weights, and it is this property, not its parameter count, that we take as our starting point\. The weights of a model*can*be made a function of the input rather than fixed constants\. Two things about the MoE realisation of that idea limit it for our purpose, however\. Its dynamic weights are bought in memory, because although only a few experts are activated for any token the router may select any of them, so all must remain resident\. And the bank it selects from is fixed once training ends, identical for every user and every moment of use, so a model serving a doctor and a novelist, at 9am and at midnight, reaches into the same unchanging palette\. An MoE varies its weight*with the token*, but it can no more learn from the interaction in front of it than a dense model can\. We keep the dynamic\-weight idea and drop both limitations, generating the weights from a compact network instead of storing a bank, and letting them keep changing after the token that produced them\.

Generation is what lets the weight carry what the prompt normally carries\. It does not expand what a small model can*store*, since the information a network holds is bounded by its parameters, near two bits per parameter for MoE and dense models alike\([Allen\-Zhu and Li, 2024](https://arxiv.org/html/2609.18842#bib.bib5)\); a compact generator obeys the same bound as any other network its size\. But storage is not the point\. The comparison generation sets up is not against a bigger model, it is against the*prompt*\. Today the knowledge and behaviour a model needs at run time are supplied in the context as retrieved facts, a task instruction, or a few demonstrations\. A generator can instead*compile that same data into the weights*, so a user’s facts become a weight that answers without the facts in the prompt\. This is the “hypernetwork as encoder” that a growing line of work has shown to work\([Charakorn et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib15)\)and to scale, with injected knowledge improving as a power law in the generator’s size and generalising*better*out of distribution than the same knowledge left in context\([Dhankhar et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib21)\)\. Behaviour compiles the same way, since an instruction or a set of demonstrations is data the generator can read into the adapter rather than into the prompt\. And because task adaptation occupies subspaces of strikingly low intrinsic dimension\([Aghajanyan et al\., 2021](https://arxiv.org/html/2609.18842#bib.bib3)\), a compact generator and a low\-dimensional code are the right\-sized tools for this, not undersized ones\. Carrying the data in the weights, rather than re\-reading it from the context on every token, is amortized in compute, frees the context window, and persists past the turn\. None of this is available to a prompt, which is re\-read whole on every request and forgotten at the end of it\.

There is, however, a gap between “generate from data” and “dynamic per token” that the existing generative work leaves open, and closing it is our contribution\. The weight generators that turn a context into an adapter do so*once*\. They read the whole context in a single pass, emit one adapter, and then hold it fixed while the model answers\([Charakorn et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib15)\)\. That is turn\-level, and it is*memoryless*, since the adapter does not evolve as more tokens arrive and nothing is carried from one turn to the next\. To match the view we started from, in which an MoE varies its weight*every token*, and to let the model keep adapting*within*a session rather than resetting each turn, the latent code cannot be read once and frozen\. It must be inferred online\. We therefore treat the code as a latent variable with its own prior and carry a*belief*over it, updated as evidence arrives, coarsely once per turn or finely every token, by an amortized recursive Bayesian filter\. The turn\-level generator supplies the belief with a strong measurement from the live facts, and the online filter keeps the belief moving between measurements\. Together they give a weight that is both generated from data and evolving in time\.

Carried to its conclusion, this produces a model whose stored footprint is fixed but whose*effective*weights are not, since it keeps compiling fresh ones from whatever data the run brings\. We call it an infinite\-parameter LLM, and intend the term precisely and narrowly\. It names the unbounded set of*effective weights and behaviours*the model can realise, a fresh expert per token, produced from a continuous code and moved by an evolving belief\. The contrast is with an ordinary model, whose weights are fixed after training and whose only channel for anything new is the prompt\. There the reachable behaviours are whatever the fixed weights plus a bounded context allow, whereas here the weights themselves are recompiled from live data at every step\. Concretely, we replace the stored expert bank with a compact generator that synthesises each token’s expert on demand as a low\-rank modulation of a shared base network, drive that generator from the run\-time data, and carry a Bayesian belief over the latent code that the generator reads, updated online over the course of a session\.[Figure1](https://arxiv.org/html/2609.18842#S1.F1)places this regime in the landscape of ways a model turns data into capability;[Figure2](https://arxiv.org/html/2609.18842#S2.F2)\([§2\.7](https://arxiv.org/html/2609.18842#S2.SS7)\) then locates our architecture against the specific prior designs it draws on\.

the model’s weights change:\(and on fresher data\)onceoccasionallycontinuouslyPre\-training *data:*static web corpus *updates:*all weights *when:*once, offline *after:*frozenPost\-training *data:*curated demos / prefs *updates:*most weights *when:*occasionally, offline *after:*frozenLive\-learning\(this work\) *data:*live interaction *updates:*a generated code *when:*continuously, at inference *after:*keeps adapting, reversibleoffline, batch — weights frozen after the updateonline, per\-interactionIn\-context learning / retrieval\(weights never change\)data lives in the prompt, re\-read on every request, then discardedboth act at inference

Figure 1:Three regimes for turning data into model capability, ordered by how often the model’s weights change and how recent the data they learn from is\.Pre\-trainingandpost\-training\(SFT, RLHF\) both update the weights offline, in batch, and leave them frozen thereafter; they differ mainly in the data they use and how often they run\.Live\-learning, the regime this work targets, updates a generated low\-rank code continuously, at inference, on the interaction data — facts, corrections, outcomes — that the others cannot reach in the loop, and keeps adapting rather than freezing\. In\-context learning and retrieval \(bottom\) also act at inference, but they leave the weights unchanged and carry the data in the prompt, where it is re\-read every request and then discarded\. The regimes are complementary, not competing: live\-learning does not replace pretraining \([§4](https://arxiv.org/html/2609.18842#S4)\), it reaches the data pretraining and prompting leave on the table\.This closes the loop we opened with\. The data that is still growing is generated by use, and a model whose weights are compiled from that data can turn it into capability at inference, in the loop, rather than only at the next pretraining run\. As an interaction accumulates, more of it is written into the weights, and the model becomes more useful on the task at hand\. We are careful about the scope of this claim\. Session adaptation is bounded, low\-dimensional, and reversible \([§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\), and it does not repeal the capacity law or substitute for pretraining\. The loop we enable is that live data can*enter the weights and be used*, closing a path that a frozen model, prompted or scaffolded, leaves open, not that a model grows without limit from its own exhaust\.

The design that follows is built around this weights\-versus\-prompt comparison\. Knowledge and behaviour enter from the run\-time data the generator reads, and what we generate and adapt is the low\-dimensional part, namely which weight best fits the current context and how it should drift as the session goes on\. Whether carrying data in the weights actually beats carrying it in the prompt, at matched budget, is what our evaluation measures \([§4](https://arxiv.org/html/2609.18842#S4)\)\.

We situate the proposal within a natural progression along two axes at once, where the weight comes from, and whether it can change after training\. Standard MoE and the bank\-free variants \(μ\\muMoE,∞\\infty\-MoE\)selectfrom a set that is fixed at deployment, and whether or not that set is unbounded, it is frozen\. The recent weight generatorsgeneratethe weight from context, but read the context once and then freeze the adapter for the turn\. We take the last step, an expert space that is bothgenerated from live data and adapted online, its weight moving with the data and with time, and make the following contributions\.

1. 1\.A generative expert architecture and its design space\([§§3\.2](https://arxiv.org/html/2609.18842#S3.SS2)and[3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\)\. We make a shared base FFN’s weights dynamic through agenerated low\-rank delta driven by a latent code, with no stored expert bank, and set out a design space that positions∞\\infty\-MoE,μ\\muMoE, DFC, HyperMoE, and MoEGen by the axis on which each departs from this structure\. The concrete choices that instantiate it — the base, and the form of the generator — are made in[§3\.5](https://arxiv.org/html/2609.18842#S3.SS5)\.
2. 2\.The infinite\-parameter view\([§3\.4](https://arxiv.org/html/2609.18842#S3.SS4)\)\. A precise statement of the sense in which the expert space is unbounded, a continuous generated family, one expert per token, extended over time by adaptation, distinguished from unbounded*knowledge*, with a guiding analogy to Bayesian\-nonparametric mixtures of experts\.
3. 3\.Online adaptation as a belief over the latent code\([§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\), the element that distinguishes us from one\-shot weight generators\. Rather than reading the context once and fixing the adapter, we carry a belief over the latent code and update it as the interaction proceeds, at three cadences \(contextual, per\-turn, per\-token\) under one probabilistic formulation, with uncertainty\-gated retention, locating in\-context learning, one\-shot hypernetworks, continual\-learning posteriors, and fast weights as points within it\.
4. 4\.An evaluation protocol\([§4](https://arxiv.org/html/2609.18842#S4)\) aimed at the comparison the design actually faces, carrying knowledge and behaviour in the weights versus carrying them in the prompt\. The headline baselines are the prompt family, in\-context learning and retrieval, with one\-shot weight generators and point\-estimate test\-time training as adaptation baselines and stored\-bank MoE as a reference point\.

The ingredients each have precedent\. Our contribution is their*coupling*, a shared\-base low\-rank generator driven from live data by a continuous latent code, made*dynamic in time*by recursive Bayesian inference over that code\. Concurrent efforts that independently articulate the generate\-instead\-of\-store thesis are discussed, and our differences delimited, in[§2\.7](https://arxiv.org/html/2609.18842#S2.SS7)\.

## 2Related Work

Our proposal touches several mature literatures; we organise them below and state our position against the closest work in[§2\.7](https://arxiv.org/html/2609.18842#S2.SS7)\. We claim none of the individual ingredients in isolation\.

### 2\.1Mixture\-of\-Experts: from stored banks to bank\-free selection

The mixture\-of\-experts idea originates with adaptive mixtures of local experts and their hierarchical, EM\-trained form\([Jacobs et al\., 1991](https://arxiv.org/html/2609.18842#bib.bib38);[Jordan and Jacobs, 1994](https://arxiv.org/html/2609.18842#bib.bib40)\), and the underlying principle of conditional computation, activating input\-dependent parts of a network for capacity without proportional cost\([Bengio et al\., 2013](https://arxiv.org/html/2609.18842#bib.bib10);[Bengio et al\., 2015](https://arxiv.org/html/2609.18842#bib.bib9)\)\. Under the scaling\-law paradigm, where capacity reliably buys capability\([Kaplan et al\., 2020](https://arxiv.org/html/2609.18842#bib.bib41);[Hoffmann et al\., 2022](https://arxiv.org/html/2609.18842#bib.bib34)\), this made sparsity the default route to cheap capacity: an MoE layer applies a per\-token effective weightWeff​\(x\)=∑igi​\(x\)​WiW\_\{\\text\{eff\}\}\(x\)=\\sum\_\{i\}g\_\{i\}\(x\)W\_\{i\}, a dense FFN whose weights are chosen conditionally on the input\. Sparsely\-gated MoE realised this at scale\([Shazeer et al\., 2017](https://arxiv.org/html/2609.18842#bib.bib77);[Fedus et al\., 2022](https://arxiv.org/html/2609.18842#bib.bib26)\), with subsequent work pursuing finer\-grained experts and an always\-on*shared expert*\(DeepSeekMoE;[Dai et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib19)\), a design that directly parallels our always\-applied shared base FFN, and very large expert counts via retrieval \(PEER;[He, 2024](https://arxiv.org/html/2609.18842#bib.bib32)\) built on product\-key memory\([Lample et al\., 2019](https://arxiv.org/html/2609.18842#bib.bib49)\)and related memory layers\([Berges et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib11)\), with the returns to sparsity themselves the subject of MoE scaling laws\([Clark and others, 2022](https://arxiv.org/html/2609.18842#bib.bib18);[Krajewski et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib46);[Abnar and others, 2025](https://arxiv.org/html/2609.18842#bib.bib2)\)\. All of these*store*their experts\. Dense\-to\-MoE “upcycling” makes this explicit, replicating a dense FFN into a stored bank\([Komatsuzaki et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib45)\), the replicate\-and\-store move we invert\. Softer relaxations reduce discreteness but not storage: soft merging of stored experts \(SMEAR;[Muqeeth et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib62)\), scaled to autoregressive pre\-training \(Lory;[Zhong et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib98)\)\. A separate line reaches an*unbounded but frozen*expert set without a stored bank:μ\\muMoE\([Oldfield et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib65),[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2)\)factorises a fixed weight tensor, and∞\\infty\-MoE\([Takashiro et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib82)\)draws a per\-token continuous latent code from a Gaussian router and uses it as a top\-N%N\\%*activation mask*over one shared FFN\.∞\\infty\-MoE shares with us a shared base steered by a low\-dimensional per\-token latent code, but the resemblance is superficial\. Its operator is a*multiplicative mask*that only reweights the existing neurons of a vanilla \(non\-gated\) FFN, essentially giving an old\-style FFN a GLU\-like gate, so on a modern SwiGLU base, which already gates multiplicatively, the mechanism is largely redundant with the architecture\. We instead generate an*additive low\-rank delta*that steers neurons along new pre\-activation directions, drive it from live data rather than a fixed router, and, the difference with no analogue in a frozen router, adapt the latent code online\. We treat∞\\infty\-MoE andμ\\muMoE as the frozen\-selection contrast, not as the precedent our method extends\.

### 2\.2Compressing and factorising experts

A large literature makes experts cheaper\. Low\-rank or vector experts over a shared base recover most of full\-expert quality at a fraction of the parameters \(MoV/MoLORA,[Zadouri et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib94); MixLoRA,[Li et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib52); MoLE,[Wu et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib89); X\-LoRA,[Buehler and Buehler, 2024](https://arxiv.org/html/2609.18842#bib.bib13); LoRAMoE,[Dou et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib24)\), but retain a stored bank\. Multilinear MoE \(μ\\muMoE;[Oldfield et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib65)\) is the closest “do not store experts” precedent: it represents the whole bank as a single CP\- or Tensor\-Ring\-factorised weight tensor that is never materialised, routed by a differentiable entmax gate, so, like us, it stores no individual experts\. The difference is thatμ\\muMoE*factorises a fixed tensor and routes linearly over it*, confining each token’s effective weight to the convex hull of a fixed atom set, whereas we*generate*the factors from a latent code that is itself produced from live data \([§3\.2](https://arxiv.org/html/2609.18842#S3.SS2)\); it is also not adaptable\. Becauseμ\\muMoE already achieves an un\-materialised bank, we do not rest our contribution on the absence of storage but on generating the code from run\-time data and adapting it online\. Orthogonally, resident memory is reduced by quantising and decoding experts on the fly \(QMoE;[Frantar and Alistarh, 2023](https://arxiv.org/html/2609.18842#bib.bib28)\), pruning or skipping experts\([Lu et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib59)\), merging them \(HC\-SMoE,[Chen et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib17); MEO,[He et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib31)\), offloading\([Eliseev and Mazur, 2023](https://arxiv.org/html/2609.18842#bib.bib25)\), or distilling an MoE into a dense model\([Xue et al\., 2022](https://arxiv.org/html/2609.18842#bib.bib92)\), as surveyed by[Liu et al\. \(2024\)](https://arxiv.org/html/2609.18842#bib.bib56)\. These compress a*stored bank*; we remove the bank and generate experts instead\.

### 2\.3Hypernetworks and generated experts

Hypernetworks generate a target network’s weights\([Ha et al\., 2017](https://arxiv.org/html/2609.18842#bib.bib30)\); more generally, the dynamic\-weight\-tensor view treats any layer whose weights are an input\-dependent function, made tractable by CP factorisation of the generated tensor \(DFC;[Babiloni et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib7)\), the general form our[§3\.2](https://arxiv.org/html/2609.18842#S3.SS2)specialises to the FFN\. Lightweight conditioning primitives such as FiLM\([Perez et al\., 2018](https://arxiv.org/html/2609.18842#bib.bib66)\)and \(IA\)3\([Liu et al\., 2022](https://arxiv.org/html/2609.18842#bib.bib55)\)modulate a shared computation from an input\-dependent signal, the same family as an additive or multiplicative weight modulation from a code\. The most direct precedents generate a PEFT module for a frozen LLM in a single forward pass over a context or task description \(HyperTuning,[Phang et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib67); Text\-to\-LoRA,[Charakorn et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib15); Doc\-to\-LoRA,[Charakorn et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib16); SHINE,[Liu et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib57); Drag\-and\-Drop LLMs,[Liang et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib54); Zhyper,[Abdalla et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib1)\), and the concurrent injection\-scaling work\([Dhankhar et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib21)\)shows this route*scales*: knowledge injected into a generated adapter improves as a power law in the hypernetwork’s size and generalises out of distribution better than a stored LoRA or full fine\-tuning\. This line is the closest to ours and the one we build on: it establishes that a hypernetwork acting as an*encoder of live data*injects knowledge that the model then uses without the data in context, the “knowledge from data, not from a bigger bank” leg of our design\. These generators differ sharply in how they read the data, and at what cost\. At one end, Text\-to\-LoRA reads only a short*task description*into a single embedding and generates the adapter from a small MLP, adding well under a percent of the base’s parameters\([Charakorn et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib15)\)\. At the other, SHINE reads the*full context*by reusing the frozen backbone itself as the encoder, appending learnable memory tokens processed under an auxiliary “Meta LoRA” and mapping their all\-layer hidden states to the adapter through a dedicated memory\-to\-parameter transformer; this reads context far more richly but adds on the order of a sixth of the base’s parameters \(roughly\(L′/L\+2​r/H\)​P\(L^\{\\prime\}/L\+2r/H\)\\,P, about17%17\\%for their Qwen3\-8B setting;[Liu et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib57)\)\. This span, from a compact description\-encoder to a backbone\-reusing context\-encoder, sets the sizing question our own encoder faces \([§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\)\. The gap we close is orthogonal to it: all of these read the context*once*and then*freeze*the adapter, so the generated weight is turn\-level and memoryless, unchanged as the model reads on and reset from one turn to the next \(SHINE’s recurrent variant chunks a long context but still produces a fixed adapter, not an evolving one\)\. We keep the encoder\-of\-data generator and add what it lacks, a belief over the latent code that keeps moving as the interaction proceeds \([§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\)\. Within MoE, HMoE\([Qu et al\., 2022](https://arxiv.org/html/2609.18842#bib.bib69)\)and HyperMoE\([Zhao et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib97)\)generate expert modulations from a low\-dimensional latent code but retain the stored bank;[Zhao et al\. \(2024\)](https://arxiv.org/html/2609.18842#bib.bib97)report that conditioning the generator directly on the token can underperform a standard MoE, the optimisation difficulty our compact latent bottleneck \([§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\) targets\. The effort closest to our generator \(see[§2\.7](https://arxiv.org/html/2609.18842#S2.SS7)\) is MoEGen\([Zeng et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib96)\), which generates instance\-specific LoRA updates from a shared hypernetwork, though on the attention projections and from a per\-prompt, top\-kkdiscrete latent code, without online adaptation\. A documented failure mode across weight generators is memorisation rather than generalisation\([Zeng et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib95)\), which we treat as a first\-class evaluation concern \([§4](https://arxiv.org/html/2609.18842#S4)\)\.

### 2\.4Inference\-time adaptation and fast weights

Adapting weights at inference descends from fast\-weight programmers\([Schmidhuber, 1992](https://arxiv.org/html/2609.18842#bib.bib75);[Ba et al\., 2016](https://arxiv.org/html/2609.18842#bib.bib6);[Schlag et al\., 2021](https://arxiv.org/html/2609.18842#bib.bib74)\)\. Test\-time training updates weights by self\-supervised gradient steps, as a sequence primitive\([Sun et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib81)\), a long\-term memory\([Behrouz et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib8)\), reinforcement\-learned self\-edits\([Zweiger et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib99)\), or per\-task adapters that surpass in\-context learning on novel structure\([Akyürek et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib4)\); all produce point estimates\. Methodologically, these approaches embed an updatable state*within the sequence\-mixing layer*and update it by a hand\-designed gradient or “surprise” rule; we instead leave attention unchanged, adapt only the generated FFN experts, and replace the hand\-designed update with an amortized approximation to an explicit Bayesian filter \([§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\), yielding calibrated retention rather than a point estimate\. Any per\-token belief we carry is a low\-dimensional FFN\-side filter, adding no recurrent state to the attention/sequence\-mixing path\. Test\-time compute can instead be spent on search or sampling against a verifier\([Snell et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib79)\), a matched\-budget baseline for us\. Closest in spirit are online MoE adaptations: continuous rerouting via gradient\-updated router\-logit deltas \(Rewiring Experts;[Su et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib80)\) and gradient\-free, uncertainty\-guided Bayesian adaptation of expert confidence in medical vision\-language models \(MoBE;[Imam et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib37)\)\. Both adapt the*usage of a fixed expert bank*, not the latent code of a generated manifold\.

### 2\.5Continual, online, and Bayesian foundations for adaptation

Continual and online learning study exactly the problem of updating a model over time without erasing what it knows, the stability–plasticity trade and its failure mode, catastrophic forgetting\([McCloskey and Cohen, 1989](https://arxiv.org/html/2609.18842#bib.bib61);[Kirkpatrick et al\., 2017](https://arxiv.org/html/2609.18842#bib.bib44)\)\. Its three families, regularisation \(EWC; online EWC in*Progress & Compress*,[Schwarz and others, 2018](https://arxiv.org/html/2609.18842#bib.bib76)\), replay \(GEM,[Lopez\-Paz and Ranzato, 2017](https://arxiv.org/html/2609.18842#bib.bib58)\), and architecture growth \(Progressive Networks,[Rusu et al\., 2016](https://arxiv.org/html/2609.18842#bib.bib73)\), together with distillation\-based variants \(Learning without Forgetting,[Li and Hoiem, 2017](https://arxiv.org/html/2609.18842#bib.bib53)\) all target durable adaptation;[van de Ven and Tolias \(2019\)](https://arxiv.org/html/2609.18842#bib.bib83)taxonomise the settings, and recent work carries the problem to LLMs \([Wu and others, 2024](https://arxiv.org/html/2609.18842#bib.bib88); O\-LoRA,[Wang et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib85)\)\. A complementary line shows that fixed\-capacity networks progressively*lose plasticity*under continual updates\([Dohare and others, 2024](https://arxiv.org/html/2609.18842#bib.bib23)\)\. We take two things from this literature\. The*framing*: our uncertainty\-gating is a stability–plasticity controller that spends plasticity where the posterior is uncertain and protects it where confident, so live adaptation increases the*diversity of weight configurations realised over a session*rather than the stored parameter count\. The*machinery*: the recursive posterior\-as\-prior update \(below\)\. We differ by relocating this from full\-weight, offline, task\-sequential training to a low\-dimensional,*generated*latent code updated online at inference, forward\-only and anchored to base, so adaptation is bounded and reversible rather than a permanent consolidation\.

Probabilistic treatments of MoE run from the original mixtures\([Jacobs et al\., 1991](https://arxiv.org/html/2609.18842#bib.bib38);[Jordan and Jacobs, 1994](https://arxiv.org/html/2609.18842#bib.bib40)\)through Bayesian hierarchical mixtures of experts\([Waterhouse et al\., 1996](https://arxiv.org/html/2609.18842#bib.bib87)\), nonparametric infinite MoE via a Dirichlet\-process gate\([Rasmussen and Ghahramani, 2002](https://arxiv.org/html/2609.18842#bib.bib71)\), feature\-allocation priors with unboundedly many latent features finitely active\([Griffiths and Ghahramani, 2011](https://arxiv.org/html/2609.18842#bib.bib29)\), and modern identifiability/convergence theory for softmax gating\([Nguyen et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib64)\)\. For LLM\-scale adaptation, Bayesian posteriors over low\-rank adapters are tractable \(Laplace\-LoRA,[Yang et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib93); BLoB,[Wang et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib86)\), and post\-hoc structured Laplace has been applied to MoE expert layers \(Bayesian\-MoE;[Dialameh et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib22)\)\. Our online update is recursive Bayesian filtering \(variational continual learning,[Nguyen et al\., 2018](https://arxiv.org/html/2609.18842#bib.bib63); online Laplace,[Ritter et al\., 2018](https://arxiv.org/html/2609.18842#bib.bib72), building on[Kirkpatrick et al\., 2017](https://arxiv.org/html/2609.18842#bib.bib44); low\-rank extended Kalman filtering,[Chang et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib14)\), but applied to the generator’s low\-dimensional per\-layer latent code rather than to full weights or expert selection\. Amortizing such a filter, training a recognition network to emit the state update in a forward pass, places us in the deep state\-space / amortized\-filtering lineage \(deep Kalman filters,[Krishnan et al\., 2015](https://arxiv.org/html/2609.18842#bib.bib47); structured inference networks,[Krishnan et al\., 2017](https://arxiv.org/html/2609.18842#bib.bib48); deep variational Bayes filters,[Karl et al\., 2017](https://arxiv.org/html/2609.18842#bib.bib42); Kalman VAEs,[Fraccaro et al\., 2017](https://arxiv.org/html/2609.18842#bib.bib27)\), and we distinguish it from Kalman methods used as*training\-time optimizers*over weights, whose observation is the loss rather than a predictive likelihood over a latent code \(KOALA\+\+;[Xia et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib90)\)\.

### 2\.6Conditioning on run\-time data through the prompt

The incumbent way to make a deployed model use run\-time data is to place that data in the context\. In\-context learning conditions a frozen model on instructions or a few demonstrations supplied at inference\([Brown et al\., 2020](https://arxiv.org/html/2609.18842#bib.bib12)\), and can be read as implicit Bayesian inference over a latent concept the context selects\([Xie et al\., 2022](https://arxiv.org/html/2609.18842#bib.bib91)\); retrieval\-augmented generation fetches relevant text into the context so the model can draw on knowledge it does not store\([Lewis et al\., 2020](https://arxiv.org/html/2609.18842#bib.bib51)\), with nearest\-neighbour language models a non\-parametric variant that interpolates an external datastore at the output\([Khandelwal et al\., 2020](https://arxiv.org/html/2609.18842#bib.bib43)\)\. Long\-context modelling and soft prompt\- or prefix\-tuning\([Lester et al\., 2021](https://arxiv.org/html/2609.18842#bib.bib50)\)are further points on the same axis, enlarging or learning the conditioning signal while the model’s own weights stay fixed\. All of these carry the run\-time knowledge and behaviour*in the context*, where it is re\-read on every request, competes for a bounded context window, and is discarded when the request ends; agent harnesses and external memories likewise manage this data around a frozen model rather than writing it into one\. Our design targets the same goal by the opposite route, compiling that data into the weights, and[§4](https://arxiv.org/html/2609.18842#S4)makes in\-context learning and retrieval the primary baselines against which the weight\-carried alternative is measured\.

### 2\.7Positioning: how this work differs

No confirmed prior work combines the full stack we describe, so we position against it on the two axes that survive the reframe:where the weight comes from, selected from a stored bank versus generated from data, and, once generation is granted,whether the weight keeps moving after it is produced, frozen for the turn versus carried as an online\-updated belief\. Underneath both sits the paradigm contrast that motivates the work, whether run\-time knowledge and behaviour are carried in the*weights*or in the*prompt*; the whole generate\-and\-adapt family lives on the weights side of that line, and in\-context learning and retrieval on the prompt side \(we treat these as the primary evaluation baselines in[§4](https://arxiv.org/html/2609.18842#S4), not as architectural precedents\)\.[Figure2](https://arxiv.org/html/2609.18842#S2.F2)lays out the two architecture axes and the single cell each prior method occupies;[Table1](https://arxiv.org/html/2609.18842#S2.T1)places the closest lineage, the context\-driven weight generators, against the axes in detail; and[Table2](https://arxiv.org/html/2609.18842#S2.T2)does the same on the adaptation axis specifically\. We give stored\-bank MoE only the two\-axis summary and not a row\-by\-row scorecard: as[§1](https://arxiv.org/html/2609.18842#S1)argued, a stored MoE is the*inspiration*our design departs from and a reference point, not a method we compete with benchmark\-for\-benchmark, so the detailed comparisons below are with the generator and test\-time\-adaptation lines that are genuinely close to us\.

generate, don’t storemake the weight live in timeWeight is*frozen*once producedWeight*evolves online*\(belief over the code\)Selected froma stored bankGeneratedfrom dataselect from a fixed bankdiscrete MoE,μ\\muMoE,∞\\infty\-MoE\(all experts resident\)adapt a fixed bank’s*usage*usage shiftsRewiring, MoBE\(bank stays; routing/usage shifts\)generate once, then freezeGGonceText\-to\-LoRA, SHINE, MoEGen\(read context once; fixed for the turn\)generate*and*adapte \(This work\)GGbeliefztz\_\{t\}evolves over the sessiontt

Figure 2:The two architecture axes of the design, and where prior work sits\.Down— where each token’s weightcomesfrom: selected from a stored, fully\-resident bank \(top\), or generated on demand from a compact resident generator \(bottom\); this is the “generate, don’t store” move, and it buys a fixed footprint\.Across— what happens to the weightafterit is produced: frozen once made \(left\), or carried as a belief over its latent code and updated online \(right\); this is the axis that makes the weightlive in time\. Stored\-bank MoE \(μ\\muMoE,∞\\infty\-MoE\) and the one\-shot weight generators \(Text\-to\-LoRA, SHINE, MoEGen\) each sit in a single cell; only the bottom\-right — generate the weight from live dataandkeep a moving belief over the code — is occupied by this work\. “Infinite parameters” is the reach this opens up: an unbounded set ofeffective weights and behavioursacross both facts and time, from a fixed resident footprint — not an unbounded store of knowledge, which the capacity laws forbid and we do not claim \([§§3\.4](https://arxiv.org/html/2609.18842#S3.SS4)and[4](https://arxiv.org/html/2609.18842#S4)\)\. Attention is unchanged throughout; the bounded\-vs\-unbounded geometry of a single generated layer is developed in[Figure4](https://arxiv.org/html/2609.18842#S3.F4)\.Table 1:The hypernetwork / generator lineage — the closest prior work — against the axes that matter once “generate rather than store” is granted\. The upper block generates a PEFT module from context in a single pass and thenfreezesit \(turn\-level, memoryless\); the middle block generates over attention, a stored bank, or the whole weight without a shared low\-rank base; the lower block selects from or adapts the usage of a fixed bank\. Only this work drives the generator fromlive, accumulatingdata and lets the produced weight keep moving, as a calibrated belief over the latent code\. “Δ\\Deltaover shared base FFN” marks our specific structure — an additive low\-rank modulation of one always\-applied base \(∼\\sim= partial:∞\\infty\-MoE masks a base rather than adding to it\); MoBE’s posterior is over labels, not weights\.WorkWhat drives the generated weightΔ\\Deltaoversharedbase FFNWeight after it is producedPost\-eriorHyperTuning\([Phang et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib67)\)task / context description✓frozen for the turn×\\timesText\-to\-LoRA\([Charakorn et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib15)\)task description✓frozen for the turn×\\timesDoc\-to\-LoRA\([Charakorn et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib16)\)a document✓frozen for the turn×\\timesSHINE\([Liu et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib57)\)in\-context prompt✓frozen for the turn×\\timesZhyper\([Abdalla et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib1)\)conditioning / task✓frozen×\\timesInjection scaling\([Dhankhar et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib21)\)fact corpus \(train\-time\)✓frozen once baked×\\timesMoEGen\([Zeng et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib96)\)per\-prompt discrete code×\\timesfrozen for the prompt×\\timesHyperMoE / HMoE\([Zhao et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib97);[Qu et al\., 2022](https://arxiv.org/html/2609.18842#bib.bib69)\)latent code×\\timesfrozen×\\timesDFC\([Babiloni et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib7)\)raw input×\\timesfrozen per input×\\timesμ\\muMoE /∞\\infty\-MoE \(frozen sel\.\)router over fixed atoms∼\\simfrozen×\\timesRewiring / MoBE \(fixed\-bank\)— \(adapts*usage*\)×\\timesevolves \(bank usage\)∼\\simInf\-params LLMs \(This work\)live data \+ running evidence✓evolves online \(belief overzz\)✓Read across these axes, the*generate\-instead\-of\-store*thesis is by now partly anticipated\. MoEGen frames the shift from expert*selection*to expert\-conditioned*generation*, DFC generates weights from the input in general, the Text\-to\-LoRA / SHINE line generates adapters from context and shows the route scales, and∞\\infty\-MoE andμ\\muMoE both reach an un\-materialised expert set, so we claim neither that thesis nor the absence of a stored bank as new\. The two genuinely unclaimed elements are \(i\) the*mechanism*as a specific point in the design space of[§3\.2](https://arxiv.org/html/2609.18842#S3.SS2), a generated low\-rank additive delta over a single shared base FFN, driven by a latent code produced from data; and \(ii\) the*coupling*, in which the generator is driven from*live data*and the latent code it reads is not fixed but carried as abelief updated onlineby recursive Bayesian inference\. The sharpest single distinction is against the generator line closest to us \(Text\-to\-LoRA, SHINE\): those read the contextonce and freezethe adapter, turn\-level and memoryless, whereas we carry an evolving belief, so the weight keeps moving within a session\. Distillation\([Hinton et al\., 2015](https://arxiv.org/html/2609.18842#bib.bib33)\), where we use it, is an*enabling*training choice and not a contribution; the reframed design does not rest on compressing a teacher bank\. Each rival misses at least one axis:∞\\infty\-MoE masks rather than generates and is frozen;μ\\muMoE factorises a fixed tensor with linear routing and is frozen; DFC generates a factor but over the input directly, with no shared base or online adaptation; Text\-to\-LoRA/SHINE generate from context but freeze the adapter; MoEGen generates over attention with a per\-prompt top\-kkcode and no online adaptation; Rewiring and MoBE adapt a fixed bank’s*usage*rather than a generated latent code\.

Table 2:Adaptation positioning: what each method adapts, where, and how\. The one\-shot weight generators \(Text\-to\-LoRA, SHINE\) sit at the top as the closest relatives on the “generate the weight” axis — they produce the adapter from context but freeze it for the turn; the test\-time\-training methods move a point estimate by gradient descent inside the sequence layer or over the whole model\. Ours is the only one to carry a calibrated posterior over a generated latent code, updated online\.WorkWhat’s adaptedWhere it livesUpdate rulePt\./dist\.GranularityUnc\.Forg\.Text\-to\-LoRA / SHINEgenerated LoRAFFN/attn adapterread context oncept\.per\-turn \(one\-shot\)—reset each turnTTT\([Sun et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib81)\)inner\-model weightsin sequence layergradientpt\.per\-token—implicitTitans\([Behrouz et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib8)\)memory MLPbranch beside attn\.gradient \+ momentumpt\.per\-token—gateαt\\alpha\_\{t\}SEAL\([Zweiger et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib99)\)full weightswhole modelRL→\\toSFTpt\.per\-task——Rewiring\([Su et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib80)\)router logitsMoE routergradientpt\.per\-segmententropy \(heur\.\)resetMoBE\([Imam et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib37)\)label statisticsfrozen expertsgradient\-free EMApost\. \(labels\)per\-sample✓—This work \(A–C\)generated latent codezzFFN\-side, attn\. frozenamortized Bayes filterdist\. overzzin\-ctx / turn / token✓\(prec\.\)✓\(QQ\)The adaptation axis tells the complementary story\. The closest relatives on the “generate the weight” axis, Text\-to\-LoRA and SHINE, produce the adapter from context but thenfreezeit for the turn and reset each turn, so the weight does not evolve as the interaction proceeds\. The test\-time weight\-adaptation methods do evolve the weight, but every LLM\-side one updates a*point estimate*by gradient descent, inside the sequence\-mixing layer \(TTT; Titans\), over the whole model by reinforcement \(SEAL\), or over a fixed bank’s router logits \(Rewiring\), while the sole Bayesian one keeps a posterior over*labels*, not parameters, by gradient\-free moment\-matching \(MoBE\)\. None both generates the weight from live data*and*carries a calibrated posterior over the generating latent code, updated by a distilled recursive filter with uncertainty\-gating and principled forgetting, on the FFN side with attention untouched\.

##### Concurrent work\.

MoEGen\([Zeng et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib96)\)appeared essentially concurrently and independently articulates part of the generate\-instead\-of\-store thesis; we cite it as concurrent, delimit our differences above, and do not claim priority over the shared framing\.

## 3Method: Generating and Adapting FFN Experts

### 3\.1Overview

We build on a standard decoder\-only transformer and leave attention untouched; only the feed\-forward \(FFN\) sub\-layer is changed, and only in a chosen subset of layers\. At each such*generative layer*, three components replace the usual FFN \([Figure3](https://arxiv.org/html/2609.18842#S3.F3)\): ashared base FFN, always applied; acompact generatorGϕG\_\{\\phi\}that maps a low\-dimensional*latent code*to a structured low\-rank modulation of that base; and abelief over the latent code, from which the code driving the generator is read and which is updated from live data and running evidence \([§§3\.2](https://arxiv.org/html/2609.18842#S3.SS2)and[3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\)\. The token’s effective expert is the base FFN plus the generated modulation\. Crucially,no expert bank is stored: each token’s expert is generated from its latent code and discarded, so the resident parameters are the base, the generator, and the small inference map, all of fixed size, while the set of experts the model can produce is unbounded \([§3\.4](https://arxiv.org/html/2609.18842#S3.SS4)\)\.

This one mechanism carries all three of our claims\.*Infinite parameters:*a fresh expert is generated from a latent code drawn from a continuous, data\-materialised space, so the model deploys an unbounded family of*effective weights*rather than reusing a finite stored bank \([§3\.4](https://arxiv.org/html/2609.18842#S3.SS4)\)\.*Knowledge and behaviour in the weights:*because the code is produced from the data supplied at run time, the effective weights come to carry what a prompt would otherwise carry, such as facts, an instruction, or a few demonstrations, entering through the weights rather than being re\-read from the context on every token \([§3\.2](https://arxiv.org/html/2609.18842#S3.SS2)\)\.*Adaptation:*because the expert comes from a latent code, and because we carry abeliefover that code rather than reading it once, the model keeps specialising as a session proceeds, with the transformer left unchanged \([§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\)\. We first fix notation and set out the belief\-over\-code framework \([§3\.2](https://arxiv.org/html/2609.18842#S3.SS2)\); describe how live data writes the belief and how per\-token inference moves it \([§§3\.2](https://arxiv.org/html/2609.18842#S3.SS2)and[3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\); make the infinite\-parameter claim precise \([§3\.4](https://arxiv.org/html/2609.18842#S3.SS4)\); and only then commit to the concrete architectural choices that instantiate the framework \([§3\.5](https://arxiv.org/html/2609.18842#S3.SS5)\)\.

beliefPPoverzzlatent codezzΔ\\Deltaweightscontrolproj\.BaseModelyyapplyData\-to\-Weight LLM\(dynamic weight generator\)per\-turn datageneratexxamortised per\-tokenposteriorq\(z∣x1:t\)q\(z\\mid x\_\{1:t\}\)per\-token signal

Figure 3:The architecture, organised around abelief over the latent codezz\. On the main \(per\-turn\) path, live data is read by theData\-to\-Weight LLM\(the encoderEϕE\_\{\\phi\}of[§3\.2](https://arxiv.org/html/2609.18842#S3.SS2)\) into a latent codezz; the code generates a low\-rank weight deltaΔ​W\\Delta Wthat modulates the frozenbase model, which produces the output\. AbeliefPPoverzzsits above the code and is what makes the weightmove: it is updated online, carried from step to step rather than re\-encoded from scratch\. The per\-token signal \(dashed\) — the running hidden state, equivalently the realised output, of the autoregressive stream — feeds the belief and is introduced only toamortise the per\-token posteriorq\(z∣x1:t\)q\(z\\mid x\_\{1:t\}\); it is not on the main data path\. The belief’s form — a Gaussian over a continuous code, or a categorical posterior over materialised codes — is the architectural choice of[§3\.5](https://arxiv.org/html/2609.18842#S3.SS5)\.
### 3\.2The belief\-over\-code framework

##### Notation and setup\.

We writeddfor the model width,hhfor the FFN hidden width,dzd\_\{z\}for the latent dimension, andrrfor the rank of a generated modulation, withr≪dr\\ll d\. Layers are indexed byℓ\\elland tokens within a sequence bytt; the input to a generative layer is the post\-attention hidden statex=hℓ,t∈ℝdx=h\_\{\\ell,t\}\\in\\mathbb\{R\}^\{d\}, which already integrates context through the layer’s attention and the residual stream\. The frozen, shared*base FFN*has weights collectively denotedW0W\_\{0\}and is initialised from a strong dense model \([§4\.1](https://arxiv.org/html/2609.18842#S4.SS1)\)\. The three moving parts are a*Data\-to\-Weight encoder*EϕE\_\{\\phi\}that reads run\-time data into a latent code, a*generator*GϕG\_\{\\phi\}that maps a codezzto a low\-rank modulationΔ​W​\(z\)\\Delta W\(z\)of the base, and a*belief*over the code that is updated online\. The code budget\(r,dz\)\(r,d\_\{z\}\), the base architecture, and the concrete values these symbols take are choices we fix in[§3\.5](https://arxiv.org/html/2609.18842#S3.SS5)\.

##### The primitive: a belief over the latent code\.

The object at the centre of the design is not a weight and not a code but abeliefover the code — a distributionP⁡\(z\)P\(z\)that the model carries and updates as it works \(we reserveqqfor the amortized approximation to it,[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\)\. Everything else is downstream of it: the code that drives the generator is a summary of the belief \(its mean, or its most probable atom\), the weight delta is a function of that code, and*adaptation is inference on the belief*\. Fixing the belief as the primitive, rather than the weight or a point code, is what lets one mechanism serve the two channels of[§1](https://arxiv.org/html/2609.18842#S1): ameasurement channel, by which run\-time data writes the belief \(this is where knowledge and behaviour enter the weights, later in this section\), and aninference channel, by which the belief moves between measurements as evidence accumulates \([§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\)\. The*form*of the belief — a Gaussian over a continuous code, or a categorical distribution over a pool of materialised codes — is an architectural choice we defer to[§3\.5](https://arxiv.org/html/2609.18842#S3.SS5); the framework, and the two channels, are the same either way\.

##### The generated weight\.

Given a codezz, the effective weight of any modulated base projectionW0W\_\{0\}is the base plus a generated low\-rank additive delta,

W⁡\(z\)=W0\+Δ​W​\(z\),Δ​W​\(z\)=B⁡\(z\)​A​\(z\)⊤,A⁡\(z\)∈ℝd×r,B⁡\(z\)∈ℝh×r,W\(z\)=W\_\{0\}\+\\Delta W\(z\),\\qquad\\Delta W\(z\)=B\(z\)\\,A\(z\)^\{\\top\},\\quad A\(z\)\\in\\mathbb\{R\}^\{d\\times r\},\\ B\(z\)\\in\\mathbb\{R\}^\{h\\times r\},\(1\)with the factorsA⁡\(z\),B⁡\(z\)A\(z\),B\(z\)produced from the code\. The delta is never materialised: we computeydelta=B⁡\(z\)​\(A​\(z\)⊤​x\)y\_\{\\text\{delta\}\}=B\(z\)\\,\\big\(A\(z\)^\{\\top\}x\\big\), so the per\-token application costs𝒪⁡\(r⁡\(d\+h\)\)\\mathcal\{O\}\(r\\,\(d\+h\)\)per layer, negligible relative to the base FFN’s𝒪⁡\(h​d\)\\mathcal\{O\}\(hd\)\. This is the framework; the specific base \(SwiGLU\), which projections carry a delta, and howA,BA,Bare produced are concrete choices made in[§3\.5](https://arxiv.org/html/2609.18842#S3.SS5)\. What matters for the framework is only that the weight is a function of a*code*, and the code is drawn from a*belief*\.

##### The measurement channel: writing the belief from live data\.

The belief is written from the data supplied for a turn — the facts, instruction, or examples — by the Data\-to\-Weight encoderEϕE\_\{\\phi\}\(the encoder\-hypernetwork of[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3)\)\. This is where the knowledge and behaviour that would otherwise sit in the prompt enters the weights:EϕE\_\{\\phi\}turns supplied data into a code \(or, categorically, into a*new materialised code*added to the pool,[§3\.5](https://arxiv.org/html/2609.18842#S3.SS5)\), and it is a strong, content\-rich measurement rather than a cheap per\-token guess\. A router over the running hidden state cannot, by itself, inject a fact the base was never given; only the measurement channel can, which is why the encoder and the per\-token inference of[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)are distinct modules\.

The encoder is where the design’s cost concentrates, and the framework spans a spectrum of realisations trading footprint against reading fidelity; we set out the axis rather than fix a point on it\. At the light end, the encoder reads the supplied data with the*base model’s own forward pass*\(which must process those tokens regardless\) and taps a small*readout head*on the resulting hidden states, so the head and generator are the only added parameters, at the scale of a description\-conditioned hypernetwork \(Text\-to\-LoRA’s smallest variant adds under a percent of the base;[Charakorn et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib15)\)\. In the middle, the backbone is reused as a*dedicated context\-encoder*with auxiliary read\-time adapters and a memory\-to\-parameter network, as in SHINE\([Liu et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib57)\), which reads long context more faithfully at the cost of on the order of a sixth of the base’s parameters\. At the heavy end, the encoder is a*completely separate hypernetwork*, not tied to the base’s weights at all, as in the knowledge\-injection hypernetworks of[Dhankhar et al\. \(2026\)](https://arxiv.org/html/2609.18842#bib.bib21), whose evidence is that injection fidelity scales with this hypernetwork’s capacity\. These are points on one axis — how much dedicated machinery reads the data into the code — and which is warranted is an empirical, footprint\-versus\-quality question \([§4\.3](https://arxiv.org/html/2609.18842#S4.SS3)\) rather than settled by fiat; a relevant consideration along the way is that the online belief \([§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\) can correct an imperfect one\-shot read that a frozen generator cannot, which can relieve a lighter encoder of carrying the whole burden in a single pass\. Attention and the base weights are frozen throughout\.

### 3\.3Moving the belief: online inference over the code

The element that separates this design from the one\-shot weight generators of[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3)is that the code is not read once and fixed; we carry the belief and update it as the interaction proceeds\. Adaptation, in every variant, is therefore*inference over the latent code*, with the generator and base frozen\. Thetransformer’s attention / sequence\-mixing path is left unchanged throughout: no variant inserts a recurrent state into the sequence layer, in contrast to test\-time\-training methods that adapt the sequence path itself\([Sun et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib81);[Behrouz et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib8)\)\. The only state carried across steps is the low\-dimensional belief, and it lives entirely on the FFN side\.

##### Why a belief and not a point\.

Existing test\-time adaptation carries a*point estimate*of the adapted weights and moves it by gradient descent \(TTT,[Sun et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib81); Titans,[Behrouz et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib8)\)\. Carrying instead a posterior over the code earns three things a point cannot\. First,calibration: the posterior’s spread is an explicit statement of how much to trust the adaptation, usable to gate, abstain, or defer when the model is uncertain\. Second,uncertainty\-gated stability–plasticity: a precision\-weighted update adapts fast where the posterior is unsure and protects what it is confident in, resisting catastrophic forgetting without a bolted\-on regulariser \(a derived analogue of elastic weight consolidation,[Kirkpatrick et al\., 2017](https://arxiv.org/html/2609.18842#bib.bib44)\)\. Third,principled forgetting: a process\-noise term gives a controlled, optionally content\-aware way to reopen plasticity when the input distribution shifts\. These benefits are carried by the posterior’s*spread*, which is exactly the fragile part under amortization, so “Bayesian” here is an empirical claim about a calibrated posterior, not a free consequence of emitting a distribution, and validating it — exact\-filter recovery, the amortization gap, and calibration of the posterior precision — is part of the continuous\-Gaussian instantiation we leave to future work \([§3\.5](https://arxiv.org/html/2609.18842#S3.SS5)\)\.

##### The exact update, and why we amortize it\.

The belief is updated by an exact recursive Bayesian filter, the same object at every cadence\. At an update, the observation over the tokens since the last update is either the model’s own log\-likelihoodℒ⁡\(z\)=∑ilog⁡pθ​\(xi∣x<i;z\)\\mathcal\{L\}\(z\)=\\sum\_\{i\}\\log p\_\{\\theta\}\(x\_\{i\}\\mid x\_\{<i\};\\,z\)\(the*self\-supervised*regime, always available\) or an explicit feedback likelihoodp⁡\(y∣context;z\)p\(y\\mid\\text\{context\};\\,z\): Boltzmann in a scalar reward,p⁡\(y∣z\)∝exp⁡\(rz/T\)p\(y\\mid z\)\\propto\\exp\(r\_\{z\}/T\), or Bradley–Terry for a pairwise preference \(the*feedback*regime\)\. The recursion is Bayes’ rule applied to the running posterior,

Pt​\(z\)∝Pt−1​\(z\)⋅p⁡\(obst∣z\),P\_\{t\}\(z\)\\;\\propto\\;P\_\{t\-1\}\(z\)\\;\\cdot\\;p\(\\text\{obs\}\_\{t\}\\mid z\),\(2\)carried from step to step rather than recomputed from scratch\. Throughout, we writePPfor this exact recursive belief andqqfor the amortized approximation to it that we actually run — the standard variational reading in which a learnedqqis fit to a targetPP\. Computing the likelihood term exactly requires a*test\-time backward pass*to the code, impractical per token at deployment, so the exact beliefPPis kept only as an offline reference \(a distillation teacher, and a comparison baseline\) and*amortized*: a trained forward mapFϕF\_\{\\phi\}emits the belief update in a single pass, its outputqqdistilled againstPP\([Putzky and Welling, 2017](https://arxiv.org/html/2609.18842#bib.bib68);[Marino et al\., 2018](https://arxiv.org/html/2609.18842#bib.bib60)\); the recognition\-network instance of a state\-space filter\([Krishnan et al\., 2015](https://arxiv.org/html/2609.18842#bib.bib47);[Karl et al\., 2017](https://arxiv.org/html/2609.18842#bib.bib42);[Fraccaro et al\., 2017](https://arxiv.org/html/2609.18842#bib.bib27)\)\. One property makes a*single*FϕF\_\{\\phi\}serve both cadences below: the exact update over a*window*of tokens is the same function of \(prior belief, accumulated observation\) whatever the window’s length, soFϕF\_\{\\phi\}reads the prior belief and a pooled summary of the window \(with a length feature\) and is distilled against the exact trajectory at both cadences\. This is the*per\-token signal*of[Figure3](https://arxiv.org/html/2609.18842#S3.F3), drawn dashed because it exists only to amortise the posteriorq\(z∣x1:t\)q\(z\\mid x\_\{1:t\}\)— it is not on the main data path, and switching it off returns the one\-shot generator\.

##### The three cadences\.

The designs place this one machinery at three points on the belief\-granularity axis, indexed by tokenttor turnτ\\tau; they are not three mechanisms but one belief updated more or less often\.

- •Design A — Contextual \(implicit belief\)\.No explicit belief is carried within a sequence; context is integrated by ordinary attention, and a routerRℓ​\(hℓ,t\)R\_\{\\ell\}\(h\_\{\\ell,t\}\)maps the contextual hidden state to the code\. Per\-token generation is then an amortized*predictive*inference, the forward pass approximating the Bayesian predictive in\-context\([Xie et al\., 2022](https://arxiv.org/html/2609.18842#bib.bib91)\)\. This is the cheapest variant and the*degenerate*member of the family — Bayesian only in the weak sense that in\-context learning implicitly approximates a posterior predictive, with none of the calibration, persistence, or controlled forgetting the explicit belief buys\. We keep it as thebaselinethe explicit\-belief designs must beat \([§4\.4](https://arxiv.org/html/2609.18842#S4.SS4)\)\.
- •Design B — Session posterior \(per\-turn update\)\.An explicit belief is maintained per layer and updatedonce per turnbyFϕF\_\{\\phi\}, from the prior belief and a pooled encoding of the turn \(and any feedback\)\. Because it fires only per turn, B can equally run the*exact*filter online — one backward pass per turn is affordable — making amortization optional here\. It gives persistent weight\-space adaptation at turn granularity and carries no per\-token state\.
- •Design C — Fast belief filter \(per\-token update\)\.The belief is carried as a side state and updatedevery tokenby the same amortized map,bt=Fϕ​\(bt−1,st\)b\_\{t\}=F\_\{\\phi\}\(b\_\{t\-1\},s\_\{t\}\), on a per\-token signalsts\_\{t\}; the generator reads its summary\. Here amortization is essential\. This is a genuine per\-token weight\-space update realised as a*benign, low\-dimensional recurrence outside the attention/sequence path*, the finest\-grained and fully persistent variant, at the cost of a small carried belief and a cheap forward\-only filter step per token\.

All three instantiate the same idea — a generated, continuously\-indexed expert space adapted by Bayesian inference over its latent code — and differ only in the granularity and persistence of that inference\. The family also locates prior work within one frame: discrete MoE and∞\\infty\-MoE are the*frozen*limit; in\-context learning is the contextual instance \(A\); and fast\-weight/TTT methods are per\-token updates placed in the*sequence layer*rather than, as in C, in a low\-dimensional FFN\-side belief\. Because test\-time gains tend to accrue with the*number*of updates rather than their size\([Sun et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib81)\), we expect C to dominate B under fine\-grained drift, with B the natural read\-out when the phenomenon and its labels live at turn granularity; since C run over the whole conversation subsumes B, our accumulation study \([§4\.4](https://arxiv.org/html/2609.18842#S4.SS4)\) updates at C and reports at the turn level\.

Table 3:The three adaptation cadences as one machinery — a belief over the code updated more or less often\. A carries no explicit belief \(the baseline\); B updates the belief once per turn; C every token\. The update rule is the recursive Bayes recursion of[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)in every case, differing only in the observation window; it is agnostic to the belief’s form \(the Gaussian or categorical realisations of[§3\.5](https://arxiv.org/html/2609.18842#S3.SS5)\)\.A — ContextualB — Session \(per\-turn\)C — Fast filter \(per\-token\)Belief updatenone \(implicit in context\)once per turnτ\\tauevery tokenttCarried statenonebeliefPτ​\(z\)P\_\{\\tau\}\(z\)beliefPt​\(z\)P\_\{t\}\(z\)Update rulerouter reads the codePτ∝Pτ−1⋅p⁡\(obsτ∣z\)P\_\{\\tau\}\\propto P\_\{\\tau\-1\}\\cdot p\(\\text\{obs\}\_\{\\tau\}\\mid z\)Pt∝Pt−1⋅p⁡\(xt∣z\)P\_\{t\}\\propto P\_\{t\-1\}\\cdot p\(x\_\{t\}\\mid z\)Wins whencontext suffices; short interactionstask shifts across turns; per\-turn feedbacklong single stream; fine\-grained drift

### 3\.4The infinite\-parameter view

We call the model an*infinite\-parameter*LLM in a precise sense: the set of experts reachable at a generative layer is\{W0\+Gϕ​\(z\):z∈𝒵\}\\\{\\,W\_\{0\}\+G\_\{\\phi\}\(z\):z\\in\\mathcal\{Z\}\\,\\\}, where𝒵\\mathcal\{Z\}is the space of codes the encoder can*materialise from data*\. The stored parameters — base, encoder, generator, and the small inference map — are finite and fixed; the reachable effective weights are not, because𝒵\\mathcal\{Z\}is not a fixed finite index but a space populated by whatever data the model is given\. Over an interaction the model instantiates a growing set of distinct weight configurations rather than reusing a fixed bank\.

This is where the categorical instantiation of[§3\.5](https://arxiv.org/html/2609.18842#S3.SS5)must be positioned carefully, because it*looks*like the finite selection the paper otherwise argues against\. The distinction is the origin of the atoms\. A classical MoE selects among afixed, storedbank of experts; its reachable set is the convex hull of those atoms — bounded, a*selection*\([Figure4](https://arxiv.org/html/2609.18842#S3.F4), left\)\. Our categorical belief is a posterior over a pool of atoms that are themselvesgenerated from databyEϕE\_\{\\phi\}: any new data materialises a new code, so the pool is unbounded and the atoms are drawn from a continuum, not enumerated in advance\. A categorical belief over a data\-materialised pool is thus the*finite, tractable working\-set representation*of a belief over an unbounded generated space — the same relationship a Dirichlet\-process mixture has to its infinite base measure, where any computation touches only a finite active set while the pool of possible components is unbounded\([Rasmussen and Ghahramani, 2002](https://arxiv.org/html/2609.18842#bib.bib71)\)\. The unboundedness the name claims therefore does not require a*continuous code at inference*; it requires that codes be*generated rather than stored*, which the measurement channel \([§3\.2](https://arxiv.org/html/2609.18842#S3.SS2)\) guarantees\. Selection over a*stored*bank is bounded; selection over a*generated*pool is not\.

Two clarifications keep the claim honest\. First, “infinite” is a statement about*reachable weight configurations*, not stored knowledge: knowledge remains bounded by the resident parameters\([Allen\-Zhu and Li, 2024](https://arxiv.org/html/2609.18842#bib.bib5)\), and “infinite” here never means a larger knowledge store\. Second, adaptation adds no parameters; it*re\-allocates plasticity*, since the belief’s uncertainty \([§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\) decides which latent directions stay plastic and which are protected, resolving the stability–plasticity trade\([Dohare and others, 2024](https://arxiv.org/html/2609.18842#bib.bib23)\)at inference rather than freezing it\. We are careful to claim only what is ours: that a layer’s weights can be made a data\-dependent function rather than a stored constant is established \(hypernetworks,[Ha et al\., 2017](https://arxiv.org/html/2609.18842#bib.bib30); dynamic layers,[Babiloni et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib7)\), and MoE is itself a dynamic\-weight layer with a finite index; our contribution is the specific coupling — a belief over a*generated*code space, written by live data and moved by online inference — not dynamic weights in the abstract\.

This positioning also separates us from the neighbouring generated\- and selected\-expert methods along one axis, the origin of the atoms and whether the belief moves:μ\\muMoE\([Oldfield et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib65)\)and discrete MoE*select*over a stored bank \(bounded\); DFC\([Babiloni et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib7)\)and MoEGen\([Zeng et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib96)\)*generate*an adapter and freeze it after one read;∞\\infty\-MoE\([Takashiro et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib82)\)masks subsets of one fixed network; and HyperMoE\([Zhao et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib97)\)generates a supplementary branch over a stored bank\. None carries an online belief over a*data\-materialised*pool, which is the coupling this paper adds\.

∑kck​\(z\)​W\(k\)\\sum\_\{k\}c\_\{k\}\(z\)\\,W^\{\(k\)\}selection\(linear routing over staticatoms:μ\\muMoE / dictionary / MoE\)reachable set==convex hull of a*finite*atom set\(bounded\)W0\+Gϕ​\(z\)W\_\{0\}\+G\_\{\\phi\}\(z\)*outside the hull*generation\(nonlinearGϕG\_\{\\phi\}over acontinuous latent codezz: ours\)reachable set==a curved manifold in*no*finite\-dim\. affine span\(unbounded\)teacher / anchor expertsexpert used for one token\- \- \-convex hull \(selection’s reach\)

Figure 4:Selection over a stored bank versus generation over a code space, on the same three anchor experts\.Left:routing over a finite set ofstoredatoms \(μ\\muMoE / discrete MoE\) reaches only their convex hull \(the triangle\); every routed expert lies strictly inside it — bounded\.Right:codesgenerated from databyEϕE\_\{\\phi\}populate a curved manifold that bulges beyond that hull \(shown dashed\), so a generated expertW0\+Gϕ​\(z\)W\_\{0\}\+G\_\{\\phi\}\(z\)can lie strictlyoutsideit — the reachable set is contained in no finite\-dimensional affine span, and is unbounded\. This is the geometric content of the infinite\-parameter claim: what matters is that the atoms aregeneratedrather thanstored, not whether the belief over them is continuous or categorical\. This paper’s categorical belief is a finite working set over this unbounded generated space — a moving slice of the right panel, not a return to the left\.
### 3\.5Architecture choice in this paper

The framework above is deliberately agnostic about the form of the belief and the shape of the generator\. We now commit to the choices this paper evaluates: the*form of the belief*\(categorical,[§3\.5\.1](https://arxiv.org/html/2609.18842#S3.SS5.SSS1)\), and the*generator and base*it drives \([§3\.5\.2](https://arxiv.org/html/2609.18842#S3.SS5.SSS2)\)\. The alternative — a continuous\-Gaussian belief with a nonlinear generator, the framework’s most expressive point — we note as a further direction at the end of this section rather than evaluate here\.

#### 3\.5\.1A categorical belief over materialised codes

We instantiate the belief overzzas acategoricaldistribution over a pool of codes\{m1,…,mK\}\\\{m\_\{1\},\\dots,m\_\{K\}\\\}, each materialised from data by the encoderEϕE\_\{\\phi\}\. The belief isPt​\(z\)=Cat⁡\(πt\)P\_\{t\}\(z\)=\\mathrm\{Cat\}\(\\pi\_\{t\}\)withπt∈ΔK−1\\pi\_\{t\}\\in\\Delta^\{K\-1\}, the code driving the generator is the posterior’s most probable atom \(top\-1\) or its mean, and the online update of[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)becomes recursive categorical Bayes,

πt,k∝πt−1,k⋅p⁡\(obst∣z=mk\),\\pi\_\{t,k\}\\;\\propto\\;\\pi\_\{t\-1,k\}\\;\\cdot\\;p\(\\text\{obs\}\_\{t\}\\mid z=m\_\{k\}\),\(3\)so that as the interaction proceeds the belief concentrates on the code that best explains the running evidence, and re\-opens when the evidence shifts\. This is the exact recursive filter of[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)specialised to a categorical latent; its amortization is a learnedselectorthat emits the posterior over the pool in a single forward pass\. Concretely, the selector scores the running hidden state against each code and normalises: at generative layerℓ\\ellwith the layer\-input activationuℓu\_\{\\ell\}as query and a learned keyκℓ​\(mk\)\\kappa\_\{\\ell\}\(m\_\{k\}\)per code,π∝exp⁡⟨uℓ,κℓ​\(mk\)⟩\\pi\\propto\\exp\\langle u\_\{\\ell\},\\kappa\_\{\\ell\}\(m\_\{k\}\)\\rangle\. Selection is top\-1 per layer, so a single generated expert is applied — not a top\-kkmixture — which keeps the operator a genuine weight rather than an averaged one; the measurement channel of[§3\.2](https://arxiv.org/html/2609.18842#S3.SS2)supplies the codes, and the selector supplies the cheap per\-step inference over them\. The two cadences of[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)carry over directly: per\-turn \(B\), the posterior is updated once per turn as questions accumulate over a fixed knowledge pool; per\-token \(C\), it is updated as the sequence streams\.

*\(The empirical study of this selector — how well the categorical posterior identifies the code that carries the answer, what signal drives it, and where in the network the routing signal lives — is the subject of[§4\.3](https://arxiv.org/html/2609.18842#S4.SS3)\.\)*

#### 3\.5\.2The low\-rank generator

For the generator and its base we adopt the concrete pipeline of SHINE\([Liu et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib57)\)essentially unchanged, and materialise the categorical pool of[§3\.5\.1](https://arxiv.org/html/2609.18842#S3.SS5.SSS1)by running it once per knowledge set\. The base is a strong dense SwiGLU model\([Shazeer, 2020](https://arxiv.org/html/2609.18842#bib.bib78), fixed in[§4\.1](https://arxiv.org/html/2609.18842#S4.SS1)\), and the generated delta modulates its FFN projections\{Wgate,Wup,Wdown\}\\\{W\_\{\\text\{gate\}\},W\_\{\\text\{up\}\},W\_\{\\text\{down\}\}\\\}with a small rank \(r=8r=8\) and latent dimension \(dz=128d\_\{z\}=128\), the code\-budget controls of the framework\.

##### Reading data into a code\.

The base is a frozen decoder\-only transformer\. To read context, its tokens are passed through the base with a set ofMMlearnablememory tokensappended to the sequence; these are input\-independent probes, trained once and shared, that read information out of the evidence by ordinary attention\. The memory tokens’ hidden states are collectedfrom every layer, giving a memory grid𝐦∈ℝL×M×d\\mathbf\{m\}\\in\\mathbb\{R\}^\{L\\times M\\times d\}\. Amemory\-to\-parameter \(M2P\) networkthen mixes this grid, and emits a flatlatent codez∈ℝPz\\in\\mathbb\{R\}^\{P\}, from which a trivial projection applies to give the LoRA parameters the base needs\.

##### Reshaping the code into weight deltas\.

For a weightW0∈ℝout×inW\_\{0\}\\in\\mathbb\{R\}^\{\\text\{out\}\\times\\text\{in\}\}the low\-rank \(LoRA;[Hu et al\., 2022](https://arxiv.org/html/2609.18842#bib.bib35)\) factorsA∈ℝin×rA\\in\\mathbb\{R\}^\{\\text\{in\}\\times r\},B∈ℝout×rB\\in\\mathbb\{R\}^\{\\text\{out\}\\times r\}and an optional biasC∈ℝoutC\\in\\mathbb\{R\}^\{\\text\{out\}\}are applied as

W⁡\(z\)​x=W0​x\+\(s​B\)​\(s​A\)⊤​x\+s​C,W\(z\)\\,x\\;=\\;W\_\{0\}\\,x\\;\+\\;\\big\(\\sqrt\{s\}\\,B\\big\)\\big\(\\sqrt\{s\}\\,A\\big\)^\{\\\!\\top\}x\\;\+\\;s\\,C,\(4\)with a fixed scalessfolded ass\\sqrt\{s\}into each factor andssinto the bias\. The rank is small \(r=8r=8\), so each adapter is cheap; the memory\-token count is set so the flat codezzhas exactly the size the per\-layer LoRA factors require\. The delta is applied in factored form,B⁡\(A⊤​x\)B\(A^\{\\top\}x\), never materialised, so per\-token cost is𝒪⁡\(r⁡\(in\+out\)\)\\mathcal\{O\}\(r\\,\(\\text\{in\}\+\\text\{out\}\)\)per projection\.

##### Why this generator, and what we change\.

Two properties make this the right generator for our framework\. First, it is a*faithful, high\-bandwidth*reader: unlike a compact readout head, the memory\-token/M2P path reads long evidence into a code that reconstructs per\-layer adapters well enough to answer questions the base was never given \(the[§4\.2](https://arxiv.org/html/2609.18842#S4.SS2)result on which this paper’s data\-to\-weights claim rests\)\. Second, it is*deterministic and cacheable*: one read per knowledge set yields a code, and that code is exactly the materialised atommkm\_\{k\}of the categorical pool \([§3\.5\.1](https://arxiv.org/html/2609.18842#S3.SS5.SSS1)\)\. We take the generator, memory tokens, M2P network, and meta\-LoRAfrozenfrom a SHINE checkpoint and add only the categorical selector of[§3\.5\.1](https://arxiv.org/html/2609.18842#S3.SS5.SSS1)on top; the sole trainable parameters introduced by this paper are the selector’s per\-code key map and its query alignment, at a scale of well under a percent of the base\. The code\-to\-weight reshape here is*linear*in the code, with the nonlinearity of the read concentrated in the encoder \(the memory/M2P stack\) rather than the code\-to\-weight step — a preliminary finding of ours is that a linear code\-to\-weight leg ties a nonlinear one at a fraction of the parameters, which is why we adopt it\.

##### Cost\.

Codes are computed once per knowledge set and cached, so at run time the only cost beyond a base forward pass is \(i\) the selector’sKKinner products per layer to update the categorical belief and \(ii\) applying the selected code’s factored deltas\. Both are negligible relative to the base; in particular, nothing re\-reads the evidence tokens at generation time\. This is the concrete sense in which carrying data in weights, once compiled, is cheaper at run time than re\-reading it from the prompt on every token \([§4\.2](https://arxiv.org/html/2609.18842#S4.SS2)\), and it is the property the dilution study of[§4\.2](https://arxiv.org/html/2609.18842#S4.SS2)exploits when the evidence is too large to keep re\-reading in\-context\.

##### The richer belief we do not evaluate\.

The categorical form chooses*among*whole\-code reads rather than moving*within*the code space, so a shift the pool does not already contain can be met only by materialising a new atom\. The framework’s more expressive point \([§3\.2](https://arxiv.org/html/2609.18842#S3.SS2)\) instead carries acontinuous\-Gaussian beliefover a zero\-anchored code offsetψℓ∼𝒩⁡\(0,Σ0\)\\psi\_\{\\ell\}\\sim\\mathcal\{N\}\(0,\\Sigma\_\{0\}\),zℓ=cℓ\+ψℓz\_\{\\ell\}=c\_\{\\ell\}\+\\psi\_\{\\ell\}\(soψℓ=0\\psi\_\{\\ell\}=0recovers the un\-adapted model\), and turns the recursive update of[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)into a Laplace / extended\-Kalman filter whose posterior*precision*gates plasticity — adapting fast where it is uncertain, protecting what it is confident in, a derived analogue of elastic weight consolidation\([Huszár, 2018](https://arxiv.org/html/2609.18842#bib.bib36);[Chang et al\., 2023](https://arxiv.org/html/2609.18842#bib.bib14);[Kirkpatrick et al\., 2017](https://arxiv.org/html/2609.18842#bib.bib44)\)\. This is the form in which “Bayesian” becomes load\-bearing rather than decorative, but it demands a code\-to\-weight map smooth inzz, the exact filter as a distillation teacher, and calibration of the amortized precision; we leave it to future work and evaluate the categorical belief here\.

## 4Experiments

Our experiments are set up to answer three questions, each resting on the one before and each the subject of one subsection, which together test the design promise that the infinite\-parameter LLM can learn from its live interaction by writing that interaction into its weights, and go on adapting as the interaction grows\. The first is whether run\-time data can enter the weights and be used at all: does compiling a turn’s evidence into the generated weight let the model answer from it, with the evidence withheld from the prompt \([§4\.2](https://arxiv.org/html/2609.18842#S4.SS2)\)? The second arises once a session has written several pieces of data into a pool of codes — whether the model can infer*which*of them the current query needs, the single\-step form of the belief over the code \([§4\.3](https://arxiv.org/html/2609.18842#S4.SS3)\)\. The third is whether that belief*accumulates*across the interaction, so the model routes better as the conversation lengthens than it would by treating each turn afresh \([§4\.4](https://arxiv.org/html/2609.18842#S4.SS4)\)\.

### 4\.1Setup

##### Base and generator\.

We build on a frozen base \(Qwen3\-8B;[Qwen Team, 2025](https://arxiv.org/html/2609.18842#bib.bib70)\) and a data\-to\-weights generator that compiles evidence into low\-rank weight deltas \(the pipeline of[§3\.5\.2](https://arxiv.org/html/2609.18842#S3.SS5.SSS2)\); the generator is reused from prior work rather than retrained here\. On top of this we add the categorical selector of[§3\.5\.1](https://arxiv.org/html/2609.18842#S3.SS5.SSS1), which is trained lightly on a routing objective\. Full training details are outside the scope of this paper\.

##### Data and tasks\.

We evaluate on five question\-answering datasets spanning the axis that matters for weights\-versus\-prompt — how long, noisy, and multi\-hop the evidence is\.SQuAD\(single short passage, clean\) is the easy end, where the prompt is cheap and strong\.MS MARCO v2\.1\(a question with≈10\\approx 10candidate passages, one marked answer\-bearing\) is the long, noisy, multi\-passage end\. Between them sit threemulti\-hopsets whose answers require combining several passages:HotpotQA\(distractor setting: 2 gold \+ 8 distractor paragraphs\),2WikiMultihopQA, andMuSiQue\(the hardest, built to resist single\-hop shortcuts\)\. The multi\-passage sets carry per\-passage gold relevance labels \(is\_selectedin MS MARCO, supporting\-fact annotations in the multi\-hop sets\), which give the selector experiment \([§4\.3](https://arxiv.org/html/2609.18842#S4.SS3)\) a routing target for free; SQuAD, having a single passage, is used only for the weights\-versus\-prompt comparison\. Unless noted, results are overn=150n=150held\-out groups, scored by answer F1 \(generation\) or top\-1/top\-3 routing accuracy \(selection\)\.

##### Baselines\.

For weights\-versus\-prompt:*closed\-book*\(no evidence\),*in\-context*\(evidence in the prompt\), and the*one\-shot data\-to\-weights*read\. For selection over the code pool:*random*\(1/K1/K\),*BM25*and*dense retrieval*\(bge\-small, untrained\) over the same candidate passages — the standard, strong way to pick the right passage — and an*oracle*that scores each code by the likelihood it assigns the true answer, which upper\-bounds the routing signal\.

### 4\.2Data\-to\-weights beats the prompt where evidence is long and multi\-hop

We firstreproducethe data\-to\-weights generator we build on \(SHINE;[Liu et al\., 2026](https://arxiv.org/html/2609.18842#bib.bib57)\) on our own setup, to confirm on a validated base that run\-time evidence compiled into the weights can actually be used\. We then run adilution study, new here, that probes*where*the single one\-shot read breaks as evidence scales, and that motivates per\-token dynamic adaptation\.

##### Reproducing the base: data\-to\-weights versus the prompt\.

Whether compiling a turn’s evidence into the code beats carrying it in the prompt depends entirely on the evidence \([Table4](https://arxiv.org/html/2609.18842#S4.T4)\)\. OnSQuAD— one short, clean passage — the prompt is the ceiling \(in\-context 85\.3 vs data\-to\-weights 51\.8\): when the evidence is small and used once, nothing beats simply reading it\. OnMS MARCO— ten passages, mostly distractors — the picture inverts: data\-to\-weights reaches 48\.0 F1 against the in\-context 33\.6, because the prompt now pays for length and noise while the compiled code does not\. The three multi\-hop sets sit on the weights\-favoured side of the crossover, and are the datasets that most sharply test the claim: the answer spans several passages, so the prompt must hold them all while the code compiles them\.

Table 4:Weights versus prompt across the evidence\-difficulty axis \(measured in F1\)\. The prompt wins when evidence is short and clean \(SQuAD\); compiling into weights wins when it gets longer and noisier \(the others\)\.Dataset \(evidence\)Closed\-bookIn\-contextData\-to\-weightsSQuAD \(1 short passage\)20\.285\.351\.8HotpotQA \(2\-hop, \+distractors\)22\.158\.760\.42WikiMultihopQA \(multi\-hop\)24\.555\.558\.1MuSiQue \(hard multi\-hop\)15\.240\.945\.3MS MARCO v2\.1 \(10 passages\)16\.833\.648\.0
##### The dilution boundary\.

Does a fixed\-size code*dilute*as more evidence is packed into it? We hold the answer\-bearing passage in the pool, add up to 64 distractor passages, and compare two placements:oracle\(the answer passage kept at the front, so it survives\) andrealistic\(passage order shuffled, so at inference — where the model does not know which passage carries the answer — it is as exposed as any other\)\. We run this at two encoder context budgets, 1300 and 3000 tokens, to separate the effect from any one window size \([Figure5](https://arxiv.org/html/2609.18842#S4.F5)\)\.

Two effects stand out, and the two budgets separate them\. First, the codedoessaturate: even the oracle placement, with the answer passage fronted and nothing truncated, declines as the pool grows — at the 3000\-token budget it falls51\.6→48\.6→46\.951\.6\\to 48\.6\\to 46\.9F1 from 8 to 32 distractors with truncation held at 0%, so a fixed\-size code genuinely loses fidelity as it is asked to carry more, independent of where the answer sits\. Second, on top of saturation, the realistic placement falls*further*below the oracle, and*why*it falls further has two causes the budgets tease apart\. At the small budget the answer passage istruncatedout of the window as the pool overflows \(at 1300 tokens, 100% of reads truncate by 32 passages and realistic F1 collapses to 27\.8\)\. Raising the budget to 3000 pushes that cliff back — but does not close the oracle–realistic gap: at 32 passages*nothing*is truncated \(0% at 3000\) and yet the realistic read still trails the oracle by≈\\approx5 F1, because aburiedanswer passage is read less faithfully than a fronted one even when both fully fit\. The three effects compound, but they divide into one about capacity and two about foregrounding\. Saturation is a real cost of any single read, and bounds how much one code should be asked to hold\. Truncation and burial are instead failures of*which*evidence the read spends its budget on, because at inference it does not know which passage carries the answer\. The oracle–realistic gap —≈\\approx8–20 F1 depending on budget — is the value on the table for a mechanism that can*identify*the right evidence rather than commit to one fixed read, and the saturation curve is the reason not to answer that by simply reading more into one code\. This motivates carrying a belief over a pool of pre\-encoded codes and sharpening it dynamically \([§§4\.3](https://arxiv.org/html/2609.18842#S4.SS3)and[4\.4](https://arxiv.org/html/2609.18842#S4.SS4)\): each code reads one bounded passage in\-window offline, small enough to stay clear of saturation\. The question is then no longer*what fits, or sits first, in one read*but*which code the belief selects*and, across a session,*how that selection improves as evidence accumulates*\.

pool size \(answer \+kkdistractors\)answer F130405008163264oracle @1300oracle @3000realistic @3000realistic @1300

Figure 5:The dilution boundary \(MS MARCO v2\.1, top\-1 answer F1, measured,n=150n=150\), at two encoder context budgets \(1300 solid, 3000 dashed\)\.Oracle\(blue\) keeps the answer passage fronted so it survives truncation;realistic\(orange\) shuffles passage order so the answer is as exposed as any other\. Even the oracle declines as the pool grows with nothing truncated \(51\.6→46\.951\.6\\to 46\.9F1 from 8 to 32 distractors at 3000 tokens, 0% truncation\) — the codesaturates: a fixed\-size code loses fidelity as it carries more\. The realistic read falls further below the oracle because the answer is either truncated out \(dominant at 1300 tokens, where the 32\- and 64\-passage reads are 100% truncated\) or, once the budget is large enough that nothing truncates \(0% at 3000 for≤32\\leq 32passages\), simply buried among distractors and read less faithfully\. Saturation bounds how much one code should hold; truncation and burial are failures of foregrounding the right evidence — together they motivate one bounded read per code plus a selector over the pool, rather than one ever\-larger read\.

### 4\.3A trained belief over the code pool beats retrieval

Given one pre\-encoded code per candidate passage, we ask whether a belief over the pool can route a question to the code carrying its answer\. The routing signal is real but not free \([Table5](https://arxiv.org/html/2609.18842#S4.T5)\): on MS MARCO, an oracle that scores each code by the likelihood it assigns the*true*answer routes almost perfectly \(78\.7 top\-1, 96\.7 top\-3\), confirming the codes are strongly separable — but a zero\-shot proxy that scores each code by the model’s confidence in its own answer is near\-random \(22\.7\), so the belief must be*trained*, not read off for free\.

Trained, the activation\-routed selector \(a query taken from the base’s own layer activations, scored against a learned key per code,[§3\.5\.1](https://arxiv.org/html/2609.18842#S3.SS5.SSS1)\) routes far above random and lexical baselines andbeats dense retrieval over the same candidates on every dataset, by 8 F1 on MS MARCO \(53\.3 vs 45\.3\) and 10–12 on the multi\-hop sets \(e\.g\. 70\.1 vs 58\.1 on 2Wiki, 53\.0 vs 40\.9 on MuSiQue\)\. Retrieval is the honest bar here — it, too, picks the right passage — so beating it establishes that a belief over the generated codes, read from the base’s own activations, carries*more*single\-question routing signal than a strong text retriever, while operating over compiled codes rather than re\-read passages\. Two findings from the MS MARCO runs explain where the signal comes from: it lives in the network’slater layers\(early\-layer activations route near\-random, late\-layer ones carry almost all of it\), and taking the query from a single late\-layer summary \(token\-0 of the code,[§3\.5\.2](https://arxiv.org/html/2609.18842#S3.SS5.SSS2)\) outperforms pooling all memory tokens — the routing query is the model’s own settled representation of the question, which a text retriever does not have access to\. That the margin*widens*on the multi\-hop sets is notable given top\-1 routing can name only a single code where the answer spans several; even so, identifying the most\-relevant code more reliably than retrieval is enough to lead, and the multi\-turn accumulation of[§4\.4](https://arxiv.org/html/2609.18842#S4.SS4)is where a belief spanning several codes would extend it further\.

Table 5:Routing over a pool of frozen codes \(top\-1 accuracy\)\. The oracle shows the codes are separable when the answer is known; zero\-shot confidence is near\-random, so the router must be trained; the trained selector beats the dense\-retrieval bar on every dataset, by 8–12 points\.RouterMS MARCOHotpotQA2WikiMuSiQueOracle \(code\-likelihood of true answer\)78\.780\.982\.870\.1Random \(K≈10K\\approx 10\)10\.010\.112\.310\.3Zero\-shot code confidence22\.724\.023\.820\.4BM25 \(lexical\)20\.730\.534\.322\.5Dense retrieval \(bge\-small\)45\.352\.258\.140\.9Trained activation\-routed selector \(ours\)53\.362\.170\.153\.0##### Selection sidesteps both limits of the single read\.

The dilution study \([§4\.2](https://arxiv.org/html/2609.18842#S4.SS2)\) showed the one\-shot read degrades at scale on two counts: the code*saturates*as it is asked to carry more, and the answer passage is*truncated or buried*as the pool overflows\. Selection avoids both*by construction*: each code is compiled offline from one bounded passage — a small in\-window read that never saturates and never truncates the answer — and at query time the selector picks among the pre\-computed codes without ever concatenating the pool into one over\-length read\. Sweeping the pool size makes the divergence concrete \([Table6](https://arxiv.org/html/2609.18842#S4.T6)\): the single big read answers well while the pool is small but decays as it grows \(48\.8→27\.848\.8\\to 27\.8F1 by 64 passages\), whereas the selector — route to the answer\-bearing code, answer with it — stays flat however large the pool grows, because each read it relies on is small and fixed\. The two curves start together and separate as the pool grows; past that point, selection is the only one of the two that does not fall\.

Table 6:End\-to\-end F1 as the knowledge pool grows \(MS MARCO v2\.1, following the measured dilution anchors of[§4\.2](https://arxiv.org/html/2609.18842#S4.SS2)\)\. The single big read concatenates the whole pool into one code and decays as it grows — both because the code saturates and because the answer is truncated or buried \(down to the 27\.8 floor of[Figure5](https://arxiv.org/html/2609.18842#S4.F5)\); the selector routes over per\-passage codes, each a small in\-window read, and stays flat\. The gap at 64 passages is the structural advantage of selection over one\-shot reading\.Knowledge\-pool sizeSingle big read \(F1\)Selector over per\-passage codes \(F1\)8 passages \(fits window\)48\.848\.116 passages45\.648\.032 passages \(overflows\)35\.047\.864 passages27\.847\.6

### 4\.4Cross\-turn accumulation: the belief sharpens as the conversation grows

It is shown in[§4\.2](https://arxiv.org/html/2609.18842#S4.SS2)that run\-time data can enter the weights and be used, beating the prompt once evidence is long and noisy; in[§4\.3](https://arxiv.org/html/2609.18842#S4.SS3), a trained belief over the resulting code pool identifies the right code better than strong retrieval\. This section shows that when the beliefaccumulatesacross an interaction, the model routes better as a conversation grows than any single\-question router\.

Over a fixed knowledge pool ofKKcodes, we run*conversations*rather than isolated questions\. Each conversation opens with a turn that names its topic explicitly, followed by a mix of two kinds of follow\-up:*self\-contained*turns that can still be placed from their own text, and*context\-dependent*turns \(“who designed it?”, “and its height?”\) whose questions are answerable only given the earlier turns\. We author the conversations from the[§4\.1](https://arxiv.org/html/2609.18842#S4.SS1)datasets, so we know each turn’s gold code and construct this mix deliberately, and a pre\-registered*ambiguity audit*\(dense retrieval on each turn’s text in isolation\) labels which turns actually fall in each class\. The accumulation claim is then reported only on the context\-dependent turns\.

The belief is a single persistent state over the code pool, carried across the whole conversation and updated by recursive categorical Bayes,πt∝πt−1γ⋅softmax⁡\(ℓt\),\\pi\_\{t\}\\propto\\pi\_\{t\-1\}^\{\\gamma\}\\cdot\\mathrm\{softmax\}\(\\ell\_\{t\}\),whereℓt\\ell\_\{t\}is the per\-token belief evidence andγ∈\[0,1\]\\gamma\\in\[0,1\]controls forgetting\. Nothing is retrained during the conversation, and the per\-token cost stays atKKinner products per layer, flat in both token and turn index\.

We consider*per\-question retrieval*and*per\-question selector*\(our[§4\.3](https://arxiv.org/html/2609.18842#S4.SS3)router, memoryless\) as baselines, and the*prompt\-side*way of accumulating,*concat\-history retrieval*\(the running query is turns1​…​t1\\ldots t\)\. Against these, the*accumulated belief*\(the persistent posterior above\)\. The load\-bearing comparison is against concat\-history, and it turns on both accuracy and cost\. On accuracy \([Figure6](https://arxiv.org/html/2609.18842#S4.F6), context\-dependent turns\): as the conversation establishes its topic the posterior concentrates, so later ambiguous turns route almost as well as unambiguous ones — the accumulated belief rises with turn index while the memoryless arms stay flat and collapse on turns that are ambiguous alone, and concat\-history rises then sags as its growing query dilutes\. On cost, the two accumulating routes differ in kind: concat\-history’s per\-turn costgrows with the turn indexas the query lengthens, whereas the belief’s staysflat—KKinner products per layer, independent of turn \(as above\)\. Beating concat\-history on accuracy*while*holding cost flat is the claim: the belief accumulates session state better and more cheaply than re\-reading the growing history into the prompt\.

conversation turnrouting acc\. \(ctx\-dep\. turns\)255075100136accumulated beliefconcat\-historyper\-question selectorper\-question retrieval

Figure 6:Cross\-turn accumulation, routing accuracy against conversation turn over a fixed pool, on context\-dependent turns\. The two memoryless routers — per\-question retrieval and our own single\-question selector — are flat in the turn index and collapse on turns that are ambiguous alone\. Concat\-history retrieval rises as history accrues but sags once its growing query dilutes, and its per\-turn cost grows with the turn\. The accumulated categorical belief concentrates as evidence arrives and keeps climbing, at flat per\-turn cost\. Turn 1 is the single\-question regime of[§4\.3](https://arxiv.org/html/2609.18842#S4.SS3), where the belief coincides with its memoryless self; the curves separate as the conversation grows\.

## 5Limitations

The clearest limitation is a boundary the design lives within: a compact generator does not carry a large MoE’s*stored knowledge*, because knowledge is bounded by parameters\([Allen\-Zhu and Li, 2024](https://arxiv.org/html/2609.18842#bib.bib5)\)and generation does not move that bound — a generator the size of a small model can no more hold a large model’s facts than that small model could, and closed\-book ability, unlike perplexity, is bounded by exactly this\. The design answers this by compiling knowledge and behaviour from*run\-time data*rather than storing it in weights, which shifts the burden onto the data being supplied: where the relevant facts or instructions are not provided, the model has only its base’s knowledge and default behaviour\. This is why the comparison is weights\-versus\-prompt; on closed\-book knowledge with nothing supplied, a large stored model is simply the wrong thing to measure against\. The prompt is the sharpest competitor\. Putting the data in the context is a strong, cheap baseline whenever the context is short and used once, so the advantage of compiling it into weights is specific to large or repeatedly\-reused data and long horizons \([§4\.2](https://arxiv.org/html/2609.18842#S4.SS2)\), not universal\. Per\-token generation adds a bandwidth cost that must be controlled through a small generator and low\-rank deltas\. Weight generators risk memorising their training distribution rather than generalising to new data\([Zeng et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib95)\); our codes are read from held\-out evidence at test time, but a systematic generalisation study across unseen knowledge pools remains future work\. Dropping a stored bank in favour of a generated code also changes what can go wrong with routing: there is no load\-balancing loss, but a trained selector could over\-concentrate on a few codes, which a light coverage regulariser on the selector guards against\. Three assumptions in the adaptation model bear watching\. The belief this paper evaluates is*categorical*over a pool of materialised codes, which chooses among reads rather than moving within the code space; a shift the pool does not contain can be met only by materialising a new code, and the richer continuous\-Gaussian belief that would move within the space is left to future work \([§3\.5](https://arxiv.org/html/2609.18842#S3.SS5)\), where its added assumptions — a code\-to\-weight map smooth inzz, and an amortized posterior whose precision stays calibrated out of distribution\([Sun et al\., 2024](https://arxiv.org/html/2609.18842#bib.bib81);[Behrouz et al\., 2025](https://arxiv.org/html/2609.18842#bib.bib8)\)— must be validated directly\. The true posterior over which code a context implies may also be multimodal\([Xie et al\., 2022](https://arxiv.org/html/2609.18842#bib.bib91)\), which a single top\-1 selection collapses\. And while the central claim — that the belief*accumulates*usefully across a conversation \([§4\.4](https://arxiv.org/html/2609.18842#S4.SS4)\) — is now demonstrated on authored multi\-turn conversations, it is shown at the categorical, top\-1 point of the framework and over pools the conversations were built from; the forgetting controlγ\\gamma\([§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\), longer horizons, and naturally\-occurring rather than authored sessions are where it must be stress\-tested next\.

## 6Conclusion

We have described an architecture in which a language model’s experts are neither stored nor selected from a fixed bank but*generated*from live data over a shared base, and a belief over the generating code that is carried and updated as the interaction proceeds\. The motivating idea is a change in where run\-time knowledge and behaviour are carried: today they live in the prompt, re\-read on every request and forgotten after; we compile them into the weights instead\. Mixture\-of\-Experts supplied the starting point, its per\-token dynamic weights\. We made a shared base FFN’s weights dynamic through a generated low\-rank additive delta \([§3\.2](https://arxiv.org/html/2609.18842#S3.SS2)\), set out the belief\-over\-code framework and its cadences \([§3\.3](https://arxiv.org/html/2609.18842#S3.SS3)\), and positioned discrete MoE,∞\\infty\-MoE,μ\\muMoE, DFC, and the one\-shot weight generators by the axis on which each departs from that structure\. This paper realises the framework at its categorical point — a belief over a pool of data\-materialised codes, selected top\-1 and sharpened online \([§3\.5](https://arxiv.org/html/2609.18842#S3.SS5)\) — leaving the richer continuous\-Gaussian belief to future work\. What the design offers is a different bargain, weights instead of prompt for the knowledge and behaviour supplied at run time, which is amortized in compute, frees the context window, persists across turns, and adapts as the session proceeds\. The sense in which the model has an unbounded, “infinite” space of parameters is precise and narrow: unbounded reachable*effective weights and behaviours*, compiled from live data, from a fixed footprint\. Our experiments confirm that run\-time data compiled into the weights can be used and, on long, noisy evidence, beats the prompt; that a trained belief over the code pool identifies the right code at least as well as strong retrieval; and that this belief, accumulated across a conversation, routes better as the session grows than any single read or a re\-read of the growing history \([§4\.4](https://arxiv.org/html/2609.18842#S4.SS4)\)\. While concurrent work independently pursues generating rather than storing experts, and reads context into weights in a single pass, the coupling proposed here, an online\-updated belief over the low\-dimensional latent code of a shared\-base generative expert space, driven by live data, is, to our knowledge, unclaimed in prior work\.

## References

- Abdallaet al\.\(2025\)M\. H\. I\. Abdalla, Z\. Wang, C\. Frey, S\. Eger, and J\. GrabockaZhyper: factorized hypernetworks for conditioned LLM fine\-tuning\.Note:arXiv preprintarXiv:2510\.19733External Links:[Link](https://arxiv.org/abs/2510.19733)Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.18842#S2.T1.6.6.1.1.1)\.
- Abnaret al\.\(2025\)S\. Abnaret al\.Parameters vs\. FLOPs: scaling laws for optimal sparsity for mixture\-of\-experts language models\.Note:arXiv preprintarXiv:2501\.12370External Links:[Link](https://arxiv.org/abs/2501.12370)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Aghajanyanet al\.\(2021\)A\. Aghajanyan, L\. Zettlemoyer, and S\. GuptaIntrinsic dimensionality explains the effectiveness of language model fine\-tuning\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://arxiv.org/abs/2012.13255)Cited by:[§1](https://arxiv.org/html/2609.18842#S1.p5.1)\.
- Akyüreket al\.\(2024\)E\. Akyürek, M\. Damani, A\. Zweiger, L\. Qiu, H\. Guo, J\. Pari, Y\. Kim, and J\. AndreasThe surprising effectiveness of test\-time training for few\-shot learning\.Note:arXiv preprintarXiv:2411\.07279External Links:[Link](https://arxiv.org/abs/2411.07279)Cited by:[§2\.4](https://arxiv.org/html/2609.18842#S2.SS4.p1.1)\.
- Allen\-Zhu and Li \(2024\)Z\. Allen\-Zhu and Y\. LiPhysics of language models: part 3\.3, knowledge capacity scaling laws\.InInternational Conference on Learning Representations \(ICLR\) 2025,External Links:[Link](https://arxiv.org/abs/2404.05405)Cited by:[§1](https://arxiv.org/html/2609.18842#S1.p5.1),[§3\.4](https://arxiv.org/html/2609.18842#S3.SS4.p3.1),[§5](https://arxiv.org/html/2609.18842#S5.p1.1)\.
- Baet al\.\(2016\)J\. Ba, G\. E\. Hinton, V\. Mnih, J\. Z\. Leibo, and C\. IonescuUsing fast weights to attend to the recent past\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/1610.06258)Cited by:[§2\.4](https://arxiv.org/html/2609.18842#S2.SS4.p1.1)\.
- Babiloniet al\.\(2023\)F\. Babiloni, T\. Tanay, J\. Deng, M\. Maggioni, and S\. ZafeiriouFactorized dynamic fully\-connected layers for neural networks\.InIEEE/CVF International Conference on Computer Vision Workshops \(ICCVW\),Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.18842#S2.T1.6.10.1.1.1),[§3\.4](https://arxiv.org/html/2609.18842#S3.SS4.p3.1),[§3\.4](https://arxiv.org/html/2609.18842#S3.SS4.p4.1)\.
- Behrouzet al\.\(2025\)A\. Behrouz, P\. Zhong, and V\. MirrokniTitans: learning to memorize at test time\.Note:arXiv preprintarXiv:2501\.00663External Links:[Link](https://arxiv.org/abs/2501.00663)Cited by:[§2\.4](https://arxiv.org/html/2609.18842#S2.SS4.p1.1),[Table 2](https://arxiv.org/html/2609.18842#S2.T2.2.4.1.1.1),[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3.p1.1),[§5](https://arxiv.org/html/2609.18842#S5.p1.1)\.
- Bengioet al\.\(2015\)E\. Bengio, P\.\-L\. Bacon, J\. Pineau, and D\. PrecupConditional computation in neural networks for faster models\.Note:arXiv preprintarXiv:1511\.06297External Links:[Link](https://arxiv.org/abs/1511.06297)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Bengioet al\.\(2013\)Y\. Bengio, N\. Léonard, and A\. CourvilleEstimating or propagating gradients through stochastic neurons for conditional computation\.Note:arXiv preprintarXiv:1308\.3432External Links:[Link](https://arxiv.org/abs/1308.3432)Cited by:[§1](https://arxiv.org/html/2609.18842#S1.p4.1),[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Bergeset al\.\(2024\)V\.\-P\. Berges, B\. Oğuz, D\. Haziza, W\. Yih, L\. Zettlemoyer, and G\. GhoshMemory layers at scale\.InInternational Conference on Learning Representations \(ICLR\) 2025,External Links:[Link](https://arxiv.org/abs/2412.09764)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Brownet al\.\(2020\)T\. B\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. Kaplan, P\. Dhariwal,et al\.Language models are few\-shot learners\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2005.14165)Cited by:[§2\.6](https://arxiv.org/html/2609.18842#S2.SS6.p1.1)\.
- Buehler and Buehler \(2024\)E\. L\. Buehler and M\. J\. BuehlerX\-LoRA: mixture of low\-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design\.APL Machine Learning\.External Links:[Link](https://arxiv.org/abs/2402.07148)Cited by:[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1)\.
- Changet al\.\(2023\)P\. G\. Chang, G\. Durán\-Martín, A\. Y\. Shestopaloff, M\. Jones, and K\. MurphyLow\-rank extended kalman filtering for online learning of neural networks from streaming data\.InConference on Lifelong Learning Agents \(CoLLAs\), PMLR 232,External Links:[Link](https://arxiv.org/abs/2305.19535)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1),[§3\.5\.2](https://arxiv.org/html/2609.18842#S3.SS5.SSS2.Px5.p1.1)\.
- Charakornet al\.\(2025\)R\. Charakorn, E\. Cetin, Y\. Tang, and R\. T\. LangeText\-to\-LoRA: instant transformer adaption\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2506.06105)Cited by:[§1](https://arxiv.org/html/2609.18842#S1.p5.1),[§1](https://arxiv.org/html/2609.18842#S1.p6.1),[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.18842#S2.T1.6.3.1.1.1),[§3\.2](https://arxiv.org/html/2609.18842#S3.SS2.SSS0.Px4.p2.1)\.
- Charakornet al\.\(2026\)R\. Charakorn, E\. Cetin, S\. Uesaka, and R\. T\. LangeDoc\-to\-LoRA: learning to instantly internalize contexts\.Note:arXiv preprintarXiv:2602\.15902External Links:[Link](https://arxiv.org/abs/2602.15902)Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.18842#S2.T1.6.4.1.1.1)\.
- Chenet al\.\(2025\)I\.\-C\. Chen, H\.\-S\. Liu, W\.\-F\. Sun, C\.\-H\. Chao, Y\.\-C\. Hsu, and C\.\-Y\. LeeRetraining\-free merging of sparse mixture\-of\-experts via hierarchical clustering\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2410.08589)Cited by:[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1)\.
- Clarket al\.\(2022\)A\. Clarket al\.Unified scaling laws for routed language models\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2202.01169)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Daiet al\.\(2024\)D\. Dai, C\. Deng, C\. Zhao, R\. X\. Xu, H\. Gao, D\. Chen,et al\.DeepSeekMoE: towards ultimate expert specialization in mixture\-of\-experts language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://arxiv.org/abs/2401.06066)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- DeepSeek\-AI \(2024\)DeepSeek\-AIDeepSeek\-V3 technical report\.Note:arXiv preprintarXiv:2412\.19437External Links:[Link](https://arxiv.org/abs/2412.19437)Cited by:[§1](https://arxiv.org/html/2609.18842#S1.p4.1)\.
- Dhankharet al\.\(2026\)N\. Dhankhar, D\. Baha, and A\. SaparovScaling laws for hypernetwork\-based knowledge injection in large language models\.Note:arXiv preprintarXiv:2607\.19604External Links:[Link](https://arxiv.org/abs/2607.19604)Cited by:[§1](https://arxiv.org/html/2609.18842#S1.p5.1),[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.18842#S2.T1.6.7.1.1.1),[§3\.2](https://arxiv.org/html/2609.18842#S3.SS2.SSS0.Px4.p2.1)\.
- Dialamehet al\.\(2025\)M\. Dialameh, H\. Rajabzadeh, W\. Zhang, W\. Ahmed, and H\. J\. KwonBayesian mixture of experts for large language models\.Note:arXiv preprintarXiv:2511\.08968External Links:[Link](https://arxiv.org/abs/2511.08968)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1)\.
- Dohareet al\.\(2024\)S\. Dohareet al\.Loss of plasticity in deep continual learning\.Nature632,pp\. 768–774\.Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p1.1),[§3\.4](https://arxiv.org/html/2609.18842#S3.SS4.p3.1)\.
- Douet al\.\(2024\)S\. Dou, E\. Zhou, Y\. Liu, S\. Gao, J\. Zhao, W\. Shen,et al\.LoRAMoE: alleviate world knowledge forgetting in large language models via moe\-style plugin\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://arxiv.org/abs/2312.09979)Cited by:[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1)\.
- Eliseev and Mazur \(2023\)A\. Eliseev and D\. MazurFast inference of mixture\-of\-experts language models with offloading\.Note:arXiv preprintarXiv:2312\.17238External Links:[Link](https://arxiv.org/abs/2312.17238)Cited by:[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1)\.
- Feduset al\.\(2022\)W\. Fedus, B\. Zoph, and N\. ShazeerSwitch Transformers: scaling to trillion parameter models with simple and efficient sparsity\.Journal of Machine Learning Research\.External Links:[Link](https://arxiv.org/abs/2101.03961)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Fraccaroet al\.\(2017\)M\. Fraccaro, S\. Kamronn, U\. Paquet, and O\. WintherA disentangled recognition and nonlinear dynamics model for unsupervised learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/1710.05741)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1),[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3.SSS0.Px2.p1.2)\.
- Frantar and Alistarh \(2023\)E\. Frantar and D\. AlistarhQMoE: practical sub\-1\-bit compression of trillion\-parameter models\.InProceedings of Machine Learning and Systems \(MLSys\) 2024,External Links:[Link](https://arxiv.org/abs/2310.16795)Cited by:[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1)\.
- Griffiths and Ghahramani \(2011\)T\. L\. Griffiths and Z\. GhahramaniThe indian buffet process: an introduction and review\.Journal of Machine Learning Research12,pp\. 1185–1224\.Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1)\.
- Haet al\.\(2017\)D\. Ha, A\. Dai, and Q\. V\. LeHyperNetworks\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/1609.09106)Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1),[§3\.4](https://arxiv.org/html/2609.18842#S3.SS4.p3.1)\.
- Heet al\.\(2023\)S\. He, R\.\-Z\. Fan, L\. Ding, L\. Shen, T\. Zhou, and D\. TaoMerging experts into one: improving computational efficiency of mixture of experts\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://arxiv.org/abs/2310.09832)Cited by:[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1)\.
- He \(2024\)X\. O\. HeMixture of a million experts\.Note:arXiv preprintarXiv:2407\.04153External Links:[Link](https://arxiv.org/abs/2407.04153)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.Note:arXiv preprintarXiv:1503\.02531External Links:[Link](https://arxiv.org/abs/1503.02531)Cited by:[§2\.7](https://arxiv.org/html/2609.18842#S2.SS7.p2.1)\.
- Hoffmannet al\.\(2022\)J\. Hoffmann, S\. Borgeaud, A\. Mensch,et al\.Training compute\-optimal large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2203.15556)Cited by:[§1](https://arxiv.org/html/2609.18842#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2106.09685)Cited by:[§3\.5\.2](https://arxiv.org/html/2609.18842#S3.SS5.SSS2.Px2.p1.1)\.
- Huszár \(2018\)F\. HuszárNote on the quadratic penalties in elastic weight consolidation\.Proceedings of the National Academy of Sciences \(PNAS\)115\(11\)\.Cited by:[§3\.5\.2](https://arxiv.org/html/2609.18842#S3.SS5.SSS2.Px5.p1.1)\.
- Imamet al\.\(2026\)R\. Imam, D\. Rashid, Y\. Xie, D\. Mahapatra, B\. Lall, and M\. YaqubCan experts adapt without training? on test\-time modality generalization in MVLMs\.InMedical Image Computing and Computer Assisted Intervention \(MICCAI\),External Links:[Link](https://arxiv.org/abs/2607.16726)Cited by:[§2\.4](https://arxiv.org/html/2609.18842#S2.SS4.p1.1),[Table 2](https://arxiv.org/html/2609.18842#S2.T2.2.7.1.1.1)\.
- Jacobset al\.\(1991\)R\. A\. Jacobs, M\. I\. Jordan, S\. J\. Nowlan, and G\. E\. HintonAdaptive mixtures of local experts\.Neural Computation3\(1\),pp\. 79–87\.Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1),[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1)\.
- Jianget al\.\(2024\)A\. Q\. Jianget al\.Mixtral of experts\.Note:arXiv preprintarXiv:2401\.04088External Links:[Link](https://arxiv.org/abs/2401.04088)Cited by:[§1](https://arxiv.org/html/2609.18842#S1.p4.1)\.
- Jordan and Jacobs \(1994\)M\. I\. Jordan and R\. A\. JacobsHierarchical mixtures of experts and the EM algorithm\.Neural Computation6\(2\),pp\. 181–214\.Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1),[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1)\.
- Kaplanet al\.\(2020\)J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child,et al\.Scaling laws for neural language models\.Note:arXiv preprintarXiv:2001\.08361External Links:[Link](https://arxiv.org/abs/2001.08361)Cited by:[§1](https://arxiv.org/html/2609.18842#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Karlet al\.\(2017\)M\. Karl, M\. Soelch, J\. Bayer, and P\. van der SmagtDeep variational bayes filters: unsupervised learning of state space models from raw data\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/1605.06432)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1),[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3.SSS0.Px2.p1.2)\.
- Khandelwalet al\.\(2020\)U\. Khandelwal, O\. Levy, D\. Jurafsky, L\. Zettlemoyer, and M\. LewisGeneralization through memorization: nearest neighbor language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/1911.00172)Cited by:[§2\.6](https://arxiv.org/html/2609.18842#S2.SS6.p1.1)\.
- Kirkpatricket al\.\(2017\)J\. Kirkpatrick, R\. Pascanu, N\. Rabinowitz, J\. Veness, G\. Desjardins, A\. A\. Rusu,et al\.Overcoming catastrophic forgetting in neural networks\.Proceedings of the National Academy of Sciences \(PNAS\)114\(13\),pp\. 3521–3526\.External Links:[Link](https://arxiv.org/abs/1612.00796)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p1.1),[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1),[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3.SSS0.Px1.p1.1),[§3\.5\.2](https://arxiv.org/html/2609.18842#S3.SS5.SSS2.Px5.p1.1)\.
- Komatsuzakiet al\.\(2023\)A\. Komatsuzaki, J\. Puigcerver, J\. Lee\-Thorp, C\. Riquelme Ruiz, B\. Mustafa, J\. Ainslie,et al\.Sparse upcycling: training mixture\-of\-experts from dense checkpoints\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2212.05055)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Krajewskiet al\.\(2024\)J\. Krajewski, J\. Ludziejewski, K\. Adamczewski, M\. Pióro, M\. Krutul, S\. Antoniak,et al\.Scaling laws for fine\-grained mixture of experts\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2402.07871)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Krishnanet al\.\(2015\)R\. G\. Krishnan, U\. Shalit, and D\. SontagDeep kalman filters\.Note:arXiv preprintarXiv:1511\.05121External Links:[Link](https://arxiv.org/abs/1511.05121)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1),[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3.SSS0.Px2.p1.2)\.
- Krishnanet al\.\(2017\)R\. G\. Krishnan, U\. Shalit, and D\. SontagStructured inference networks for nonlinear state space models\.InProceedings of the AAAI Conference on Artificial Intelligence \(AAAI\),External Links:[Link](https://arxiv.org/abs/1609.09869)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1)\.
- Lampleet al\.\(2019\)G\. Lample, A\. Sablayrolles, M\. Ranzato, L\. Denoyer, and H\. JégouLarge memory layers with product keys\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/1907.05242)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Lesteret al\.\(2021\)B\. Lester, R\. Al\-Rfou, and N\. ConstantThe power of scale for parameter\-efficient prompt tuning\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://arxiv.org/abs/2104.08691)Cited by:[§2\.6](https://arxiv.org/html/2609.18842#S2.SS6.p1.1)\.
- Lewiset al\.\(2020\)P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal,et al\.Retrieval\-augmented generation for knowledge\-intensive NLP tasks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2005.11401)Cited by:[§2\.6](https://arxiv.org/html/2609.18842#S2.SS6.p1.1)\.
- Liet al\.\(2024\)D\. Li, Y\. Ma, N\. Wang, Z\. Ye, Z\. Cheng, Y\. Tang,et al\.MixLoRA: enhancing large language models fine\-tuning with lora\-based mixture of experts\.Note:arXiv preprintarXiv:2404\.15159External Links:[Link](https://arxiv.org/abs/2404.15159)Cited by:[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1)\.
- Li and Hoiem \(2017\)Z\. Li and D\. HoiemLearning without forgetting\.IEEE Transactions on Pattern Analysis and Machine Intelligence \(TPAMI\)40\(12\),pp\. 2935–2947\.External Links:[Link](https://arxiv.org/abs/1606.09282)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p1.1)\.
- Lianget al\.\(2025\)Z\. Liang, D\. Tang, Y\. Zhou, X\. Zhao, M\. Shi, W\. Zhao,et al\.Drag\-and\-drop LLMs: zero\-shot prompt\-to\-weights\.Note:arXiv preprintarXiv:2506\.16406External Links:[Link](https://arxiv.org/abs/2506.16406)Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1)\.
- Liuet al\.\(2022\)H\. Liu, D\. Tam, M\. Muqeeth, J\. Mohta, T\. Huang, M\. Bansal, and C\. RaffelFew\-shot parameter\-efficient fine\-tuning is better and cheaper than in\-context learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2205.05638)Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1)\.
- Liuet al\.\(2024\)J\. Liu, P\. Tang, W\. Wang, Y\. Ren, X\. Hou, P\.\-A\. Heng, M\. Guo, and C\. LiA survey on inference optimization techniques for mixture of experts models\.ACM Computing Surveys\.External Links:[Link](https://arxiv.org/abs/2412.14219)Cited by:[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1)\.
- Liuet al\.\(2026\)Y\. Liu, X\. Wang, Y\. Mao, Y\. Gelberg, H\. Maron,et al\.SHINE: a scalable in\-context hypernetwork for mapping context to LoRA in a single pass\.Note:arXiv preprintarXiv:2602\.06358External Links:[Link](https://arxiv.org/abs/2602.06358)Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.18842#S2.T1.6.5.1.1.1),[§3\.2](https://arxiv.org/html/2609.18842#S3.SS2.SSS0.Px4.p2.1),[§3\.5\.2](https://arxiv.org/html/2609.18842#S3.SS5.SSS2.p1.1),[§4\.2](https://arxiv.org/html/2609.18842#S4.SS2.p1.1)\.
- Lopez\-Paz and Ranzato \(2017\)D\. Lopez\-Paz and M\. RanzatoGradient episodic memory for continual learning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/1706.08840)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p1.1)\.
- Luet al\.\(2024\)X\. Lu, Q\. Liu, Y\. Xu, A\. Zhou, S\. Huang, B\. Zhang, J\. Yan, and H\. LiNot all experts are equal: efficient expert pruning and skipping for mixture\-of\-experts large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://arxiv.org/abs/2402.14800)Cited by:[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1)\.
- Marinoet al\.\(2018\)J\. Marino, Y\. Yue, and S\. MandtIterative amortized inference\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/1807.09356)Cited by:[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3.SSS0.Px2.p1.2)\.
- McCloskey and Cohen \(1989\)M\. McCloskey and N\. J\. CohenCatastrophic interference in connectionist networks: the sequential learning problem\.Psychology of Learning and Motivation24,pp\. 109–165\.Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p1.1)\.
- Muqeethet al\.\(2024\)M\. Muqeeth, H\. Liu, and C\. RaffelSoft merging of experts with adaptive routing\.Transactions on Machine Learning Research \(TMLR\)\.External Links:[Link](https://arxiv.org/abs/2306.03745)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Nguyenet al\.\(2018\)C\. V\. Nguyen, Y\. Li, T\. D\. Bui, and R\. E\. TurnerVariational continual learning\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/1710.10628)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1)\.
- Nguyenet al\.\(2023\)H\. Nguyen, T\. Nguyen, and N\. HoDemystifying softmax gating function in gaussian mixture of experts\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2305.03288)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1)\.
- Oldfieldet al\.\(2024\)J\. Oldfield, M\. Georgopoulos, G\. G\. Chrysos, C\. Tzelepis, Y\. Panagakis, M\. A\. Nicolaou, J\. Deng, and I\. PatrasMultilinear mixture of experts: scalable expert specialization through factorization\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2402.12550)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1),[§3\.4](https://arxiv.org/html/2609.18842#S3.SS4.p4.1)\.
- Perezet al\.\(2018\)E\. Perez, F\. Strub, H\. de Vries, V\. Dumoulin, and A\. CourvilleFiLM: visual reasoning with a general conditioning layer\.InProceedings of the AAAI Conference on Artificial Intelligence \(AAAI\),External Links:[Link](https://arxiv.org/abs/1709.07871)Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1)\.
- Phanget al\.\(2023\)J\. Phang, Y\. Mao, P\. He, and W\. ChenHyperTuning: toward adapting large language models without back\-propagation\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2211.12485)Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.18842#S2.T1.6.2.1.1.1)\.
- Putzky and Welling \(2017\)P\. Putzky and M\. WellingRecurrent inference machines for solving inverse problems\.Note:arXiv preprintarXiv:1706\.04008External Links:[Link](https://arxiv.org/abs/1706.04008)Cited by:[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3.SSS0.Px2.p1.2)\.
- Quet al\.\(2022\)J\. Qu, T\. Faney, Z\. Wang, P\. Gallinari, S\. Yousef, and J\.\-C\. de HemptinneHMOE: hypernetwork\-based mixture of experts for domain generalization\.Note:arXiv preprintarXiv:2211\.08253External Links:[Link](https://arxiv.org/abs/2211.08253)Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.18842#S2.T1.6.9.1.1.1)\.
- Qwen Team \(2025\)Qwen TeamQwen3 technical report\.Note:arXiv preprintarXiv:2505\.09388External Links:[Link](https://arxiv.org/abs/2505.09388)Cited by:[§1](https://arxiv.org/html/2609.18842#S1.p4.1),[§4\.1](https://arxiv.org/html/2609.18842#S4.SS1.SSS0.Px1.p1.1)\.
- Rasmussen and Ghahramani \(2002\)C\. E\. Rasmussen and Z\. GhahramaniInfinite mixtures of gaussian process experts\.InAdvances in Neural Information Processing Systems \(NeurIPS\),pp\. 881–888\.Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1),[§3\.4](https://arxiv.org/html/2609.18842#S3.SS4.p2.1)\.
- Ritteret al\.\(2018\)H\. Ritter, A\. Botev, and D\. BarberOnline structured laplace approximations for overcoming catastrophic forgetting\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/1805.07810)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1)\.
- Rusuet al\.\(2016\)A\. A\. Rusu, N\. C\. Rabinowitz, G\. Desjardins,et al\.Progressive neural networks\.Note:arXiv preprintarXiv:1606\.04671External Links:[Link](https://arxiv.org/abs/1606.04671)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p1.1)\.
- Schlaget al\.\(2021\)I\. Schlag, K\. Irie, and J\. SchmidhuberLinear transformers are secretly fast weight programmers\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2102.11174)Cited by:[§2\.4](https://arxiv.org/html/2609.18842#S2.SS4.p1.1)\.
- Schmidhuber \(1992\)J\. SchmidhuberLearning to control fast\-weight memories: an alternative to dynamic recurrent networks\.Neural Computation4\(1\),pp\. 131–139\.Cited by:[§2\.4](https://arxiv.org/html/2609.18842#S2.SS4.p1.1)\.
- Schwarzet al\.\(2018\)J\. Schwarzet al\.Progress & compress: a scalable framework for continual learning\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/1805.06370)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p1.1)\.
- Shazeeret al\.\(2017\)N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. DeanOutrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/1701.06538)Cited by:[§1](https://arxiv.org/html/2609.18842#S1.p4.1),[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Shazeer \(2020\)N\. ShazeerGLU variants improve transformer\.Note:arXiv preprintarXiv:2002\.05202External Links:[Link](https://arxiv.org/abs/2002.05202)Cited by:[§3\.5\.2](https://arxiv.org/html/2609.18842#S3.SS5.SSS2.p1.1)\.
- Snellet al\.\(2024\)C\. Snell, J\. Lee, K\. Xu, and A\. KumarScaling LLM test\-time compute optimally can be more effective than scaling model parameters\.Note:arXiv preprintarXiv:2408\.03314External Links:[Link](https://arxiv.org/abs/2408.03314)Cited by:[§2\.4](https://arxiv.org/html/2609.18842#S2.SS4.p1.1)\.
- Suet al\.\(2025\)G\. Su, Y\. Yang, L\. Shen, L\. Yin, S\. Liu, and J\. GeipingRewiring experts on the fly: continuous rerouting for better online adaptation in mixture\-of\-expert models\.InInternational Conference on Machine Learning \(ICML\) 2026,External Links:[Link](https://arxiv.org/abs/2510.14853)Cited by:[§2\.4](https://arxiv.org/html/2609.18842#S2.SS4.p1.1),[Table 2](https://arxiv.org/html/2609.18842#S2.T2.2.6.1.1.1)\.
- Sunet al\.\(2024\)Y\. Sun, X\. Li, K\. Dalal, J\. Xu, A\. Vikram, G\. Zhang,et al\.Learning to \(learn at test time\): RNNs with expressive hidden states\.InInternational Conference on Machine Learning \(ICML\) 2025,External Links:[Link](https://arxiv.org/abs/2407.04620)Cited by:[§2\.4](https://arxiv.org/html/2609.18842#S2.SS4.p1.1),[Table 2](https://arxiv.org/html/2609.18842#S2.T2.2.3.1.1.1),[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3.SSS0.Px1.p1.1),[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3.SSS0.Px3.p3.1),[§3\.3](https://arxiv.org/html/2609.18842#S3.SS3.p1.1),[§5](https://arxiv.org/html/2609.18842#S5.p1.1)\.
- Takashiroet al\.\(2026\)S\. Takashiro, T\. Kojima, S\. Taniguchi, Y\. Iwasawa, and Y\. Matsuo∞\\infty\-MoE: generalizing mixture of experts to infinite experts\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(EACL\),External Links:[Link](https://arxiv.org/abs/2601.17680)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1),[§3\.4](https://arxiv.org/html/2609.18842#S3.SS4.p4.1)\.
- van de Ven and Tolias \(2019\)G\. M\. van de Ven and A\. S\. ToliasThree scenarios for continual learning\.Note:arXiv preprintarXiv:1904\.07734External Links:[Link](https://arxiv.org/abs/1904.07734)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p1.1)\.
- Villaloboset al\.\(2024\)P\. Villalobos, A\. Ho, J\. Sevilla, T\. Besiroglu, L\. Heim, and M\. HobbhahnWill we run out of data? limits of LLM scaling based on human\-generated data\.InInternational Conference on Machine Learning \(ICML\),External Links:[Link](https://arxiv.org/abs/2211.04325)Cited by:[§1](https://arxiv.org/html/2609.18842#S1.p1.1)\.
- Wanget al\.\(2023\)X\. Wang, T\. Chen, Q\. Ge,et al\.Orthogonal subspace learning for language model continual learning\.InFindings of the Association for Computational Linguistics: EMNLP 2023,External Links:[Link](https://arxiv.org/abs/2310.14152)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p1.1)\.
- Wanget al\.\(2024\)Y\. Wang, H\. Shi, L\. Han, D\. Metaxas, and H\. WangBLoB: bayesian low\-rank adaptation by backpropagation for large language models\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2406.11675)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1)\.
- Waterhouseet al\.\(1996\)S\. R\. Waterhouse, D\. J\. C\. MacKay, and A\. J\. RobinsonBayesian methods for mixtures of experts\.InAdvances in Neural Information Processing Systems \(NeurIPS\),pp\. 351–357\.Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1)\.
- Wuet al\.\(2024\)T\. Wuet al\.Continual learning for large language models: a survey\.Note:arXiv preprintarXiv:2402\.01364External Links:[Link](https://arxiv.org/abs/2402.01364)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p1.1)\.
- Wuet al\.\(2024\)X\. Wu, S\. Huang, and F\. WeiMixture of LoRA experts\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2404.13628)Cited by:[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1)\.
- Xiaet al\.\(2025\)Z\. Xia, A\. Davtyan, and P\. FavaroKOALA\+\+: efficient kalman\-based optimization with gradient\-covariance products\.Note:arXiv preprintarXiv:2506\.04432External Links:[Link](https://arxiv.org/abs/2506.04432)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1)\.
- Xieet al\.\(2022\)S\. M\. Xie, A\. Raghunathan, P\. Liang, and T\. MaAn explanation of in\-context learning as implicit bayesian inference\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2111.02080)Cited by:[§2\.6](https://arxiv.org/html/2609.18842#S2.SS6.p1.1),[1st item](https://arxiv.org/html/2609.18842#S3.I1.i1.p1.1),[§5](https://arxiv.org/html/2609.18842#S5.p1.1)\.
- Xueet al\.\(2022\)F\. Xue, X\. He, X\. Ren, Y\. Lou, and Y\. YouOne student knows all experts know: from sparse to dense\.Note:arXiv preprintarXiv:2201\.10890External Links:[Link](https://arxiv.org/abs/2201.10890)Cited by:[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1)\.
- Yanget al\.\(2024\)A\. X\. Yang, M\. Robeyns, X\. Wang, and L\. AitchisonBayesian low\-rank adaptation for large language models\.InInternational Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2308.13111)Cited by:[§2\.5](https://arxiv.org/html/2609.18842#S2.SS5.p2.1)\.
- Zadouriet al\.\(2023\)T\. Zadouri, A\. Üstün, A\. Ahmadian, B\. Ermiş, A\. Locatelli, and S\. HookerPushing mixture of experts to the limit: extremely parameter efficient MoE for instruction tuning\.Note:arXiv preprintarXiv:2309\.05444External Links:[Link](https://arxiv.org/abs/2309.05444)Cited by:[§2\.2](https://arxiv.org/html/2609.18842#S2.SS2.p1.1)\.
- Zenget al\.\(2025\)B\. Zeng, Y\. Yin, Z\. Xu, and Z\. LiuGenerative modeling of weights: generalization or memorization?\.Note:arXiv preprintarXiv:2506\.07998External Links:[Link](https://arxiv.org/abs/2506.07998)Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1),[§5](https://arxiv.org/html/2609.18842#S5.p1.1)\.
- Zenget al\.\(2026\)Y\. Zeng, L\. Lu, Z\. Li, Z\. Li, S\. Li, S\. Liao,et al\.MoEGen: mixture\-of\-experts for instance\-adaptive LoRA generation\.Note:arXiv preprintarXiv:2608\.03275External Links:[Link](https://arxiv.org/abs/2608.03275)Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1),[§2\.7](https://arxiv.org/html/2609.18842#S2.SS7.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2609.18842#S2.T1.6.8.1.1.1),[§3\.4](https://arxiv.org/html/2609.18842#S3.SS4.p4.1)\.
- Zhaoet al\.\(2024\)H\. Zhao, Z\. Qiu, H\. Wu, Z\. Wang, Z\. He, and J\. FuHyperMoE: towards better mixture of experts via transferring among experts\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://arxiv.org/abs/2402.12656)Cited by:[§2\.3](https://arxiv.org/html/2609.18842#S2.SS3.p1.1),[Table 1](https://arxiv.org/html/2609.18842#S2.T1.6.9.1.1.1),[§3\.4](https://arxiv.org/html/2609.18842#S3.SS4.p4.1)\.
- Zhonget al\.\(2024\)Z\. Zhong, M\. Xia, D\. Chen, and M\. LewisLory: fully differentiable mixture\-of\-experts for autoregressive language model pre\-training\.InConference on Language Modeling \(COLM\),External Links:[Link](https://arxiv.org/abs/2405.03133)Cited by:[§2\.1](https://arxiv.org/html/2609.18842#S2.SS1.p1.1)\.
- Zweigeret al\.\(2025\)A\. Zweiger, J\. Pari, H\. Guo, E\. Akyürek, Y\. Kim, and P\. AgrawalSelf\-adapting language models\.Note:arXiv preprintarXiv:2506\.10943External Links:[Link](https://arxiv.org/abs/2506.10943)Cited by:[§2\.4](https://arxiv.org/html/2609.18842#S2.SS4.p1.1),[Table 2](https://arxiv.org/html/2609.18842#S2.T2.2.5.1.1.1)\.