Open-world AI: a Tale of Three Requirements

# a tale of three Requirements
Source: [https://arxiv.org/html/2610.20782](https://arxiv.org/html/2610.20782)
## Open\-world AI: a tale of three RequirementsThanks:This research, including all access and use of models and datasets, was completed in author’s personal capacity\.

Léon BottouAffiliation:Flatiron InstituteEmail:[leon@bottou\.org](mailto:)

###### Abstract

The closed world asks AI for specialized competence and lets it manufacture theiiddata its theory assumes: a fixed criterion licenses collecting examples until the task is covered\. The open world asks for versatility — taking up tasks a machine was not built for, from few examples and with little prior from a human designer — and offers data that is grouped, not shuffled\. Its tasks are not known in advance, so a machine must do*inference\-time**learning*\(1st requirement\)\. Modeling tasks as sharing latent factors and decomposing the training objective, we derive two more: a*rich representation*, retaining more thaniidgeneralization requires, and a*predictive disentanglement*that aligns inputs and outputs into matched low\-dimensional groups\. The analysis further predicts the need for a consistent inner learner, and the benefit of long contexts spanning related tasks\. We test these claims on language models up to88b111Code:[https://github\.com/facebookresearch/MemoryMosaics](https://github.com/facebookresearch/MemoryMosaics), asking whether a model has a property \(qualitative\) rather than how much it wins \(quantitative\)\. Models that meet the requirements improve as data from related tasks accumulates; the rest degrade, and neither scale nor layer\-type interleaving repairs this\.

## 1Introduction

We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 \[…\] An attempt will be made to find how to make machines \[…\] solve kinds of problems now reserved for humans \[…\]\. We think that a significant advance can be made \[…\] for a summer\.— McCarthy et al\., 1955

Seventy years after the Dartmouth proposal\([McCarthy et al\., 1955](https://arxiv.org/html/2610.20782#bib.bib12)\), artificial intelligence has produced a long sequence of successes associated with the*closed world*: artificial environments\([Samuel, 1959](https://arxiv.org/html/2610.20782#bib.bib13);[Tesauro, 1995](https://arxiv.org/html/2610.20782#bib.bib14);[Silver et al\., 2016](https://arxiv.org/html/2610.20782#bib.bib15)\)and specific real\-world tasks\([Matan et al\., 1991](https://arxiv.org/html/2610.20782#bib.bib16);[Taigman et al\., 2014](https://arxiv.org/html/2610.20782#bib.bib17)\)\. Each rests on a clear criterion of success, which does two things at once\. 1\) The criterion specifies the distribution the machine is measured against, licensing the collection of more iid examples, with a generalization gap falling asO\(l−1/2\)O\(\\smash\[t\]\{l^\{\-1/2\}\}\)in the number of examplesll\([Vapnik, 1991](https://arxiv.org/html/2610.20782#bib.bib6)\)\. 2\) It reveals not merely*whether*a machine fails but*how*it fails, so that designers can repair the problems one after another\. Together these give a loop along which capability accumulates: locate a failure, collect data against it, repair, repeat\. On tasks other than the one it was trained on — even one that looks close — the machine may fail\([Zech et al\., 2018](https://arxiv.org/html/2610.20782#bib.bib26);[Koh et al\., 2021](https://arxiv.org/html/2610.20782#bib.bib2)\), and the loop must be run again, by a human, for each new task\. After building AI this way, rather than proving that our machines are intelligent, we have more often found satisfaction in proving that we are\.

Do large language models escape the*closed world*? Their competence across many specialized domains is real and impressive, and the in\-context learning ability it comes with is genuinely remarkable\. But two questions are still worth asking: how that competence was obtained, and what kind of in\-context learning it is\. On the first, the training practice is eager to consume all available data and to annotate as much task\-specific data as possible until the tasks of interest are likely to be covered \([Yang et al\., 2025a](https://arxiv.org/html/2610.20782#bib.bib30), §4\.1;[DeepSeek\-AI, 2026](https://arxiv.org/html/2610.20782#bib.bib38), §5\.1\)\.222Web crawls, source code, book corpora, chain\-of\-thought for mathematics, queries for reasoning\. The practice is general; these two reports document it explicitly, which most do not\.Benchmarks then serve as both criterion and measurement; a failure identified is a dataset collected, and the loop continues\. This is the closed\-world loop,*industrialized*rather than escaped\. On the second, modern in\-context learning is impressive but fragile\([Li et al\., 2024](https://arxiv.org/html/2610.20782#bib.bib11)\): supplying*more*demonstrations frequently makes performance*worse*rather than better; and frontier models \(e\.g\.,gpt\-4o\) may under\-perform smaller & older ones \(e\.g\.,llama\-2\-7b\)\. Neither more task\-specificiiddata nor more scale reliably improves performance on the task at hand\. This is not yet learning\.

The Dartmouth proposal set out to produce versatility rather than specialized competence — a machine that can take up a task it was not built for\. We take that as the target:

AI for the open world requires a machine to learn awide rangeof new tasks, usingfewer examplesandless task\-specific prior knowledgefrom human designers\.

Unlike closed\-world AI where the*tasks of interest*were already known at training time, open\-world AI does not come with predefined lists of tasks\. Thus, models must be able to adapt to new tasks on the fly\.333We consider this well understood and broadly accepted\.This gives the first requirement,*inference\-time learning*\.

Just as closed\-world AI relies on theiidsampling assumption to connect training and test*examples*, open\-world AI needs connections between training and test*tasks*\(groups of examples\)\([Baxter, 2000](https://arxiv.org/html/2610.20782#bib.bib46);[Maurer et al\., 2016](https://arxiv.org/html/2610.20782#bib.bib24);[Tripuraneni et al\., 2020](https://arxiv.org/html/2610.20782#bib.bib36);[Du et al\., 2020](https://arxiv.org/html/2610.20782#bib.bib37)\)\. Fortunately, the world is structured, and tasks are not arbitrary\. For example, in language modeling, poetry and mathematics are different tasks following very different statistics\. Yet these two tasks share substantial latent factors, e\.g\., language and grammatical construction\. This observation suggests viewing tasks and latent factors as a bipartite graph \(as in Figure[1](https://arxiv.org/html/2610.20782#S1.F1)\), and assuming two tasks to be more similar when they share more latent factors\.444Without this assumption, the world would be arbitrary and unpredictable\.

Figure 1:Latent factors, task, and related tasks\.In this work, Section[2](https://arxiv.org/html/2610.20782#S2), we first analyze*inference\-time learning*under the assumption that the latent\-factor overlap of*similar tasks*can be expressed as an overlap over internal representation of either the model inputs or the model outputs\. We argue that this analysis suggests two additional requirements:

- •*rich representation*\(gather broadly\): learns features richer thaniidgeneralization requires, making a wide range of new tasks reachable;
- •*predictive disentanglement*\(group predictively\): disentangles input and output representations into matching groups, and performs prediction within each group, making a new task learnable from fewer examples\.

Furthermore, this analysis allows us to make interesting predictions about learning algorithms for building open\-world AI:

- •the importance of a consistent inner learner for inference\-time learning;
- •the benefit of long contexts that span multiple related tasks, provided the target and related tasks are handled separately\.

Finally, Section[3](https://arxiv.org/html/2610.20782#S3)tests these requirements and predictions on language models up to88b\. What we look for is a*qualitative*difference — whether a model has a given property or lacks it — rather than a*quantitative*margin\. A margin in today’s language modeling depends sensitively on engineering detail that is largely unshared and would not fit in a scientific paper in any case; a qualitative difference is robust to that detail\.

What we find is that models meeting the requirements improve as data from related tasks accumulates, while the rest degrade\. Neither scale nor layer\-type interleaving repairs it\.

## 2Inference\-time learning

In closed\-world AI, both training and test examples are assumed to be drawniid\. There is then no need to learn at inference time, provided the data are actually sampled that way\. Real\-world data, however, has been observed for half a century to be “grouped”, where a group of examples \(a*task*, writtenτ\\tau\) is sampled conditionally on the choice of certain latent factors\([Heckman, 1979](https://arxiv.org/html/2610.20782#bib.bib28);[Zadrozny, 2004](https://arxiv.org/html/2610.20782#bib.bib27);[Torralba and Efros, 2011](https://arxiv.org/html/2610.20782#bib.bib23);[Koh et al\., 2021](https://arxiv.org/html/2610.20782#bib.bib2);[Liu and He, 2025](https://arxiv.org/html/2610.20782#bib.bib22)\)\. The latent factors are not fully observable and sparsely changed\([Schölkopf et al\., 2021](https://arxiv.org/html/2610.20782#bib.bib7)\)over tasks, resulting in a series of similar but not identical tasks at inference time\. Thus, in open\-world AI, adapting to new tasks on the fly —*inference\-time learning*— is necessary\.

### 2\.1Modeling inference\-time learning

To model inference\-time learning, we make two assumptions: 1\) inputs and outputs both have internal representations that are shared across tasks\([Finn et al\., 2017](https://arxiv.org/html/2610.20782#bib.bib1);[Garnelo et al\., 2018](https://arxiv.org/html/2610.20782#bib.bib47);[Kim et al\., 2019](https://arxiv.org/html/2610.20782#bib.bib48)\); 2\) the connection between those representations is task\-specific, and must therefore be learned at inference time\. The first says what is carried over between tasks; the second says what is not\. Together, they ask a machine to have four parts:

1. ➊input feature extractorϕθ,\\phi\_\{\\theta\},on inputxx,z=\[z1,…,zG\]=ϕθ​\(x\)z=\[z\_\{1\},\\ldots,z\_\{G\}\]=\\phi\_\{\\theta\}\(x\), shared across tasks\.
2. ➋output feature extractorψ\\psi, on outputyy,γ=\[γ1,…,γG\]=ψ⁡\(y\)\\gamma=\[\\gamma\_\{1\},\\ldots,\\gamma\_\{G\}\]=\\psi\(y\), shared across tasks\.ϕ\\phiandψ\\psilearn representation and partition them into matching groups\{\(zg,γg\)\}\\\{\(z\_\{g\},\\gamma\_\{g\}\)\\\}\.
3. ➌decoder\(affine\)AAmapsγ\\gammaback toyy:y=A​γy=A\\gamma\.555An affineAAdoes not makeψ\\psiaffine, sinceγ\\gammalives in a larger space thanyy\.
4. ➍inner learnerff, the only task\-specific part,*adapted at inference time*\. It is fitted on thellsupport\-set examplesS\(l\)=\{\(xi,yi\)\}i=1l∼τS^\{\(l\)\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{l\}\\sim\\tau, carried into latent space as\{\(zg,γg\)\}\\\{\(z\_\{g\},\\gamma\_\{g\}\)\\\}, and predicts group\-wise on a queryx∗∼τx^\{\*\}\\sim\\tau:γ^g=fg​\(zg\),y^=A⁡\[γ^1,…,γ^G\],\\hat\{\\gamma\}\_\{g\}=f\_\{g\}\(z\_\{g\}\)\\,,\\hat\{y\}=A\[\\hat\{\\gamma\}\_\{1\},\\ldots,\\hat\{\\gamma\}\_\{G\}\],where eachfgf\_\{g\}is fitted on the group\-ggpairs\{\(zg,γg\)\}\\\{\(z\_\{g\},\\gamma\_\{g\}\)\\\}alone\.

In practice a model stacks many such units; the analysis concerns the functional decomposition, not its allocation across layers\.

##### Training objective

The training process minimizes the error \(for ease of analysis we take the squared error throughout\) on a query\(x∗,y∗\)\(x^\{\*\},y^\{\*\}\)given support\-setS=\{\(x,y\)\}S=\\\{\(x,y\)\\\}drawn from the same task, averaged over tasks:

𝔼τ,S,\(x∗,y∗\)​\[\(y^∗−y∗\)2\]=𝔼l∼p⁡\(l\)​𝔼τ,S\(l\),\(x∗,y∗\)​\[\(y^∗−y∗\)2\],\\mathbb\{E\}\_\{\\tau,S,\(x^\{\*\},y^\{\*\}\)\}\\bigl\[\(\\hat\{y\}^\{\*\}\-y^\{\*\}\)^\{2\}\\bigr\]\\;=\\;\\mathbb\{E\}\_\{l\\sim p\(l\)\}\\;\\mathbb\{E\}\_\{\\tau,S^\{\(l\)\},\(x^\{\*\},y^\{\*\}\)\}\\bigl\[\(\\hat\{y\}^\{\*\}\-y^\{\*\}\)^\{2\}\\bigr\],\(1\)wherep⁡\(l\)p\(l\)is the distribution of support\-set sizes, which deployment does not let a machine choose\.

### 2\.2Decomposing the training objective

The reconstruction ofyydirectly supervises decoder and output extractor to avoid collapse: ifγ\\gammahas dropped information aboutyy, and equally ifAAcannot read it back, the reconstruction fails\. We therefore leave the decoderAAout of the analysis below\. We decompose the training objective around the*Bayes\-optimal prediction*on queryx∗x^\{\*\}given the support setS\(l\)S^\{\(l\)\}, formed in the latent space thatϕθ\\phi\_\{\\theta\}andψ\\psiprovide:

y¯∗=A𝔼\[γ∗∣z∗,\{\(z,γ\)\}\(l\)\]\.\\bar\{y\}^\{\*\}\\;=\\;A\\;\\mathbb\{E\}\\bigl\[\\gamma^\{\*\}\\mid z^\{\*\},\\\{\(z,\\gamma\)\\\}^\{\(l\)\}\\bigr\]\.\(2\)It is the best any inner learner could do with what the representations hand it\. SinceAAis affine it commutes with the conditional expectation, and sinceAArecoversyyfromγ\\gamma, conditioning on\{\(z,γ\)\}\(l\)\\\{\(z,\\gamma\)\\\}^\{\(l\)\}is conditioning on\{\(z,y\)\}\(l\)\\\{\(z,y\)\\\}^\{\(l\)\}; hencey¯∗=𝔼\[y∗∣z∗,\{\(z,y\)\}\(l\)\]\\bar\{y\}^\{\*\}=\\mathbb\{E\}\[y^\{\*\}\\mid z^\{\*\},\\\{\(z,y\)\\\}^\{\(l\)\}\]\. Predicting in latent space and predicting in output space share the same reference, and the decomposition can be written onyy:

𝔼τ,S\(l\),\(x∗,y∗\)​\[\(y^∗−y∗\)2\]\\displaystyle\\mathbb\{E\}\_\{\\tau,S^\{\(l\)\},\(x^\{\*\},y^\{\*\}\)\}\\bigl\[\(\\hat\{y\}^\{\*\}\-y^\{\*\}\)^\{2\}\\bigr\]=𝔼τ,S\(l\),\(x∗,y∗\)​\[\(y^∗−y¯∗\+y¯∗−y∗\)2\]\\displaystyle=\\mathbb\{E\}\_\{\\tau,S^\{\(l\)\},\(x^\{\*\},y^\{\*\}\)\}\\bigl\[\(\\hat\{y\}^\{\*\}\-\\bar\{y\}^\{\*\}\+\\bar\{y\}^\{\*\}\-y^\{\*\}\)^\{2\}\\bigr\]=𝔼τ​𝔼S\(l\),x∗​\[\(y¯∗−y∗\)2\]⏟ℓθ​\(l\)\+𝔼τ​𝔼S\(l\),x∗​\[\(y^∗−y¯∗\)2\]⏟ℓf​\(θ,l\)\\displaystyle=\\mathbb\{E\}\_\{\\tau\}\\,\\underbrace\{\\mathbb\{E\}\_\{S^\{\(l\)\},x^\{\*\}\}\\bigl\[\(\\bar\{y\}^\{\*\}\-y^\{\*\}\)^\{2\}\\bigr\]\}\_\{\\ell\_\{\\theta\}\(l\)\}\\;\+\\;\\mathbb\{E\}\_\{\\tau\}\\,\\underbrace\{\\mathbb\{E\}\_\{S^\{\(l\)\},x^\{\*\}\}\\bigl\[\(\\hat\{y\}^\{\*\}\-\\bar\{y\}^\{\*\}\)^\{2\}\\bigr\]\}\_\{\\ell\_\{f\}\(\\theta,l\)\}=𝔼τ​στ2\+𝔼τ​\[ℓθ​\(l\)−στ2\]\+𝔼τ​ℓf​\(θ,l\),\\displaystyle=\\mathbb\{E\}\_\{\\tau\}\\sigma\_\{\\tau\}^\{2\}\+\\mathbb\{E\}\_\{\\tau\}\[\\ell\_\{\\theta\}\(l\)\-\\sigma\_\{\\tau\}^\{2\}\]\+\\mathbb\{E\}\_\{\\tau\}\\ell\_\{f\}\(\\theta,l\)\\,,\(3\)whereστ2:=𝔼x∗​\[Var⁡\(y∗∣x∗,τ\)\]\\sigma^\{2\}\_\{\\tau\}:=\\mathbb\{E\}\_\{x^\{\*\}\}\\bigl\[\\mathrm\{Var\}\(y^\{\*\}\\mid x^\{\*\},\\tau\)\\bigr\]is the irreducible error of taskτ\\tau\. The cross term𝔼τ,S\(l\),x∗​\[\(y^∗−y¯∗\)​\(y¯∗−y∗\)\]=0\\mathbb\{E\}\_\{\\tau,S^\{\(l\)\},x^\{\*\}\}\[\(\\hat\{y\}^\{\*\}\-\\bar\{y\}^\{\*\}\)\(\\bar\{y\}^\{\*\}\-y^\{\*\}\)\]=0vanishes by the definition ofy¯∗\\bar\{y\}^\{\*\}:y^∗−y¯∗\\hat\{y\}^\{\*\}\-\\bar\{y\}^\{\*\}is a function of\(z∗,S\(l\)\)\(z^\{\*\},S^\{\(l\)\}\), andy¯∗−y∗\\bar\{y\}^\{\*\}\-y^\{\*\}has zero mean given the same\. This orthogonality is not special to squared error: it holds for any Bregman divergence, hence for cross\-entropy\.

Neither term in Eq\. \([3](https://arxiv.org/html/2610.20782#S2.Ex1)\) can be optimized on its own:ℓf\\ell\_\{f\}is defined against the Bayes\-optimaly¯∗\\bar\{y\}^\{\*\}, which is unknown\. Only the full objective is available, andlldecides which term it reaches\. At largell,ℓf\\ell\_\{f\}vanishes for any consistent inner learner, so reducing the objective reducesℓθ\\ell\_\{\\theta\}— the information the representation has permanently discarded — and builds a*rich representation*\. At smallll,ℓf\\ell\_\{f\}dominates instead, and reducing the objective pressesϕ\\phiandψ\\psitoward coordinates in which the predictionzg→γgz\_\{g\}\\rightarrow\\gamma\_\{g\}factorizes group\-wise:*predictive disentanglement*\. The two are ordered: becausey¯∗\\bar\{y\}^\{\*\}is defined relative toϕ\\phiandψ\\psi, the small\-llpressure is worth applying only once the representation is rich666This matches the finding of[Zhang et al\. \(2022\)](https://arxiv.org/html/2610.20782#bib.bib3);[Chen et al\. \(2023\)](https://arxiv.org/html/2610.20782#bib.bib8), where algorithms that seek an invariant representation directly fail, and a rich representation restores them\.\(built at largell\)\.

##### I\.𝔼τ​\[ℓθ​\(l\)−στ2\]\\mathbb\{E\}\_\{\\tau\}\[\\ell\_\{\\theta\}\(l\)\-\\sigma\_\{\\tau\}^\{2\}\]at largell: rich representation\.

Bayes\-optimal predictiony¯∗\\bar\{y\}^\{\*\}uses the exact prior of each task \(ideal case\), and thus depends only on the information retained inϕθ\\phi\_\{\\theta\}, not on how it is arranged\.𝔼τ​\[ℓθ​\(l\)−στ2\]\\mathbb\{E\}\_\{\\tau\}\[\\ell\_\{\\theta\}\(l\)\-\\sigma\_\{\\tau\}^\{2\}\]removes the effect of tasks’ irreducible error; at largellit measures the information aboutp⁡\(y∣x,τ\)p\(y\\mid x,\\tau\)thatϕθ\\phi\_\{\\theta\}has permanently discarded\. A smaller𝔼τ​\[ℓθ​\(l\)−στ2\]\\mathbb\{E\}\_\{\\tau\}\[\\ell\_\{\\theta\}\(l\)\-\\sigma\_\{\\tau\}^\{2\}\]indicates a richer representation\([Zhang and Bottou, 2023](https://arxiv.org/html/2610.20782#bib.bib5)\), which may be redundant for one task but helpful for a broader range of tasks\.

Furthermore, this work provides a way to learn rich representation\. Call an inner learner*consistent*ifℓf​\(θ,l\)→0\\ell\_\{f\}\(\\theta,l\)\\to 0asl→∞l\\to\\infty— given enough examples from a task, its prediction approaches the Bayes\-optimaly¯∗\\bar\{y\}^\{\*\}\([Györfi et al\., 2002](https://arxiv.org/html/2610.20782#bib.bib29)\)\.777This excludes learners whose effective memory is limited, whether by a fixed\-size state \(as in standard recurrent and state\-space models\), a sliding window, or strongly decaying positional encodings\.With a consistent inner learner𝔼τ​ℓf\\mathbb\{E\}\_\{\\tau\}\\ell\_\{f\}vanishes at largell, so the reduction of𝔼τ,S\(l\),\(x∗,y∗\)​\[\(y^∗−y∗\)2\]\\mathbb\{E\}\_\{\\tau,S^\{\(l\)\},\(x^\{\*\},y^\{\*\}\)\}\\bigl\[\(\\hat\{y\}^\{\*\}\-y^\{\*\}\)^\{2\}\\bigr\]operates directly on𝔼τ​\[ℓθ​\(l\)−στ2\]\\mathbb\{E\}\_\{\\tau\}\[\\ell\_\{\\theta\}\(l\)\-\\sigma\_\{\\tau\}^\{2\}\]\.*It implies that a rich representation can be constructed by minimizing𝔼τ,S\(l\),\(x∗,y∗\)​\[\(y^∗−y∗\)2\]\\mathbb\{E\}\_\{\\tau,S^\{\(l\)\},\(x^\{\*\},y^\{\*\}\)\}\\bigl\[\(\\hat\{y\}^\{\*\}\-y^\{\*\}\)^\{2\}\\bigr\]at largell\.*A linear Gaussian case analysis on rich representation is shown in Appendix[A](https://arxiv.org/html/2610.20782#A1)\.

##### II\.ℓf\\ell\_\{f\}at smallll: predictive disentanglement\.

The termℓf\\ell\_\{f\}measures how far the actual inner learner is from the Bayes\-optimal predictor \(which uses the exact task prior\) under the learned representation\. Once the representation is rich,ℓf\\ell\_\{f\}dominates in the small\-llregime\. So the goal is no longer to refine the representation, but to let the inner learner reachy¯∗\\bar\{y\}^\{\*\}from as few examples as possible\. Without being given a task\-specific prior, this asks for a prior shared across the tasks of interest\. One way to supply it is to reduce the inner learner from a single joint predictionz→γz\\rightarrow\\gammawithz,γ∈ℝmz,\\gamma\\in\\mathbb\{R\}^\{m\}, toGGseparate predictionszg→γgz\_\{g\}\\rightarrow\\gamma\_\{g\}withzg,γg∈ℝm/Gz\_\{g\},\\gamma\_\{g\}\\in\\mathbb\{R\}^\{m/G\}\.888In analogy to the sparse mechanism shift hypothesis\([Schölkopf et al\., 2021](https://arxiv.org/html/2610.20782#bib.bib7)\): tasks vary along sparse, independent directions\.Feature groups are shared across tasks; tasks differ in the function mapping within each group:

z=ϕ⁡\(x\)=\(z1,…,zG\),γ=ψ⁡\(y\)=\(γ1,…,γG\),\\displaystyle z=\\phi\(x\)=\(z\_\{1\},\\ldots,z\_\{G\}\),\\qquad\\gamma=\\psi\(y\)=\(\\gamma\_\{1\},\\ldots,\\gamma\_\{G\}\),𝔼\[γ∣z,τ\]=\(ητ​1\(z1\),…,ητ​G\(zG\)\),ητ​g\(zg\):=𝔼\[γg∣zg,τ\]\.\\displaystyle\\mathbb\{E\}\[\\gamma\\mid z,\\tau\]=\\bigl\(\\eta\_\{\\tau 1\}\(z\_\{1\}\),\\ldots,\\eta\_\{\\tau G\}\(z\_\{G\}\)\\bigr\),\\qquad\\eta\_\{\\tau g\}\(z\_\{g\}\):=\\mathbb\{E\}\[\\gamma\_\{g\}\\mid z\_\{g\},\\tau\]\.\(4\)Under this structure the Bayes predictor itself factorizes group\-wise, so a*block*\-diagonal inner learner loses nothing\. Therefore, minimizingℓf\\ell\_\{f\}at smallllwith such a learner pressures the representationsϕ,ψ\\phi,\\psito find the coordinates in which this form holds\. Since\(zg,γg\)\(z\_\{g\},\\gamma\_\{g\}\)are aligned for*prediction*,\(z1,…,zG\)\(z\_\{1\},\\dots,z\_\{G\}\)and\(γ1,…,γG\)\(\\gamma\_\{1\},\\dots,\\gamma\_\{G\}\)are*disentangled*in groups; we call this*predictive disentanglement*\.

##### Prediction in latent space\.

Predictingγ=ψ⁡\(y\)\\gamma=\\psi\(y\)rather thanyyis the move made explicit by joint\-embedding predictive architectures \(jepa\)\([LeCun, 2022](https://arxiv.org/html/2610.20782#bib.bib21)\)999Contrastive predictive coding\([Oord et al\., 2018](https://arxiv.org/html/2610.20782#bib.bib40)\)makes the same move with a contrastive objective\. Joint\-embedding methods \(jea\)\([Chen et al\., 2020](https://arxiv.org/html/2610.20782#bib.bib4);[Bardes et al\., 2022](https://arxiv.org/html/2610.20782#bib.bib10)\)match the embeddings of two views rather than predicting one from the other; they have no predictor for Eq\. \([4](https://arxiv.org/html/2610.20782#S2.Ex3)\) to factorize\., which encode the output and measure prediction error in embedding space to avoid spending capacity on unpredictable detail\.*Predictive disentanglement*shares that step but differs in two respects\. First, the predictor is not dense, but group\-wise as in Eq\. \([4](https://arxiv.org/html/2610.20782#S2.Ex3)\)\. Its sample complexity is governed by the sub\-problem dimensionm/Gm/Grather than the full dimensionmm\([Stone, 1985](https://arxiv.org/html/2610.20782#bib.bib39)\)\. Second, the motivation of predictive disentanglement is sample complexity at smallllrather than the difficulty of the output space\.jepais recovered with a single dense group, no task conditioning, and no decoder\.

### 2\.3Two predictions

#### 2\.3\.1Consistency of the inner learner

The rich\-representation analysis depends on the consistency of the inner learner: a consistent inner learner drivesℓf\\ell\_\{f\}to zero at largell, so the optimization pressure falls onϕθ\\phi\_\{\\theta\}and enriches it\. This rich representation in turn prepares features for learning predictive disentanglement at smallll\.

#### 2\.3\.2Task size, and the use of related tasks

The analysis of rich representation starts from largell\. Without largell, a rich representation cannot be constructed, and predictive disentanglement is therefore hard to learn\. The training process thus requires each task to contain enough examples\.

Could we sample arbitrarily many examples from the target task, the inner learner would need only to be consistent — predictive disentanglement having already made the sample complexity manageable — and would handle new tasks at inference time without a designer’s task\-specific prior\. We cannot\. As the ImageNetv2–ImageNet comparison shows\([Recht et al\., 2019](https://arxiv.org/html/2610.20782#bib.bib9)\), unknown hidden factors shift the distribution even under a faithful replication protocol, so data from the*target*taskτt\\tau\_\{t\}is hard to accumulate\. What accumulates instead is data from*related*tasksτr\\tau\_\{r\}, and using them when the target task does not supply enough examples becomes essential\.

We now analyze that case\. In the feature space defined by Eq\. \([4](https://arxiv.org/html/2610.20782#S2.Ex3)\), a related task helps with the target task’s groupggwhen its group\-wise conditional mean is close to the target task’s,ητr​g\(zg\):=𝔼\[γg∣zg,τr\]≈ητt​g\(zg\)\\eta\_\{\\tau\_\{r\}g\}\(z\_\{g\}\):=\\mathbb\{E\}\[\\gamma\_\{g\}\\mid z\_\{g\},\\tau\_\{r\}\]\\approx\\eta\_\{\\tau\_\{t\}g\}\(z\_\{g\}\)\(droppinggg,zzandτ\\tauwhen unambiguous:ηr≈ηt\\eta\_\{r\}\\approx\\eta\_\{t\}\), and hurts it otherwise\. Pooling every example into one estimate cannot tell the two cases apart, and the damage grows as related tasks accumulate\. Two things are therefore required: estimating the target and related tasks*separately*, asη^t\\hat\{\\eta\}\_\{t\}andη^r\\hat\{\\eta\}\_\{r\}, and mixing them*dynamically*,

η^=α​η^t\+\(1−α\)​η^r,\\hat\{\\eta\}=\\alpha\\,\\hat\{\\eta\}\_\{t\}\+\\bigl\(1\-\\alpha\\bigr\)\\hat\{\\eta\}\_\{r\}\\,,whereη^t\\hat\{\\eta\}\_\{t\}uses target examples alone,η^r\\hat\{\\eta\}\_\{r\}uses related\-task examples, andα\\alphaindicates how to mix\. The optimalα\\alphagrows as related tasks move further from the target task, and the learner effectively stops borrowing once they become too distant\([Grant et al\., 2018](https://arxiv.org/html/2610.20782#bib.bib49)\)\.

###### Proof\. Writeη^t=ηt\+ξt\\hat\{\\eta\}\_\{t\}=\\eta\_\{t\}\+\\xi\_\{t\}andη^r=ηr\+ξr\\hat\{\\eta\}\_\{r\}=\\eta\_\{r\}\+\\xi\_\{r\}with independent zero\-mean estimation errors,Vart:=𝔼⁡\[ξt2\]\\mathrm\{Var\}\_\{t\}:=\\mathbb\{E\}\[\\xi\_\{t\}^\{2\}\],Varr:=𝔼⁡\[ξr2\]\\mathrm\{Var\}\_\{r\}:=\\mathbb\{E\}\[\\xi\_\{r\}^\{2\}\], andδ:=ηr−ηt\\delta:=\\eta\_\{r\}\-\\eta\_\{t\}the mechanism discrepancy between them \(ηr\\eta\_\{r\}being the pooled mean over related tasks when several are available\)\. Minimizing the squared error𝔼⁡\[\(η^−ηt\)2\]=α2​Vart\+\(1−α\)2​\(Varr\+δ2\)\\mathbb\{E\}\\bigl\[\(\\hat\{\\eta\}\-\\eta\_\{t\}\)^\{2\}\\bigr\]=\\alpha^\{2\}\\mathrm\{Var\}\_\{t\}\+\(1\-\\alpha\)^\{2\}\(\\mathrm\{Var\}\_\{r\}\+\\delta^\{2\}\)gives the optimal weight:α∗=\(Varr\+δ2\)/\(Vart\+Varr\+δ2\),\\alpha^\{\*\}=\(\{\\mathrm\{Var\}\_\{r\}\+\\delta^\{2\}\}\)/\(\{\\mathrm\{Var\}\_\{t\}\+\\mathrm\{Var\}\_\{r\}\+\\delta^\{2\}\}\)\\,,which depends only on the statistics of the target and related task estimates\.δ\\deltais not observed at inference\. Treating tasks within a neighborhood as exchangeable makes𝔼⁡\[δ2\]\\mathbb\{E\}\[\\delta^\{2\}\]the between\-task variance, and spread among the related tasks estimates it\([Gelman and Hill, 2007](https://arxiv.org/html/2610.20782#bib.bib25)\)\. ∎

## 3Experiments

##### Qualitative effects, not quantitative ones\.

Sections[2\.2](https://arxiv.org/html/2610.20782#S2.SS2)and[2\.3](https://arxiv.org/html/2610.20782#S2.SS3)derive three requirements and two predictions\. The experiments in this section ask, for each, whether a model has it or lacks it — a*qualitative*comparison \(“does a model have a property needed for open\-world AI?”\) rather than a*quantitative*one \(“the proposed model outperforms baselines byxxpercent\.”\)\. A quantitative margin in modern language modeling depends sensitively on engineering details — data curation, training schedules, evaluation schemes — that are largely unpublished, making the comparison fragile\. By contrast, a qualitative comparison asks for the sign of an effect rather than its size, and a sign is robust to those details\.

### 3\.1Connecting the experiments to the theory

##### Language data\.

The theory is stated over examples and tasks and does not depend on modality\. We instantiate it in language because language corpora are large, widely shared, and the architectures we compare are established in that setting\. Every model in this work is trained on exactly the same mixture: Cosmopedia\([Ben Allal et al\., 2024](https://arxiv.org/html/2610.20782#bib.bib31)\), NemotronCC\-high\([Su et al\., 2025](https://arxiv.org/html/2610.20782#bib.bib32)\), open\-web\-math & arxiv\([Azerbayev et al\., 2024](https://arxiv.org/html/2610.20782#bib.bib34)\), Gutenberg books\([Manu, 2023](https://arxiv.org/html/2610.20782#bib.bib41)\), and ShareGPT\([Vicuna, 2023](https://arxiv.org/html/2610.20782#bib.bib33)\)\.101010Topical diversity is what matters, not the choice of sources: the same experiments on Cosmopedia alone — a single synthetic corpus, but a topically diverse one — give the same conclusions\.

##### Building tasks and related tasks\.

Following the illustration of*tasks*and*related tasks*in Figure[1](https://arxiv.org/html/2610.20782#S1.F1), we use natural documents as*tasks*, and semantic clusters of documents as*related tasks*\. We applykk\-means clustering to document embeddings\([Nussbaum et al\., 2024](https://arxiv.org/html/2610.20782#bib.bib35)\), getting11501150clusters for training \(each contains related tasks\)111111The exact number of clusters does not affect the conclusions; anything from a few hundred to a few thousand works\.\. Similarly, we partition the Simple English Wikipedia \(SimpleWiki\)121212[https://simple\.wikipedia\.org/wiki/Simple\_English\_Wikipedia](https://simple.wikipedia.org/wiki/Simple_English_Wikipedia)evaluation corpus into 19 clusters, after removing template\-stub clusters \(Appendix[B\.3](https://arxiv.org/html/2610.20782#A2.SS3)\)\.

The clusters built this way are semantically meaningful and reflect real\-world groupings\. To rule out artifacts of thekk\-means clustering, we also run the same evaluation without any post\-clustering, treating Simple English Wikipedia as a single natural cluster\. See Appendix[C](https://arxiv.org/html/2610.20782#A3)for details\.

##### Training and evaluation\.

Unless stated otherwise, every model at each scale uses the same hyper\-parameters, optimizer, and amount of training data, with no per\-model tuning\. Each sequence is built by sampling documents \(tasks\) from one randomly drawn cluster \(related tasks\), delimiting each with\[bos\]and\[eos\]131313Every model can therefore tell the target task apart from the related tasks in its context\., concatenating them, and truncating at lengthllfor training, and at lengthl′l^\{\\prime\}for evaluation\. The training objective, as in Eq\. \([1](https://arxiv.org/html/2610.20782#S2.E1)\), averages the loss from position11toll, sollis the maximum support set size seen in training; we call it the*maximum training length*, or*training length*for short\.

##### How to read the evaluation\.

To keep the comparison*qualitative*, and so robust to engineering choices and to the environment, we never directly compare raw performance numbers across model families\. Instead, for each model or model family, we vary the other factors — training lengthll, evaluation lengthl′l^\{\\prime\}, how to use related tasks — and ask whether the model has a property needed for open\-world AI, yes or no\.

### 3\.2Models to compare

Figure 2:Model architecture groups\. Architectures within each group differ only in the dark box\.\(a\)*Basic*: the dark box is one of\{\\\{Mosaics, attention, DeltaNet\}\\\}\.\(b\)*Modern hybrid*:\{\\\{DeltaNet, sliding window attention \(swa\)\}\\\}, interleaved with full attention \(nope\)\.\(c\)*\+\+*:\{\\\{Mosaics, attention\}\\\}; the dark box skips thessmost recent positions, theswabelow it uses a small windowww, and the leaky average in Mosaics is removed\.\(d\)*Two\-stage*:\{\\\{Mosaics, attention\}\\\}; as \(c\), but the target task and the related tasks are handled separately\.##### Architecture\.

We use two scales,∼\\sim1\.51\.5B and∼\\sim88B\. Every model has the same architecture layout: a token embedding, a stack of blocks each pairing a*learning module*\(the “dark box” of Figure[2](https://arxiv.org/html/2610.20782#S3.F2)\) with aswiglufeed\-forward network, and a linear decoder\. A learning module contains key and value extractors \(and, except in the Memory Mosaics family, a query extractor\)141414The Memory Mosaics family computes keys and queries with a single shared projection, so its learning module has 25% fewer parameters \(3​d23d^\{2\}instead of4​d24d^\{2\}\) than its counterparts\.and an inner learner that predicts from them\. It runsGGheads in parallel, one per group\(zg,γg\)\(z\_\{g\},\\gamma\_\{g\}\)of Eq\. \([4](https://arxiv.org/html/2610.20782#S2.Ex3)\): each head has its own extractors and its own group\-wise inner learnerfgf\_\{g\}\. Other detailed settings are in Appendix[B](https://arxiv.org/html/2610.20782#A2)\.

- a\)Basic architecture\. - –*Memory Mosaics*: The learning module’s inner learner is Gaussian kernel regression over keys and values, with the query equal to the key — per head, a single*long memory*spanning the whole context\. This inner learner is consistent\([Györfi et al\., 2002](https://arxiv.org/html/2610.20782#bib.bib29)\)\. Keysktk\_\{t\}and valuesvtv\_\{t\}are built from linear projectionsWk​xtW\_\{k\}x\_\{t\}andWv​xtW\_\{v\}x\_\{t\}of the inputxtx\_\{t\}\. The key is a leaky average of past projections,k¯t=gt​Wk​xt\+λt​k¯t−1\\bar\{k\}\_\{t\}=g\_\{t\}W\_\{k\}x\_\{t\}\+\\lambda\_\{t\}\\bar\{k\}\_\{t\-1\},kt=k¯t/‖k¯t‖k\_\{t\}=\\bar\{k\}\_\{t\}/\\\|\\bar\{k\}\_\{t\}\\\|, with input\-dependentλt∈\(0,1\)\\lambda\_\{t\}\\in\(0,1\)andgt\>0g\_\{t\}\>0\([Zhang and Bottou, 2026](https://arxiv.org/html/2610.20782#bib.bib44)\); the valuevtv\_\{t\}is a weighted average of the projections atttandt\+1t\+1\(‖vt‖=1\\\|v\_\{t\}\\\|=1\)\([Zhang et al\., 2025](https://arxiv.org/html/2610.20782#bib.bib18)\)\. Keys summarize the past and values the near future, so each stored pair is a*predictive*association rather than a retrieval of one position’s own representation\. The kernel regression predicts: 1Zt​∑i=1t−1eβ​kt⊤​ki​vi,Zt=∑i=1t−1eβ​kt⊤​ki,β=β1​tβ2\+β0,\\textstyle\\frac\{1\}\{Z\_\{t\}\}\\sum\_\{i=1\}^\{t\-1\}e^\{\\,\\beta\\,k\_\{t\}^\{\\top\}k\_\{i\}\}\\,v\_\{i\}\\,,\\qquad Z\_\{t\}=\\sum\_\{i=1\}^\{t\-1\}e^\{\\,\\beta\\,k\_\{t\}^\{\\top\}k\_\{i\}\}\\,,\\qquad\\beta=\\beta\_\{1\}t^\{\\beta\_\{2\}\}\+\\beta\_\{0\}\\,,whereeβ​kt⊤​kie^\{\\beta k\_\{t\}^\{\\top\}k\_\{i\}\}is a Gaussian kernel since keys have unit norm, with a bandwidth1/β\\sqrt\{1/\\beta\}that scales with positiontt\([Zhang and Bottou, 2026](https://arxiv.org/html/2610.20782#bib.bib44)\)\. This prediction is then scaled by a sigmoid gate computed from the input\. In terms of Section[2\.1](https://arxiv.org/html/2610.20782#S2.SS1), the key and value extractors contribute to the input and output feature extractorsϕ\\phi\(➊\) andψ\\psi\(➋\), and the kernel regression is the inner learnerff\(➍\)\. - –*Transformer*: replaces the learning module \(dark box\) with softmax attention \(rope\)\. Compared with Memory Mosaics, its inner learner is not consistent:rope’s positional decay bounds its effective memory, and its kerneleqt⊤​ki/dhe^\{q\_\{t\}^\{\\top\}k\_\{i\}/\\sqrt\{d\_\{h\}\}\}has no bandwidth schedule that narrows as the context grows, so its bias does not vanish asl→∞l\\to\\infty\([Györfi et al\., 2002](https://arxiv.org/html/2610.20782#bib.bib29)\)\. Its examples are also not predictive: keys and values are projections of the same token, so one attention layer retrieves a position’s own representation, and forming a predictive association takes at least two layers\([Olsson et al\., 2022](https://arxiv.org/html/2610.20782#bib.bib19);[Bietti et al\., 2023](https://arxiv.org/html/2610.20782#bib.bib20)\)\. - –*DeltaNet*: replaces the learning module with a gated delta\-rule recurrence\([Yang et al\., 2025b](https://arxiv.org/html/2610.20782#bib.bib50)\); its fixed\-size state makes it inconsistent once a task outgrows that state\.
- b\)Modern hybrids, interleaving DeltaNet/sliding window attention \(swa, 2k window length\)151515A 4k window gives the same conclusions; see Appendix[F](https://arxiv.org/html/2610.20782#A6)\.and full attention \(nope, no position encoding\) in successive blocks\. This gives two architectures:*Hybrid DeltaNet*and*Hybridswa*\. Both are configurations that frontier language models are converging on\([Agarwal et al\., 2025](https://arxiv.org/html/2610.20782#bib.bib51);[Qiu et al\., 2026](https://arxiv.org/html/2610.20782#bib.bib52)\)\.
- c\)\+ architecture\. - –*Memory Mosaics\+*: uses a separate*short memory*— a small sliding window attention module of sizeww\(or a mosaics or deltanet module\) — to summarize the near past below the*long memory*\. This allows the long memory to drop the leaky average from its keys and simply use normalized projectionskt=Wk​xt/‖Wk​xt‖k\_\{t\}=W\_\{k\}x\_\{t\}/\\\|W\_\{k\}x\_\{t\}\\\|\. It also allows the long memory to skip thessmost recent\(k,v\)\(k,v\)pairs, which are strongly dependent on the current position in sequential data\([Yu, 1994](https://arxiv.org/html/2610.20782#bib.bib43)\), since the recent information is already captured by the short memory: 1Zt​∑i=1t−seβ​kt⊤​ki​vi,Zt=∑i=1t−seβ​kt⊤​ki\.\\textstyle\\frac\{1\}\{Z\_\{t\}\}\\sum\_\{i=1\}^\{t\-s\}e^\{\\,\\beta\\,k\_\{t\}^\{\\top\}k\_\{i\}\}\\,v\_\{i\}\\,,\\qquad Z\_\{t\}=\\sum\_\{i=1\}^\{t\-s\}e^\{\\,\\beta\\,k\_\{t\}^\{\\top\}k\_\{i\}\}\\,\.Following[Zhang and Bottou \(2026\)](https://arxiv.org/html/2610.20782#bib.bib44),ssis drawn uniformly from\[64,256\]\[64,256\]during training and fixed at6464at inference, andw=256w=256; randomizingsshelps prevent the long memory from taking over the role of the short memory\. - –*Transformer\+*: as Memory Mosaics\+, with the same short memory, windowww, and skip schedule forss, but with attention \(nope, no position encoding\)161616Using attention with position encoding, e\.g\.rope, degrades performance\. See Appendix[E](https://arxiv.org/html/2610.20782#A5)\.as the long memory\.
- d\)Two\-stage architecture\. - –*Two\-stage Memory Mosaics\+*: as Memory Mosaics\+, but the long memory is split in two, one over the target task \(the current document, starting from positioncc\) and one over the related tasks \(the earlier documents in the context, ending atc−1c\-1\):171717Splitting the memory does not duplicate inner\-learner computation: the two sums run over disjoint parts of the context\. 1Ztrelated​∑i=1c−1eβ​kt⊤​ki​vi\+1Zttarget​∑i=ct−seβ​kt⊤​ki​vi,\\textstyle\\frac\{1\}\{Z\_\{t\}^\{\\mathrm\{related\}\}\}\\sum\_\{i=1\}^\{c\-1\}e^\{\\,\\beta\\,k\_\{t\}^\{\\top\}k\_\{i\}\}\\,v\_\{i\}\\;\+\\;\\frac\{1\}\{Z\_\{t\}^\{\\mathrm\{target\}\}\}\\sum\_\{i=c\}^\{t\-s\}e^\{\\,\\beta\\,k\_\{t\}^\{\\top\}k\_\{i\}\}\\,v\_\{i\}\\,,where eachZZnormalizes its own sum\. Each sum is a weighted average of unit\-norm values, so its magnitude is close to11when the retrieved values agree and shrinks when they disagree; the two contributions therefore enter in proportion to how concordant each body of evidence is, an implicit form of the mixing weightα\\alphaof Section[2\.3\.2](https://arxiv.org/html/2610.20782#S2.SS3.SSS2)\. Every structural choice in this model is one that Section[2](https://arxiv.org/html/2610.20782#S2)calls for: a consistent inner learner, \(k,vk,v\) example construction separated from prediction, and target task evidence held apart from related task evidence\. - –*Two\-stage Transformer\+*: as two\-stage Memory Mosaics\+, but with attention \(nope\) as each of the two parts of the long memory\.

Groups \(a\) and \(b\) are what the field currently builds\. Group \(c\) adds a separate short memory for strongly dependent neighboring tokens, and group \(d\) further separates the target task from related tasks \(Section[2\.3\.2](https://arxiv.org/html/2610.20782#S2.SS3.SSS2)\)\.

### 3\.3Experiments on rich representation and predictive disentanglement

Table 1:Memory Mosaics top\-1 accuracy \(%\) on SimpleWiki\. Rows are the training lengthll, columns the evaluation lengthl′l^\{\\prime\}\. Cells withl<l′l<l^\{\\prime\}are omitted\.Δ\\Deltais the64k rowminus thematched\-length entry\. Every column increases downward\.Figure 3:gsm8kaccuracy \(%\) of Memory Mosaics after fine\-tuning on math, against training lengthll\. Accuracy tends to increase withll\.
##### The positive impact of large training lengthll\.

At largell,ℓf\\ell\_\{f\}vanishes for a consistent inner learner, so the training objective of Eq\. \([3](https://arxiv.org/html/2610.20782#S2.Ex1)\) is dominated by what the representation has discarded\. Minimizing it produces a rich representation, which provides the features that predictive disentanglement then groups\. That is, a large maximum training lengthll, holding the total amount of training data fixed, benefits tasks other than the ones trained on\.

Table[1](https://arxiv.org/html/2610.20782#S3.T1)trains Memory Mosaics models at various maximum training lengthslland compares their performance at fixed evaluation lengthsl′l^\{\\prime\}on SimpleWiki\. Every column increases downward: the model trained atl=64l=64k is the best at every evaluation length, including lengths at which the extra context it saw during training is never available\.

Figure[3](https://arxiv.org/html/2610.20782#S3.F3)fine\-tunes181818We fine\-tune each model on 262k math examples, sweep the learning rate over\{3,6,9\}×10−5\\\{3,6,9\\\}\\times 10^\{\-5\}and the number of epochs over\{4,6,8\}\\\{4,6,8\\\}, and report the best configuration for each model\.the Memory Mosaics models trained atl=2l=2k–6464k \(those of Table[1](https://arxiv.org/html/2610.20782#S3.T1), plus one at22k\) on mathematics\([Yu et al\., 2023](https://arxiv.org/html/2610.20782#bib.bib45)\)and evaluates them ongsm8k, whose problems are shorter than11k tokens\. The result shows a positive correlation betweenlland downstream math reasoning performance\.

##### The importance of a consistent inner learner\.

The gain above depends on consistency\. If the inner learner is not consistent,ℓf\\ell\_\{f\}does not vanish at largell, soℓθ\\ell\_\{\\theta\}loses the direct optimization pressure, and disentanglement loses the rich features it needs\.

Table[2](https://arxiv.org/html/2610.20782#S3.T2)repeats the protocol of Table[1](https://arxiv.org/html/2610.20782#S3.T1)for four architectures whose inner learners lack consistency: the Transformer \(attention withrope\), DeltaNet \(a delta\-rule recurrence\), Hybrid DeltaNet, and Hybridswa\. The sign flips from beneficial to harmful\. None shows a systematic gain, while Memory Mosaics in Table[1](https://arxiv.org/html/2610.20782#S3.T1)benefits from a larger maximum training length\. For all four, the best configuration is to train at or just beyond the evaluation length\. The downstream results agree: fine\-tuned on math in the same way as Memory Mosaics \(Figure[3](https://arxiv.org/html/2610.20782#S3.F3)\), none shows a systematic gain ingsm8kaccuracy asllgrows \(Figure[4](https://arxiv.org/html/2610.20782#S3.F4)\)\.

Table 2:Top\-1 accuracy \(%\) on SimpleWiki for architectures whose inner learners lack consistency, under the same protocol as Table[1](https://arxiv.org/html/2610.20782#S3.T1)\. Red:l=l′l=l^\{\\prime\}\(trained at the evaluation length\)\. Blue:l=64l=64k\. A longer max training lengthlldoes not help these models learn new tasks; in almost every case it hurts them\.Figure 4:GSM8K accuracy \(%\) after fine\-tuning on math, against training lengthll, for the four architectures of Table[2](https://arxiv.org/html/2610.20782#S3.T2)\. Accuracy shows no systematic gain asllgrows, unlike Memory Mosaics \(Figure[3](https://arxiv.org/html/2610.20782#S3.F3)\)\. We do not compare accuracy*quantitatively*across architectures, since it is easily altered by small engineering details\. For example, Memory Mosaics has 25% fewer learning\-module parameters[14](https://arxiv.org/html/2610.20782#footnote14); filling this gap by adjusting its heads raises its accuracy to 28\.73% atl=4l=4k and 30\.78% atl=64l=64k, while its trend withllis unchanged \(Appendix[D](https://arxiv.org/html/2610.20782#A4)\)\.
##### Predictive disentanglement\.

Predictive disentanglement reduces the sample complexity of a new task: theGG\-group partition of the latent is learned from the training tasks and reused, and only the prediction within each group is recomputed for the new task \(Eq\. \([4](https://arxiv.org/html/2610.20782#S2.Ex3)\)\)\. A consistent inner learner, which keeps the optimization pressure on the representation \(Section[2\.2](https://arxiv.org/html/2610.20782#S2.SS2)\), lets training find such a partition and therefore benefits from a largeGG; an inconsistent one cannot\.

Table[3](https://arxiv.org/html/2610.20782#S3.T3)sweeps the number of heads for Memory Mosaics, Transformer, and Hybridswa\.191919DeltaNet and Hybrid DeltaNet are omitted, because changing the number of heads also affects the total recurrent state size\.The sign splits along consistency: Memory Mosaics gains as the number of heads rises,202020The per\-head gate is not the cause\. See Appendix[H](https://arxiv.org/html/2610.20782#A8)for ablation with gate removed\.while the Transformer212121Theropeposition encoding is not the cause\. See Appendix[H](https://arxiv.org/html/2610.20782#A8)for ablation withroperemoved\.and Hybridswalose\.

Table 3:Effect of the number of groupsGG\(heads\) at fixed model width: raising the number of heads raisesGGand shrinks each group\. All models are pre\-trained at lengthl=16l=16k\.Left:top\-1 accuracy on SimpleWiki, evaluated at lengthl′=4l^\{\\prime\}=4k\.Right:gsm8kaccuracy after fine\-tuning as in Figure[3](https://arxiv.org/html/2610.20782#S3.F3)\.Δ\\Deltais \(64 heads−\-4 heads\)\.

### 3\.4Inference\-time learning

##### Scaling the per\-task length at inference\.

Consider first the simple case in which there are no related tasks, in training or at evaluation — only isolated ones\. The inner learner of Section[2\.1](https://arxiv.org/html/2610.20782#S2.SS1)reads a support set of examples drawn independently from one task\. A token sequence does not come this way: neighboring tokens are strongly dependent, whereas sufficiently separated blocks behave as approximately independent\([Yu, 1994](https://arxiv.org/html/2610.20782#bib.bib43);[Mohri and Rostamizadeh, 2008](https://arxiv.org/html/2610.20782#bib.bib42)\)\. This is why in “\+\+” architectures \(Figure[2](https://arxiv.org/html/2610.20782#S3.F2)c\) we handle neighboring tokens separately and delay the long memory to ignore neighboring tokens\.

Figure 5:Left:Transformer vs Transformer\+\+\.Right:Memory Mosaics vs Memory Mosaics\+\+\.xx\-axis: evaluation per\-document length\. The gap widens with length\.To test this, we train four models \(Transformer, Transformer\+\+, Memory Mosaics, Memory Mosaics\+\+\) with a maximum sequence length of1616k under a document\-wise causal mask, which confines attention computation within each document even if it is very short \(never across documents\)\. We evaluate them on synthetic long documents formed by concatenating Simple English Wikipedia articles within each cluster, delimited by\\n\\n\.

Figure[5](https://arxiv.org/html/2610.20782#S3.F5)shows the result\. Transformer\+\+improves on Transformer, and Memory Mosaics\+\+improves substantially on Memory Mosaics\. Both “\+\+” models extrapolate beyond the training length\. The trend does not depend on how the short memory is implemented\. See Appendix[G](https://arxiv.org/html/2610.20782#A7)for ablation\.

##### Scaling the number of related tasks at inference\.

We now return to the realistic case: few examples from the target task, and many from related tasks\. Pooling all of them is biased; Section[2\.3](https://arxiv.org/html/2610.20782#S2.SS3)prescribes modeling them separately and combining them dynamically, which is what the two\-stage models do\. The dynamic mixing termα\\alphais implicitly controlled via the magnitudes of the two parts\. See Appendix[I](https://arxiv.org/html/2610.20782#A9)for details\.

Figure[6](https://arxiv.org/html/2610.20782#S3.F6)shows the result\. One\-stage models peak around their training length and then decline or flatten\. Two\-stage Transformer\+\+keeps improving, though slowly; two\-stage Memory Mosaics\+\+improves quickly\. The rightmost panel repeats the comparison at∼\\sim88bscale, over established architectures, hybrids, and two\-stage\+\+variants, and every curve keeps its shape — even though each member of the Memory Mosaics family is smaller than its counterpart[14](https://arxiv.org/html/2610.20782#footnote14)\. Capacity is not the missing ingredient; structure is\.

Figure 6:Accuracy as the number of related tasks in the context grows\. Training lengthl=16l=16k\. At1\.51\.5B:1\)RNN and hybrid architectures;2\)Transformer versus two\-stage Transformer\+\+;3\)Memory Mosaics versus two\-stage Memory Mosaics\+\+\. Panel4\)repeats the comparison at88b\.

## 4Conclusion

Open\-world AI asks for a machine that takes up tasks it was not built for — the first requirement,*inference\-time learning*\. Modeling the problem with tasks that share latent factors, and decomposing the training objective, this work derives two more requirements, a*rich representation*and a*predictive disentanglement*, and two predictions: that the inner learner must be consistent, and that the target task and the many related tasks in a long context must be handled separately\. The experiments test each qualitatively, asking whether a model has the property or lacks it\. Models with these properties improve as the training lengthllor the number of related tasks grows, while the others degrade\.

## References

- Agarwalet al\.\(2025\)S\. Agarwal, L\. Ahmad, J\. Ai, S\. Altman, A\. Applebaum, E\. Arbus, R\. K\. Arora, Y\. Bai, B\. Baker, H\. Bao,et al\.Gpt\-oss\-120b & gpt\-oss\-20b model card\.arXiv preprint arXiv:2508\.10925\.Cited by:[item b\)](https://arxiv.org/html/2610.20782#S3.I1.ix2.p1.1)\.
- Azerbayevet al\.\(2024\)Z\. Azerbayev, H\. Schoelkopf, K\. Paster, M\. Dos Santos, S\. McAleer, Q\. Jiang, J\. Deng, S\. R\. Biderman, and S\. WelleckLlemma: an open language model for mathematics\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 40622–40649\.Cited by:[Table 5](https://arxiv.org/html/2610.20782#A2.T5.2.4.1),[Table 5](https://arxiv.org/html/2610.20782#A2.T5.2.5.1),[§3\.1](https://arxiv.org/html/2610.20782#S3.SS1.SSS0.Px1.p1.1)\.
- Bardeset al\.\(2022\)A\. Bardes, J\. Ponce, and Y\. LeCunVICReg: variance\-invariance\-covariance regularization for self\-supervised learning\.InInternational Conference on Learning Representations,Cited by:[footnote 9](https://arxiv.org/html/2610.20782#footnote9)\.
- Baxter \(2000\)J\. BaxterA model of inductive bias learning\.Journal of artificial intelligence research12,pp\. 149–198\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p7.1)\.
- Ben Allalet al\.\(2024\)L\. Ben Allal, A\. Lozhkov, G\. Penedo, T\. Wolf, and L\. von WerraCosmopedia\.External Links:[Link](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia)Cited by:[Table 5](https://arxiv.org/html/2610.20782#A2.T5.2.2.1),[§3\.1](https://arxiv.org/html/2610.20782#S3.SS1.SSS0.Px1.p1.1)\.
- Biettiet al\.\(2023\)A\. Bietti, V\. Cabannes, D\. Bouchacourt, H\. Jegou, and L\. BottouBirth of a transformer: a memory viewpoint\.Advances in Neural Information Processing Systems36,pp\. 1560–1588\.Cited by:[2nd item](https://arxiv.org/html/2610.20782#S3.I1.ix1.I1.i2.p1.1)\.
- Chenet al\.\(2020\)T\. Chen, S\. Kornblith, M\. Norouzi, and G\. HintonA simple framework for contrastive learning of visual representations\.InInternational conference on machine learning,pp\. 1597–1607\.Cited by:[footnote 9](https://arxiv.org/html/2610.20782#footnote9)\.
- Chenet al\.\(2023\)Y\. Chen, W\. Huang, K\. Zhou, Y\. Bian, B\. Han, and J\. ChengTowards understanding feature learning in out\-of\-distribution generalization\.arXiv preprint arXiv:2304\.11327\.Cited by:[footnote 6](https://arxiv.org/html/2610.20782#footnote6)\.
- DeepSeek\-AI \(2026\)DeepSeek\-AIDeepSeek\-v4\.1\-flash: pushing the limits of kv cache compression\.Technical reportDeepSeek\-AI\.External Links:[Link](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash)Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p3.1)\.
- Duet al\.\(2020\)S\. S\. Du, W\. Hu, S\. M\. Kakade, J\. D\. Lee, and Q\. LeiFew\-shot learning via learning the representation, provably\.arXiv preprint arXiv:2002\.09434\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p7.1)\.
- Finnet al\.\(2017\)C\. Finn, P\. Abbeel, and S\. LevineModel\-agnostic meta\-learning for fast adaptation of deep networks\.InInternational conference on machine learning,pp\. 1126–1135\.Cited by:[§2\.1](https://arxiv.org/html/2610.20782#S2.SS1.p1.1)\.
- Garneloet al\.\(2018\)M\. Garnelo, D\. Rosenbaum, C\. Maddison, T\. Ramalho, D\. Saxton, M\. Shanahan, Y\. W\. Teh, D\. Rezende, and S\. A\. EslamiConditional neural processes\.InInternational conference on machine learning,pp\. 1704–1713\.Cited by:[§2\.1](https://arxiv.org/html/2610.20782#S2.SS1.p1.1)\.
- Gelman and Hill \(2007\)A\. Gelman and J\. HillData analysis using regression and multilevel/hierarchical models\.Vol\.1,Cambridge university press\.Cited by:[§2\.3\.2](https://arxiv.org/html/2610.20782#S2.SS3.SSS2.p4.pic1.p1.2.1)\.
- Grantet al\.\(2018\)E\. Grant, C\. Finn, S\. Levine, T\. Darrell, and T\. GriffithsRecasting gradient\-based meta\-learning as hierarchical bayes\.arXiv preprint arXiv:1801\.08930\.Cited by:[§2\.3\.2](https://arxiv.org/html/2610.20782#S2.SS3.SSS2.p3.2)\.
- Györfiet al\.\(2002\)L\. Györfi, M\. Kohler, A\. Krzyżak, and H\. WalkA distribution\-free theory of nonparametric regression\.Springer\.Cited by:[§2\.2](https://arxiv.org/html/2610.20782#S2.SS2.SSS0.Px1.p2.1),[1st item](https://arxiv.org/html/2610.20782#S3.I1.ix1.I1.i1.p1.1),[2nd item](https://arxiv.org/html/2610.20782#S3.I1.ix1.I1.i2.p1.1)\.
- Heckman \(1979\)J\. J\. HeckmanSample selection bias as a specification error\.Econometrica: Journal of the econometric society,pp\. 153–161\.Cited by:[§2](https://arxiv.org/html/2610.20782#S2.p1.1)\.
- Kimet al\.\(2019\)H\. Kim, A\. Mnih, J\. Schwarz, M\. Garnelo, A\. Eslami, D\. Rosenbaum, O\. Vinyals, and Y\. W\. TehAttentive neural processes\.arXiv preprint arXiv:1901\.05761\.Cited by:[§2\.1](https://arxiv.org/html/2610.20782#S2.SS1.p1.1)\.
- Kohet al\.\(2021\)P\. W\. Koh, S\. Sagawa, H\. Marklund, S\. M\. Xie, M\. Zhang, A\. Balsubramani, W\. Hu, M\. Yasunaga, R\. L\. Phillips, I\. Gao,et al\.Wilds: a benchmark of in\-the\-wild distribution shifts\.InInternational Conference on Machine Learning,pp\. 5637–5664\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p2.1),[§2](https://arxiv.org/html/2610.20782#S2.p1.1)\.
- LeCun \(2022\)Y\. LeCunA path towards autonomous machine intelligence version 0\.9\. 2, 2022\-06\-27\.Open Review62\(1\),pp\. 1–62\.Cited by:[§2\.2](https://arxiv.org/html/2610.20782#S2.SS2.SSS0.Px3.p1.1)\.
- Liet al\.\(2024\)T\. Li, G\. Zhang, Q\. D\. Do, X\. Yue, and W\. ChenLong\-context llms struggle with long in\-context learning\.arXiv preprint arXiv:2404\.02060\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p3.1)\.
- Liu and He \(2025\)Z\. Liu and K\. HeA decade’s battle on dataset bias: are we there yet?\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 41433–41449\.Cited by:[§2](https://arxiv.org/html/2610.20782#S2.p1.1)\.
- Manu \(2023\)ManuProject gutenberg\.Hugging Face\.Note:[https://huggingface\.co/datasets/manu/project\_gutenberg](https://huggingface.co/datasets/manu/project_gutenberg)Cited by:[Table 5](https://arxiv.org/html/2610.20782#A2.T5.2.7.1),[§3\.1](https://arxiv.org/html/2610.20782#S3.SS1.SSS0.Px1.p1.1)\.
- Matanet al\.\(1991\)O\. Matan, C\. J\. Burges, Y\. LeCun, and J\. DenkerMulti\-digit recognition using a space displacement neural network\.Advances in neural information processing systems4\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p2.1)\.
- Maureret al\.\(2016\)A\. Maurer, M\. Pontil, and B\. Romera\-ParedesThe benefit of multitask representation learning\.Journal of Machine Learning Research17\(81\),pp\. 1–32\.External Links:[Link](http://jmlr.org/papers/v17/15-242.html)Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p7.1)\.
- McCarthyet al\.\(1955\)J\. McCarthy, M\. Minsky, N\. Rochester, and C\. E\. ShannonA proposal for the dartmouth summer research project on artificial intelligence\.Note:Unpublished manuscript, Dartmouth CollegeCited by:[§1](https://arxiv.org/html/2610.20782#S1.p2.1)\.
- Mohri and Rostamizadeh \(2008\)M\. Mohri and A\. RostamizadehRademacher complexity bounds for non\-i\.i\.d\. processes\.InAdvances in Neural Information Processing Systems,D\. Koller, D\. Schuurmans, Y\. Bengio, and L\. Bottou \(Eds\.\),Vol\.21,pp\.\.Cited by:[§3\.4](https://arxiv.org/html/2610.20782#S3.SS4.SSS0.Px1.p1.1)\.
- Nussbaumet al\.\(2024\)Z\. Nussbaum, J\. X\. Morris, B\. Duderstadt, and A\. MulyarNomic embed: training a reproducible long context text embedder\.arXiv preprint arXiv:2402\.01613\.Cited by:[§B\.2](https://arxiv.org/html/2610.20782#A2.SS2.SSS0.Px2.p2.1),[§3\.1](https://arxiv.org/html/2610.20782#S3.SS1.SSS0.Px2.p1.1)\.
- Olssonet al\.\(2022\)C\. Olsson, N\. Elhage, N\. Nanda, N\. Joseph, N\. DasSarma, T\. Henighan, B\. Mann, A\. Askell, Y\. Bai, A\. Chen,et al\.In\-context learning and induction heads\.arXiv preprint arXiv:2209\.11895\.Cited by:[2nd item](https://arxiv.org/html/2610.20782#S3.I1.ix1.I1.i2.p1.1)\.
- Oordet al\.\(2018\)A\. v\. d\. Oord, Y\. Li, and O\. VinyalsRepresentation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.Cited by:[footnote 9](https://arxiv.org/html/2610.20782#footnote9)\.
- Qiuet al\.\(2026\)Z\. Qiu, Z\. Wang, X\. Li, Y\. Li, Y\. Xu, Y\. Wang, H\. Zhang, R\. Men, B\. Mao, C\. Zhang,et al\.On the design of qwen3\.8\-next architecture: evaluation, efficiency, and training stability\.arXiv preprint arXiv:2608\.30320\.Cited by:[item b\)](https://arxiv.org/html/2610.20782#S3.I1.ix2.p1.1)\.
- Rechtet al\.\(2019\)B\. Recht, R\. Roelofs, L\. Schmidt, and V\. ShankarDo imagenet classifiers generalize to imagenet?\.InInternational conference on machine learning,pp\. 5389–5400\.Cited by:[§2\.3\.2](https://arxiv.org/html/2610.20782#S2.SS3.SSS2.p2.1)\.
- Samuel \(1959\)A\. L\. SamuelSome studies in machine learning using the game of checkers\.IBM Journal of research and development3\(3\),pp\. 210–229\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p2.1)\.
- Schölkopfet al\.\(2021\)B\. Schölkopf, F\. Locatello, S\. Bauer, N\. R\. Ke, N\. Kalchbrenner, A\. Goyal, and Y\. BengioToward causal representation learning\.Proceedings of the IEEE109\(5\),pp\. 612–634\.Cited by:[§2](https://arxiv.org/html/2610.20782#S2.p1.1),[footnote 8](https://arxiv.org/html/2610.20782#footnote8)\.
- Silveret al\.\(2016\)D\. Silver, A\. Huang, C\. J\. Maddison, A\. Guez, L\. Sifre, G\. Van Den Driessche, J\. Schrittwieser, I\. Antonoglou, V\. Panneershelvam, M\. Lanctot,et al\.Mastering the game of go with deep neural networks and tree search\.nature529\(7587\),pp\. 484–489\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p2.1)\.
- Stone \(1985\)C\. J\. StoneAdditive regression and other nonparametric models\.The annals of Statistics,pp\. 689–705\.Cited by:[§2\.2](https://arxiv.org/html/2610.20782#S2.SS2.SSS0.Px3.p1.1)\.
- Suet al\.\(2025\)D\. Su, K\. Kong, Y\. Lin, J\. Jennings, B\. Norick, M\. Kliegl, M\. Patwary, M\. Shoeybi, and B\. CatanzaroNemotron\-cc: transforming common crawl into a refined long\-horizon pretraining dataset\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2459–2475\.Cited by:[Table 5](https://arxiv.org/html/2610.20782#A2.T5.2.3.1),[§3\.1](https://arxiv.org/html/2610.20782#S3.SS1.SSS0.Px1.p1.1)\.
- Taigmanet al\.\(2014\)Y\. Taigman, M\. Yang, M\. Ranzato, and L\. WolfDeepface: closing the gap to human\-level performance in face verification\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 1701–1708\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p2.1)\.
- Tesauro \(1995\)G\. TesauroTemporal difference learning and td\-gammon\.Communications of the ACM38\(3\),pp\. 58–68\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p2.1)\.
- Torralba and Efros \(2011\)A\. Torralba and A\. A\. EfrosUnbiased look at dataset bias\.InCVPR 2011,Vol\.,pp\. 1521–1528\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2011.5995347)Cited by:[§2](https://arxiv.org/html/2610.20782#S2.p1.1)\.
- Tripuraneniet al\.\(2020\)N\. Tripuraneni, M\. Jordan, and C\. JinOn the theory of transfer learning: the importance of task diversity\.Advances in neural information processing systems33,pp\. 7852–7862\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p7.1)\.
- Vapnik \(1991\)V\. VapnikPrinciples of risk minimization for learning theory\.Advances in neural information processing systems4\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p2.1)\.
- Vicuna \(2023\)VicunaShareGPT vicuna unfiltered\.Hugging Face\.Note:[https://huggingface\.co/datasets/anon8231489123/ShareGPT\_Vicuna\_unfiltered](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered)Cited by:[Table 5](https://arxiv.org/html/2610.20782#A2.T5.2.6.1),[§3\.1](https://arxiv.org/html/2610.20782#S3.SS1.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025a\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p3.1)\.
- Yanget al\.\(2025b\)S\. Yang, J\. Kautz, and A\. HatamizadehGated delta networks: improving mamba2 with delta rule\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 29687–29707\.Cited by:[3rd item](https://arxiv.org/html/2610.20782#S3.I1.ix1.I1.i3.p1.1)\.
- Yu \(1994\)B\. YuRates of convergence for empirical processes of stationary mixing sequences\.The Annals of Probability,pp\. 94–116\.Cited by:[1st item](https://arxiv.org/html/2610.20782#S3.I1.ix3.I2.i1.p1.1),[§3\.4](https://arxiv.org/html/2610.20782#S3.SS4.SSS0.Px1.p1.1)\.
- Yuet al\.\(2023\)L\. Yu, W\. Jiang, H\. Shi, J\. Yu, Z\. Liu, Y\. Zhang, J\. T\. Kwok, Z\. Li, A\. Weller, and W\. LiuMetaMath: bootstrap your own mathematical questions for large language models\.arXiv preprint arXiv:2309\.12284\.Cited by:[§3\.3](https://arxiv.org/html/2610.20782#S3.SS3.SSS0.Px1.p3.1)\.
- Zadrozny \(2004\)B\. ZadroznyLearning and evaluating classifiers under sample selection bias\.InProceedings of the twenty\-first international conference on Machine learning,pp\. 114\.Cited by:[§2](https://arxiv.org/html/2610.20782#S2.p1.1)\.
- Zechet al\.\(2018\)J\. R\. Zech, M\. A\. Badgeley, M\. Liu, A\. B\. Costa, J\. J\. Titano, and E\. K\. OermannVariable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross\-sectional study\.PLoS medicine15\(11\),pp\. e1002683\.Cited by:[§1](https://arxiv.org/html/2610.20782#S1.p2.1)\.
- Zhang and Bottou \(2023\)J\. Zhang and L\. BottouLearning useful representations for shifting tasks and distributions\.InInternational Conference on Machine Learning,pp\. 40830–40850\.Cited by:[§2\.2](https://arxiv.org/html/2610.20782#S2.SS2.SSS0.Px1.p1.1)\.
- Zhang and Bottou \(2026\)J\. Zhang and L\. BottouMemory mosaics at scale\.Advances in Neural Information Processing Systems38,pp\. 38929–38956\.Cited by:[1st item](https://arxiv.org/html/2610.20782#S3.I1.ix1.I1.i1.p2.1),[1st item](https://arxiv.org/html/2610.20782#S3.I1.ix1.I1.i1.p2.2),[1st item](https://arxiv.org/html/2610.20782#S3.I1.ix3.I2.i1.p1.2)\.
- Zhanget al\.\(2022\)J\. Zhang, D\. Lopez\-Paz, and L\. BottouRich feature construction for the optimization\-generalization dilemma\.InInternational Conference on Machine Learning,pp\. 26397–26411\.Cited by:[footnote 6](https://arxiv.org/html/2610.20782#footnote6)\.
- Zhanget al\.\(2025\)J\. Zhang, N\. Nolte, R\. Sadhukhan, B\. Chen, and L\. BottouMemory mosaics\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=IiagjrJNwF)Cited by:[1st item](https://arxiv.org/html/2610.20782#S3.I1.ix1.I1.i1.p2.1)\.

Open\-world AI: a Tale of Three Requirements

Supplementary Material

## Appendix AA linear Gaussian case study

This section makes𝔼τ​\[ℓθ​\(l\)−στ2\]\\mathbb\{E\}\_\{\\tau\}\[\\ell\_\{\\theta\}\(l\)\-\\sigma^\{2\}\_\{\\tau\}\]of Eq\. \([3](https://arxiv.org/html/2610.20782#S2.Ex1)\) explicit in a case where every quantity can be computed, and shows what a*rich representation*is there\.

##### Data generation\.

Each taskτ\\tauis a parameterω∈ℝd\\omega\\in\\mathbb\{R\}^\{d\}:

y=ω⊤​x\+ϵ,x∼𝒩⁡\(0,Σx\),ϵ∼𝒩⁡\(0,σϵ2\),ω∼p⁡\(ω\),\\displaystyle y=\\omega^\{\\top\}x\+\\epsilon,\\qquad x\\sim\\mathcal\{N\}\(0,\\Sigma\_\{x\}\),\\qquad\\epsilon\\sim\\mathcal\{N\}\(0,\\sigma\_\{\\epsilon\}^\{2\}\),\\qquad\\omega\\sim p\(\\omega\),withω\\omega,xxandϵ\\epsilonmutually independent,Σω:=𝔼⁡\[ω​ω⊤\]\\Sigma\_\{\\omega\}:=\\mathbb\{E\}\[\\omega\\omega^\{\\top\}\]finite, andrank⁡\(Σω\)≥m\\mathrm\{rank\}\(\\Sigma\_\{\\omega\}\)\\geq m,m≤dm\\leq d\. The irreducible error is the same for every task,στ2=σϵ2\\sigma^\{2\}\_\{\\tau\}=\\sigma^\{2\}\_\{\\epsilon\}\.

Theinput feature extractoris a rank\-mmlinear bottleneck,ϕ=V∈ℝm×d\\phi=V\\in\\mathbb\{R\}^\{m\\times d\},z=V​x∈ℝmz=Vx\\in\\mathbb\{R\}^\{m\}\. Theoutput feature extractoranddecoderare set to identity\. Theinner learnerffis ordinary least squares \(OLS\) with weightμ^∈ℝm\\hat\{\\mu\}\\in\\mathbb\{R\}^\{m\}and predictiony^∗=μ^⊤​z∗\\hat\{y\}^\{\*\}=\\hat\{\\mu\}^\{\\top\}z^\{\*\}\. OLS is chosen because atl→∞l\\to\\inftyit attains the Bayes prediction givenzz:ℓf​\(θ,∞\)=0\\ell\_\{f\}\(\\theta,\\infty\)=0exactly, so the whole of the remaining error isℓθ\\ell\_\{\\theta\}\.

Work in the whitened coordinatesx~=Σx−1/2x∼𝒩\(0,I\)\\tilde\{x\}=\\Sigma\_\{x\}^\{\-1/2\}x\\sim\\mathcal\{N\}\(0,I\),ω~=Σx1/2​ω\\tilde\{\\omega\}=\\Sigma\_\{x\}^\{1/2\}\\omegaandV~=V​Σx1/2∈ℝm×d\\tilde\{V\}=V\\Sigma\_\{x\}^\{1/2\}\\in\\mathbb\{R\}^\{m\\times d\}, so thaty=ω~⊤​x~\+ϵy=\\tilde\{\\omega\}^\{\\top\}\\tilde\{x\}\+\\epsilonandz=V~​x~z=\\tilde\{V\}\\tilde\{x\}\. Write

M:=𝔼⁡\[ω~​ω~⊤\]=Σx1/2​Σω​Σx1/2M:=\\mathbb\{E\}\\bigl\[\\tilde\{\\omega\}\\tilde\{\\omega\}^\{\\top\}\\bigr\]=\\Sigma\_\{x\}^\{1/2\}\\,\\Sigma\_\{\\omega\}\\,\\Sigma\_\{x\}^\{1/2\}for the whitened task\-signal covariance, andPV~=V~⊤​\(V~​V~⊤\)\+​V~P\_\{\\tilde\{V\}\}=\\tilde\{V\}^\{\\top\}\(\\tilde\{V\}\\tilde\{V\}^\{\\top\}\)^\{\+\}\\tilde\{V\}for the orthogonal projection ontorow⁡\(V~\)\\mathrm\{row\}\(\\tilde\{V\}\), where\+is the Moore–Penrose pseudoinverse\.

##### What the representation discards\.

Proposition 1\.*With OLS as the inner learner,*

𝔼τ​\[ℓθ​\(∞\)−στ2\]=𝔼ω​\[ω~⊤​\(I−PV~\)​ω~\]=tr⁡\(M⁡\(I−PV~\)\)\.\\mathbb\{E\}\_\{\\tau\}\\bigl\[\\ell\_\{\\theta\}\(\\infty\)\-\\sigma^\{2\}\_\{\\tau\}\\bigr\]\\;=\\;\\mathbb\{E\}\_\{\\omega\}\\bigl\[\\tilde\{\\omega\}^\{\\top\}\(I\-P\_\{\\tilde\{V\}\}\)\\tilde\{\\omega\}\\bigr\]\\;=\\;\\mathrm\{tr\}\\bigl\(M\\,\(I\-P\_\{\\tilde\{V\}\}\)\\bigr\)\.This is the expected squared task signal lying in the whitened directionsorthogonal to the row space ofV~\\tilde\{V\}\. It does not depend onll: no amount of task data recovers it, only a betterVVdoes\. This is the quantity Section[2\.2](https://arxiv.org/html/2610.20782#S2.SS2)calls the information the representation has permanently discarded\.

###### Proof\. PV~P\_\{\\tilde\{V\}\}is symmetric and idempotent withrank⁡\(PV~\)=rank⁡\(V~\)\\mathrm\{rank\}\(P\_\{\\tilde\{V\}\}\)=\\mathrm\{rank\}\(\\tilde\{V\}\), and\(I−PV~\)​x~⟂row⁡\(V~\)\(I\-P\_\{\\tilde\{V\}\}\)\\tilde\{x\}\\perp\\mathrm\{row\}\(\\tilde\{V\}\)\.Asl→∞l\\to\\inftythe OLS normal equations converge,\(Z⊤​Z\)/l→𝔼⁡\[z​z⊤\]\(Z^\{\\top\}Z\)/l\\to\\mathbb\{E\}\[zz^\{\\top\}\]andZ⊤​y/l→𝔼⁡\[z​y\]Z^\{\\top\}y/l\\to\\mathbb\{E\}\[zy\], so for a taskω\\omega,μ^ω=\(Z⊤​Z\)\+​Z⊤​y⟶𝔼​\[z​z⊤\]\+​𝔼​\[z​y\]=\(V​Σx​V⊤\)\+​V​Σx​ω,\\hat\{\\mu\}\_\{\\omega\}=\(Z^\{\\top\}Z\)^\{\+\}Z^\{\\top\}y\\;\\longrightarrow\\;\\mathbb\{E\}\[zz^\{\\top\}\]^\{\+\}\\,\\mathbb\{E\}\[zy\]=\(V\\Sigma\_\{x\}V^\{\\top\}\)^\{\+\}V\\Sigma\_\{x\}\\omega,giving the prediction and the residualy^∞∗\\displaystyle\\hat\{y\}^\{\*\}\_\{\\infty\}=μ^ω⊤​V​x∗=ω~⊤​PV~​x~∗,\\displaystyle=\\hat\{\\mu\}\_\{\\omega\}^\{\\top\}Vx^\{\*\}=\\tilde\{\\omega\}^\{\\top\}P\_\{\\tilde\{V\}\}\\tilde\{x\}^\{\*\},y∗−y^∞∗\\displaystyle y^\{\*\}\-\\hat\{y\}^\{\*\}\_\{\\infty\}=ω~⊤​x~∗\+ϵ−y^∞∗=ω~⊤​\(I−PV~\)​x~∗⏟⟂row⁡\(V~\)\+ϵ\.\\displaystyle=\\tilde\{\\omega\}^\{\\top\}\\tilde\{x\}^\{\*\}\+\\epsilon\-\\hat\{y\}^\{\*\}\_\{\\infty\}=\\tilde\{\\omega\}^\{\\top\}\\underbrace\{\(I\-P\_\{\\tilde\{V\}\}\)\\tilde\{x\}^\{\*\}\}\_\{\\perp\\;\\mathrm\{row\}\(\\tilde\{V\}\)\}\+\\epsilon\.Squaring and taking expectations,𝔼ω,x∗,ϵ​\[\(y∗−y^∞∗\)2\]\\displaystyle\\mathbb\{E\}\_\{\\omega,x^\{\*\},\\epsilon\}\\bigl\[\(y^\{\*\}\-\\hat\{y\}^\{\*\}\_\{\\infty\}\)^\{2\}\\bigr\]=𝔼⁡\[\(ω~⊤​\(I−PV~\)​x~∗\)2\]\+2​𝔼⁡\[ω~⊤​\(I−PV~\)​x~∗​ϵ\]⏟=0\+𝔼⁡\[ϵ2\]⏟=σϵ2\\displaystyle=\\mathbb\{E\}\\bigl\[\\bigl\(\\tilde\{\\omega\}^\{\\top\}\(I\-P\_\{\\tilde\{V\}\}\)\\tilde\{x\}^\{\*\}\\bigr\)^\{2\}\\bigr\]\+2\\,\\underbrace\{\\mathbb\{E\}\\bigl\[\\tilde\{\\omega\}^\{\\top\}\(I\-P\_\{\\tilde\{V\}\}\)\\tilde\{x\}^\{\*\}\\,\\epsilon\\bigr\]\}\_\{=\\,0\}\+\\underbrace\{\\mathbb\{E\}\[\\epsilon^\{2\}\]\}\_\{=\\,\\sigma^\{2\}\_\{\\epsilon\}\}=𝔼ω​\[ω~⊤​\(I−PV~\)​𝔼⁡\[x~∗​x~∗⁣⊤\]⏟=I​\(I−PV~\)​ω~\]\+σϵ2\\displaystyle=\\mathbb\{E\}\_\{\\omega\}\\bigl\[\\tilde\{\\omega\}^\{\\top\}\(I\-P\_\{\\tilde\{V\}\}\)\\underbrace\{\\mathbb\{E\}\[\\tilde\{x\}^\{\*\}\\tilde\{x\}^\{\*\\top\}\]\}\_\{=\\,I\}\(I\-P\_\{\\tilde\{V\}\}\)\\tilde\{\\omega\}\\bigr\]\+\\sigma^\{2\}\_\{\\epsilon\}=𝔼ω​\[ω~⊤​\(I−PV~\)​ω~\]\+σϵ2=tr⁡\(M⁡\(I−PV~\)\)\+σϵ2\.\\displaystyle=\\mathbb\{E\}\_\{\\omega\}\\bigl\[\\tilde\{\\omega\}^\{\\top\}\(I\-P\_\{\\tilde\{V\}\}\)\\tilde\{\\omega\}\\bigr\]\+\\sigma^\{2\}\_\{\\epsilon\}\\;=\\;\\mathrm\{tr\}\\bigl\(M\\,\(I\-P\_\{\\tilde\{V\}\}\)\\bigr\)\+\\sigma^\{2\}\_\{\\epsilon\}\.The cross term vanishes becauseϵ\\epsilonis zero\-mean and independent of\(ω,x∗\)\(\\omega,x^\{\*\}\)\. The second\-to\-last equality uses idempotency,\(I−PV~\)2=I−PV~\(I\-P\_\{\\tilde\{V\}\}\)^\{2\}=I\-P\_\{\\tilde\{V\}\}, and the last uses𝔼ω​\[ω~⊤​A​ω~\]=tr⁡\(𝔼⁡\[ω~​ω~⊤\]​A\)\\mathbb\{E\}\_\{\\omega\}\[\\tilde\{\\omega\}^\{\\top\}A\\tilde\{\\omega\}\]=\\mathrm\{tr\}\(\\mathbb\{E\}\[\\tilde\{\\omega\}\\tilde\{\\omega\}^\{\\top\}\]A\)for symmetricAA\. Subtractingστ2=σϵ2\\sigma^\{2\}\_\{\\tau\}=\\sigma^\{2\}\_\{\\epsilon\}gives the claim\. ∎

##### What the best representation retains\.

Proposition 2\.*LetM=U​Λ​U⊤M=U\\Lambda U^\{\\top\}withΛ=diag⁡\(λ1≥⋯≥λd≥0\)\\Lambda=\\mathrm\{diag\}\(\\lambda\_\{1\}\\geq\\cdots\\geq\\lambda\_\{d\}\\geq 0\)\. Then*

minV∈ℝm×d⁡tr⁡\(M⁡\(I−PV~\)\)=∑i\>mλi,\\min\_\{V\\in\\mathbb\{R\}^\{m\\times d\}\}\\;\\mathrm\{tr\}\\bigl\(M\(I\-P\_\{\\tilde\{V\}\}\)\\bigr\)\\;=\\;\\sum\_\{i\>m\}\\lambda\_\{i\},*attained by anyVVwhose whitened row spaceV~=V​Σx1/2\\tilde\{V\}=V\\Sigma\_\{x\}^\{1/2\}spans the top\-mmeigenvectors ofMM\.*

###### Proof\. Sincetr⁡\(M⁡\(I−PV~\)\)=tr⁡\(M\)−tr⁡\(M​PV~\)\\mathrm\{tr\}\(M\(I\-P\_\{\\tilde\{V\}\}\)\)=\\mathrm\{tr\}\(M\)\-\\mathrm\{tr\}\(MP\_\{\\tilde\{V\}\}\), minimizing the former is maximizingtr⁡\(M​PV~\)\\mathrm\{tr\}\(MP\_\{\\tilde\{V\}\}\), subject torank⁡\(PV~\)=rank⁡\(V~\)≤m\\mathrm\{rank\}\(P\_\{\\tilde\{V\}\}\)=\\mathrm\{rank\}\(\\tilde\{V\}\)\\leq m\.PutQ:=U⊤​PV~​UQ:=U^\{\\top\}P\_\{\\tilde\{V\}\}U\. It is again an orthogonal projection — symmetric and idempotent,Q⊤=Q=Q2Q^\{\\top\}=Q=Q^\{2\}— of the same rank\. Withwi:=Qi​iw\_\{i\}:=Q\_\{ii\},tr⁡\(M​PV~\)=tr⁡\(U​Λ​U⊤​PV~\)=tr⁡\(Λ​Q\)=∑i=1dλi​wi\.\\mathrm\{tr\}\(MP\_\{\\tilde\{V\}\}\)=\\mathrm\{tr\}\(U\\Lambda U^\{\\top\}P\_\{\\tilde\{V\}\}\)=\\mathrm\{tr\}\(\\Lambda Q\)=\\sum\_\{i=1\}^\{d\}\\lambda\_\{i\}w\_\{i\}\.BecauseQQis an orthogonal projection,wi∈\[0,1\]w\_\{i\}\\in\[0,1\]and∑iwi=tr⁡\(Q\)=tr⁡\(PV~​U​U⊤\)=rank⁡\(PV~\)≤m\\sum\_\{i\}w\_\{i\}=\\mathrm\{tr\}\(Q\)=\\mathrm\{tr\}\(P\_\{\\tilde\{V\}\}UU^\{\\top\}\)=\\mathrm\{rank\}\(P\_\{\\tilde\{V\}\}\)\\leq m\. The sum∑iλi​wi\\sum\_\{i\}\\lambda\_\{i\}w\_\{i\}is therefore at most∑i≤mλi\\sum\_\{i\\leq m\}\\lambda\_\{i\}, attained atw1=⋯=wm=1w\_\{1\}=\\cdots=w\_\{m\}=1andwm\+1=⋯=wd=0w\_\{m\+1\}=\\cdots=w\_\{d\}=0, that isQ=diag⁡\(Im,0\)Q=\\mathrm\{diag\}\(I\_\{m\},0\)andPV~=U​diag​\(Im,0\)​U⊤P\_\{\\tilde\{V\}\}=U\\,\\mathrm\{diag\}\(I\_\{m\},0\)\\,U^\{\\top\}\. This is exactly the projection onto the span of the top\-mmeigenvectors ofMM, and the minimum value istr⁡\(M\)−∑i≤mλi=∑i\>mλi\\mathrm\{tr\}\(M\)\-\\sum\_\{i\\leq m\}\\lambda\_\{i\}=\\sum\_\{i\>m\}\\lambda\_\{i\}\. ∎

##### Why a second principle is needed\.

Proposition 2 determines a*subspace*, not a basis: replacingV~\\tilde\{V\}byR​V~R\\tilde\{V\}for any invertibleR∈ℝm×mR\\in\\mathbb\{R\}^\{m\\times m\}leavesPV~P\_\{\\tilde\{V\}\}unchanged, and with it the loss\. The large\-llpressure of Section[2\.2](https://arxiv.org/html/2610.20782#S2.SS2)therefore fixes what the representation must retain and says nothing about how thosemmdirections are arranged into groups\. Choosing the arrangement is the separate job of predictive disentanglement at smallll— the same invariance that makesy¯∗\\bar\{y\}^\{\*\}independent of howψ\\psiorganizesγ\\gamma\.

## Appendix BExperiment details

### B\.1Architecture

Every model shares the layout of Section[3\.2](https://arxiv.org/html/2610.20782#S3.SS2): a token embedding, a stack of blocks each pairing a learning module with a SwiGLU feed\-forward network, and a linear decoder\. Embeddings are untied\. Table[4](https://arxiv.org/html/2610.20782#A2.T4)gives the settings held fixed at each of the two scales\.

Table 4:Architecture settings shared by all models at each scale\. The feed\-forward width follows the Llama SwiGLU convention \(see text\), and the input and output embeddings are not tied\.The feed\-forward width follows the Llama SwiGLU convention: a nominal4​d4dexpansion is multiplied by2/32/3to compensate for SwiGLU’s three weight matrices, then by a size\-dependent factor \(1\.01\.0at1\.51\.5B,1\.31\.3at88B\) and rounded up to a multiple of256256\. The hidden dimension is therefore2\.75​d2\.75\\,dand3\.5​d3\.5\\,d, not4​d4d\.

##### Position encoding\.

ropesplits each head intodh/2d\_\{h\}/2channel pairs and rotates pairiiof the query and key by an anglet​ωit\\,\\omega\_\{i\}at positiontt, withωi=θ−2i/dh\\omega\_\{i\}=\\theta^\{\-2i/d\_\{h\}\}andi=0,…,dh/2−1i=0,\\ldots,d\_\{h\}/2\-1, wheredhd\_\{h\}is the head dimension\.

Whereverropeis used, its baseθ\\thetais scaled linearly with the span the layer attends over, at a fixed ratioθ/context length≈122\\theta/\\text\{context length\}\\approx 122\.

This applies to the short sliding window \(256256\), to the sliding\-window blocks of the hybrids \(22k\), and to the Transformer of Table[2](https://arxiv.org/html/2610.20782#S3.T2)at each of its training lengths\. The Transformer baseline is therefore given a correctly rescaledropeat everyll, so its behaviour in Table[2](https://arxiv.org/html/2610.20782#S3.T2)is not an artifact of a mistunedθ\\theta\.

##### Hybrids\.

The hybrids interleave local and global blocks1:11\{:\}1: even\-numbered layers are local, odd\-numbered layers are full attention without position encoding, so layer00is local and the last layer is full attention\. That gives1212of each at1\.51\.5B and1616of each at88B\. The local block differs by family:*Hybridswa*uses sliding\-window attention withrope, while*Hybrid DeltaNet*uses a recurrence and therefore carries no position encoding at all\.

### B\.2Training

All models are trained with AdamW \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.95\\beta\_\{2\}=0\.95,ϵ=10−8\\epsilon=10^\{\-8\}\) and weight decay0\.10\.1, peak learning rate3×10−43\\times 10^\{\-4\}with200200warmup steps and a cosine decay to1%1\\%of peak, and gradient clipping at1\.01\.0\. Weight decay is not applied to the learned bandwidth and damping coefficients, whose optima are not at zero\. Training is inbf16underfsdp\. Every run uses the same seed; no pre\-training hyper\-parameter is tuned per model\.

##### Token budget\.

The1\.51\.5B models train for40004000steps and the88B models for80008000, at a global batch of222=4\.192^\{22\}=4\.19M tokens per step\. For the training\-length sweeps of Tables[1](https://arxiv.org/html/2610.20782#S3.T1)and[2](https://arxiv.org/html/2610.20782#S3.T2), the number ofgpus and the per\-device batch are chosen together so that this global batch is identical at everyll:

Every model in that comparison therefore sees the same number of tokens in the same number of optimizer steps, and only the length of the sequences differs\.

##### Data\.

Documents are drawn from six sources in the proportions of Table[5](https://arxiv.org/html/2610.20782#A2.T5)\. The percentages are shares of*documents*, not of tokens\. Cosmopedia and NemotronCC\-high mainly contain short documents, so their token contribution is well below their94%94\\%share of the documents\.

Table 5:Training mixture, as a share of documents\.Documents are embedded with[Nussbaum et al\. \(2024\)](https://arxiv.org/html/2610.20782#bib.bib35)and partitioned bykk\-means into12001200clusters \(11501150for training,5050held out for validation\)\. Each training step draws from3232clusters; within a step, a sequence is built from documents of a single cluster, delimited by\[bos\]and\[eos\], concatenated and truncated atll\.

### B\.3Evaluation clusters

The same clustering procedure gives5050clusters on Simple English Wikipedia\. The corpus contains large families of near\-template stubs — United States counties, Italian*comuni*, French communes — andkk\-means naturally collects each family into a cluster of its own:

> *Saint\-Créac is a commune in the Gers department\. It is in southwestern France\. Related pages: Communes of the Gers department\.* *Saint\-Georges is a commune in the Gers department\. It is in southwestern France\. Related pages: Communes of the Gers department\.* *Tourdun is a commune in the Gers department\. It is in southwestern France\. Related pages: Communes of the Gers department\.*

We remove these clusters, identified from the documents alone and not from any model’s predictions, to avoid trivial solutions\. That leaves1919clusters, used in all main\-text SimpleWiki evaluations\. Figure[7](https://arxiv.org/html/2610.20782#A2.F7)repeats the comparison on all5050: the conclusion is unchanged\.

Figure 7:Memory Mosaics and two\-stage Memory Mosaics\+\+on all5050Simple English Wikipedia clusters, at1\.51\.5B\. As Figure[6](https://arxiv.org/html/2610.20782#S3.F6), without the cluster selection\.

## Appendix CEvaluation on Simple English Wikipedia without clustering

The whole of Simple English Wikipedia is treated as one pool of related tasks, with no clustering\. Only the evaluation changes\. The training and the evaluation sequence construction are the same as in Figure[6](https://arxiv.org/html/2610.20782#S3.F6)\. The pattern is unchanged — two\-stage models keep improving, one\-stage models peak and decline — so the conclusion does not rest on the clustering step\. The two figures cover different portions of the evaluation corpus, so their shapes are comparable and their magnitudes are not\.

Figure 8:Accuracy as the number of related tasks in the context grows, at1\.51\.5B\. As Figure[6](https://arxiv.org/html/2610.20782#S3.F6), but the whole of Simple English Wikipedia is treated as related tasks, with no clustering\.1\)RNN and hybrid architectures;2\)Transformer versus two\-stage Transformer\+\+;3\)Memory Mosaics versus two\-stage Memory Mosaics\+\+\.
## Appendix DMemory Mosaics with matched parameters

Memory Mosaics has no query projection, so with the same head configuration its learning module has 25% fewer parameters than in the other architectures[14](https://arxiv.org/html/2610.20782#footnote14)\. Table[6](https://arxiv.org/html/2610.20782#A4.T6)closes that gap at the 1\.5B scale by using 43 heads of dimension 64 instead of 16 heads of dimension 128\. The key, value, and output projections then have3×2048×\(43⋅64\)≈16\.93\\times 2048\\times\(43\\cdot 64\)\\approx 16\.9M parameters per block, against4×20482≈16\.84\\times 2048^\{2\}\\approx 16\.8M for attention in the other architectures, a match within 1%\. Data, optimizer, and the fine\-tuning protocol are unchanged\. The table shows that a small engineering change shifts the accuracy level while leaving the sign ofΔ\\Deltaintact, which is why the main text makes qualitative comparisons rather than quantitative ones\.

Table 6:gsm8kaccuracy \(%\) after fine\-tuning as in Figure[3](https://arxiv.org/html/2610.20782#S3.F3), at 1\.5B\.Δ\\Deltais \(64k−\-4k\)\.
## Appendix EPosition encoding on the long\-range attention

Transformer\+\+appliesropeto the short sliding window and none to the long\-range attention above it\. Figure[9](https://arxiv.org/html/2610.20782#A5.F9)repeats the setting of Figure[5](https://arxiv.org/html/2610.20782#S3.F5)with a third curve, a Transformer\+\+whose long\-range attention also carriesrope\. Transformer\+\+withropeis worse than Transformer\+\+, confirming the choice of usingnopein the long\-range attention, as in the main text\.

Figure 9:As Figure[5](https://arxiv.org/html/2610.20782#S3.F5), at1\.51\.5B, with a third curve: Transformer\+\+whose long\-range attention also carriesrope\.
## Appendix FSliding\-window length in Hybridswa

Table 7:Top\-1 accuracy \(%\) on SimpleWiki for Hybridswawith a sliding\-window length of 4k vs 2k, under the same protocol as Table[1](https://arxiv.org/html/2610.20782#S3.T1)\. The window length does not change the conclusion: for both, a longer max training lengthlldoes not help the model learn new tasks\.
## Appendix GWhat implements the short memory

Section[3\.4](https://arxiv.org/html/2610.20782#S3.SS4)argues for a short\-memory component to handle highly dependent local tokens, but says nothing about its type\. Figure[10](https://arxiv.org/html/2610.20782#A7.F10)sweeps that short memory in Memory Mosaics\+\+over attention withrope, a Mosaics module with the leaky\-average key extractor, and DeltaNet, in the setting of Figure[5](https://arxiv.org/html/2610.20782#S3.F5)\. The curve “None” indicates no short memory, i\.e\. Memory Mosaics\. The other three curves follow the same trend: what matters is a separate short memory, not how it is implemented\.

Figure 10:As Figure[5](https://arxiv.org/html/2610.20782#S3.F5), at1\.51\.5B, sweeping the short\-memory implementation in Memory Mosaics\+\+: attention withrope, a Mosaics module with the leaky\-average key extractor, and DeltaNet\.
## Appendix HPer\-head gate and position encoding

Table[8](https://arxiv.org/html/2610.20782#A8.T8)repeats the comparison of Table[3](https://arxiv.org/html/2610.20782#S3.T3)with two ablations: Memory Mosaics without the per\-head sigmoid gate, and a Transformer without any position encoding \[nope\]\. Neither changes the sign\. Without the gate, Memory Mosaics still gains asGGgrows, so the gate is not what produces the gain\. Withoutrope, the Transformer still does not gain, so position encoding is not what prevents it; the same holds for Hybridswa, whose full\-attention layers already carry no position encoding\. Transformer \[nope\] is lower overall, as expected: with no position encoding anywhere, it cannot model the dependence between nearby tokens\. Only the sign matters here\.

Table 8:Top\-1 accuracy \(%\) on SimpleWiki, evaluated at lengthl′=4l^\{\\prime\}=4k; all models trained at lengthl=16l=16k\. As Table[3](https://arxiv.org/html/2610.20782#S3.T3), with two ablations added: Memory Mosaics without the per\-head gate, and Transformer without position encoding \[nope\]\. Model width is fixed, so raising the number of heads raises the number of groupsGGand shrinks each group\. Row maxima in bold\.Δ\\Deltais \(64 heads−\-4 heads\)\.
## Appendix IThe implicit balancing weight

The two\-stage models never compute the optimal weightα∗=\(Varr\+δ2\)/\(Vart\+Varr\+δ2\)\\alpha^\{\*\}=\(\\mathrm\{Var\}\_\{r\}\+\\delta^\{2\}\)/\(\\mathrm\{Var\}\_\{t\}\+\\mathrm\{Var\}\_\{r\}\+\\delta^\{2\}\)of Section[2\.3](https://arxiv.org/html/2610.20782#S2.SS3); each part is normalized on its own, so the two contributions enter in proportion to their output magnitudes\. That magnitude is informative because each part is a normalized average of unit\-norm values: its norm approaches11when the retrieved values agree and falls towards00when they disagree\. Figure[11](https://arxiv.org/html/2610.20782#A9.F11)plots the two against evaluation length\. The target\-task part stays flat while the related\-task part grows — the related evidence is becoming more concordant as it accumulates — so the balance shifts towards it with nothing computing a weight\. This reproduces the direction in whichα∗\\alpha^\{\*\}moves as related tasks accumulate, though not its value; estimating the weight directly is left to future work\.

Figure 11:Output magnitude of the target\-task and related\-task parts of the long memory in two\-stage Memory Mosaics\+\+at1\.51\.5B, against evaluation length\.