理解与构建具备“任意任务推理”能力的基础模型的理论框架

# A Theoretical Framework for Understanding and Building “Any-task inference” Capable Foundation Models
Source: [https://arxiv.org/html/2610.40934](https://arxiv.org/html/2610.40934)
Matthew McDermottmm6677@cumc\.columbia\.eduFlorent PolletAffiliation:Columbia University Department of Biomedical InformaticsPayal ChandakAffiliation:Harvard Medical SchoolGregory KondasAffiliation:Columbia University Department of Biomedical InformaticsLiat Friedman AntwargAffiliation:Harvard Medical SchoolNoa DaganAffiliation:Clalit Health Services and Ben Gurion UniversityIsaac KohaneAffiliation:Harvard Medical School

###### Abstract

In this work, we provide a formal framework and mathematical definition for the subset of foundation models focused on enabling the property of “any\-task inference” \(ATI\) \(similar to zero\-shot inference\), rather than those designed for pre\-training fine\-tuning\. To achieve this mathematical definition, we formalize the notion of a “space of tasks of interest,” which we include in the problem domain, then provide an interface\-based formal definition for ATI foundation models as those that map samples of dataxxand tasksttto predictions over the space of labels directly\. While simple, this definition allows us to show that all ATI foundation models can be decomposed through a “universal task reduction” lens into encoder, model, and decoder aspects\. We can leverage this decomposition to also decompose foundation model error into terms corresponding to different aspects of this reduction, stratified into a collection of “design terms,” which are theoretical properties that can inform one of whether a particular proposed ATI foundation model can achieve optimality or not due to the structure of the proposed task reduction, and “realized model excess” terms which capture interactions between learned and programmatic components of foundation model systems\. Lastly, we can use this perspective to offer two “prototypical” recipes for realizing ATI foundation models in diverse domains, which we term generative and discriminative variants of ATI foundation models\. This exercise is grounded throughout by 5 real\-world examples of foundation models in different settings, varying greatly in how these capabilities are realized, showing the applicability of this definition to real\-world settings\.

## 1Introduction

This work aims to provide a formal definition of the subset of “foundation models” that are designed to enable the practical property typically called “zero\-shot inference,” which we recognize more directly as the ability to use the foundation model to perform inference for any arbitrary task in a given domain, on any input sample, with no additional training required\. A formal definition for this class of foundation models will reduce ambiguity when this term is used in research, allow for more comprehensive theoretical understanding of these foundation models, and help us see how we can build them in diverse domains\.

### 1\.1Focusing on Foundation Models that enable Direct “Any\-task inference\.”

The term “foundation model” was originally introduced by[Bommasani et al\. \(2021\)](https://arxiv.org/html/2610.40934#bib.bib13)as “*models … trained on broad data \(generally using self\-supervision at scale\) that can be adapted to a wide range of downstream tasks*”\([Bommasani et al\., 2021](https://arxiv.org/html/2610.40934#bib.bib13)\)\. This definition has since been applied to many different kinds of models that share different traits and usage goals\. In this work, we assume that the key property in this definition, and the property that matters most for something to be a “foundation model” is that of its data\-efficient utility on a “wide range of downstream tasks\.” In the modern era, this utility is best epitomized through models that permit*immediate, no\-fine\-tuning required inference*across an*arbitrary range of reasonable tasks over a domain of interest\.*We will use the term ‘‘any\-task inference \(ATI\) capability’’ to refer to this ability \-\- being able to immediately perform inference on an arbitrary task of interest without any additional fine\-tuning\.111We use this term rather than the term “zero\-shot inference” because our primary interest is not in*how a model was trained*\(i\.e\., what particular style of learning objective was used\) but rather in*what you can do with it at inference time*\(i\.e\., make predictions for arbitrary tasks of interest\), though in many cases these two capabilities will intersect\.

In this work, we will focus our analyses specifically on foundation models that are capable of ATI, which we will call ATI foundation models, rather than other kinds of foundation models\. In so doing, we are explicitly not studying other notions of what may make something a “foundation model” that are commonly used today\. In particular, in this work we will take the following positions:

1. 1\.We will assume that foundation models need not be generative models\. Instead, the focus on downstream task utility is paramount, not their ability to estimate the likelihood of or generate samples from the data domain\.
2. 2\.We will assume that foundation models need not be trained with any particular style of self\-supervision or neural network architecture\. What makes foundation models unique is how they can be used at inference time, not how they were trained\.
3. 3\.We will focus our analyses on ATI foundation models,*rather than foundation models that require fine\-tuning and do not permit ATI\.*
4. 4\.We will assume that foundation models need not solve problems of distribution shift over their domains, provided they still show any\-task adaptability to diverse downstream tasks in the domain of interest\. Critically, as we will formalize later in the work, any\-task adaptability itself need not require distribution shift, so this is not in conflict\.

Note that we do not argue that these topics are not interesting, useful, and important in their own right – just that we can separate them from the concept of an “ATI foundation model,” \(and arguably from the term “foundation model” more generally\) and reserve this new term to refer more specifically to these any\-task capable models\. The established terms for each of these capabilities \(e\.g\., generative modeling, self\-supervised modeling, transfer learning, and distribution shift\) can instead be used to refer to those specific areas individually\. Note that we may at times just use the term “foundation model” in the text, but we will near universally use this to mean an ATI foundation model unless otherwise specified\.

### 1\.2This Definition Permits Meaningful Theoretical Contributions

With this narrowed focus, we can provide a formal mathematical definition of ATI foundation models that permits us to make several theoretical advances\. Concretely, to summarize the contributions of this work:

1. 1\.We re\-introduce the notion of a “distribution of tasks” over a space𝒯\\mathcal\{T\}that is jointly distributed with one’s data over𝒳\\mathcal\{X\}and labels over𝒴\\mathcal\{Y\}\.222To the best of our knowledge, this concept was first introduced in[Baxter \(2000\)](https://arxiv.org/html/2610.40934#bib.bib15)\.We further introduce the notion of stochastic representationsρ\\rhoof tasks that map from the \(potentially intractable and unobserved\) space𝒯\\mathcal\{T\}to a usable space for modeling\.
2. 2\.Given this joint domain, we define any\-task inference capable foundation models as*procedures*that operate jointly over pairs of data samples and tasks to predict the target label:𝙿:𝒳×𝒯→𝒫⁡\(𝒴\)\\mathtt\{P\}:\\mathcal\{X\}\\times\\mathcal\{T\}\\to\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)\.
3. 3\.We show that any foundation model under this definition can be decomposed into “encoder”, “model”, and “decoder” stages, which formalize the view that foundation models enable any\-task inference by performing an algorithmic reduction in the space of tasks to some “universal task\.”
4. 4\.We further show that this decomposition gives rise to a clear decomposition of the sources of error any foundation model has, across terms dependent on each of these components, including the following error terms: - •Irreducible additional uncertainty due to low fidelity representations of the input task\. - •Limitations in how effectively the universal task can approximate the true task\. - •Error in how effectively the model component actually models the universal task\. - •Error arising from the choice of labeling function used to translate universal\-task outputs into downstream labels\. - •Error arising from imperfect approximation of the target decoding distribution by the realized decoder\.
5. 5\.We give two practical recipes for how to build foundation models in diverse domains\.
6. 6\.We show that foundation model evaluation is mathematically analogous to evaluation in the context of algorithmic fairness, yielding identical impossibility criteria\.

### 1\.3Running examples

We will use several running examples of ATI foundation models in the literature today throughout this work, including the following:

#### Large language models

We will consider large language models \(LLMs\) as ATI foundation models in the sense that, through prompting and/or specific decoding strategies, diverse \(and arguably all\) tasks in natural language processing \(NLP\) can be reframed as next\-token prediction problems, and thus a well trained LLM arguably should be useful for solving arbitrary examples of such tasks in NLP\. Examples include[Radford et al\. \(2019\)](https://arxiv.org/html/2610.40934#bib.bib16);[Brown et al\. \(2020\)](https://arxiv.org/html/2610.40934#bib.bib17);[Achiam et al\. \(2023\)](https://arxiv.org/html/2610.40934#bib.bib18)\. We separate these any\-task capable examples from PT/FT LLMs such as BERT\([Devlin et al\., 2019](https://arxiv.org/html/2610.40934#bib.bib20)\), which we will not consider here \(though we note that masked language models can in theory be used for any\-task inference as well, though this is less well explored in practice\)\.

#### Autoregressive Foundation Models for Electronic Health Record Data

An emerging area of foundation models today concerns autoregressive \(AR\) sequence models applied over longitudinal electronic health record \(EHR\) datasets \(typically expressed in event\-stream format, e\.g\., in the MEDS standard\([McDermott et al\., 2026](https://arxiv.org/html/2610.40934#bib.bib7)\)which is then tokenized into some categorical sequence\([Guo et al\., 2026](https://arxiv.org/html/2610.40934#bib.bib4);[Lee et al\., 2026](https://arxiv.org/html/2610.40934#bib.bib2);[Attrach et al\., 2026](https://arxiv.org/html/2610.40934#bib.bib3)\)\)\. These models appear superficially similar to LLMs, but perform any\-task inference in a distinct pathway\. Whereas LLMs use prompting to modify the “input” for an any\-task problem in some controlled fashion, EHR AR FMs do not modify the input \(a stream of patient level data\) at all \(and are generally not capable of prompting\)\. Instead, they simply forecast possible “simulated futures” of the patient data, then infer the task label via some statistical form \(e\.g\., Monte Carlo sampling\) over those simulated futures directly\. Examples include[Renc et al\. \(2024\)](https://arxiv.org/html/2610.40934#bib.bib23);[Waxler et al\. \(2025\)](https://arxiv.org/html/2610.40934#bib.bib22)\.

#### Prior Fitted Networks

Another emerging area of foundation models concerns prior fitted networks\([Müller et al\., 2022](https://arxiv.org/html/2610.40934#bib.bib14)\), which frame foundation models as a Bayesian inference problem to learn arbitrary posteriors given samples of that posterior learned through a dataset derived from a prior over distributions\. Recent works such as TabPFN\([Hollmann et al\., 2025](https://arxiv.org/html/2610.40934#bib.bib12)\)have realized that these models can be used as powerful foundation models over, e\.g\., tabular data, though further recent work has also applied this method to other data modalities, such as imaging data\([Woerner et al\., 2026](https://arxiv.org/html/2610.40934#bib.bib11)\)\. Unlike other foundation model variants, the input for a PFN looks quite different from a traditional model that mapsxxtoyy\. In particular, PFNs explicitly take as input a sequence of sampled pairs\(x,y\)\(x,y\)that represent the task of interest at inference time, followed by a final, unpairedxxsample for which the target labelyyshould be inferred\. During pre\-training, these sampled sequences are generated from a defined “prior” over possible tasks, enabling many diverse relationships betweenxxandyyto be learned\. This makes PFNs extremely flexible, but also imposes computational resource limitations on when they can be used given a sequence of many pairs\(x,y\)\(x,y\)must fit on the GPU for a single sample\. While one may argue that this sequence of\(x,y\)\(x,y\)pairs used as input to the model violates the “any\-task” contract of interest here, we will permit this, and justify it more formally later in the work\.

#### Task\-conditioned or Multi\-task Foundation Models over an Explicit Task Space

Classically, multi\-task learning\([McDermott et al\., 2021](https://arxiv.org/html/2610.40934#bib.bib6)\)or even single\-task \(but typically multi\-class\) supervised pre\-training have been widely used in representation learning settings\([Deng et al\., 2009](https://arxiv.org/html/2610.40934#bib.bib19)\)\. More recently, many variants of multi\-task pre\-training algorithms have been presented as “foundation models”, in particular in health domains\([Steinberg et al\., 2024](https://arxiv.org/html/2610.40934#bib.bib21);[Jing et al\., 2026](https://arxiv.org/html/2610.40934#bib.bib5)\)\. A superficially related approach, in which task\-conditioned labels are produced, rather than producing labels for multiple tasks simultaneously, has also recently been proposed as an alternative to AR foundation models in the health domain\([Chandak et al\., 2026](https://arxiv.org/html/2610.40934#bib.bib1)\)\.

In multi\-task models, a model outputs a separate prediction for a finite set of tasks simultaneously\. In task\-conditioned models, the model takes as input a samplexxbut also some “representation of the task of interest” in some structured form, then outputs a prediction solely for that task and input\. In this way, it takes many inference steps for a task\-conditioned model to produce the same output as a multi\-task model, but the task\-conditioned model can model a much larger \(potentially infinite\) set of tasks simultaneously, in contrast to the finite set of tasks expressible by a multi\-task model\. These models realize “any\-task” capabilities over the structured space of tasks over which labels are produced\. If either this space is sufficiently broad or a procedure on how to adapt other tasks into this structured space of tasks is provided, it is reasonable to consider these models as “foundation models” in our more restricted sense \(though we note that often these models are presented in a manner requiring PT/FT or transfer learning\)\.

A summary of the key characteristics of these examples is provided in Section[5\.3](https://arxiv.org/html/2610.40934#S5.SS3)and Table[3](https://arxiv.org/html/2610.40934#S5.T3)\.

### 1\.4Outline of the Rest of the Work

Our notation uses precise conventions for typeface and notational elements in several places: for example, we write random variables in sans\-serif typeface and distinguish them from their values, write Markov kernels asρ:𝒯↝ℛ\\rho:\\mathcal\{T\}\\rightsquigarrow\\mathcal\{R\}, and use push\-forward distributionsf♯​qf\_\{\\sharp\}q\. These conventions, together with the standing assumptions we rely on, are collected in Appendix[A](https://arxiv.org/html/2610.40934#A1)\.We strongly recommend that readers review Appendix[A](https://arxiv.org/html/2610.40934#A1)before proceeding to the rest of the work, as the formal development begins there and uses this notation without further introduction\.

In the rest of this work, we will first formalize the notion of the “distribution of tasks” and of “task representations” in Section[2](https://arxiv.org/html/2610.40934#S2)\. Then, in Section[3](https://arxiv.org/html/2610.40934#S3), we will formally define ATI foundation models and demonstrate that this definition universally permits a decomposition through a “universal task reduction” lens\. Building on this definition, in Section[4](https://arxiv.org/html/2610.40934#S4), we will decompose the error of any given foundation model into meaningful terms dictated by the universal task decomposition\. In Section[5](https://arxiv.org/html/2610.40934#S5), we will further show that we can define two concrete recipes to build foundation models, building on the same basic assumptions and describe how they translate error across these different error sources\. Finally, in Section[6](https://arxiv.org/html/2610.40934#S6), we will discuss evaluating foundation models, and show that it is mathematically equivalent to evaluation of supervised models under fairness constraints\.

## 2Formalizing Tasks in Our Notation and Problem Formulation

A key ingredient of our eventual ATI foundation model definition will be the elevation of a “task” to a part of the input to the procedure, and thus to the probabilistic formulation of the problem\. In this section, we provide the necessary mathematical and notational set\-up to do this formally\.

### 2\.1The Extended Task\-Data Domain

Let𝒳\\mathcal\{X\}be some data domain\. Let𝒴\\mathcal\{Y\}be a label domain, such as\{0,1\}\\\{0,1\\\}for binary classification, though𝒴\\mathcal\{Y\}need not be discrete\. Typically, in machine learning, we assume that we have random variables𝗑,𝗒\\mathsf\{x\},\\mathsf\{y\}over𝒳×𝒴\\mathcal\{X\}\\times\\mathcal\{Y\}that define our space of interest\. In this work, we argue this view is insufficient\. As was first introduced \(to the best of our knowledge\) by[Baxter](https://arxiv.org/html/2610.40934#bib.bib15), we argue that we should also include a notion of a standard Borel space𝒯\\mathcal\{T\}of “tasks” requiring us to predict𝒴\\mathcal\{Y\}from𝒳\\mathcal\{X\}333Note this relies on assuming that all tasks share the same label space – we will revisit this later but it is a more comfortable assumption than it seems at first glance\.\. This notion requires us to similarly extend our \(joint\) distribution to𝗑,𝗒,𝗍\\mathsf\{x\},\\mathsf\{y\},\\mathsf\{t\}over data inputs𝗑\\mathsf\{x\}, labels𝗒\\mathsf\{y\}and the*task to which that label corresponds*𝗍\\mathsf\{t\}\. We will explicitly assume that this joint random variable is well defined for any valid domain, and further that the random variable𝗍\\mathsf\{t\}always has full support over𝒯\\mathcal\{T\}\.

To make our notion of a taskt∈𝒯t\\in\\mathcal\{T\}clear, we will identify a taskttwith its associated posterior distribution of labels given data by assuming that there exists a jointly measurable kernelη:𝒳×𝒯↝𝒴\\eta:\{\\mathcal\{X\}\\times\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{Y\}\}, such that𝗒\|𝗑=x,𝗍=t∼η\(⋅\|x,t\)\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\}=x,\\mathsf\{t\}=t\\right\.\\sim\\eta\(\\cdot\|x,t\)\. We will further assume that the mappingt↦ηt=η\(⋅\|⋅,t\)t\\mapsto\\eta\_\{t\}=\\eta\(\\cdot\|\\cdot,t\)is injective – i\.e\., each task defines a unique posterior distribution of𝗒t\|𝗑\\left\.\\mathsf\{y\}\_\{t\}\\middle\|\\mathsf\{x\}\\right\.\. Note that while, mathematically, we assert thatη\\etaexists, we are not claiming that it is ever known – indeed, ifη\\etawere known, we would not need to do any learning as it defines the true posterior distributions\.

#### Distribution Shift in the Task Space

Note that formalizing this space of tasks𝒯\\mathcal\{T\}and distribution over tasks𝗍\\mathsf\{t\}implies we may observe distribution shift over tasks between training and evaluation\. This is an important area of study, but not, we argue, essential to “any\-task inference” capabilities or to “foundation models,” and we leave it as an area of future work\.

#### Existence of Trivial Task Distributions

Note that we are not claiming that there is only “one” true task space𝒯\\mathcal\{T\}or task distribution𝗍\\mathsf\{t\}\. Instead, just as we have many data domains𝒳\\mathcal\{X\}or data distributions, we can have many task domains or distributions, and define new ones to suit certain applications or contexts\. This does mean that we can define some “trivial” task spaces – e\.g\., a single element task distribution𝒯=\{t\}\\mathcal\{T\}=\\\{t\\\}\. This is valid in much the same way that it is valid to define my data domain as a single point𝒳=\{x\}\\mathcal\{X\}=\\\{x\\\}\. It is technically true, but neither interesting nor useful\. Given this flexibility, specifying𝒯\\mathcal\{T\}is therefore an important part of specifying the domain of interest for any claims made about associated foundation models\.

#### Single\-task Models in this Extended Domain

Note that this formulation does not preclude us from studying single\-task models for a given taskt∈𝒯t\\in\\mathcal\{T\}\. In this setting, we are interested solely in modeling𝗑t,𝗒t\\mathsf\{x\}\_\{t\},\\mathsf\{y\}\_\{t\}\(in either a generative or discriminative𝗒t\|𝗑\\left\.\\mathsf\{y\}\_\{t\}\\middle\|\\mathsf\{x\}\\right\.setting\)\. In such a setting, we can safely discardttentirely, define our “effective” variables𝗑,𝗒\\mathsf\{x\},\\mathsf\{y\}to be given by𝗑t,𝗒t\\mathsf\{x\}\_\{t\},\\mathsf\{y\}\_\{t\}, and model as normal without issue\.

#### Any\-task Inference Capabilities in this Extended Domain

If we are not interested in a single\-task model, however, and instead interested in models that display “any\-task inference capabilities,” then this extended domain is precisely the machinery we need to formalize this capability\. In particular, for a model to support any\-task inference over𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}it is clear that this model must permit, through some procedural form, one to map*any*pair\(x,t\)∈𝒳×𝒯\(x,t\)\\in\\mathcal\{X\}\\times\\mathcal\{T\}to some prediction distribution over𝒴\\mathcal\{Y\}– e\.g\., the model must be usable through a procedure that lives in𝒫​\(𝒴\)\(𝒳×𝒯\)\{\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)\}^\{\(\\mathcal\{X\}\\times\\mathcal\{T\}\)\}\.

Critically, note here that we are not asserting anything about the model’s actual performance\. It may, for example, yield terrible performance over every taskt∈𝒯t\\in\\mathcal\{T\}, or be constant in its predictions across allt∈𝒯t\\in\\mathcal\{T\}, and that need not violate its ability to perform “any\-task inference\.” Instead, the claim we make here is operational – if someone describes their model as being able to make inference over any task within this extended domain, then there must be a mechanism by which you can perform inference for any taskt∈𝒯t\\in\\mathcal\{T\}, which we can formalize in this functional form\.

Note that we are also not requiring that𝒯\\mathcal\{T\}be directly observable for a model to support any\-task inference – instead, the only requirement is that if you have access to a taskt∈𝒯t\\in\\mathcal\{T\}or some representation of such a task, you must somehow be able to use the model to make inference for that tasktt\. Note that this definition shares many similarities with the informal definition of “zero\-shot” learning; for more commentary on why we use “any\-task inference” vs “zero\-shot learning” here, see Appendix[B](https://arxiv.org/html/2610.40934#A2)\.

### 2\.2Task Representations

In general,𝒯\\mathcal\{T\}will be an unobserved, latent space which is not represented in a tractable format\. This will make it challenging to operationalize how we will realize these “procedures” defined on𝒳×𝒯\\mathcal\{X\}\\times\\mathcal\{T\}referenced above\. Instead, in practice, where “tasks” are used today in an operational manner, we instead must “represent” a taskt∈𝒯t\\in\\mathcal\{T\}in some more tractable manner\. We formalize this through the notion of a Markov Kernel \(i\.e\., a stochastic function\) from𝒯\\mathcal\{T\}into a spaceℛ\\mathcal\{R\}viaρ:𝒯↝ℛ\\rho:\{\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{R\}\}\. As this kernel defines a probability distribution for eacht∈𝒯t\\in\\mathcal\{T\}, we can introduce the notion of a random variable𝗋\\mathsf\{r\}, jointly distributed with𝗑,𝗒,𝗍\\mathsf\{x\},\\mathsf\{y\},\\mathsf\{t\}such that the conditional random variable𝗋t=𝗋\|𝗍=t\\mathsf\{r\}\_\{t\}=\\left\.\\mathsf\{r\}\\middle\|\\mathsf\{t\}=t\\right\.is given precisely by the action ofρ\\rhoontt\. If we wish forρ\\rhoto be given by a deterministic functiong:𝒯→ℛg:\\mathcal\{T\}\\to\\mathcal\{R\}, we can represent that with a point mass distribution, such thatρ⁡\(t\)=δg⁡\(t\)\\rho\(t\)=\\delta\_\{g\(t\)\}\. In this case, we may \(at a slight abuse in notation\) useρ⁡\(t\)=g⁡\(t\)\\rho\(t\)=g\(t\)as well when this distinction is clear from context\. Note that even in the case thatρ\\rhois deterministic,𝗋\\mathsf\{r\}may still be stochastic given the stochasticity of𝗍\\mathsf\{t\}\. Below, we list some general properties and/or definitional requirements of task representations \(all of which are assumed to be held in the context of a joint domain over𝗑,𝗍,𝗒\\mathsf\{x\},\\mathsf\{t\},\\mathsf\{y\}\)\.

Given a representationρ:𝒯↝ℛ\\rho:\{\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{R\}\}, we can also define several induced kernels that will help with further results\. In particular, Definition[2\.1](https://arxiv.org/html/2610.40934#S2.Thmtheorem1)definesκρ:ℛ↝𝒯\\kappa\_\{\\rho\}:\{\\mathcal\{R\}\}\\rightsquigarrow\{\\mathcal\{T\}\}to be the posterior kernel of tasks given representations and Definition[2\.2](https://arxiv.org/html/2610.40934#S2.Thmtheorem2)defines the kernelϕρ:𝒳×ℛ↝𝒴\\phi\_\{\\rho\}:\{\\mathcal\{X\}\\times\\mathcal\{R\}\}\\rightsquigarrow\{\\mathcal\{Y\}\}to be the distribution of labels conditioned on a data sample and a represented task \(much asη\\etais the kernel for true tasks\)\.

###### Definition 2\.1\.

Given extended domain𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\},ρ:𝒯↝ℛ\\rho:\{\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{R\}\}a representation of𝒯\\mathcal\{T\}, and associated joint random variables𝗑,𝗍,𝗒,𝗋\\mathsf\{x\},\\mathsf\{t\},\\mathsf\{y\},\\mathsf\{r\}, we can define the induced kernelκρ:ℛ↝𝒯\\kappa\_\{\\rho\}:\{\\mathcal\{R\}\}\\rightsquigarrow\{\\mathcal\{T\}\}to be the posterior distribution of𝗍\|𝗋\\mathsf\{t\}\|\\mathsf\{r\}\.

###### Definition 2\.2\.

Given extended domain𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\},ρ:𝒯↝ℛ\\rho:\{\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{R\}\}a representation of𝒯\\mathcal\{T\}, and associated joint random variables𝗑,𝗍,𝗒,𝗋\\mathsf\{x\},\\mathsf\{t\},\\mathsf\{y\},\\mathsf\{r\}, we can define the induced kernelϕρ:𝒳×ℛ↝𝒴\\phi\_\{\\rho\}:\{\\mathcal\{X\}\\times\\mathcal\{R\}\}\\rightsquigarrow\{\\mathcal\{Y\}\}to be the induced distribution of𝗒\|𝗑,𝗋\\mathsf\{y\}\|\\mathsf\{x\},\\mathsf\{r\}\.

To maximize the utility of these task representations, there are some basic properties, assumptions, and definitions we will introduce\. Concretely, we take as a definitional assumption that𝗋⟂𝗑,𝗒\|𝗍\\mathsf\{r\}\\perp\\mathsf\{x\},\\mathsf\{y\}\|\\mathsf\{t\}\. Similarly, if we have two distinct representationsρ1\\rho\_\{1\}andρ2\\rho\_\{2\}, then we will likewise assume that𝗋1⟂𝗋2\|𝗍\\mathsf\{r\}\_\{1\}\\perp\\mathsf\{r\}\_\{2\}\|\\mathsf\{t\}\. Given two such representations, we may also define a composite representationρ=ρ1⊗ρ2\\rho=\\rho\_\{1\}\\otimes\\rho\_\{2\}such thatρ:𝒯↝ℛ1×ℛ2\\rho:\{\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{R\}\_\{1\}\\times\\mathcal\{R\}\_\{2\}\}yields a random variable𝗋\\mathsf\{r\}taking values\(r1,r2\)\(r\_\{1\},r\_\{2\}\)such that each component of𝗋t\\mathsf\{r\}\_\{t\}is independent of one another and defined byρ1\(⋅\|t\)\\rho\_\{1\}\(\\cdot\|t\)andρ2\(⋅\|t\)\\rho\_\{2\}\(\\cdot\|t\)respectively\. Additionally, some representationsρ\\rhomay be “degenerate” in the sense that they may offer no information about the task in question, such that𝗋⟂𝗍\\mathsf\{r\}\\perp\\mathsf\{t\}\. For exampleρ⁡\(t\)=δ0\\rho\(t\)=\\delta\_\{0\}\(ρ\\rhomaps all tasks to a point\-mass at00\) is degenerate\.

Conversely, a representationρ\\rhowill be called “complete” if it contains all the information about the task\. Concretely,ρ\\rhois complete ifκρ\(⋅\|𝗋\)=δ𝗍\\kappa\_\{\\rho\}\(\\cdot\|\\mathsf\{r\}\)=\\delta\_\{\\mathsf\{t\}\}almost everywhere\. For example,t↦δtt\\mapsto\\delta\_\{t\}is complete\.

A graphical model of the relationship between data, tasks, labels, and representations can be shown in Figure[1](https://arxiv.org/html/2610.40934#S2.F1)\.

𝗑\\mathsf\{x\}𝗍\\mathsf\{t\}𝗒\\mathsf\{y\}𝗋\\mathsf\{r\}ρ\\rhoFigure 1:A graphical model of data samples𝗑\\mathsf\{x\}, tasks𝗍\\mathsf\{t\}, labels𝗒\\mathsf\{y\}, and representations of tasks induced byρ\\rho,𝗋\\mathsf\{r\}\.#### Examples

While it may seem counterintuitive at first, all task spaces permit tractable representations through a variety of means\. Concretely, as we will explore in more depth below, each of our running examples leverages distinct tractable representations of tasks to achieve their foundation model behaviors\. In particular, LLMs represent tasks via free\-text prompts and natural language questions\. AR foundation models for EHR data represent tasks in the form of structured programs they use to extract labels from generated trajectories during inference\. PFNs represent a taskttstochastically via samples \(assumed to be iid\) drawn from the distribution of𝗑,𝗒\|𝗍=t\\mathsf\{x\},\\mathsf\{y\}\|\\mathsf\{t\}=t, which are directly passed to models as a sequence of\(x,y\)\(x,y\)pairs\. Task\-conditioned, direct prediction foundation models like EveryQuery\([Chandak et al\., 2026](https://arxiv.org/html/2610.40934#bib.bib1)\)directly represent tasks via structured programs \(much like AR EHR FMs use\), and, finally, multi\-task foundation models consider only a discrete space of tasks and represent them via categorical indices\.

### 2\.3Irreducible Error due to Task Representation Ambiguity

It is important to note thatρ\\rhowill often produce “lossy” representations of tasks\. In particular, it is clear by the data processing inequality and the fact that𝗑,𝗒→𝗍→𝗋\\mathsf\{x\},\\mathsf\{y\}\\to\\mathsf\{t\}\\to\\mathsf\{r\}is a Markov chain, thenI⁡\(𝗑,𝗒,𝗍\)≥I⁡\(𝗑,𝗒,𝗋\)I\(\\mathsf\{x\},\\mathsf\{y\};\\mathsf\{t\}\)\\geq I\(\\mathsf\{x\},\\mathsf\{y\};\\mathsf\{r\}\)\. In other words, a representation of a task has less mutual information with the joint distribution of𝗑,𝗒\\mathsf\{x\},\\mathsf\{y\}than does the true task\.

We can generalize this notion slightly to apply to arbitrary generalized entropies and divergences by introducing the strictly proper scoring lossℓ\\ellover𝒴\\mathcal\{Y\}and the associated generalized entropies introduced in Appendix[A](https://arxiv.org/html/2610.40934#A1), through Definition[2\.3](https://arxiv.org/html/2610.40934#S2.Thmtheorem3)\.

###### Definition 2\.3\.

Givenℓ\\ell,ρ\\rhoas defined above, letΔρ\\Delta\_\{\\rho\}\(implicitly dependent onℓ\\ell, which we will assume is specified in context\) to be the increase in the generalizedℓ\\ell\-entropyhℓh\_\{\\ell\}of𝗒\|𝗑\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\}\\right\.when conditioning on𝗋\\mathsf\{r\}rather than𝗍\\mathsf\{t\}:

Δρ\\displaystyle\\Delta\_\{\\rho\}=hℓ\(𝗒\|𝗑,𝗋\)−hℓ\(𝗒\|𝗑,𝗍\)\\displaystyle=h\_\{\\ell\}\(\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.\)\-h\_\{\\ell\}\(\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\},\\mathsf\{t\}\\right\.\)=hℓ\(𝗒\|𝗑,𝗋\)−hℓ\(𝗒\|𝗑,𝗍,𝗋\)\\displaystyle=h\_\{\\ell\}\(\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.\)\-h\_\{\\ell\}\(\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\},\\mathsf\{t\},\\mathsf\{r\}\\right\.\)=Iℓ\(𝗒;𝗍\|𝗑,𝗋\)\\displaystyle=I\_\{\\ell\}\(\\left\.\\mathsf\{y\};\\mathsf\{t\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.\)≥0\\displaystyle\\geq 0

Definition[2\.3](https://arxiv.org/html/2610.40934#S2.Thmtheorem3)shows thatΔρ\\Delta\_\{\\rho\}is precisely \(1\) the amount of information that remains between𝗒\\mathsf\{y\}and𝗍\\mathsf\{t\}when conditioning on𝗋\\mathsf\{r\}and𝗑\\mathsf\{x\}, and \(2\) the increase in Bayes\-optimal error were we forced to predict𝗒\\mathsf\{y\}from𝗑\\mathsf\{x\}and𝗋\\mathsf\{r\}alone vs\. if we could use the true𝗍\\mathsf\{t\}directly\.

Note that we can see that ifρ\\rhois complete, thenhℓ\(𝗒\|𝗑,𝗋\)=hℓ\(𝗒\|𝗑,𝗍\)h\_\{\\ell\}\(\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.\)=h\_\{\\ell\}\(\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\},\\mathsf\{t\}\\right\.\), soΔρ=0\\Delta\_\{\\rho\}=0\. Conversely, ifρ\\rhois degenerate, thenΔρ=Iℓ​\(𝗒;𝗍\|𝗑\)\\Delta\_\{\\rho\}=I\_\{\\ell\}\(\\mathsf\{y\};\\mathsf\{t\}\|\\mathsf\{x\}\)and is maximized\. Additionally, suppose thatρ=ρ1⊗ρ2\\rho=\\rho\_\{1\}\\otimes\\rho\_\{2\}\. Then, by independence ofρ1\\rho\_\{1\}andρ2\\rho\_\{2\}and the construction of the composite representationρ\\rho, we have thatΔρ≤Δℓ​\(ρ1\)\\Delta\_\{\\rho\}\\leq\\Delta\_\{\\ell\}\(\\rho\_\{1\}\)andΔρ≤Δℓ​\(ρ2\)\\Delta\_\{\\rho\}\\leq\\Delta\_\{\\ell\}\(\\rho\_\{2\}\)\.

#### Example

Consider two tasks: predicting whether a patient will be readmitted within 7 days or within 30 days\. If both are represented simply as “predict readmission,” the time horizon is lost\. A readmission on day 20 would have different labels under the two tasks\. Thus, even a Bayes\-optimal predictor cannot fully resolve this ambiguity, resulting in irreducible representation errorΔρ\\Delta\_\{\\rho\}\.

## 3Defining Foundation Models

With this machinery in place, we can now define an ATI foundation model formally\. In fact, in this work, we will make the property of “any\-task inference capable” identical with being a “foundation model,” as this aligns precisely with the desiderata introduced in the Introduction and the spirit of maximally data\-efficient adaptability introduced by[Bommasani et al\.](https://arxiv.org/html/2610.40934#bib.bib13)We will first introduce a direct definition based on any\-task inference capability in Subsection[3\.1](https://arxiv.org/html/2610.40934#S3.SS1), then further show that we can equivalently define foundation models as reductions to a universal task and establish some basic properties of this definition in Subsection[3\.2](https://arxiv.org/html/2610.40934#S3.SS2)\.

### 3\.1Formal Definition of an ATI Foundation Model

We first define ATI foundation models formally in Definition[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1)\.

###### Definition 3\.1\(Foundation Model\)\.

Let𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}be an extended data, task, label domain of interest, satisfying the normal, previously mentioned assumptions\. We will define an ‘‘ATI foundation model’’ over this domain to be a procedure444Note that we use the term “procedure” rather than “function” or “model” here simply to reflect that this may, operationally, involve classical, algorithmic and modeled components, not to differentiate this mathematically from any other ordinary function\.𝙿:𝒳×𝒯→𝒫⁡\(𝒴\)\\mathtt\{P\}:\\mathcal\{X\}\\times\\mathcal\{T\}\\to\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)\. Implicitly, if one is interested in an ATI foundation model over this domain, but is only permitted to access tasks through a single\-draw of a representationρ:𝒯↝ℛ\\rho:\{\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{R\}\}, then we should understand this as a procedure over𝒳×ℛ\\mathcal\{X\}\\times\\mathcal\{R\}instead of𝒳×𝒯\\mathcal\{X\}\\times\\mathcal\{T\}to reflect this single\-draw requirement\.

Much as we said for “any\-task inference capability,” Definition[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1)does not assert a specific performance guarantee, nor does it require generalization outside of the specified domain𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}\. It does, however, assert that there must be an operational manner by which one can map an inputxxand a given taskttto a predicted distribution over𝒴\\mathcal\{Y\}\. Note that we also do not assert that the foundation model will be able to access the raw tasktt– instead, as we will introduce later, we will assume there is a specific representation of the task that the foundation model can leverage, which will factor into eventual error decompositions\.

While Definition[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1)may seem weak, we will see in this section that it permits meaningful theoretical and practical results\. Concretely, in what remains of this section we will first remark that this definition trivially demonstrates what the “Bayes\-optimal” foundation model is over this domain, and the implications of this, then we will show that all foundation models under this definition can be decomposed into terms under a “universal\-task reduction” lens\.

#### Bayes\-optimal Foundation Models

It is clear from Definition[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1)that over a given domain with joint random variables𝗑,𝗍,𝗒\\mathsf\{x\},\\mathsf\{t\},\\mathsf\{y\}, the Bayes\-optimal foundation model𝙿∗\\mathtt\{P\}^\{\*\}under any strictly proper scoring lossℓ\\ellis given by the true posterior of𝗒\\mathsf\{y\}given𝗑,𝗍\\mathsf\{x\},\\mathsf\{t\}:

𝙿∗:\(x,t\)↦η⁡\(x,t\)\.\\mathtt\{P\}^\{\*\}:\(x,t\)\\mapsto\\eta\(x,t\)\.
We can also use this to see, near trivially, that the best possible foundation model cannot outperform the best possible single\-task model for a taskt∈𝒯t\\in\\mathcal\{T\}, both in general and when under matched model\-permissibility constraints, thereby formally clarifying that foundation model improvements over supervised models must come from other mechanisms \(Appendix[C](https://arxiv.org/html/2610.40934#A3)\)\.

#### Foundation Models over Represented Task Spaces

Definition[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1)defines a foundation model over an extended𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}domain and over the𝒳,ℛ,𝒴\\mathcal\{X\},\\mathcal\{R\},\\mathcal\{Y\}space whereℛ\\mathcal\{R\}is the representation space of some representationρ:𝒯↝ℛ\\rho:\{\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{R\}\}in the case that the user is only allowed to access a single\-draw ofρ\\rhofor any taskt∈𝒯t\\in\\mathcal\{T\}\. Note that even in this single\-draw case, we still consider the foundation model to be “over the𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}domain” not for the purposes of its definition/API, but instead for the purposes of optimality, as were the user to use a different representationρ′:𝒯↝ℛ′\\rho^\{\\prime\}:\{\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{R\}^\{\\prime\}\}for the root space of𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}, they might be able to obtain a “better” model than is obtainable underρ\\rhoin a manner that is lost if we consider optimality to be in the𝒳,ℛ,𝒴\\mathcal\{X\},\\mathcal\{R\},\\mathcal\{Y\}space\. We will show later that the cost of a given choiceρ\\rhoin this single\-draw case is precisely the irreducible representation errorΔρ\\Delta\_\{\\rho\}as defined in Definition[2\.3](https://arxiv.org/html/2610.40934#S2.Thmtheorem3)\.

#### Example

A multi\-task model predicting a fixed set of clinical outcomes satisfies Definition[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1)when𝒯\\mathcal\{T\}is restricted to those outcomes: given a task index, the corresponding prediction can be returned without retraining\. However, this guarantees no inference capability beyond the specified task space, emphasizing that the expressiveness of𝒯\\mathcal\{T\}is central to interpreting foundation model claims\.

### 3\.2Foundation Models as Universal Task Reductions

Building on only Definition[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1), we can decompose any foundation model into a procedure that leverages a form of*reduction*in the space of tasks to achieve any\-task inference capabilities\. Concretely, we will break a foundation model𝙿\\mathtt\{P\}down into an “encoder”, a “universal task” and associated predictive “model”, and a “decoder”, such that the encoder, limited by an encoder\-specific task representation, converts inputs to𝙿\\mathtt\{P\}into universal task model inputs, the predictive model attempts to solve the universal task over the modified inputs, and the decoder \(similarly limited by a decoder\-specific task representation\) converts predictions for the universal task into predictions over the true labels𝒴\\mathcal\{Y\}\.

To define this decomposition formally, we introduce the following variables, introduced in Notation[1](https://arxiv.org/html/2610.40934#Thmnotation1)\.

###### Notation 1\.

Let𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}be the normal extended domain, with𝗑,𝗍,𝗒\\mathsf\{x\},\\mathsf\{t\},\\mathsf\{y\}the normal jointly distributed random variables\. Then, define the following terminology and notational elements:

Encoder and Decoder Task RepresentationsLetℛE\\mathcal\{R\}\_\{E\}andℛD\\mathcal\{R\}\_\{D\}be encoder and decoder specific task representation spaces, withρE:𝒯↝ℛE\\rho\_\{E\}:\{\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{R\}\_\{E\}\}andρD:𝒯↝ℛD\\rho\_\{D\}:\{\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{R\}\_\{D\}\}defining encoder and decoder representations, and𝗋𝙴\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\}and𝗋𝙳\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}the induced encoder/decoder representation random variables, respectively\.

Global RepresentationDefineρ=ρE⊗ρD\\rho=\\rho\_\{E\}\\otimes\\rho\_\{D\}, with𝗋\\mathsf\{r\}the induced random variable\.

Universal\-task DomainLet𝒱\\mathcal\{V\},𝒵\\mathcal\{Z\}be alternative, standard\-Borel spaces\. Intuitively,𝒱\\mathcal\{V\}and𝒵\\mathcal\{Z\}define the input and output spaces of the universal prediction problem to which the original task is reduced: the encoder maps the original input and task representation into𝒱\\mathcal\{V\}, and predictions in𝒵\\mathcal\{Z\}can subsequently be transformed by the decoder into predictions over the original label space𝒴\\mathcal\{Y\}\.

EncoderLet𝙴:𝒳×ℛE→𝒱\\mathtt\{E\}:\\mathcal\{X\}\\times\\mathcal\{R\}\_\{E\}\\to\\mathcal\{V\}\. In the context of a known encoder𝙴\\mathtt\{E\}, we can introduce a random variable𝗏\\mathsf\{v\}defined as𝙴⁡\(𝗑,𝗋𝙴\)\\mathtt\{E\}\(\\mathsf\{x\},\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\}\), jointly distributed with all others\.

Universal TaskLetuudefine a probability distribution for a “universal task” predicting elements from𝒵\\mathcal\{Z\}given inputs from𝒱\\mathcal\{V\}, such thatu:𝒱↝𝒵u:\{\\mathcal\{V\}\}\\rightsquigarrow\{\\mathcal\{Z\}\}\. In the context of a defined taskuuand encoder𝙴\\mathtt\{E\}, we will let the random variable𝗓\\mathsf\{z\}denote the random variable induced fromu⁡\(𝗏\)u\(\\mathsf\{v\}\), jointly distributed and independent of all others given𝗏\\mathsf\{v\}\.

LabelerDefine a per\-task deterministic labelerd:ℛD×𝒵→𝒴d:\\mathcal\{R\}\_\{D\}\\times\\mathcal\{Z\}\\to\\mathcal\{Y\}, a measurable function that maps a universal\-task outputz∈𝒵z\\in\\mathcal\{Z\}to a downstream label for any represented taskr𝙳∈ℛD\{r\_\{\\mathtt\{D\}\}\}\\in\\mathcal\{R\}\_\{D\}\. We writedr𝙳​\(⋅\)=d⁡\(r𝙳,⋅\)d\_\{\{r\_\{\\mathtt\{D\}\}\}\}\(\\cdot\)=d\(\{r\_\{\\mathtt\{D\}\}\},\\cdot\)\.

ModelLetf:𝒱→𝒫⁡\(𝒵\)f:\\mathcal\{V\}\\to\\mathcal\{P\}\\left\(\\mathcal\{Z\}\\right\)\. In the context of a known𝙴\\mathtt\{E\}we can define a random variable induced byffas𝗓^\\hat\{\\mathsf\{z\}\}sampled viaf⁡\(𝗏\)f\(\\mathsf\{v\}\), jointly distributed and independent with all others given𝗏\\mathsf\{v\}\.

DecoderLet𝙳:𝒫⁡\(𝒵\)×ℛD→𝒫⁡\(𝒴\)\\mathtt\{D\}:\\mathcal\{P\}\\left\(\\mathcal\{Z\}\\right\)\\times\\mathcal\{R\}\_\{D\}\\to\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)be the realized decoder\. It is intended to approximate the*target decoder*𝙳⋆:𝒫⁡\(𝒵\)×ℛD→𝒫⁡\(𝒴\)\\mathtt\{D\}^\{\\star\}:\\mathcal\{P\}\\left\(\\mathcal\{Z\}\\right\)\\times\\mathcal\{R\}\_\{D\}\\to\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)induced by the labeler,

𝙳⋆​\(q,r𝙳\):=dr𝙳♯​q,\\mathtt\{D\}^\{\\star\}\(q,\{r\_\{\\mathtt\{D\}\}\}\):=\{d\_\{\{r\_\{\\mathtt\{D\}\}\}\}\}\_\{\\sharp\}\{q\},i\.e\., the distribution ofdr𝙳​\(z\)d\_\{\{r\_\{\\mathtt\{D\}\}\}\}\(z\)forz∼qz\\sim q\.

Unlike the modelffand decoder𝙳\\mathtt\{D\}, which can be compared respectively to the target universal taskuuand the stochastic labeling operation induced bydd, we do not posit a target encoder\. Rather, the choice of encoderEEis part of the specification of the universal\-task reduction itself and therefore jointly determines, together withuu, the universal prediction problem\.

With these introduced, we can formally state and prove Theorem[3\.2](https://arxiv.org/html/2610.40934#S3.Thmtheorem2), which informally states that all foundation models permit a reduction through this universal task lens\.

###### Theorem 3\.2\(All Foundation Models are Universal Task Reductions\)\.

Let𝙿\\mathtt\{P\}be a foundation model over an extended domain𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}\. Then there exists a representation, encoder, model, and decoder \(in the sense of the previously introduced terminology\), such that almost surely

𝙿⁡\(x,𝗍\)=𝙳⁡\(f⁡\(𝙴⁡\(x,𝗋𝙴\)\),𝗋𝙳\)\.\\mathtt\{P\}\(x,\\mathsf\{t\}\)=\\mathtt\{D\}\(f\(\\mathtt\{E\}\(x,\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\}\)\),\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\)\.

###### Proof\.

We can realize this trivially by construction\. Fort∈𝒯t\\in\\mathcal\{T\}, let the encoder representationρE​\(t\)=δt\\rho\_\{E\}\(t\)=\\delta\_\{t\}be complete, let𝒱=𝒳×𝒯\\mathcal\{V\}=\\mathcal\{X\}\\times\\mathcal\{T\},𝒵=𝒴\\mathcal\{Z\}=\\mathcal\{Y\},𝙴=I\\mathtt\{E\}=Ibe the identity,f=𝙿f=\\mathtt\{P\}directly, and let𝙳⁡\(q,⋅\)=q\\mathtt\{D\}\(q,\\cdot\)=qreturn the first argument only\. Then this equality holds almost surely\. ∎

Note that Theorem[3\.2](https://arxiv.org/html/2610.40934#S3.Thmtheorem2)does not suggest that all foundation models*uniquely*decompose among these components – merely that there exists*a*decomposition of this form\. Additionally, note that Theorem[3\.2](https://arxiv.org/html/2610.40934#S3.Thmtheorem2)leverages a consistent, deterministic representationρ\\rho, and thus does not permit us to say that we can decompose a generic foundation model over𝒯\\mathcal\{T\}in a manner leveraging an inconsistent or stochastic representation\. However, in such settings, if we assume the user is only permitted a single draw from the given representationρ\\rho\(which we generally will\), then we really are operating over the induced domain of𝒳,ℛ,𝒴\\mathcal\{X\},\\mathcal\{R\},\\mathcal\{Y\}\(rather than𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}\) with a consistent, deterministic representation without loss of generality\. Finally, and most critically, note that the construction used in Theorem[3\.2](https://arxiv.org/html/2610.40934#S3.Thmtheorem2)is trivial, and does not assign any special properties to any components in a particular sense\. However, it does show that we can, without loss of generality almost surely, use this decomposed view of a foundation model as a general definition\. In particular, we can use Definition[3\.3](https://arxiv.org/html/2610.40934#S3.Thmtheorem3)instead of Definition[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1)as our working definition of ATI foundation models, and conclude by Theorem[3\.2](https://arxiv.org/html/2610.40934#S3.Thmtheorem2)that this is almost surely without loss of generality\.

###### Definition 3\.3\(Foundation Models as Reductions\)\.

Defineρ\\rho,𝙴\\mathtt\{E\},ff,𝙳\\mathtt\{D\}in the prior terminology block\. Then a foundation model𝙿:𝒳×𝒯→𝒫⁡\(𝒴\)\\mathtt\{P\}:\\mathcal\{X\}\\times\\mathcal\{T\}\\to\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)can be defined as the tuple\(ρ,𝙴,f,𝙳\)\(\\rho,\\mathtt\{E\},f,\\mathtt\{D\}\), realizing the function

𝙿⁡\(x,𝗍\)=𝙳⁡\(f⁡\(𝙴⁡\(x,𝗋𝙴\)\),𝗋𝙳\)\.\\mathtt\{P\}\(x,\\mathsf\{t\}\)=\\mathtt\{D\}\(f\(\\mathtt\{E\}\(x,\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\}\)\),\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\)\.

#### Example

A nontrivial instance of Theorem[3\.2](https://arxiv.org/html/2610.40934#S3.Thmtheorem2)and Definition[3\.3](https://arxiv.org/html/2610.40934#S3.Thmtheorem3)is given by autoregressive foundation models for longitudinal EHR data\([Renc et al\., 2024](https://arxiv.org/html/2610.40934#bib.bib23)\)\. Here, the model need not receive the downstream task as input\. Instead, we may takeρE\\rho\_\{E\}to be degenerate and let𝙴⁡\(x,⋅\)=x\\mathtt\{E\}\(x,\\cdot\)=x, while choosing the universal taskuuto be prediction of a future patient trajectory𝗓\\mathsf\{z\}given the observed history𝗑\\mathsf\{x\}\. The learned modelffapproximates this trajectory distribution\. A downstream tasktt, such as predicting whether a particular clinical event occurs within a specified time horizon, is represented on the decoder side throughρD\\rho\_\{D\}, and𝙳\\mathtt\{D\}maps the predicted trajectory distribution to a distribution over the corresponding task label\. Thus, many distinct task\-specific predictions are obtained by reducing them to the same universal task of forecasting future patient trajectories\.

#### Optimality Relative To a Target Universal TaskuuAnd Labelerdd

Note that thus far, we have not explicitly used the universal taskuuor the labelerdd\. To incorporate these into our definitions, we will define the notions ofuuordd\-optimal foundation models in Definition[3\.4](https://arxiv.org/html/2610.40934#S3.Thmtheorem4)\. Informally, a foundation model isuu\-optimal ifffperfectly reflects the distributionuu\(e\.g\.,f⁡\(v\)=u⁡\(v\)f\(v\)=u\(v\)\)\. It isdd\-optimal if the realized decoder coincides with the target decoder,𝙳=𝙳⋆\\mathtt\{D\}=\\mathtt\{D\}^\{\\star\}, i\.e\.,𝙳⁡\(q,r𝙳\)=dr𝙳♯​q\\mathtt\{D\}\(q,\{r\_\{\\mathtt\{D\}\}\}\)=\{d\_\{\{r\_\{\\mathtt\{D\}\}\}\}\}\_\{\\sharp\}\{q\}\.

###### Definition 3\.4\(Foundation Model Optimality\)\.

Let\(ρ,𝙴,f,𝙳\)\(\\rho,\\mathtt\{E\},f,\\mathtt\{D\}\)define a foundation model𝙿\\mathtt\{P\}, withuu,ddand the induced target decoder𝙳⋆\\mathtt\{D\}^\{\\star\}as defined previously\. Then𝙿\\mathtt\{P\}is*uu\-optimal*iff=uf=u\(ifffperfectly models the universal task\) and*dd\-optimal*if𝙳⁡\(q,r𝙳\)=𝙳⋆​\(q,r𝙳\)\\mathtt\{D\}\(q,\{r\_\{\\mathtt\{D\}\}\}\)=\\mathtt\{D\}^\{\\star\}\(q,\{r\_\{\\mathtt\{D\}\}\}\)for all\(q,r𝙳\)\(q,\{r\_\{\\mathtt\{D\}\}\}\)\(if𝙳\\mathtt\{D\}exactly implements the labelerdd\)\.

In general,uuanddd\-optimality are only meaningful for a pre\-specifieduuordd; e\.g\., the construction used in Theorem[3\.2](https://arxiv.org/html/2610.40934#S3.Thmtheorem2)actually establishesuu\-optimality foruubeing the kernel for𝙿⁡\(𝗑,𝗍\)\\mathtt\{P\}\(\\mathsf\{x\},\\mathsf\{t\}\)anddd\-optimality withdr𝙳​\(z\)=zd\_\{\{r\_\{\\mathtt\{D\}\}\}\}\(z\)=z\(the identity, since𝒵=𝒴\\mathcal\{Z\}=\\mathcal\{Y\}\), thus showing that one can always choose auuandddto establish optimality\. However, this will not imply that for anyuuanddd, a given foundation model is optimal\.

#### Notational Elements for the Decomposed Foundation Model

In addition to the variables introduced in Notation[1](https://arxiv.org/html/2610.40934#Thmnotation1), now that we have introduceduuandddoptimality, it makes sense to introduce explicit random variables for the various combinations of the components of𝙿\\mathtt\{P\}under Definition[3\.3](https://arxiv.org/html/2610.40934#S3.Thmtheorem3), as they will ultimately help us describe𝙿\\mathtt\{P\}in more precise ways\. These elements are defined in Notation[2](https://arxiv.org/html/2610.40934#Thmnotation2)and are represented as an augmented graphical model in Figure[2](https://arxiv.org/html/2610.40934#S3.F2)\. Tables[1](https://arxiv.org/html/2610.40934#S3.T1)and[2](https://arxiv.org/html/2610.40934#S3.T2)summarize all the introduced objects and how they relate to each other\.

###### Notation 2\.

Recall the notation introduced in Notation[1](https://arxiv.org/html/2610.40934#Thmnotation1)\. We further introduce the following notational conventions: \(1\) The use of a⋅^\\hat\{\\cdot\}will indicate a random variable that has been approximated viaffrather than the true universal taskuu– this is consistent with𝗓^\\hat\{\\mathsf\{z\}\}already introduced in Notation[1](https://arxiv.org/html/2610.40934#Thmnotation1)\. \(2\) The use of a⋅~\\tilde\{\\cdot\}will indicate a distribution being approximated through the action of the realized decoder𝙳\\mathtt\{D\}rather than the target decoder𝙳⋆\\mathtt\{D\}^\{\\star\}\.

Consistent with these, we have:

Input Variables𝗑,𝗍,𝗒,𝗋𝙴,𝗋𝙳\\mathsf\{x\},\\mathsf\{t\},\\mathsf\{y\},\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\},\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}as defined in Notation[1](https://arxiv.org/html/2610.40934#Thmnotation1)\.

Encoder ActionLet𝗏=𝙴⁡\(𝗑,𝗋𝙴\)\\mathsf\{v\}=\\mathtt\{E\}\(\\mathsf\{x\},\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\}\)be the random variable induced by𝙴\\mathtt\{E\}\.

Universal Task OutputsLet𝗓\\mathsf\{z\}be the true universal task label drawn fromu⁡\(𝗏\)u\(\\mathsf\{v\}\), and𝗓^\\hat\{\\mathsf\{z\}\}be the approximated universal task label drawn fromf⁡\(𝗏\)f\(\\mathsf\{v\}\)\. Auu\-optimal model has𝗓\|𝗏\\left\.\\mathsf\{z\}\\middle\|\\mathsf\{v\}\\right\.and𝗓^\|𝗏\\left\.\\hat\{\\mathsf\{z\}\}\\middle\|\\mathsf\{v\}\\right\.identically distributed\.

Decoded OutputsWe will let the “w” character represent the𝒴\\mathcal\{Y\}valued random variable obtained by applying the target decoder𝙳⋆\\mathtt\{D\}^\{\\star\}or the realized decoder𝙳\\mathtt\{D\}to a distribution over𝒵\\mathcal\{Z\}\. Concretely,

- •Let𝗐\\mathsf\{w\}represent the random variable obtained by applying the target decoder to the true universal\-task distribution: 𝗐∼𝙳⋆​\(u⁡\(𝗏\),𝗋𝙳\)\.\\mathsf\{w\}\\sim\\mathtt\{D\}^\{\\star\}\\\!\\left\(u\(\\mathsf\{v\}\),\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\\right\)\.
- •Let𝗐~\\tilde\{\\mathsf\{w\}\}represent the random variable obtained by applying the realized decoder to the true universal\-task distribution: 𝗐~∼𝙳⁡\(u⁡\(𝗏\),𝗋𝙳\)\.\\tilde\{\\mathsf\{w\}\}\\sim\\mathtt\{D\}\\\!\\left\(u\(\\mathsf\{v\}\),\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\\right\)\.
- •Let𝗐^\\hat\{\\mathsf\{w\}\}represent the random variable obtained by applying the target decoder to the model\-approximated universal\-task distribution: 𝗐^∼𝙳⋆​\(f⁡\(𝗏\),𝗋𝙳\)\.\\hat\{\\mathsf\{w\}\}\\sim\\mathtt\{D\}^\{\\star\}\\\!\\left\(f\(\\mathsf\{v\}\),\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\\right\)\.
- •Instead of using the awkward notation𝗐~^\\hat\{\\tilde\{\\mathsf\{w\}\}\}for our final term, we use𝗒^\\hat\{\\mathsf\{y\}\}to represent the true output of𝙿\\mathtt\{P\}, obtained by applying the realized decoder to the realized model output: 𝗒^∼𝙳⁡\(f⁡\(𝗏\),𝗋𝙳\)\.\\hat\{\\mathsf\{y\}\}\\sim\\mathtt\{D\}\\\!\\left\(f\(\\mathsf\{v\}\),\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\\right\)\.

Note that add\-optimal model has𝗐\|𝗑,𝗋\\left\.\\mathsf\{w\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.and𝗐~\|𝗑,𝗋\\left\.\\tilde\{\\mathsf\{w\}\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.identically distributed, and likewise𝗐^\|𝗑,𝗋\\left\.\\hat\{\\mathsf\{w\}\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.and𝗒^\|𝗑,𝗋\\left\.\\hat\{\\mathsf\{y\}\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.\. Ifffisuu\-optimal andDDisdd\-optimal, then

𝗐\|𝗑,𝗋=𝗐~\|𝗑,𝗋=𝗐^\|𝗑,𝗋=𝗒^\|𝗑,𝗋a\.s\.\\left\.\\mathsf\{w\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.=\\left\.\\tilde\{\\mathsf\{w\}\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.=\\left\.\\hat\{\\mathsf\{w\}\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.=\\left\.\\hat\{\\mathsf\{y\}\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.\\quad\\text\{a\.s\.\}
The conditional independence relationships between these random variables, and their generating kernels/relationships, are shown via the augmented graphical model in Figure[2](https://arxiv.org/html/2610.40934#S3.F2)\.

𝗑\\mathsf\{x\}𝗍\\mathsf\{t\}η\\eta𝗒\\mathsf\{y\}ρE\\rho\_\{E\}𝗋𝙴\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\}ρD\\rho\_\{D\}𝗋𝙳\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}𝙴\\mathtt\{E\}𝗏\\mathsf\{v\}uu𝗓\\mathsf\{z\}ff𝗓^\\hat\{\\mathsf\{z\}\}𝙳⋆\\mathtt\{D\}^\{\\star\}𝗐\\mathsf\{w\}𝙳\\mathtt\{D\}𝗐~\\tilde\{\\mathsf\{w\}\}𝙳⋆\\mathtt\{D\}^\{\\star\}𝗐^\\hat\{\\mathsf\{w\}\}𝙳\\mathtt\{D\}𝗒^\\hat\{\\mathsf\{y\}\}Figure 2:An augmented schematic graphical representation of the random variables and distribution\-level operations induced by different combinations of our foundation model components\. Blue circular nodes correspond to random variables and orange nodes correspond to stochastic or deterministic kernels or procedures\. Solid directed edges through kernels indicate sample\-level generation\. Dashed edges from universal\-task output random variables to𝙳⋆\\mathtt\{D\}^\{\\star\}or𝙳\\mathtt\{D\}indicate dependence on the corresponding conditional distribution rather than pointwise application to the realized random variable\. Each decoder \(target𝙳⋆\\mathtt\{D\}^\{\\star\}or realized𝙳\\mathtt\{D\}\) separately receives the decoder\-side task representation𝗋𝙳\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\. This figure should be interpreted as a probabilistic graphical representation rather than a causal DAG\.Table 1:Spaces used in the universal\-task reduction and their associated realizations and random variables\.RoleObjectDefinition / typeInduced object\(s\)InterpretationBase problem and reductionBaseTask kernelη\\etaη:𝒳×𝒯↝𝒴\\eta:\\mathcal\{X\}\\times\\mathcal\{T\}\\rightsquigarrow\\mathcal\{Y\}𝗒∣𝗑=x,𝗍=t∼η\(⋅∣x,t\)\\mathsf\{y\}\\mid\\mathsf\{x\}=x,\\mathsf\{t\}=t\\sim\\eta\(\\cdot\\mid x,t\)True task\-specific conditional distribution of labels\.ReductionEncoder task representationρE\\rho\_\{E\}ρE:𝒯↝ℛE\\rho\_\{E\}:\\mathcal\{T\}\\rightsquigarrow\\mathcal\{R\}\_\{E\}𝗋𝙴\|𝗍=t∼ρE​\(t\)\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\}\\mid\\mathsf\{t\}=t\\sim\\rho\_\{E\}\(t\)Represents the task for the purpose of constructing the universal\-task input\.ReductionDecoder task representationρD\\rho\_\{D\}ρD:𝒯↝ℛD\\rho\_\{D\}:\\mathcal\{T\}\\rightsquigarrow\\mathcal\{R\}\_\{D\}𝗋𝙳\|𝗍=t∼ρD​\(t\)\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\\mid\\mathsf\{t\}=t\\sim\\rho\_\{D\}\(t\)Represents the task for the purpose of interpreting universal\-task outputs in the original label space\.ReductionGlobal representationρ\\rhoρ=ρE⊗ρD:𝒯↝ℛE×ℛD\\rho=\\rho\_\{E\}\\otimes\\rho\_\{D\}:\\mathcal\{T\}\\rightsquigarrow\\mathcal\{R\}\_\{E\}\\times\\mathcal\{R\}\_\{D\}𝗋=\(𝗋𝙴,𝗋𝙳\)\\mathsf\{r\}=\(\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\},\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\)Joint representation containing the encoder\- and decoder\-side task representations\.ReductionEncoder𝙴\\mathtt\{E\}𝙴:𝒳×ℛE→𝒱\\mathtt\{E\}:\\mathcal\{X\}\\times\\mathcal\{R\}\_\{E\}\\to\\mathcal\{V\}𝗏=𝙴⁡\(𝗑,𝗋𝙴\)\\mathsf\{v\}=\\mathtt\{E\}\(\\mathsf\{x\},\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\}\)Maps the original input and encoder\-side task representation into the input space of the universal task\.Universal taskTargetUniversal taskuuu:𝒱↝𝒵u:\\mathcal\{V\}\\rightsquigarrow\\mathcal\{Z\}𝗓\|𝗏=v∼u⁡\(v\)\\mathsf\{z\}\\mid\\mathsf\{v\}=v\\sim u\(v\)Target conditional distribution defining the universal task\.RealizedModelfff:𝒱→𝒫⁡\(𝒵\)f:\\mathcal\{V\}\\to\\mathcal\{P\}\(\\mathcal\{Z\}\)𝗓^\|𝗏=v∼f⁡\(v\)\\hat\{\\mathsf\{z\}\}\\mid\\mathsf\{v\}=v\\sim f\(v\)Realized predictive model intended to approximate the universal taskuu\.Task\-specific labeling and decodingTargetLabelerddd:ℛD×𝒵→𝒴d:\\mathcal\{R\}\_\{D\}\\times\\mathcal\{Z\}\\to\\mathcal\{Y\}dr𝙳​\(z\)=d⁡\(r𝙳,z\)d\_\{\{r\_\{\\mathtt\{D\}\}\}\}\(z\)=d\(\{r\_\{\\mathtt\{D\}\}\},z\)
Task\-specific deterministic map from a universal\-task outcome to a downstream label\.TargetTarget decoder𝙳⋆\\mathtt\{D\}^\{\\star\}𝙳⋆:𝒫⁡\(𝒵\)×ℛD→𝒫⁡\(𝒴\)\\mathtt\{D\}^\{\\star\}:\\mathcal\{P\}\(\\mathcal\{Z\}\)\\times\\mathcal\{R\}\_\{D\}\\to\\mathcal\{P\}\(\\mathcal\{Y\}\),𝙳⋆​\(q,r𝙳\)=dr𝙳♯​q\\mathtt\{D\}^\{\\star\}\(q,\{r\_\{\\mathtt\{D\}\}\}\)=\{d\_\{\{r\_\{\\mathtt\{D\}\}\}\}\}\_\{\\sharp\}\{q\}𝗐∼𝙳⋆​\(u⁡\(𝗏\),𝗋𝙳\)\\mathsf\{w\}\\sim\\mathtt\{D\}^\{\\star\}\(u\(\\mathsf\{v\}\),\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\),𝗐^∼𝙳⋆​\(f⁡\(𝗏\),𝗋𝙳\)\\hat\{\\mathsf\{w\}\}\\sim\\mathtt\{D\}^\{\\star\}\(f\(\\mathsf\{v\}\),\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\)Decoding operation implied by the target labeler\.RealizedDecoder𝙳\\mathtt\{D\}𝙳:𝒫⁡\(𝒵\)×ℛD→𝒫⁡\(𝒴\)\\mathtt\{D\}:\\mathcal\{P\}\(\\mathcal\{Z\}\)\\times\\mathcal\{R\}\_\{D\}\\to\\mathcal\{P\}\(\\mathcal\{Y\}\)𝗐~∼𝙳⁡\(u⁡\(𝗏\),𝗋𝙳\)\\tilde\{\\mathsf\{w\}\}\\sim\\mathtt\{D\}\(u\(\\mathsf\{v\}\),\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\),𝗒^∼𝙳⁡\(f⁡\(𝗏\),𝗋𝙳\)\\hat\{\\mathsf\{y\}\}\\sim\\mathtt\{D\}\(f\(\\mathsf\{v\}\),\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\)Realized decoder intended to approximate the target decoder𝙳⋆\\mathtt\{D\}^\{\\star\}\.Complete foundation modelRealizedFoundation model𝙿\\mathtt\{P\}𝙿:𝒳×𝒯→𝒫⁡\(𝒴\)\\mathtt\{P\}:\\mathcal\{X\}\\times\\mathcal\{T\}\\to\\mathcal\{P\}\(\\mathcal\{Y\}\)𝗒^∼𝙿⁡\(𝗑,𝗍\)\\hat\{\\mathsf\{y\}\}\\sim\\mathtt\{P\}\(\\mathsf\{x\},\\mathsf\{t\}\),𝙿⁡\(x,t\)=𝙳⁡\(f⁡\(𝙴⁡\(x,rE\)\),rD\)\\mathtt\{P\}\(x,t\)=\\mathtt\{D\}\\\!\\left\(f\\\!\\left\(\\mathtt\{E\}\(x,r\_\{E\}\)\\right\),r\_\{D\}\\right\)Complete any\-task inference procedure obtained by composing the task representations, encoder, realized universal\-task model, and decoder\.Table 2:Components of the universal\-task reduction\. Reduction components specify how the original task and input are mapped into the universal\-task domain\. Target components, shown in blue, specify the intended universal task, labeler, and the target decoder they induce; realized components, shown in orange, denote their implemented counterparts\.

## 4Decomposing Foundation Model Error

In this section, we will use Definition[3\.3](https://arxiv.org/html/2610.40934#S3.Thmtheorem3)to decompose foundation model errors into terms corresponding to different aspects of this decomposition\. There are many such decompositions possible, and as we will define them here, many are near\-trivial in complexity, as we will define individual error terms such that by construction they sum to the total foundation model error\. Nevertheless, the decompositions we introduce will allow us to understand what parts of the error we can optimize and control under different algorithms\. For example, suppose a foundation model is trained in a “typical” manner, where the developer’s intent is to train the modelffto approximate a pre\-specified universal taskuuand the decoder𝙳\\mathtt\{D\}to approximate the target decoder𝙳⋆\\mathtt\{D\}^\{\\star\}, while the other components of the process are held fixed\. A typical training procedure in this setting would then optimizeffto better approximateuuand/or𝙳\\mathtt\{D\}to better approximate𝙳⋆\\mathtt\{D\}^\{\\star\}\. By separating such error components out from other aspects of the error, we can identify what remaining components exist in the total error and when they can be bounded or identified to be zero\.

To decompose error in this way, we will first formally define what we mean by “foundation model error” in Subsection[4\.1](https://arxiv.org/html/2610.40934#S4.SS1), then introduce a large set of terms quantifying different aspects of the total error in Subsection[4\.2](https://arxiv.org/html/2610.40934#S4.SS2), before finally formally introducing our error decompositions and arguing for their utility in Subsection[4\.3](https://arxiv.org/html/2610.40934#S4.SS3)\.

### 4\.1Defining Error for a Single\-Draw Model

We define the error of a foundation model under strictly proper scoring lossℓ\\ellformally in Definition[4\.1](https://arxiv.org/html/2610.40934#S4.Thmtheorem1)\.

###### Definition 4\.1\(Foundation Modelℓ\\ell\-Error\)\.

Let𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}be an extended domain,ρ:𝒯↝ℛ\\rho:\{\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{R\}\}be a task representation,𝗑,𝗍,𝗋,𝗒\\mathsf\{x\},\\mathsf\{t\},\\mathsf\{r\},\\mathsf\{y\}be the normal joint random variables, and letℓ\\ellbe a strictly proper scoring loss\. Let𝙿\\mathtt\{P\}be a single\-ρ\\rho\-draw foundation model over this domain, defined therefore via Definition[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1)as a procedure over𝒳×ℛ\\mathcal\{X\}\\times\\mathcal\{R\}\. Let𝗒^\\hat\{\\mathsf\{y\}\}be a random variable induced by𝙿⁡\(𝗑,𝗋\)\\mathtt\{P\}\(\\mathsf\{x\},\\mathsf\{r\}\), which we can therefore understand to be jointly distributed with all other random variables\. We define the foundation model𝙿\\mathtt\{P\}’sℓ\\ellerror over𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}as the normal regret divergence forℓ\\ellbetween the predicted distribution given by𝙿\\mathtt\{P\}and the true distribution given by𝙿∗=η\\mathtt\{P\}^\{\*\}=\\etain expectation over samples and tasks:

Errℓ​\(𝙿\)\\displaystyle\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\):=dℓ​\(η⁡\(𝗑,𝗍\),𝙿⁡\(𝗑,𝗋\|𝗍\)\)\\displaystyle:=d\_\{\\ell\}\(\\eta\(\\mathsf\{x\},\\mathsf\{t\}\),\\mathtt\{P\}\(\\mathsf\{x\},\\mathsf\{r\}\|\\mathsf\{t\}\)\)=𝔼𝗑,𝗍​\[𝔼𝗋\|𝗍=t​\[dℓ​\(η⁡\(x,t\),𝙿⁡\(x,r\)\)\]\]\\displaystyle=\\mathbb\{E\}\_\{\\mathsf\{x\},\\mathsf\{t\}\}\\left\[\\mathbb\{E\}\_\{\\mathsf\{r\}\|\\mathsf\{t\}=t\}\\left\[d\_\{\\ell\}\(\\eta\(x,t\),\\mathtt\{P\}\(x,r\)\)\\right\]\\right\]

Definition[4\.1](https://arxiv.org/html/2610.40934#S4.Thmtheorem1)introduces the error of the𝒳×ℛ\\mathcal\{X\}\\times\\mathcal\{R\}procedure against the ideal model achievable over𝒳×𝒯\\mathcal\{X\}\\times\\mathcal\{T\}\(to respect the single\-draw constraint\)\. To capture the impact the single\-draw criteria has, Lemma[4\.2](https://arxiv.org/html/2610.40934#S4.Thmtheorem2)decomposesErrℓ​\(𝙿\)\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\)into a term describing the error in comparison to the achievable error of𝙿\\mathtt\{P\}vs\. the ideal predictor over𝒳×ℛ\\mathcal\{X\}\\times\\mathcal\{R\}\(not𝒳×𝒯\\mathcal\{X\}\\times\\mathcal\{T\}\) and a term capturing remaining error due to task\-representation error\.

###### Lemma 4\.2\.

Given Definition[4\.1](https://arxiv.org/html/2610.40934#S4.Thmtheorem1)andϕρ:𝒳×ℛ↝𝒴\\phi\_\{\\rho\}:\{\\mathcal\{X\}\\times\\mathcal\{R\}\}\\rightsquigarrow\{\\mathcal\{Y\}\}introduced in Definition[2\.2](https://arxiv.org/html/2610.40934#S2.Thmtheorem2), we can re\-organizeErrℓ​\(𝙿\)\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\)as defined in Definition[4\.1](https://arxiv.org/html/2610.40934#S4.Thmtheorem1)as follows:

Errℓ​\(𝙿\)\\displaystyle\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\)=dℓ​\(ϕρ,𝙿\)⏟achievable error\+Δρ\\displaystyle=\\underbrace\{d\_\{\\ell\}\\left\(\{\\phi\_\{\\rho\}\};\{\\mathtt\{P\}\}\\right\)\}\_\{\\text\{achievable error\}\}\+\\Delta\_\{\\rho\}=dℓ\(𝗒;𝗒^\|𝗑,𝗋\)⏟achievable error\+Δρ\\displaystyle=\\underbrace\{d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{y\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\}\_\{\\text\{achievable error\}\}\+\\Delta\_\{\\rho\}

###### Proof\.

Errℓ​\(𝙿\)\\displaystyle\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\)=𝔼𝗑,𝗍​\[𝔼𝗋\|𝗍=t​\[dℓ​\(η⁡\(x,t\),𝙿⁡\(x,r\)\)\]\]\\displaystyle=\\mathbb\{E\}\_\{\\mathsf\{x\},\\mathsf\{t\}\}\\left\[\\mathbb\{E\}\_\{\\mathsf\{r\}\|\\mathsf\{t\}=t\}\\left\[d\_\{\\ell\}\(\\eta\(x,t\),\\mathtt\{P\}\(x,r\)\)\\right\]\\right\]=𝔼𝗑,𝗍\[𝔼𝗋\|𝗍=t\[𝔼𝗒\|𝗑=x,𝗍=t,𝗋=r\[ℓ\(𝙿\(x,r\),y\)−ℓ\(η\(x,t\),y\)\]\]\]\\displaystyle=\\mathbb\{E\}\_\{\\mathsf\{x\},\\mathsf\{t\}\}\\left\[\\mathbb\{E\}\_\{\\mathsf\{r\}\|\\mathsf\{t\}=t\}\\left\[\\mathbb\{E\}\_\{\\mathsf\{y\}\|\\mathsf\{x\}=x,\\mathsf\{t\}=t,\\mathsf\{r\}=r\}\\left\[\\ell\(\\mathtt\{P\}\(x,r\),y\)\-\\ell\(\\eta\(x,t\),y\)\\right\]\\right\]\\right\]=𝔼𝗑,𝗍\[𝔼𝗋\|𝗍=t\[𝔼𝗒\|𝗑=x,𝗍=t,𝗋=r\[ℓ\(𝙿\(x,r\),y\)\]\]−𝔼𝗒\|𝗑=x,𝗍=t\[ℓ\(η\(x,t\),y\)\]\]\\displaystyle=\\mathbb\{E\}\_\{\\mathsf\{x\},\\mathsf\{t\}\}\\left\[\\mathbb\{E\}\_\{\\mathsf\{r\}\|\\mathsf\{t\}=t\}\\left\[\\mathbb\{E\}\_\{\\mathsf\{y\}\|\\mathsf\{x\}=x,\\mathsf\{t\}=t,\\mathsf\{r\}=r\}\\left\[\\ell\(\\mathtt\{P\}\(x,r\),y\)\\right\]\\right\]\-\\mathbb\{E\}\_\{\\mathsf\{y\}\|\\mathsf\{x\}=x,\\mathsf\{t\}=t\}\\left\[\\ell\(\\eta\(x,t\),y\)\\right\]\\right\]=𝔼𝗑,𝗍\[𝔼𝗋\|𝗍=t,𝗑=x\[𝔼𝗒\|𝗑=x,𝗋=r\[ℓ\(𝙿\(x,r\),y\)\]\]\]−𝔼𝗑,𝗍\[𝔼𝗒\|𝗑=x,𝗍=t\[ℓ\(η\(x,t\),y\)\]\]\\displaystyle=\\mathbb\{E\}\_\{\\mathsf\{x\},\\mathsf\{t\}\}\\left\[\\mathbb\{E\}\_\{\\mathsf\{r\}\|\\mathsf\{t\}=t,\\mathsf\{x\}=x\}\\left\[\\mathbb\{E\}\_\{\\mathsf\{y\}\|\\mathsf\{x\}=x,\\mathsf\{r\}=r\}\\left\[\\ell\(\\mathtt\{P\}\(x,r\),y\)\\right\]\\right\]\\right\]\-\\mathbb\{E\}\_\{\\mathsf\{x\},\\mathsf\{t\}\}\\left\[\\mathbb\{E\}\_\{\\mathsf\{y\}\|\\mathsf\{x\}=x,\\mathsf\{t\}=t\}\\left\[\\ell\(\\eta\(x,t\),y\)\\right\]\\right\]=𝔼𝗍​\[𝔼𝗋,𝗑\|𝗍=t​\[dℓ​\(ϕρ​\(x,r\),𝙿⁡\(x,r\)\)\+hℓ​\(ϕρ​\(x,r\)\)\]\]−hℓ​\(𝗒\|𝗑,𝗍\)\\displaystyle=\\mathbb\{E\}\_\{\\mathsf\{t\}\}\\left\[\\mathbb\{E\}\_\{\\mathsf\{r\},\\mathsf\{x\}\|\\mathsf\{t\}=t\}\\left\[d\_\{\\ell\}\(\\phi\_\{\\rho\}\(x,r\),\\mathtt\{P\}\(x,r\)\)\+h\_\{\\ell\}\(\\phi\_\{\\rho\}\(x,r\)\)\\right\]\\right\]\-h\_\{\\ell\}\(\\mathsf\{y\}\|\\mathsf\{x\},\\mathsf\{t\}\)=𝔼𝗋,𝗑\[𝔼𝗍=t\|𝗋=r,𝗑=x\[dℓ\(ϕρ\(x,r\),𝙿\(x,r\)\)\+hℓ\(ϕρ\(x,r\)\)\]\]−hℓ\(𝗒\|𝗑,𝗍\)\\displaystyle=\\mathbb\{E\}\_\{\\mathsf\{r\},\\mathsf\{x\}\}\\left\[\\mathbb\{E\}\_\{\\mathsf\{t\}=t\|\\mathsf\{r\}=r,\\mathsf\{x\}=x\}\\left\[d\_\{\\ell\}\(\\phi\_\{\\rho\}\(x,r\),\\mathtt\{P\}\(x,r\)\)\+h\_\{\\ell\}\(\\phi\_\{\\rho\}\(x,r\)\)\\right\]\\right\]\-h\_\{\\ell\}\(\\mathsf\{y\}\|\\mathsf\{x\},\\mathsf\{t\}\)=dℓ​\(ϕρ​\(𝗑,𝗋\),𝙿⁡\(𝗑,𝗋\)\)\+hℓ​\(𝗒\|𝗑,𝗋\)−hℓ​\(𝗒\|𝗑,𝗍\)\\displaystyle=d\_\{\\ell\}\\left\(\{\\phi\_\{\\rho\}\(\\mathsf\{x\},\\mathsf\{r\}\)\};\{\\mathtt\{P\}\(\\mathsf\{x\},\\mathsf\{r\}\)\}\\right\)\+h\_\{\\ell\}\(\\mathsf\{y\}\|\\mathsf\{x\},\\mathsf\{r\}\)\-h\_\{\\ell\}\(\\mathsf\{y\}\|\\mathsf\{x\},\\mathsf\{t\}\)=dℓ​\(ϕρ,𝙿\)\+Δρ,\\displaystyle=d\_\{\\ell\}\\left\(\{\\phi\_\{\\rho\}\};\{\\mathtt\{P\}\}\\right\)\+\\Delta\_\{\\rho\},as desired\. ∎

### 4\.2Components of Foundation Model Error

To provide a useful decomposition ofErrℓ​\(𝙿\)\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\), in this section we will introduce a variety of terms of different types of error which we will eventually combine in Theorem[4\.11](https://arxiv.org/html/2610.40934#S4.Thmtheorem11)\.

These terms will be organized to quantify the errors of different components of𝙿\\mathtt\{P\}in various arrangements, motivated by the assumption that the developer is “intending” to approximate a pre\-specified universal taskuuand target decoderD⋆D^\{\\star\}\. For example, under the assumption that a model builder may be training the modelffto approximate a taskuu, it would be of interest to compute the error offf’s approximation ofuu\(to determine how well their training process worked\) or how much error they would incur were they able to approximateuuperfectly, then use the rest of their foundation model as specified \(to evaluate the choice of universal taskuu\)\.

We will introduce these terms in three sections\. First, a section introducing “error” terms capturing unambiguous, strictly non\-negative measures of error that would be obtained were elements of𝙿\\mathtt\{P\}isolated from the system and considered in a standalone manner\. Second, a section introducing signed “excess” terms that capture how real\-world interactions amongst these components in a full foundation model can either exacerbate or ameliorate total expected error\. Finally, third, we will note that asℓ\\ellis a strictly proper scoring loss, there are natural, information\-geometric, signed Bregman interaction terms canonical via the Bregman three\-point identity between some combinations of our terms\. All of these sections will rely heavily on the notation introduced in Appendix[A](https://arxiv.org/html/2610.40934#A1)and the specific random variables introduced to correspond to foundation model components defined in Notation[1](https://arxiv.org/html/2610.40934#Thmnotation1)and[2](https://arxiv.org/html/2610.40934#Thmnotation2)and Figure[2](https://arxiv.org/html/2610.40934#S3.F2)\. In addition, all of these terms will judge the foundation model \(for now\) against the achievable total error, and thus condition on the observed\(𝗑,𝗋\)\(\\mathsf\{x\},\\mathsf\{r\}\)pair rather than the base\(𝗑,𝗍\)\(\\mathsf\{x\},\\mathsf\{t\}\)pair, consistent with the single\-draw criteria\.

#### Error Terms

In Definition[4\.3](https://arxiv.org/html/2610.40934#S4.Thmtheorem3), we introduce various direct error measurements corresponding to parts of𝙿\\mathtt\{P\}\. While we will combine each of these terms later, informally, they are all directly useful quantities when designing or evaluating a concrete foundation model\. Specifically:

- •A target universal taskuuthat has large deficiencyΔu\\Delta\_\{u\}is likely a poor target for foundation model development, as solving the task perfectly will still not enable production of a consistent, high\-performance foundation model\.
- •While often unmeasurable, the observed model deficiencyΔf\\Delta\_\{f\}directly captures how much error is inescapable with a fixed modelff, and will be useful in our decomposition later\.
- •If the model errorεf\\varepsilon\_\{f\}is large, then the model poorly approximatesuu, limiting internal validity if the choice of universal taskuuis important\.
- •If labeler error w\.r\.t\. eitheruuorffis large, then the choice of labelerddcan be seen to be problematic at a design level\.
- •Finally, if the observed decoder error w\.r\.t\. eitheruuorffis large, it means that the realized decoder𝙳\\mathtt\{D\}is unable to well approximate the target decoder𝙳⋆\\mathtt\{D\}^\{\\star\}, limiting internal validity if the choice ofddis important\.

All terms will also be w\.r\.t\. a strictly proper𝒴\\mathcal\{Y\}\-scoring lossℓ\\ell\. In addition, we letℓu:𝒫⁡\(𝒵\)×𝒵→ℝ¯\\ell\_\{u\}:\\mathcal\{P\}\\left\(\\mathcal\{Z\}\\right\)\\times\\mathcal\{Z\}\\to\\overline\{\\mathbb\{R\}\}denote a strictly proper scoring loss over the universal\-task output space𝒵\\mathcal\{Z\}, which we use to measure discrepancies between the target universal taskuuand the modelff\.

###### Definition 4\.3\.

Given the setup introduced previously, define the following error terms, which are all non\-negative:

uu\-deficiencyΔu\\Delta\_\{u\}The*universal\-task deficiency*is the minimal possible error auu\-optimal foundation model𝙿\\mathtt\{P\}could ever achieve when its decoder is restricted to a labeling step:555Meaning that𝙿\\mathtt\{P\}must bedd\-optimal for some labeling kernelλ∈ℳ⁡\(𝒴\|ℛD×𝒵\)\\lambda\\in\\mathcal\{M\}\\left\(\\mathcal\{Y\}\\middle\|\\mathcal\{R\}\_\{D\}\\times\\mathcal\{Z\}\\right\)\. We allowλ\\lambdato be stochastic so thatΔu\\Delta\_\{u\}is the best error attainable from𝗓\\mathsf\{z\}at all; the cost of restricting to a deterministic labelerddis then part ofΔdu\\Delta\_\{d\}^\{u\}\.

Δu:=δℓ\(𝗒;𝗓,𝗋𝙳\|𝗑,𝗋\)\.\\Delta\_\{u\}:=\\delta\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\mathsf\{z\},\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\.
ff\-deficiencyΔf\\Delta\_\{f\}The*model\-deficiency*is analogous toΔu\\Delta\_\{u\}, but when we use the modelffrather than the target universal taskuu:

Δf:=δℓ\(𝗒;𝗓^,𝗋𝙳\|𝗑,𝗋\)\.\\Delta\_\{f\}:=\\delta\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{z\}\},\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\.
Model Errorεf\\varepsilon\_\{f\}The*model\-error*captures how effectivelyffmodelsuu, in the typical way:

εf:=dℓu\(𝗓;𝗓^\|𝗑,𝗋\)\.\\varepsilon\_\{f\}:=d\_\{\\ell\_\{u\}\}\\left\(\{\\mathsf\{z\}\};\{\\hat\{\\mathsf\{z\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\.
Labeler ErrorΔdu\\Delta\_\{d\}^\{u\}The*labeler\-error w\.r\.t\.uu*computes how much excess error is induced through the use of a specific labelerddover the minimal error possible to obtain when predicting from the true universal taskuu:

Δdu:=dℓ\(𝗒;𝗐\|𝗑,𝗋\)−Δu\.\\Delta\_\{d\}^\{u\}:=d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\mathsf\{w\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\-\\Delta\_\{u\}\.
Labeler ErrorΔdf\\Delta\_\{d\}^\{f\}Similarly, the*labeler error w\.r\.t\.ff*computes the excess error due toddwhen predicting from real modelff:

Δdf:=dℓ\(𝗒;𝗐^\|𝗑,𝗋\)−Δf\.\\Delta\_\{d\}^\{f\}:=d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{w\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\-\\Delta\_\{f\}\.
Decoder\-labeler Error w\.r\.tuuΔ𝙳u\\Delta\_\{\\mathtt\{D\}\}^\{u\}The*Decoder\-error w\.r\.t\.uu*computes how well the realized decoder𝙳\\mathtt\{D\}approximates the target decoder𝙳⋆\\mathtt\{D\}^\{\\star\}on the outputs of the universal taskuu:

Δ𝙳u:=dℓ\(𝗐;𝗐~\|𝗑,𝗋\)\\Delta\_\{\\mathtt\{D\}\}^\{u\}:=d\_\{\\ell\}\\left\(\{\\mathsf\{w\}\};\{\\tilde\{\\mathsf\{w\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)
Decoder\-labeler Error w\.r\.tffΔ𝙳f\\Delta\_\{\\mathtt\{D\}\}^\{f\}The*Decoder\-error w\.r\.tff*is analogous toΔ𝙳u\\Delta\_\{\\mathtt\{D\}\}^\{u\}but over the outputs of realized modelffrather thanuu666Recall that𝗒^\\hat\{\\mathsf\{y\}\}denotes the realized decoder applied to the realized model, i\.e\., the output of𝙿\\mathtt\{P\}\.:

Δ𝙳f:=dℓ\(𝗐^;𝗒^\|𝗑,𝗋\)\.\\Delta\_\{\\mathtt\{D\}\}^\{f\}:=d\_\{\\ell\}\\left\(\{\\hat\{\\mathsf\{w\}\}\};\{\\hat\{\\mathsf\{y\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\.

With these error terms, we can additionally introduce Definition[4\.4](https://arxiv.org/html/2610.40934#S4.Thmtheorem4), which states that a foundation model is*well\-founded*w\.r\.t\. eitheruuorddif it has no universal\-task deficiency or labeler error, respectively\. Informally, this definition is advisable as if one can justify that a foundation model setup is well\-founded, it indicates they have chosen a universal task and labeler that theoretically permit any\-task inference at optimal achievable performance, in the event that one can learn auu\-optimal modelffand leverage add\-optimal decoder𝙳\\mathtt\{D\}\.

###### Definition 4\.4\.

A foundation model𝙿\\mathtt\{P\}is*well\-founded w\.r\.t\.uu*ifΔu=0\\Delta\_\{u\}=0\. It is*well\-founded w\.r\.t\.dd*ifΔdu=0\\Delta\_\{d\}^\{u\}=0\. It is simply*well\-founded*if it is well\-founded w\.r\.t\. bothuuanddd\.

In addition, Lemma[4\.5](https://arxiv.org/html/2610.40934#S4.Thmtheorem5)relatesuu\- anddd\-optimality to the corresponding error terms, showing thatuu\-optimality is characterized by zero model error, whiledd\-optimality guarantees zero decoder error\.

###### Lemma 4\.5\.

Recall Definitions[3\.4](https://arxiv.org/html/2610.40934#S3.Thmtheorem4)and[4\.3](https://arxiv.org/html/2610.40934#S4.Thmtheorem3)\. Two properties hold:

1. 1\.𝙿\\mathtt\{P\}isuu\-optimal iffεf=0\\varepsilon\_\{f\}=0\.
2. 2\.If𝙿\\mathtt\{P\}isdd\-optimal, thenΔ𝙳f=Δ𝙳u=0\\Delta\_\{\\mathtt\{D\}\}^\{f\}=\\Delta\_\{\\mathtt\{D\}\}^\{u\}=0\.

###### Proof\.

The first property follows directly from the definition ofuu\-optimality meaning thatf=uf=uand thatℓu\\ell\_\{u\}is strictly proper\. The second follows from the definition ofdd\-optimality meaning that𝙳=𝙳⋆\\mathtt\{D\}=\\mathtt\{D\}^\{\\star\}, so𝗐~\\tilde\{\\mathsf\{w\}\}and𝗐\\mathsf\{w\}\(resp\.𝗒^\\hat\{\\mathsf\{y\}\}and𝗐^\\hat\{\\mathsf\{w\}\}\) have the same conditional distribution given\(𝗑,𝗋\)\(\\mathsf\{x\},\\mathsf\{r\}\)\. ∎

#### Excess Terms

All of the direct error terms in Definition[4\.3](https://arxiv.org/html/2610.40934#S4.Thmtheorem3)are non\-negative, and in a strict sense only a subset of them will directly contribute to total foundation model error\. This discrepancy is because these error terms measure direct comparisons on individual components of𝙿\\mathtt\{P\}, and when these components are combined, the result can have greater or lesser error than that introduced by the sum of its parts\. To capture these additional interactions, Definition[4\.6](https://arxiv.org/html/2610.40934#S4.Thmtheorem6)defines these signed “excess terms” which compute what remains between the true foundation model error and the direct error terms defined in Definition[4\.3](https://arxiv.org/html/2610.40934#S4.Thmtheorem3)\.

###### Definition 4\.6\.

Given the setup introduced previously and the terms in Definition[4\.3](https://arxiv.org/html/2610.40934#S4.Thmtheorem3), define the following “excess terms”, which compare the relative errors that one obtains in reality \(via modelff\) vs\. those they would expect during model design \(via universal taskuu\):

Excess model deficiencyΦf\\Phi\_\{f\}Even thoughffis an imperfect approximator ofuu, it may exacerbate or ameliorate the total possible error obtainable through a permissible foundation model in comparison touu, andΦf\\Phi\_\{f\}captures this tradeoff:

Φf:=Δf−Δu\.\\Phi\_\{f\}:=\\Delta\_\{f\}\-\\Delta\_\{u\}\.
Excess labeler errorΦd\\Phi\_\{d\}Similarly, beyond the optimal possible discrepancy in error,ffmay induce different observed error relative to the possible error in comparison to what would be expected at design time w\.r\.t\. the universal taskuu:

Φd:=Δdf−Δdu\.\\Phi\_\{d\}:=\\Delta\_\{d\}^\{f\}\-\\Delta\_\{d\}^\{u\}\.
Excess Decoder\-labeler errorΦ𝙳\\Phi\_\{\\mathtt\{D\}\}Similarly, the realized decoder can exacerbate or ameliorate error in matching the target decoder𝙳⋆\\mathtt\{D\}^\{\\star\}differently underffin comparison to what would be expected at design time w\.r\.tuu:

Φ𝙳:=Δ𝙳f−Δ𝙳u\\Phi\_\{\\mathtt\{D\}\}:=\\Delta\_\{\\mathtt\{D\}\}^\{f\}\-\\Delta\_\{\\mathtt\{D\}\}^\{u\}

Building on these definitions, Lemma[4\.7](https://arxiv.org/html/2610.40934#S4.Thmtheorem7)remarks that all excess termsΦ∗\\Phi\_\{\*\}vanish underuu\-optimality andΦ𝙳\\Phi\_\{\\mathtt\{D\}\}vanishes underdd\-optimality\.

###### Lemma 4\.7\.

If𝙿\\mathtt\{P\}isuu\-optimal, thenΦf=Φd=Φ𝙳=0\\Phi\_\{f\}=\\Phi\_\{d\}=\\Phi\_\{\\mathtt\{D\}\}=0\. If𝙿\\mathtt\{P\}isdd\-optimal, thenΦ𝙳=0\\Phi\_\{\\mathtt\{D\}\}=0\.

###### Proof\.

If𝙿\\mathtt\{P\}isuu\-optimal, thenffis preciselyuu, so by construction these differences in response betweenffanduuvanish\. For point 2, Lemma[4\.5](https://arxiv.org/html/2610.40934#S4.Thmtheorem5)shows that underdd\-optimality, bothΔ𝙳f=Δ𝙳u=0\\Delta\_\{\\mathtt\{D\}\}^\{f\}=\\Delta\_\{\\mathtt\{D\}\}^\{u\}=0, which implies thatΦ𝙳=0\\Phi\_\{\\mathtt\{D\}\}=0, as desired\. ∎

#### Bregman Interaction Terms

Lastly, in Definition[4\.8](https://arxiv.org/html/2610.40934#S4.Thmtheorem8)we introduce some notation for the remaining signed Bregman interaction terms inherent from Bregman’s three\-point identity and the fact thatℓ\\ellis a strictly proper scoring loss\.

###### Definition 4\.8\.

We will explicitly useB𝙳fB\_\{\\mathtt\{D\}\}^\{f\}to denote the specific Bregman interaction term capturing the difference in divergence between𝗒\\mathsf\{y\}and𝗒^\\hat\{\\mathsf\{y\}\}vs\. that captured consecutively between𝗒\\mathsf\{y\}then𝗐^\\hat\{\\mathsf\{w\}\}then𝗒^\\hat\{\\mathsf\{y\}\}:

B𝙳f\\displaystyle B\_\{\\mathtt\{D\}\}^\{f\}:=Bℓ\(𝗒;𝗐^;𝗒^\|𝗑,𝗋\):=dℓ\(𝗒;𝗒^\|𝗑,𝗋\)−dℓ\(𝗒;𝗐^\|𝗑,𝗋\)−dℓ\(𝗐^;𝗒^\|𝗑,𝗋\)\\displaystyle:=B\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{w\}\}\};\{\\hat\{\\mathsf\{y\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\):=d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{y\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\-d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{w\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\-d\_\{\\ell\}\\left\(\{\\hat\{\\mathsf\{w\}\}\};\{\\hat\{\\mathsf\{y\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)

Building on this definition, Lemma[4\.9](https://arxiv.org/html/2610.40934#S4.Thmtheorem9)identifies thatB𝙳fB\_\{\\mathtt\{D\}\}^\{f\}vanishes fordd\-optimal models\.

###### Lemma 4\.9\.

If𝙿\\mathtt\{P\}isdd\-optimal, thenB𝙳f=0B\_\{\\mathtt\{D\}\}^\{f\}=0\.

###### Proof\.

This follows directly fromdd\-optimality implying that𝗐^\\hat\{\\mathsf\{w\}\}will be identically distributed to𝗒^\\hat\{\\mathsf\{y\}\}given\(𝗑,𝗋\)\(\\mathsf\{x\},\\mathsf\{r\}\), as𝙳=𝙳⋆\\mathtt\{D\}=\\mathtt\{D\}^\{\\star\}\. ∎

### 4\.3General Decompositions of Foundation Model Error

In this section, we leverage the terms defined in Definitions[4\.3](https://arxiv.org/html/2610.40934#S4.Thmtheorem3),[4\.6](https://arxiv.org/html/2610.40934#S4.Thmtheorem6), and[4\.8](https://arxiv.org/html/2610.40934#S4.Thmtheorem8)to formally decompose foundation model error\. We will express this decomposed error first via Lemma[4\.10](https://arxiv.org/html/2610.40934#S4.Thmtheorem10), which shows a clear, telescoping error decomposition among our model error terms and our Bregman interaction term\. However, our more ultimately useful result will be Theorem[4\.11](https://arxiv.org/html/2610.40934#S4.Thmtheorem11), which will leverage the excess terms as well to divide the error between design\-space quantities dependent onuu,dd, and𝙳\\mathtt\{D\}– which can be analyzed even before a specific modelffis trained to motivate a foundation model design – and our signed Bregman and excess terms, which can adjust total error in either direction, but generally cannot be fully controlled, and can only be bounded or analyzed post training or in specific contexts\.

###### Lemma 4\.10\.

Given a single\-draw foundation model𝙿\\mathtt\{P\}using the notation/variable conventions introduced previously, then

Errℓ​\(𝙿\)=Δρ\+Δf\+Δdf\+Δ𝙳f\+B𝙳f\.\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\)=\\Delta\_\{\\rho\}\+\\Delta\_\{f\}\+\\Delta\_\{d\}^\{f\}\+\\Delta\_\{\\mathtt\{D\}\}^\{f\}\+B\_\{\\mathtt\{D\}\}^\{f\}\.

###### Proof\.

Recall from Lemma[4\.2](https://arxiv.org/html/2610.40934#S4.Thmtheorem2)thatErrℓ​\(𝙿\)−Δρ=dℓ​\(ϕρ,𝙿\)\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\)\-\\Delta\_\{\\rho\}=d\_\{\\ell\}\\left\(\{\\phi\_\{\\rho\}\};\{\\mathtt\{P\}\}\\right\), wheredℓ\(ϕρ;𝙿\)=dℓ\(𝗒;𝗒^\|𝗑,𝗋\)\.d\_\{\\ell\}\\left\(\{\\phi\_\{\\rho\}\};\{\\mathtt\{P\}\}\\right\)=d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{y\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\.Further, by definition,B𝙳f=dℓ\(𝗒;𝗒^\|𝗑,𝗋\)−Δ𝙳f−\(Δdf\+Δf\)B\_\{\\mathtt\{D\}\}^\{f\}=d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{y\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\-\\Delta\_\{\\mathtt\{D\}\}^\{f\}\-\(\\Delta\_\{d\}^\{f\}\+\\Delta\_\{f\}\)\. Therefore,B𝙳f\+Δ𝙳f\+Δdf\+Δf\+Δρ=Errℓ​\(𝙿\)B\_\{\\mathtt\{D\}\}^\{f\}\+\\Delta\_\{\\mathtt\{D\}\}^\{f\}\+\\Delta\_\{d\}^\{f\}\+\\Delta\_\{f\}\+\\Delta\_\{\\rho\}=\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\), as desired\. ∎

###### Theorem 4\.11\(Foundation Model Error Decomposition\)\.

Let𝙿\\mathtt\{P\}be a foundation model with the usual notation\. Then,

Errℓ​\(𝙿\)\\displaystyle\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\)=Δρ\\displaystyle=\\Delta\_\{\\rho\}Representation Error\+\(Δu\+Δdu\+Δ𝙳u\)\\displaystyle\\quad\+\\left\(\\Delta\_\{u\}\+\\Delta\_\{d\}^\{u\}\+\\Delta\_\{\\mathtt\{D\}\}^\{u\}\\right\)Design Error\+\(Φf\+Φd\+Φ𝙳\)\\displaystyle\\quad\+\\left\(\\Phi\_\{f\}\+\\Phi\_\{d\}\+\\Phi\_\{\\mathtt\{D\}\}\\right\)Model Excess\+B𝙳f\\displaystyle\\quad\+B\_\{\\mathtt\{D\}\}^\{f\}Geometric Bregman Interaction Term\.

###### Proof\.

This follows directly from Lemma[4\.10](https://arxiv.org/html/2610.40934#S4.Thmtheorem10)and the observation that the excess terms are precisely the difference between the model error terms used in Lemma[4\.10](https://arxiv.org/html/2610.40934#S4.Thmtheorem10)and the design\-space error terms introduced here\. ∎

Note that Theorem[4\.11](https://arxiv.org/html/2610.40934#S4.Thmtheorem11)does not precisely tell us anything distinct from the generic expression of error\. The terms used in its decomposition are constructed precisely to enable a decomposition of this form, and the model excess terms and the Bregman interaction terms are both signed, so it is not the case that we can even say that the design\-space error terms lower bound total error in general \(though by Lemma[4\.2](https://arxiv.org/html/2610.40934#S4.Thmtheorem2)we can say thatΔρ\\Delta\_\{\\rho\}does lower bound total error\)\.

Nevertheless, when designing a foundation model algorithm, especially under current AI systems, we typically can only directly control the design error terms and cannot intentionally influence model excess or the Bregman interaction terms directly\. E\.g\., if we design a foundation model with a fixed encoder𝙴\\mathtt\{E\}and decoder𝙳\\mathtt\{D\}and a target labelerddand universal taskuuin mind, then train a modelffto approximateuu, it is not the case that we will be able to directly or even indirectly dictate the sign or magnitude of the model excess or Bregman interaction terms at all – so identifying how they split from the design error terms, which we can control, allows us to meaningfully and transparently justify our design choices and their limitations\. Beyond this benefit, in select settings we will be able to enforce more concrete bounds on some of these non\-design terms, enabling us to bound or otherwise more richly understand the total error, which we will explore next\.

#### Example

The decomposition highlights distinct limitations across foundation model paradigms\. An underspecified LLM prompt can induce representation errorΔρ\\Delta\_\{\\rho\}, while an autoregressive EHR model approximating task probabilities through finite Monte Carlo sampling may incur decoder errorΔ𝙳u\\Delta\_\{\\mathtt\{D\}\}^\{u\}\. In contrast, task\-conditioned models such as EveryQuery use an identity decoder, eliminating this decoder approximation error\. Thus, different architectures can concentrate error in fundamentally different components of the same decomposition\.

#### Practical Settings in Which Model Excess and Bregman Interaction Terms can be Bounded\.

It is not practical to insist onuu\-optimality, as this cannot be reliably achieved in settings whereffis learned \(though we note trivially by Lemma[4\.7](https://arxiv.org/html/2610.40934#S4.Thmtheorem7)that this zeros all excess terms\)\. However, it may be practical to insist upon a model being both well\-founded anddd\-optimal – e\.g\., were you able to perfectly learn the universal taskuu, the total error would reduce to the irreducible representation errorΔρ\\Delta\_\{\\rho\}\. Lemma[4\.12](https://arxiv.org/html/2610.40934#S4.Thmtheorem12)shows that in this setting, all error terms can be reduced solely toΔρ\+Φf\+Φd=Δρ\+dℓ\(𝗒;𝗐^\|𝗑,𝗋\)\\Delta\_\{\\rho\}\+\\Phi\_\{f\}\+\\Phi\_\{d\}=\\Delta\_\{\\rho\}\+d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{w\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\.

###### Lemma 4\.12\.

Let𝙿\\mathtt\{P\}be a foundation model that is well\-founded in the sense of Definition[4\.4](https://arxiv.org/html/2610.40934#S4.Thmtheorem4)anddd\-optimal in sense of Definition[3\.4](https://arxiv.org/html/2610.40934#S3.Thmtheorem4)\. Then

Errℓ​\(𝙿\)\\displaystyle\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\)=Δρ\+Φf\+Φd\\displaystyle=\\Delta\_\{\\rho\}\+\\Phi\_\{f\}\+\\Phi\_\{d\}=Δρ\+Δf\+Δdf\\displaystyle=\\Delta\_\{\\rho\}\+\\Delta\_\{f\}\+\\Delta\_\{d\}^\{f\}=Δρ\+dℓ\(𝗒;𝗐^\|𝗑,𝗋\)\.\\displaystyle=\\Delta\_\{\\rho\}\+d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{w\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)\.

###### Proof\.

The first equality follows directly from Definition[4\.4](https://arxiv.org/html/2610.40934#S4.Thmtheorem4)of well\-founded \(which setsΔu=Δdu=0\\Delta\_\{u\}=\\Delta\_\{d\}^\{u\}=0\), Lemma[4\.7](https://arxiv.org/html/2610.40934#S4.Thmtheorem7), which establishes thatΦ𝙳=0\\Phi\_\{\\mathtt\{D\}\}=0, Lemma[4\.5](https://arxiv.org/html/2610.40934#S4.Thmtheorem5), which establishes thatΔ𝙳u=0\\Delta\_\{\\mathtt\{D\}\}^\{u\}=0, and Lemma[4\.9](https://arxiv.org/html/2610.40934#S4.Thmtheorem9), which establishes thatB𝙳f=0B\_\{\\mathtt\{D\}\}^\{f\}=0, all givendd\-optimality\. The second two equalities simply expand the definitions of these excess terms and ofΔdf\\Delta\_\{d\}^\{f\}\. ∎

However, in such a setting, we can do even better by re\-introducing the model errorεf\\varepsilon\_\{f\}\. Recall from Definition[4\.3](https://arxiv.org/html/2610.40934#S4.Thmtheorem3)thatεf\\varepsilon\_\{f\}may be measured via a different strictly proper scoring lossℓu\\ell\_\{u\}thanℓ\\ell, which can make it challenging to compare these two losses\. To remedy this, Definition[4\.13](https://arxiv.org/html/2610.40934#S4.Thmtheorem13)first introduces the notion of a loss inflation term,ιℓ←ℓu\\iota\_\{\\ell\\leftarrow\\ell\_\{u\}\}that bounds how much a stochastic kernel can possibly inflate a loss in spaceℓu\\ell\_\{u\}to one in spaceℓ\\ell\.

###### Definition 4\.13\.

Given \(1\) spaces𝒴\\mathcal\{Y\}and𝒵\\mathcal\{Z\}; \(2\) strictly proper scoring lossesℓ\\ellandℓu\\ell\_\{u\}over these spaces, respectively; and \(3\) a stochastic kernelλ:𝒵↝𝒴\\lambda:\{\\mathcal\{Z\}\}\\rightsquigarrow\{\\mathcal\{Y\}\}, define the divergence inflation coefficient as

ιℓ←ℓu\(λ\):=inf\{C∈\[0,∞\):dℓ\(λ∘p;λ∘q\)≤Cdℓu\(p;q\),∀p,q∈𝒫\(𝒵\)\},\\iota\_\{\\ell\\leftarrow\\ell\_\{u\}\}\(\\lambda\):=\\inf\\left\\\{C\\in\[0,\\infty\):d\_\{\\ell\}\\left\(\{\\lambda\\circ p\};\{\\lambda\\circ q\}\\right\)\\leq Cd\_\{\\ell\_\{u\}\}\\left\(\{p\};\{q\}\\right\),\\quad\\forall p,q\\in\\mathcal\{P\}\\left\(\\mathcal\{Z\}\\right\)\\right\\\},with the conventioninf∅=∞\\inf\\varnothing=\\infty\.

Armed with Definition[4\.13](https://arxiv.org/html/2610.40934#S4.Thmtheorem13)and Lemma[4\.12](https://arxiv.org/html/2610.40934#S4.Thmtheorem12), Theorem[4\.14](https://arxiv.org/html/2610.40934#S4.Thmtheorem14)shows that in this well\-founded,dd\-optimal setting, we can leverage this to boundErrℓ​\(𝙿\)\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\)in terms ofΔρ\\Delta\_\{\\rho\}andεf\\varepsilon\_\{f\}directly\.

###### Theorem 4\.14\.

Let𝙿\\mathtt\{P\}be a well\-founded,dd\-optimal foundation model, withℓu\\ell\_\{u\}a strictly proper scoring loss over𝒵\\mathcal\{Z\}in addition to the strictly proper scoring lossℓ\\ellover𝒴\\mathcal\{Y\}\. Forr𝙳∈ℛD\{r\_\{\\mathtt\{D\}\}\}\\in\\mathcal\{R\}\_\{D\}, letλr𝙳:𝒵↝𝒴\\lambda\_\{\{r\_\{\\mathtt\{D\}\}\}\}:\{\\mathcal\{Z\}\}\\rightsquigarrow\{\\mathcal\{Y\}\},λr𝙳​\(z\):=δdr𝙳​\(z\)\\lambda\_\{\{r\_\{\\mathtt\{D\}\}\}\}\(z\):=\\delta\_\{d\_\{\{r\_\{\\mathtt\{D\}\}\}\}\(z\)\}, be the kernel induced by the labeler, and letCd:=ess​sup𝗋𝙳⁡ιℓ←ℓu​\(λr𝙳\)C\_\{d\}:=\\esssup\_\{\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\}\\iota\_\{\\ell\\leftarrow\\ell\_\{u\}\}\(\\lambda\_\{\{r\_\{\\mathtt\{D\}\}\}\}\)\.

Assume thatεf<∞\\varepsilon\_\{f\}<\\inftyandCd<∞C\_\{d\}<\\infty\.

Then,

Errℓ​\(𝙿\)≤Δρ\+Cd​εf\.\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\)\\leq\\Delta\_\{\\rho\}\+C\_\{d\}\\varepsilon\_\{f\}\.

###### Proof\.

Letp:=u∘𝙴p:=u\\circ\\mathtt\{E\}andq:=f∘𝙴q:=f\\circ\\mathtt\{E\}, both kernels𝒳×ℛE↝𝒵\{\\mathcal\{X\}\\times\\mathcal\{R\}\_\{E\}\}\\rightsquigarrow\{\\mathcal\{Z\}\}\. Thenppis the conditional kernel of𝗓\|𝗑,𝗋𝙴\\left\.\\mathsf\{z\}\\middle\|\\mathsf\{x\},\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\}\\right\.andqqthat of𝗓^\|𝗑,𝗋𝙴\\left\.\\hat\{\\mathsf\{z\}\}\\middle\|\\mathsf\{x\},\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\}\\right\.\.

Note that by well\-foundedness \(Δu=Δdu=0\\Delta\_\{u\}=\\Delta\_\{d\}^\{u\}=0\),𝗒\|𝗑=x,𝗋𝙴=r𝙴,𝗋𝙳=r𝙳\\mathsf\{y\}\|\\mathsf\{x\}=x,\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\}=\{r\_\{\\mathtt\{E\}\}\},\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}=\{r\_\{\\mathtt\{D\}\}\}has law𝙳⋆​\(p⁡\(x,r𝙴\),r𝙳\)=λr𝙳∘p⁡\(x,r𝙴\)\\mathtt\{D\}^\{\\star\}\(p\(x,\{r\_\{\\mathtt\{E\}\}\}\),\{r\_\{\\mathtt\{D\}\}\}\)=\\lambda\_\{\{r\_\{\\mathtt\{D\}\}\}\}\\circ p\(x,\{r\_\{\\mathtt\{E\}\}\}\), and by definition𝗐^\|𝗑=x,𝗋𝙴=r𝙴,𝗋𝙳=r𝙳\\left\.\\hat\{\\mathsf\{w\}\}\\middle\|\\mathsf\{x\}=x,\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\}=\{r\_\{\\mathtt\{E\}\}\},\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}=\{r\_\{\\mathtt\{D\}\}\}\\right\.has lawλr𝙳∘q⁡\(x,r𝙴\)\\lambda\_\{\{r\_\{\\mathtt\{D\}\}\}\}\\circ q\(x,\{r\_\{\\mathtt\{E\}\}\}\)\. Hence, by Definition[4\.13](https://arxiv.org/html/2610.40934#S4.Thmtheorem13),

dℓ\(𝗒;𝗐^\|𝗑,𝗋\)\\displaystyle d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{w\}\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)=𝔼𝗑,𝗋𝙴,𝗋𝙳​\[dℓ​\(λr𝙳∘p⁡\(x,r𝙴\),λr𝙳∘q⁡\(x,r𝙴\)\)\]\\displaystyle=\\mathbb\{E\}\_\{\\mathsf\{x\},\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\},\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\}\\left\[d\_\{\\ell\}\\left\(\{\\lambda\_\{\{r\_\{\\mathtt\{D\}\}\}\}\\circ p\(x,\{r\_\{\\mathtt\{E\}\}\}\)\};\{\\lambda\_\{\{r\_\{\\mathtt\{D\}\}\}\}\\circ q\(x,\{r\_\{\\mathtt\{E\}\}\}\)\}\\right\)\\right\]≤𝔼𝗑,𝗋𝙴,𝗋𝙳​\[ιℓ←ℓu​\(λr𝙳\)​dℓu​\(p⁡\(x,r𝙴\),q⁡\(x,r𝙴\)\)\]\\displaystyle\\leq\\mathbb\{E\}\_\{\\mathsf\{x\},\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\},\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\}\\left\[\\iota\_\{\\ell\\leftarrow\\ell\_\{u\}\}\(\\lambda\_\{\{r\_\{\\mathtt\{D\}\}\}\}\)\\,d\_\{\\ell\_\{u\}\}\\left\(\{p\(x,\{r\_\{\\mathtt\{E\}\}\}\)\};\{q\(x,\{r\_\{\\mathtt\{E\}\}\}\)\}\\right\)\\right\]≤𝔼𝗑,𝗋𝙴,𝗋𝙳​\[\(ess​sup𝗋𝙳⁡ιℓ←ℓu​\(λr𝙳\)\)​dℓu​\(p⁡\(x,r𝙴\),q⁡\(x,r𝙴\)\)\]\\displaystyle\\leq\\mathbb\{E\}\_\{\\mathsf\{x\},\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\},\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\}\\left\[\\Big\(\\esssup\_\{\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\}\\iota\_\{\\ell\\leftarrow\\ell\_\{u\}\}\(\\lambda\_\{\{r\_\{\\mathtt\{D\}\}\}\}\)\\Big\)\\,d\_\{\\ell\_\{u\}\}\\left\(\{p\(x,\{r\_\{\\mathtt\{E\}\}\}\)\};\{q\(x,\{r\_\{\\mathtt\{E\}\}\}\)\}\\right\)\\right\]=Cd​𝔼𝗑,𝗋𝙴,𝗋𝙳​\[dℓu​\(p⁡\(x,r𝙴\),q⁡\(x,r𝙴\)\)\]\\displaystyle=C\_\{d\}\\,\\mathbb\{E\}\_\{\\mathsf\{x\},\\mathsf\{\{r\_\{\\mathtt\{E\}\}\}\},\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}\}\\left\[d\_\{\\ell\_\{u\}\}\\left\(\{p\(x,\{r\_\{\\mathtt\{E\}\}\}\)\};\{q\(x,\{r\_\{\\mathtt\{E\}\}\}\)\}\\right\)\\right\]=Cd​εf\.\\displaystyle=C\_\{d\}\\varepsilon\_\{f\}\.Combining with Lemma[4\.12](https://arxiv.org/html/2610.40934#S4.Thmtheorem12)gives the result\. ∎

We can go one step farther than Theorem[4\.14](https://arxiv.org/html/2610.40934#S4.Thmtheorem14)in settings whereℓu\\ell\_\{u\}is informally equivalent toℓ\\elland areff\-divergences\. In that setting, Lemma[4\.15](https://arxiv.org/html/2610.40934#S4.Thmtheorem15)states thatιℓ←ℓu≤1\\iota\_\{\\ell\\leftarrow\\ell\_\{u\}\}\\leq 1for all possible kernels\.

###### Lemma 4\.15\.

Supposeℓ:𝒫⁡\(𝒴\)×𝒴→ℝ¯\\ell:\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)\\times\\mathcal\{Y\}\\to\\overline\{\\mathbb\{R\}\}andℓu:𝒫⁡\(𝒵\)×𝒵→ℝ¯\\ell\_\{u\}:\\mathcal\{P\}\\left\(\\mathcal\{Z\}\\right\)\\times\\mathcal\{Z\}\\to\\overline\{\\mathbb\{R\}\}induce the same normalizedff\-divergence over their respective families\. Then for every Markov kernelλ:𝒵↝𝒴\\lambda:\{\\mathcal\{Z\}\}\\rightsquigarrow\{\\mathcal\{Y\}\},ιℓ←ℓu​\(λ\)≤1\\iota\_\{\\ell\\leftarrow\\ell\_\{u\}\}\(\\lambda\)\\leq 1\.

###### Proof\.

This follows exactly from the well\-established data\-processing inequality, which states that theff\-divergence between two random variables must always be at least that formed through stochastic garbling of those random variables by the same kernel\. ∎

In settings where our strictly proper scoring losses can also satisfy this property, Lemma[4\.15](https://arxiv.org/html/2610.40934#S4.Thmtheorem15)would thus let us offer an even tighter, more natural bound\. The log loss satisfies it, as its regret divergence is the KL divergence; moreover, on label spaces with at least three elements, the KL divergence is the only divergence that is both a Bregman divergence \(and hence the regret of a strictly proper scoring loss\) and anff\-divergence\([Amari, 2009](https://arxiv.org/html/2610.40934#bib.bib26);[Jiao et al\., 2014](https://arxiv.org/html/2610.40934#bib.bib27)\), so the log loss is essentially the only case in which Lemma[4\.15](https://arxiv.org/html/2610.40934#S4.Thmtheorem15)applies\. Theorem[4\.16](https://arxiv.org/html/2610.40934#S4.Thmtheorem16)uses this to state that ifℓ\\ellandℓu\\ell\_\{u\}are the ordinary log loss and𝙿\\mathtt\{P\}is both well\-founded anddd\-optimal, thenErrℓ​\(𝙿\)≤Δρ\+εf\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\)\\leq\\Delta\_\{\\rho\}\+\\varepsilon\_\{f\}\.

###### Theorem 4\.16\.

Let𝙿\\mathtt\{P\}be a well\-founded,dd\-optimal foundation model and letℓ\\ellandℓu\\ell\_\{u\}both be the ordinary log\-loss\. Then

Errℓ​\(𝙿\)≤Δρ\+εf\.\\mathrm\{Err\}\_\{\\ell\}\\left\(\\mathtt\{P\}\\right\)\\leq\\Delta\_\{\\rho\}\+\\varepsilon\_\{f\}\.

###### Proof\.

The regret divergence of the log loss is the KL divergence, anff\-divergence, so Lemma[4\.15](https://arxiv.org/html/2610.40934#S4.Thmtheorem15)givesιℓ←ℓu​\(λr𝙳\)≤1\\iota\_\{\\ell\\leftarrow\\ell\_\{u\}\}\(\\lambda\_\{\{r\_\{\\mathtt\{D\}\}\}\}\)\\leq 1for everyr𝙳\{r\_\{\\mathtt\{D\}\}\}, henceCd≤1C\_\{d\}\\leq 1, and the result follows from Theorem[4\.14](https://arxiv.org/html/2610.40934#S4.Thmtheorem14)\. ∎

### 4\.4Conclusion

In this Section, we have shown in Theorem[4\.11](https://arxiv.org/html/2610.40934#S4.Thmtheorem11)how to decompose single\-draw foundation model error into the irreducible representation error; design\-error terms corresponding to universal taskuu, target labelerddand realized decoder𝙳\\mathtt\{D\}; excess terms capturing how the approximate modelffinteracts with the various system components to exacerbate or ameliorate design errors; and a Bregman interaction term dictated by the geometries induced by the strictly proper scoring lossℓ\\elland distributions in use\. This decomposition permits intentional, justifiable design choices when constructing foundation models and explains what aspects of the system contribute to overall error\.

We further showed in Theorem[4\.14](https://arxiv.org/html/2610.40934#S4.Thmtheorem14)that in the case of a well\-founded,dd\-optimal model, we can further bound the total error by the sum of the irreducible representation error and a scaled version of the normal model error capturing how wellffapproximatesuu, where the scaling factor is dictated by geometric properties of both the target labelerddand the different losses between the model target space𝒵\\mathcal\{Z\}and the overall label space𝒴\\mathcal\{Y\}\. Next, in Section[5](https://arxiv.org/html/2610.40934#S5), we will show how to use Definition[3\.3](https://arxiv.org/html/2610.40934#S3.Thmtheorem3)to realize practical recipes to build foundation models in the real world\.

## 5Recipes to Build Foundation Models

When considering building foundation models, an astute reader may observe that Definition[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1)frames foundation models in a way that is identical to how one might frame a supervised\-learning, single\-task model to predict𝗒\\mathsf\{y\}from the joint input domain\(𝗑,𝗍\)\(\\mathsf\{x\},\\mathsf\{t\}\)\. This analogy would suggest that to train a foundation model, we should simply sample triples\(x,t,y\)\(x,t,y\)from𝗑,𝗍,𝗒\\mathsf\{x\},\\mathsf\{t\},\\mathsf\{y\}and train a model as we would normally, using all the advancements in ML/AI for supervised learning to power this endeavor\. This is, of course, not possible in general because we rarely, if ever, have overt access to the ability to sample from𝗑,𝗍,𝗒\\mathsf\{x\},\\mathsf\{t\},\\mathsf\{y\}\(though in some strange setting where this were possible, this would be a viable strategy\)\. In general, we will not even be able to sample from𝗑,𝗋,𝗒\\mathsf\{x\},\\mathsf\{r\},\\mathsf\{y\}, which would also permit us to learn a \(reduced\) foundation model\.

In this section, we instead ask, powered by Definitions[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1)or[3\.3](https://arxiv.org/html/2610.40934#S3.Thmtheorem3), how we can practically construct a foundation model for a domain𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}when𝒯\\mathcal\{T\}may be latent and𝗍\\mathsf\{t\}is unknown and cannot be sampled directly\. To answer this question, we provide two recipes, that each build on the following assumptions\.

Firstly, for the purposes of these recipes, we will assume thatρ\\rhois complete \(and in particular assume that𝒯\\mathcal\{T\}is naturally expressed in a tractable form, such that we can reliably uset∈𝒯t\\in\\mathcal\{T\}as an input to functions, etc\.\)\. As was shown in Section[4](https://arxiv.org/html/2610.40934#S4), this is equivalent to simply acknowledging that there is irreducible error due toρ\\rho, then simply modeling over the𝒳,ℛ,𝒴\\mathcal\{X\},\\mathcal\{R\},\\mathcal\{Y\}space instead of𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}and considering error in that lossy domain, so we can do this without loss of generality, and it will only serve to make algebra simpler\.

In addition, we assume two properties:

1. 1\.There exists a𝒵\\mathcal\{Z\}valued random variable𝗓\\mathsf\{z\}\(jointly distributed with all others\) such that𝗑,𝗓\\mathsf\{x\},\\mathsf\{z\}is sampleable, such that𝗑,𝗍,𝗒,𝗓\\mathsf\{x\},\\mathsf\{t\},\\mathsf\{y\},\\mathsf\{z\}are related via the Bayesian DAG shown in Figure[3](https://arxiv.org/html/2610.40934#S5.F3)\.
2. 2\.There exists a known, computable labeling functiong:𝒯×𝒵→𝒴g:\\mathcal\{T\}\\times\\mathcal\{Z\}\\to\\mathcal\{Y\}such that𝗒=g⁡\(𝗍,𝗓\)\\mathsf\{y\}=g\(\\mathsf\{t\},\\mathsf\{z\}\)almost surely\. Note that𝗓⟂𝗍\|𝗑\\mathsf\{z\}\\perp\\mathsf\{t\}\\mid\\mathsf\{x\}by Figure[3](https://arxiv.org/html/2610.40934#S5.F3), soggrecovers the true task kernel throughη\(x,t\)=g\(t,⋅\)♯p𝗓\|𝗑=x\\eta\(x,t\)=\{g\(t,\\cdot\)\}\_\{\\sharp\}\{p\_\{\\left\.\\mathsf\{z\}\\middle\|\\mathsf\{x\}=x\\right\.\}\}\.777If𝗒\|𝗍,𝗓\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{t\},\\mathsf\{z\}\\right\.is stochastic, all results below carry over by appending an independent seedξ∼U⁡\[0,1\]\\xi\\sim U\[0,1\]to𝗓\\mathsf\{z\}\.

𝗑\\mathsf\{x\}𝗓\\mathsf\{z\}𝗍\\mathsf\{t\}𝗒\\mathsf\{y\}Figure 3:An extended graphical model where labels𝗒\\mathsf\{y\}are generated from𝗑\\mathsf\{x\}through a random variable𝗓\\mathsf\{z\}\.### 5\.1\(Recipe 1\) “Generative” Foundation Models: Model𝗓\|𝗑\\mathsf\{z\}\|\\mathsf\{x\}, and Decode Labels Explicitly

In Recipe 1, which we will call our ‘‘generative foundation model recipe,’’888Note that the model here is actually learning a*discriminative*model over𝗓\\mathsf\{z\}in the classical sense, not a generative model, but we still call this recipe the “generative” recipe as this is the vehicle by which most “generative” foundation models today operate\.we do the following:

- •Define the universal taskuuto be predicting𝗓\|𝗑\\mathsf\{z\}\\mid\\mathsf\{x\}, with𝒱=𝒳\\mathcal\{V\}=\\mathcal\{X\}\.
- •Let𝙴⁡\(x,⋅\)=x\\mathtt\{E\}\(x,\\cdot\)=xsimply forward the first argument and discard the task\.
- •Letffbe any model trained on samples of\(x,z\)\(x,z\)to approximateuu\.
- •Let the decoder representation be complete, with the canonical choiceρD​\(t\)=δt\\rho\_\{D\}\(t\)=\\delta\_\{t\}, so thatrD=tr\_\{D\}=talmost surely\.
- •Let the labeler be the known labeling function,d⁡\(t,z\):=g⁡\(t,z\)d\(t,z\):=g\(t,z\)\. The target decoder is then𝙳⋆​\(q,t\)=g​\(t,⋅\)♯​q\\mathtt\{D\}^\{\\star\}\(q,t\)=\{g\(t,\\cdot\)\}\_\{\\sharp\}\{q\}\.
- •Let the decoder𝙳\\mathtt\{D\}be any procedure to approximate𝙳⋆\\mathtt\{D\}^\{\\star\}\(e\.g\., Monte Carlo: drawz∼qz\\sim qand outputg⁡\(t,z\)g\(t,z\), up to some budget\)\.

The autoregressive EHR foundation models discussed in Section[3\.2](https://arxiv.org/html/2610.40934#S3.SS2)illustrate this construction, with future patient trajectories serving as the universal\-task output\.

Then, as is shown in Theorem[5\.1](https://arxiv.org/html/2610.40934#S5.Thmtheorem1), the resulting foundation model given by\(ρ,𝙴,f,𝙳\)\(\\rho,\\mathtt\{E\},f,\\mathtt\{D\}\)is well\-founded in the sense of Definition[4\.4](https://arxiv.org/html/2610.40934#S4.Thmtheorem4), which means that the only remaining sources of error will be the excess terms, which capture error introduced byff’s poor approximation ofuu, the Decoder\-labeler ErrorΔ𝙳u\\Delta\_\{\\mathtt\{D\}\}^\{u\}, capturing𝙳\\mathtt\{D\}’s poor approximation of𝙳⋆\\mathtt\{D\}^\{\\star\}, and the Bregman interaction term\.

###### Theorem 5\.1\.

If a foundation model is built as in Recipe 1, then this model is well\-founded\.

###### Proof\.

By Assumption 2 and𝗓⟂𝗍\|𝗑\\mathsf\{z\}\\perp\\mathsf\{t\}\\mid\\mathsf\{x\}, with probability one under the joint law of\(x,t\)\(x,t\),

𝙳⋆\(u\(x\),t\)=g\(t,⋅\)♯p𝗓\|𝗑=x=g\(t,⋅\)♯p𝗓\|𝗑=x,𝗍=t=η\(x,t\)\.\\mathtt\{D\}^\{\\star\}\(u\(x\),t\)=\{g\(t,\\cdot\)\}\_\{\\sharp\}\{p\_\{\\left\.\\mathsf\{z\}\\middle\|\\mathsf\{x\}=x\\right\.\}\}=\{g\(t,\\cdot\)\}\_\{\\sharp\}\{p\_\{\\left\.\\mathsf\{z\}\\middle\|\\mathsf\{x\}=x,\\mathsf\{t\}=t\\right\.\}\}=\\eta\(x,t\)\.As𝗋𝙳=𝗍\\mathsf\{\{r\_\{\\mathtt\{D\}\}\}\}=\\mathsf\{t\}almost surely,𝗐\|𝗑,𝗋\\left\.\\mathsf\{w\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.and𝗒\|𝗑,𝗋\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\},\\mathsf\{r\}\\right\.are identically distributed, sodℓ\(𝗒;𝗐\|𝗑,𝗋\)=0d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\mathsf\{w\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)=0\. Since𝗐\\mathsf\{w\}is feasible in the deficiency definingΔu\\Delta\_\{u\}and deficiencies are non\-negative,Δu=0\\Delta\_\{u\}=0, and thereforeΔdu=0\\Delta\_\{d\}^\{u\}=0, establishing well\-foundedness\. ∎

#### Advantages to Recipe 1

This recipe has multiple advantages\. Firstly, it is guaranteed to be well\-founded, and thus iff=uf=uand𝙳=𝙳⋆\\mathtt\{D\}=\\mathtt\{D\}^\{\\star\}, the resulting procedure achieves Bayes\-optimal predictions under the stated assumptions\. In addition, if𝗑,𝗓\\mathsf\{x\},\\mathsf\{z\}are sampleable, this model requires no additional assumptions to learn\. Further, note that this model is universal in the sense that*any*task with a label inferrable from𝗓\\mathsf\{z\}can be inferred from this foundation model – not just those tasks in𝒯\\mathcal\{T\}, which will differ from the capabilities of models in Recipe 2\.

Because Recipe 1 models the full distribution of𝗓\|𝗑\\left\.\\mathsf\{z\}\\middle\|\\mathsf\{x\}\\right\., it can also answer*conditional*queries without retraining, by rejection sampling at inference time\. For instance, given taskst,t1,…,tk∈𝒯t,t\_\{1\},\\ldots,t\_\{k\}\\in\\mathcal\{T\}and labelsy1,…,yk∈𝒴y\_\{1\},\\ldots,y\_\{k\}\\in\\mathcal\{Y\}, letE:=\{z:g\(ti,z\)=yi,i=1,…,k\}⊆𝒵E:=\\\{z:g\(t\_\{i\},z\)=y\_\{i\},\\ i=1,\\ldots,k\\\}\\subseteq\\mathcal\{Z\}\. Drawingz∼f⁡\(x\)z\\sim f\(x\), discarding draws outsideEE, and labeling the accepted draws withg⁡\(t,⋅\)g\(t,\\cdot\)yieldsg\(t,⋅\)♯f\(x\)\(⋅∣E\)\{g\(t,\\cdot\)\}\_\{\\sharp\}\{f\(x\)\(\\cdot\\mid E\)\}wheneverf​\(x\)​\(E\)\>0f\(x\)\(E\)\>0, an approximation of the conditional label distributiong\(t,⋅\)♯u\(x\)\(⋅∣E\)\{g\(t,\\cdot\)\}\_\{\\sharp\}\{u\(x\)\(\\cdot\\mid E\)\}of taskttgiven that taskst1,…,tkt\_\{1\},\\ldots,t\_\{k\}take labelsy1,…,yky\_\{1\},\\ldots,y\_\{k\}\. Recipe 2 offers no analogous mechanism unless such conditional tasks are included in𝒯\\mathcal\{T\}during training\. A formal treatment of conditional tasks, including how the acceptance probabilityu​\(x\)​\(E\)u\(x\)\(E\)affects the error bounds of Section[4](https://arxiv.org/html/2610.40934#S4), is left to future work\.

#### Detriments to Recipe 1

There are also several key disadvantages to this recipe\. In particular, while this property of universality is nice, it also may induce needless inefficiency\. If the only tasks of interest are those in𝒯\\mathcal\{T\}, then any additional learning effort spent on how to transform from𝗑\\mathsf\{x\}to𝗓\\mathsf\{z\}in a manner that does not benefit learning𝗒\|𝗍,𝗑\\mathsf\{y\}\|\\mathsf\{t\},\\mathsf\{x\}is wasted\.

In addition, while well\-foundedness is nice, it is a relatively weak property\. Withoutdd\-optimality, which these models may not have, we cannot even relate total error to underlying model error in a principled manner, meaning that there may be interactions that exacerbate or ameliorate error betweenffand𝙳\\mathtt\{D\}that we cannot predict in advance with this theory\. More generally, the lack ofdd\-optimality and the requirement thatggbe used at inference time to translate from𝗓\\mathsf\{z\}to𝗒\\mathsf\{y\}impose additional sources of error and computational costs to the user, which can be severe\.

#### Non well\-founded Generative Foundation Models

Recipe 1 is constructed such that it is well\-founded, by virtue of the assumed Bayesian DAG shown in Figure[3](https://arxiv.org/html/2610.40934#S5.F3)\. We could, however, generalize our notion of this recipe to apply in settings even when there is an unobserved, “confounder” pathway to𝗒\\mathsf\{y\}from𝗑\\mathsf\{x\}independent of𝗓\\mathsf\{z\}, thereby violating the assumption that𝗒⟂𝗑\|𝗓,𝗍\\mathsf\{y\}\\perp\\mathsf\{x\}\|\\mathsf\{z\},\\mathsf\{t\}\. In such a setting, we could still build a model as in Recipe 1 – let𝙴⁡\(x,t\)=x\\mathtt\{E\}\(x,t\)=x, setuuto be our task of predicting𝗓\|𝗑\\mathsf\{z\}\|\\mathsf\{x\}, setρ𝙳​\(t\)=δt\{\\rho\_\{\\mathtt\{D\}\}\}\(t\)=\\delta\_\{t\}, and useggas our labeler, with𝙳\\mathtt\{D\}approximating the induced target decoder\. This model would still be a valid foundation model under Definition[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1), but it would no longer be guaranteed to be well\-founded, due precisely to the error induced by ignoring the pathway𝗑→𝗒\\mathsf\{x\}\\to\\mathsf\{y\}independent of𝗓\\mathsf\{z\}\. We will use the term “generative foundation models” in general to refer to any model that satisfies this form – in particular have𝙴⁡\(x,t\)=x\\mathtt\{E\}\(x,t\)=x, which permits an interpretation of the model in this way, and reserve*well\-founded generative foundation model*towards one that satisfies Recipe 1 as described fully \(and is therefore well\-founded\)\.

### 5\.2\(Recipe 2\) “Discriminative” Foundation Models — Sample over a proxy distributionqt∈𝒫⁡\(𝒯\)q\_\{t\}\\in\\mathcal\{P\}\\left\(\\mathcal\{T\}\\right\), then learng⁡\(𝗍′,𝗓\)\|𝗑,𝗍′g\(\\mathsf\{t\}^\{\\prime\},\\mathsf\{z\}\)\|\\mathsf\{x\},\\mathsf\{t\}^\{\\prime\}for𝗍′∼qt\\mathsf\{t\}^\{\\prime\}\\sim q\_\{t\}

In this recipe, which we will call the “task\-conditioned” recipe, we assume that we also have access to a proxy distributionqt∈𝒫⁡\(𝒯\)q\_\{t\}\\in\\mathcal\{P\}\\left\(\\mathcal\{T\}\\right\), from which we sample𝗍′\\mathsf\{t\}^\{\\prime\}independently of\(𝗑,𝗓\)\(\\mathsf\{x\},\\mathsf\{z\}\)\. We assume that, with probability one over the input𝗑\\mathsf\{x\}, the conditional distribution of𝗍\\mathsf\{t\}given𝗑\\mathsf\{x\}is absolutely continuous with respect toqtq\_\{t\}\. In other words, for such inputs, any measurable set of tasks assigned zero probability byqtq\_\{t\}also has zero probability under the original conditional task distribution\.

In this setting, we can sample\(x,z,t′\)\(x,z,t^\{\\prime\}\)from𝗑,𝗓,𝗍′\\mathsf\{x\},\\mathsf\{z\},\\mathsf\{t\}^\{\\prime\}, then computey′=g⁡\(t′,z\)y^\{\\prime\}=g\(t^\{\\prime\},z\); we write𝗒′:=g⁡\(𝗍′,𝗓\)\\mathsf\{y\}^\{\\prime\}:=g\(\\mathsf\{t\}^\{\\prime\},\\mathsf\{z\}\)for the induced label random variable\. To construct a foundation model, we then

- •Define the universal taskuuto be predicting𝗒′\|𝗑,𝗍′\\mathsf\{y\}^\{\\prime\}\\mid\\mathsf\{x\},\\mathsf\{t\}^\{\\prime\}, with𝒱=𝒳×𝒯\\mathcal\{V\}=\\mathcal\{X\}\\times\\mathcal\{T\}and𝒵=𝒴\\mathcal\{Z\}=\\mathcal\{Y\}\.
- •Let the encoder representationρE\\rho\_\{E\}be complete \(and canonicallyρE​\(𝗍\)=δ𝗍\\rho\_\{E\}\(\\mathsf\{t\}\)=\\delta\_\{\\mathsf\{t\}\}\), andE=IE=Ibe the identity\.
- •Letffbe any model trained on samples\(x,t′,y′\)\(x,t^\{\\prime\},y^\{\\prime\}\)to approximateuu\.
- •Let the decoder representation be degenerate, the labeler be the identity,dr𝙳​\(y\)=yd\_\{\{r\_\{\\mathtt\{D\}\}\}\}\(y\)=y, and let the realized decoder simply return the output offfdirectly:𝙳⁡\(q,⋅\)=q\\mathtt\{D\}\(q,\\cdot\)=q\.

Task\-conditioned models such as EveryQuery illustrate this construction, using structured task queries to directly predict downstream labels, subject to the stated assumptions on the proxy task distribution\.

Then, as we will show in Theorem[5\.2](https://arxiv.org/html/2610.40934#S5.Thmtheorem2), this construction yields a foundation model that is not only well\-founded but alsodd\-optimal, permitting us to reduce all error to model error through Theorem[4\.14](https://arxiv.org/html/2610.40934#S4.Thmtheorem14)\.

###### Theorem 5\.2\.

If a foundation model is built as in Recipe 2, then this model will be both well\-founded anddd\-optimal\.

###### Proof\.

dd\-optimality is clear, as𝙳⋆​\(q,r𝙳\)=id♯​q=q=𝙳⁡\(q,r𝙳\)\\mathtt\{D\}^\{\\star\}\(q,\{r\_\{\\mathtt\{D\}\}\}\)=\{\\mathrm\{id\}\}\_\{\\sharp\}\{q\}=q=\\mathtt\{D\}\(q,\{r\_\{\\mathtt\{D\}\}\}\)\.

Since𝗍′\\mathsf\{t\}^\{\\prime\}is independent of\(𝗑,𝗓\)\(\\mathsf\{x\},\\mathsf\{z\}\)and𝗒′=g⁡\(𝗍′,𝗓\)\\mathsf\{y\}^\{\\prime\}=g\(\\mathsf\{t\}^\{\\prime\},\\mathsf\{z\}\), the definition ofuugives

u\(x,t\)=g\(t,⋅\)♯p𝗓\|𝗑=xu\(x,t\)=\{g\(t,\\cdot\)\}\_\{\\sharp\}\{p\_\{\\left\.\\mathsf\{z\}\\middle\|\\mathsf\{x\}=x\\right\.\}\}with probability one when\(x,t\)\(x,t\)is sampled from the joint law of\(𝗑,𝗍′\)\(\\mathsf\{x\},\\mathsf\{t\}^\{\\prime\}\)\.

By the absolute continuity assumption, this equality also holds with probability one when\(x,t\)\(x,t\)is sampled from the original joint law of\(𝗑,𝗍\)\(\\mathsf\{x\},\\mathsf\{t\}\)\.

Since, by the labeling assumption,𝗒=g⁡\(𝗍,𝗓\)\\mathsf\{y\}=g\(\\mathsf\{t\},\\mathsf\{z\}\)almost surely, and𝗓⟂𝗍\|𝗑\\mathsf\{z\}\\perp\\mathsf\{t\}\\mid\\mathsf\{x\}, the right\-hand side equalsη⁡\(x,t\)\\eta\(x,t\)under that same joint law\. Thus,u⁡\(x,t\)=η⁡\(x,t\)u\(x,t\)=\\eta\(x,t\)almost surely\.

Since the task representation is complete and the encoder and labeler are identities,dℓ\(𝗒;𝗐\|𝗑,𝗋\)=0d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\mathsf\{w\}\}\\middle\|\{\\mathsf\{x\},\\mathsf\{r\}\}\\right\)=0\. As𝗐\\mathsf\{w\}is feasible in the deficiency definingΔu\\Delta\_\{u\}and deficiencies are non\-negative,Δu=0\\Delta\_\{u\}=0, and thereforeΔdu=0\\Delta\_\{d\}^\{u\}=0, establishing well\-foundedness\. ∎

#### Advantages of Recipe 2

Unlike Recipe 1, Recipe 2 is both well\-founded anddd\-optimal, meaning we can bound total error solely via model error, via Theorem[4\.14](https://arxiv.org/html/2610.40934#S4.Thmtheorem14)\. This means that we can rely on the extensive theory and practice of ordinary machine learning to learn foundation models via this recipe without concern for unmeasurable interactions between different error terms\. It also has the significant benefit that there is no additional computational burden on users at inference time beyond the modelffitself, as𝙳\\mathtt\{D\}is trivial\. It also does not waste learning energy on parts of𝗓\|𝗑\\mathsf\{z\}\|\\mathsf\{x\}that do not matter for any task – though this efficiency gain may be counteracted by the distribution shift ofqtq\_\{t\}discussed more below\.

#### Disadvantages of Recipe 2

The main detriment to this recipe is that this induces a source of distribution shift between training and testing, in that𝗍′≠𝗍\\mathsf\{t\}^\{\\prime\}\\neq\\mathsf\{t\}\. At optimality under the sampling distribution, the absolute continuity assumption transfers the almost\-sure equalityf=uf=uto the original joint law of\(𝗑,𝗍\)\(\\mathsf\{x\},\\mathsf\{t\}\), whereu=ηu=\\etaalmost surely by the proof of Theorem[5\.2](https://arxiv.org/html/2610.40934#S5.Thmtheorem2), but in a real\-world setting, this means that we cannot actually measureεf\\varepsilon\_\{f\}in an unbiased manner without the ability to sample from𝗍\\mathsf\{t\}directly, which we assume is impossible\. As such, though we can bound total error in theory, in practice we may still be left with uncertainty around what the true inference time error will be in practice, though we may be able to rely on the existing literature around distribution shift and limited ability to sample from𝗍\\mathsf\{t\}to ameliorate this in practice, which we leave as an area of future work\. In addition, this recipe is only universal to𝒯\\mathcal\{T\}– not to𝗓\\mathsf\{z\}as Recipe 1 is\.

Lastly, this recipe also has a source of learning inefficiency similar to Recipe 1 – but instead of learning parts of𝗓\|𝗑\\mathsf\{z\}\|\\mathsf\{x\}that do not matter for any tasks, Recipe 2 may introduce learning inefficiency by requiring the model to learn labels for proxy pairs\(x,t′\)\(x,t^\{\\prime\}\)that have very low probability under the original joint distribution of\(𝗑,𝗍\)\(\\mathsf\{x\},\\mathsf\{t\}\), since𝗍′\\mathsf\{t\}^\{\\prime\}is sampled independently of𝗑\\mathsf\{x\}\.

#### General Task\-conditioned foundation models\.

Much like how we generalize our notion of a generative model to include any model with an unused encoder representation, we can generalize our notion of a task\-conditioned model to non\-well\-founded settings by calling any foundation model with an unused*decoder*representation a “task\-conditioned” foundation model\.

### 5\.3Examples

Table[3](https://arxiv.org/html/2610.40934#S5.T3)illustrates how the universal\-task reduction applies to several representative foundation\-model designs introduced earlier\. For each example, the table gives one natural decomposition into encoder\-side task representationρE\\rho\_\{E\}, encoder𝙴\\mathtt\{E\}, universal\-task input space𝒱\\mathcal\{V\}, predictive modelff, universal\-task output space𝒵\\mathcal\{Z\}, decoder\-side task representationρD\\rho\_\{D\}, and decoder𝙳\\mathtt\{D\}\. These decompositions are not unique: in particular, preprocessing steps such as serialization or tokenization, as well as architectural encoder or decoder components, may be absorbed into𝙴\\mathtt\{E\}or𝙳\\mathtt\{D\}depending on the level of abstraction chosen\. The examples are therefore intended to illustrate how the framework can represent common foundation\-model designs rather than to prescribe a unique decomposition for each architecture\.

The designs in Table[3](https://arxiv.org/html/2610.40934#S5.T3)are also not intended to exhaust the possible realizations of foundation models under our definition\. Prompted language models provide one broad family of examples whenever natural\-language task specifications support any\-task inference over the domain of interest\. At the other extreme, recent decision models such as JEV expose a more explicitly task\-conditioned interface: a model is provided with an input state together with one or more structured, typed questions and returns probability distributions over their possible answers\. JEV is additionally trained using reinforcement learning for calibrated decisions \(RLCD\), which explicitly optimizes the quality and calibration of these probabilistic decisions\([Almeida, 2026](https://arxiv.org/html/2610.40934#bib.bib25)\)\. From the perspective of our framework, such systems are naturally interpreted as instances of, or closely related to, the task\-conditioned construction of Recipe 2\. These emerging designs further illustrate that foundation\-model behavior need not be tied to autoregressive text generation, but can instead arise from models trained to directly answer a sufficiently expressive space of task specifications\.

Clinical LLMEHR ARPFNEQMultitaskρE\\rho\_\{E\}
Encoder task repr\.Natural\-language prompt / instruction∅\\varnothingContext set\{\(xi,yi\)\}i=1n\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}Structured program / queryCategorical task index𝙴\\mathtt\{E\}
EncoderPair EHR inputxxwith prompt / instructionForward EHR inputxxPair queryxxwith context setPair EHR inputxxwith structured queryPair EHR inputxxwith task index𝒱\\mathcal\{V\}/vv
Universal\-task input𝒳×ℛE\\mathcal\{X\}\\times\\mathcal\{R\}\_\{E\}𝒳\\mathcal\{X\}𝒳×ℛE\\mathcal\{X\}\\times\\mathcal\{R\}\_\{E\}𝒳×ℛE\\mathcal\{X\}\\times\\mathcal\{R\}\_\{E\}𝒳×ℛE\\mathcal\{X\}\\times\\mathcal\{R\}\_\{E\}Training data
Typical pretraining dataClinical text / EHR notesLongitudinal structured EHR event sequencesTask episodes sampled from a prior: context set plus queryxxEHR examples paired with structured tasks and labelsEHR examples with labels for a fixed finite task setff
ModelAutoregressive clinical language modelAutoregressive patient\-trajectory modelPFN predictorTask\-conditioned EHR predictorShared multitask predictor𝒵\\mathcal\{Z\}/zz
Universal\-task outputGenerated clinical text / answer sequenceGenerated EHR trajectoryPrediction in𝒴\\mathcal\{Y\}Prediction in𝒴\\mathcal\{Y\}Prediction in𝒴\\mathcal\{Y\}ρD\\rho\_\{D\}
Decoder task repr\.∅\\varnothing, or output schema / formatStructured task program∅\\varnothing∅\\varnothing∅\\varnothing𝙳\\mathtt\{D\}
DecoderIdentity, or parse generated text into task outputApply task program to generated trajectory distributionIdentityIdentityIdentity𝒯\\mathcal\{T\}
Task spaceNatural\-language clinical tasks / instructionsEHR prediction / extraction tasksSupervised prediction tasks / data\-generating distributionsStructured EHR queries / programsFinite discrete task setExample papers
Representative works\([Peng et al\., 2023](https://arxiv.org/html/2610.40934#bib.bib24)\)\([Renc et al\., 2024](https://arxiv.org/html/2610.40934#bib.bib23)\)\([Hollmann et al\., 2025](https://arxiv.org/html/2610.40934#bib.bib12)\)\([Chandak et al\., 2026](https://arxiv.org/html/2610.40934#bib.bib1)\)\([Steinberg et al\., 2024](https://arxiv.org/html/2610.40934#bib.bib21)\)Recipe
Closest Sec\.[5](https://arxiv.org/html/2610.40934#S5)constructionHybrid: Recipe 2\-like input, Recipe 1\-like outputRecipe 1 \(Generative\)Recipe 2\-likeRecipe 2 \(Task\-conditioned\)Recipe 2\-likeTable 3:Illustrative decompositions of several foundation\-model paradigms under the universal\-task reduction framework\. Here,x∈𝒳x\\in\\mathcal\{X\}denotes the original input,y∈𝒴y\\in\\mathcal\{Y\}the downstream label,rE∈ℛEr\_\{E\}\\in\\mathcal\{R\}\_\{E\}andrD∈ℛDr\_\{D\}\\in\\mathcal\{R\}\_\{D\}the encoder\- and decoder\-side task representations,v∈𝒱v\\in\\mathcal\{V\}the universal\-task input, andz∈𝒵z\\in\\mathcal\{Z\}the universal\-task output\. We use a minimal functional decomposition in which𝙴\\mathtt\{E\}only assembles the original input and encoder\-side task representation, while𝙳\\mathtt\{D\}only performs the task\-specific transformation needed to map universal\-task predictions into the downstream label space\. Preprocessing, tokenization, embeddings, and architectural encoder or decoder components are therefore absorbed intoff\. The “Training data” and “Example papers” rows are included for comparison and are not themselves components of the formal reduction\. These decompositions are not unique\. The “Recipe” row indicates the closest structural correspondence to the constructions in Section[5](https://arxiv.org/html/2610.40934#S5)rather than an exact identification with a particular recipe\. In particular, clinical LLMs are shown as hybrid constructions: task information enters on the encoder side through natural\-language prompting, as in Recipe 2, while the model generates a general text output that may subsequently be mapped into the downstream label space, as in Recipe 1\.

## 6Foundation Model Evaluation is Mathematically Similar to Evaluation under Algorithmic Fairness

Given a foundation model𝙿\\mathtt\{P\}, we may also ask how we can evaluate𝙿\\mathtt\{P\}in a useful manner, given we care about all tasks in𝒯\\mathcal\{T\}in accordance with𝗍\\mathsf\{t\}\. In fact, this evaluation shows a striking mathematical similarity to evaluation of supervised learners in settings with constraints on algorithmic fairness \(Theorem[6\.1](https://arxiv.org/html/2610.40934#S6.Thmtheorem1)\)\. The motivation for this theorem is, informally, that many fairness results can be seen as general statements about how a model behaves over a partitioned input domain, where the sensitive attribute defines the partition, and we can realize that in evaluation of foundation models, we likewise care about a partitioned domain, where tasks determine the partition\.

###### Theorem 6\.1\.

Let𝙿\\mathtt\{P\}be a foundation model over extended domain𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}as defined throughout\.

Additionally, suppose we have a distinct domain𝒳′,𝒴′\\mathcal\{X\}^\{\\prime\},\\mathcal\{Y\}^\{\\prime\}, such that a modelf′:𝒳′→𝒫⁡\(𝒴′\)f^\{\\prime\}:\\mathcal\{X\}^\{\\prime\}\\to\\mathcal\{P\}\\left\(\\mathcal\{Y\}^\{\\prime\}\\right\)is a model of some task on that domain, with induced score𝗌′:=f′​\(𝗑′\)\\mathsf\{s\}^\{\\prime\}:=f^\{\\prime\}\(\\mathsf\{x\}^\{\\prime\}\)\. In this domain, supposeAAdefines a set of “attributes,” such that for each samplex′,y′x^\{\\prime\},y^\{\\prime\}, we can identifyaadictating the value of the “attribute” for this samplex′x^\{\\prime\}, realized through the random variables𝗑′,𝗒′,𝖺\\mathsf\{x\}^\{\\prime\},\\mathsf\{y\}^\{\\prime\},\\mathsf\{a\}\.

Suppose one can make a logical statementSSor formal theoretical proof about properties off′f^\{\\prime\}and its evaluation as dictated across the full partition of possible valuesa∈Aa\\in A– e\.g\., splitting the dataset into separate partitions, each defined by the observation of a given valueaafor𝖺\\mathsf\{a\}\. Then one can make an analogous statementSSabout𝙿\\mathtt\{P\}over𝒳,𝒴\\mathcal\{X\},\\mathcal\{Y\}, such that the partitions are induced by𝒯\\mathcal\{T\}instead ofAA, provided that any required assumptions for said logical statement also hold over the𝒳,𝒯,𝒴\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}domain with𝒯\\mathcal\{T\}standing in forAA\.

###### Proof\.

Such a statement is expressed entirely in terms of the joint distribution of the prediction, label, and partitioning variable\. Replacing the attribute𝖺\\mathsf\{a\}by the task𝗍\\mathsf\{t\}, the label𝗒′\\mathsf\{y\}^\{\\prime\}by𝗒\\mathsf\{y\}, and the score𝗌′\\mathsf\{s\}^\{\\prime\}by the foundation\-model score𝗌\\mathsf\{s\}preserves the form of all probabilities, conditional independences, and group\-conditioned quantities, provided that any required assumptions for the theorem are likewise met over this space \(as is an assumption in the theorem\)\. Hence the same argument applies with tasks indexing the partitions instead of attributes\. ∎

While basic, the utility of Theorem[6\.1](https://arxiv.org/html/2610.40934#S6.Thmtheorem1)is that many results stemming from the literature on algorithmic fairness can now be seen to have an analogous result for general foundation models, over “subpopulations” defined by disparate tasks, rather than protected attributes\. For example, Lemma[6\.2](https://arxiv.org/html/2610.40934#S6.Thmtheorem2)establishes an analog of the Impossibility Theorem in fairness for foundation models:

###### Lemma 6\.2\.

Suppose for simplicity that𝒴=\{0,1\}\\mathcal\{Y\}=\\\{0,1\\\}\. Then, let𝗌\\mathsf\{s\}be the random variable corresponding to the induced output of a foundation model𝙿⁡\(𝗑,𝗋\)\\mathtt\{P\}\(\\mathsf\{x\},\\mathsf\{r\}\)\(or𝙿⁡\(𝗑,𝗍\)\\mathtt\{P\}\(\\mathsf\{x\},\\mathsf\{t\}\)whenρ\\rhois complete\), realized as a distribution\-valued random variable jointly distributed with all other relevant variables, with𝗌\\mathsf\{s\}being treated as analogous to a “score” of the output of𝙿\\mathtt\{P\}\. In analogy to algorithmic fairness, define*sufficiency*to mean that𝗒⟂𝗍\|𝗌\\mathsf\{y\}\\perp\\mathsf\{t\}\|\\mathsf\{s\}and*separation*to mean that𝗌⟂𝗍\|𝗒\\mathsf\{s\}\\perp\\mathsf\{t\}\|\\mathsf\{y\}\.

Then, provided𝗒⟂̸𝗍\\mathsf\{y\}\\not\\perp\\mathsf\{t\}and𝙿\\mathtt\{P\}is not a perfect predictor \(H⁡\(𝗒\|𝗌\)\>0H\(\\mathsf\{y\}\|\\mathsf\{s\}\)\>0\), then𝙿\\mathtt\{P\}cannot simultaneously satisfy sufficiency and separation\.

###### Proof\.

This is a direct application of Theorem[6\.1](https://arxiv.org/html/2610.40934#S6.Thmtheorem1)to the well\-established impossibility theorem in algorithmic fairness\([Chouldechova, 2017](https://arxiv.org/html/2610.40934#bib.bib8)\)\. ∎

#### Example

Consider a multi\-task EHR model predicting outcomes with different prevalences, such as mortality and hospital readmission\. Treating task identity as the partitioning variable reveals that, under the conditions of Lemma[6\.2](https://arxiv.org/html/2610.40934#S6.Thmtheorem2), sufficiency and separation cannot generally hold simultaneously across tasks\. Thus, tradeoffs familiar from algorithmic fairness arise naturally when evaluating foundation models across heterogeneous prediction tasks\.

The novelty of this section is therefore not a new impossibility theorem in itself, but rather the observation that known fairness\-style tradeoffs can be reinterpreted as limitations of cross\-task evaluation, and hence apply not only to foundation models but to multi\-task evaluation more broadly\.

## References

- Achiamet al\.\(2023\)J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px1.p1.1)\.
- Almeida \(2026\)D\. AlmeidaIntroducing system one models & jev\.Note:TypeSafe AI BlogPublished September 15, 2026External Links:[Link](https://typesafe.ai/blog/introducing-system-one-models-and-jev)Cited by:[§5\.3](https://arxiv.org/html/2610.40934#S5.SS3.p2.1)\.
- Amari \(2009\)S\. Amariα\\alpha\-Divergence is unique, belonging to bothff\-divergence and Bregman divergence classes\.IEEE Transactions on Information Theory55\(11\),pp\. 4925–4931\.Cited by:[§4\.3](https://arxiv.org/html/2610.40934#S4.SS3.SSS0.Px2.p9.1)\.
- Attrachet al\.\(2026\)R\. A\. Attrach, R\. Fani, Y\. jia, D\. Restrepo, L\. A\. Celi, and P\. SchuefflerRethinking tokenization for clinical time series: when less is more\.InMachine Learning for Health 2025,External Links:[Link](https://openreview.net/forum?id=kdzXH4yxKs)Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px2.p1.1)\.
- Baxter \(2000\)J\. BaxterA model of inductive bias learning\.Journal of artificial intelligence research12,pp\. 149–198\.Cited by:[§2\.1](https://arxiv.org/html/2610.40934#S2.SS1.p1.1),[footnote 2](https://arxiv.org/html/2610.40934#footnote2)\.
- Bommasaniet al\.\(2021\)R\. Bommasani, D\. A\. Hudson, E\. Adeli, R\. B\. Altman, S\. Arora, S\. von Arx, M\. S\. Bernstein, J\. Bohg, A\. Bosselut, E\. Brunskill, E\. Brynjolfsson, S\. Buch, D\. Card, R\. Castellon, N\. S\. Chatterji, A\. S\. Chen, K\. Creel, J\. Q\. Davis, D\. Demszky, C\. Donahue, M\. Doumbouya, E\. Durmus, S\. Ermon, J\. Etchemendy, K\. Ethayarajh, L\. Fei\-Fei, C\. Finn, T\. Gale, L\. E\. Gillespie, K\. Goel, N\. D\. Goodman, S\. Grossman, N\. Guha, T\. Hashimoto, P\. Henderson, J\. Hewitt, D\. E\. Ho, J\. Hong, K\. Hsu, J\. Huang, T\. Icard, S\. Jain, D\. Jurafsky, P\. Kalluri, S\. Karamcheti, G\. Keeling, F\. Khani, O\. Khattab, P\. W\. Koh, M\. S\. Krass, R\. Krishna, R\. Kuditipudi, and et al\.On the opportunities and risks of foundation models\.CoRRabs/2108\.07258\.External Links:[Link](https://arxiv.org/abs/2108.07258),2108\.07258Cited by:[§1\.1](https://arxiv.org/html/2610.40934#S1.SS1.p1.1),[§3](https://arxiv.org/html/2610.40934#S3.p1.1)\.
- Brownet al\.\(2020\)T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px1.p1.1)\.
- Chandaket al\.\(2026\)P\. Chandak, G\. Kondas, L\. A\. Friedman, I\. Kohane, and M\. McDermottEveryQuery: zero\-shot clinical prediction via task\-conditioned pretraining over electronic health records\.External Links:2603\.07900,[Link](https://arxiv.org/abs/2603.07900)Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px4.p1.1),[§2\.2](https://arxiv.org/html/2610.40934#S2.SS2.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2610.40934#S5.T3.3.11.5.1.1)\.
- Chouldechova \(2017\)A\. ChouldechovaFair prediction with disparate impact: a study of bias in recidivism prediction instruments\.Big data5\(2\),pp\. 153–163\.Cited by:[§6](https://arxiv.org/html/2610.40934#S6.p4.1.1)\.
- Denget al\.\(2009\)J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and L\. Fei\-FeiImagenet: a large\-scale hierarchical image database\.In2009 IEEE conference on computer vision and pattern recognition,pp\. 248–255\.Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px4.p1.1)\.
- Devlinet al\.\(2019\)J\. Devlin, M\. Chang, K\. Lee, and K\. ToutanovaBert: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long and short papers\),pp\. 4171–4186\.Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px1.p1.1)\.
- Guoet al\.\(2026\)L\. L\. Guo, S\. E\. Arciniegas, J\. J\. Lee, A\. P\. Yan, G\. Tomlinson, J\. Fries, and L\. SungTokenization tradeoffs in structured ehr foundation models\.External Links:2603\.15644,[Link](https://arxiv.org/abs/2603.15644)Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px2.p1.1)\.
- Hollmannet al\.\(2025\)N\. Hollmann, S\. Müller, L\. Purucker, A\. Krishnakumar, M\. Körfer, S\. B\. Hoo, R\. T\. Schirrmeister, and F\. HutterAccurate predictions on small data with a tabular foundation model\.Nature637\(8045\),pp\. 319–326\.Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px3.p1.1),[Table 3](https://arxiv.org/html/2610.40934#S5.T3.3.11.4.1.1)\.
- Jiaoet al\.\(2014\)J\. Jiao, T\. A\. Courtade, A\. No, K\. Venkat, and T\. WeissmanInformation measures: the curious case of the binary alphabet\.IEEE Transactions on Information Theory60\(12\),pp\. 7616–7626\.Cited by:[§4\.3](https://arxiv.org/html/2610.40934#S4.SS3.SSS0.Px2.p9.1)\.
- Jinget al\.\(2026\)Z\. Jing, V\. Jeanselme, Y\. Kobayashi, S\. A\. Lee, C\. Pang, A\. Kashyap, Y\. Li, X\. Jiang, and S\. JoshiOne loss to rule them all: marked time\-to\-event for structured ehr foundation models\.External Links:2602\.00541,[Link](https://arxiv.org/abs/2602.00541)Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px4.p1.1)\.
- Leeet al\.\(2026\)I\. Lee, L\. Solo, M\. C\. Burkhart, B\. Ramadan, S\. Sethi, S\. Jabbour, W\. F\. Parker, and B\. K\. Beaulieu\-JonesRepresentation before training: a practical benchmark for generative medical event model tokenization\.External Links:2604\.16775,[Link](https://arxiv.org/abs/2604.16775)Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px2.p1.1)\.
- Liese and Vajda \(2006\)F\. Liese and I\. VajdaOn divergences and informations in statistics and information theory\.IEEE Transactions on Information Theory52\(10\),pp\. 4394–4412\.Cited by:[item 10](https://arxiv.org/html/2610.40934#A1.I4.i10.p1.1)\.
- McDermottet al\.\(2026\)M\. B\. McDermott, E\. Steinberg, J\. A\. Fries, R\. P\. van de Water, C\. Pang, P\. Rockenschaub, P\. Renc, J\. Oh, K\. Stankevičiūtė, J\. Xu,et al\.MEDS—an emerging data standard and ecosystem for health ai research\.NEJM AI3\(6\),pp\. AIra2501253\.Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px2.p1.1)\.
- McDermottet al\.\(2021\)M\. McDermott, B\. Nestor, E\. Kim, W\. Zhang, A\. Goldenberg, P\. Szolovits, and M\. GhassemiA comprehensive ehr timeseries pre\-training benchmark\.InProceedings of the Conference on Health, Inference, and Learning,CHIL ’21,New York, NY, USA,pp\. 257–278\.External Links:ISBN 9781450383592,[Link](https://doi.org/10.1145/3450439.3451877),[Document](https://dx.doi.org/10.1145/3450439.3451877)Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px4.p1.1)\.
- Mülleret al\.\(2022\)S\. Müller, N\. Hollmann, S\. P\. Arango, J\. Grabocka, and F\. HutterTransformers can do bayesian inference\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=KSugKcbNf9)Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px3.p1.1)\.
- Penget al\.\(2023\)C\. Peng, X\. Yang, A\. Chen, K\. E\. Smith, N\. PourNejatian, A\. B\. Costa, C\. Martin, M\. G\. Flores, Y\. Zhang, T\. Magoc,et al\.A study of generative large language model for medical research and healthcare\.npj Digital Medicine6\.Cited by:[Table 3](https://arxiv.org/html/2610.40934#S5.T3.3.11.2.1.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.Language models are unsupervised multitask learners\.OpenAI blog1\(8\),pp\. 9\.Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px1.p1.1)\.
- Raginsky \(2011\)M\. RaginskyShannon meets blackwell and le cam: channels, codes, and statistical experiments\.In2011 IEEE International Symposium on Information Theory Proceedings,pp\. 1220–1224\.Cited by:[item 10](https://arxiv.org/html/2610.40934#A1.I4.i10.p1.1)\.
- Rencet al\.\(2024\)P\. Renc, Y\. Jia, M\. K\. Grzeszczyk, T\. Pawlowski, E\. Alsentzer, M\. B\. McDermott, and I\. S\. KohaneZero shot health trajectory prediction using transformer\.npj Digital Medicine7,pp\. 260\.Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px2.p1.1),[§3\.2](https://arxiv.org/html/2610.40934#S3.SS2.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2610.40934#S5.T3.3.11.3.1.1)\.
- Steinberget al\.\(2024\)E\. Steinberg, J\. A\. Fries, Y\. Xu, and N\. ShahMOTOR: a time\-to\-event foundation model for structured medical records\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=NialiwI2V6)Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px4.p1.1),[Table 3](https://arxiv.org/html/2610.40934#S5.T3.3.11.6.1.1)\.
- Waxleret al\.\(2025\)S\. Waxler, P\. Blazek, D\. White, D\. Sneider, K\. Chung, M\. Nagarathnam, P\. Williams, H\. Voeller, K\. Wong, M\. Swanhorst, S\. Zhang, N\. Usuyama, C\. Wong, T\. Naumann, H\. Poon, A\. Loza, D\. Meeker, S\. Hain, and R\. ShahGenerative medical event models improve with scale\.External Links:2508\.12104,[Link](https://arxiv.org/abs/2508.12104)Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px2.p1.1)\.
- Woerneret al\.\(2026\)S\. Woerner, S\. J\. Oh, and C\. F\. BaumgartnerUniversal algorithm\-implicit learning\.External Links:2602\.14761,[Link](https://arxiv.org/abs/2602.14761)Cited by:[§1\.3](https://arxiv.org/html/2610.40934#S1.SS3.SSS0.Px3.p1.1)\.

## Appendix ANotation and Assumptions

#### Sets

1. 1\.We will use the notationℝ\\mathbb\{R\}to refer to the real numbers,ℝ\+\\mathbb\{R\}\_\{\+\}to refer to the strictly positive real numbers, andℝ¯\\overline\{\\mathbb\{R\}\}to refer to the extended reals including±∞\\pm\\infty\.

#### Typefaces / Styling

1. 1\.We will use a standard typeface \(xx\) to denote ordinary variables or functions\. These may be scalars or vectors or functions depending on context, as that distinction will be less essential for our purposes\.
2. 2\.We will use a calligraphic font \(𝒳\\mathcal\{X\}\) to denote generic, measurable sets, which we will typically use as the domain over which we will construct random variables\.
3. 3\.We will use a sans\-serif typeface \(𝗑\\mathsf\{x\}\) to denote random variables\. As a general rule, if we define multiple random variables, they should be understood to be jointly defined, even when that joint relationship has not yet been defined\.
4. 4\.We will use the monospace typeface𝙴\\mathtt\{E\}to represent a “procedure”, which is, mathematically, simply a function like any other, but we will denote it differently to reflect functions that will often be realized through more traditional computational algorithms rather than models that we might train via ML/AI methods, which we will denote with the more traditional notationfforgg\.

#### Probability

1. 1\.Given a set or boolean relationshipf:𝒳→\{True,False\}f:\\mathcal\{X\}\\to\\\{\\text\{True\},\\text\{False\}\\\}, we will use the notationP𝗑​\(f\)P\_\{\\mathsf\{x\}\}\\left\(f\\right\)to represent the ordinary probability thatx∈f−1​\(True\)x\\in f^\{\-1\}\(\\text\{True\}\)\.
2. 2\.We will use the notation𝒫⁡\(𝒴\)\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)to denote the set of all probability density/measure functions over the set𝒴\\mathcal\{Y\}\. Note that if we state thatq∈𝒫⁡\(𝒴\)q\\in\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\), that does not necessarily mean that we have easy ways to perform all operations overqq, such as sample or compute a conditional or expectation, but we will assume the ability to evaluate the probability law on given inputs\.
3. 3\.We will use the notationx∼𝗑x\\sim\\mathsf\{x\}to denote that the valuexxis drawn from the random variable𝗑\\mathsf\{x\}\. At times, we may also use the notationx∼px\\sim pfor somep∈𝒫⁡\(𝒳\)p\\in\\mathcal\{P\}\\left\(\\mathcal\{X\}\\right\)to denote thatxxis drawn from a random variable with probability density/mass functionpp\.
4. 4\.Unless otherwise specified, we will usep𝗑p\_\{\\mathsf\{x\}\}to denote the probability mass or density function of a random variable\.
5. 5\.We will writeℒ⁡\(𝗑\)\\mathcal\{L\}\(\\mathsf\{x\}\)for the law \(probability distribution\) of a random variable𝗑\\mathsf\{x\}, andℒ⁡\(𝗑∣E\)\\mathcal\{L\}\(\\mathsf\{x\}\\mid E\)orℒ⁡\(𝗑∣𝗐=w,E\)\\mathcal\{L\}\(\\mathsf\{x\}\\mid\\mathsf\{w\}=w,E\)for its conditional law given an eventEEof positive probability \(and, possibly, another random variable𝗐\\mathsf\{w\}\)\.
6. 6\.Given a functionffover the domain𝒳\\mathcal\{X\}, we will use the notation𝔼𝗑​\[f​\(x\)\]\\mathbb\{E\}\_\{\\mathsf\{x\}\}\\left\[f\(x\)\\right\]or𝔼𝗑​\[f\]\\mathbb\{E\}\_\{\\mathsf\{x\}\}\\left\[f\\right\]to denote the expectation of a functionffthat maps samples of𝗑\\mathsf\{x\}to a measurable output space over the random variable𝗑\\mathsf\{x\}\. In the case that the random variable or distribution does not clearly delineate the function’s input variable, we can use the expanded notation𝔼t∼𝗑​\[f​\(t\)\]\\mathbb\{E\}\_\{t\\sim\\mathsf\{x\}\}\\left\[f\(t\)\\right\]to indicate how the input function is applied to samples drawn from the random variable\.
7. 7\.Given a suitably nice functionffover a domain𝒳\\mathcal\{X\}and an𝒳\\mathcal\{X\}valued random variable𝗑\\mathsf\{x\}, we will letf⁡\(𝗑\)f\(\\mathsf\{x\}\)denote the random variable given by applying the functionffto samples drawn from𝗑\\mathsf\{x\}\. Relatedly, iff:𝒳→𝒴f:\\mathcal\{X\}\\to\\mathcal\{Y\}andq∈𝒫⁡\(𝒳\)q\\in\\mathcal\{P\}\\left\(\\mathcal\{X\}\\right\)is a probability distribution over𝒳\\mathcal\{X\}, then we will use the pushforward notationf♯​q∈𝒫⁡\(𝒴\)\{f\}\_\{\\sharp\}\{q\}\\in\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)to denote the distribution obtained were one to sample fromqqthen applyff\(equivalent to the distribution off⁡\(𝗑\)f\(\\mathsf\{x\}\)if𝗑∼q\\mathsf\{x\}\\sim q\)\.
8. 8\.We will use a vertical bar to indicate conditional relationships, applied both to random variables \(e\.g\.,𝗑\|𝗍=t\\left\.\\mathsf\{x\}\\middle\|\\mathsf\{t\}=t\\right\.indicates the conditional random variable of𝗑\\mathsf\{x\}given the random variable𝗍\\mathsf\{t\}takes valuet∈𝒯t\\in\\mathcal\{T\}, defined over a joint distribution over𝗑,𝗍\\mathsf\{x\},\\mathsf\{t\}\) or probability density functions \(e\.g\.,p𝗑\|𝗍=t\(x\)p\_\{\\left\.\\mathsf\{x\}\\middle\|\\mathsf\{t\}=t\\right\.\}\(x\)\)\.
9. 9\.Extending the conditional notation above to events, givenq∈𝒫⁡\(𝒴\)q\\in\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)and a measurable setS⊆𝒴S\\subseteq\\mathcal\{Y\}withq⁡\(S\)\>0q\(S\)\>0, we writeq\(⋅∣S\)∈𝒫\(𝒴\)q\(\\cdot\\mid S\)\\in\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)forqqconditioned onSS, i\.e\.,q⁡\(A∣S\):=q⁡\(A∩S\)/q⁡\(S\)q\(A\\mid S\):=q\(A\\cap S\)/q\(S\)for any measurableA⊆𝒴A\\subseteq\\mathcal\{Y\}\.
10. 10\.When it is clear from letter identity, we may simplify our conditional notation through the use of a subscript – e\.g\., let𝗑t=𝗑\|𝗍=t\\mathsf\{x\}\_\{t\}=\\left\.\\mathsf\{x\}\\middle\|\\mathsf\{t\}=t\\right\.\. If we do this for multiple random variables over the same conditional, they should be understood to be considered jointly conditioned with respect to the given variable, not two independently conditioned random variables – e\.g\., if we discuss𝗑t,𝗒t\\mathsf\{x\}\_\{t\},\\mathsf\{y\}\_\{t\}, this is equivalent to𝗑,𝗒\|𝗍=t\\left\.\\mathsf\{x\},\\mathsf\{y\}\\middle\|\\mathsf\{t\}=t\\right\.\.
11. 11\.When used without a target of the conditional, e\.g\.,𝗑\|𝗍\\left\.\\mathsf\{x\}\\middle\|\\mathsf\{t\}\\right\., this should be understood to be a stochastic kernel from𝒯\\mathcal\{T\}to the space of probability distributions over𝒳\\mathcal\{X\}, such that on inputt∈𝒯t\\in\\mathcal\{T\}, this evaluates to the typical𝗑\|𝗍=t\\left\.\\mathsf\{x\}\\middle\|\\mathsf\{t\}=t\\right\.\.
12. 12\.Given an𝒳\\mathcal\{X\}\-valued random variable𝗑\\mathsf\{x\}, we will letess​sup𝗑\\esssup\_\{\\mathsf\{x\}\}denote the essential supremum overx∼𝗑x\\sim\\mathsf\{x\}, which we can recall is defined for any measurable functionf:𝒳→ℝf:\\mathcal\{X\}\\to\\mathbb\{R\}to be ess​sup𝗑⁡f:=inf\{M∈ℝ¯:P𝗑​\(f⁡\(x\)\>M\)=0\}\.\\esssup\_\{\\mathsf\{x\}\}f:=\\inf\\left\\\{M\\in\\overline\{\\mathbb\{R\}\}:P\_\{\\mathsf\{x\}\}\\left\(f\(x\)\>M\\right\)=0\\right\\\}\.Informally, this is the supremum over the values thatffcan take on𝗑\\mathsf\{x\}that will be observed with non\-zero probability\.

#### Information Theory

1. 1\.We will use the traditional notationρ:𝒯↝ℛ\\rho:\{\\mathcal\{T\}\}\\rightsquigarrow\{\\mathcal\{R\}\}to denote thatρ\\rhodefines a jointly measurable Markov kernel from𝒯\\mathcal\{T\}toℛ\\mathcal\{R\}\(i\.e\., a stochastic function\), such that we could introduce a random variable𝗋t\\mathsf\{r\}\_\{t\}for eacht∈𝒯t\\in\\mathcal\{T\}drawn from𝗋t∼ρ⁡\(t\)\\mathsf\{r\}\_\{t\}\\sim\\rho\(t\)\. Given a random variable𝗍\\mathsf\{t\}over𝒯\\mathcal\{T\}, then we can further considerρ\\rhoto induce a random variable𝗋\\mathsf\{r\}\(without a subscript\) overℛ\\mathcal\{R\}such that the conditional distribution given𝗍=t\\mathsf\{t\}=tis defined precisely byρ\\rho\. In this case, we can see thatρ\\rhois precisely𝗋\|𝗍\\left\.\\mathsf\{r\}\\middle\|\\mathsf\{t\}\\right\.as introduced previously\. Similarly, in this context the use of𝗋t\\mathsf\{r\}\_\{t\}is consistent with our simplified subscript form for conditionals discussed above\. We will also use the notationℳ⁡\(𝗒\|𝗑\)\\mathcal\{M\}\\left\(\\mathsf\{y\}\\middle\|\\mathsf\{x\}\\right\)to represent the set of all possible Markov kernels from𝒳\\mathcal\{X\}to𝒴\\mathcal\{Y\}equipped with the measures induced by the given random variables\. At times we may use the notationℳ⁡\(𝒴\|𝒳\)\\mathcal\{M\}\\left\(\\mathcal\{Y\}\\middle\|\\mathcal\{X\}\\right\)equivalently when the measures are clear in context\.
2. 2\.We will use the variableℓ\\ellto represent a strictly proper scoring loss for𝒴\\mathcal\{Y\}, such thatℓ:𝒫⁡\(𝒴\)×𝒴→ℝ¯\\ell:\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)\\times\\mathcal\{Y\}\\to\\overline\{\\mathbb\{R\}\}\. We permit scoring rules and generalized entropies to take extended\-real values and assume all relevant expectations and differences are well defined and finite whenever invoked\.
3. 3\.Given a proper scoring ruleℓ\\elland a random variable𝗒\\mathsf\{y\}over𝒴\\mathcal\{Y\}, we will let the generalized entropyhℓ​\(𝗒\)=infq∈𝒫⁡\(𝒴\)𝔼𝗒​\[ℓ⁡\(q,y\)\]=𝔼𝗒​\[ℓ⁡\(p𝗒,y\)\]h\_\{\\ell\}\(\\mathsf\{y\}\)=\\inf\_\{q\\in\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)\}\\mathbb\{E\}\_\{\\mathsf\{y\}\}\\left\[\\ell\(q,y\)\\right\]=\\mathbb\{E\}\_\{\\mathsf\{y\}\}\\left\[\\ell\(p\_\{\\mathsf\{y\}\},y\)\\right\]\(where the last equality is guaranteed asℓ\\ellis proper\)\. In the event thatℓ\\ellis the log loss, then this is the Shannon entropy,hℓ=Hh\_\{\\ell\}=H\.
4. 4\.To produce a generalized notion of conditionalℓ\\ellentropy, we simply compute theℓ\\ellentropy over the conditional random variable in expectation over all possible conditioned values:hℓ​\(𝗑\|𝗍\)=𝔼𝗍​\[hℓ​\(𝗑\|𝗍=t\)\]h\_\{\\ell\}\(\\left\.\\mathsf\{x\}\\middle\|\\mathsf\{t\}\\right\.\)=\\mathbb\{E\}\_\{\\mathsf\{t\}\}\\left\[h\_\{\\ell\}\(\\left\.\\mathsf\{x\}\\middle\|\\mathsf\{t\}=t\\right\.\)\\right\]\. Note that this conditional entropy is*exactly*the score assigned byℓ\\ellfor a Bayes optimal predictor when predicting𝗑\\mathsf\{x\}from𝗍\\mathsf\{t\}over the joint distribution of𝗑,𝗍\\mathsf\{x\},\\mathsf\{t\}\. Similarly, we define the conditional generalizedℓ\\ell\-information asIℓ\(𝗒;𝗑∣𝗍\):=hℓ\(𝗒\|𝗍\)−hℓ\(𝗒\|𝗍,𝗑\)≥0\.I\_\{\\ell\}\(\\mathsf\{y\};\\mathsf\{x\}\\mid\\mathsf\{t\}\):=h\_\{\\ell\}\(\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{t\}\\right\.\)\-h\_\{\\ell\}\(\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{t\},\\mathsf\{x\}\\right\.\)\\geq 0\.
5. 5\.Similarly, we can define the associated regret divergence forℓ\\ellover two probability distributionsp,q∈𝒫⁡\(𝒴\)p,q\\in\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)asdℓ​\(p,q\):=𝔼𝗒∼p​\[ℓ⁡\(q,y\)−ℓ⁡\(p,y\)\]≥0d\_\{\\ell\}\\left\(\{p\};\{q\}\\right\):=\\mathbb\{E\}\_\{\\mathsf\{y\}\\sim p\}\\left\[\\ell\(q,y\)\-\\ell\(p,y\)\\right\]\\geq 0by propriety\. In the case of the log loss, this is the normal KL\-divergence\. At a slight risk of an abuse of notation, if𝗒\\mathsf\{y\}and𝗒^\\hat\{\\mathsf\{y\}\}are two𝒴\\mathcal\{Y\}valued random variables, we can use the random variables rather than probability laws to reflect the expected quantity:dℓ​\(𝗒,𝗒^\):=𝔼𝗒​\[ℓ⁡\(p𝗒^,y\)−ℓ⁡\(p𝗒,y\)\]d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{y\}\}\}\\right\):=\\mathbb\{E\}\_\{\\mathsf\{y\}\}\\left\[\\ell\(p\_\{\\hat\{\\mathsf\{y\}\}\},y\)\-\\ell\(p\_\{\\mathsf\{y\}\},y\)\\right\]\.
6. 6\.We can take the typical convention of supporting divergences between conditional kernels as reflecting an expectation over the conditioning value in a context where all variables are jointly defined:dℓ\(𝗒\|𝗑;𝗒^\|𝗑\):=𝔼𝗑\[dℓ\(𝗒\|𝗑=x;𝗒^\|𝗑=x\)\]d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\|\\mathsf\{x\}\};\{\\hat\{\\mathsf\{y\}\}\|\\mathsf\{x\}\}\\right\):=\\mathbb\{E\}\_\{\\mathsf\{x\}\}\\left\[d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\|\\mathsf\{x\}=x\};\{\\hat\{\\mathsf\{y\}\}\|\\mathsf\{x\}=x\}\\right\)\\right\]\. We will also take as the convention thatdℓ\(𝗒;𝗒^\|𝗑\):=dℓ\(𝗒\|𝗑;𝗒^\|𝗑\)d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\hat\{\\mathsf\{y\}\}\}\\middle\|\{\\mathsf\{x\}\}\\right\):=d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\|\\mathsf\{x\}\};\{\\hat\{\\mathsf\{y\}\}\|\\mathsf\{x\}\}\\right\), to simplify notation\.999Note that this notation is unambiguous as we cannot take the divergence between a distribution over𝒴\\mathcal\{Y\}alone and a conditional kernel inℳ⁡\(𝒴\|𝒳\)\\mathcal\{M\}\\left\(\\mathcal\{Y\}\\middle\|\\mathcal\{X\}\\right\), which is what the interpretation of this expression as being a divergence between\(𝗒\)\(\\mathsf\{y\}\)and\(𝗒\|𝗑\)\(\\mathsf\{y\}\|\\mathsf\{x\}\)would imply\.
7. 7\.As these divergences are Bregman regrets, we can also recall the three\-point rule, which relates the pairwise divergences between three random variables over the same space via a \(signed\) Bregman interaction term\. Concretely, if𝗐\\mathsf\{w\},𝗎\\mathsf\{u\}, and𝗏\\mathsf\{v\}are all random variables over𝒴\\mathcal\{Y\}, all jointly distributed with𝗑\\mathsf\{x\}over a potentially different space, then we can define the unconditioned and conditioned Bregman interaction terms via: Bℓ​\(𝗐,𝗎,𝗏\)\\displaystyle B\_\{\\ell\}\\left\(\{\\mathsf\{w\}\};\{\\mathsf\{u\}\};\{\\mathsf\{v\}\}\\right\):=dℓ​\(𝗐,𝗏\)−\(dℓ​\(𝗐,𝗎\)\+dℓ​\(𝗎,𝗏\)\)\\displaystyle:=d\_\{\\ell\}\\left\(\{\\mathsf\{w\}\};\{\\mathsf\{v\}\}\\right\)\-\(d\_\{\\ell\}\\left\(\{\\mathsf\{w\}\};\{\\mathsf\{u\}\}\\right\)\+d\_\{\\ell\}\\left\(\{\\mathsf\{u\}\};\{\\mathsf\{v\}\}\\right\)\)Bℓ\(𝗐;𝗎;𝗏\|𝗑\)\\displaystyle B\_\{\\ell\}\\left\(\{\\mathsf\{w\}\};\{\\mathsf\{u\}\};\{\\mathsf\{v\}\}\\middle\|\{\\mathsf\{x\}\}\\right\):=dℓ\(𝗐;𝗏\|𝗑\)−\(dℓ\(𝗐;𝗎\|𝗑\)\+dℓ\(𝗎;𝗏\|𝗑\)\)\\displaystyle:=d\_\{\\ell\}\\left\(\{\\mathsf\{w\}\};\{\\mathsf\{v\}\}\\middle\|\{\\mathsf\{x\}\}\\right\)\-\(d\_\{\\ell\}\\left\(\{\\mathsf\{w\}\};\{\\mathsf\{u\}\}\\middle\|\{\\mathsf\{x\}\}\\right\)\+d\_\{\\ell\}\\left\(\{\\mathsf\{u\}\};\{\\mathsf\{v\}\}\\middle\|\{\\mathsf\{x\}\}\\right\)\)Note that in the differentiable case, these Bregman interaction terms correspond explicitly to geometric quantities that can be independently expressed, but this interpretation will not be essential for us here\.
8. 8\.We can further define the generalized mutual information forℓ\\ellvia: Iℓ​\(𝗒,𝗑\)\\displaystyle I\_\{\\ell\}\(\\mathsf\{y\};\\mathsf\{x\}\)=hℓ​\(𝗒\)−hℓ​\(𝗒\|𝗑\)\\displaystyle=h\_\{\\ell\}\(\\mathsf\{y\}\)\-h\_\{\\ell\}\(\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\}\\right\.\)=𝔼𝗑\[dℓ\(p𝗒\|𝗑=x;p𝗒\)\]\.\\displaystyle=\\mathbb\{E\}\_\{\\mathsf\{x\}\}\\left\[d\_\{\\ell\}\\left\(\{p\_\{\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\}=x\\right\.\}\};\{p\_\{\\mathsf\{y\}\}\}\\right\)\\right\]\.which is the improvement in Bayes\-optimal predictive loss underℓ\\ellfrom using𝗑\\mathsf\{x\}to predict𝗒\\mathsf\{y\}vs\. using𝗒\\mathsf\{y\}alone\.
9. 9\.Given two kernels,k1:𝒳↝𝒴k\_\{1\}:\{\\mathcal\{X\}\}\\rightsquigarrow\{\\mathcal\{Y\}\}andk2:𝒴↝𝒵k\_\{2\}:\{\\mathcal\{Y\}\}\\rightsquigarrow\{\\mathcal\{Z\}\}, we will use the term “stochastic push\-forward” ofk1k\_\{1\}throughk2k\_\{2\}to refer to the compositionk2∘k1:𝒳↝𝒵k\_\{2\}\\circ k\_\{1\}:\{\\mathcal\{X\}\}\\rightsquigarrow\{\\mathcal\{Z\}\}such thatk2∘k1:x↦𝔼y∼k1​\(x\)​\[k2​\(y\)\]k\_\{2\}\\circ k\_\{1\}:x\\mapsto\\mathbb\{E\}\_\{y\\sim k\_\{1\}\(x\)\}\\left\[k\_\{2\}\(y\)\\right\]\. We extend this notation in two ways\. First, we identify a distributionp∈𝒫⁡\(𝒴\)p\\in\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)with the constant kernel from a one\-point space, so thatk2∘p:=𝔼y∼p​\[k2​\(y\)\]∈𝒫⁡\(𝒵\)k\_\{2\}\\circ p:=\\mathbb\{E\}\_\{y\\sim p\}\\left\[k\_\{2\}\(y\)\\right\]\\in\\mathcal\{P\}\\left\(\\mathcal\{Z\}\\right\)\. Second, given a𝒴\\mathcal\{Y\}\-valued random variable𝗒\\mathsf\{y\}, we writek2∘𝗒k\_\{2\}\\circ\\mathsf\{y\}for the𝒵\\mathcal\{Z\}\-valued random variable obtained by drawing fromk2​\(𝗒\)k\_\{2\}\(\\mathsf\{y\}\), jointly distributed with all others and conditionally independent of them given𝗒\\mathsf\{y\}; its law isk2∘p𝗒k\_\{2\}\\circ p\_\{\\mathsf\{y\}\}, and, by this conditional independence,\(k2∘𝗒\)\|𝗑=k2∘\(𝗒\|𝗑\)\\left\.\(k\_\{2\}\\circ\\mathsf\{y\}\)\\middle\|\\mathsf\{x\}\\right\.=k\_\{2\}\\circ\(\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\}\\right\.\)for any other random variable𝗑\\mathsf\{x\}\. Whenk2k\_\{2\}is induced by a measurable functiong:𝒴→𝒵g:\\mathcal\{Y\}\\to\\mathcal\{Z\}, i\.e\.,k2​\(y\)=δg⁡\(y\)k\_\{2\}\(y\)=\\delta\_\{g\(y\)\}, these reduce tog♯​k1​\(x\)\{g\}\_\{\\sharp\}\{k\_\{1\}\(x\)\},g♯​p\{g\}\_\{\\sharp\}\{p\}, and the random variableg⁡\(𝗒\)g\(\\mathsf\{y\}\)\(with lawg♯​p𝗒\{g\}\_\{\\sharp\}\{p\_\{\\mathsf\{y\}\}\}\), respectively\.
10. 10\.We can further define the notion of anℓ\\ell\-deficiency, in the sense of statistical deficiencies in stochastic channels, a la[Liese and Vajda \(2006\)](https://arxiv.org/html/2610.40934#bib.bib9);[Raginsky \(2011\)](https://arxiv.org/html/2610.40934#bib.bib10)\. Let𝗑,𝗒,𝗓\\mathsf\{x\},\\mathsf\{y\},\\mathsf\{z\}be jointly defined random variables taking values in𝒳,𝒴,𝒵\\mathcal\{X\},\\mathcal\{Y\},\\mathcal\{Z\}, respectively\. Note that the typical conditional distributions𝗒\|𝗑\\mathsf\{y\}\|\\mathsf\{x\}and𝗓\|𝗑\\mathsf\{z\}\|\\mathsf\{x\}form kernelskx→y:𝒳↝𝒴k\_\{x\\to y\}:\{\\mathcal\{X\}\}\\rightsquigarrow\{\\mathcal\{Y\}\}such thatkx→y​\(x\)=p𝗒\|𝗑=xk\_\{x\\to y\}\(x\)=p\_\{\\mathsf\{y\}\|\\mathsf\{x\}=x\}andkx→z:𝒳↝𝒵k\_\{x\\to z\}:\{\\mathcal\{X\}\}\\rightsquigarrow\{\\mathcal\{Z\}\}mapsx↦p𝗓\|𝗑=xx\\mapsto p\_\{\\mathsf\{z\}\|\\mathsf\{x\}=x\}\. We define theℓ\\elldeficiency of𝗓\\mathsf\{z\}relative to𝗒\\mathsf\{y\}to be the minimum possible error that could be obtained were one to predict𝗒\|𝗑\\mathsf\{y\}\|\\mathsf\{x\}through the action of any stochastic kernel inℳ⁡\(𝒴\|𝒵\)\\mathcal\{M\}\\left\(\\mathcal\{Y\}\\middle\|\\mathcal\{Z\}\\right\): δℓ\(𝗒;𝗓\|𝗑\)\\displaystyle\\delta\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\mathsf\{z\}\}\\middle\|\{\\mathsf\{x\}\}\\right\):=δℓ​\(kx→y,kx→z\)\\displaystyle:=\\delta\_\{\\ell\}\\left\(\{k\_\{x\\to y\}\};\{k\_\{x\\to z\}\}\\right\)=infλ∈ℳ⁡\(𝒴\|𝒵\)dℓ\(𝗒;λ∘𝗓\|𝗑\)\\displaystyle=\\inf\_\{\\lambda\\in\\mathcal\{M\}\\left\(\\mathcal\{Y\}\\middle\|\\mathcal\{Z\}\\right\)\}d\_\{\\ell\}\\left\(\{\\mathsf\{y\}\};\{\\lambda\\circ\\mathsf\{z\}\}\\middle\|\{\\mathsf\{x\}\}\\right\)=infλ∈ℳ⁡\(𝒴\|𝒵\)𝔼𝗑\[dℓ\(p𝗒\|𝗑=x;λ∘kx→z\(x\)\)\]\.\\displaystyle=\\inf\_\{\\lambda\\in\\mathcal\{M\}\\left\(\\mathcal\{Y\}\\middle\|\\mathcal\{Z\}\\right\)\}\\mathbb\{E\}\_\{\\mathsf\{x\}\}\\left\[d\_\{\\ell\}\\left\(\{p\_\{\\left\.\\mathsf\{y\}\\middle\|\\mathsf\{x\}=x\\right\.\}\};\{\\lambda\\circ k\_\{x\\to z\}\(x\)\}\\right\)\\right\]\.

#### Functions

1. 1\.We will slightly modify the traditional notation𝒴𝒳\{\\mathcal\{Y\}\}^\{\\mathcal\{X\}\}to denote not the set of functions from𝒳\\mathcal\{X\}to𝒴\\mathcal\{Y\}, but instead, in cases where𝒳\\mathcal\{X\}and𝒴\\mathcal\{Y\}are measurable, which will nearly universally be true in this work, the set of measurable functions from𝒳\\mathcal\{X\}to𝒴\\mathcal\{Y\},Meas⁡\(𝒳,𝒴\)\\mathrm\{Meas\}\(\\mathcal\{X\},\\mathcal\{Y\}\)\. We will also use the notationg:𝒳→𝒴g:\\mathcal\{X\}\\to\\mathcal\{Y\}as is typical to denote thatg∈𝒴𝒳g\\in\{\\mathcal\{Y\}\}^\{\\mathcal\{X\}\}\(again assuming measurability where applicable\)\.

#### Assumptions

In general, where applicable, we will assume that sets/spaces are not only measurable, but are standard Borel spaces, and assume that random variables / probability distributions defined in the context of a known joint can be used interchangeably \(e\.g\., we can identify a random variable with its probability distribution\), that conditionals are well defined, and that statements made will be assumed to hold only almost surely \(a\.s\.\) in the probabilistic sense unless otherwise specified\.

## Appendix BAny\-task vs\. Zero\-shot

We mentioned in the Introduction that we use the term “any\-task inference” rather than “zero\-shot inference” to avoid confusion, but here we can actually say more formally that these two requirements are very similar, and arguably identical in all domains where𝗍\\mathsf\{t\}is a continuous random variable everywhere\. Concretely, in such a case, if we satisfy the two restrictions given above \(can support inference for any task, no fine\-tuning\), then every evaluation we perform will be on tasks𝗍\\mathsf\{t\}that were almost surely101010Meaning that the statement holds with probability one\.never seen during training \(by continuity of𝗍\\mathsf\{t\}\), implying that every task we assess will be assessed in a zero\-shot manner\. In a setting where𝗍\\mathsf\{t\}is not continuous \(e\.g\.,𝒯\\mathcal\{T\}is finite\), then this may not hold\. However, in this setting, the notion of “true zero\-shot capabilities” \(without assuming some notion of distribution shift to a new space of task𝒯\\mathcal\{T\}\) is somewhat strange, as the space of tasks is fixed and finite, so there is nothing preventing a model from literally memorizing every single task that could ever exist directly with no risk of generalization error\. Regardless, we use the term “any\-task inference” here to focus squarely on the evaluation side, rather than on the training side, and avoid any confusion on this point\.

## Appendix CFoundation Models vs\. Supervised, Single\-task Models

Definition[3\.1](https://arxiv.org/html/2610.40934#S3.Thmtheorem1)implies the trivial, known results that the best possible foundation model has the same Bayes\-optimal solution as the best possible supervised, single\-task model for a given taskt∈𝒯t\\in\\mathcal\{T\}, and thus has the same optimal error/performance\. This is an obvious result, but it is worth stating explicitly to dispel any confusion that foundation models are somehow better than supervised learning models at optimality\. Instead, their advantage must come in efficiency, learnability, flexibility, or along some other axis in practice\.

In fact, we can go one step further than this\. Suppose we have𝒯′⊆𝒯\\mathcal\{T\}^\{\\prime\}\\subseteq\\mathcal\{T\}also measurable, with𝗍′\\mathsf\{t\}^\{\\prime\}defined such that𝗍\|t∈𝒯′\\mathsf\{t\}\|t\\in\\mathcal\{T\}^\{\\prime\}, and𝗑,𝗒\\mathsf\{x\},\\mathsf\{y\}as usual, yielding two possible domains,D=\(𝒳,𝒯,𝒴\)D=\(\\mathcal\{X\},\\mathcal\{T\},\\mathcal\{Y\}\)andD′=\(𝒳,𝒯′,𝒴\)D^\{\\prime\}=\(\\mathcal\{X\},\\mathcal\{T\}^\{\\prime\},\\mathcal\{Y\}\)with the usual associated random variables\. Further, supposeℱ⊆𝒫​\(𝒴\)𝒳×𝒯\\mathcal\{F\}\\subseteq\{\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)\}^\{\\mathcal\{X\}\\times\\mathcal\{T\}\}be some function class of interest within the space of all foundation models forDDandℱ′⊆𝒫​\(𝒴\)𝒳×𝒯′\\mathcal\{F\}^\{\\prime\}\\subseteq\{\\mathcal\{P\}\\left\(\\mathcal\{Y\}\\right\)\}^\{\\mathcal\{X\}\\times\\mathcal\{T\}^\{\\prime\}\}be a function class of interest within the space of foundation models forD′D^\{\\prime\}\. For example, suppose that these function classes are defined both by operational constraints \(e\.g\., neural networks with no more than a certain number of parameters\), but also by “admissibility” constraints based on performance \(e\.g\., the resulting foundation model will only be used if it achieves a certain average performance across all tasks within the domain, and otherwise it would not be considered successful\)\.

In this setting, if we were to consider a foundation model overDDto be “permitted” in general, it must be withinℱ\\mathcal\{F\}\. If we were to then evaluate this model over𝒯′⊆𝒯\\mathcal\{T\}^\{\\prime\}\\subseteq\\mathcal\{T\}alone and determine if it is “permitted” for the restricted domainD′D^\{\\prime\}, we would make this conclusion based on whether or not it is also withinℱ′\\mathcal\{F\}^\{\\prime\}\. As such, we can say that a foundation model overDDthen considered as a solution forD′D^\{\\prime\}would be allowed only if it lives withinℱ∩ℱ′⊆ℱ′\\mathcal\{F\}\\cap\\mathcal\{F\}^\{\\prime\}\\subseteq\\mathcal\{F\}^\{\\prime\}\. This therefore trivially implies that were we to consider the set of all foundation models permitted underD′D^\{\\prime\}alone vs\. those permitted underDDandD′D^\{\\prime\}, the best possible performance of a model overD′D^\{\\prime\}alone must be at least as good as that of one permitted under bothDDandD′D^\{\\prime\}\.

Concretely, as an example, this implies that if we train a single task model for a taskt∈𝒯t\\in\\mathcal\{T\}under any acceptance criteria \(e\.g\., it must obtain a certain performance and have a certain capacity limitation\), and alternatively train a foundation model over𝒯\\mathcal\{T\}, asserting that it must likewise satisfy the capacity limitation and attain a similar performance ontt, but also satisfy some more general constraints due to its “foundation model” nature, then it is theoretically clear that it is impossible for the optimal such foundation model to outperform the optimal such single\-task model\. Instead, any performance difference we hope to achieve between single\-task and foundation models must be accompanied by differences in prescribed criteria, such as permitting the foundation model to leverage additional capacity\.