大型语言模型的概率结构

# The Probabilistic Structure of Large Language Models
Source: [https://arxiv.org/html/2609.25134](https://arxiv.org/html/2609.25134)
###### Abstract

This paper presents a probabilistic perspective on large language models \(LLMs\), developed with the aim of bringing together, in a single self\-contained account, tools that are usually treated separately across the literature\. LLMs are described through probability measures on the set of sequences of tokens, specified via their autoregressive conditional distributions\. Training is formulated as a maximum\-likelihood estimation problem, addressed by stochastic gradient methods, while text generation is viewed as the sequential simulation of the resulting stochastic process\. The role of the asymmetry of the Kullback–Leibler divergence in text generation is examined in relation with characteristic phenomena such as hallucination and the distinction between statistical plausibility and truth\. As a complementary illustration of the same viewpoint, we also discuss diffusion models, built around the score function, which cast generation not as sequential token prediction but as the simulation of a reverse\-time stochastic process transforming noise into data both in discrete and continuous time\.

Key\-words: Autoregressive Processes, Markov Chains, Maximum Likelihood Estimation, Stochastic Simulation, Kullback–Leibler Divergence, Diffusion Models, Score\-Based Generative Modeling, Stochastic Differential Equations, Large Language Models

2020 Mathematics Subject Classification\.Primary 60J10, 60J60, 62F10; Secondary 60H10, 94A17, 68T50\.

## 1Introduction

Generative artificial intelligence has moved, within a few years, from a research topic to an instrument of work\. Systems that produce text, code, images and speech are now used daily across science, engineering, education, medicine and in many service and industrial sectors ; the economic and social consequences of this diffusion are already substantial and still unfolding\.

Generative AI encompasses different probabilistic paradigms, and two families of models account for most of this activity: on one hand, autoregressive language models, which generate text and code one token at a time via sequential sampling from conditional distributions; on the other hand, diffusion models, which generate images, audio and video via the reverse simulation of a stochastic process transforming noise into data\. This paper is concerned with that first family\. A later section extends the discussion to diffusion models, developed in enough depth to exhibit their own probabilistic machinery — score functions, stochastic differential equations, and reverse\-time simulation — so as to illustrate, from a different generative paradigm, the same interplay between a measure to be estimated and a stochastic process to be simulated\.

At its core, the LLM concept admits a natural formulation within the framework of probability theory\. Let𝒱\\mathcal\{V\}denote a finite vocabulary and𝒱∗\\mathcal\{V\}^\{\*\}the set of finite sequences of tokens over𝒱\\mathcal\{V\}\. A language model defines a probability distribution on such sequences and, in the autoregressive setting, is characterized by conditional probabilities of the form

ℙ⁡\(Xt=wt∣X<t=w<t\),\\mathbb\{P\}\\\!\\left\(X\_\{t\}=w\_\{t\}\\mid X\_\{<t\}=w\_\{<t\}\\right\),whereXtX\_\{t\}denotes the token generated at positionttandX<tX\_\{<t\}its preceding context\. A corpus𝒟\\mathcal\{D\}provides an empirical distributionP^N\\widehat\{P\}\_\{N\}, while an underlying, generally unknown distributionP⋆P^\{\\star\}may be regarded as the data\-generating distribution\. A parametric family\{Pθ:θ∈Θ\}\\\{P\_\{\\theta\}:\\theta\\in\\Theta\\\}is then introduced to model this distribution\. At the sequence level, the autoregressive factorization reads

Pθ​\(w1,…,wT\)=∏t=1TPθ​\(wt∣w1,…,wt−1\)\.P\_\{\\theta\}\(w\_\{1\},\\ldots,w\_\{T\}\)=\\prod\_\{t=1\}^\{T\}P\_\{\\theta\}\(w\_\{t\}\\mid w\_\{1\},\\ldots,w\_\{t\-1\}\)\.Training consists, broadly speaking, in selecting parametersθ\\thetaso thatPθP\_\{\\theta\}provides a good fit to the empirical distributionP^N\\widehat\{P\}\_\{N\}, with the underlying objective of approximating the unknown data\-generating distributionP⋆P^\{\\star\}\.

This probabilistic formulation provides a natural framework for understanding both the training and the inference processes of large language models\. It also highlights a fundamental distinction between statistical plausibility and truth: the model assigns likelihood to possible sequences or continuations according to statistical regularities learned from data, but no term in its objective refers to the world, and nothing in it represents whether a generated statement is in fact correct\.

Stating this framework is easy; stating it precisely is less so\. A reader coming from probability or statistics, and wishing to know exactly*which*measure, on*which*space, estimated in*which*sense and sampled by*which*rule, will find the answer harder to assemble than one might expect\. The difficulty is not a shortage of expositions but the shape of the two literatures that could supply one\. Accounts written for a machine\-learning audience are concise and operational, but they routinely leave the central objects unseparated: the unknown law of the language, the empirical law of the corpus and the parametric family are all writtenPP, and the training criterion is described as minimizing a “distance” whose direction is left unspecified — although, as we shall see, that direction governs the behavior of the fitted model in its tail, and with it the model’s propensity to hallucinate\. Accounts written for a probabilistic audience are rigorous but partial: the ingredients — autoregressive processes on a finite alphabet, the asymptotic equipartition property, exponential tilting of a measure, decision rules under a loss — are each treated somewhere in the classical literature, but to the best of the author’s knowledge they have not been assembled into a single account of what a language model is, as a probabilistic object\.

This paper attempts that assembly\. It is written for an expository purpose, and it retains the marks of the quest of understanding the machinery through the questions that actually cause difficulties ; and these questions are raised where they arise rather than where a systematic treatment would place them\. Why not use the empirical distribution, since it is already known? What is missing from the expansion of the divergence? In what sense is the generated text a Markov chain, given that the process is plainly not memoryless? Once the model has produced a measure on the vocabulary, how is one word actually chosen from it? Each of these is elementary, and each is passed over in most presentations\. More subtle points are addressed, in particular, why the word of the best probability is not the one chosen from a probabilistic standpoint and why it is not desirable to be chosen at all\. Concerning these issues and others, the exposition privileges transparency over concision\.

For completeness on the probabilistic aspects of generative AI, both LLMs and diffusion models are treated in this article; the emphasis nonetheless differs between the two: for diffusion models, whose probabilistic reading is already the standard one and calls for no defense, what is offered here is essentially depth and completeness of exposition; for language models, disentangling the relevant probabilistic objects is itself the point of the exercise, which is why they give the paper its title and its guiding thread\. For accounts on concepts and techniques of generative AI and LLMs we refer to\[[6](https://arxiv.org/html/2609.25134#bib.bib6)\],\[[8](https://arxiv.org/html/2609.25134#bib.bib8)\],\[[7](https://arxiv.org/html/2609.25134#bib.bib7)\],\[[9](https://arxiv.org/html/2609.25134#bib.bib9)\],\[[10](https://arxiv.org/html/2609.25134#bib.bib10)\], and for a concise presentation, see\[[11](https://arxiv.org/html/2609.25134#bib.bib11)\]\.

The present paper follows the three stages a language model passes through\. Sections[2](https://arxiv.org/html/2609.25134#S2)and[3](https://arxiv.org/html/2609.25134#S3)set up the object: a measure on sequences, specified by its predictive kernels, together with the parametric family that approximates it\. Section[4](https://arxiv.org/html/2609.25134#S4)treats estimation — the criterion, its asymmetry, and the algorithms that minimize it\. Section[5](https://arxiv.org/html/2609.25134#S5)treats generation, which we develop at greater length than is customary, since it is where the probabilistic content is densest and where expositions are most elliptical; we have in particular tried to make explicit the two distinct operations that choosing a word involves, the transformation of the measure and the draw from it, the second of which is usually passed over in silence\. Section[7](https://arxiv.org/html/2609.25134#S7)asks in what sense, and at which levels, the whole enterprise deserves to be called probabilistic, and where the adjective misleads\. Section[8](https://arxiv.org/html/2609.25134#S8)extends the discussion to diffusion models, as a complementary illustration: starting from the discrete\-time Gaussian construction, it introduces the score function and its connection to denoising, and ends with the continuous\-time formulation in terms of stochastic differential equations\.

## 2A fundamental goal: construction of a probability measure on sequences of words

By its very definition, a large language model is a languageLLendowed with a probability measure\. The measure assigns to every finite sequence of words a number quantifying how likely that sequence is to occur as a passage ofLL; by conditioning, it delivers the object of practical interest, the law of the next word given those already written\. Building an LLM amounts to finding a good such measure by exploring a large corpus and extracting its probabilistic structure, in three steps:

1. 1\.fix a large corpus of text inLL;
2. 2\.train a machine learning system — transformer\-based, this architecture having proved the most effective — to find the probability measure that best governs the behavior of word sequences inLL;
3. 3\.use the resulting probabilistic system for the desired tasks: translation, text generation, summarization, question answering\.

The sections below will make these steps precise and clarify the probabilistic nature of Gen AI\.

Fix a finite vocabulary𝒱=\{w1,…,w\|𝒱\|\}\\mathcal\{V\}=\\\{w\_\{1\},\\dots,w\_\{\|\\mathcal\{V\}\|\}\\\}\(in practice, a set of*tokens*: sub\-word units produced by a byte\-pair\-encoding scheme\[[12](https://arxiv.org/html/2609.25134#bib.bib12)\], with\|𝒱\|\|\\mathcal\{V\}\|of the order of10410^\{4\}–10510^\{5\}\)\.

A sequence of words to be learned or generated is described byX1,X2,…:Ω→𝒱X\_\{1\},X\_\{2\},\\dots:\\Omega\\to\\mathcal\{V\}, which are random variables defined on a probability space\(Ω,ℱ,ℙ\)\(\\Omega,\\mathcal\{F\},\\mathbb\{P\}\), and their values being in𝒱\\mathcal\{V\}\. Then, the object we wish to model is the law of this process, i\.e\. the family of finite\-dimensional distributions

P⋆​\(wi1,…,wim\):=ℙ⁡\(X1=wi1,…,Xm=wim\),m≥1\.P^\{\\star\}\(w\_\{i\_\{1\}\},\\dots,w\_\{i\_\{m\}\}\)\\;:=\\;\\mathbb\{P\}\\\!\\left\(X\_\{1\}=w\_\{i\_\{1\}\},\\dots,X\_\{m\}=w\_\{i\_\{m\}\}\\right\),\\qquad m\\geq 1\.\(1\)
Technical note\.To obtain a probability measure on sequences of finite but unbounded length, we augment the vocabulary with an absorbing symbol,𝒱¯=𝒱∪\{<eos\>\}\\overline\{\\mathcal\{V\}\}=\\mathcal\{V\}\\cup\\\{\\texttt\{<eos\>\}\\\}, takeΩ=⋃m≥0𝒱¯m\\Omega=\\bigcup\_\{m\\geq 0\}\\overline\{\\mathcal\{V\}\}^\{m\}with theXtX\_\{t\}the coordinate maps, and define the length as the stopping timeτ=inf\{t≥1:Xt=<eos\>\}\\tau=\\inf\\\{t\\geq 1:X\_\{t\}=\\texttt\{<eos\>\}\\\}\. The construction yields a probability measure on finite strings if and only ifτ<∞\\tau<\\inftyalmost surely; a model violating this leaks mass to infinite sequences and generates without terminating\. Models satisfyingτ<∞\\tau<\\inftya\.s\. are called*tight*; a measure\-theoretic treatment of this condition, and a proof that the usual model families are tight, is given in\[[16](https://arxiv.org/html/2609.25134#bib.bib16)\]\.

##### Why considering these probabilities: the predictive objective ?

The quantity of operational interest is the*one\-step predictive distribution*: given a past oft−1t\-1words, the conditional law of the next one,

ℙ\(Xt=⋅∣X1=wi1,…,Xt−1=wit−1\)=:P⋆\(⋅∣w<t\),\\mathbb\{P\}\\\!\\left\(X\_\{t\}=\\cdot\\mid X\_\{1\}=w\_\{i\_\{1\}\},\\dots,X\_\{t\-1\}=w\_\{i\_\{t\-1\}\}\\right\)\\;=:\\;P^\{\\star\}\\\!\\left\(\\cdot\\mid w\_\{<t\}\\right\),\(2\)where we writew<t=\(wi1,…,wit−1\)w\_\{<t\}=\(w\_\{i\_\{1\}\},\\dots,w\_\{i\_\{t\-1\}\}\)with the usual abuse of notation\. For each fixed contextw<tw\_\{<t\}, expression \([2](https://arxiv.org/html/2609.25134#S2.E2)\) is a probability measure on the finite set𝒱\\mathcal\{V\}\.

The joint law and the family of predictive kernels determine each other\. In one direction, the chain rule gives

P⋆​\(wi1,…,wim\)=∏t=1mP⋆​\(wit∣wi1,…,wit−1\),P^\{\\star\}\(w\_\{i\_\{1\}\},\\dots,w\_\{i\_\{m\}\}\)\\;=\\;\\prod\_\{t=1\}^\{m\}P^\{\\star\}\\\!\\left\(w\_\{i\_\{t\}\}\\mid w\_\{i\_\{1\}\},\\dots,w\_\{i\_\{t\-1\}\}\\right\),\(3\)with the convention that thet=1t=1factor is the marginalP⋆​\(wi1\)P^\{\\star\}\(w\_\{i\_\{1\}\}\)\.

## 3The statistical model

We introduce a parametric family of probability measures:

𝒫Θ=\{Pθ\(⋅∣w<t\):θ∈Θ⊂ℝp\},\\mathcal\{P\}\_\{\\Theta\}=\\left\\\{P\_\{\\theta\}\(\\cdot\\mid w\_\{<t\}\):\\theta\\in\\Theta\\subset\\mathbb\{R\}^\{p\}\\right\\\},\(4\)realized by atransformer: the ideal goal is to determine

P⋆P^\{\\star\}, but we can not\. So the idea is to find a probability measure

PθP\_\{\\theta\}that is quite close to

P⋆P^\{\\star\}which governs the probabilities of sequences of words in the language

LL, given what we have observed in the corpus or data set considered\. One should note that

P⋆∉𝒫ΘP^\{\\star\}\\notin\\mathcal\{P\}\_\{\\Theta\}in general: the family is misspecified, and

θ^\\hat\{\\theta\}converges not to

P⋆P^\{\\star\}but to the*pseudo\-true*parameter

arg​minθD\(P⋆∥Pθ\)\\argmin\_\{\\theta\}D\(P^\{\\star\}\\\|P\_\{\\theta\}\), the information projection of

P⋆P^\{\\star\}onto the family\[[18](https://arxiv.org/html/2609.25134#bib.bib18)\]\. Note\. Transformers\[[6](https://arxiv.org/html/2609.25134#bib.bib6)\]form the AI architecture behind the Gen AI move\. They should be viewed as computing systems whose outputs depend on both the inputs and a huge set of parameters

θ\\theta\. Learning amounts to adjust these parameters\. We refer the reader to\[[6](https://arxiv.org/html/2609.25134#bib.bib6),[7](https://arxiv.org/html/2609.25134#bib.bib7),[8](https://arxiv.org/html/2609.25134#bib.bib8),[9](https://arxiv.org/html/2609.25134#bib.bib9),[10](https://arxiv.org/html/2609.25134#bib.bib10),[11](https://arxiv.org/html/2609.25134#bib.bib11)\]for these topics\.

The practical result obtained will be the predictive or conditional probabilities on contexts

w<tw\_\{<t\}\. It is convenient to consider

PθP\_\{\\theta\}of the form :

Pθ​\(wt=v∣w<t\)=exp⁡\(zt,v\)∑u∈𝒱exp⁡\(zt,u\)=s​o​f​t​m​a​x​\(zt\)v,P\_\{\\theta\}\(w\_\{t\}=v\\mid w\_\{<t\}\)=\\frac\{\\exp\\\!\\big\(z\_\{t,v\}\\big\)\}\{\\sum\_\{u\\in\\mathcal\{V\}\}\\exp\\\!\\big\(z\_\{t,u\}\\big\)\}\\;=\\;softmax\\big\(z\_\{t\}\\big\)\_\{v\},\(5\)\(exponential family\)\. Thezt,v=zt​\(θ,w<t\)∈ℝ\|𝒱\|z\_\{t,v\}=z\_\{t\}\(\\theta,w\_\{<t\}\)\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}are the so\-calledlogits\. Herepp, the dimension ofθ\\theta, is of order10910^\{9\}–101210^\{12\}; GPT\-3\[[13](https://arxiv.org/html/2609.25134#bib.bib13)\], for instance, has1\.75×10111\.75\\times 10^\{11\}parameters\.

##### Logits usage:

ztz\_\{t\}carries one real*score*per candidate token\. The reason to seekztz\_\{t\}is that they are unconstrained: the network may output any vector ofℝ\|𝒱\|\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}, positivity and normalization being restored by the softmax\. Parametrizing the simplex directly would require enforcingpv≥0p\_\{v\}\\geq 0and∑vpv=1\\sum\_\{v\}p\_\{v\}=1at every step, which is awkward\.

Taking logarithms in \([5](https://arxiv.org/html/2609.25134#S3.E5)\),logPθ\(v∣w<t\)=zt,v−log∑uezt,u\\log P\_\{\\theta\}\(v\\mid w\_\{<t\}\)=z\_\{t,v\}\-\\log\\sum\_\{u\}e^\{z\_\{t,u\}\}: the logits are the log\-probabilities up to one additive constant common to all tokens\. They are accordingly defined only up to a shift,ztz\_\{t\}andzt\+c​𝟏z\_\{t\}\+c\\mathbf\{1\}giving the same measure\.

*How the logits are obtained ?*The network compresses the context into a single vectorht∈ℝdh\_\{t\}\\in\\mathbb\{R\}^\{d\}withd∼103d\\sim 10^\{3\}–10410^\{4\}, and each tokenvvcarries its own vectoruv∈ℝdu\_\{v\}\\in\\mathbb\{R\}^\{d\}; then

zt,v=⟨ht,uv⟩\.z\_\{t,v\}=\\langle h\_\{t\},u\_\{v\}\\rangle\.\(6\)The score of a candidate is thus the inner product between a summary of the context and a representation of the token, both living in the samedd\-dimensional space\. Section[6](https://arxiv.org/html/2609.25134#S6)describes howhth\_\{t\}is built; the essential consequence is already visible here, and is taken up in Question 1: since contexts with similar continuations are mapped to nearbyhth\_\{t\}, an observation made in one context transfers to contexts never observed\[[1](https://arxiv.org/html/2609.25134#bib.bib1)\]\.

## 4The learning phase

### 4\.1Setting

The corpus that serves for the learning phase is written as:𝒟=\{x\(1\),…,x\(N\)\}\\mathcal\{D\}=\\\{x^\{\(1\)\},\\dots,x^\{\(N\)\}\\\}\. Thex\(i\)x^\{\(i\)\}are to be viewed asNNrealizations drawn \(idealized as i\.i\.d\.\) fromP⋆P^\{\\star\}\. Then the following 3 distinct objects are considered:

- •P⋆P^\{\\star\}: the true law, unknown — the*target*\.
- •P^N\\widehat\{P\}\_\{N\}: the empirical law which is observed, but not suitable \(see below\): P^N=1N​∑i=1Nδx\(i\)\\widehat\{P\}\_\{N\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\delta\_\{x^\{\(i\)\}\}\(7\)be the empirical measure on𝒱∗\\mathcal\{V\}^\{\*\}\.
- •PθP\_\{\\theta\}: the family of probabilities depending on \(huge\) set of parametersθ\\thetawhich are computable, and from which we have to choose the best one that approaches the targetP⋆P^\{\\star\}\.

### 4\.2The estimation criterion

We chooseθ\\thetaso as to minimize some “distance”d⁡\(P^N,Pθ\)d\(\\widehat\{P\}\_\{N\},P\_\{\\theta\}\)betweenP^N\\widehat\{P\}\_\{N\}andPθP\_\{\\theta\}

θ^=arg​minθ∈Θd\(P^N∥Pθ\)\.\\hat\{\\theta\}=\\argmin\_\{\\theta\\in\\Theta\}\\;d\(\\widehat\{P\}\_\{N\}\\,\\\|\\,P\_\{\\theta\}\)\.\(8\)We take :

d\(P^N,Pθ\)=D\(P∥Q\)=∑xP\(x\)logP⁡\(x\)Q⁡\(x\)\.d\(\\widehat\{P\}\_\{N\},P\_\{\\theta\}\)=D\(P\\\|Q\)=\\sum\_\{x\}P\(x\)\\log\\frac\{P\(x\)\}\{Q\(x\)\}\.\(9\)D\(P∥Q\)D\(P\\\|Q\)is the Kullback–Leibler divergence \(sometimes written asKL\(P∥Q\)KL\(P\\\|Q\)\) which is to be viewed here as a sort of distance between the two probabilitiesP,QP,Q\.D\(P∥Q\)≥0D\(P\\\|Q\)\\geq 0with equality if and only ifP=QP=Q\. This choice is not a convention of convenience: minimizingD\(P^N∥Pθ\)D\(\\widehat\{P\}\_\{N\}\\\|P\_\{\\theta\}\)is equivalent to scoring the model by the logarithmic loss, which is, up to affine transformation, the unique*local*strictly proper scoring rule\[[17](https://arxiv.org/html/2609.25134#bib.bib17)\]— “local” meaning that the score depends only on the probability assigned to the event actually observed, which is precisely the property allowing the criterion to decompose along the chain rule \([3](https://arxiv.org/html/2609.25134#S2.E3)\) without a partition function\. LetH⁡\(P\)H\(P\)be the entropy ofPP:

H\(P\)=−∑xP\(x\)logP\(x\),H\(P\)=\-\\sum\_\{x\}P\(x\)\\log P\(x\),\(10\)then we have:

D\(P^N∥Pθ\)=−H⁡\(P^N\)⏟independent of​θ−1N∑i=1NlogPθ\(x\(i\)\),D\\\!\\left\(\\widehat\{P\}\_\{N\}\\\|P\_\{\\theta\}\\right\)=\\underbrace\{\-H\(\\widehat\{P\}\_\{N\}\)\}\_\{\\text\{independent of \}\\theta\}\\;\-\\;\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\log P\_\{\\theta\}\\\!\\left\(x^\{\(i\)\}\\right\),\(11\)so that \([8](https://arxiv.org/html/2609.25134#S4.E8)\) is*exactly*maximum likelihood\. Applying the chain rule \([3](https://arxiv.org/html/2609.25134#S2.E3)\) inside the logarithm yields the loss actually implemented,

ℒ\(θ\)=−1N∑i=1N∑t=1TilogPθ\(xt\(i\)∣x<t\(i\)\)\.\\mathcal\{L\}\(\\theta\)=\-\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\sum\_\{t=1\}^\{T\_\{i\}\}\\log P\_\{\\theta\}\\\!\\left\(x^\{\(i\)\}\_\{t\}\\mid x^\{\(i\)\}\_\{<t\}\\right\)\.\(12\)
Equivalently, by the chain rule for the KL divergence,

D\(P^N∥Pθ\)=∑t≥1𝔼P^N\[D\(P^N\(⋅∣X<t\)∥Pθ\(⋅∣X<t\)\)\],D\\\!\\left\(\\widehat\{P\}\_\{N\}\\\|P\_\{\\theta\}\\right\)=\\sum\_\{t\\geq 1\}\\mathbb\{E\}\_\{\\widehat\{P\}\_\{N\}\}\\\!\\left\[\\,D\\\!\\left\(\\widehat\{P\}\_\{N\}\(\\cdot\\mid X\_\{<t\}\)\\,\\big\\\|\\,P\_\{\\theta\}\(\\cdot\\mid X\_\{<t\}\)\\right\)\\right\],\(13\)which makes explicit that fitting the joint law is the same as fitting all the predictive kernels simultaneously, weighted by the frequency of their contexts\.

### Why not simply usingP^N\\widehat\{P\}\_\{N\}?

The reader may wonder why not usingP^N\\widehat\{P\}\_\{N\}which is already known\. The main reason is thatP^N\\widehat\{P\}\_\{N\}covers only the corpus already known: it will not enable to generate new contents, for any new content will have zero probability underP^N\\widehat\{P\}\_\{N\}\. Even if we smooth it,P^N\\widehat\{P\}\_\{N\}suffers the curse of dimensionality : the number of possible sequences grows exponentially with the lengthm⁡\(\|𝒱\|m\)m\(\|\{\\cal V\}\|^\{m\}\)\. The training set covers only an infinitesimally small fraction of this space\. The parametric family escapes this because it shares statistical strength across contexts: semantically similar contexts having nearby representations, an observation transfers to contexts never seen, which a lookup table cannot do\[[1](https://arxiv.org/html/2609.25134#bib.bib1)\]\.

### The minimization procedure for \([8](https://arxiv.org/html/2609.25134#S4.E8)\):

We have seen that the learning phase amounts to finding parameters

θ^=arg⁡minθ⁡ℒ⁡\(θ\),\\hat\{\\theta\}=\\arg\\min\_\{\\theta\}\\mathcal\{L\}\(\\theta\),whereℒ⁡\(θ\)\\mathcal\{L\}\(\\theta\)measures how far the probabilities predicted by the model are from those observed in the training data\. For a large language model,θ\\thetamay contain billions of parameters andℒ\\mathcal\{L\}is a highly non\-linear and non\-convex function\. A direct minimization is therefore impossible in practice\.

The basic tool is*gradient descent*\. Starting from an initial valueθ0\\theta\_\{0\}, the parameters are progressively modified in the direction in which the loss decreases:

θk\+1=θk−ηk​∇θℒ​\(θk\),\\theta\_\{k\+1\}=\\theta\_\{k\}\-\\eta\_\{k\}\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\_\{k\}\),where∇θℒ\\nabla\_\{\\theta\}\\mathcal\{L\}is the gradient of the loss andηk\>0\\eta\_\{k\}\>0, called the*learning rate*, determines the size of the modification\. The intuition is the same as that of neural networks \(explained above\): the gradient points locally in the direction of greatest increase of the loss, so that its opposite,

−∇θℒ,\-\\nabla\_\{\\theta\}\\mathcal\{L\},provides a direction in which the loss should decrease\.

Why stochastic gradient descent?There is, however, a major practical difficulty\. The training loss is an average over an extremely large numberNNof training examples:

ℒ⁡\(θ\)=1N​∑i=1Nℓi​\(θ\),\\mathcal\{L\}\(\\theta\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\ell\_\{i\}\(\\theta\),whereℓi​\(θ\)\\ell\_\{i\}\(\\theta\)is the loss associated with theii\-th training example\. Consequently,

∇θℒ​\(θ\)=1N​∑i=1N∇θℓi​\(θ\)\.\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\nabla\_\{\\theta\}\\ell\_\{i\}\(\\theta\)\.Computing this exact gradient before every parameter update would require processing the entire training corpus, which would be prohibitively expensive\. Instead, at each iteration one selects a relatively small random subset of training examples,

Bk⊂\{1,…,N\},B\_\{k\}\\subset\\\{1,\\ldots,N\\\},called a*mini\-batch*\. For example, instead of computing the gradient from billions of training tokens, one computes it from a much smaller group of examples and defines

∇θℒBk​\(θ\)=1\|Bk\|​∑i∈Bk∇θℓi​\(θ\)\.\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{B\_\{k\}\}\(\\theta\)=\\frac\{1\}\{\|B\_\{k\}\|\}\\sum\_\{i\\in B\_\{k\}\}\\nabla\_\{\\theta\}\\ell\_\{i\}\(\\theta\)\.The parameters are then updated according to

θk\+1=θk−ηk​∇θℒBk​\(θk\)\.\\theta\_\{k\+1\}=\\theta\_\{k\}\-\\eta\_\{k\}\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{B\_\{k\}\}\(\\theta\_\{k\}\)\.Because the mini\-batch is randomly selected, this gradient is random, hence the terminology*stochastic gradient descent*\(SGD\)\. If the examples are sampled uniformly, then

𝔼⁡\[∇θℒBk​\(θ\)\]=∇θℒ​\(θ\)\.\\mathbb\{E\}\\left\[\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{B\_\{k\}\}\(\\theta\)\\right\]=\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta\)\.
Thus the mini\-batch gradient fluctuates around the true gradient, but is much cheaper to compute\. The resulting recursion is a*stochastic approximation*scheme in the sense of Robbins and Monro\[[21](https://arxiv.org/html/2609.25134#bib.bib21)\], whose classical step\-size conditions∑kηk=∞\\sum\_\{k\}\\eta\_\{k\}=\\infty,∑kηk2<∞\\sum\_\{k\}\\eta\_\{k\}^\{2\}<\\inftyare relaxed in practice to engineered schedules\. The basic learning procedure can therefore be summarized as follows:

Select a small part of the training data→\\rightarrowCompute the model predictions→\\rightarrowcompute the corresponding loss→\\rightarrowcompute its gradient→\\rightarrowslightly modify→\\rightarrowselect another group of examples and repeat…\\ldots

The entire corpus is therefore progressively processed, but the parameters are updated many times during this process, rather than only once after processing the complete corpus\.

##### How is the gradient computed?

There remains an apparently big problem\. If

θ=\(θ1,…,θp\),p∼1011,\\theta=\(\\theta\_\{1\},\\ldots,\\theta\_\{p\}\),\\qquad p\\sim 10^\{11\},then the gradient contains

∇θℒ=\(∂ℒ∂θ1,…,∂ℒ∂θp\),\\nabla\_\{\\theta\}\\mathcal\{L\}=\\left\(\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\theta\_\{1\}\},\\ldots,\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\theta\_\{p\}\}\\right\),possibly hundreds of billions of partial derivatives\.These derivatives are not computed separately\. As we saw, neural networks use*backpropagation*, which is an efficient application of the chain rule of differentiation

##### From SGD to Adam\.

The elementary SGD update

θk\+1=θk−ηk​gk,gk=∇θℒBk​\(θk\),\\theta\_\{k\+1\}=\\theta\_\{k\}\-\\eta\_\{k\}g\_\{k\},\\qquad g\_\{k\}=\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{B\_\{k\}\}\(\\theta\_\{k\}\),uses essentially the same learning\-rate factorηk\\eta\_\{k\}for every parameter\. The learning rateηk\\eta\_\{k\}is itself usually varied during training\.

In a large neural network, however, different parameters can have gradients with very different magnitudes and behaviors\. Modern LLM training therefore commonly uses adaptive variants of stochastic gradient descent, particularly*Adam*and*AdamW*\. Adam was introduced by Kingma and Ba\[[22](https://arxiv.org/html/2609.25134#bib.bib22)\], and the decoupled weight\-decay version AdamW by Loshchilov and Hutter\[[23](https://arxiv.org/html/2609.25134#bib.bib23)\]\. We omit the details\.

##### Remark on the parametersθ\\theta\.

While the numerical optimization procedure determines a parameter vectorθ^\\widehat\{\\theta\}, the object of interest is ultimately the probability modelPθ^P\_\{\\widehat\{\\theta\}\}defined by these parameters\. The parameter vector itself is generally not uniquely identifiable\. Indeed, because of symmetries in neural network parameterizations \(such as permutations of hidden units or certain compensating rescalings\), two different parameter vectors may define the same, or essentially the same, probability model:

θ≠θ′while possiblyPθ=Pθ′\.\\theta\\neq\\theta^\{\\prime\}\\qquad\\text\{while possibly\}\\qquad P\_\{\\theta\}=P\_\{\\theta^\{\\prime\}\}\.Thus, one should not interpret training as recovering a unique “true” parameter vector\. Moreover, since the loss function is highly non\-convex and the optimization is performed over a finite number of iterations, the training procedure is not guaranteed to reach a global minimizer\. The resultingθ^\\widehat\{\\theta\}should rather be viewed as one parameter configuration reached by the optimization procedure and providing a satisfactory fitted probability model\.

Once training is completed, the parametersθ^\\widehat\{\\theta\}are fixed\. For a given contextw<tw\_\{<t\}, the Transformer uses them to compute the conditional probability distribution

Pθ^\(⋅∣w<t\)\.P\_\{\\widehat\{\\theta\}\}\\bigl\(\\,\\cdot\\mid w\_\{<t\}\\bigr\)\.The collection of these conditional distributions defines the learned autoregressive probability model

Pθ^\(w1:T\)=∏t=1TPθ^\(wt∣w<t\)\.P\_\{\\widehat\{\\theta\}\}\(w\_\{1:T\}\)=\\prod\_\{t=1\}^\{T\}P\_\{\\widehat\{\\theta\}\}\(w\_\{t\}\\mid w\_\{<t\}\)\.Hence,θ^\\widehat\{\\theta\}is primarily an internal numerical parameterization: the probabilistic object of interest is the modelPθ^P\_\{\\widehat\{\\theta\}\}that it defines\.

To end this paragraph let us recall, the complete learning process :

Training corpusdefines the empirical distribution​P^N⟶Define the training loss​ℒ​\(θ\)and choose initial parameters​θ0⟶Select a small random subset​Bkof the training data\\begin\{subarray\}\{c\}\\text\{Training corpus\}\\\\ \\text\{defines the empirical distribution \}\\widehat\{P\}\_\{N\}\\end\{subarray\}\\;\\longrightarrow\\;\\begin\{subarray\}\{c\}\\text\{Define the training loss \}\\mathcal\{L\}\(\\theta\)\\\\ \\text\{and choose initial parameters \}\\theta\_\{0\}\\end\{subarray\}\\;\\longrightarrow\\;\\begin\{subarray\}\{c\}\\text\{Select a small random subset \}B\_\{k\}\\\\ \\text\{of the training data\}\\end\{subarray\}⟶Compute the loss on​Bkand its gradient​∇θℒBk​\(θk\)⟶Update the parameters usinga stochastic gradient algorithmθk⟶θk\+1⟶Repeat over successive subsetsuntil the end of training\\;\\longrightarrow\\;\\begin\{subarray\}\{c\}\\text\{Compute the loss on \}B\_\{k\}\\\\ \\text\{and its gradient \}\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{B\_\{k\}\}\(\\theta\_\{k\}\)\\end\{subarray\}\\;\\longrightarrow\\;\\begin\{subarray\}\{c\}\\text\{Update the parameters using\}\\\\ \\text\{a stochastic gradient algorithm\}\\\\ \\theta\_\{k\}\\longrightarrow\\theta\_\{k\+1\}\\end\{subarray\}\\;\\longrightarrow\\;\\begin\{subarray\}\{c\}\\text\{Repeat over successive subsets\}\\\\ \\text\{until the end of training\}\\end\{subarray\}⟶Obtain a fitted parameter vector​θ^defining the learned model​Pθ^\\;\\longrightarrow\\;\\begin\{subarray\}\{c\}\\text\{Obtain a fitted parameter vector \}\\widehat\{\\theta\}\\\\ \\text\{defining the learned model \}P\_\{\\widehat\{\\theta\}\}\\end\{subarray\}

## 5The inference phase or how to generate texts and next words from the probability measure obtained

Having obtainedθ^\\hat\{\\theta\}, we have at our disposal, for every context, the predictive measure

πt\(⋅\):=Pθ^\(⋅∣w<t\)∈ℳ1\(𝒱\),\\pi\_\{t\}\(\\cdot\)\\;:=\\;P\_\{\\hat\{\\theta\}\}\\\!\\left\(\\cdot\\mid w\_\{<t\}\\right\)\\;\\in\\;\\mathcal\{M\}\_\{1\}\(\\mathcal\{V\}\),\(14\)a probability measure on the vocabulary\.

##### Decoding: from a probability measure to a word

At each step the model hands us a probability measureπt\\pi\_\{t\}on𝒱\\mathcal\{V\}, not a word\. Producing one decomposes into two stages:\(a\)a transformation of the measure,πt↦π~t\\pi\_\{t\}\\mapsto\\tilde\{\\pi\}\_\{t\}, still a probability on𝒱\\mathcal\{V\}: the aim here is to optimize further the measureπt\\pi\_\{t\}which will guide the choice of the next word : for instance by restricting it to some of the best candidates;\(b\)the realization of one draw fromπ~t\\tilde\{\\pi\}\_\{t\}\. Stage \(a\) distinguishes the strategies; stage \(b\) is common to all and is where the randomness enters\.

### 5\.1A visual analogy of the decoding process

Suppose the measure has been restricted to 20 words candidates\. Cut a segment of length11into twenty pieces, the length of each being the probability of its word: a word of probability0\.450\.45occupies45%45\\%of the segment\. Throw a dart randomly \(and uniformly\) at the segment; the word whose piece is hit is the next word to be generated\.

The most probable word is not necessarily the*chosen*one, although it owns the largest piece and is therefore hit most often\. Among the choice strategies that will be reviewed below, let us mention :*Greedy decoding*throws no dart and takes the largest piece ;*Truncation*deletes the minuscule pieces before throwing and stretches the rest back to unit length — the renormalization ;*TemperatureTT*rescales the pieces before throwing: with lowTT\(cold\), the large pieces grow and the small shrink; with highTT\(hot\), they tend to equalize\. Truncation and temperature alter the sizes of the pieces\.

### 5\.2Stage \(a\): transforming the measure

Writeπ=πt\\pi=\\pi\_\{t\}, with logitszz,π=s​o​f​t​m​a​x​\(z\)\\pi=softmax\(z\)\.

##### Greedy decoding :

Xt=arg​maxv⁡π​\(v\)X\_\{t\}=\\argmax\_\{v\}\\pi\(v\); It is deterministic\. It maximizes the probability of the*next*token only, not of the sequence, since the maximum does not factorize into a product of individual measures\. It is a myopic choice for it considers only what is the right word for the next step \(only\) and not for whole text to be generated\. It produces flat text \(lack of diversity\) and falls into repetitive loops \(“I see I see …”\) because once a high\-probability token is chosen, the context shifts to reinforce the same choice\.

##### Beam search :

it is a deterministic strategy that approximates the global mode of the sequence:

w^1:T=argmaxw1:TPθ^\(w1:T\),\\hat\{w\}\_\{1:T\}=\\arg\\max\_\{w\_\{1:T\}\}P\_\{\\hat\{\\theta\}\}\(w\_\{1:T\}\),
or more commonly, a length\-normalized variantT−αlogPθ^\(w1:T\)T^\{\-\\alpha\}\\log P\_\{\\hat\{\\theta\}\}\(w\_\{1:T\}\)\. At each step, it maintains a fixed numberkk\(the beam width\) of the most probable partial sequences, extends each by one token, and keeps thekkbest among the resultingk​\|𝒱\|k\|\\mathcal\{V\}\|candidates\. The advantage is that itsearches globallyand is therefore excellent for tasks where the output is largely determined by the input, such as machine translation, speech recognition, and summarization — precisely the tasks for which it was originally designed\.

For open\-ended text generation, however, beam search performs poorly\. The reason is thelikelihood trap: the mode of a high\-dimensional distribution is systematically atypical\. The most probable sequence is the most predictable, hence least informative, and often degenerates into short, repetitive, or bland text\. Moreover, beam search collapses diversity across different prompts; a fixed beam width produces similar outputs regardless of the prompt’s uncertainty\. Consequently, beam search has been largely abandoned in conversational and creative generation systems as of 2026\.

##### Ancestral \(or pure\) sampling:

here stage \(a\) is the identity,π~=π\\tilde\{\\pi\}=\\pi, and one simply drawsXt∼πtX\_\{t\}\\sim\\pi\_\{t\}\. The name records the order of the draws: one follows the factorization \([3](https://arxiv.org/html/2609.25134#S2.E3)\) in its natural direction, drawingX1X\_\{1\}fromπ1\\pi\_\{1\}, thenX2X\_\{2\}fromπ2\(⋅∣X1\)\\pi\_\{2\}\(\\cdot\\mid X\_\{1\}\), and so on, each variable being drawn only after its “ancestors” in the chain rule\. It is the only rule producing an*exact*realization of the fitted processPθ^P\_\{\\hat\{\\theta\}\}— see §[5\.5](https://arxiv.org/html/2609.25134#S5.SS5)— and is the correct choice if one’s aim is to studyPθ^P\_\{\\hat\{\\theta\}\}itself\.

Its weakness in application follows from the mass\-covering property established in Section[4](https://arxiv.org/html/2609.25134#S4)\. By \([5](https://arxiv.org/html/2609.25134#S3.E5)\) all\|𝒱\|≈5×104\|\\mathcal\{V\}\|\\approx 5\\times 10^\{4\}tokens carry strictly positive probability, and the aggregate mass of the tail is typically a few percent, say3%3\\%\. Over a generation of500500tokens the probability of drawing at least one implausible token is then1−\(1−0\.03\)500≈11\-\(1\-0\.03\)^\{500\}\\approx 1; and a single such token derails everything that follows, since the model conditions on it\. Hence the truncation schemes below\.

##### Temperature Scaling or Exponential Tilting:

temperature scaling modifies the predictive measure by raising the logits \(or equivalently, exponentiating the probabilities\) with a temperature parameterT\>0T\>0:

π\(T\)​\(v\)=exp⁡\(zv/T\)∑u∈𝒱exp⁡\(zu/T\)∝π​\(v\)1/T\.\\pi^\{\(T\)\}\(v\)=\\frac\{\\exp\(z\_\{v\}/T\)\}\{\\sum\_\{u\\in\\mathcal\{V\}\}\\exp\(z\_\{u\}/T\)\}\\propto\\pi\(v\)^\{1/T\}\.This is an exponential tilting of the original distribution: it is a Gibbs measure with the log\-likelihood as energy andTTas the inverse temperature\. AsT→0\+T\\to 0^\{\+\}, the distribution concentrates on the mode \(recovering greedy decoding in the limit\); asT=1T=1, it leavesπ\\piunchanged; asT→∞T\\to\\infty, it flattens to the uniform measure\. The entropyH⁡\(π\(T\)\)H\(\\pi^\{\(T\)\}\)increases monotonically withTT\.

The advantage isfine\-grained controlover the randomness of the output: Lower temperatures \(e\.g\.,T=0\.6T=0\.6\) reduce the probability of rare tokens, producing more focused and deterministic text; higher temperatures \(e\.g\.,T=1\.2T=1\.2\) increase diversity and surprise\. The disadvantage is that temperaturedoes not remove the tailof the distribution: every token, no matter how improbable, retains strictly positive probability\. In practice, it is almost always combined with truncation \(top\-kkor top\-pp\) to avoid drawing from the inflated tail\. Furthermore, optimalTTis context\-dependent and requires manual tuning\.

##### Top\-kksampling \(fixed truncation\) :

this sampling mode restricts the support to thekkmost probable tokens, renormalises, and then samples from the truncated distribution:

π~​\(v\)=π⁡\(v\)∑u∈Akπ⁡\(u\)for​v∈Ak,Ak=the​k​most probable tokens\.\\tilde\{\\pi\}\(v\)=\\frac\{\\pi\(v\)\}\{\\sum\_\{u\\in A\_\{k\}\}\\pi\(u\)\}\\quad\\text\{for \}v\\in A\_\{k\},\\quad A\_\{k\}=\\text\{the \}k\\text\{ most probable tokens\}\.
The advantage is that it removes the long, noisy tail of the distribution, drastically reducing the chance of sampling rare or nonsensical tokens\. It is more diverse than greedy and straightforward to implement\. The disadvantage is thatkkisfixed, while the entropy ofπt\\pi\_\{t\}varies dramatically across contexts\. At a highly predictable position \(e\.g\., after “The capital of France is”\), the distribution is peaked; a fixedkkkeepsk−1k\-1implausible candidates\. At a highly uncertain position \(e\.g\., after “The story begins with”\), the distribution is flat; the samekkmay discard legitimate alternatives\. This rigidity is the primary motivation for the adaptive strategy that follows\.

##### Top\-pp\(nucleus or dynamic truncation\):

also known as top\-ppsampling, it is an improvement of Top\-kkmethod, by allowing variations ofkk: it dynamically selects the smallest setApA\_\{p\}of tokens whose cumulative probability exceeds a thresholdp∈\[0\.9,0\.95\]p\\in\[0\.9,0\.95\]:

Ap=argminA\{π\(A\)≥p\},π~\(v\)=π⁡\(v\)π⁡\(Ap\)forv∈Ap\.A\_\{p\}=\\arg\\min\_\{A\}\\left\\\{\\pi\(A\)\\geq p\\right\\\},\\qquad\\tilde\{\\pi\}\(v\)=\\frac\{\\pi\(v\)\}\{\\pi\(A\_\{p\}\)\}\\quad\\text\{for \}v\\in A\_\{p\}\.
Introduced in\[[24](https://arxiv.org/html/2609.25134#bib.bib24)\], the setApA\_\{p\}is the “nucleus” of the distribution: the minimal set that carries a fractionppof the mass\. The advantage is that\|Ap\|\|A\_\{p\}\|adapts to the entropy ofπt\\pi\_\{t\}\. When the model is confident, the nucleus is small \(sometimes just 1 or 2 tokens\); when uncertain, the nucleus is large\. This adaptivity makes it the default in deployed conversational systems \(OpenAI, Anthropic, Google\)\. The disadvantage is that the thresholdppitself still requires tuning, and the procedure may include rare tokens if the probability mass is smoothly distributed\. Nevertheless, it has largely superseded top\-kkin practice, and is typically combined with temperature scaling \(order: temperature, then nucleus conditioning\)\.

##### Typical sampling \(information\-theoretic truncation\):

introduced by Meister*et al\.*\[[25](https://arxiv.org/html/2609.25134#bib.bib25)\], this is a more recent information\-theoretic strategy\. Instead of retaining the most probable tokens, it retains tokens whose negative log\-probability−log⁡π⁡\(v\)\-\\log\\pi\(v\)is close to the entropyH⁡\(π\)H\(\\pi\)of the distribution\. The intuition comes from the asymptotic equipartition property\[[15](https://arxiv.org/html/2609.25134#bib.bib15)\]: typical sequences are those whose log\-probability is near the entropy rate\. Specifically, one selects a set

Atypical=\{v:\|−log⁡π⁡\(v\)−H⁡\(π\)\|≤δ\},A\_\{\\text\{typical\}\}=\\left\\\{v:\\left\|\-\\log\\pi\(v\)\-H\(\\pi\)\\right\|\\leq\\delta\\right\\\},
for some thresholdδ\\delta, and then samples from the renormalized distribution overAtypicalA\_\{\\text\{typical\}\}\. The advantage is that it deliberatelyavoids the mode\(see the note below\), which is systematically atypical, and produces text that is more representative of the model’s “typical” outputs rather than its “most likely” ones\. This can reduce repetitive and dull generations\. The disadvantage is that the thresholdδ\\deltais non\-trivial to tune, and the method is less widely adopted than nucleus sampling due to its relative novelty and conceptual complexity\.

### 5\.3The mode, and why it is avoided ?

Themodeof a probability measure is its most probable outcome:arg​maxv⁡π​\(v\)\\argmax\_\{v\}\\pi\(v\)for a single token — the word greedy decoding selects — andarg​maxw1:TPθ^\(w1:T\)\\argmax\_\{w\_\{1:T\}\}P\_\{\\hat\{\\theta\}\}\(w\_\{1:T\}\)for a whole text, which beam search approximates\. Two distinct points must be kept apart here\. The first is probabilistic and explains why a draw does not produce the mode; the second is semantic and explains why one would not want it even if it did\.

*First point: being the most probable at each step confers almost no advantage over many steps\.*Return to the measure of § 4\.1, whose leading word carries probability0\.450\.45\. Greedy decoding selects it at every step\. Over77steps the modal path keeps its rank — no other particular sequence is more probable — and yet its probability is only0\.457≈0\.00230\.45^\{7\}\\approx 0\.0023, that is, fewer than one draw in400400; over1010steps, one in5 9005\\,900; over100100steps, one in103710^\{37\}\. A random draw therefore almost never lands exactly on the mode\. In fact

An elementary comparison: letXiX\_\{i\}be a sequence of i\.i\.d\. random variables with valuesA=1A=1\(head\) with probabilitypA=0\.9p\_\{A\}=0\.9andB=−1B=\-1\(tails\) with probabilitypB=0\.1p\_\{B\}=0\.1\. The mean theXiX\_\{i\}isM=A​pA\+B​pB=0\.8M=Ap\_\{A\}\+Bp\_\{B\}=0\.8and by the law of large numbers, all the pathsX1,X2,…​XnX\_\{1\},X\_\{2\},…X\_\{n\}whose empirical means\(X1\+…\+Xn\)/n\(X\_\{1\}\+\\ldots\+X\_\{n\}\)/nisveryclose to the expectationM=0\.8M=0\.8are eligible to happen; but the sequence111111​…111111\\ldotsof the most probable outcome \(A\) at each step, has11as empirical mean, which is notveryclose to the expectationM=0\.8M=0\.8\. We know already that the sequence111111​…111111\\ldotsis very unlikely to occur by the law of large numbers, and in fact the probability of events that are around this sequence is even exponentially small w\.r\.tnn\. In other words, in a head and tails game, obtaining a sequence of head innntosses is very unlikely to happen even if the probability to obtain head is90%90\\%at each step\.

*Second point: why the mode should be avoided anyway\.*This second argument is not probabilistic but semantic and informational\. A text that is maximally probable at every step is entirely predictable: it maximizes likelihood and minimizes surprise, and therefore carries almost no information\. This is exactly the flat, repetitive, loop\-prone output observed with greedy decoding and beam search, and it is known as the*likelihood trap*\[[24](https://arxiv.org/html/2609.25134#bib.bib24)\]\.

Human text is neither maximally predictable nor random; it holds a balance between the two\. It sits in what information theory calls the*typical set*: the region where the probability per token is close to the entropy of the language\[[15](https://arxiv.org/html/2609.25134#bib.bib15)\]\. Typical sampling is the decoding rule that targets that region directly, instead of climbing toward a peak that is both unreachable by sampling and undesirable as text\.

##### Composition and interpretation\.

These compose, conventionally in the order temperature, top\-kk, top\-pp, renormalization; the order matters and is a convention\. Truncation should be read as a*correction for misspecification in the tail*\.

### 5\.4Stage \(b\): performing the draw or how the next word Is finally chosen

Once the predictive measureπt\\pi\_\{t\}has been transformed into a modified distributionπ~t\\tilde\{\\pi\}\_\{t\}via temperature scaling, truncation \(top\-kk, top\-pp\), or typical set selection, the final step is torealize a single drawfromπ~t\\tilde\{\\pi\}\_\{t\}\. Except for greedy decoding \(which bypasses this step by deterministically taking thearg⁡max\\arg\\max\), all other strategies require a concrete algorithmic procedure to select a token according to the probabilities specified byπ~t\\tilde\{\\pi\}\_\{t\}\. The mechanics of this selection fall into two standard methods:inverse transform sampling\(the conceptual gold standard\) and theGumbel\-max trick\(the computationally efficient workhorse\)\.

LetA=\{v1,…,vm\}A=\\\{v\_\{1\},\\dots,v\_\{m\}\\\}be the support ofπ~\\tilde\{\\pi\}, that is, the set of tokens still carrying positive probability after stage \(a\)\. Thusm=\|A\|=\|𝒱\|m=\|A\|=\|\\mathcal\{V\}\|if no truncation was applied,m=km=kunder top\-kk, andm=\|Ap\|m=\|A\_\{p\}\|under top\-pp— in which casemmvaries from step to step, being small whereπt\\pi\_\{t\}is concentrated and large where it is spread out\.

The enumerationv1,…,vmv\_\{1\},\\dots,v\_\{m\}is merely a labeling of the elements ofAA: we number them in order to write down cumulative sums\. The order is neither random nor prescribed, and the law ofXtX\_\{t\}does not depend on it — the segment below is cut into the samemmpieces whatever the arrangement, and a uniform dart is indifferent to how they are arranged\. BothAAand its labeling are of course recomputed at every steptt, sinceπt\\pi\_\{t\}changes with the context\. In practice the decreasing order is used, because top\-pphas already sortedπ\\piin order to determineApA\_\{p\}, so the cumulative sums come free and the search below is a binary search\.

##### Inverse transform\.

WithF0=0F\_\{0\}=0andFj=∑i≤jπ~​\(vi\)F\_\{j\}=\\sum\_\{i\\leq j\}\\tilde\{\\pi\}\(v\_\{i\}\)forj=1,…,mj=1,\\dots,m, so thatFm=1F\_\{m\}=1, letUUbe a random variable uniformly distributed on\[0,1\]\[0,1\], drawn independently at each step\. Given its realizationU⁡\(ω\)U\(\\omega\), set

Xt=vJ,J=min⁡\{j:U⁡\(ω\)≤Fj\}\.X\_\{t\}=v\_\{J\},\\qquad J=\\min\\\{j:\\;U\(\\omega\)\\leq F\_\{j\}\\\}\.\(18\)The rule is correct because\{J=j\}=\{Fj−1<U≤Fj\}\\\{J=j\\\}=\\\{F\_\{j\-1\}<U\\leq F\_\{j\}\\\}, an interval of lengthFj−Fj−1F\_\{j\}\-F\_\{j\-1\}, and a uniform variable falls in an interval with probability equal to its length; hence

ℙ⁡\(Xt=vj\)=Fj−Fj−1=π~​\(vj\)\.\\mathbb\{P\}\(X\_\{t\}=v\_\{j\}\)=F\_\{j\}\-F\_\{j\-1\}=\\tilde\{\\pi\}\(v\_\{j\}\)\.\(19\)This is the dart and the segment, literally: theFjF\_\{j\}are the cut points andU⁡\(ω\)U\(\\omega\)is the dart\.

Two points deserve emphasis\. First, the successive drawsU1,U2,…U\_\{1\},U\_\{2\},\\dotsare independent of one another and of everything else; this independence is what makes the sequential procedure produce an exact sample from the joint law, as shown in §[5\.5](https://arxiv.org/html/2609.25134#S5.SS5)\. Second, all the modeling has already been done by the time we reach this step: the transformer determines the lengths of the pieces, and the dart is a plain uniform variable carrying no information about language\.

##### Gumbel\-max\.

In practice, particularly on parallel hardware such as GPUs, inverse transform sampling is not optimal because it requires sorting the tokens or computing cumulative sums sequentially\. The Gumbel\-max trick provides a mathematically equivalent but parallelisable alternative: it avoids both the sorting and the normalization: perturb each logit by independent noise and take the largest\.

LetGGfollow the standard*Gumbel*distribution,ℙ⁡\(G≤x\)=exp⁡\(−e−x\)\\mathbb\{P\}\(G\\leq x\)=\\exp\(\-e^\{\-x\}\)— the classical extreme\-value law of type I, and easy to simulate, sinceG=−log⁡\(−log⁡U\)G=\-\\log\(\-\\log U\)withUUuniform on\[0,1\]\[0,1\]\. Draw\(Gv\)v∈𝒱\(G\_\{v\}\)\_\{v\\in\\mathcal\{V\}\}i\.i\.d\. from it and set

Xt=arg​maxv∈𝒱⁡\(zvT\+Gv\)\.X\_\{t\}=\\argmax\_\{v\\in\\mathcal\{V\}\}\\left\(\\frac\{z\_\{v\}\}\{T\}\+G\_\{v\}\\right\)\.\(20\)Then thenXtX\_\{t\}is distributed exactly according toπ~\\tilde\{\\pi\}:Xt∼π\(T\)X\_\{t\}\\sim\\pi^\{\(T\)\}\. The verification is easy: Writesv=zv/Ts\_\{v\}=z\_\{v\}/TandMv=sv\+GvM\_\{v\}=s\_\{v\}\+G\_\{v\}, so thatℙ⁡\(Mv≤x\)=exp⁡\(−esv​e−x\)\\mathbb\{P\}\(M\_\{v\}\\leq x\)=\\exp\(\-e^\{s\_\{v\}\}e^\{\-x\}\)andMvM\_\{v\}has densityesv​e−x​exp⁡\(−esv​e−x\)e^\{s\_\{v\}\}e^\{\-x\}\\exp\(\-e^\{s\_\{v\}\}e^\{\-x\}\)\. Then

ℙ⁡\(arg​maxu⁡Mu=v\)=∫ℝesv​e−x​exp⁡\(−\(∑uesu\)​e−x\)​𝑑x=esv∑uesu=π\(T\)​\(v\),\\mathbb\{P\}\\\!\\left\(\\argmax\_\{u\}M\_\{u\}=v\\right\)=\\int\_\{\\mathbb\{R\}\}e^\{s\_\{v\}\}e^\{\-x\}\\exp\\\!\\Big\(\-\\Big\(\\textstyle\\sum\_\{u\}e^\{s\_\{u\}\}\\Big\)e^\{\-x\}\\Big\)\\,dx=\\frac\{e^\{s\_\{v\}\}\}\{\\sum\_\{u\}e^\{s\_\{u\}\}\}=\\pi^\{\(T\)\}\(v\),\(21\)the middle equality following from the substitutiony=e−xy=e^\{\-x\}, which turns the integral into∫0∞esv​e−S​y​𝑑y\\int\_\{0\}^\{\\infty\}e^\{s\_\{v\}\}e^\{\-Sy\}dywithS=∑uesuS=\\sum\_\{u\}e^\{s\_\{u\}\}\.

The Gumbel\-max trick is a direct consequence of the fact that the Gumbel distribution is the max\-stable distribution for the exponential family \([20](https://arxiv.org/html/2609.25134#S5.E20)\)\. Its advantage is computational: it requires only one pass over the vocabulary \(to add noise and take a maximum\), with no sorting, no cumulative sums, and no binary search\. Every coordinate is treated independently — so the whole operation is one parallel pass followed by a maximum — and that the normalizing constant never appears\. Truncation is incorporated simply by settingzv=−∞z\_\{v\}=\-\\inftyforv∉Av\\notin Abefore applying \([20](https://arxiv.org/html/2609.25134#S5.E20)\)\. As a by\-product, the maximum itself is Gumbel with locationlog∑uesu\\log\\sum\_\{u\}e^\{s\_\{u\}\}and is independent of the argmax, which is what allows the construction to be extended to sampling several tokens without replacement\. This is why modern implementations of temperature scaling and nucleus sampling use this trick internally; it is particularly efficient when the support ofπ~\\tilde\{\\pi\}is the full vocabulary and the distribution is evaluated in vectorized operations\.

### 5\.5Why truncated sampling does not collapse to greedy decoding

We rephrase here the arguments discussed above on the mode\. The most probable word keeps the largest piece under every scheme, so sampling might seem a mere softening of greedy decoding, inheriting its myopia\. But it is not the case:

\(1\) Greedy decoding takes the most probable word at each step\. It is tempting to think that sampling will reproduce this most probable word\. But it does not : it writes it with probability0\.450\.45, but writes something else58%58\\%of the time : the advantage of the most probable word is no guarantee that it will be chosen\. Furthermore, the probability for choosing the most probable words inTTsteps, is, e\.g\.,0\.45T0\.45^\{T\}which will be very small ifTTis just1010or2020; here we assume the same probability0\.450\.45for the most probable words in theTTsteps\.

\(2\) The aim of not choosing,with certainty, the most probable next word \(i\.e\. not adopting greedy decoding\) is not variety for its own sake\. If a continuation is correct in45%45\\%of such contexts, writing it45%45\\%of the time is right; writing it always over\-represents it, writing it10%10\\%of the time may under\-represents it\.

\(3\) Greedy decoding is myopic because

arg​maxw1:T∏tπt\(wt\)≠\(arg​maxπ1,…,arg​maxπT\):\\argmax\_\{w\_\{1:T\}\}\\prod\_\{t\}\\pi\_\{t\}\(w\_\{t\}\)\\;\\neq\\;\\left\(\\argmax\\pi\_\{1\},\\dots,\\argmax\\pi\_\{T\}\\right\):\(22\)maximizing stepwise does not maximize the product\. Sampling, by contrast, factorizes*exactly*: drawingX1∼π1X\_\{1\}\\sim\\pi\_\{1\}, thenX2∼π2\(⋅∣X1\)X\_\{2\}\\sim\\pi\_\{2\}\(\\cdot\\mid X\_\{1\}\), and so on, the chain rule \([3](https://arxiv.org/html/2609.25134#S2.E3)\) gives\(X1,…,XT\)∼Pθ^​\(w1,…,wT\)\(X\_\{1\},\\dots,X\_\{T\}\)\\sim P\_\{\\hat\{\\theta\}\}\(w\_\{1\},\\dots,w\_\{T\}\)with no approximation\. A purely local procedure yields an exact sample from the global joint law\.

### 5\.6Practice in deployed systems

As of 2026, beam search has disappeared from conversational systems; the norm is ancestral sampling with temperature and nucleus truncation\. The API defaults of OpenAI, Anthropic and Google all set temperature to1\.01\.0, and all three recommend adjusting either temperature or top\-pp, not both — a recommendation without theoretical basis, composing a tilting and a conditioning being perfectly well defined, but hard to tune blind\. Open\-weight models are the only case where the settings of a deployed product are documented: DeepSeek reportsT=0\.6T=0\.6andp=0\.95p=0\.95, the former being the value used in its web interface, with the range0\.50\.5–0\.70\.7recommended to avoid both endless repetition and incoherence — precisely the two pathologies of a tilting that is too cold or too hot\. Closed products \(ChatGPT, Claude, Gemini\) do not disclose theirs\. A recent trend removes the knobs altogether: several frontier models now reject non\-default sampling parameters, the decoder ceasing to be a user choice and becoming a component of the model, co\-tuned with post\-training\.

### 5\.7The mass\-covering property of the forward KL divergence

The mass\-covering property, or zero\-avoiding property is a behavior stemming from the mathematical asymmetry of the Kullback–Leibler divergence, specifically the Forward KL,D\(P\|\|Q\)D\(P\|\|Q\)\. The contrast between the forward direction, which is mass\-covering \(or*inclusive*\), and the reverse directionD\(Q\|\|P\)D\(Q\|\|P\), which is mode\-seeking \(or*exclusive*\), is standard; see Bishop\[[19](https://arxiv.org/html/2609.25134#bib.bib19), §10\.1\.2\]for a graphical account and Minka\[[20](https://arxiv.org/html/2609.25134#bib.bib20)\]for the general treatment within theα\\alpha\-divergence family\. In the context of training models like LLMs, ifPeP\_\{e\}represents the empirical data distribution of the training corpus \(ourP^N\\widehat\{P\}\_\{N\}\), the sum in the formula:

D\(Pe\|\|Pθ\)=∑xPe\(x\)logPe​\(x\)Pθ​\(x\)D\(P\_\{e\}\|\|P\_\{\\theta\}\)=\\sum\_\{x\}P\_\{e\}\(x\)\\log\\frac\{P\_\{e\}\(x\)\}\{P\_\{\\theta\}\(x\)\}
is weighted by the true data distributionPe​\(x\)P\_\{e\}\(x\)\. If there is any specific data point or sequence where the empirical distribution has mass \(Pe​\(x\)\>0P\_\{e\}\(x\)\>0\), but the model assigns it a probability near zero \(Pθ​\(x\)→0P\_\{\\theta\}\(x\)\\to 0\), the ratioPe​\(x\)/Pθ​\(x\)P\_\{e\}\(x\)/P\_\{\\theta\}\(x\)grows infinitely large\. This drives the divergence, and therefore the training loss, toward\+∞\+\\infty\. The Result \(Mass\-Covering\): To avoid this infinite penalty, the modelPθP\_\{\\theta\}isforcedto assign non\-zero probability everywhere the true dataPeP\_\{e\}has mass and this is what happens:

- •Support Inclusion vs\. Support Restriction: at a first sight one may think that this will favor the trained corpus, but it is the opposite that happens\. The penalty enforcesPe​\(x\)\>0⟹Pθ​\(x\)\>0P\_\{e\}\(x\)\>0\\implies P\_\{\\theta\}\(x\)\>0, setting a strict floor onPθ​\(x\)P\_\{\\theta\}\(x\)across the training set\. Critically, it places zero penalty onPθ​\(x\)P\_\{\\theta\}\(x\)being non\-zero wherePe​\(x\)=0P\_\{e\}\(x\)=0\. It compelsPθP\_\{\\theta\}to cover the empirical corpus, but never forbidsPeP\_\{e\}from expanding beyond it\.
- •Forced Smoothing and Generalization: BecausePθP\_\{\\theta\}is parametrized by a continuous Transformer rather than a discrete lookup table, it cannot assign non\-zero probability to every training corpus sequencex∈𝒟x\\in\\mathcal\{D\}without smoothly assigning non\-zero probability to the continuous space surrounding them\. The requirement to cover all training points forcesPθP\_\{\\theta\}to "fill in the gaps," assigning probability mass to unseen, newly generated sequences \(x∉𝒱∗x\\notin\\mathcal\{V\}^\{\*\}\)\.
- •By forcingPθP\_\{\\theta\}to maintain non\-zero probability across all empirical data, Forward KL forces the model to maintain broad, permissive support\. This open support is the exact probabilistic mechanism that allows generative models to sample valid, novel token sequences during inference\.

## 6Summary of the decoder transformer procedure

For completeness we review here the decoder transformer\[[6](https://arxiv.org/html/2609.25134#bib.bib6)\]procedure which is the part used in Gen AI systems\.

In a mathematical form, the decoder transformer can be considered as a map from a token sequence to logits\. We describe its procedure and the learning and inference phases\.

##### Representations\.

Letddbe the model dimension\. To each positionttand each layerℓ\\ellis attached a vectorht\(ℓ\)∈ℝdh\_\{t\}^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{d\}, the*representation*of positionttafter layerℓ\\ell: a summary ofw≤tw\_\{\\leq t\}, refined layer by layer, from which the logits will eventually be read off\. Stacking them row\-wise givesH\(ℓ\)∈ℝT×dH^\{\(\\ell\)\}\\in\\mathbb\{R\}^\{T\\times d\}\. WithE∈ℝ\|𝒱\|×dE\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}the embedding matrix andptp\_\{t\}a positional encoding,

ht\(0\)=Ewt\+pt\.h^\{\(0\)\}\_\{t\}=E\_\{w\_\{t\}\}\+p\_\{t\}\.\(24\)

##### Layers\.

Forℓ=1,…,Ld\\ell=1,\\dots,L\_\{\\mathrm\{d\}\},

h~\(ℓ\)\\displaystyle\\tilde\{h\}^\{\(\\ell\)\}=h\(ℓ−1\)\+M​H​A​\(L​N​\(h\(ℓ−1\)\)\),\\displaystyle=h^\{\(\\ell\-1\)\}\+MHA\\\!\\left\(LN\\big\(h^\{\(\\ell\-1\)\}\\big\)\\right\),\(25\)h\(ℓ\)\\displaystyle h^\{\(\\ell\)\}=h~\(ℓ\)\+F​F​N​\(L​N​\(h~\(ℓ\)\)\),\\displaystyle=\\tilde\{h\}^\{\(\\ell\)\}\+FFN\\\!\\left\(LN\\big\(\\tilde\{h\}^\{\(\\ell\)\}\\big\)\\right\),\(26\)where the three operators are:

- •L​NLN\(*layer normalization*\) standardizes each vector across itsddcoordinates and rescales it,L​N​\(h\)=γ⊙\(h−μ\)/σ\+βLN\(h\)=\\gamma\\odot\(h\-\\mu\)/\\sigma\+\\betawithμ,σ\\mu,\\sigmathe mean and standard deviation of the entries ofhhandγ,β∈ℝd\\gamma,\\beta\\in\\mathbb\{R\}^\{d\}learned; it keeps activation scales stable with depth\.
- •M​H​AMHA\(*multi\-head attention*\) is the only operator mixing positions\. For one head, withQ=H​WQQ=HW\_\{Q\},K=H​WKK=HW\_\{K\},V=H​WVV=HW\_\{V\}, A​t​t​n​\(Q,K,V\)=s​o​f​t​m​a​x​\(Q​K⊤dk\+M\)​V,Mt​s=\{0s≤t,−∞s\>t;Attn\(Q,K,V\)=softmax\\\!\\left\(\\frac\{QK^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\+M\\right\)V,\\qquad M\_\{ts\}=\\begin\{cases\}0&s\\leq t,\\\\ \-\\infty&s\>t;\\end\{cases\}\(27\)M​H​AMHAconcatenates several such heads with distinct projections and recombines them byWOW\_\{O\}\.
- •F​F​NFFN\(*feed\-forward*\) acts on each position separately,F​F​N​\(h\)=W2​ϕ​\(W1​h\)FFN\(h\)=W\_\{2\}\\,\\phi\(W\_\{1\}h\)withW1∈ℝdf×dW\_\{1\}\\in\\mathbb\{R\}^\{d\_\{f\}\\times d\},W2∈ℝd×dfW\_\{2\}\\in\\mathbb\{R\}^\{d\\times d\_\{f\}\},df≈4​dd\_\{f\}\\approx 4dandϕ\\phia nonlinearity\.

Here⊙\\odotdenotes the componentwise product: fora,b∈ℝda,b\\in\\mathbb\{R\}^\{d\},\(a⊙b\)i=ai​bi\(a\\odot b\)\_\{i\}=a\_\{i\}b\_\{i\}\. Thusγ\\gammarescales each of theddcoordinates separately, andβ\\betashifts each separately, after the vector has been standardized\.

Finally, withWU∈ℝ\|𝒱\|×dW\_\{U\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}the unembedding matrix,

zt=WULN\(ht\(Ld\)\),Pθ\(⋅∣w≤t\)=softmax\(zt\),z\_\{t\}=W\_\{U\}\\,LN\\\!\\left\(h^\{\(L\_\{\\mathrm\{d\}\}\)\}\_\{t\}\\right\),\\qquad P\_\{\\theta\}\(\\cdot\\mid w\_\{\\leq t\}\)=softmax\(z\_\{t\}\),\(28\)which is the inner product⟨ht,uv⟩\\langle h\_\{t\},u\_\{v\}\\rangleof Section[3](https://arxiv.org/html/2609.25134#S3), the rows ofWUW\_\{U\}being the token vectorsuvu\_\{v\}\. The parameter is the whole collectionθ=\{E,p⋅,WQ,WK,WV,WO,W1,W2,γ,β,WU\}\\theta=\\\{E,\\,p\_\{\\cdot\},\\,W\_\{Q\},W\_\{K\},W\_\{V\},W\_\{O\},\\,W\_\{1\},W\_\{2\},\\,\\gamma,\\beta,\\,W\_\{U\}\\\}over all layers and heads\. NB\.The causal mask described in the § on the transformers is implemented byMMin \([27](https://arxiv.org/html/2609.25134#S6.E27)\) is the architectural encoding of the chain rule: it forcesztz\_\{t\}to depend onw≤tw\_\{\\leq t\}only, so that the network computes genuinely*predictive*kernels\. Without it the factorization \([3](https://arxiv.org/html/2609.25134#S2.E3)\) would not be satisfied and the model would “predict” a token by looking at it\.

##### Learning phase :

this was described in Section[4](https://arxiv.org/html/2609.25134#S4)\. The above procedure just presented the function being differentiated\. The composition above is smooth inθ\\theta, so∇θℒ\\nabla\_\{\\theta\}\\mathcal\{L\}is obtained by the chain rule of calculus applied backwards through the layers — backpropagation — andθ\\thetais updated by \([12](https://arxiv.org/html/2609.25134#S4.E12)\) and the minimization algorithm ofℒ\\mathcal\{L\}\. This paragraph provides an important practical fact: thanks to the causal mask, a*single*forward pass onw1:Tw\_\{1:T\}produces allTTlogit vectors in parallel, hence allTTconditionals of \([12](https://arxiv.org/html/2609.25134#S4.E12)\) at once, differentiated in one backward pass\. The contexts used are always the true corpus prefixes, never the model’s own output — this is*teacher forcing*\. Recurrent architectures cannot do this, which is the practical reason transformers scale\.

##### Inference phase :

the situation is the reverse of the preceding one\. During learning the sequence is known in advance, so every position can be processed at once\. During generation the sequence is being created as we go:zt\+1z\_\{t\+1\}cannot be computed beforewtw\_\{t\}has been drawn, sincewtw\_\{t\}is part of the context on whichzt\+1z\_\{t\+1\}depends\. Generation therefore requires one forward pass*per token*,TTpasses forTTtokens\.

## 7On the probabilistic nature of LLMs

The probabilistic nature of Gen AI is multi\-tiered: first, it stems fundamentally from the underlying probabilistic language model, which maps semantic properties to joint probability distributions\. Second, it is further shaped at the operational level by stochastic techniques and sampling applied during text generation\. More precisely:

\(1\) The object learned is a probability measure\.The basic objects of the model are probability measures on the spaceΩ\\Omegaof finite sequences of words\. It leads to a family of Markov kernelsw<t↦Pθ\(⋅∣w<t\)w\_\{<t\}\\mapsto P\_\{\\theta\}\(\\cdot\\mid w\_\{<t\}\), which by Section[3](https://arxiv.org/html/2609.25134#S3)is used to predict the next wordwtw\_\{t\}given the set of word generated untilt−1t\-1:w<t=\{w1,…,wt−1\}w\_\{<t\}=\\\{w\_\{1\},\\ldots,w\_\{t\-1\}\\\}\. The reason is not computational convenience but the nature of language: for a given context several continuations are simultaneously correct, and a measure is the only object able to represent a set of admissible answers together with their relative plausibility\. A deterministic map would have to choose arbitrarily among them and would be penalized for whichever it did not choose\.

\(2\) The estimation is statistical inference\.The corpus is a finite sample from an unknownP⋆P^\{\\star\}; the criterion \([8](https://arxiv.org/html/2609.25134#S4.E8)\) is maximum likelihood, i\.e\. an information projection ofP^N\\widehat\{P\}\_\{N\}onto𝒫Θ\\mathcal\{P\}\_\{\\Theta\}; the algorithm is stochastic approximation\. The quality of fit is measured by*perplexity*,

PP=exp\(−1T∑t=1TlogPθ^\(wt∣w<t\)\),\\mathrm\{PP\}=\\exp\\\!\\left\(\-\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\log P\_\{\\hat\{\\theta\}\}\(w\_\{t\}\\mid w\_\{<t\}\)\\right\),\(29\)the exponential of the empirical cross\-entropy, interpretable as an effective branching factor\. As the entropy, \([29](https://arxiv.org/html/2609.25134#S7.E29)\) means that when the probabilityPθ^\(\.∣w<t\)\)P\_\{\\hat\{\\theta\}\}\(\.\\mid w\_\{<t\}\)\)becomes dispersed \(i\.e\. more random\) the perplexity becomes higher: thePP=10\\mathrm\{PP\}=10means the model is on average as uncertain as if choosing uniformly among ten tokens\. Were the language a stationary ergodic source of entropy ratehh, no model could achievePP<eh\\mathrm\{PP\}<e^\{h\}\[[15](https://arxiv.org/html/2609.25134#bib.bib15)\]\.

\(3\) The generation is a random draw\.The output is a realization of the Markov chain of Remark[4](https://arxiv.org/html/2609.25134#Thmremark4), started at the prompt and sampled by the rules of Section[5](https://arxiv.org/html/2609.25134#S5)\. Two identical prompts yield different outputs because two independent simulations of the same chain differ — not because the model “reasons differently”\. Temperature and top\-ppare respectively an exponential tilting and a conditioning of the transition kernel: they modify the law of the chain being simulated\. Familiar behaviors follow as corollaries\.*Non\-determinism*is \(3\)\.*Sensitivity to the prompt*is the dependence of a Markov chain on its initial condition\.*Hallucination*is the conjunction of \(1\) with the mass\-covering property of forward KL: a false but linguistically plausible statement is a high\-Pθ^P\_\{\\hat\{\\theta\}\}string, and nothing in \([8](https://arxiv.org/html/2609.25134#S4.E8)\) distinguishes truth from plausibility — there is no term in the objective referring to the world or the truth\. This is more than an informal argument: Kalai and Vempala\[[27](https://arxiv.org/html/2609.25134#bib.bib27)\]prove a statistical lower bound on the hallucination rate of a calibrated model, of the order of the fraction of facts appearing exactly once in the training corpus — a Good–Turing missing\-mass estimate — independently of the architecture or of the quality of the data\. Related to this,Pθ^P\_\{\\hat\{\\theta\}\}is a measure*on all possible strings*, not only the meaningful ones;Pθ^​\(wt∣w<t\)=0\.7P\_\{\\hat\{\\theta\}\}\(w\_\{t\}\\mid w\_\{<t\}\)=0\.7is an estimated token frequency, not a degree of belief in a proposition, and is not automatically calibrated as one\. Post\-training with human feedback deformsPθ^P\_\{\\hat\{\\theta\}\}towards a rewardrr\[[14](https://arxiv.org/html/2609.25134#bib.bib14)\]via

maxθ𝔼y∼πθ\(⋅∣x\)\[r\(x,y\)\]−βD\(πθ\(⋅∣x\)∥πref\(⋅∣x\)\),\\max\_\{\\theta\}\\;\\mathbb\{E\}\_\{y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}\\\!\\left\[r\(x,y\)\\right\]\-\\beta\\,D\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid x\)\\,\\\|\\,\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)\\right\),\(30\)whose solutionπ∗​\(y∣x\)∝πref​\(y∣x\)​exp⁡\{r⁡\(x,y\)/β\}\\pi^\{\*\}\(y\\mid x\)\\propto\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\\exp\\\{r\(x,y\)/\\beta\\\}is again a Gibbs measure — a deformation of a probability law, not a departure from the probabilistic framework\.

##### Generative AI and its predecessors\.

It is tempting to summarize the development by saying that classical neural networks are deterministic and generative models probabilistic\. This is partially true\. The first half is right, and holds of both: a network — RNN or transformer — is a deterministic map, and given a context andθ\\thetathe logitsztz\_\{t\}are entirely determined\. What is probabilistic is not the computation but its*output*, sinceztz\_\{t\}is read through a softmax as a measure on𝒱\\mathcal\{V\}; and this too is common to both, an RNN language model\[[2](https://arxiv.org/html/2609.25134#bib.bib2)\]having the same terminal softmax, the same factorization \([3](https://arxiv.org/html/2609.25134#S2.E3)\) and the same loss \([12](https://arxiv.org/html/2609.25134#S4.E12)\) as a transformer\. The two architectures differ in how the context is summarized — recurrence against attention — not in the nature of what they output\. Attention\[[5](https://arxiv.org/html/2609.25134#bib.bib5)\]was itself introduced within sequence\-to\-sequence models that were already conditional language models\[[3](https://arxiv.org/html/2609.25134#bib.bib3),[4](https://arxiv.org/html/2609.25134#bib.bib4)\]\. The shift lies in the following places:

*The status of the measure\.*In discriminative usage, the distribution is a means: one reads off its mode and discards the rest, so only the ranking of the top candidates matters\. In generative usage, the distribution is the product: one*samples*from it, and the whole shape of the measure matters, its tail included — which is why in Sections[4](https://arxiv.org/html/2609.25134#S4)and[5](https://arxiv.org/html/2609.25134#S5)we were led to maintain consideration of that tail\.

*The output space\.*One passes from a measure on a small label set to a measure on𝒱∗\\mathcal\{V\}^\{\*\}, an unbounded combinatorial space\. It is precisely this that makes the autoregressive factorization necessary rather than optional\. A classifier over a thousand labels can be specified exhaustively: one lists the thousand probabilities and the measure is fully described\. A language model admits no such list\. With\|𝒱\|≈5×104\|\\mathcal\{V\}\|\\approx 5\\times 10^\{4\}there are roughly1047010^\{470\}possible texts of a hundred tokens \(a huge number\)\.

The measure is therefore specified indirectly\. At each position the model produces only the conditional law of the next token, a vector of\|𝒱\|≈5×104\|\\mathcal\{V\}\|\\approx 5\\times 10^\{4\}numbers; a hundred such vectors, some5×1065\\times 10^\{6\}numbers in all, determine the probability of every one of the1047010^\{470\}sequences through the chain rule \([3](https://arxiv.org/html/2609.25134#S2.E3)\)\. The factorization is thus not a modeling convenience but the only possibility by which a measure on this space can be written\.

*The feedback loop\.*The output is appended to the input\. Hence, the model’s own randomness enters the conditioning of its future draws, and the object produced is not a single conditional law but a trajectory of a stochastic process — the Markov chain of Remark[4](https://arxiv.org/html/2609.25134#Thmremark4)\. This is what has no counterpart in a classifier, and it is the reason the whole apparatus of simulation, and not merely of estimation, is required\. The reorientation is thus from function approximation to*the simulation of a stochastic process*, the estimation of a conditional law being the common ground rather than the novelty\.

## 8The Diffusion Models Approach : From Gaussian Transitions to Stochastic Differential Equations

Diffusion models provide another probabilistic paradigm for generative artificial intelligence\. While autoregressive language models generate a sequence of tokens successively by sampling from conditional probability distributions, diffusion models formulate generation differently: noise is added progressively to a data point until it becomes indistinguishable from pure Gaussian noise; a network is then trained to reverse this process step by step, so that, once trained, it can transform a fresh sample of pure noise into a new, realistic data point\. The training itself only ever sees noise added to real data; it is at generation time that the process is run in reverse, starting from noise that has no data underneath it\. See the example below\. This approach has become successful for image generation and has subsequently been extended to audio, video, three\-dimensional data, and scientific and medical imaging\[[28](https://arxiv.org/html/2609.25134#bib.bib28),[29](https://arxiv.org/html/2609.25134#bib.bib29),[33](https://arxiv.org/html/2609.25134#bib.bib33)\]\.

### 8\.1Object and Motivations

The goal of generative modeling is to learn an approximation of an unknown data distributionpdata​\(𝐱\)p\_\{\\text\{data\}\}\(\\mathbf\{x\}\)defined over a high\-dimensional space, and then to draw new samples𝐱∼pdata\\mathbf\{x\}\\sim p\_\{\\text\{data\}\}\. Here, the symbol𝐱∈ℝd\\mathbf\{x\}\\in\\mathbb\{R\}^\{d\}denotes a single data point \(for instance, an image flattened into a vector ofddpixel intensities\), andpdatap\_\{\\text\{data\}\}is the probability density function we wish to model\. Classical approaches—variational autoencoders, generative adversarial networks, and normalizing flows—try to solve this problem differently but face the same fundamental difficulty: directly parameterizing and sampling from a highly complex, multimodal distribution in high dimension is intractable\.

Diffusion models rely on the following idea, which is adapted to contents like images: instead of learning to map noise to data in one shot, they learn to*gradually*transform a simple distribution \(a standard Gaussian\) into the data distribution through a sequence of small, invertible\-in\-distribution steps\. This idea was introduced by Sohl\-Dickstein et al\.\[[28](https://arxiv.org/html/2609.25134#bib.bib28)\]and developed by Ho et al\.\[[29](https://arxiv.org/html/2609.25134#bib.bib29)\]with Denoising Diffusion Probabilistic Models \(DDPM\), and by Song et al\.\[[30](https://arxiv.org/html/2609.25134#bib.bib30)\], who showed that the discrete construction has a continuous\-time limit expressed as a stochastic differential equation \(SDE\)\. The remainder of this section follows this order: we begin with the discrete\-time formulation and the crucial role played by Gaussian transition kernels, and at the end we take the continuous\-time limit that yields the SDE formulation\.

### 8\.2The Intuitive Idea

Let us consider a clean image𝐱0\\mathbf\{x\}\_\{0\}and add a small amount of Gaussian noise to it, over and over again, forTTsteps\. After many steps, the image becomes unrecognizable before being statistically indistinguishable from pure Gaussian noise\. This is theforward process: it is fixed and requires no learning\. Now we reverse this operation by starting from pure noise and removing a tiny bit of noise at each step\. If we knew exactly how the noise was added, we could invert the process and recover a clean image\. Thereverse processis what we learn: a network is trained to undo one step of noising at a time\. Once this training is done on a big data set, we can generate new data by starting from a fresh Gaussian sample and applying the learned reverse steps\. The whole construction is a probabilistic analogue of annealing: we destroy structure gradually, then learn to rebuild it gradually\.

As an example, suppose we have a dataset of500500cat images\. We first run the forward process on each of them: every cat image is progressively randomized by Gaussian noise until, afterTTsteps, nothing remains but noise\. We then run the learning process: a network is trained to perform the reverse operation, i\.e\. to remove noise step by step\. Once training is complete, we can use the model to generate a brand\-new image of a cat that never existed: we start from a fresh Gaussian noise sample and apply the learned reverse process\. Moreover, if we feed the network additional information—such as a text caption—at every denoising step, the model can learn to generate images conditioned on text, which is the basis of text\-to\-image systems \(see Section[8\.4](https://arxiv.org/html/2609.25134#S8.SS4)\)\.

It is worth being precise about what the symbols𝐱0,𝐱1,…,𝐱T\\mathbf\{x\}\_\{0\},\\mathbf\{x\}\_\{1\},\\dots,\\mathbf\{x\}\_\{T\}represent\. Each𝐱t\\mathbf\{x\}\_\{t\}encodes an image as a vector of pixel intensities:

𝐱t=\(y1,y2,…,yd\),\\mathbf\{x\}\_\{t\}=\(y\_\{1\},y\_\{2\},\\dots,y\_\{d\}\),whereyjy\_\{j\}is the intensity \(or color value\) at pixeljj, andddis the total number of pixels in the image\. For a1024×10241024\\times 1024RGB image, for instance,d=1024×1024×3≈3\.1d=1024\\times 1024\\times 3\\approx 3\.1million\. Soddcan be several million, which is precisely what makes direct density modeling so difficult and what makes the sequential, Gaussian\-based approach of diffusion models so attractive\.

### 8\.3Discrete\-Time Formulation: Gaussian Transition Probabilities

##### Notation\.

Let us fix the notation that will be used in the sequel\.

- •𝐱0∈ℝd\\mathbf\{x\}\_\{0\}\\in\\mathbb\{R\}^\{d\}denotes a clean data point sampled from the unknown data distributionpdatap\_\{\\text\{data\}\}\. The subscript00indicates “time zero,” i\.e\. the clean, uncorrupted data\.
- •𝐱1,𝐱2,…,𝐱T\\mathbf\{x\}\_\{1\},\\mathbf\{x\}\_\{2\},\\dots,\\mathbf\{x\}\_\{T\}denote successively noisier versions of𝐱0\\mathbf\{x\}\_\{0\}\. Each𝐱t∈ℝd\\mathbf\{x\}\_\{t\}\\in\\mathbb\{R\}^\{d\}has the same dimension as the data\. The integert∈\{0,1,…,T\}t\\in\\\{0,1,\\dots,T\\\}is a discrete time index, andTTis the total number of diffusion steps \(e\.g\.,T=1000T=1000in image applications\)\.
- •q⁡\(⋅\)q\(\\cdot\)denotes the*forward*\(noising\) distribution, which is fixed and known\.
- •pθ​\(⋅\)p\_\{\\theta\}\(\\cdot\)denotes the*reverse*\(denoising\) distribution, parameterized by a neural network with weightsθ\\theta\.
- •𝒩⁡\(𝝁,𝚺\)\\mathcal\{N\}\(\\bm\{\\mu\},\\bm\{\\Sigma\}\)denotes a multivariate Gaussian distribution with mean vector𝝁\\bm\{\\mu\}and covariance matrix𝚺\\bm\{\\Sigma\}\. When𝚺=σ2​𝐈\\bm\{\\Sigma\}=\\sigma^\{2\}\\mathbf\{I\}is isotropic, we write𝒩⁡\(𝝁,σ2​𝐈\)\\mathcal\{N\}\(\\bm\{\\mu\},\\sigma^\{2\}\\mathbf\{I\}\)\. Here𝐈∈ℝd×d\\mathbf\{I\}\\in\\mathbb\{R\}^\{d\\times d\}is the identity matrix\.
- •β1,…,βT∈\(0,1\)\\beta\_\{1\},\\dots,\\beta\_\{T\}\\in\(0,1\)is a sequence of small positive scalars called thenoise schedule\(or variance schedule\)\. It controls how much noise is injected at each step\.

Remark\.The long form𝒩⁡\(𝐱,𝝁,𝚺\)\\mathcal\{N\}\(\\mathbf\{x\};\\bm\{\\mu\},\\bm\{\\Sigma\}\)gives the*density*of that distribution evaluated at the point𝐱\\mathbf\{x\}; the semicolon separates the variable𝐱\\mathbf\{x\}from the parameters𝝁\\bm\{\\mu\}and𝚺\\bm\{\\Sigma\}\. Explicitly,

𝒩⁡\(𝐱,𝝁,𝚺\)=1\(2​π\)d/2​\|𝚺\|1/2​exp⁡\(−12​\(𝐱−𝝁\)⊤​𝚺−1​\(𝐱−𝝁\)\),\\mathcal\{N\}\(\\mathbf\{x\};\\bm\{\\mu\},\\bm\{\\Sigma\}\)=\\frac\{1\}\{\(2\\pi\)^\{d/2\}\|\\bm\{\\Sigma\}\|^\{1/2\}\}\\exp\\\!\\left\(\-\\tfrac\{1\}\{2\}\(\\mathbf\{x\}\-\\bm\{\\mu\}\)^\{\\top\}\\bm\{\\Sigma\}^\{\-1\}\(\\mathbf\{x\}\-\\bm\{\\mu\}\)\\right\),which is a scalar \(a density value\), whereas𝒩⁡\(𝝁,𝚺\)\\mathcal\{N\}\(\\bm\{\\mu\},\\bm\{\\Sigma\}\)is a distribution\. In what follows we use the long form when we want to be explicit about the variable—especially for conditional densities such asq⁡\(𝐱t∣𝐱t−1\)q\(\\mathbf\{x\}\_\{t\}\\mid\\mathbf\{x\}\_\{t\-1\}\)—and the short form when we only want to name the law\.

#### 8\.3\.1The Forward Process

The forward process is a Markov chain defined by Gaussian transition kernels:

q\(𝐱t∣𝐱t−1\)=𝒩\(𝐱t;1−βt𝐱t−1,βt𝐈\),t=1,…,T\.q\(\\mathbf\{x\}\_\{t\}\\mid\\mathbf\{x\}\_\{t\-1\}\)=\\mathcal\{N\}\\\!\\left\(\\mathbf\{x\}\_\{t\};\\ \\sqrt\{1\-\\beta\_\{t\}\}\\,\\mathbf\{x\}\_\{t\-1\},\\ \\beta\_\{t\}\\mathbf\{I\}\\right\),\\qquad t=1,\\dots,T\.\(32\)
Let us now examine this expression\. The notationq⁡\(𝐱t∣𝐱t−1\)q\(\\mathbf\{x\}\_\{t\}\\mid\\mathbf\{x\}\_\{t\-1\}\)denotes the conditional probability density of the noisy variable𝐱t\\mathbf\{x\}\_\{t\}given the previous variable𝐱t−1\\mathbf\{x\}\_\{t\-1\}\. The right\-hand side says that𝐱t\\mathbf\{x\}\_\{t\}is drawn from a Gaussian whose mean is a scaled copy of𝐱t−1\\mathbf\{x\}\_\{t\-1\}—namely1−βt​𝐱t−1\\sqrt\{1\-\\beta\_\{t\}\}\\,\\mathbf\{x\}\_\{t\-1\}—and whose covariance isβt​𝐈\\beta\_\{t\}\\mathbf\{I\}, i\.e\. isotropic noise with varianceβt\\beta\_\{t\}in every coordinate\. In the notation𝒩⁡\(𝐱t,1−βt​𝐱t−1,βt​𝐈\)\\mathcal\{N\}\(\\mathbf\{x\}\_\{t\};\\sqrt\{1\-\\beta\_\{t\}\}\\,\\mathbf\{x\}\_\{t\-1\},\\beta\_\{t\}\\mathbf\{I\}\), the first argument𝐱t\\mathbf\{x\}\_\{t\}is the variable, the second is the mean \(which depends on𝐱t−1\\mathbf\{x\}\_\{t\-1\}, hence the conditional nature of the kernel\), and the third is the covariance\.

An equivalent and often more convenient way to write the same transition is via the*reparameterization trick*:

𝐱t=1−βt​𝐱t−1\+βt​ϵt,ϵt∼𝒩⁡\(𝟎,𝐈\)\.\\mathbf\{x\}\_\{t\}=\\sqrt\{1\-\\beta\_\{t\}\}\\,\\mathbf\{x\}\_\{t\-1\}\+\\sqrt\{\\beta\_\{t\}\}\\,\\bm\{\\epsilon\}\_\{t\},\\qquad\\bm\{\\epsilon\}\_\{t\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)\.\(33\)
Hereϵt\\bm\{\\epsilon\}\_\{t\}is a standard Gaussian noise vector \(mean𝟎∈ℝd\\mathbf\{0\}\\in\\mathbb\{R\}^\{d\}, covariance𝐈\\mathbf\{I\}\), and the two coefficients1−βt\\sqrt\{1\-\\beta\_\{t\}\}andβt\\sqrt\{\\beta\_\{t\}\}are chosen so that the*variance is preserved*:

Var⁡\(𝐱t\)=\(1−βt\)​Var⁡\(𝐱t−1\)\+βt\.\\operatorname\{Var\}\(\\mathbf\{x\}\_\{t\}\)=\(1\-\\beta\_\{t\}\)\\,\\operatorname\{Var\}\(\\mathbf\{x\}\_\{t\-1\}\)\+\\beta\_\{t\}\.If𝐱t−1\\mathbf\{x\}\_\{t\-1\}has unit variance, then𝐱t\\mathbf\{x\}\_\{t\}also has unit variance\. Thisvariance\-preservingproperty ensures that the signal does not blow up or vanish asttgrows, and it is why the mean is scaled by1−βt\\sqrt\{1\-\\beta\_\{t\}\}rather than by11\.

##### The Gaussian transitions choice\.

It is motivated by the universal character of Gaussian distributions which provides several features and advantages :

Closed\-form marginals\.Because the composition of Gaussian transitions is again Gaussian, we can write the distribution of

𝐱t\\mathbf\{x\}\_\{t\}conditioned on the original data

𝐱0\\mathbf\{x\}\_\{0\}in closed form\. Defining

αt=1−βt\\alpha\_\{t\}=1\-\\beta\_\{t\}and

α¯t=∏s=1tαs\\bar\{\\alpha\}\_\{t\}=\\prod\_\{s=1\}^\{t\}\\alpha\_\{s\}, one obtains

q⁡\(𝐱t∣𝐱0\)=𝒩⁡\(𝐱t,α¯t​𝐱0,\(1−α¯t\)​𝐈\),q\(\\mathbf\{x\}\_\{t\}\\mid\\mathbf\{x\}\_\{0\}\)=\\mathcal\{N\}\\\!\\left\(\\mathbf\{x\}\_\{t\};\\ \\sqrt\{\\bar\{\\alpha\}\_\{t\}\}\\,\\mathbf\{x\}\_\{0\},\\ \(1\-\\bar\{\\alpha\}\_\{t\}\)\\,\\mathbf\{I\}\\right\),\(34\)or equivalently, in reparameterized form,

𝐱t=α¯t​𝐱0\+1−α¯t​ϵ,ϵ∼𝒩⁡\(𝟎,𝐈\)\.\\mathbf\{x\}\_\{t\}=\\sqrt\{\\bar\{\\alpha\}\_\{t\}\}\\,\\mathbf\{x\}\_\{0\}\+\\sqrt\{1\-\\bar\{\\alpha\}\_\{t\}\}\\,\\bm\{\\epsilon\},\\qquad\\bm\{\\epsilon\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)\.\(35\)This single\-shot formula is essential: it allows us to sample

𝐱t\\mathbf\{x\}\_\{t\}directly from

𝐱0\\mathbf\{x\}\_\{0\}without simulating all intermediate steps, which makes training efficient\. As

t→Tt\\to Tand

α¯t→0\\bar\{\\alpha\}\_\{t\}\\to 0, the marginal

q⁡\(𝐱t∣𝐱0\)q\(\\mathbf\{x\}\_\{t\}\\mid\\mathbf\{x\}\_\{0\}\)converges to

𝒩⁡\(𝟎,𝐈\)\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\), so the forward process indeed destroys all information about the data\. The central limit theorem\.Even if the individual transitions were not Gaussian, the cumulative effect of many small independent perturbations tends toward a Gaussian by the central limit theorem\. Choosing Gaussian transitions from the start makes the limiting distribution exactly Gaussian and analytically tractable\.

Tractable reverse process\.The reverse transition

q⁡\(𝐱t−1∣𝐱t\)q\(\\mathbf\{x\}\_\{t\-1\}\\mid\\mathbf\{x\}\_\{t\}\)is also Gaussian when the forward transitions are Gaussian and

βt\\beta\_\{t\}is small\. Specifically, a standard computation using Bayes’ rule gives

q⁡\(𝐱t−1∣𝐱t,𝐱0\)=𝒩⁡\(𝐱t−1,𝝁~t​\(𝐱t,𝐱0\),β~t​𝐈\),q\(\\mathbf\{x\}\_\{t\-1\}\\mid\\mathbf\{x\}\_\{t\},\\mathbf\{x\}\_\{0\}\)=\\mathcal\{N\}\\\!\\left\(\\mathbf\{x\}\_\{t\-1\};\\ \\tilde\{\\bm\{\\mu\}\}\_\{t\}\(\\mathbf\{x\}\_\{t\},\\mathbf\{x\}\_\{0\}\),\\ \\tilde\{\\beta\}\_\{t\}\\mathbf\{I\}\\right\),\(36\)with

𝝁~t​\(𝐱t,𝐱0\)\\displaystyle\\tilde\{\\bm\{\\mu\}\}\_\{t\}\(\\mathbf\{x\}\_\{t\},\\mathbf\{x\}\_\{0\}\)=α¯t−1​βt1−α¯t​𝐱0\+αt​\(1−α¯t−1\)1−α¯t​𝐱t,\\displaystyle=\\frac\{\\sqrt\{\\bar\{\\alpha\}\_\{t\-1\}\}\\,\\beta\_\{t\}\}\{1\-\\bar\{\\alpha\}\_\{t\}\}\\,\\mathbf\{x\}\_\{0\}\+\\frac\{\\sqrt\{\\alpha\_\{t\}\}\\,\(1\-\\bar\{\\alpha\}\_\{t\-1\}\)\}\{1\-\\bar\{\\alpha\}\_\{t\}\}\\,\\mathbf\{x\}\_\{t\},\(37\)β~t\\displaystyle\\tilde\{\\beta\}\_\{t\}=1−α¯t−11−α¯t​βt\.\\displaystyle=\\frac\{1\-\\bar\{\\alpha\}\_\{t\-1\}\}\{1\-\\bar\{\\alpha\}\_\{t\}\}\\,\\beta\_\{t\}\.\(38\)This closed\-form reverse kernel is what makes the training objective simple and the sampling procedure well\-defined\.

Connection to score matching and Langevin dynamics\.A Gaussian perturbation kernel

q⁡\(𝐱t∣𝐱0\)q\(\\mathbf\{x\}\_\{t\}\\mid\\mathbf\{x\}\_\{0\}\)has a score function

∇𝐱t​log​q​\(𝐱t∣𝐱0\)\\nabla\_\{\\mathbf\{x\}\_\{t\}\}\\log q\(\\mathbf\{x\}\_\{t\}\\mid\\mathbf\{x\}\_\{0\}\)that is linear in

𝐱t\\mathbf\{x\}\_\{t\}\(see the digression below for the interest and usage of score functions\)\. This linearity is what allows denoising score matching to reduce to simple noise prediction, and it is the discrete precursor of the score\-based formulation that later becomes an SDE\.

##### Digression on the Score Function

For a probability densityp⁡\(𝐱\)p\(\\mathbf\{x\}\)onℝd\\mathbb\{R\}^\{d\}, the*score function*is defined as the gradient of the logarithm of the density with respect to the variable,

s⁡\(𝐱\)=∇𝐱​log​p​\(𝐱\)=\(∂log⁡p∂x1​\(𝐱\),…,∂log⁡p∂xd​\(𝐱\)\)∈ℝd\.s\(\\mathbf\{x\}\)=\\nabla\_\{\\mathbf\{x\}\}\\log p\(\\mathbf\{x\}\)=\\left\(\\frac\{\\partial\\log p\}\{\\partial x\_\{1\}\}\(\\mathbf\{x\}\),\\ \\dots,\\ \\frac\{\\partial\\log p\}\{\\partial x\_\{d\}\}\(\\mathbf\{x\}\)\\right\)\\in\\mathbb\{R\}^\{d\}\.\(39\)It is a vector field : Two properties account for its role in generative modeling\.

First, the score is independent of the normalizing constant\. Ifp​\(𝐱\)=p~​\(𝐱\)/Zp\(\\mathbf\{x\}\)=\\tilde\{p\}\(\\mathbf\{x\}\)/ZwithZ=∫p~​\(𝐱\)​𝑑𝐱Z=\\int\\tilde\{p\}\(\\mathbf\{x\}\)\\,d\\mathbf\{x\}, then∇𝐱​log​p​\(𝐱\)=∇𝐱​log​p~​\(𝐱\)\\nabla\_\{\\mathbf\{x\}\}\\log p\(\\mathbf\{x\}\)=\\nabla\_\{\\mathbf\{x\}\}\\log\\tilde\{p\}\(\\mathbf\{x\}\), sinceZZdoes not depend on𝐱\\mathbf\{x\}\. The score is therefore accessible even when the density itself is intractable, as in energy\-based models wherep⁡\(𝐱\)∝e−E⁡\(𝐱\)p\(\\mathbf\{x\}\)\\propto e^\{\-E\(\\mathbf\{x\}\)\}\.

Second, the score suffices to generate samples via Langevin dynamics,

𝐱k\+1=𝐱k\+ε​∇𝐱​log⁡p⁡\(𝐱k\)\+2​ε​𝐳k,𝐳k∼𝒩⁡\(𝟎,𝐈\),\\mathbf\{x\}\_\{k\+1\}=\\mathbf\{x\}\_\{k\}\+\\varepsilon\\,\\nabla\_\{\\mathbf\{x\}\}\\log p\(\\mathbf\{x\}\_\{k\}\)\+\\sqrt\{2\\varepsilon\}\\,\\mathbf\{z\}\_\{k\},\\qquad\\mathbf\{z\}\_\{k\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\),\(40\)where the gradient term drives samples toward high\-density regions and the injected noise prevents collapse onto a single mode\. Equation \([40](https://arxiv.org/html/2609.25134#S8.E40)\) is the discrete counterpart of the overdamped Langevin stochastic differential equation, which constitutes the elementary building block of score\-based generative models\.

Within the diffusion framework, the relevant quantity is the score of the Gaussian transition kernel\. From the closed\-form marginalq⁡\(𝐱t∣𝐱0\)=𝒩⁡\(𝐱t,α¯t​𝐱0,\(1−α¯t\)​𝐈\)q\(\\mathbf\{x\}\_\{t\}\\mid\\mathbf\{x\}\_\{0\}\)=\\mathcal\{N\}\(\\mathbf\{x\}\_\{t\};\\sqrt\{\\bar\{\\alpha\}\_\{t\}\}\\,\\mathbf\{x\}\_\{0\},\(1\-\\bar\{\\alpha\}\_\{t\}\)\\mathbf\{I\}\), differentiation of the log\-density with respect to𝐱t\\mathbf\{x\}\_\{t\}, combined with the reparameterization𝐱t−α¯t​𝐱0=1−α¯t​ϵ\\mathbf\{x\}\_\{t\}\-\\sqrt\{\\bar\{\\alpha\}\_\{t\}\}\\,\\mathbf\{x\}\_\{0\}=\\sqrt\{1\-\\bar\{\\alpha\}\_\{t\}\}\\,\\bm\{\\epsilon\}, yields

∇𝐱t​log​q​\(𝐱t∣𝐱0\)=−ϵ1−α¯t\.\\nabla\_\{\\mathbf\{x\}\_\{t\}\}\\log q\(\\mathbf\{x\}\_\{t\}\\mid\\mathbf\{x\}\_\{0\}\)=\-\\frac\{\\bm\{\\epsilon\}\}\{\\sqrt\{1\-\\bar\{\\alpha\}\_\{t\}\}\}\.\(41\)Thus, up to the scalar factor−1/1−α¯t\-1/\\sqrt\{1\-\\bar\{\\alpha\}\_\{t\}\}, the score of the conditional Gaussian coincides with the noiseϵ\\bm\{\\epsilon\}that was added\. Estimating the noise is therefore equivalent to estimating the score\. When the networkϵθ​\(𝐱t,t\)\\bm\{\\epsilon\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t\)is trained to predictϵ\\bm\{\\epsilon\}, it implicitly learnssθ\(𝐱t,t\)≈−ϵθ\(𝐱t,t\)/1−α¯ts\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t\)\\approx\-\\bm\{\\epsilon\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t\)/\\sqrt\{1\-\\bar\{\\alpha\}\_\{t\}\}, where the marginal score∇𝐱t​log​q​\(𝐱t\)\\nabla\_\{\\mathbf\{x\}\_\{t\}\}\\log q\(\\mathbf\{x\}\_\{t\}\)is recovered from the conditional one through the denoising score matching identity, also known as Tweedie’s formula,

∇𝐱t​log​q​\(𝐱t\)=𝔼q⁡\(𝐱0∣𝐱t\)​\[∇𝐱t​log​q​\(𝐱t∣𝐱0\)\]\.\\nabla\_\{\\mathbf\{x\}\_\{t\}\}\\log q\(\\mathbf\{x\}\_\{t\}\)=\\mathbb\{E\}\_\{q\(\\mathbf\{x\}\_\{0\}\\mid\\mathbf\{x\}\_\{t\}\)\}\\big\[\\nabla\_\{\\mathbf\{x\}\_\{t\}\}\\log q\(\\mathbf\{x\}\_\{t\}\\mid\\mathbf\{x\}\_\{0\}\)\\big\]\.\(42\)This identity justifies the substitution of the intractable marginal score by a regression on the tractable conditional score and explains the explicit appearance of the score in the reverse SDE in the stochastic differential equation approach \(see below\), whose drift isf⁡\(𝐗t,t\)−g​\(t\)2​∇𝐱​log⁡pt​\(𝐗t\)f\(\\mathbf\{X\}\_\{t\},t\)\-g\(t\)^\{2\}\\nabla\_\{\\mathbf\{x\}\}\\log p\_\{t\}\(\\mathbf\{X\}\_\{t\}\)\.

#### 8\.3\.2The Reverse Process and the Training Objective

Since the true reverse kernelq⁡\(𝐱t−1∣𝐱t\)q\(\\mathbf\{x\}\_\{t\-1\}\\mid\\mathbf\{x\}\_\{t\}\)depends on the unknown data distribution, we approximate it with a learned Gaussian:

pθ​\(𝐱t−1∣𝐱t\)=𝒩⁡\(𝐱t−1,𝝁θ​\(𝐱t,t\),σt2​𝐈\),p\_\{\\theta\}\(\\mathbf\{x\}\_\{t\-1\}\\mid\\mathbf\{x\}\_\{t\}\)=\\mathcal\{N\}\\\!\\left\(\\mathbf\{x\}\_\{t\-1\};\\ \\bm\{\\mu\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t\),\\ \\sigma\_\{t\}^\{2\}\\mathbf\{I\}\\right\),\(43\)where𝝁θ​\(𝐱t,t\)\\bm\{\\mu\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t\)is the output of a network with weightsθ\\theta, andσt2\\sigma\_\{t\}^\{2\}is a fixed or learned variance\. Training maximizes the variational lower bound onlog⁡pθ​\(𝐱0\)\\log p\_\{\\theta\}\(\\mathbf\{x\}\_\{0\}\), which after simplification\[[29](https://arxiv.org/html/2609.25134#bib.bib29)\]reduces to a mean\-squared error on the noise:

ℒsimple​\(θ\)=𝔼t,𝐱0,ϵ​\[‖ϵ−ϵθ​\(α¯t​𝐱0\+1−α¯t​ϵ,t\)‖2\],\\mathcal\{L\}\_\{\\text\{simple\}\}\(\\theta\)=\\mathbb\{E\}\_\{t,\\mathbf\{x\}\_\{0\},\\bm\{\\epsilon\}\}\\left\[\\left\\\|\\bm\{\\epsilon\}\-\\bm\{\\epsilon\}\_\{\\theta\}\\\!\\left\(\\sqrt\{\\bar\{\\alpha\}\_\{t\}\}\\,\\mathbf\{x\}\_\{0\}\+\\sqrt\{1\-\\bar\{\\alpha\}\_\{t\}\}\\,\\bm\{\\epsilon\},\\ t\\right\)\\right\\\|^\{2\}\\right\],\(44\)whereϵ∼𝒩⁡\(𝟎,𝐈\)\\bm\{\\epsilon\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)is the noise used to form𝐱t\\mathbf\{x\}\_\{t\}, andϵθ\\bm\{\\epsilon\}\_\{\\theta\}is the network’s prediction of that noise\. Intuitively, the network learns to look at a noisy image𝐱t\\mathbf\{x\}\_\{t\}and guess which noise vector was added—equivalently, to estimate the score∇𝐱t​log​q​\(𝐱t\)\\nabla\_\{\\mathbf\{x\}\_\{t\}\}\\log q\(\\mathbf\{x\}\_\{t\}\)up to a scaling factor ; we refer to\[[29](https://arxiv.org/html/2609.25134#bib.bib29)\]for the calculation details\.

### 8\.4From Unconditional Generation to Text\-to\-Image

#### 8\.4\.1Unconditional generation

Start with the unconditional case\. We train a networkϵθ​\(𝐱t,t\)\\bm\{\\epsilon\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t\)to predict the noise added at steptt\. Once trained, generation proceeds as follows:

1. 1\.Sample𝐱T∼𝒩⁡\(𝟎,𝐈\)\\mathbf\{x\}\_\{T\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)\. This is add\-dimensional vector of pure Gaussian noise—for an image, think of a512×512×3512\\times 512\\times 3tensor of independent Gaussian pixels\. Visually, it looks like television static\.
2. 2\.Fort=T,T−1,…,1t=T,T\-1,\\dots,1, compute the predicted noiseϵθ​\(𝐱t,t\)\\bm\{\\epsilon\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t\)and use it to form an estimate of𝐱t−1\\mathbf\{x\}\_\{t\-1\}according to the reverse Gaussian kernel\.
3. 3\.AfterTTsteps, we get the output𝐱0\\mathbf\{x\}\_\{0\}\.

The fact is that𝐱0\\mathbf\{x\}\_\{0\}is a*new sample*from \(an approximation of\)pdatap\_\{\\text\{data\}\}\. If the model was trained on a dataset of cat photos,𝐱0\\mathbf\{x\}\_\{0\}will look like a cat that never existed\. If it was trained on handwritten digits,𝐱0\\mathbf\{x\}\_\{0\}will look like a digit\. The Gaussian noise is progressively sculpted into a structured image\.

But this unconditional process gives us no control: we cannot ask for a cat specifically, or a cat wearing a hat\. That control comes fromconditioning\.

#### 8\.4\.2Conditioning on text

In text\-to\-image generation, we want to sample from the conditional distributionp⁡\(𝐱∣𝐜\)p\(\\mathbf\{x\}\\mid\\mathbf\{c\}\), where𝐜\\mathbf\{c\}is a text prompt such as “a photo of futuristic cars” The prompt is first encoded into a vector \(or a sequence of vectors\)𝐜\\mathbf\{c\}by a text encoder—for example CLIP’s text encoder or a T5 encoder\. The denoising network then takes both the noisy image𝐱t\\mathbf\{x\}\_\{t\}and the text embedding𝐜\\mathbf\{c\}as inputs:

ϵθ​\(𝐱t,t,𝐜\)\.\\bm\{\\epsilon\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t,\\mathbf\{c\}\)\.Training uses pairs\(𝐱0,𝐜\)\(\\mathbf\{x\}\_\{0\},\\mathbf\{c\}\)of images and their captions\. At each step, noiseϵ\\bm\{\\epsilon\}is added to𝐱0\\mathbf\{x\}\_\{0\}to form𝐱t\\mathbf\{x\}\_\{t\}, and the network is trained to predictϵ\\bm\{\\epsilon\}given𝐱t\\mathbf\{x\}\_\{t\},tt, and𝐜\\mathbf\{c\}\. The loss is the same mean\-squared error as before, but now the network sees the text as an additional input\.

At generation time, the procedure is identical to the unconditional case, except that at every denoising step the network is told the prompt𝐜\\mathbf\{c\}\. Concretely:

1. 1\.Encode the prompt “a photo of futuristic cars” into𝐜\\mathbf\{c\}\.
2. 2\.Sample𝐱T∼𝒩⁡\(𝟎,𝐈\)\\mathbf\{x\}\_\{T\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)\.
3. 3\.Fort=T,…,1t=T,\\dots,1, predictϵθ​\(𝐱t,t,𝐜\)\\bm\{\\epsilon\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t,\\mathbf\{c\}\)and update𝐱t→𝐱t−1\\mathbf\{x\}\_\{t\}\\to\\mathbf\{x\}\_\{t\-1\}\.
4. 4\.Output𝐱0\\mathbf\{x\}\_\{0\}, which is now a sample fromp⁡\(𝐱∣𝐜\)p\(\\mathbf\{x\}\\mid\\mathbf\{c\}\)\.

The same noise seed combined with different prompts yields different images; the same prompt with different noise seeds yields different images\. This is the mechanism behindStable Diffusion, DALL⋅\\cdotE 2, Imagen, and Midjourney\.

#### 8\.4\.3Why this works: the score interpretation

Recall that predicting the noiseϵ\\bm\{\\epsilon\}is equivalent \(up to a scaling factor\) to estimating the score∇𝐱t​log​pt​\(𝐱t∣𝐜\)\\nabla\_\{\\mathbf\{x\}\_\{t\}\}\\log p\_\{t\}\(\\mathbf\{x\}\_\{t\}\\mid\\mathbf\{c\}\)\. In the conditional case, the score points in the direction in which the noisy image should be modified to become more likely under the text\-conditioned distribution\. Each denoising step is a small move along this score, plus a bit of injected noise to keep the process stochastic\. Over many steps, these small moves accumulate into a coherent image that matches the prompt\.

#### 8\.4\.4Classifier\-free guidance

A crucial practical ingredient isclassifier\-free guidance\[[31](https://arxiv.org/html/2609.25134#bib.bib31)\]\. At each step, the network is evaluated twice: once with the prompt𝐜\\mathbf\{c\}and once with an empty prompt∅\\varnothing\. The two predictions are combined as

ϵ~θ​\(𝐱t,t,𝐜\)=ϵθ​\(𝐱t,t,∅\)\+s⁡\[ϵθ​\(𝐱t,t,𝐜\)−ϵθ​\(𝐱t,t,∅\)\],\\tilde\{\\bm\{\\epsilon\}\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t,\\mathbf\{c\}\)=\\bm\{\\epsilon\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t,\\varnothing\)\+s\\left\[\\bm\{\\epsilon\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t,\\mathbf\{c\}\)\-\\bm\{\\epsilon\}\_\{\\theta\}\(\\mathbf\{x\}\_\{t\},t,\\varnothing\)\\right\],\(45\)wheres\>1s\>1is the guidance scale\. This sharpens the alignment between the generated image and the prompt, at the cost of some diversity\. Without it, text\-to\-image models tend to produce images that are only loosely related to the prompt\.

#### 8\.4\.5Latent diffusion

In practice, the denoising is often not performed in pixel space\. Stable Diffusion\[[33](https://arxiv.org/html/2609.25134#bib.bib33)\]uses alatent diffusionapproach: a variational autoencoder first compresses the image into a lower\-dimensional latent representation𝐳0∈ℝh×w×c\\mathbf\{z\}\_\{0\}\\in\\mathbb\{R\}^\{h\\times w\\times c\}withh,wh,wmuch smaller than the original image dimensions\. The diffusion process is then run in this latent space, and the final latent𝐳0\\mathbf\{z\}\_\{0\}is decoded back to pixel space by the VAE decoder\. This makes training and sampling far more computationally efficient while preserving perceptual quality\.

#### 8\.4\.6Summary

The reverse diffusion process*starts*from noise and*transforms*it into data\. In text\-to\-image generation, the transformation is guided at every step by a text embedding, so the final image is a sample from the conditional distributionp⁡\(𝐱∣𝐜\)p\(\\mathbf\{x\}\\mid\\mathbf\{c\}\)\. The same mathematical machinery—Gaussian forward transitions, learned reverse transitions, and score estimation—underlies unconditional image generation, text\-to\-image, image editing, super\-resolution, and many other tasks\.

### 8\.5Continuous\-Time Limit: Stochastic Differential Equations

The discrete formulation above becomes difficult to manipulate whenTTis large\. Taking the limitT→∞T\\to\\inftywithβt→0\\beta\_\{t\}\\to 0yields a continuous\-time description in terms of stochastic differential equations \(Song et al\.\[[30](https://arxiv.org/html/2609.25134#bib.bib30)\]\)\.

Lett∈\[0,T\]t\\in\[0,T\]be a continuous time variable, let𝐗t∈ℝd\\mathbf\{X\}\_\{t\}\\in\\mathbb\{R\}^\{d\}be a continuous\-time stochastic process,f⁡\(𝐗t,t\)∈ℝdf\(\\mathbf\{X\}\_\{t\},t\)\\in\\mathbb\{R\}^\{d\}a drift,g⁡\(t\)∈ℝg\(t\)\\in\\mathbb\{R\}a scalar diffusion coefficient, and𝐁t∈ℝd\\mathbf\{B\}\_\{t\}\\in\\mathbb\{R\}^\{d\}a standard Brownian motion\. The forward SDE is

d​𝐗t=f⁡\(𝐗t,t\)​d​t\+g⁡\(t\)​d​𝐁t,𝐗0∼pdata\.d\\mathbf\{X\}\_\{t\}=f\(\\mathbf\{X\}\_\{t\},t\)\\,dt\+g\(t\)\\,d\\mathbf\{B\}\_\{t\},\\qquad\\mathbf\{X\}\_\{0\}\\sim p\_\{\\text\{data\}\}\.\(46\)The reverse\-time SDE, obtained from Anderson’s theorem\[[32](https://arxiv.org/html/2609.25134#bib.bib32)\], is, with𝐗←t\\overleftarrow\{\\mathbf\{X\}\}\_\{t\}denoting the reverse process,

d​𝐗←t=\[f⁡\(𝐗←t,t\)−g​\(t\)2​∇𝐱​log⁡pt​\(𝐗←t\)\]​d​t\+g⁡\(t\)​d​𝐁¯t,𝐗←T∼𝒩⁡\(𝟎,𝐈\),d\\overleftarrow\{\\mathbf\{X\}\}\_\{t\}=\\left\[f\(\\overleftarrow\{\\mathbf\{X\}\}\_\{t\},t\)\-g\(t\)^\{2\}\\nabla\_\{\\mathbf\{x\}\}\\log p\_\{t\}\(\\overleftarrow\{\\mathbf\{X\}\}\_\{t\}\)\\right\]dt\+g\(t\)\\,d\\bar\{\\mathbf\{B\}\}\_\{t\},\\qquad\\overleftarrow\{\\mathbf\{X\}\}\_\{T\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\),\(47\)whereptp\_\{t\}is the marginal density of𝐗t\\mathbf\{X\}\_\{t\}and∇𝐱​log​pt\\nabla\_\{\\mathbf\{x\}\}\\log p\_\{t\}is its score\. The reverse SDE has a deterministic counterpart, the*probability flow ODE*: Using the Itô formula, Song et al\. showed that the solution𝐘t\\mathbf\{Y\}\_\{t\}of the \(deterministic\) ODE:

d​𝐘td​t=f⁡\(𝐘t,t\)−12​g​\(t\)2​∇𝐱​log⁡pt​\(𝐘t\),𝐘T∼𝒩⁡\(𝟎,𝐈\)\.\\frac\{d\\mathbf\{Y\}\_\{t\}\}\{dt\}=f\(\\mathbf\{Y\}\_\{t\},t\)\-\\frac\{1\}\{2\}g\(t\)^\{2\}\\nabla\_\{\\mathbf\{x\}\}\\log p\_\{t\}\(\\mathbf\{Y\}\_\{t\}\),\\qquad\\mathbf\{Y\}\_\{T\}\\sim\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)\.\(48\)has the same marginal law as that of𝐗t\\mathbf\{X\}\_\{t\}at every timett: the two share the same marginals, but not the same paths\. It is understood that this ODE is also solved backward with the same conditionYT=XT=𝒩⁡\(𝟎,𝐈\)Y\_\{T\}=X\_\{T\}=\\mathcal\{N\}\(\\mathbf\{0\},\\mathbf\{I\}\)\. Integrating \([48](https://arxiv.org/html/2609.25134#S8.E48)\) backwards fromt=Tt=Ttot=0t=0produces samples frompdatap\_\{\\text\{data\}\}, and because the map is deterministic and invertible, it also yields exact log\-likelihoods via the change\-of\-variables formula\. We omit the details and refer to\[[30](https://arxiv.org/html/2609.25134#bib.bib30)\],\[[34](https://arxiv.org/html/2609.25134#bib.bib34)\]\. The discrete analogue of this ODE is the DDIM sampler\[[35](https://arxiv.org/html/2609.25134#bib.bib35)\]\.

### 8\.6Current Usage and comments

Diffusion models have achieved high quality results in image generation \(Stable Diffusion, DALL⋅\\cdotE 2, Imagen\), image editing, super\-resolution, medical imaging, audio and speech synthesis, video generation, reinforcement learning, and computational biology \(protein structure and molecule design\)\. Their appeal lies in training stability, mode coverage, and the principled probabilistic framework that connects them to score matching, Langevin dynamics, and continuous normalizing flows\. A recent survey is given by Ahsan et al\.\[[36](https://arxiv.org/html/2609.25134#bib.bib36)\]\.

The power of diffusion models comes from the probabilistic features of the Gaussian transitions: they are analytically tractable, compose into closed\-form marginals, and admit a Gaussian reverse kernel that a neural network can learn\. The discrete construction of DDPM is the practical realization of this insight; the continuous\-time SDE formulation is its mathematically natural limit, unifying discrete and continuous views and linking generative modeling to the classical theory of stochastic processes\. In text\-to\-image systems, the same machinery, augmented with a text encoder and classifier\-free guidance, produces images conditioned on natural\-language prompts—one of the most visible successes of generative AI today\.

## References

- \[1\]Bengio, Y\., Ducharme, R\., Vincent, P\. and Jauvin, C\., 2003\. A Neural Probabilistic Language Model\.Journal of Machine Learning Research, 3, pp\. 1137–1155\.
- \[2\]Mikolov, T\., Karafiát, M\., Burget, L\., Černocký, J\. and Khudanpur, S\., 2010\. Recurrent Neural Network Based Language Model\.Proceedings of Interspeech 2010, pp\. 1045–1048\.
- \[3\]Cho, K\., van Merriënboer, B\., Gulcehre, C\., Bahdanau, D\., Bougares, F\., Schwenk, H\. and Bengio, Y\., 2014\. Learning Phrase Representations Using RNN Encoder–Decoder for Statistical Machine Translation\.Proceedings of EMNLP 2014, pp\. 1724–1734\.
- \[4\]Sutskever, I\., Vinyals, O\. and Le, Q\. V\., 2014\. Sequence to Sequence Learning with Neural Networks\.Advances in Neural Information Processing Systems \(NeurIPS 2014\)\.
- \[5\]Bahdanau, D\., Cho, K\. and Bengio, Y\., 2014\. Neural Machine Translation by Jointly Learning to Align and Translate\.arXiv:1409\.0473;Proceedings of ICLR 2015\.
- \[6\]Vaswani, A\., Shazeer, N\., Parmar, N\., Uszkoreit, J\., Jones, L\., Gomez, A\. N\., Kaiser, L\. and Polosukhin, I\., 2017\. Attention Is All You Need\.Advances in Neural Information Processing Systems \(NeurIPS 2017\), 30\.
- \[7\]Rush, S\., Huang, A\., Subramanian, S\., Sum, J\., Almubarak, K\. and Biderman, S\.The Annotated Transformer\.[https://nlp\.seas\.harvard\.edu/annotated\-transformer/](https://nlp.seas.harvard.edu/annotated-transformer/)
- \[8\]Alammar, J\.The Illustrated Transformer\.[https://jalammar\.github\.io/illustrated\-transformer/](https://jalammar.github.io/illustrated-transformer/)
- \[9\]Alammar, J\. and Grootendorst, M\., 2024\.Hands\-On Large Language Models: Language Understanding and Generation\. O’Reilly Media\.
- \[10\]Brownlee, J\., Cristina, S\. and Saeed, M\., 2022\.Building Transformer Models with Attention: Implementing a Neural Machine Translator from Scratch in Keras\. Machine Learning Mastery\.
- \[11\]Aboulalaa A\.,Generative AI foundations: Attention Mechanisms, Transformers, LLMs and All That\. To appear\.
- \[12\]Radford, A\., Wu, J\., Child, R\., Luan, D\., Amodei, D\. and Sutskever, I\., 2019\. Language Models are Unsupervised Multitask Learners\. OpenAI Technical Report\.
- \[13\]Brown, T\. B\., Mann, B\., Ryder, N\., Subbiah, M\., Kaplan, J\., Dhariwal, P\., Neelakantan, A\., Shyam, P\., Sastry, G\., Askell, A\. et al\., 2020\. Language Models are Few\-Shot Learners\.Advances in Neural Information Processing Systems \(NeurIPS 2020\)\.
- \[14\]Ouyang, L\., Wu, J\., Jiang, X\., Almeida, D\., Wainwright, C\., Mishkin, P\., Zhang, C\., Agarwal, S\., Slama, K\., Ray, A\. et al\., 2022\. Training Language Models to Follow Instructions with Human Feedback\.Advances in Neural Information Processing Systems \(NeurIPS 2022\)\.
- \[15\]Cover, T\. M\. and Thomas, J\. A\., 2006\.Elements of Information Theory\. 2nd ed\., Wiley\. Entropy rate, AEP, typical sets\.
- \[16\]Du, L\., Torroba Hennigen, L\., Pimentel, T\., Meister, C\., Eisner, J\. and Cotterell, R\., 2023\. A Measure\-Theoretic Characterization of Tight Language Models\.Proceedings of ACL 2023, pp\. 9744–9770\.
- \[17\]Gneiting, T\. and Raftery, A\. E\., 2007\. Strictly Proper Scoring Rules, Prediction, and Estimation\.Journal of the American Statistical Association, 102\(477\), pp\. 359–378\.
- \[18\]White, H\., 1982\. Maximum Likelihood Estimation of Misspecified Models\.Econometrica, 50\(1\), pp\. 1–25\.
- \[19\]Bishop, C\. M\., 2006\.Pattern Recognition and Machine Learning\. Springer\. See §10\.1\.2 for the mass\-covering vs\. mode\-seeking behavior of the two KL directions\.
- \[20\]Minka, T\., 2005\. Divergence Measures and Message Passing\. Microsoft Research Technical Report MSR\-TR\-2005\-173\.
- \[21\]Robbins, H\. and Monro, S\., 1951\. A Stochastic Approximation Method\.The Annals of Mathematical Statistics, 22\(3\), pp\. 400–407\.
- \[22\]Kingma, D\. P\. and Ba, J\., 2015\. Adam: A Method for Stochastic Optimization\.Proceedings of ICLR 2015\.
- \[23\]Loshchilov, I\. and Hutter, F\., 2019\. Decoupled Weight Decay Regularization\.Proceedings of ICLR 2019\.
- \[24\]Holtzman, A\., Buys, J\., Du, L\., Forbes, M\. and Choi, Y\., 2020\. The Curious Case of Neural Text Degeneration\.Proceedings of ICLR 2020\. Nucleus sampling and the likelihood trap\.
- \[25\]Meister, C\., Pimentel, T\., Wiher, G\. and Cotterell, R\., 2023\. Locally Typical Sampling\.Transactions of the ACL, 11, pp\. 102–121\.
- \[26\]Eikema, B\. and Aziz, W\., 2020\. Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation\.Proceedings of COLING 2020, pp\. 4506–4520\.
- \[27\]Kalai, A\. T\. and Vempala, S\. S\., 2024\. Calibrated Language Models Must Hallucinate\.Proceedings of the 56th Annual ACM Symposium on Theory of Computing \(STOC 2024\), pp\. 160–171\.
- \[28\]J\. Sohl\-Dickstein, E\. A\. Weiss, N\. Maheswaranathan, and S\. Ganguli,Deep Unsupervised Learning using Nonequilibrium Thermodynamics,*ICML*, 2015\.
- \[29\]J\. Ho, A\. Jain, and P\. Abbeel,Denoising Diffusion Probabilistic Models,*NeurIPS*, 2020\.
- \[30\]Y\. Song, J\. Sohl\-Dickstein, D\. P\. Kingma, A\. Kumar, S\. Ermon, and B\. Poole,Score\-Based Generative Modeling through Stochastic Differential Equations,*ICLR*, 2021\.
- \[31\]J\. Ho and T\. Salimans,Classifier\-Free Diffusion Guidance,*arXiv:2207\.12598*, 2022\.
- \[32\]B\. D\. O\. Anderson,Reverse\-time diffusion equation models,*Stochastic Processes and their Applications*, 12\(3\):313–326, 1982\.
- \[33\]R\. Rombach, A\. Blattmann, D\. Lorenz, P\. Esser, and B\. Ommer,High\-Resolution Image Synthesis with Latent Diffusion Models,*CVPR*, 2022\.
- \[34\]W\. Tang and H\. Zhao,Score\-based Diffusion Models via Stochastic Differential Equations—a Technical Tutorial,*arXiv:2402\.07487*, 2024\.
- \[35\]J\. Song, C\. Meng, and S\. Ermon,Denoising Diffusion Implicit Models,*ICLR*, 2021\.
- \[36\]M\. M\. Ahsan, S\. Raman, and Y\. Liu,A comprehensive survey of diffusion models and their applications,*Applied Soft Computing*, 181:113470, 2025\.