神经期望最大化与元模型

# Neural Expectation-Maximization and meta-models
Source: [https://arxiv.org/html/2610.12741](https://arxiv.org/html/2610.12741)
###### Abstract

We interpret Transformer networks as learned approximations to a hybrid probabilistic\-inference procedure that combines max\-product Loopy Belief Propagation \(LBP\) with Expectation–Maximization \(EM\)\. In this view, a Transformer performs inference in partially observed Markov Random Fields \(MRFs\) whose structure and parameters are not given a priori but are inferred from data\. We show that self\-attention implements a differentiable approximation to max\-product LBP on an input\-conditioned, dynamically structured MRF, with attention weights encoding learned structural affinities\. We further establish an EM correspondence at the level of depth: each layer updates latent representations in an E\-step\-like forward pass, while the transitionθ\(l\)→θ\(l\+1\)\\theta^\{\(l\)\}\\to\\theta^\{\(l\+1\)\}across layers acts as an implicit M\-step\. Building on this interpretation, we introduce*meta\-models*, a design paradigm in which inputs generate instance\-specific parameters, unifying mechanisms such as self\-attention, mixture\-of\-experts, and gated units within a single framework\. Empirical results support the proposed perspective\.

*K*eywordsSelf\-attention⋅\\cdotTransformer⋅\\cdotLoopy Belief Propagation⋅\\cdotExpectation\-Maximization⋅\\cdotMeta\-Model

## 1Introduction

Figure 1:Expectation\-Maximization interpretation of DNNs\. A layer corresponds to an iteration of the EM\. The forward pass corresponds to the E\-step\. The transition from parameterθ′\\theta^\{\\prime\}to the next layerθ\\thetaimplicitly corresponds to the M\-step\.Figure 2:Illustration of a meta\-model\. The input vectorxxfirst interacts with the 3D meta\-parameterΘ\\Thetaand outputs the 2D parameterθ\\theta\. Then,xxinteracts withθ\\thetaand produces the outputzz\. Meta\-model is what makes a model capable of in\-context learning\. Self\-attention, MoE, GLUs, etc\. are members of meta\-models\.
Transformers have become the dominant architecture for modern sequence modeling, language understanding, and large\-scale generative modeling since the introduction of self\-attention\[[26](https://arxiv.org/html/2610.12741#bib.bib1)\]\. Despite their empirical success, the internal computation performed by a Transformer remains difficult to characterize in algorithmic terms\. A growing body of work has therefore shifted from treating Transformers as black\-box function approximators toward explaining what their layers, attention heads, and residual streams compute\[[8](https://arxiv.org/html/2610.12741#bib.bib2),[22](https://arxiv.org/html/2610.12741#bib.bib3)\]\. This paper contributes to that line of work by asking whether Transformer computation can be understood as a familiar inference procedure rather than as a collection of isolated architectural mechanisms\.

Existing explanations illuminate important aspects of Transformer behavior, but they remain partial\. Studies of in\-context learning show that Transformers can implement gradient\-descent\-like or Bayesian updates; and message\-passing views relate attention to graph computation\[[1](https://arxiv.org/html/2610.12741#bib.bib4),[5](https://arxiv.org/html/2610.12741#bib.bib5),[28](https://arxiv.org/html/2610.12741#bib.bib6),[27](https://arxiv.org/html/2610.12741#bib.bib7)\]\. However, these perspectives do not fully account for how self\-attention, depth, latent\-variable refinement, graph\-structure selection, and layerwise parameter changes fit into a single probabilistic picture\.

We propose such an interpretation by viewing Transformers as neuralized Expectation\-Maximization\. Under this view, the underlying model is a partially observed Markov Random Field \(MRF\) with both unknown structure and parameters\. Self\-attention approximates Loopy Belief Propagation \(LBP\) over a dynamically inferred graph structure, where the attention between two tokens acts as the soft connection strength within such MRF and the value vector acts as the maximizer of the aggregated message \(aka\. belief\)\. Successive layers play the role of Expectation\-Maximization \(EM\) iterations\[[7](https://arxiv.org/html/2610.12741#bib.bib8),[29](https://arxiv.org/html/2610.12741#bib.bib9),[16](https://arxiv.org/html/2610.12741#bib.bib10)\]for parameter learning\. Specifically, a layer of a Transformer corresponds to an iteration of the EM\. The forward pass of a layer corresponds to the E\-step, where the latent variables are updated based on the last iteration\. The change from the parametersθ\(l\)\\theta^\{\(l\)\}of layerlltoθ\(l\+1\)\\theta^\{\(l\+1\)\}of the next layer implicitly corresponds to the M\-step, where the parameters are updated\.

This paper makes the following contributions:

- •We derive a correspondence between self\-attention and approximate LBP on a dynamically structured MRF, where attention weights act as learned structural affinities and values approximate individual message maximizers\.
- •We interpret the depth of a Transformer as an EM\-like iterative procedure in which the forward pass updates latent variables and the transition between layer parameters corresponds to parameter refinement\.
- •We use this EM perspective to motivate meta\-models, a design paradigm in which the parametersθ\\thetaof a forward pass are generated from the input by meta\-parametersΘ\\Theta\.
- •We show that self\-attention, Mixture\-of\-Experts layers, gated mechanisms, LSTM, and selective state\-space models can be viewed as members or close relatives of this meta\-model principle\.

## 2Related Work

Our work connects three lines of research: interpretations of transformers as probabilistic inference, interpretations of transformer layers as expectation–maximization \(EM\) steps, the view of attention as a meta\-model that generates parameters from the input\.

#### Transformers and expectation–maximization\.

A line of work interprets transformer depth as iterative probabilistic fitting\.\[[4](https://arxiv.org/html/2610.12741#bib.bib30)\]prove that transformers can approximate the EM algorithm, together with spectral \(tensor power\) iterations, on Gaussian mixtures\. The first is an interpretation tied to a specific modeling choice, and the second is an expressivity result for a particular model class\. The distinction is that our analysis frames the algorithm as probabilistic latent\-variable inference: the E\-step updates latent variableszz, while changes in layer parametersθ\(l\)\\theta^\{\(l\)\}are interpreted as an implicit M\-step\.

#### Transformers and belief propagation\.

Message passing on graphs has long been a template for neural computation\. Neural message passing\[[10](https://arxiv.org/html/2610.12741#bib.bib27)\]unified many graph architectures under one framework, and Factor Graph Neural Networks\[[32](https://arxiv.org/html/2610.12741#bib.bib26)\]extend this to learned message passing on factor graphs\. Closer to attention,\[[33](https://arxiv.org/html/2610.12741#bib.bib25)\]introduce graphical models with attention to capture context\-specific independence, applying them to perceptual grouping\. Several recent papers connect transformers directly to BP\. Other work uses BP as a design principle:\[[18](https://arxiv.org/html/2610.12741#bib.bib28)\]refine self\-attention with a one\-step BP update to mitigate attention localization in small\-scale models, and\[[17](https://arxiv.org/html/2610.12741#bib.bib29)\]propose a differential\-attention message\-passing transformer\. These papers either establish an equivalence under specific architectural assumptions or use BP to improve the architecture\. The connection between attention and graph computation has also been studied through graph neural networks and graph attention mechanisms\[[27](https://arxiv.org/html/2610.12741#bib.bib7)\]\. These methods treat attention coefficients as learned edge weights and thereby resemble message passing on graphs\.

The present paper differs in its use of BP as the organizing principle: the messagesmi,j\(t\)​\(xj\)m\_\{i,j\}^\{\(t\)\}\(x\_\{j\}\), beliefsβi\(t\)​\(xi\)\\beta\_\{i\}^\{\(t\)\}\(x\_\{i\}\), and structural indicatorsσj,i\\sigma\_\{j,i\}are mapped directly to the components of self\-attention\.

#### Dynamic Parameters and Meta\-Model Architectures

Many successful neural architectures can be interpreted as using input\-dependent computation\. Hypernetworks\[[12](https://arxiv.org/html/2610.12741#bib.bib31)\]established the principle that one network \(the meta\-parameters\) can generate the weights of another\. Mixture\-of\-Experts models route tokens to different experts using learned gates\[[25](https://arxiv.org/html/2610.12741#bib.bib11)\], while gated convolutional and recurrent models modulate information flow through input\-conditioned gates\[[6](https://arxiv.org/html/2610.12741#bib.bib12),[14](https://arxiv.org/html/2610.12741#bib.bib13)\]\. More recently, selective state\-space models such as Mamba use input\-dependent state\-space parameters to obtain efficient sequence modeling with dynamic behavior\[[11](https://arxiv.org/html/2610.12741#bib.bib14)\]\.

These architectures are usually studied as separate design families: routing models, gated models, recurrent models, or state\-space models\. The meta\-model view proposed here treats them as instances of a broader principle suggested by EM\.

## 3Method

LBP and EM are two classic algorithms specialized at doing inference in different situations\. LBP make inferences on partially observed models with known structure and parameters, while EM on partially observed models with known structure but unknown parameters\. However, with real\-life data, not only are they often partially observed but also the underlying structure and parameters are both unknown\. Applying either LPB or EM to real\-life data will not work effectively\. We show that the architecture of Transformers combines LPB and EM to make inferences on such complex data\. This combination is what makes Transformers effective\.

### 3\.1The Underlying Model

Figure 3:The PGM that underlies a encoder Transformer\. 1\) On the left is the original BN: tokensXXare generated by a sequence of latent variablesZZ\.ZZare further connected to an unknown hierarchy of intermediate layersSS, which further traces back to a root causeYY\. 2\) In the middle is the model after moralization: we abstract awaySSby moralizing the BN overSSinto a chain graph\. 3\) On the right is the model after variable elimination: we perform variable elimination on observed tokensXX\. The final model is a fully connected MRF onZZandYY\.Let us consider a encoder Transformer and examine the probabilistic graphic model \(PGM\) that underlies it\. Our assumption is that the underlying PGM is a Bayesian Network \(BN\) as shown on the left of Fig\.[3](https://arxiv.org/html/2610.12741#S3.F3)\. The sequence of observed tokensX=\(X1,…,Xn\)X=\(X\_\{1\},\\dots,X\_\{n\}\)is generated by a sequence of latent variablesZ=\(Z1,…,Zn\)Z=\(Z\_\{1\},\\dots,Z\_\{n\}\)\.ZZis further connected to an unknown hierarchical of intermediate layersSS, which finally trace back to a latent root causeYY\. The full factorization is

p⁡\(x,y,z,s\)=p⁡\(y\)​p​\(s\|y\)​∏ip⁡\(xi\|zi\)​p​\(zi\|s\)\.p\(x,y,z,s\)=p\(y\)p\(s\|y\)\\prod\_\{i\}p\(x\_\{i\}\|z\_\{i\}\)p\(z\_\{i\}\|s\)\.\(1\)The structure ofSSis unknown and dynamic, as it can vary from instance to instance\. However, we can abstract it away by moralizing the BN overSSinto a chain graph, as shown in the middle of Fig\.[3](https://arxiv.org/html/2610.12741#S3.F3)\. As a last step, we can perform Variable Elimination on observed leaf variablesXX, as shown to the right of Fig\.[3](https://arxiv.org/html/2610.12741#S3.F3)\. In this way, we are able to bake the information of an observationXiX\_\{i\}into its latent variableZiZ\_\{i\}\. This can be seen as imposing a pseudo prior onZiZ\_\{i\}:

ϕi​\(zi\)=p⁡\(xi\|zi\),\\phi\_\{i\}\(z\_\{i\}\)=p\(x\_\{i\}\|z\_\{i\}\),\(2\)wherexix\_\{i\}is observed\. As we shall see later, the embeddingeie\_\{i\}of a tokenxix\_\{i\}can be seen as the maximizer of this pseudo\-prior:

ei=arg⁡max​ϕi​\(zi\)\.e\_\{i\}=\\arg\\max\\phi\_\{i\}\(z\_\{i\}\)\.\(3\)After the above operations, we are left with a fully connected MRF onZZandYY\. We can unify our notation and treatYYas a special latent variable

Zn\+1≐Y\.Z\_\{n\+1\}\\doteq Y\.\(4\)Let the set of resultant factors on this MRF be\{Φi∣i=1,…,m\}\\\{\\Phi\_\{i\}\\mid i=1,\\dots,m\\\}for somemm\. We make the following assumption:

###### Assumption 1

Any factorΦi\\Phi\_\{i\}whose arity is larger than 2 can be reasonably approximated by the product of a finite number of pairwise factors\.

With this assumption,\{Φi∣i=1,…,m\}\\\{\\Phi\_\{i\}\\mid i=1,\\dots,m\\\}can be approximated by a set of pairwise factors\{ϕi,j\(zi,zj\)∣i,j=1,…,n\+1\}\\\{\\phi\_\{i,j\}\(z\_\{i\},z\_\{j\}\)\\mid i,j=1,\\dots,n\+1\\\}\. We posit that this MRF with pairwise factors is the model that Transformer works on implicitly\.

###### Assumption 2

Encoder Transformers implicitly make inferences on MRFs with pairwise factors that vary from instance to instance\.

### 3\.2Attention as Belief Propagation

We show that the attention mechanism can be mapped to the max\-product BP algorithm\. Given the MRF above, with pairwise factors\{ϕi,j\(zi,zj\)∣i,j=1,…,n\+1\}\\\{\\phi\_\{i,j\}\(z\_\{i\},z\_\{j\}\)\\mid i,j=1,\\dots,n\+1\\\}, where we defineϕi,i​\(zi,zi\)=ϕi​\(zi\)\\phi\_\{i,i\}\(z\_\{i\},z\_\{i\}\)=\\phi\_\{i\}\(z\_\{i\}\)as its node potential\. In BP, at time steptt, every variableZjZ\_\{j\}passes a \(max\-product\) messagemj,i\(t\)m\_\{j,i\}^\{\(t\)\}to every other variableZiZ\_\{i\}

mj,i\(t\)​\(zi\)=maxzj⁡\[ϕj,i​\(zj,zi\)​∏k∈Nb​\(j\)\\\{i\}mk,j\(t−1\)​\(zj\)\],m\_\{j,i\}^\{\(t\)\}\(z\_\{i\}\)=\\max\_\{z\_\{j\}\}\\left\[\\phi\_\{j,i\}\(z\_\{j\},z\_\{i\}\)\\prod\_\{k\\in\\text\{Nb\}\(j\)\\backslash\\\{i\\\}\}m\_\{k,j\}^\{\(t\-1\)\}\(z\_\{j\}\)\\right\],\(5\)whereNb​\(j\)\\text\{Nb\}\(j\)is the set of neighbors of nodejj\. Then, at the end of iteration, every variable updates its own belief

βi\(t\)​\(zi\)=∏j∈Nb​\(i\)mj,i\(t\)​\(zi\)\.\\beta\_\{i\}^\{\(t\)\}\(z\_\{i\}\)=\\prod\_\{j\\in\\text\{Nb\}\(i\)\}m\_\{j,i\}^\{\(t\)\}\(z\_\{i\}\)\.\(6\)Note that for the sake of uniformity, hereNb​\(i\)\\text\{Nb\}\(i\)includesiiitself and that the self\-messagemi,i​\(zi\)≐ϕi​\(zi\)m\_\{i,i\}\(z\_\{i\}\)\\doteq\\phi\_\{i\}\(z\_\{i\}\)\. Taking log from both sides,

log⁡βi\(t\)​\(zi\)\\displaystyle\\log\\beta\_\{i\}^\{\(t\)\}\(z\_\{i\}\)=∑j∈Nb​\(i\)log⁡mj,i\(t\)​\(zi\)\\displaystyle=\\sum\_\{j\\in\\text\{Nb\}\(i\)\}\\log m\_\{j,i\}^\{\(t\)\}\(z\_\{i\}\)\(7\)=∑j=1n\+1σj,i​log⁡mj,i\(t\)​\(zi\),\\displaystyle=\\sum\_\{j=1\}^\{n\+1\}\\sigma\_\{j,i\}\\log m\_\{j,i\}^\{\(t\)\}\(z\_\{i\}\),\(8\)whereσj,i∈\{0,1\}\\sigma\_\{j,i\}\\in\\\{0,1\\\}indicates whether nodesjjandiiare neighbors\. In Transformers, each node \(or token position\) stores the argumentz˙i\(t\)\\dot\{z\}\_\{i\}^\{\(t\)\}that maximizes its log belief:

z˙i\(t\)=arg⁡max​∑j=1n\+1σj,i​log⁡mj,i\(t\)​\(zi\)\.\\dot\{z\}\_\{i\}^\{\(t\)\}=\\arg\\max\\sum\_\{j=1\}^\{n\+1\}\\sigma\_\{j,i\}\\log m\_\{j,i\}^\{\(t\)\}\(z\_\{i\}\)\.\(9\)If we expand each log messagelog⁡mj,i\(t\)​\(zi\)\\log m\_\{j,i\}^\{\(t\)\}\(z\_\{i\}\)at its maximizerz˙j,i\(t\)\\dot\{z\}\_\{j,i\}^\{\(t\)\}using second\-order Taylor expansion, then we can write the belief maximizer as the precision\-weighted average of the individual message maximizers:

z˙i\(t\)=∑j=1n\+1σj,i​Hj,i​z˙j,i\(t\)∑j=1n\+1σj,i​Hj,i,\\dot\{z\}\_\{i\}^\{\(t\)\}=\\frac\{\\sum\_\{j=1\}^\{n\+1\}\\sigma\_\{j,i\}H\_\{j,i\}\\dot\{z\}\_\{j,i\}^\{\(t\)\}\}\{\\sum\_\{j=1\}^\{n\+1\}\\sigma\_\{j,i\}H\_\{j,i\}\},\(10\)whereHj,i=−∇2​log​mj,i\(t\)​\(zi\)H\_\{j,i\}=\-\\nabla^\{2\}\\log m\_\{j,i\}^\{\(t\)\}\(z\_\{i\}\)is the negative Hessian\. This equation should look familiar by far\. If we further assume thatHj,iH\_\{j,i\}is isotropic,

Hj,i=αj,i​I,H\_\{j,i\}=\\alpha\_\{j,i\}I,\(11\)then,

z˙i\(t\)=∑j=1n\+1σj,i​αj,i​z˙j,i\(t\)∑j=1n\+1σj,i​αj,i\.\\dot\{z\}\_\{i\}^\{\(t\)\}=\\frac\{\\sum\_\{j=1\}^\{n\+1\}\\sigma\_\{j,i\}\\alpha\_\{j,i\}\\dot\{z\}\_\{j,i\}^\{\(t\)\}\}\{\\sum\_\{j=1\}^\{n\+1\}\\sigma\_\{j,i\}\\alpha\_\{j,i\}\}\.\(12\)Self\-attention uses the following equations to compute the terms in Eq\.[12](https://arxiv.org/html/2610.12741#S3.E12):

σj,i​αj,i\\displaystyle\\sigma\_\{j,i\}\\alpha\_\{j,i\}=exp⁡\(qiT​kj\)=exp⁡\(\(Wq​z˙i\(t−1\)\)T​\(Wk​z˙j\(t−1\)\)\)\\displaystyle=\\exp\{\(q\_\{i\}^\{T\}k\_\{j\}\)\}=\\exp\{\(\(W\_\{q\}\\dot\{z\}\_\{i\}^\{\(t\-1\)\}\)^\{T\}\(W\_\{k\}\\dot\{z\}\_\{j\}^\{\(t\-1\)\}\)\)\}\(13\)z˙j,i\(t\)\\displaystyle\\dot\{z\}\_\{j,i\}^\{\(t\)\}=vj=Wv​z˙j\(t−1\),\\displaystyle=v\_\{j\}=W\_\{v\}\\dot\{z\}\_\{j\}^\{\(t\-1\)\},\(14\)whereq,k,vq,k,vare feature vectors for query, key, and value, andWq,Wk,WvW\_\{q\},W\_\{k\},W\_\{v\}are their weight matrices\. We note here that the initial beliefβi\(0\)​\(zi\)=ϕi​\(zi\)\\beta\_\{i\}^\{\(0\)\}\(z\_\{i\}\)=\\phi\_\{i\}\(z\_\{i\}\), mentioned in the last section\. Thus, the embedding of each token can be seen as the maximizer of the prior\.

Note thatσj,i\\sigma\_\{j,i\}is capable of learning and adjusting the strength of the connection between two variables, which is equivalent to learning the structure of the underlying MRF\. Consequently, the repetitive applications of layers of self\-attention can be seen as performing LBP to infer the latent variables and inferring the structure of the MRF at the same time\.

### 3\.3Transformers as Expectation\-Maximization

In the last section, we see how self\-attention handles hidden variables and unknown structure\. There is one last loose thread: the unknown parameters\. In this section, we show that a Transformer instantiates itself as a hard EM\[[20](https://arxiv.org/html/2610.12741#bib.bib32)\]to infer the MAP parameters\.

Formally, consider supervised learning\. Letxxbe the input,yythe label,zzthe unobserved latent variables, andθ\\thetathe model parameters\. The goal of the model is

arg⁡maxθ⁡log⁡p⁡\(y\|x;θ\)\.\\arg\\max\_\{\\theta\}\\log p\(y\|x;\\theta\)\.\(15\)We have

logp\(y\|x;θ\)=logp\(y,z\|x;θ\)−logp\(z\|y,x;θ\)\.\\displaystyle\\log p\(y\|x;\\theta\)=\\log p\(y,z\|x;\\theta\)\-\\log p\(z\|y,x;\\theta\)\.\(16\)Letθ\(l\)\\theta^\{\(l\)\}andz\(l\)z^\{\(l\)\}be, respectively, the parameter and the output of layerllof a Transformer\. Then

log⁡p⁡\(y\|x;θ\)\\displaystyle\\log p\(y\|x;\\theta\)=𝔼p⁡\(z\|y,x;θ\(l\)\)logp\(y,z\|x;θ\)−𝔼p⁡\(z\|y,x;θ\(l\)\)logp\(z\|y,x;θ\)\\displaystyle=\\mathbb\{E\}\_\{p\(z\|y,x;\\theta^\{\(l\)\}\)\}\\log p\(y,z\|x;\\theta\)\-\\mathbb\{E\}\_\{p\(z\|y,x;\\theta^\{\(l\)\}\)\}\\log p\(z\|y,x;\\theta\)\(17\)=Q⁡\(θ\|θ\(l\)\)\+H⁡\(θ\|θ\(l\)\),\\displaystyle=Q\(\\theta\|\\theta^\{\(l\)\}\)\+H\(\\theta\|\\theta^\{\(l\)\}\),\(18\)where we have defined

Q⁡\(θ\|θ\(l\)\)\\displaystyle Q\(\\theta\|\\theta^\{\(l\)\}\)=𝔼p⁡\(z\|y,x;θ\(l\)\)logp\(y,z\|x;θ\),\\displaystyle=\\mathbb\{E\}\_\{p\(z\|y,x;\\theta^\{\(l\)\}\)\}\\log p\(y,z\|x;\\theta\),\(19\)H⁡\(θ\|θ\(l\)\)\\displaystyle H\(\\theta\|\\theta^\{\(l\)\}\)=−𝔼p⁡\(z\|y,x;θ\(l\)\)​log⁡p⁡\(z\|y,x;θ\)\.\\displaystyle=\-\\mathbb\{E\}\_\{p\(z\|y,x;\\theta^\{\(l\)\}\)\}\\log p\(z\|y,x;\\theta\)\.\(20\)With Gibbs inequality,

log⁡p⁡\(y\|x;θ\)−log⁡p⁡\(y\|x;θ\(l\)\)\\displaystyle\\log p\(y\|x;\\theta\)\-\\log p\(y\|x;\\theta^\{\(l\)\}\)\(21\)=Q⁡\(θ\|θ\(l\)\)−Q⁡\(θ\(l\)\|θ\(l\)\)\+H⁡\(θ\|θ\(l\)\)−H⁡\(θ\(l\)\|θ\(l\)\)\\displaystyle=Q\(\\theta\|\\theta^\{\(l\)\}\)\-Q\(\\theta^\{\(l\)\}\|\\theta^\{\(l\)\}\)\+H\(\\theta\|\\theta^\{\(l\)\}\)\-H\(\\theta^\{\(l\)\}\|\\theta^\{\(l\)\}\)\(22\)≥Q⁡\(θ\|θ\(l\)\)−Q⁡\(θ\(l\)\|θ\(l\)\),\\displaystyle\\geq Q\(\\theta\|\\theta^\{\(l\)\}\)\-Q\(\\theta^\{\(l\)\}\|\\theta^\{\(l\)\}\),\(23\)which implies that improvement inQ⁡\(θ\|θ\(l\)\)Q\(\\theta\|\\theta^\{\(l\)\}\)results in at least as much improvement in the log likelihood\.

The forward pass of layerllcomputesQ⁡\(θ\|θ\(l\)\)Q\(\\theta\|\\theta^\{\(l\)\}\), i\.e\., the E step\. However, instead of computing the exact posteriorp⁡\(z\|y,x;θ\(l\)\)p\(z\|y,x;\\theta^\{\(l\)\}\), layerllonly runs one iteration of approximate max\-product BP to get an estimated maximizerz\(l\)z^\{\(l\)\}\. Then we have

Q\(θ\|θ\(l\)\)=logp\(y,z\(l\)\|x;θ\)\.Q\(\\theta\|\\theta^\{\(l\)\}\)=\\log p\(y,z^\{\(l\)\}\|x;\\theta\)\.\(24\)The job of the next layerl\+1l\+1is to maximizeQ⁡\(θ\|θ\(l\)\)Q\(\\theta\|\\theta^\{\(l\)\}\), i\.e\., the M step\. Specifically, The parameterθ\(l\+1\)\\theta^\{\(l\+1\)\}of the next layer corresponds to the parameterθ\\thetathat maximizesQ⁡\(θ\|θ\(l\)\)Q\(\\theta\|\\theta^\{\(l\)\}\):

θ\(l\+1\)=argmaxθlogp\(y,z\(l\)\|x;θ\)\.\\theta^\{\(l\+1\)\}=\\arg\\max\_\{\\theta\}\\log p\(y,z^\{\(l\)\}\|x;\\theta\)\.\(25\)
See Fig\.[2](https://arxiv.org/html/2610.12741#S1.F2)for an illustration\. In summary, we can view a layer of a Transformer as an iteration of the EM\. The forward pass of a layer corresponds to the E step, where the latent variables for each token are updated on the basis of last layer\. The change in the parameters of a layer compared to its previous layer implicitly corresponds to the M step, where the model parameters are updated\.

The EM point of view has several consequences\. Here, we list a few we can think of\.

- •It provides key insight into the in\-context learning capability of Transformers, to be elaborated in the next section\.
- •Training a model is equivalent to learning an EM algorithm\.
- •It partly explains the efficacy of ResNet\[[13](https://arxiv.org/html/2610.12741#bib.bib16)\], where each layer is only responsible for learning an increment to latent variables\.
- •The parameters of different layers are essentially different estimates of the same set of parameters\. This may partly explain why DNNs do not suffer from significant overfitting\[[30](https://arxiv.org/html/2610.12741#bib.bib17)\], because the effective number of parameters remains constant as layers grow\. This constitutes a complementary point of view to some previous ones\[[21](https://arxiv.org/html/2610.12741#bib.bib15),[23](https://arxiv.org/html/2610.12741#bib.bib18),[2](https://arxiv.org/html/2610.12741#bib.bib19)\]\.

### 3\.4Meta\-Models

An important consequence of the EM point of view lies in the M\-step:θ\(l\+1\)=argmaxθlogp\(y,z\(l\)\|x;θ\)\\theta^\{\(l\+1\)\}=\\arg\\max\_\{\\theta\}\\log p\(y,z^\{\(l\)\}\|x;\\theta\)\. The M\-step says that the parametersθ\(l\+1\)\\theta^\{\(l\+1\)\}of layerl\+1l\+1should be dependent on its inputz\(l\)z^\{\(l\)\}, which implies a procedure that generates parameters based on the input, prior to the forward pass\. This leads to a natural model design paradigm defined by the following two sequential passes:

θ\\displaystyle\\theta=FΘ​\(z\),\\displaystyle=F\_\{\\Theta\}\(z\),\(26\)z′\\displaystyle z^\{\\prime\}=fθ​\(z\),\\displaystyle=f\_\{\\theta\}\(z\),whereffis the forward pass,FFthe meta\-forward pass,θ\\thetathe parameters,Θ\\Thetathe meta\-parameters, andzzandz′z^\{\\prime\}respectively the input and output\. In the meta\-forward pass,FFtakeszzas input and determines what parameters to use in the forward pass\. This dynamic generation of parameters is what makes in\-context learning possible\. In fact, many familiar deep architectures are members of meta\-models\.

#### Self\-attention as a meta\-model

\. Letz,z′∈ℝd×nz,z^\{\\prime\}\\in\\mathbb\{R\}^\{d\\times n\}be the input and output andQ,K,V∈ℝd×dQ,K,V\\in\\mathbb\{R\}^\{d\\times d\}be the query, key, and value matrices\. The following setup is equivalent to self\-attention:

A\\displaystyle A≐F\{Q,K\}​\(z\)=σ⁡\(\(Q​z\)T​\(K​z\)/d\),\\displaystyle\\doteq F\_\{\\\{Q,K\\\}\}\(z\)=\\sigma\(\(Qz\)^\{T\}\(Kz\)/\\sqrt\{d\}\),\(27\)z′\\displaystyle z^\{\\prime\}≐fA​\(z\)=A​z​V,\\displaystyle\\doteq f\_\{A\}\(z\)=AzV,whereσ\\sigmais the Softmax operation\.

#### MoE as a meta\-model

\. In the Mixture of Experts \(MoE\) models, letKKbe the number of experts,W∈ℝK×d′×dW\\in\\mathbb\{R\}^\{K\\times d^\{\\prime\}\\times d\}the parameters for each expert,R∈ℝK×dR\\in\\mathbb\{R\}^\{K\\times d\}the routing parameters,z∈ℝdz\\in\\mathbb\{R\}^\{d\}andz′∈ℝd′z^\{\\prime\}\\in\\mathbb\{R\}^\{d^\{\\prime\}\}respectively the input and output\. The following setup is equivalent to MoE:

w\\displaystyle w≐FR​\(z\)=σ⁡\(R​z\),\\displaystyle\\doteq F\_\{R\}\(z\)=\\sigma\(Rz\),\(28\)z′\\displaystyle z^\{\\prime\}≐fw​\(z\)=wT​W​z,\\displaystyle\\doteq f\_\{w\}\(z\)=w^\{T\}Wz,whereσ\\sigmais the Softmax operation\. Note that MoE is essentially a meta version of the linear layer\.

Meta\-models also include LSTM\[[14](https://arxiv.org/html/2610.12741#bib.bib13)\]and Mamba\[[11](https://arxiv.org/html/2610.12741#bib.bib14)\], which we will leave to interested readers\. Meta\-models entail a two\-pass procedure ofthinking about your own thinking, which could possibly be likened to the meta\-cognition\[[9](https://arxiv.org/html/2610.12741#bib.bib33)\]in psychology\.

#### Meta\-Linear model

\. Here, we also propose a ordinary design for the Meta\-Linear layer\. Letffbe a linear operation with parameterθ∈ℝd×d′\\theta\\in\\mathbb\{R\}^\{d\\times d^\{\\prime\}\}andFFits meta\-linear operation with parameterΘ∈ℝk×d′×d\\Theta\\in\\mathbb\{R\}^\{k\\times d^\{\\prime\}\\times d\}\. Then

θ\\displaystyle\\theta=FΘ​\(z\)=ΘT​σ​\(z\),\\displaystyle=F\_\{\\Theta\}\(z\)=\\Theta^\{T\}\\sigma\(z\),\(29\)z′\\displaystyle z^\{\\prime\}=fθ​\(z\)=θT​z,\\displaystyle=f\_\{\\theta\}\(z\)=\\theta^\{T\}z,wherez∈ℝdz\\in\\mathbb\{R\}^\{d\}andz′∈ℝd′z^\{\\prime\}\\in\\mathbb\{R\}^\{d^\{\\prime\}\}\.σ\\sigmais some \(parameterized\) nonlinearity that projectsdd\-dimension tokk\-dimension, ifk≠dk\\neq d\. If we letσ⁡\(x\)=Softmax​\(xT​R\)\\sigma\(x\)=\\text\{Softmax\}\(x^\{T\}R\), whereRRisdd\-by\-kk, and interpretkkas the number of experts, then this is equivalent to akk\-expert MoE model\. The parameter count \(excluding nonlinearity\) of a Meta\-Linear layer iskktimes that of a linear layer\. See Fig\.[2](https://arxiv.org/html/2610.12741#S1.F2)for an illustration\.

### 3\.5Generalized Meta\-Models

We can further generalize the idea of meta\-models to higher orders\. Here, we take the linear layer as an example\. For annn\-th order meta\-linear \(or metan\-linear\) layer with inputz∈ℝdz\\in\\mathbb\{R\}^\{d\}and outputz′∈ℝd′z^\{\\prime\}\\in\\mathbb\{R\}^\{d^\{\\prime\}\}, the procedure becomes

Θ\(n−1\)=F\(n\)​\(z\)=Θ\(n\)​σ​\(z\),\\displaystyle\\Theta^\{\(n\-1\)\}=F^\{\(n\)\}\(z\)=\\Theta^\{\(n\)\}\\sigma\(z\),\(30\)Θ\(i−1\)=F\(i\)​\(z\)=Θ\(i\)​σ​\(z\),\\displaystyle\\Theta^\{\(i\-1\)\}=F^\{\(i\)\}\(z\)=\\Theta^\{\(i\)\}\\sigma\(z\),\(31\)θ=Θ\(0\)=F\(1\)​\(z\)=Θ\(1\)​σ​\(z\),\\displaystyle\\theta=\\Theta^\{\(0\)\}=F^\{\(1\)\}\(z\)=\\Theta^\{\(1\)\}\\sigma\(z\),\(32\)z′=Θ\(−1\)=F\(0\)​\(z\)=Θ\(0\)​z,\\displaystyle z^\{\\prime\}=\\Theta^\{\(\-1\)\}=F^\{\(0\)\}\(z\)=\\Theta^\{\(0\)\}z,\(33\)whereΘ\(i\)∈ℝd′×d×k1​⋯×ki\\Theta^\{\(i\)\}\\in\\mathbb\{R\}^\{d^\{\\prime\}\\times d\\times k\_\{1\}\\dots\\times k\_\{i\}\}is called metai\-parameter, andF\(i\)F^\{\(i\)\}the metai\-forward pass\. Note, especially, thatΘ\(−1\)=z′\\Theta^\{\(\-1\)\}=z^\{\\prime\}is the output,Θ\(0\)=θ\\Theta^\{\(0\)\}=\\thetaandF\(0\)=fF^\{\(0\)\}=f\. The parameter count of a generalized meta\-linear layer of ordernnis

\#parameter=d′​d​kn,\\text\{\\\#parameter\}=d^\{\\prime\}dk^\{n\},\(34\)which grows exponentially with its order\.

## 4Experiments

### 4\.1Retrieving the Probabilistic Graphs

If Transformers are indeed making inferences in the PGMs mentioned in Section[3\.1](https://arxiv.org/html/2610.12741#S3.SS1), we can somehow retrieve the PGMs from their outputs\. In an attempt to do this, we do the following experiments\. Given an input text, we take the output feature vector of each token from the last layer of a Transformer, and remove vectors of those non\-semantic or special tokens, e\.g\. punctuations, newlines, </s\>, and <pad\>\. However, the class tokens, e\.g\. <s\> and \[CLS\], used for final classification, if present, are retained\. We compute the similarity matrix𝒮\\mathcal\{S\}, where each entry is the cosine similarity between two feature vectors\.

𝒮i,j=cosine\_similarity​\(zi,zj\)\.\\mathcal\{S\}\_\{i,j\}=\\text\{cosine\\\_similarity\}\(z\_\{i\},z\_\{j\}\)\.\(35\)Then we run single\-linkage agglomerative hierarchical clustering algorithm on the matrix to detect hierarchical communities and plot the results as dendrograms\. We do experiments on multiple movie reviews and real\-life texts from X \(formerly Twitter\) with multiple Transformers from Huggingface\. Models include:

- •twitter\-roberta\-base\-sentiment\-latest\[[19](https://arxiv.org/html/2610.12741#bib.bib21)\]\. A RoBERTa\-base model pretrained on about 124M tweets \(2018\-2021\) and fine\-tuned for three\-class sentiment classification \(negative, neutral, positive\) on the TweetEval benchmark \(Loureiro et al\., 2022; Barbieri et al\., 2020\)\. Its class token is <s\>\.
- •distilbert\-base\-uncased\[[24](https://arxiv.org/html/2610.12741#bib.bib22)\]\. A compact \(66M parameters\), general\-purpose English encoder obtained by distilling BERT\-base \(Sanh et al\., 2019\)\. Its class token is \[CLS\]\.
- •BGE\-M3\[[3](https://arxiv.org/html/2610.12741#bib.bib20)\]\. A multilingual embedding model built on XLM\-RoBERTa \(about 568M parameters\)\. Its class token is <s\>\.
- •Qwen3\-Embedding\-0\.6B\[[31](https://arxiv.org/html/2610.12741#bib.bib23)\]\. A 0\.6B\-parameter text embedding model derived from the Qwen3 foundation model\. It does not have a class token\.

The results are shown in Fig\.[4](https://arxiv.org/html/2610.12741#S4.F4)\. A few important observations can be made from the results\.

- •Most compound words, phrases, and collocations, e\.g\. "like\-minded", "cleaned up", are immediately grouped together by Transformers\. This indicates Transformers are aware of the fact that they form a single entity and each word in a compound is generated by that entity\.
- •These recognized entities are further merged with other related tokens or entities, forming a hierarchy\.
- •Some pronouns are immediate neighbors to some other entities to which they refer in the original texts\.
- •The class token is significantly distant from other communities and is lastly merged into the hierarchy, which indicates that Transformers recognize it as the root cause\.

Figure 4:The agglomerative hierarchical clustering dendrograms for various models and texts\. We run single\-linkage agglomerative hierarchical clustering algorithm on the features of Transformers’ last layer to detect hierarchical communities and plot the results as dendrograms\. We can see that the class tokens are lastly merged into the hierarchy, supporting our theory\.
### 4\.2Meta\-Linear models

We evaluate the expressiveness and classification capabilities of Linear and Meta\-Linear models on synthetic data generated from Gaussian Mixture Models \(GMMs\)\.

To evaluate model performance across multiple sub\-distributions, we generatedMMdistinct 2D datasets \(d=2d=2\), each consisting of two Gaussian mixture distributions representing Class 0 \(𝒩1\\mathcal\{N\}\_\{1\}\) and Class 1 \(𝒩2\\mathcal\{N\}\_\{2\}\)\. We repeat the experiments for various number of GMM models ofM=1,2,4,8M=1,2,4,8\. The generated data are visualized in Fig\.[5](https://arxiv.org/html/2610.12741#S4.F5)\.

Covariance Sampling: For each datasetc∈\{0,…,M−1\}c\\in\\\{0,\\dots,M\-1\\\}, a shared covariance matrix𝚺c\\mathbf\{\\Sigma\}\_\{c\}is sampled from a Wishart distribution \(df=d\+3=5\\text\{df\}=d\+3=5\) with scale matrix𝐒=0\.8​𝐈2\\mathbf\{S\}=0\.8\\mathbf\{I\}\_\{2\}\.

Class Separation: The difference in class means𝜹=𝝁1−𝝁0\\boldsymbol\{\\delta\}=\\boldsymbol\{\\mu\}\_\{1\}\-\\boldsymbol\{\\mu\}\_\{0\}is scaled according to a target Mahalanobis distancedM=3\.2d\_\{M\}=3\.2, yielding high linear separability \( 93–98%\) per sub\-model\.

Data Splitting: Each class containsN=4000,N=4000,samples per dataset\. The combined dataset is split into training \(75%\), evaluation \(15%\), and test \(15%\) using stratified sampling\.

Theoretical classification accuracy: We also compute the theoretical upper bound for classification accuracy on test set using the optimal Bayes decision boundary from training set\. For each sub\-modelcc, the decision boundary is given by the linear discriminant scoref⁡\(𝐱\)=𝐰cT​𝐱\+bcf\(\\mathbf\{x\}\)=\\mathbf\{w\}\_\{c\}^\{T\}\\mathbf\{x\}\+b\_\{c\}, where

𝐰c=𝚺c−1​\(𝝁1−𝝁0\)\\mathbf\{w\}\_\{c\}=\\mathbf\{\\Sigma\}\_\{c\}^\{\-1\}\(\\boldsymbol\{\\mu\}\_\{1\}\-\\boldsymbol\{\\mu\}\_\{0\}\)bc=−12​\(𝝁1\+𝝁0\)T​𝐰cb\_\{c\}=\-\\frac\{1\}\{2\}\(\\boldsymbol\{\\mu\}\_\{1\}\+\\boldsymbol\{\\mu\}\_\{0\}\)^\{T\}\\mathbf\{w\}\_\{c\}Predictions are assigned based onsign​\(f​\(𝐱\)\)\\text\{sign\}\(f\(\\mathbf\{x\}\)\), establishing the ground\-truth classification accuracy on the test split\.

Models compared: We compare several custom neural network layer formulations designed to dynamically weight input features:

1. 1\.Linear: A vanilla linear layer of the formf⁡\(𝐱\)=W​𝐱\+bf\(\\mathbf\{x\}\)=W\\mathbf\{x\}\+b\.
2. 2\.Meta\-Linear: A implementation evaluatingf⁡\(𝐱\)=σ⁡\(𝐱T\)​\(W​𝐱\+B\)f\(\\mathbf\{x\}\)=\\sigma\(\\mathbf\{x\}^\{T\}\)\(W\\mathbf\{x\}\+B\), where meta\-weightWWis of shape\(k,d′,d\)\(k,d^\{\\prime\},d\)and meta\-biasBBof\(k,d′\)\(k,d^\{\\prime\}\)\. We compare different choices of non\-linearityσ\\sigma: - •Identityσ⁡\(x\)=x\\sigma\(x\)=x - •Swish​\(x\)=x⊙Sigmoid​\(x\)\\text\{Swish\}\(x\)=x\\odot\\text\{Sigmoid\}\(x\) - •ParametrizedSwish​\(x\)=x⊙Sigmoid​\(V​x\)\\text\{Swish\}\(x\)=x\\odot\\text\{Sigmoid\}\(Vx\) - •SwiGLU​\(x\)=V​x⊙Sigmoid​\(V​x\)⊙W​x\\text\{SwiGLU\}\(x\)=Vx\\odot\\text\{Sigmoid\}\(Vx\)\\odot Wx - •Ground\-truth MoE: We can make Meta\-Linear equivalent to MoE byσ​\(x\)=Softmax​\(R​x\)\\sigma\(x\)=\\text\{Softmax\}\(Rx\)\. We let number of experts be the ground\-truthk=Mk=M\.

We use Binary Cross\-Entropy optimized via ADAM\[[15](https://arxiv.org/html/2610.12741#bib.bib24)\]with a learning rateη=0\.0001\\eta=0\.0001and batch size 32 for 1000 epochs\. Loss and accuracy are monitored at each epoch on the training and evaluation set, with final generalization accuracy evaluated on the held\-out test dataset\. The results are shown in Fig\.[6](https://arxiv.org/html/2610.12741#S4.F6)\.

![Refer to caption](https://arxiv.org/html/2610.12741v1/media/experiments/metamodels/M1.png)\(a\)Number of GMMsM=1M=1
![Refer to caption](https://arxiv.org/html/2610.12741v1/media/experiments/metamodels/M2.png)\(b\)M=2M=2
![Refer to caption](https://arxiv.org/html/2610.12741v1/media/experiments/metamodels/M4.png)\(c\)M=4M=4
![Refer to caption](https://arxiv.org/html/2610.12741v1/media/experiments/metamodels/M8.png)\(d\)M=8M=8

Figure 5:The generated synthetic multi\-GMM data sets consisting of different number of GMM modelsM=1,2,4,8M=1,2,4,8\.with ground\-truth Bayes decision boundaries\.

![Refer to caption](https://arxiv.org/html/2610.12741v1/media/experiments/metamodels/result_1_4000_1000_0.0001_32_0_3.2_23.png)\(a\)M=1M=1
![Refer to caption](https://arxiv.org/html/2610.12741v1/media/experiments/metamodels/result_2_2000_1000_0.0001_32_5_3.2_23.png)\(b\)M=2M=2
![Refer to caption](https://arxiv.org/html/2610.12741v1/media/experiments/metamodels/result_4_1500_1000_0.0001_32_8_3.2_23.png)\(c\)M=4M=4
![Refer to caption](https://arxiv.org/html/2610.12741v1/media/experiments/metamodels/result_8_1000_1000_0.0001_32_12_3.2_23.png)\(d\)M=8M=8

Figure 6:Training results for comparing different linear models for differentMM\.Based on the experimental evaluation, the following conclusions can be drawn:

- •Baseline Degradation with Multi\-Modal Complexity: The vanilla Linear layer is highly effective \(accuracy 0\.9383\) only for simple, linearly separable data \(M=1M=1\)\. AsMMincreases toM=2,4,M=2,4,and88, the classification task becomes significantly non\-linear, causing the vanilla Linear model’s accuracy to degrade to 0\.6283, 0\.5361, and ultimately 0\.5171, respectively\.
- •Superior Expressiveness of Meta\-Linear Models: Dense Meta\-Linear formulations successfully overcome the limitations of the baseline by dynamically weighting weight matrix\. Across all multi\-modal datasets \(M\>1M\>1\), the Meta\-Linear variants consistently outperform the standard Linear layer, demonstrating a greater capacity to model complex decision boundaries\.
- •Efficacy of Non\-Linear Gating Functions: The specific choice of non\-linearity \(σ\\sigma\) plays a critical role in the model’s classification capability\. Advanced gating mechanisms, particularly SwiGLU and Parametrized Swish \(PSwish\), provide better adaptability than simpler formulations like the identity \(no\-gate\) function\. For instance, at the highest complexity \(M=8M=8\), the Meta\-Linear SwiGLU variant achieves an evaluation accuracy of 0\.6842, substantially outperforming the vanilla baseline\.
- •Ground\-Truth MoE as the Performance Upper Bound: The Ground\-truth MoE, which explicitly sets the number of experts equal to the number of sub\-models \(k=Mk=M\), establishes the theoretical upper limit for classification accuracy in these experiments\. It remains highly robust against dataset complexity, maintaining top\-tier accuracies of 0\.9400 atM=2M=2, 0\.9022 atM=4M=4, and 0\.8079 atM=8M=8\.

## 5Conclusion

This paper has developed a probabilistic\-inference interpretation of Transformers as Loopy Belief Propagated Expectation\-Maximization\. The central problem is to explain how self\-attention, depth, latent representation updates, graph\-structure selection, and layerwise parameter variation can be understood within one theoretical framework\. We argue that self\-attention approximates LBP in a dynamically structured MRF, where attention weightsσj,i\\sigma\_\{j,i\}encode learned structural affinities and value vectors approximate message\-specific maximizers\. We further interpreted successive Transformer layers as an EM\-like process in which the forward pass updates latent variables and the transition fromθ\(l\)\\theta^\{\(l\)\}toθ\(l\+1\)\\theta^\{\(l\+1\)\}acts as an implicit M\-step\.

This perspective suggests that several known properties of deep architectures have a common origin\. Residual depth can be understood as iterative refinement of latent variables, while dynamic computation mechanisms such as self\-attention, MoE, gated units, LSTM, and selective state\-space models can be viewed as forms of input\-conditioned parameter generation\. Empirical findings are strongly supportive of our claims\. Even without empirical claims, the derivations provide a unified language via graphical\-model views of Transformer computation\.

Future work should make this interpretation quantitatively testable\. First, one can measure whether attention maps behave like inferred MRF structures by comparing them with known dependency graphs in synthetic graphical\-model tasks\. Second, one can test the EM interpretation by tracking whether layer\-wise representations and parameters improve a surrogateQ⁡\(θ\|θ\(l\)\)Q\(\\theta\|\\theta^\{\(l\)\}\)objective across depth\. Third, the meta\-model principle motivates new architectures that generate parameters with controlled order and sparsity, allowing a direct study of the tradeoff between expressive dynamic computation and the exponential parameter growth of generalized meta\-models\.

## References

- \[1\]E\. Akyürek, D\. Schuurmans, J\. Andreas, T\. Ma, and D\. Zhou\(2023\)What learning algorithm is in\-context learning? investigations with linear models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=0g0X4H8yN4I)Cited by:[§1](https://arxiv.org/html/2610.12741#S1.p2.1)\.
- \[2\]M\. Belkin, D\. Hsu, S\. Ma, and S\. Mandal\(2019\)Reconciling modern machine\-learning practice and the classical bias–variance trade\-off\.Proceedings of the National Academy of Sciences116\(32\),pp\. 15849–15854\.External Links:ISSN 1091\-6490,[Link](http://dx.doi.org/10.1073/pnas.1903070116),[Document](https://dx.doi.org/10.1073/pnas.1903070116)Cited by:[4th item](https://arxiv.org/html/2610.12741#S3.I1.i4.p1.1)\.
- \[3\]J\. Chen, S\. Xiao, P\. Zhang, K\. Luo, D\. Lian, and Z\. Liu\(2024\)Bge m3\-embedding: multi\-lingual, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation\.arXiv preprint arXiv:2402\.032164\(5\)\.Cited by:[3rd item](https://arxiv.org/html/2610.12741#S4.I1.i3.p1.1)\.
- \[4\]Z\. Chen, R\. Wu, and G\. Fang\(2025\)Transformers as unsupervised learning algorithms: a study on gaussian mixtures\.arXiv preprint arXiv:2505\.11918\.Note:Proves transformers can approximate the EM algorithm and spectral \(tensor power\) iterations\.Cited by:[§2](https://arxiv.org/html/2610.12741#S2.SS0.SSS0.Px1.p1.1)\.
- \[5\]D\. Dai, Y\. Sun, L\. Dong, Y\. Hao, Z\. Sui, and F\. Wei\(2023\)Why can GPT learn in\-context? language models secretly perform gradient descent as meta\-optimizers\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 4005–4019\.Cited by:[§1](https://arxiv.org/html/2610.12741#S1.p2.1)\.
- \[6\]Y\. N\. Dauphin, A\. Fan, M\. Auli, and D\. Grangier\(2017\)Language modeling with gated convolutional networks\.InProceedings of the 34th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.70,pp\. 933–941\.Cited by:[§2](https://arxiv.org/html/2610.12741#S2.SS0.SSS0.Px3.p1.1)\.
- \[7\]A\. P\. Dempster, N\. M\. Laird, and D\. B\. Rubin\(1977\)Maximum likelihood from incomplete data via the EM algorithm\.Journal of the Royal Statistical Society: Series B \(Methodological\)39\(1\),pp\. 1–38\.Cited by:[§1](https://arxiv.org/html/2610.12741#S1.p3.1)\.
- \[8\]N\. Elhage, N\. Nanda, C\. Olsson, T\. Henighan, N\. Joseph, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly, N\. DasSarma, D\. Drain, D\. Ganguli, Z\. Hatfield\-Dodds, D\. Hernandez, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, T\. Brown, J\. Clark, J\. Kaplan, S\. McCandlish, and C\. Olah\(2021\)A mathematical framework for transformer circuits\.Note:Transformer Circuits ThreadExternal Links:[Link](https://transformer-circuits.pub/2021/framework/index.html)Cited by:[§1](https://arxiv.org/html/2610.12741#S1.p1.1)\.
- \[9\]J\. H\. Flavell\(1979\)Metacognition and cognitive monitoring: a new area of cognitive–developmental inquiry\.\.American psychologist34\(10\),pp\. 906\.Cited by:[§3\.4](https://arxiv.org/html/2610.12741#S3.SS4.SSS0.Px2.p2.1)\.
- \[10\]J\. Gilmer, S\. S\. Schoenholz, P\. F\. Riley, O\. Vinyals, and G\. E\. Dahl\(2017\)Neural message passing for quantum chemistry\.International Conference on Machine Learning \(ICML\)\.Note:arXiv:1704\.01212\. Canonical message\-passing neural network \(MPNN\)\.Cited by:[§2](https://arxiv.org/html/2610.12741#S2.SS0.SSS0.Px2.p1.1)\.
- \[11\]A\. Gu and T\. Dao\(2023\)Mamba: linear\-time sequence modeling with selective state spaces\.Note:arXiv preprint arXiv:2312\.00752External Links:[Link](https://arxiv.org/abs/2312.00752)Cited by:[§2](https://arxiv.org/html/2610.12741#S2.SS0.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2610.12741#S3.SS4.SSS0.Px2.p2.1)\.
- \[12\]D\. Ha, A\. Dai, and Q\. V\. Le\(2017\)HyperNetworks\.International Conference on Learning Representations \(ICLR\)\.Note:arXiv:1609\.09106\. Foundational: a network \(meta\-parameters\) generates the weights \(parameters\) of another network — the meta\-model principle\.Cited by:[§2](https://arxiv.org/html/2610.12741#S2.SS0.SSS0.Px3.p1.1)\.
- \[13\]K\. He, X\. Zhang, S\. Ren, and J\. Sun\(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 770–778\.Cited by:[3rd item](https://arxiv.org/html/2610.12741#S3.I1.i3.p1.1)\.
- \[14\]S\. Hochreiter and J\. Schmidhuber\(1997\)Long short\-term memory\.Neural Computation9\(8\),pp\. 1735–1780\.Cited by:[§2](https://arxiv.org/html/2610.12741#S2.SS0.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2610.12741#S3.SS4.SSS0.Px2.p2.1)\.
- \[15\]D\. P\. Kingma and J\. Ba\(2014\)Adam: a method for stochastic optimization\.arXiv preprint arXiv:1412\.6980\.Cited by:[§4\.2](https://arxiv.org/html/2610.12741#S4.SS2.p8.1)\.
- \[16\]D\. Koller and N\. Friedman\(2009\)Probabilistic graphical models: principles and techniques\.MIT Press,Cambridge, MA\.Cited by:[§1](https://arxiv.org/html/2610.12741#S1.p3.1)\.
- \[17\]C\. W\. Lau, X\. Shi, Z\. Zheng, H\. Cao, and N\. Guo\(2025\)Interplay between belief propagation and transformer: differential\-attention message passing transformer\.arXiv preprint arXiv:2509\.15637\.Cited by:[§2](https://arxiv.org/html/2610.12741#S2.SS0.SSS0.Px2.p1.1)\.
- \[18\]N\. Lee, Y\. Kim, M\. Oh, S\. Kim, J\. W\. Koo, H\. Jo, and J\. Lee\(2025\)Mitigating attention localization in small scale: self\-attention refinement via one\-step belief propagation\.arXiv preprint arXiv:2509\.07324\.Cited by:[§2](https://arxiv.org/html/2610.12741#S2.SS0.SSS0.Px2.p1.1)\.
- \[19\]D\. Loureiro, F\. Barbieri, L\. Neves, L\. E\. Anke, and J\. Camacho\-Collados\(2022\)TimeLMs: diachronic language models from twitter\.InProceedings of the 60th annual meeting of the association for computational linguistics: System demonstrations,pp\. 251–260\.Cited by:[1st item](https://arxiv.org/html/2610.12741#S4.I1.i1.p1.1)\.
- \[20\]R\. M\. Neal and G\. E\. Hinton\(1998\)A view of the em algorithm that justifies incremental, sparse, and other variants\.InLearning in graphical models,pp\. 355–368\.Cited by:[§3\.3](https://arxiv.org/html/2610.12741#S3.SS3.p1.1)\.
- \[21\]B\. Neyshabur, S\. Bhojanapalli, D\. McAllester, and N\. Srebro\(2017\)Exploring generalization in deep learning\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\. 5947–5956\.Cited by:[4th item](https://arxiv.org/html/2610.12741#S3.I1.i4.p1.1)\.
- \[22\]C\. Olsson, N\. Elhage, N\. Nanda, N\. Joseph, N\. DasSarma, T\. Henighan, B\. Mann, A\. Askell, Y\. Bai, A\. Chen, T\. Conerly, D\. Drain, D\. Ganguli, Z\. Hatfield\-Dodds, D\. Hernandez, S\. Johnston, A\. Jones, J\. Kernion, L\. Lovitt, K\. Ndousse, D\. Amodei, T\. Brown, J\. Clark, J\. Kaplan, S\. McCandlish, and C\. Olah\(2022\)In\-context learning and induction heads\.Note:arXiv preprint arXiv:2209\.11895External Links:[Link](https://arxiv.org/abs/2209.11895)Cited by:[§1](https://arxiv.org/html/2610.12741#S1.p1.1)\.
- \[23\]T\. Poggio, Q\. Liao, and A\. Banburski\(2020\)Complexity control by gradient descent in deep networks\.Nature communications11\(1\),pp\. 1027\.Cited by:[4th item](https://arxiv.org/html/2610.12741#S3.I1.i4.p1.1)\.
- \[24\]V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf\(2019\)DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter\.arXiv preprint arXiv:1910\.01108\.Cited by:[2nd item](https://arxiv.org/html/2610.12741#S4.I1.i2.p1.1)\.
- \[25\]N\. Shazeer, A\. Mirhoseini, K\. Maziarz, A\. Davis, Q\. Le, G\. Hinton, and J\. Dean\(2017\)Outrageously large neural networks: the sparsely\-gated mixture\-of\-experts layer\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=B1ckMDqlg)Cited by:[§2](https://arxiv.org/html/2610.12741#S2.SS0.SSS0.Px3.p1.1)\.
- \[26\]A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. Polosukhin\(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\. 5998–6008\.Cited by:[§1](https://arxiv.org/html/2610.12741#S1.p1.1)\.
- \[27\]P\. Veličković, G\. Cucurull, A\. Casanova, A\. Romero, P\. Liò, and Y\. Bengio\(2018\)Graph attention networks\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=rJXMpikCZ)Cited by:[§1](https://arxiv.org/html/2610.12741#S1.p2.1),[§2](https://arxiv.org/html/2610.12741#S2.SS0.SSS0.Px2.p1.1)\.
- \[28\]S\. M\. Xie, A\. Raghunathan, P\. Liang, and T\. Ma\(2022\)An explanation of in\-context learning as implicit Bayesian inference\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=RdJVFCHjUMI)Cited by:[§1](https://arxiv.org/html/2610.12741#S1.p2.1)\.
- \[29\]J\. S\. Yedidia, W\. T\. Freeman, and Y\. Weiss\(2003\)Understanding belief propagation and its generalizations\.InExploring Artificial Intelligence in the New Millennium,G\. Lakemeyer and B\. Nebel \(Eds\.\),pp\. 239–269\.Cited by:[§1](https://arxiv.org/html/2610.12741#S1.p3.1)\.
- \[30\]C\. Zhang, S\. Bengio, M\. Hardt, B\. Recht, and O\. Vinyals\(2021\)Understanding deep learning \(still\) requires rethinking generalization\.Communications of the ACM64\(3\),pp\. 107–115\.Cited by:[4th item](https://arxiv.org/html/2610.12741#S3.I1.i4.p1.1)\.
- \[31\]Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin,et al\.\(2025\)Qwen3 embedding: advancing text embedding and reranking through foundation models\.arXiv preprint arXiv:2506\.05176\.Cited by:[4th item](https://arxiv.org/html/2610.12741#S4.I1.i4.p1.1)\.
- \[32\]Z\. Zhang, M\. H\. Dupty, F\. Wu, J\. Q\. Shi, and W\. S\. Lee\(2023\)Factor graph neural networks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2308.00887)Cited by:[§2](https://arxiv.org/html/2610.12741#S2.SS0.SSS0.Px2.p1.1)\.
- \[33\]G\. Zhou, W\. Lehrach, A\. Dedieu, M\. Lázaro\-Gredilla, and D\. George\(2022\)Graphical models with attention for context\-specific independence and an application to perceptual grouping\.InInternational Conference on Artificial Intelligence and Statistics \(AISTATS\),External Links:[Link](https://arxiv.org/abs/2112.03371)Cited by:[§2](https://arxiv.org/html/2610.12741#S2.SS0.SSS0.Px2.p1.1)\.