Three Costs of Amortizing Gaussian Process Inference with Neural Processes

# 1 Introduction
Source: [https://arxiv.org/html/2605.21798](https://arxiv.org/html/2605.21798)
Gaussian processes\(rasmussen2005gaussian\)define exact posterior distributions over functions given observed data\. The posterior mean and variance are available in closed form, but computing them costsO​\(n3\)O\(n^\{3\}\)in the number of context pointsnn111Specifically,O​\(n3\)O\(n^\{3\}\)for the one\-time Cholesky factorization ofK\+σϵ2​IK\+\\sigma\_\{\\epsilon\}^\{2\}I, subsequent predictions costO​\(n2\)O\(n^\{2\}\)per query\., limiting scalability\.

Neural processes\(garnelo2018cnp;garnelo2018neuralprocesses\)address this by learning an amortized map from context sets to predictive distributions\. Subsequent work has extended the framework through attention mechanisms\(kim2018attentive\), convolutional structure\(Gordon2020Convolutional\), and autoregressive factorizations\(bruinsma2023autoregressive\)\. A latent neural process \(LNP\) maps each context pair through an encoder, aggregates the resulting features into a finite\-dimensional representation, passes this representation through a recognition network producing a latent distribution, and decodes samples from the latent into a predictive distribution, all inO​\(n\)O\(n\)time for fixed architecture\. When trained on samples from a GP prior, the LNP is implicitly approximating the GP posterior\.

Sparse variational methods\(titsias2009sparse\)offer one path to scaling GP inference, reducing the cubic cost toO​\(M2​n\)O\(M^\{2\}n\)by summarizing the posterior throughMMinducing points\. However, the resulting predictive still requiresO​\(M2\)O\(M^\{2\}\)computation at test time, and the inducing locations must be optimized jointly with kernel hyperparameters for each new dataset\. In settings that demand predictions across many related tasks or under real\-time constraints such as sequential experimental design\(shahriari2016taking\), robotics\(deisenroth2015gaussian\), or simulation\-based inference\(cranmer2020frontier\)this per\-task optimization becomes a bottleneck of its own\. Neural processes eliminate per\-task optimization entirely with a single forward pass mapping any context set to a predictive distribution inO​\(n\)O\(n\)time, with no matrix inversion and no task\-specific parameters\. This amortization is the source of their practical appeal, but it introduces approximation costs that have no counterpart in the sparse GP framework\.

It is known that this approximation introduces error as the variational inference underlying the LNP tends to underestimate uncertainty, and the finite\-dimensional bottleneck cannot capture arbitrary context geometries\. However, no quantitative characterization of these errors exists\.foong2020metaidentify qualitative limitations of mean aggregation, showing it leads to underfitting, but do not provide rates\. People choose representation dimensions, encoder architectures, and variance parameterizations without formal guidance on the resulting approximation quality\. In the sparse GP literature,burt2020convergencebound the KL divergence between sparse variational and exact GP posteriors, showing thatM=O​\(\(log⁡N\)dx\)M=O\(\(\\log N\)^\{d\_\{x\}\}\)inducing points suffice for squared\-exponential kernels\. Our bottleneck analysis plays an analogous role for neural processes, with the representation dimensionddreplacing the number of inducing pointsMM\.

We provide such a characterization\. Our main contributions are: \(i\) a decomposition of the predictive KL divergenceKL​\(pGP∥pLNP\)\\mathrm\{KL\}\(p\_\{\\mathrm\{GP\}\}\\\|p\_\{\\mathrm\{LNP\}\}\)into three terms with distinct architectural and statistical origins; \(ii\) an upper bound on the truncation component of the bottleneck term that decays exponentially inddfor SE kernels and polynomially for Matérn kernels, connecting representation dimension to kernel smoothness; \(iii\) characterization of the label contamination term asO​\(1\)\+O​\(1/n\)O\(1\)\+O\(1/n\), identifying a structural mismatch in how neural processes estimate predictive variance; and \(iv\) two architectural recommendations to predict variance from locations alone in the GP\-amortization regime, and use second\-order rather than mean aggregation\.

## 2Setup

### 2\.1Gaussian Process

Letf∼𝒢​𝒫​\(0,k\)f\\sim\\mathcal\{GP\}\(0,k\)be a zero\-mean GP with kernelk:𝒳×𝒳→ℝk:\\mathcal\{X\}\\times\\mathcal\{X\}\\to\\mathbb\{R\}on a compact domain𝒳⊂ℝdx\\mathcal\{X\}\\subset\\mathbb\{R\}^\{d\_\{x\}\}\. Given contextC=\{\(xi,yi\)\}i=1nC=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}withyi=f​\(xi\)\+ϵiy\_\{i\}=f\(x\_\{i\}\)\+\\epsilon\_\{i\},ϵi∼𝒩​\(0,σϵ2\)\\epsilon\_\{i\}\\sim\\mathcal\{N\}\(0,\\sigma\_\{\\epsilon\}^\{2\}\), the GP predictive at a targetx∗x\_\{\*\}is:

pGP​\(y∗∣x∗,C\)=𝒩​\(μGP,σGP2\)p\_\{\\mathrm\{GP\}\}\(y\_\{\*\}\\mid x\_\{\*\},C\)=\\mathcal\{N\}\(\\mu\_\{\\mathrm\{GP\}\},\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\)\(1\)where

μGP\\displaystyle\\mu\_\{\\mathrm\{GP\}\}=𝒌∗⊤​\(K\+σϵ2​I\)−1​𝒚,𝒚=\(y1,…,yn\)⊤\\displaystyle=\\bm\{k\}\_\{\*\}^\{\\top\}\(K\+\\sigma\_\{\\epsilon\}^\{2\}I\)^\{\-1\}\\bm\{y\},\\quad\\bm\{y\}=\(y\_\{1\},\\dots,y\_\{n\}\)^\{\\top\}\(2\)σGP2\\displaystyle\\sigma^\{2\}\_\{\\mathrm\{GP\}\}=k​\(x∗,x∗\)−𝒌∗⊤​\(K\+σϵ2​I\)−1​𝒌∗,\\displaystyle=k\(x\_\{\*\},x\_\{\*\}\)\-\\bm\{k\}\_\{\*\}^\{\\top\}\(K\+\\sigma\_\{\\epsilon\}^\{2\}I\)^\{\-1\}\\bm\{k\}\_\{\*\},\(3\)with\[𝒌∗\]i=k​\(x∗,xi\)\[\\bm\{k\}\_\{\*\}\]\_\{i\}=k\(x\_\{\*\},x\_\{i\}\)and\[K\]i​j=k​\(xi,xj\)\[K\]\_\{ij\}=k\(x\_\{i\},x\_\{j\}\)\.

A structural property is that the GP predictive variance \([3](https://arxiv.org/html/2605.21798#S2.E3)\) depends on the context locationsX=\{xi\}X=\\\{x\_\{i\}\\\}but not on the labels𝒚\\bm\{y\}\.

### 2\.2Latent Neural Process

###### Definition 1\(Latent Neural Process\)\.

A latent neural process with mean aggregation consists of:

1. 1\.An encoderh:𝒳×𝒴→ℝdh:\\mathcal\{X\}\\times\\mathcal\{Y\}\\to\\mathbb\{R\}^\{d\}mapping context pairs to representations\.
2. 2\.Mean aggregation:rC=1n​∑i=1nh​\(xi,yi\)r\_\{C\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}h\(x\_\{i\},y\_\{i\}\)\.
3. 3\.A latent encoder:q​\(z∣C\)=𝒩​\(μz​\(rC\),Σz​\(rC\)\)q\(z\\mid C\)=\\mathcal\{N\}\(\\mu\_\{z\}\(r\_\{C\}\),\\Sigma\_\{z\}\(r\_\{C\}\)\)withz∈ℝdzz\\in\\mathbb\{R\}^\{d\_\{z\}\}\.
4. 4\.A decoder:p​\(y∗∣x∗,z\)=𝒩​\(w​\(x∗\)⊤​z\+b​\(x∗\),σd2\)p\(y\_\{\*\}\\mid x\_\{\*\},z\)=\\mathcal\{N\}\(w\(x\_\{\*\}\)^\{\\top\}z\+b\(x\_\{\*\}\),\\sigma\_\{d\}^\{2\}\)\.

The mapsμz:ℝd→ℝdz\\mu\_\{z\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\_\{z\}\}andΣz:ℝd→ℝ≻0dz×dz\\Sigma\_\{z\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\_\{z\}\\times d\_\{z\}\}\_\{\\succ 0\}are typically MLPs with the covariance output parameterized via a Cholesky factor to enforce positive definiteness\. The functionsw:𝒳→ℝdzw:\\mathcal\{X\}\\to\\mathbb\{R\}^\{d\_\{z\}\}andb:𝒳→ℝb:\\mathcal\{X\}\\to\\mathbb\{R\}are learned \(typically MLP\-parameterized\) functions of the target location, not network weights and biases\. The linear\-in\-zzdecoder is a tractability choice that yields a closed\-form Gaussian posteriorp​\(z∣C\)p\(z\\mid C\)amenable to the amortization\-gap analysis of[Section3\.3](https://arxiv.org/html/2605.21798#S3.SS3); standard NPs use nonlinear MLP decoders\. See[AppendixA](https://arxiv.org/html/2605.21798#A1)for a fuller introduction to NPs in case they are unfamiliar\.

The marginal predictive is Gaussian:

pLNP​\(y∗∣x∗,C\)\\displaystyle p\_\{\\mathrm\{LNP\}\}\(y\_\{\*\}\\mid x\_\{\*\},C\)=𝒩​\(μLNP,σLNP2\),\\displaystyle=\\mathcal\{N\}\(\\mu\_\{\\mathrm\{LNP\}\},\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\),\(4\)μLNP\\displaystyle\\mu\_\{\\mathrm\{LNP\}\}=w​\(x∗\)⊤​μz​\(rC\)\+b​\(x∗\),\\displaystyle=w\(x\_\{\*\}\)^\{\\top\}\\mu\_\{z\}\(r\_\{C\}\)\+b\(x\_\{\*\}\),\(5\)σLNP2\\displaystyle\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}=w​\(x∗\)⊤​Σz​\(rC\)​w​\(x∗\)\+σd2\.\\displaystyle=w\(x\_\{\*\}\)^\{\\top\}\\Sigma\_\{z\}\(r\_\{C\}\)\\,w\(x\_\{\*\}\)\+\\sigma\_\{d\}^\{2\}\.\(6\)
Unlike \([3](https://arxiv.org/html/2605.21798#S2.E3)\), the LNP variance \([6](https://arxiv.org/html/2605.21798#S2.E6)\) depends onrCr\_\{C\}, which encodes both context locations and labels\.

### 2\.3Predictive KL Divergence

Both predictives are Gaussian, so the KL divergence has a closed form:

KL​\(pGP∥pLNP\)=12​\[σGP2σLNP2−1\+\(μGP−μLNP\)2σLNP2\+log⁡σLNP2σGP2\]\\mathrm\{KL\}\(p\_\{\\mathrm\{GP\}\}\\\|p\_\{\\mathrm\{LNP\}\}\)=\\frac\{1\}\{2\}\\left\[\\frac\{\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\}\{\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\}\-1\+\\frac\{\(\\mu\_\{\\mathrm\{GP\}\}\-\\mu\_\{\\mathrm\{LNP\}\}\)^\{2\}\}\{\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\}\+\\log\\frac\{\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\}\{\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\}\\right\]\(7\)
We bound the expected gap𝔼C​\[KL​\(pGP∥pLNP\)\]\\mathbb\{E\}\_\{C\}\[\\mathrm\{KL\}\(p\_\{\\mathrm\{GP\}\}\\\|p\_\{\\mathrm\{LNP\}\}\)\]where the expectation is over contexts drawn from the GP prior\.

###### Assumption 1\.

There exist constants0<σℓ2≤σu2<∞0<\\sigma\_\{\\ell\}^\{2\}\\leq\\sigma\_\{u\}^\{2\}<\\inftysuch thatσℓ2≤σLNP2​\(x∗;C\)≤σu2\\sigma\_\{\\ell\}^\{2\}\\leq\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\(x\_\{\*\};C\)\\leq\\sigma\_\{u\}^\{2\}andσℓ2≤σGP2​\(x∗;X\)≤σu2\\sigma\_\{\\ell\}^\{2\}\\leq\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\(x\_\{\*\};X\)\\leq\\sigma\_\{u\}^\{2\}for all contextsCC, locationsXX, and targetsx∗∈𝒳x\_\{\*\}\\in\\mathcal\{X\}\.

## 3Decomposition of the Variance Error

The variance gap is the primary source of the variational inference \(VI\) bias\. We decompose it into three sources\. Define the variance error asΔ​σ2=σLNP2−σGP2\\Delta\\sigma^\{2\}=\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\-\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\.

### 3\.1Label Contamination

The GP varianceσGP2​\(x∗;X\)\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\(x\_\{\*\};X\)is a function of context locations only\. The LNP varianceσLNP2​\(x∗;rC\)\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\(x\_\{\*\};r\_\{C\}\)depends on labels throughrCr\_\{C\}\. This mismatch introduces noise in the variance estimate\.

###### Assumption 2\.

The encoder is affine inyy:h​\(x,y\)=ϕ​\(x\)\+ψ​\(x\)​yh\(x,y\)=\\phi\(x\)\+\\psi\(x\)y, with positive constantsBϕ,BψB\_\{\\phi\},B\_\{\\psi\}such that‖ϕ​\(x\)‖≤Bϕ\\\|\\phi\(x\)\\\|\\leq B\_\{\\phi\}and‖ψ​\(x\)‖≤Bψ\\\|\\psi\(x\)\\\|\\leq B\_\{\\psi\}for allx∈𝒳x\\in\\mathcal\{X\}\.

Under this assumption, the representation decomposes as:

rC=1n​∑i=1nϕ​\(xi\)⏟ϕ¯X\+1n​∑i=1nψ​\(xi\)​yi⏟δy\.r\_\{C\}=\\underbrace\{\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\phi\(x\_\{i\}\)\}\_\{\\bar\{\\phi\}\_\{X\}\}\+\\underbrace\{\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\psi\(x\_\{i\}\)y\_\{i\}\}\_\{\\delta\_\{y\}\}\.\(8\)
The first term depends only on locations; the second introduces label dependence\.

###### Assumption 3\.

The mapsμz:ℝd→ℝdz\\mu\_\{z\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\_\{z\}\}andΣz:ℝd→ℝdz×dz\\Sigma\_\{z\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\_\{z\}\\times d\_\{z\}\}areLμL\_\{\\mu\}\-Lipschitz andLΣL\_\{\\Sigma\}\-Lipschitz respectively\. The decoder satisfies‖w​\(x∗\)‖≤Bw\\\|w\(x\_\{\*\}\)\\\|\\leq B\_\{w\}\.

###### Theorem 1\(Label Contamination Bound\)\.

Under Assumptions[2](https://arxiv.org/html/2605.21798#Thmassumption2)and[3](https://arxiv.org/html/2605.21798#Thmassumption3), for contexts drawn from a GP prior with kernelkkand noise varianceσϵ2\\sigma\_\{\\epsilon\}^\{2\}:

𝔼C​\[Var𝒚\|X​\[σLNP2​\(x∗;C\)\]\]≤LΣ2​Bw4​Bψ2​\(σϵ2n\+κk\)\\mathbb\{E\}\_\{C\}\\\!\\left\[\\mathrm\{Var\}\_\{\\bm\{y\}\|X\}\\\!\\left\[\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\(x\_\{\*\};C\)\\right\]\\right\]\\leq L\_\{\\Sigma\}^\{2\}B\_\{w\}^\{4\}B\_\{\\psi\}^\{2\}\\\!\\left\(\\frac\{\\sigma\_\{\\epsilon\}^\{2\}\}\{n\}\+\\kappa\_\{k\}\\right\)\(9\)whereκk=supx∈𝒳k​\(x,x\)\\kappa\_\{k\}=\\sup\_\{x\\in\\mathcal\{X\}\}k\(x,x\)bounds the signal variance\. The first term arises from observation noise and decays asO​\(1/n\)O\(1/n\); the second arises from correlations under the GP prior and isO​\(1\)O\(1\)\.

###### Proof sketch\.

Decompose the label\-dependent component asδy=δf\+δϵ\\delta\_\{y\}=\\delta\_\{f\}\+\\delta\_\{\\epsilon\}whereδf=1n​∑iψ​\(xi\)​f​\(xi\)\\delta\_\{f\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\psi\(x\_\{i\}\)f\(x\_\{i\}\)andδϵ=1n​∑iψ​\(xi\)​ϵi\\delta\_\{\\epsilon\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\psi\(x\_\{i\}\)\\epsilon\_\{i\}\. The noise term is an i\.i\.d\. sum, so𝔼​\[‖δϵ‖2∣X\]≤Bψ2​σϵ2/n\\mathbb\{E\}\[\\\|\\delta\_\{\\epsilon\}\\\|^\{2\}\\mid X\]\\leq B\_\{\\psi\}^\{2\}\\sigma\_\{\\epsilon\}^\{2\}/n\. The signal term has covariance1n2​∑i,jψ​\(xi\)​ψ​\(xj\)⊤​k​\(xi,xj\)\\frac\{1\}\{n^\{2\}\}\\sum\_\{i,j\}\\psi\(x\_\{i\}\)\\psi\(x\_\{j\}\)^\{\\top\}k\(x\_\{i\},x\_\{j\}\)\. Sincekkis positive definite with bounded diagonalκk\\kappa\_\{k\}, the row\-sum bound gives𝔼​\[‖δf‖2∣X\]≤Bψ2​κk\\mathbb\{E\}\[\\\|\\delta\_\{f\}\\\|^\{2\}\\mid X\]\\leq B\_\{\\psi\}^\{2\}\\kappa\_\{k\}, which isO​\(1\)O\(1\)because the GP\-correlatedf​\(xi\)f\(x\_\{i\}\)do not concentrate\. Lipschitz continuity ofΣz\\Sigma\_\{z\}then transfers these bounds toσLNP2\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\. Full proof in[SectionB\.1](https://arxiv.org/html/2605.21798#A2.SS1)\. ∎

### 3\.2Information Bottleneck

Even removing label dependence \(conditioning on𝒚\\bm\{y\}or takingϕ¯X\\bar\{\\phi\}\_\{X\}as the representation\), thedd\-dimensional summaryϕ¯X\\bar\{\\phi\}\_\{X\}cannot capture the full context geometry that determines the GP variance\.222The truncation component of this bottleneck error follows from standard Mercer approximation theory\. We include it as the baseline against which the additional NP\-specific costs \(label contamination, amortization gap\) are measured\. These additional costs have no analogue in sparse variational GP theory\.

###### Definition 2\(Mercer Expansion\)\.

Let\{\(λj,ej\)\}j=1∞\\\{\(\\lambda\_\{j\},e\_\{j\}\)\\\}\_\{j=1\}^\{\\infty\}be the eigendecomposition of the kernel integral operatorTkT\_\{k\}with respect to a base measureμ\\muon𝒳\\mathcal\{X\}, taken throughout to coincide with the distribution from which context locations are drawn:

k​\(x,x′\)=∑j=1∞λj​ej​\(x\)​ej​\(x′\)\.k\(x,x^\{\\prime\}\)=\\sum\_\{j=1\}^\{\\infty\}\\lambda\_\{j\}\\,e\_\{j\}\(x\)\\,e\_\{j\}\(x^\{\\prime\}\)\.\(10\)

###### Assumption 4\.

The location encoder uses the firstddkernel eigenfunctionsϕ​\(x\)=\(λ1​e1​\(x\),…,λd​ed​\(x\)\)⊤\\phi\(x\)=\(\\sqrt\{\\lambda\_\{1\}\}\\,e\_\{1\}\(x\),\\dots,\\sqrt\{\\lambda\_\{d\}\}\\,e\_\{d\}\(x\)\)^\{\\top\}\.

This is the optimaldd\-dimensional encoder in the sense of minimizing integrated squared error in kernel approximation\. The mean representation becomesϕ¯X=1n​∑iϕ​\(xi\)\\bar\{\\phi\}\_\{X\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\phi\(x\_\{i\}\), which captures the projection of the empirical context measure onto the firstddkernel eigenfunctions\.

###### Assumption 5\.

The representation dimensionddis large enough that the tail operator norm satisfies

η​\(X\)=‖A−1/2​KT​A−1/2‖op<1,\\eta\(X\)\\;=\\;\\bigl\\\|A^\{\-1/2\}\\,K\_\{T\}\\,A^\{\-1/2\}\\bigr\\\|\_\{\\mathrm\{op\}\}\\;<\\;1,\(11\)whereA=KH\+σϵ2​IA=K\_\{H\}\+\\sigma\_\{\\epsilon\}^\{2\}Iwith head kernel matrix\[KH\]i​l=∑j≤dλj​ej​\(xi\)​ej​\(xl\)\[K\_\{H\}\]\_\{il\}=\\sum\_\{j\\leq d\}\\lambda\_\{j\}\\,e\_\{j\}\(x\_\{i\}\)\\,e\_\{j\}\(x\_\{l\}\), and tail kernel matrix\[KT\]i​l=∑j\>dλj​ej​\(xi\)​ej​\(xl\)\[K\_\{T\}\]\_\{il\}=\\sum\_\{j\>d\}\\lambda\_\{j\}\\,e\_\{j\}\(x\_\{i\}\)\\,e\_\{j\}\(x\_\{l\}\)\.

###### Theorem 2\(Information Bottleneck Bound\)\.

Under Assumptions[4](https://arxiv.org/html/2605.21798#Thmassumption4)and[5](https://arxiv.org/html/2605.21798#Thmassumption5), for a targetx∗x\_\{\*\}and context locationsX=\{xi\}i=1nX=\\\{x\_\{i\}\\\}\_\{i=1\}^\{n\}drawn i\.i\.d\. fromμ\\mu:

𝔼X​\[\(σGP2​\(x∗;X\)−gd∗​\(ϕ¯X\)\)2\]≤Rd​\(k,n\)\+Sd​\(k,x∗\),\\mathbb\{E\}\_\{X\}\\\!\\left\[\\left\(\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\(x\_\{\*\};X\)\-g\_\{d\}^\{\*\}\(\\bar\{\\phi\}\_\{X\}\)\\right\)^\{2\}\\right\]\\leq R\_\{d\}\(k,n\)\+S\_\{d\}\(k,x\_\{\*\}\),\(12\)wheregd∗g\_\{d\}^\{\*\}is the optimal estimator ofσGP2\\sigma^\{2\}\_\{\\mathrm\{GP\}\}fromϕ¯X\\bar\{\\phi\}\_\{X\}, and:

Rd​\(k,n\)\\displaystyle R\_\{d\}\(k,n\)=O​\(d2n\)\+Rdinfo,\\displaystyle=O\\\!\\left\(\\frac\{d^\{2\}\}\{n\}\\right\)\+R\_\{d\}^\{\\mathrm\{info\}\},\(13\)Sd​\(k,x∗\)\\displaystyle S\_\{d\}\(k,x\_\{\*\}\)≤CS​∑j\>dλj​ej​\(x∗\)2,\\displaystyle\\leq C\_\{S\}\\sum\_\{j\>d\}\\lambda\_\{j\}\\,e\_\{j\}\(x\_\{\*\}\)^\{2\},\(14\)withCS=C​κk/σϵ4C\_\{S\}=C\\,\\kappa\_\{k\}/\\sigma\_\{\\epsilon\}^\{4\}for a constantCCdepending only onκk/σϵ2\\kappa\_\{k\}/\\sigma\_\{\\epsilon\}^\{2\}\. HereRdinfoR\_\{d\}^\{\\mathrm\{info\}\}is the irreducible estimation loss fromϕ¯X\\bar\{\\phi\}\_\{X\}being an insufficient statistic for the head kernel matrix, and theO​\(d2/n\)O\(d^\{2\}/n\)term is the variance ofϕ¯X\\bar\{\\phi\}\_\{X\}about its population mean\. For second\-order aggregation \(using1n​∑iϕ​\(xi\)​ϕ​\(xi\)⊤\\frac\{1\}\{n\}\\sum\_\{i\}\\phi\(x\_\{i\}\)\\phi\(x\_\{i\}\)^\{\\top\}in place ofϕ¯X\\bar\{\\phi\}\_\{X\}\),Rdinfo=0R\_\{d\}^\{\\mathrm\{info\}\}=0and the estimation error isO​\(d2/n\)O\(d^\{2\}/n\)\.

###### Proof sketch\.

Split the Mercer series into head \(j≤dj\\leq d\) and tail \(j\>dj\>d\) components, withA=KH\+σϵ2​IA=K\_\{H\}\+\\sigma\_\{\\epsilon\}^\{2\}IandM=A\+KTM=A\+K\_\{T\}\. The truncated GP varianceσd2=kH−\(𝒌∗H\)⊤​A−1​𝒌∗H\\sigma\_\{d\}^\{2\}=k\_\{H\}\-\(\\bm\{k\}\_\{\*\}^\{H\}\)^\{\\top\}A^\{\-1\}\\bm\{k\}\_\{\*\}^\{H\}depends only on head eigenfunction evaluations\. Since the optimal estimatorgd∗​\(ϕ¯X\)g\_\{d\}^\{\*\}\(\\bar\{\\phi\}\_\{X\}\)minimizes MSE over measurable functions ofϕ¯X\\bar\{\\phi\}\_\{X\}, andσd2\\sigma\_\{d\}^\{2\}is measurable with respect to a finerσ\\sigma\-algebra,SdS\_\{d\}is bounded by𝔼X​\[\(σGP2−σd2\)2\]\\mathbb\{E\}\_\{X\}\[\(\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\-\\sigma\_\{d\}^\{2\}\)^\{2\}\]\. A Neumann expansion ofM−1M^\{\-1\}aroundA−1A^\{\-1\}, valid under[5](https://arxiv.org/html/2605.21798#Thmassumption5), gives\|σGP2−σd2\|=O​\(kT\)\|\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\-\\sigma\_\{d\}^\{2\}\|=O\(\\sqrt\{k\_\{T\}\}\), hence the squared error isO​\(kT\)O\(k\_\{T\}\)with constantCS=C​κk/σϵ4C\_\{S\}=C\\kappa\_\{k\}/\\sigma\_\{\\epsilon\}^\{4\}\. The estimation termRdR\_\{d\}contains anO​\(d2/n\)O\(d^\{2\}/n\)CLT contribution fromϕ¯X\\bar\{\\phi\}\_\{X\}converging to its population mean, plus the irreducibleRdinfoR\_\{d\}^\{\\mathrm\{info\}\}from mean aggregation being insufficient for the head kernel matrix; under second\-order aggregationRdinfo=0R\_\{d\}^\{\\mathrm\{info\}\}=0\. Full proof in[SectionB\.2](https://arxiv.org/html/2605.21798#A2.SS2)\. ∎

The truncation errorSdS\_\{d\}inherits the eigenvalue decay of the kernel:

###### Corollary 1\(Kernel\-Dependent Bottleneck Rates\)\.

Under Assumptions[4](https://arxiv.org/html/2605.21798#Thmassumption4)and[5](https://arxiv.org/html/2605.21798#Thmassumption5):

1. 1\.Squared\-exponential kernelon\[0,1\]dx\[0,1\]^\{d\_\{x\}\}: Eigenvalues satisfyλj≤CSE​e−c​j2/dx\\lambda\_\{j\}\\leq C\_\{\\mathrm\{SE\}\}\\,e^\{\-c\\,j^\{2/d\_\{x\}\}\}for constantsCSE,c\>0C\_\{\\mathrm\{SE\}\},c\>0depending on the lengthscaleℓ\\ell\. Thus: Sd≤CSE′​e−c′​d2/dx\.S\_\{d\}\\leq C^\{\\prime\}\_\{\\mathrm\{SE\}\}\\,e^\{\-c^\{\\prime\}\\,d^\{2/d\_\{x\}\}\}\.\(15\)
2. 2\.Matérn\-ν\\nukernelon\[0,1\]dx\[0,1\]^\{d\_\{x\}\}: Eigenvalues satisfyλj≤Cν​j−\(2​ν\+dx\)/dx\\lambda\_\{j\}\\leq C\_\{\\nu\}\\,j^\{\-\(2\\nu\+d\_\{x\}\)/d\_\{x\}\}\. Thus: Sd≤Cν′​d−2​ν/dx\.S\_\{d\}\\leq C^\{\\prime\}\_\{\\nu\}\\,d^\{\-2\\nu/d\_\{x\}\}\.\(16\)

The constantsCSE′C^\{\\prime\}\_\{\\mathrm\{SE\}\}andCν′C^\{\\prime\}\_\{\\nu\}absorb the factorCS=C​κk/σϵ4C\_\{S\}=C\\,\\kappa\_\{k\}/\\sigma\_\{\\epsilon\}^\{4\}from[Theorem2](https://arxiv.org/html/2605.21798#Thmtheorem2)\.

###### Proof\.

For the SE kernel, the tail sum∑j\>dλj​ej​\(x∗\)2≤‖e‖∞2​∑j\>dλj\\sum\_\{j\>d\}\\lambda\_\{j\}e\_\{j\}\(x\_\{\*\}\)^\{2\}\\leq\\\|e\\\|\_\{\\infty\}^\{2\}\\sum\_\{j\>d\}\\lambda\_\{j\}\. The eigenvalues of the SE kernel on compact domains decay asλj≤C​e−c​j2/dx\\lambda\_\{j\}\\leq C\\,e^\{\-c\\,j^\{2/d\_\{x\}\}\}\(SteinwartScovel2012\)\. Thus∑j\>dλj≤C​e−c​d2/dx\\sum\_\{j\>d\}\\lambda\_\{j\}\\leq C\\,e^\{\-c\\,d^\{2/d\_\{x\}\}\}\. Substituting into the linear truncation bound gives the result\.

For the Matérn kernel, the eigenvalue asymptoticsλj=Θ​\(j−\(2​ν\+dx\)/dx\)\\lambda\_\{j\}=\\Theta\(j^\{\-\(2\\nu\+d\_\{x\}\)/d\_\{x\}\}\)follow from the spectral theory of Sobolev\-equivalent reproducing kernels\(wendland2004scattered\)\. Then∑j\>dλj≤C​∫d∞t−\(2​ν\+dx\)/dx​𝑑t=C′​d−2​ν/dx\\sum\_\{j\>d\}\\lambda\_\{j\}\\leq C\\int\_\{d\}^\{\\infty\}t^\{\-\(2\\nu\+d\_\{x\}\)/d\_\{x\}\}\\,dt=C^\{\\prime\}\\,d^\{\-2\\nu/d\_\{x\}\}\. Substituting into the linear truncation bound givesSd=O​\(d−2​ν/dx\)S\_\{d\}=O\(d^\{\-2\\nu/d\_\{x\}\}\)\. ∎

### 3\.3Amortization Gap

The LNP uses a single encoder network\(μz​\(⋅\),Σz​\(⋅\)\)\(\\mu\_\{z\}\(\\cdot\),\\Sigma\_\{z\}\(\\cdot\)\)for all contexts\. For any specific contextCC, the optimal variational parameters may differ from what the amortized encoder produces\. The concept of an amortization gap was introduced in the VAE literature bycremer2018inference\.

###### Definition 3\(Amortization Gap\)\.

For a fixed contextCC, letq∗\(z∣C\)=argminq∈𝒬KL\(q\(z\)∥p\(z∣C\)\)q^\{\*\}\(z\\mid C\)=\\arg\\min\_\{q\\in\\mathcal\{Q\}\}\\mathrm\{KL\}\(q\(z\)\\\|p\(z\\mid C\)\)be the optimal per\-task variational distribution\. The amortization gap is:

A\(C\)=KL\(qamort\(z∣C\)∥p\(z∣C\)\)−KL\(q∗\(z∣C\)∥p\(z∣C\)\)A\(C\)=\\mathrm\{KL\}\(q\_\{\\mathrm\{amort\}\}\(z\\mid C\)\\\|p\(z\\mid C\)\)\-\\mathrm\{KL\}\(q^\{\*\}\(z\\mid C\)\\\|p\(z\\mid C\)\)\(17\)

Under the linear decoder \(item 4 of[Definition1](https://arxiv.org/html/2605.21798#Thmdefinition1)\) with priorp​\(z\)=𝒩​\(0,I\)p\(z\)=\\mathcal\{N\}\(0,I\), the posterior is Gaussian:

p​\(z∣C\)=𝒩​\(μp,Σp\),Σp−1=I\+σd−2​∑i=1nw​\(xi\)​w​\(xi\)⊤p\(z\\mid C\)=\\mathcal\{N\}\(\\mu\_\{p\},\\Sigma\_\{p\}\),\\quad\\Sigma\_\{p\}^\{\-1\}=I\+\\sigma\_\{d\}^\{\-2\}\\textstyle\\sum\_\{i=1\}^\{n\}w\(x\_\{i\}\)w\(x\_\{i\}\)^\{\\top\}\(18\)Since the variational family𝒬\\mathcal\{Q\}contains all Gaussians,q∗​\(z∣C\)=p​\(z∣C\)q^\{\*\}\(z\\mid C\)=p\(z\\mid C\), soA\(C\)=KL\(qamort\(z∣C\)∥p\(z∣C\)\)A\(C\)=\\mathrm\{KL\}\(q\_\{\\mathrm\{amort\}\}\(z\\mid C\)\\\|p\(z\\mid C\)\)\. Crucially, the posterior covarianceΣp\\Sigma\_\{p\}depends on the*second\-order*empirical features∑iw​\(xi\)​w​\(xi\)⊤\\sum\_\{i\}w\(x\_\{i\}\)w\(x\_\{i\}\)^\{\\top\}, but mean aggregation provides only the*first\-order*summaryϕ¯X=1n​∑iϕ​\(xi\)\\bar\{\\phi\}\_\{X\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\phi\(x\_\{i\}\)\. This mismatch is the source of an irreducible amortization gap\.

We make this precise in the scalar case and then state the general result\.

###### Proposition 1\(Amortization Gap for Mean Aggregation,dz=d=1d\_\{z\}=d=1\)\.

Letdz=d=1d\_\{z\}=d=1with the Mercer encoderϕ​\(x\)=λ1​e1​\(x\)\\phi\(x\)=\\sqrt\{\\lambda\_\{1\}\}\\,e\_\{1\}\(x\)and linear decoderw​\(x\)=ϕ​\(x\)w\(x\)=\\phi\(x\), wheree1e\_\{1\}is the first eigenfunction of the kernel integral operator with respect to a measureμ\\muon𝒳\\mathcal\{X\}\. Let context locationsx1,…,xn​i\.i\.d\.​μx\_\{1\},\\dots,x\_\{n\}\\text\{ i\.i\.d\. \}\\mu\. Defineα=n​λ1/σd2\\alpha=n\\lambda\_\{1\}/\\sigma\_\{d\}^\{2\}\. Then the variance component of the amortization gap satisfies:

𝔼X​\[KL​\(𝒩​\(0,Σz∗​\(ϕ¯X\)\)∥𝒩​\(0,Σp​\(X\)\)\)\]\\displaystyle\\mathbb\{E\}\_\{X\}\\\!\\left\[\\mathrm\{KL\}\\\!\\left\(\\mathcal\{N\}\(0,\\,\\Sigma\_\{z\}^\{\*\}\(\\bar\{\\phi\}\_\{X\}\)\)\\;\\big\\\|\\;\\mathcal\{N\}\(0,\\,\\Sigma\_\{p\}\(X\)\)\\right\)\\right\]\(19\)=α28​n​\(1\+α\)2​\(1\+O​\(1/n\)\)\.\\displaystyle=\\frac\{\\alpha^\{2\}\}\{8n\(1\+\\alpha\)^\{2\}\}\\left\(1\+O\(1/n\)\\right\)\.\(20\)whereΣz∗​\(ϕ¯X\)\\Sigma\_\{z\}^\{\*\}\(\\bar\{\\phi\}\_\{X\}\)is the optimal posterior covariance estimator given only the mean representation\. Asymptotically,𝔼X​\[Aσ​\(X\)\]→18​n\\mathbb\{E\}\_\{X\}\[A\_\{\\sigma\}\(X\)\]\\to\\frac\{1\}\{8n\}asn​λ1/σd2→∞n\\lambda\_\{1\}/\\sigma\_\{d\}^\{2\}\\to\\infty\.

###### Proof sketch\.

The posterior precision isΣp−1=1\+α​v^\\Sigma\_\{p\}^\{\-1\}=1\+\\alpha\\hat\{v\}wherev^=1n​∑ie1​\(xi\)2\\hat\{v\}=\\frac\{1\}\{n\}\\sum\_\{i\}e\_\{1\}\(x\_\{i\}\)^\{2\}\. The mean representatione¯=1n​∑ie1​\(xi\)\\bar\{e\}=\\frac\{1\}\{n\}\\sum\_\{i\}e\_\{1\}\(x\_\{i\}\)is uncorrelated withv^\\hat\{v\}becauseCov​\(e1​\(x\)2,e1​\(x\)\)=𝔼μ​\[e13\]−𝔼μ​\[e12\]​𝔼μ​\[e1\]=0\\mathrm\{Cov\}\(e\_\{1\}\(x\)^\{2\},e\_\{1\}\(x\)\)=\\mathbb\{E\}\_\{\\mu\}\[e\_\{1\}^\{3\}\]\-\\mathbb\{E\}\_\{\\mu\}\[e\_\{1\}^\{2\}\]\\mathbb\{E\}\_\{\\mu\}\[e\_\{1\}\]=0\(both terms vanish under standard symmetric eigenfunction bases\)\. So the Bayes\-optimal estimator ofΣp\\Sigma\_\{p\}givene¯\\bar\{e\}uses𝔼​\[v^\]=1\\mathbb\{E\}\[\\hat\{v\}\]=1, givingΣz∗=\(1\+α\)−1\\Sigma\_\{z\}^\{\*\}=\(1\+\\alpha\)^\{\-1\}\. A second\-order expansion of the resulting KL inv^−1=Op​\(n−1/2\)\\hat\{v\}\-1=O\_\{p\}\(n^\{\-1/2\}\)yields the rate\. Full proof in[SectionB\.3](https://arxiv.org/html/2605.21798#A2.SS3)\. ∎

The result extends to generaldd:

###### Proposition 2\(Amortization Gap, Generaldd\)\.

Under Assumptions[4](https://arxiv.org/html/2605.21798#Thmassumption4)and[1](https://arxiv.org/html/2605.21798#Thmassumption1), withdz=dd\_\{z\}=d, decoderw​\(x\)=ϕ​\(x\)w\(x\)=\\phi\(x\), and mean aggregation:

𝔼X​\[Aσ​\(X\)\]=1n​∑j=1dcj​\(λj,σd2,n\)\+O​\(d2/n2\),\\mathbb\{E\}\_\{X\}\[A\_\{\\sigma\}\(X\)\]=\\frac\{1\}\{n\}\\sum\_\{j=1\}^\{d\}c\_\{j\}\(\\lambda\_\{j\},\\sigma\_\{d\}^\{2\},n\)\+O\(d^\{2\}/n^\{2\}\),\(21\)where eachcj=Θ​\(1\)c\_\{j\}=\\Theta\(1\)for eigenvalues bounded away from zero, arising from the conditional varianceVar​\[v^j∣e¯\]\\mathrm\{Var\}\[\\hat\{v\}\_\{j\}\\mid\\bar\{e\}\]of thejj\-th diagonal second momentv^j=1n​∑iej​\(xi\)2\\hat\{v\}\_\{j\}=\\frac\{1\}\{n\}\\sum\_\{i\}e\_\{j\}\(x\_\{i\}\)^\{2\}given the mean representation\. For the Mercer encoder with decaying eigenvalues:

𝔼X​\[Aσ​\(X\)\]=Θ​\(deff/n\),\\mathbb\{E\}\_\{X\}\[A\_\{\\sigma\}\(X\)\]=\\Theta\(d\_\{\\mathrm\{eff\}\}/n\),\(22\)wheredeff=\(∑jλj2\)/\(∑jλj\)2d\_\{\\mathrm\{eff\}\}=\\bigl\(\\sum\_\{j\}\\lambda\_\{j\}^\{2\}\\bigr\)/\\bigl\(\\sum\_\{j\}\\lambda\_\{j\}\\bigr\)^\{2\}is the spectrum\-weighted effective dimension\.

###### Proof sketch\.

The posterior precision is

Σp−1\\displaystyle\\Sigma\_\{p\}^\{\-1\}=I\+nσd2​Λ​V^​Λ,\\displaystyle=I\+\\frac\{n\}\{\\sigma\_\{d\}^\{2\}\}\\,\\Lambda\\,\\hat\{V\}\\,\\Lambda,V^j​k\\displaystyle\\hat\{V\}\_\{jk\}=1n​∑iej​\(xi\)​ek​\(xi\),\\displaystyle=\\frac\{1\}\{n\}\\sum\_\{i\}e\_\{j\}\(x\_\{i\}\)\\,e\_\{k\}\(x\_\{i\}\),Λ\\displaystyle\\Lambda=diag⁡\(λ1,…,λd\)\.\\displaystyle=\\operatorname\{diag\}\(\\sqrt\{\\lambda\_\{1\}\},\\dots,\\sqrt\{\\lambda\_\{d\}\}\)\.\(23\)Mean aggregation providese¯j=1n​∑iej​\(xi\)\\bar\{e\}\_\{j\}=\\frac\{1\}\{n\}\\sum\_\{i\}e\_\{j\}\(x\_\{i\}\)forj=1,…,dj=1,\\dots,d, which does not determineV^\\hat\{V\}\.

#### General claim\.

The conditional varianceVar​\[v^j∣e¯\]\\mathrm\{Var\}\[\\hat\{v\}\_\{j\}\\mid\\bar\{e\}\]is bounded below byVar​\[v^j\]​\(1−ρj2\)\\mathrm\{Var\}\[\\hat\{v\}\_\{j\}\]\(1\-\\rho\_\{j\}^\{2\}\)whereρj\\rho\_\{j\}measures how wellv^j\\hat\{v\}\_\{j\}is predicted by linear functionals ofe¯\\bar\{e\}\. Under broad conditions on the eigenfunction basis \(orthogonality of degree\-2 monomials in the eigenfunctions to the linear span\{e1,…,ed\}\\\{e\_\{1\},\\dots,e\_\{d\}\\\}outside a sparse set of indices\),ρj\\rho\_\{j\}is bounded away from11, so eachjjcontributesΘ​\(1/n\)\\Theta\(1/n\)to the gap\. Weighting by the sensitivity∂Σp/∂v^j∝λj2\\partial\\Sigma\_\{p\}/\\partial\\hat\{v\}\_\{j\}\\propto\\lambda\_\{j\}^\{2\}gives the effective\-dimension scalingΘ​\(deff/n\)\\Theta\(d\_\{\\mathrm\{eff\}\}/n\)\.

#### Worked example cosine basis on\[0,1\]\[0,1\]\.

Forej​\(x\)=2​cos⁡\(j​π​x\)e\_\{j\}\(x\)=\\sqrt\{2\}\\cos\(j\\pi x\)under the uniform measure,

Covμ​\(ej​\(x\)2,eℓ​\(x\)\)=𝔼μ​\[ej2​eℓ\]=0\\mathrm\{Cov\}\_\{\\mu\}\(e\_\{j\}\(x\)^\{2\},e\_\{\\ell\}\(x\)\)=\\mathbb\{E\}\_\{\\mu\}\[e\_\{j\}^\{2\}e\_\{\\ell\}\]=0\(24\)unlessℓ=2​j\\ell=2j\(the product\-to\-sum selection rule\)\. When2​j\>d2j\>d,v^j⟂e¯\\hat\{v\}\_\{j\}\\perp\\bar\{e\}completely; for2​j≤d2j\\leq d, partial information is captured throughe¯2​j\\bar\{e\}\_\{2j\}alone, soρj\\rho\_\{j\}remains bounded away from11\. This makes the general claim concrete in the canonical setting\.

The off\-diagonal termsV^j​k\\hat\{V\}\_\{jk\}forj≠kj\\neq kcontribute additionalO​\(d2/n\)O\(d^\{2\}/n\)but, weighted byλj​λk\\lambda\_\{j\}\\lambda\_\{k\}which decays, are absorbed intodeff/nd\_\{\\mathrm\{eff\}\}/n\. ∎

## 4Combined Upper Bound

The three error sources of[Section3](https://arxiv.org/html/2605.21798#S3)combine into an upper bound on the predictive KL divergence\. We state the bound as a structural decomposition where each term traces to a specific architectural mechanism\.

###### Theorem 3\(Decomposition of the predictive KL\)\.

Under Assumptions[1](https://arxiv.org/html/2605.21798#Thmassumption1)–[5](https://arxiv.org/html/2605.21798#Thmassumption5), for a latent neural process with mean aggregation, representation dimensiondd, and Mercer\-feature encoder, trained on a GP with kernelkkand observation noiseσϵ2\\sigma\_\{\\epsilon\}^\{2\}:

𝔼C​\[KL​\(pGP∥pLNP\)\]\\displaystyle\\mathbb\{E\}\_\{C\}\\\!\\left\[\\mathrm\{KL\}\(p\_\{\\mathrm\{GP\}\}\\\|p\_\{\\mathrm\{LNP\}\}\)\\right\]≤32​σℓ4\(Λ2​κk⏟label, signal\+Λ2​σϵ2n⏟label, noise\\displaystyle\\quad\\leq\\frac\{3\}\{2\\sigma\_\{\\ell\}^\{4\}\}\\Bigl\(\\underbrace\{\\Lambda^\{2\}\\kappa\_\{k\}\}\_\{\\text\{label, signal\}\}\+\\underbrace\{\\tfrac\{\\Lambda^\{2\}\\sigma\_\{\\epsilon\}^\{2\}\}\{n\}\}\_\{\\text\{label, noise\}\}\+CS​τd​\(k,x∗\)⏟bottleneck\+Rd​\(k,n\)⏟estimation\)\\displaystyle\\qquad\\qquad\+\\underbrace\{C\_\{S\}\\,\\tau\_\{d\}\(k,x\_\{\*\}\)\}\_\{\\text\{bottleneck\}\}\+\\underbrace\{R\_\{d\}\(k,n\)\}\_\{\\text\{estimation\}\}\\Bigr\)\+𝒜​\(d,n\),\\displaystyle\\quad\\quad\+\\mathcal\{A\}\(d,n\),\(25\)whereΛ=LΣ​Bw2​Bψ\\Lambda=L\_\{\\Sigma\}B\_\{w\}^\{2\}B\_\{\\psi\}collects the architectural constants from Assumptions[2](https://arxiv.org/html/2605.21798#Thmassumption2)–[3](https://arxiv.org/html/2605.21798#Thmassumption3),CS=C​κk/σϵ4C\_\{S\}=C\\kappa\_\{k\}/\\sigma\_\{\\epsilon\}^\{4\}is the truncation constant from[Theorem2](https://arxiv.org/html/2605.21798#Thmtheorem2),τd​\(k,x∗\)=∑j\>dλj​ej​\(x∗\)2\\tau\_\{d\}\(k,x\_\{\*\}\)=\\sum\_\{j\>d\}\\lambda\_\{j\}e\_\{j\}\(x\_\{\*\}\)^\{2\}is the kernel tail at the target,Rd​\(k,n\)R\_\{d\}\(k,n\)is the estimation error fromϕ¯X\\bar\{\\phi\}\_\{X\}as defined in[Theorem2](https://arxiv.org/html/2605.21798#Thmtheorem2), and𝒜​\(d,n\)\\mathcal\{A\}\(d,n\)is the amortization contribution to the KL\.

###### Proof\.

The KL between two univariate Gaussians \([7](https://arxiv.org/html/2605.21798#S2.E7)\) satisfies

KL​\(pGP∥pLNP\)≤\(Δ​σ2\)22​σℓ4\+\(μGP−μLNP\)22​σℓ2,\\mathrm\{KL\}\(p\_\{\\mathrm\{GP\}\}\\\|p\_\{\\mathrm\{LNP\}\}\)\\leq\\frac\{\(\\Delta\\sigma^\{2\}\)^\{2\}\}\{2\\sigma\_\{\\ell\}^\{4\}\}\+\\frac\{\(\\mu\_\{\\mathrm\{GP\}\}\-\\mu\_\{\\mathrm\{LNP\}\}\)^\{2\}\}\{2\\sigma\_\{\\ell\}^\{2\}\},\(26\)usingσLNP2≥σℓ2\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\\geq\\sigma\_\{\\ell\}^\{2\}from[1](https://arxiv.org/html/2605.21798#Thmassumption1)and\|log⁡\(1\+x\)−x\|≤x2\|\\log\(1\+x\)\-x\|\\leq x^\{2\}for smallxx\. The variance error decomposes by the law of total variance over𝒚\\bm\{y\}givenXX:

𝔼​\[\(Δ​σ2\)2\]\\displaystyle\\mathbb\{E\}\[\(\\Delta\\sigma^\{2\}\)^\{2\}\]≤3​𝔼​\[Var𝒚\|X​\[σLNP2\]\]\\displaystyle\\leq 3\\mathbb\{E\}\\\!\\left\[\\mathrm\{Var\}\_\{\\bm\{y\}\|X\}\[\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\]\\right\]\+3​𝔼​\[\(σGP2​\(x∗;X\)−gd∗​\(ϕ¯X\)\)2\]\\displaystyle\\quad\+3\\mathbb\{E\}\\\!\\left\[\(\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\(x\_\{\*\};X\)\-g\_\{d\}^\{\*\}\(\\bar\{\\phi\}\_\{X\}\)\)^\{2\}\\right\]\+3​𝔼​\[\(gd∗​\(ϕ¯X\)−𝔼​\[σLNP2∣X\]\)2\]\.\\displaystyle\\quad\+3\\mathbb\{E\}\\\!\\left\[\(g\_\{d\}^\{\*\}\(\\bar\{\\phi\}\_\{X\}\)\-\\mathbb\{E\}\[\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\\mid X\]\)^\{2\}\\right\]\.\(27\)The first term is bounded by[Theorem1](https://arxiv.org/html/2605.21798#Thmtheorem1), contributingΛ2​\(κk\+σϵ2/n\)\\Lambda^\{2\}\(\\kappa\_\{k\}\+\\sigma\_\{\\epsilon\}^\{2\}/n\)\. The second is bounded by[Theorem2](https://arxiv.org/html/2605.21798#Thmtheorem2), contributingCS​τd​\(k,x∗\)\+Rd​\(k,n\)C\_\{S\}\\,\\tau\_\{d\}\(k,x\_\{\*\}\)\+R\_\{d\}\(k,n\)\. The third is the variance\-pathway amortization contribution, which under mean aggregation isΘ​\(deff/n\)\\Theta\(d\_\{\\mathrm\{eff\}\}/n\)by[Proposition2](https://arxiv.org/html/2605.21798#Thmproposition2)\. Combining and bounding the mean error similarly yields \([3](https://arxiv.org/html/2605.21798#S4.Ex3)\); the mean contributions are absorbed into𝒜​\(d,n\)\\mathcal\{A\}\(d,n\)\. ∎

### 4\.1Numerical illustration

We illustrate the architectural implications of the bound numerically with four findings summarized below\. The full methods, additional sample sizes, and the verification of[Proposition1](https://arxiv.org/html/2605.21798#Thmproposition1)’s closed\-form prediction \(accurate to within1%1\\%forn≥50n\\geq 50\) appear in[AppendixC](https://arxiv.org/html/2605.21798#A3)\. The amortization\-gap experiments use the SE kernel on\[0,1\]\[0,1\]withℓ=0\.3\\ell=0\.3and the Mercer encoder of[4](https://arxiv.org/html/2605.21798#Thmassumption4); the label contamination experiment uses a trained MLP\-based LNP described in[AppendixD](https://arxiv.org/html/2605.21798#A4)\.

#### Mean vs\. second\-order aggregation\.

Table[1](https://arxiv.org/html/2605.21798#S4.T1)compares the empirical amortization gap under mean aggregationrC=1n​∑iϕ​\(xi\)r\_\{C\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\phi\(x\_\{i\}\)against the second\-order alternative1n​∑iϕ​\(xi\)​ϕ​\(xi\)⊤\\frac\{1\}\{n\}\\sum\_\{i\}\\phi\(x\_\{i\}\)\\phi\(x\_\{i\}\)^\{\\top\}atd=3d=3\. Mean aggregation decays asn−0\.81n^\{\-0\.81\}, second\-order asn−1\.75n^\{\-1\.75\}, matching theΘ​\(1/n\)\\Theta\(1/n\)versusO​\(1/n2\)O\(1/n^\{2\}\)separation predicted by[Proposition2](https://arxiv.org/html/2605.21798#Thmproposition2)\. The ratio between the two grows linearly innn, so the architectural choice is increasingly consequential at the context sizes typical of NP applications\.

Table 1:Empirical amortization gap under mean vs\. second\-order aggregation \(d=3d=3, SE kernel\)\. Selected rows; full table in[AppendixC](https://arxiv.org/html/2605.21798#A3)\.
#### Effective dimension\.

The amortization gap under mean aggregation depends onddnon\-monotonically \(Table[2](https://arxiv.org/html/2605.21798#S4.T2)\): it grows fromd=1d=1tod=3d=3as more second\-order structure becomes available to miss, then declines as eigenvalue decay reduces the sensitivity ofΣp\\Sigma\_\{p\}to the unobserved features\. Byd=8d=8the two aggregation schemes converge, sinceλj\\lambda\_\{j\}forj\>3j\>3is small enough under the SE kernel \(ℓ=0\.3\\ell=0\.3\) that the corresponding second\-order terms contribute negligibly\. This tracks the spectrum\-weighted effective dimensiondeffd\_\{\\mathrm\{eff\}\}rather thandditself, consistent with theΘ​\(deff/n\)\\Theta\(d\_\{\\mathrm\{eff\}\}/n\)scaling of[Proposition2](https://arxiv.org/html/2605.21798#Thmproposition2)\. Under second\-order aggregation the gap remains uniformly small across allddtested\.

Table 2:Amortization gap vs\. representation dimension \(n=50n=50, SE kernel\)\.
#### The mechanism: identical representations, different posteriors\.

The mean\-aggregation pathology has a concrete instantiation\. Ford=1d=1on\[0,1\]\[0,1\]withe1​\(x\)=2​cos⁡\(π​x\)e\_\{1\}\(x\)=\\sqrt\{2\}\\cos\(\\pi x\), any pair\(x1,x2\)=\(a,1−a\)\(x\_\{1\},x\_\{2\}\)=\(a,1\-a\)yieldsϕ¯=0\\bar\{\\phi\}=0by symmetry\. Yet the posterior precisionΣp−1=1\+λ1σd2​\[e1​\(x1\)2\+e1​\(x2\)2\]\\Sigma\_\{p\}^\{\-1\}=1\+\\frac\{\\lambda\_\{1\}\}\{\\sigma\_\{d\}^\{2\}\}\[e\_\{1\}\(x\_\{1\}\)^\{2\}\+e\_\{1\}\(x\_\{2\}\)^\{2\}\]varies substantially across such pairs:\(0\.10,0\.90\)\(0\.10,0\.90\)givesΣp=0\.30\\Sigma\_\{p\}=0\.30\(informative spread\-out contexts\), while\(0\.45,0\.55\)\(0\.45,0\.55\)givesΣp=0\.94\\Sigma\_\{p\}=0\.94\(clustered, uninformative contexts\)\. The amortized encoder must produce a single output for all pairs sharingϕ¯=0\\bar\{\\phi\}=0, incurring an average KL of0\.100\.10between the extreme cases\. This is the dimension\-counting deficit of[Proposition2](https://arxiv.org/html/2605.21798#Thmproposition2)\(ddscalars summarizing ad​\(d\+1\)/2d\(d\+1\)/2\-parameter posterior covariance\) made concrete in the smallest non\-trivial case\. Full pathology table in[AppendixC](https://arxiv.org/html/2605.21798#A3)\.

#### Label contamination on a trained LNP\.

[Theorem1](https://arxiv.org/html/2605.21798#Thmtheorem1)’s two\-component prediction holds empirically on a trained MLP\-based LNP that violates the encoder\-affinity[2](https://arxiv.org/html/2605.21798#Thmassumption2)\. Resampling labels at fixed context locationsXX, the variance ofσLNP2​\(x∗;C\)\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\(x\_\{\*\};C\)separates into a signal\-induced floor that plateaus at≈0\.21\\approx 0\.21acrossn∈\[5,1000\]n\\in\[5,1000\]and a noise\-induced component that decays from1\.5×10−21\.5\\times 10^\{\-2\}to8×10−48\\times 10^\{\-4\}\. The floor\-to\-noise ratio grows from20×20\\timesatn=5n=5to256×256\\timesatn=1000n=1000, demonstrating the regime change predicted by[Theorem1](https://arxiv.org/html/2605.21798#Thmtheorem1): at smallnnboth components contribute, while at largennthe irreducibleO​\(1\)O\(1\)signal floor entirely dominates the contamination\. The qualitative pattern survives the violation of[2](https://arxiv.org/html/2605.21798#Thmassumption2), suggesting the structural mechanism \(label dependence in the variance pathway\) is general even when the precise constants are not\. Full setup, results, and slope fits are in[AppendixD](https://arxiv.org/html/2605.21798#A4)\.

#### Mercer alignment of trained encoders\.

[4](https://arxiv.org/html/2605.21798#Thmassumption4)posits that the encoder recovers the top kernel eigenfunctions\. We verify this by measuring the principal angles between the trained encoder’s 64\-dimensional feature subspace and the top\-ddMercer eigenfunction subspace across three SE kernels \(ℓ∈\{0\.1,0\.2,0\.5\}\\ell\\in\\\{0\.1,0\.2,0\.5\\\}\)\. The encoder recovers each eigenfunction withR2\>0\.95R^\{2\}\>0\.95whenever the corresponding eigenvalue exceeds the noise floor, and alignment degrades precisely where the spectrum becomes negligible: forℓ=0\.5\\ell=0\.5the subspace is near\-perfectly aligned \(cos⁡θmin\>0\.95\\cos\\theta\_\{\\min\}\>0\.95\) throughd=7d=7, collapsing atd=9d=9whereλj\\lambda\_\{j\}drops below10−810^\{\-8\}\. Shorter lengthscales produce slower spectral decay, and the encoder accordingly recovers more eigenfunctions before alignment degrades\. No explicit spectral regularization is applied; the alignment arises because the ELBO training objective implicitly rewards capturing the directions of maximum variation under the GP prior\. This suggests the bottleneck rates in[Corollary1](https://arxiv.org/html/2605.21798#Thmcorollary1)are approximately predictive of trained\-NP behavior in this setup\. Full methods and tables are in[AppendixE](https://arxiv.org/html/2605.21798#A5)\.

## 5Discussion

We provided the quantitative decomposition of the gap between neural process and Gaussian process predictives\. The three error sources, label contamination, information bottleneck, and amortization, have distinct origins and distinct architectural remedies\.

The decomposition provides concrete guidance for neural process design\. The bottleneck term governs the minimumddneeded for a given kernel\. For SE kernels,d=O​\(log⁡\(1/ϵ\)\)d=O\(\\sqrt\{\\log\(1/\\epsilon\)\}\)suffices for errorϵ\\epsilon, meaning even small representations work well\. For Matérn\-ν\\nukernels with smallν\\nu\(rough functions\),ddmust scale asO​\(ϵ−dx/\(2​ν\)\)O\(\\epsilon^\{\-d\_\{x\}/\(2\\nu\)\}\), motivating substantially larger representations\. Whether the linear dependence on the tail sum can be improved to quadratic under distributional assumptions on the context locations remains open\. Our bounds characterize pointwise predictive quality; a complementary question is whether NP predictives are consistent as stochastic processes, whichyoung2026consistencyanalyzes for CNPs\.

The label contamination term arises because the LNP estimates variance through a label\-dependent representation\. The signal component isO​\(1\)O\(1\)and does not vanish with increasing context size, making this a persistent source of error rather than a finite\-sample artifact\. An architecture that computes variance from context locations alone, without access to labels, would eliminate this term entirely\. This provides strong theoretical motivation for architectures that separate the mean and variance estimation pathways, predicting uncertainty from the input geometry alone\.

The above architectural recommendation applies to the GP\-amortization use case considered here, where data is drawn from a fixed known GP prior and exact GP inference is the ground truth\. In meta\-learning regimes where NPs are trained across distributions of related tasks \(or across GPs with varying hyperparameters\), the same label\-dependence we identify as “contamination” may carry Bayes\-relevant information about task structure, and a location\-only variance head may underperform\. Characterizing this tradeoff requires bounding the approximation gap under varying or misspecified priors, which we leave to future work\.

Attentive neural processes replace mean aggregation with cross\-attention\. The bottleneck analysis extends since attention weights can approximate the GP kernel weightsαi​\(x∗\)∝k​\(x∗,xi\)\\alpha\_\{i\}\(x\_\{\*\}\)\\propto k\(x\_\{\*\},x\_\{i\}\), potentially reducing the mean approximation error\. However, the variance still flows through a finite\-dimensional latent, so the bottleneck boundSdS\_\{d\}for the variance pathway is unchanged\. Our analysis suggests attention primarily addresses the mean pathway, and whether attention can implicitly capture the second\-order context statistics relevant to the variance remains open\.

The costs identified above should be weighed against the per\-task optimization cost that amortization eliminates\. Sparse variational GPs require jointly optimizing inducing locations, kernel hyperparameters, and variational parameters for each new context set, which is a non\-convex problem typically solved by gradient descent on the ELBO\. In sequential settings where the context changes at each iteration, or in real\-time settings where predictions are needed in milliseconds, this per\-task optimization may dominate the approximation quality gap\. The bounds we derive characterize what is lost in exchange for reducing inference to a single forward pass, and whether this tradeoff is favorable depends on the deployment regime\. In large\-scale spatial or environmental monitoring tasks for example, where GP\-quality predictions are needed across thousands or millions of related subdomains, per\-task sparse GP optimization is infeasible regardless of the per\-task cost, and amortization is the only viable path to posterior inference at scale\.

The bottleneck result connects neural process architecture to kernel theory\. The representation dimensionddmust be matched to the smoothness of the function class, as measured by the eigenvalue decay of the kernel\. This parallels both classical results onnn\-widths in approximation theory\(pinkus1985nwidths\)and convergence rates for sparse variational GPs\(burt2020convergence\), where the number of inducing points required for a given approximation quality is governed by the same eigenvalue decay\. The neural process bottleneck bound can be viewed as an amortized analogue of the sparse GP result, with the additional label contamination and amortization terms reflecting costs specific to the learned encoder architecture\.

Our analysis assumes the neural process is trained on data drawn from a known GP prior, which is the setting where exact GP inference is available\. The practical value of the bounds lies in their structural character\. The three error sources arise from architectural choices \(mean aggregation, finite bottleneck, amortized encoder\) that are present regardless of the data\-generating process\. We conjecture that the bottleneck smoothness correspondence extends to misspecified settings where the true function has regularity comparable to a given kernel class, but formalizing this requires bounding the approximation gap under model misspecification, which we leave to future work\.

[1](https://arxiv.org/html/2605.21798#Thmassumption1)is mild as the GP variance is automatically bounded for any positive\-definite kernel with bounded diagonal and positive noise, and the LNP variance is bounded whenever the decoder has a positive floorσd2\>0\\sigma\_\{d\}^\{2\}\>0and bounded weights\. In practice, all standard neural process implementations satisfy these conditions\.

Several directions remain open\. Our bounds are upper bounds and matching lower bounds would establish whether the rates are tight\. Whether attention\-based aggregation can implicitly capture second\-order statistics, thereby closing the amortization gap identified earlier, deserves investigation\. Finally, extending the analysis to convolutional neural processes\(Gordon2020Convolutional\), which exploit translation equivariance, could yield tighter bottleneck bounds when the kernel is stationary\.

## References

## Appendix APrimer on Neural Processes

This appendix provides a brief background on neural processes for readers more familiar with Gaussian processes than with the NP literature, and discusses how the assumptions used in our analysis relate to NP architectures used in practice\.

### A\.1Architecture and forward pass

A latent neural process\(garnelo2018neuralprocesses\)maps a context setC=\{\(xi,yi\)\}i=1nC=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{n\}and a target locationx∗x\_\{\*\}to a predictive distribution overy∗y\_\{\*\}inO​\(n\)O\(n\)time\. Figure[1](https://arxiv.org/html/2605.21798#A1.F1)shows the forward\-pass pipeline annotated with the three error sources of[Section3](https://arxiv.org/html/2605.21798#S3)\. For the full graphical model view that also distinguishes generative from inference paths, see Figure 1 fromgarnelo2018neuralprocesses\.

context pipeline \(operates onnnpairs\)\{xi\}\\\{x\_\{i\}\\\}\{yi\}\\\{y\_\{i\}\\\}h​\(xi,yi\)h\(x\_\{i\},y\_\{i\}\)rCr\_\{C\}μz,Σz\\mu\_\{z\},\\Sigma\_\{z\}zzdecoderx∗x\_\{\*\}p​\(y∗∣x∗,z\)p\(y\_\{\*\}\\mid x\_\{\*\},z\)labelcontaminationinformationbottleneckamortizationgapFigure 1:Forward\-pass schematic of a latent neural process with the three error sources annotated\. The dashed box encloses the context pipeline\. The encoderhhis applied independently to each of thenncontext pairs\(xi,yi\)\(x\_\{i\},y\_\{i\}\)producing per\-point features that mixxxandyyinputs \(label contamination,[Section3\.1](https://arxiv.org/html/2605.21798#S3.SS1)\), mean aggregationrC=1n​∑ih​\(xi,yi\)r\_\{C\}=\\frac\{1\}\{n\}\\sum\_\{i\}h\(x\_\{i\},y\_\{i\}\)compresses these to a finite\-dimensional summary \(information bottleneck,[Section3\.2](https://arxiv.org/html/2605.21798#S3.SS2)\), the recognition network\(μz,Σz\)\(\\mu\_\{z\},\\Sigma\_\{z\}\)mapsrCr\_\{C\}to a latent posterior using a single map shared across all contexts \(amortization gap,[Section3\.3](https://arxiv.org/html/2605.21798#S3.SS3)\)\. The target locationx∗x\_\{\*\}enters only at the decoder\.The components are typically parameterized as follows:

- •Encoderh:𝒳×𝒴→ℝdh:\\mathcal\{X\}\\times\\mathcal\{Y\}\\to\\mathbb\{R\}^\{d\}is an MLP applied independently to each context pair\(xi,yi\)\(x\_\{i\},y\_\{i\}\)\.
- •Aggregationreduces the variable\-size set\{hi\}i=1n\\\{h\_\{i\}\\\}\_\{i=1\}^\{n\}to a fixed\-size representationrC∈ℝdr\_\{C\}\\in\\mathbb\{R\}^\{d\}\. Mean aggregationrC=1n​∑ihir\_\{C\}=\\frac\{1\}\{n\}\\sum\_\{i\}h\_\{i\}is the original choice\(garnelo2018cnp;garnelo2018neuralprocesses\)\. Cross\-attention\(kim2018attentive\)and convolutional pooling\(Gordon2020Convolutional\)are common alternatives\.
- •Recognition network\(μz,Σz\):ℝd→ℝdz×ℝ≻0dz×dz\(\\mu\_\{z\},\\Sigma\_\{z\}\):\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{d\_\{z\}\}\\times\\mathbb\{R\}^\{d\_\{z\}\\times d\_\{z\}\}\_\{\\succ 0\}mapsrCr\_\{C\}to the parameters of a Gaussian latent distribution\. The covariance is typically diagonal or low\-rank, with positive definiteness enforced via a softplus or Cholesky parameterization\.
- •Decodergθ:𝒳×ℝdz→ℝ×ℝ\>0g\_\{\\theta\}:\\mathcal\{X\}\\times\\mathbb\{R\}^\{d\_\{z\}\}\\to\\mathbb\{R\}\\times\\mathbb\{R\}\_\{\>0\}produces a predictive mean and variance at each target\. In practicegθg\_\{\\theta\}is a nonlinear MLP; our linear decoderw​\(x∗\)⊤​z\+b​\(x∗\)w\(x\_\{\*\}\)^\{\\top\}z\+b\(x\_\{\*\}\)in[Definition1](https://arxiv.org/html/2605.21798#Thmdefinition1)is a tractability choice\.

The marginal predictive atx∗x\_\{\*\}is obtained by integrating outzz:

pLNP​\(y∗∣x∗,C\)=∫p​\(y∗∣x∗,z\)​q​\(z∣C\)​𝑑z\.p\_\{\\mathrm\{LNP\}\}\(y\_\{\*\}\\mid x\_\{\*\},C\)=\\int p\(y\_\{\*\}\\mid x\_\{\*\},z\)\\,q\(z\\mid C\)\\,dz\.\(28\)Under our linear decoder this integral is closed\-form Gaussian \(equations \([5](https://arxiv.org/html/2605.21798#S2.E5)\)–\([6](https://arxiv.org/html/2605.21798#S2.E6)\)\); under nonlinear decoders it is typically estimated by Monte Carlo overzz\.

### A\.2Training objective

NPs are trained by maximizing an evidence lower bound \(ELBO\) over context/target splits sampled from a task distribution\. Given a task with combined dataD=\{\(xi,yi\)\}i=1ND=\\\{\(x\_\{i\},y\_\{i\}\)\\\}\_\{i=1\}^\{N\}, randomly partitioned into contextCCand targetTT:

ℒ\(θ;D\)=𝔼q​\(z∣C∪T\)\[logp\(yT∣xT,z\)\]−KL\(q\(z∣C∪T\)∥q\(z∣C\)\)\.\\mathcal\{L\}\(\\theta;D\)=\\mathbb\{E\}\_\{q\(z\\mid C\\cup T\)\}\\\!\\left\[\\log p\(y\_\{T\}\\mid x\_\{T\},z\)\\right\]\-\\mathrm\{KL\}\\bigl\(q\(z\\mid C\\cup T\)\\,\\\|\\,q\(z\\mid C\)\\bigr\)\.\(29\)The first term encourages accurate target predictions and the second regularizes the context\-only encoder toward the full\-data encoder\. Tasks may be drawn from a fixed GP prior \(the regime our analysis targets\) or from a heterogeneous task distribution \(the meta\-learning regime discussed in[Section5](https://arxiv.org/html/2605.21798#S5)\)\.

### A\.3Relationship of the assumptions to practice

Our analysis makes three substantive idealizations of the LNP architecture beyond the basic[Definition1](https://arxiv.org/html/2605.21798#Thmdefinition1): the linear\-in\-zzdecoder \(item 4 of the definition\), the affine\-in\-yyencoder \([2](https://arxiv.org/html/2605.21798#Thmassumption2)\), and the Mercer\-feature encoder \([4](https://arxiv.org/html/2605.21798#Thmassumption4)\)\. The bounded\-variance[1](https://arxiv.org/html/2605.21798#Thmassumption1), the Lipschitz[3](https://arxiv.org/html/2605.21798#Thmassumption3), and the tail[5](https://arxiv.org/html/2605.21798#Thmassumption5)are technical conditions that hold under standard NP implementations and reasonable regularity\. We discuss the realism of each below\.

#### Linear decoder \([Definition1](https://arxiv.org/html/2605.21798#Thmdefinition1), item 4\)\.

Real NPs use nonlinear MLP decodersgθ​\(x∗,z\)g\_\{\\theta\}\(x\_\{\*\},z\)\. The linear formw​\(x∗\)⊤​z\+b​\(x∗\)w\(x\_\{\*\}\)^\{\\top\}z\+b\(x\_\{\*\}\)is a tractability choice with two consequences: it makes the marginal predictive Gaussian in closed form \(equations \([5](https://arxiv.org/html/2605.21798#S2.E5)\)–\([6](https://arxiv.org/html/2605.21798#S2.E6)\)\), and it makes the posteriorp​\(z∣C\)p\(z\\mid C\)Gaussian with closed\-form precisionΣp−1=I\+σd−2​∑iw​\(xi\)​w​\(xi\)⊤\\Sigma\_\{p\}^\{\-1\}=I\+\\sigma\_\{d\}^\{\-2\}\\sum\_\{i\}w\(x\_\{i\}\)w\(x\_\{i\}\)^\{\\top\}\(equation \([18](https://arxiv.org/html/2605.21798#S3.E18)\)\)\. The latter is what makes the amortization gap characterizable in Propositions[1](https://arxiv.org/html/2605.21798#Thmproposition1)–[2](https://arxiv.org/html/2605.21798#Thmproposition2); without it,p​\(z∣C\)p\(z\\mid C\)is intractable and the gap cannot be expressed in closed form\.

The dimension\-counting argument that motivates the second\-order aggregation recommendation, that mean aggregation providesddscalars whileΣp\\Sigma\_\{p\}hasd​\(d\+1\)/2d\(d\+1\)/2free parameters, is structural and does not depend on the linear decoder\. We expect the qualitative recommendation to transfer to nonlinear decoders, though the constants do not\.

#### Encoder affine inyy\([2](https://arxiv.org/html/2605.21798#Thmassumption2)\)\.

Real NPs use general MLPsh​\(x,y\)h\(x,y\)that are non\-affine inyy\. The affine assumptionh​\(x,y\)=ϕ​\(x\)\+ψ​\(x\)​yh\(x,y\)=\\phi\(x\)\+\\psi\(x\)yallows the clean decompositionrC=ϕ¯X\+δyr\_\{C\}=\\bar\{\\phi\}\_\{X\}\+\\delta\_\{y\}in equation \([8](https://arxiv.org/html/2605.21798#S3.E8)\), isolating the label\-dependent componentδy\\delta\_\{y\}\. The qualitative conclusion of[Theorem1](https://arxiv.org/html/2605.21798#Thmtheorem1)is a structural property of any encoder that mixesxxandyyinputs into a representation that determines the predictive variance\. The empirical verification in[AppendixD](https://arxiv.org/html/2605.21798#A4)uses an MLP encoder that is non\-affine inyyand observes the predicted two\-component decomposition, suggesting the qualitative result transfers\. The constants in[Theorem1](https://arxiv.org/html/2605.21798#Thmtheorem1)are specific to the affine setting\.

#### Mercer\-feature encoder \([4](https://arxiv.org/html/2605.21798#Thmassumption4)\)\.

Real NPs do not necessarily learn Mercer eigenfunctions as the standard ELBO contains no term that would explicitly encourage this alignment\.[4](https://arxiv.org/html/2605.21798#Thmassumption4)should be read as characterizing the best\-case spectral alignment\. If the encoder achieves the optimaldd\-dimensional kernel approximation, then the bottleneck rates in[Corollary1](https://arxiv.org/html/2605.21798#Thmcorollary1)apply\. A learned encoder achieving suboptimal alignment would incur additional approximation error inϕ¯X\\bar\{\\phi\}\_\{X\}relative to the eigenfunction projection, beyond what[Theorem2](https://arxiv.org/html/2605.21798#Thmtheorem2)characterizes\.

We do not bound this gap analytically\. However, the empirical verification in[AppendixE](https://arxiv.org/html/2605.21798#A5)shows that trained NP encoders do approximately recover the top Mercer eigenfunctions, withR2\>0\.95R^\{2\}\>0\.95for all eigenfunctions whose eigenvalues exceed the noise floor\. The alignment degrades where the spectrum becomes negligible, and the effective aligned dimension tracks the kernel lengthscale\. This suggests the bottleneck rates in[Corollary1](https://arxiv.org/html/2605.21798#Thmcorollary1)are approximately predictive of trained\-NP behavior in the GP\-amortization regime, though the formal gap between the Mercer encoder and a learned encoder remains an open problem\.

### A\.4NP variants and which assumptions they satisfy

- •Conditional NPs\(garnelo2018cnp\): no latentzz; deterministic context\-to\-prediction map\. Our amortization\-gap analysis does not apply but the bottleneck and label\-contamination analyses do\.
- •Latent NPs\(garnelo2018neuralprocesses\): the architecture our analysis targets, modulo the linear\-decoder and Mercer\-feature idealizations\.
- •Attentive NPs\(kim2018attentive\): replace mean aggregation with cross\-attention\. The bottleneck bound on the variance pathway is unchanged \(variance still flows through a finite\-dimensional latent\)\. Whether attention can implicitly capture second\-order context statistics relevant to the variance is open\.
- •Convolutional NPs\(Gordon2020Convolutional\): exploit translation equivariance for stationary kernels\. Our bottleneck analysis does not directly apply, though we conjecture the spectral correspondence extends with modified rates\.

## Appendix BProofs

This appendix contains the full proofs of Theorems[1](https://arxiv.org/html/2605.21798#Thmtheorem1)and[2](https://arxiv.org/html/2605.21798#Thmtheorem2)and[Proposition1](https://arxiv.org/html/2605.21798#Thmproposition1)\.

### B\.1Proof of[Theorem1](https://arxiv.org/html/2605.21798#Thmtheorem1)\(Label Contamination Bound\)

###### Proof\.

Conditional onXX, the label\-dependent componentδy=1n​∑i=1nψ​\(xi\)​yi\\delta\_\{y\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\psi\(x\_\{i\}\)y\_\{i\}has:

𝔼​\[δy∣X\]=1n​∑iψ​\(xi\)​𝔼​\[yi\]=0\\mathbb\{E\}\[\\delta\_\{y\}\\mid X\]=\\frac\{1\}\{n\}\\sum\_\{i\}\\psi\(x\_\{i\}\)\\,\\mathbb\{E\}\[y\_\{i\}\]=0\(30\)under the zero\-mean GP prior\. The covariance is:

𝔼​\[δy​δy⊤∣X\]\\displaystyle\\mathbb\{E\}\[\\delta\_\{y\}\\delta\_\{y\}^\{\\top\}\\mid X\]=1n2​∑i,jψ​\(xi\)​ψ​\(xj\)⊤​𝔼​\[yi​yj\]\\displaystyle=\\frac\{1\}\{n^\{2\}\}\\sum\_\{i,j\}\\psi\(x\_\{i\}\)\\psi\(x\_\{j\}\)^\{\\top\}\\mathbb\{E\}\[y\_\{i\}y\_\{j\}\]\(31\)=1n2​∑i,jψ​\(xi\)​ψ​\(xj\)⊤​\(k​\(xi,xj\)\+σϵ2​δi​j\)\.\\displaystyle=\\frac\{1\}\{n^\{2\}\}\\sum\_\{i,j\}\\psi\(x\_\{i\}\)\\psi\(x\_\{j\}\)^\{\\top\}\(k\(x\_\{i\},x\_\{j\}\)\+\\sigma\_\{\\epsilon\}^\{2\}\\delta\_\{ij\}\)\.\(32\)Decomposeyi=f​\(xi\)\+ϵiy\_\{i\}=f\(x\_\{i\}\)\+\\epsilon\_\{i\}to splitδy=δf\+δϵ\\delta\_\{y\}=\\delta\_\{f\}\+\\delta\_\{\\epsilon\}where

δf=1n​∑iψ​\(xi\)​f​\(xi\),δϵ=1n​∑iψ​\(xi\)​ϵi\.\\delta\_\{f\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\psi\(x\_\{i\}\)f\(x\_\{i\}\),\\qquad\\delta\_\{\\epsilon\}=\\frac\{1\}\{n\}\\sum\_\{i\}\\psi\(x\_\{i\}\)\\epsilon\_\{i\}\.\(33\)Since theϵi\\epsilon\_\{i\}are i\.i\.d\. with varianceσϵ2\\sigma\_\{\\epsilon\}^\{2\}and independent offf:

𝔼​\[‖δϵ‖2∣X\]=σϵ2n2​∑i‖ψ​\(xi\)‖2≤Bψ2​σϵ2n\.\\mathbb\{E\}\[\\\|\\delta\_\{\\epsilon\}\\\|^\{2\}\\mid X\]=\\frac\{\\sigma\_\{\\epsilon\}^\{2\}\}\{n^\{2\}\}\\sum\_\{i\}\\\|\\psi\(x\_\{i\}\)\\\|^\{2\}\\leq\\frac\{B\_\{\\psi\}^\{2\}\\sigma\_\{\\epsilon\}^\{2\}\}\{n\}\.\(34\)For the signal term, the covariance is

𝔼​\[δf​δf⊤∣X\]=1n2​∑i,jψ​\(xi\)​ψ​\(xj\)⊤​k​\(xi,xj\)\.\\mathbb\{E\}\[\\delta\_\{f\}\\delta\_\{f\}^\{\\top\}\\mid X\]=\\frac\{1\}\{n^\{2\}\}\\sum\_\{i,j\}\\psi\(x\_\{i\}\)\\psi\(x\_\{j\}\)^\{\\top\}k\(x\_\{i\},x\_\{j\}\)\.\(35\)Taking the trace and using‖ψ​\(x\)‖≤Bψ\\\|\\psi\(x\)\\\|\\leq B\_\{\\psi\}:

𝔼​\[‖δf‖2∣X\]\\displaystyle\\mathbb\{E\}\[\\\|\\delta\_\{f\}\\\|^\{2\}\\mid X\]≤Bψ2n2​∑i,j\|k​\(xi,xj\)\|\\displaystyle\\leq\\frac\{B\_\{\\psi\}^\{2\}\}\{n^\{2\}\}\\sum\_\{i,j\}\|k\(x\_\{i\},x\_\{j\}\)\|\(36\)≤Bψ2n​supi∑j=1n\|k​\(xi,xj\)\|n≤Bψ2​κk,\\displaystyle\\leq\\frac\{B\_\{\\psi\}^\{2\}\}\{n\}\\sup\_\{i\}\\sum\_\{j=1\}^\{n\}\\frac\{\|k\(x\_\{i\},x\_\{j\}\)\|\}\{n\}\\leq B\_\{\\psi\}^\{2\}\\kappa\_\{k\},\(37\)the last inequality using1n​∑j\|k​\(xi,xj\)\|≤supxk​\(x,x\)=κk\\frac\{1\}\{n\}\\sum\_\{j\}\|k\(x\_\{i\},x\_\{j\}\)\|\\leq\\sup\_\{x\}k\(x,x\)=\\kappa\_\{k\}, which holds sincekkis positive definite with bounded diagonal\. Note that‖K‖op=O​\(n\)\\\|K\\\|\_\{\\mathrm\{op\}\}=O\(n\)for i\.i\.d\. context points drawn fromμ\\mu, so thisO​\(1\)O\(1\)bound cannot be improved in general\.

Sinceδf\\delta\_\{f\}andδϵ\\delta\_\{\\epsilon\}are independent conditional onXX\(the noiseϵi\\epsilon\_\{i\}is independent offf\), the cross term vanishes:𝔼​\[δf⊤​δϵ∣X\]=𝔼​\[δf∣X\]⊤​𝔼​\[δϵ∣X\]=0\\mathbb\{E\}\[\\delta\_\{f\}^\{\\top\}\\delta\_\{\\epsilon\}\\mid X\]=\\mathbb\{E\}\[\\delta\_\{f\}\\mid X\]^\{\\top\}\\mathbb\{E\}\[\\delta\_\{\\epsilon\}\\mid X\]=0\. Therefore:

𝔼​\[‖δy‖2∣X\]=𝔼​\[‖δf‖2∣X\]\+𝔼​\[‖δϵ‖2∣X\]≤Bψ2​\(κk\+σϵ2n\)\.\\mathbb\{E\}\[\\\|\\delta\_\{y\}\\\|^\{2\}\\mid X\]=\\mathbb\{E\}\[\\\|\\delta\_\{f\}\\\|^\{2\}\\mid X\]\+\\mathbb\{E\}\[\\\|\\delta\_\{\\epsilon\}\\\|^\{2\}\\mid X\]\\leq B\_\{\\psi\}^\{2\}\\\!\\left\(\\kappa\_\{k\}\+\\frac\{\\sigma\_\{\\epsilon\}^\{2\}\}\{n\}\\right\)\.\(38\)
SinceΣz\\Sigma\_\{z\}isLΣL\_\{\\Sigma\}\-Lipschitz and the LNP variance isσLNP2=w​\(x∗\)⊤​Σz​\(rC\)​w​\(x∗\)\+σd2\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}=w\(x\_\{\*\}\)^\{\\top\}\\Sigma\_\{z\}\(r\_\{C\}\)w\(x\_\{\*\}\)\+\\sigma\_\{d\}^\{2\}:

\|σLNP2​\(rC\)−σLNP2​\(ϕ¯X\)\|≤LΣ​Bw2​‖δy‖\.\|\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\(r\_\{C\}\)\-\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\(\\bar\{\\phi\}\_\{X\}\)\|\\leq L\_\{\\Sigma\}B\_\{w\}^\{2\}\\\|\\delta\_\{y\}\\\|\.\(39\)Taking the conditional variance:

Var𝒚\|X​\[σLNP2\]\\displaystyle\\mathrm\{Var\}\_\{\\bm\{y\}\|X\}\[\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\]≤𝔼𝒚\|X​\[\(σLNP2​\(rC\)−σLNP2​\(ϕ¯X\)\)2\]\\displaystyle\\leq\\mathbb\{E\}\_\{\\bm\{y\}\|X\}\\bigl\[\(\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\(r\_\{C\}\)\-\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\(\\bar\{\\phi\}\_\{X\}\)\)^\{2\}\\bigr\]≤LΣ2​Bw4​𝔼​\[‖δy‖2∣X\]\.\\displaystyle\\leq L\_\{\\Sigma\}^\{2\}B\_\{w\}^\{4\}\\,\\mathbb\{E\}\\bigl\[\\\|\\delta\_\{y\}\\\|^\{2\}\\mid X\\bigr\]\.\(40\)Taking the expectation overXXand substituting the bound on𝔼​\[‖δy‖2∣X\]\\mathbb\{E\}\[\\\|\\delta\_\{y\}\\\|^\{2\}\\mid X\]yields the result\. ∎

### B\.2Proof of[Theorem2](https://arxiv.org/html/2605.21798#Thmtheorem2)\(Information Bottleneck Bound\)

###### Proof\.

Write the GP variance using the Mercer expansion\. The kernel vector and matrix satisfy

\[𝒌∗\]i\\displaystyle\[\\bm\{k\}\_\{\*\}\]\_\{i\}=∑j=1∞λj​ej​\(x∗\)​ej​\(xi\),\\displaystyle=\\sum\_\{j=1\}^\{\\infty\}\\lambda\_\{j\}\\,e\_\{j\}\(x\_\{\*\}\)\\,e\_\{j\}\(x\_\{i\}\),\(41\)\[K\]i​l\\displaystyle\[K\]\_\{il\}=∑j=1∞λj​ej​\(xi\)​ej​\(xl\)\.\\displaystyle=\\sum\_\{j=1\}^\{\\infty\}\\lambda\_\{j\}\\,e\_\{j\}\(x\_\{i\}\)\\,e\_\{j\}\(x\_\{l\}\)\.\(42\)Define the empirical kernel featurev^j=1n​∑i=1nej​\(xi\)2\\hat\{v\}\_\{j\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}e\_\{j\}\(x\_\{i\}\)^\{2\}\. For fixed context locations, the GP variance can be written via the Woodbury identity as a function of the spectrum and the empirical features\{v^j\}j=1∞\\\{\\hat\{v\}\_\{j\}\\\}\_\{j=1\}^\{\\infty\}\. Note thatϕ¯X\\bar\{\\phi\}\_\{X\}is add\-dimensional vector encoding1n​∑iλj​ej​\(xi\)\\frac\{1\}\{n\}\\sum\_\{i\}\\sqrt\{\\lambda\_\{j\}\}\\,e\_\{j\}\(x\_\{i\}\)forj=1,…,dj=1,\\dots,d, which does not determine the diagonal empirical featuresv^j=1n​∑iej​\(xi\)2\\hat\{v\}\_\{j\}=\\frac\{1\}\{n\}\\sum\_\{i\}e\_\{j\}\(x\_\{i\}\)^\{2\}forj≤dj\\leq d\. The optimal estimatorgd∗​\(ϕ¯X\)g\_\{d\}^\{\*\}\(\\bar\{\\phi\}\_\{X\}\)is the best predictor ofσGP2\\sigma^\{2\}\_\{\\mathrm\{GP\}\}given this limited statistic\.

#### Setup: head/tail decomposition\.

Write the Mercer series as head \(j≤dj\\leq d\) and tail \(j\>dj\>d\) components\. Define the head and tail prior variances

kH​\(x∗\)=∑j≤dλj​ej​\(x∗\)2,kT​\(x∗\)=∑j\>dλj​ej​\(x∗\)2,k\_\{H\}\(x\_\{\*\}\)=\\sum\_\{j\\leq d\}\\lambda\_\{j\}\\,e\_\{j\}\(x\_\{\*\}\)^\{2\},\\qquad k\_\{T\}\(x\_\{\*\}\)=\\sum\_\{j\>d\}\\lambda\_\{j\}\\,e\_\{j\}\(x\_\{\*\}\)^\{2\},\(43\)the corresponding splits𝐤∗H\\mathbf\{k\}\_\{\*\}^\{H\},𝐤∗T\\mathbf\{k\}\_\{\*\}^\{T\}of the kernel vector andKHK\_\{H\},KTK\_\{T\}of the kernel matrix, and let

A=KH\+σϵ2​I,M=A\+KT\.A=K\_\{H\}\+\\sigma\_\{\\epsilon\}^\{2\}I,\\qquad M=A\+K\_\{T\}\.\(44\)The truncated GP variance is then

σd2=kH−\(𝐤∗H\)⊤​A−1​𝐤∗H,\\sigma\_\{d\}^\{2\}=k\_\{H\}\-\(\\mathbf\{k\}\_\{\*\}^\{H\}\)^\{\\top\}A^\{\-1\}\\,\\mathbf\{k\}\_\{\*\}^\{H\},\(45\)which depends only on head eigenfunction evaluations\{ej​\(xi\)\}j≤d\\\{e\_\{j\}\(x\_\{i\}\)\\\}\_\{j\\leq d\}at the context points\. Since the optimal estimatorgd∗​\(ϕ¯X\)g\_\{d\}^\{\*\}\(\\bar\{\\phi\}\_\{X\}\)minimizes MSE over all measurable functions ofϕ¯X\\bar\{\\phi\}\_\{X\}, andσd2\\sigma\_\{d\}^\{2\}is measurable with respect to a finerσ\\sigma\-algebra \(the full head evaluations\), we have

Sd≤𝔼X​\[\(σGP2−σd2\)2\]\.S\_\{d\}\\leq\\mathbb\{E\}\_\{X\}\\\!\\bigl\[\(\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\-\\sigma\_\{d\}^\{2\}\)^\{2\}\\bigr\]\.\(46\)

#### Bounding\|σGP2−σd2\|\|\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\-\\sigma\_\{d\}^\{2\}\|via Neumann expansion\.

The full GP variance isσGP2=\(kH\+kT\)−𝒌∗⊤​M−1​𝒌∗\\sigma^\{2\}\_\{\\mathrm\{GP\}\}=\(k\_\{H\}\+k\_\{T\}\)\-\\bm\{k\}\_\{\*\}^\{\\top\}M^\{\-1\}\\bm\{k\}\_\{\*\}\. WritingΔ​I=𝒌∗⊤​M−1​𝒌∗−\(𝒌∗H\)⊤​A−1​𝒌∗H\\Delta I=\\bm\{k\}\_\{\*\}^\{\\top\}M^\{\-1\}\\bm\{k\}\_\{\*\}\-\(\\bm\{k\}\_\{\*\}^\{H\}\)^\{\\top\}A^\{\-1\}\\bm\{k\}\_\{\*\}^\{H\}, we have

σGP2−σd2=kT−Δ​I\.\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\-\\sigma\_\{d\}^\{2\}=k\_\{T\}\-\\Delta I\.\(47\)
We boundΔ​I\\Delta Iusing a Neumann expansion\. Pass to the whitened coordinates𝒂=A−1/2​𝒌∗H\\bm\{a\}=A^\{\-1/2\}\\bm\{k\}\_\{\*\}^\{H\}and𝒃=A−1/2​𝒌∗T\\bm\{b\}=A^\{\-1/2\}\\bm\{k\}\_\{\*\}^\{T\}, and defineP=A−1/2​KT​A−1/2P=A^\{\-1/2\}K\_\{T\}A^\{\-1/2\}\. Then‖𝒂‖2=\(𝒌∗H\)⊤​A−1​𝒌∗H\\\|\\bm\{a\}\\\|^\{2\}=\(\\bm\{k\}\_\{\*\}^\{H\}\)^\{\\top\}A^\{\-1\}\\bm\{k\}\_\{\*\}^\{H\}and∥𝒃∥2=:β=\(𝒌∗T\)⊤A−1𝒌∗T\\\|\\bm\{b\}\\\|^\{2\}=:\\beta=\(\\bm\{k\}\_\{\*\}^\{T\}\)^\{\\top\}A^\{\-1\}\\bm\{k\}\_\{\*\}^\{T\}\. SinceA⪰σϵ2​IA\\succeq\\sigma\_\{\\epsilon\}^\{2\}I:

‖𝒂‖2≤kH/σϵ2,β≤kT/σϵ2\.\\\|\\bm\{a\}\\\|^\{2\}\\leq k\_\{H\}/\\sigma\_\{\\epsilon\}^\{2\},\\qquad\\beta\\leq k\_\{T\}/\\sigma\_\{\\epsilon\}^\{2\}\.\(48\)NowM−1=A−1/2​\(I\+P\)−1​A−1/2M^\{\-1\}=A^\{\-1/2\}\(I\+P\)^\{\-1\}A^\{\-1/2\}, and under[5](https://arxiv.org/html/2605.21798#Thmassumption5)\(η=‖P‖op<1\\eta=\\\|P\\\|\_\{\\mathrm\{op\}\}<1\):

\(I\+P\)−1=I−P\+Q,Q=∑m=2∞\(−P\)m,‖Q‖op≤η21−η\.\(I\+P\)^\{\-1\}=I\-P\+Q,\\qquad Q=\\sum\_\{m=2\}^\{\\infty\}\(\-P\)^\{m\},\\qquad\\\|Q\\\|\_\{\\mathrm\{op\}\}\\leq\\frac\{\\eta^\{2\}\}\{1\-\\eta\}\.\(49\)ExpandingΔ​I\\Delta Iin these coordinates:

Δ​I\\displaystyle\\Delta I=\(𝐚\+𝐛\)⊤​\(I\+P\)−1​\(𝐚\+𝐛\)−‖𝐚‖2\\displaystyle=\(\\mathbf\{a\}\+\\mathbf\{b\}\)^\{\\top\}\(I\+P\)^\{\-1\}\(\\mathbf\{a\}\+\\mathbf\{b\}\)\-\\\|\\mathbf\{a\}\\\|^\{2\}=2​𝐚⊤​𝐛\+‖𝐛‖2⏟T1−𝐚⊤​P​𝐚\+2​𝐚⊤​P​𝐛\+𝐛⊤​P​𝐛⏟T2\+\(𝐚\+𝐛\)⊤​Q​\(𝐚\+𝐛\)⏟T3\.\\displaystyle=\\underbrace\{2\\mathbf\{a\}^\{\\top\}\\mathbf\{b\}\+\\\|\\mathbf\{b\}\\\|^\{2\}\}\_\{T\_\{1\}\}\-\\underbrace\{\\mathbf\{a\}^\{\\top\}P\\,\\mathbf\{a\}\+2\\,\\mathbf\{a\}^\{\\top\}P\\,\\mathbf\{b\}\+\\mathbf\{b\}^\{\\top\}P\\,\\mathbf\{b\}\}\_\{T\_\{2\}\}\+\\underbrace\{\(\\mathbf\{a\}\+\\mathbf\{b\}\)^\{\\top\}Q\\,\(\\mathbf\{a\}\+\\mathbf\{b\}\)\}\_\{T\_\{3\}\}\.\(50\)
By Cauchy–Schwarz:

\|T1\|\\displaystyle\|T\_\{1\}\|≤2​‖𝒂‖​β\+β,\\displaystyle\\leq 2\\\|\\bm\{a\}\\\|\\sqrt\{\\beta\}\+\\beta,\(51\)\|T2\|\\displaystyle\|T\_\{2\}\|≤η​\(‖𝒂‖\+β\)2,\\displaystyle\\leq\\eta\\bigl\(\\\|\\bm\{a\}\\\|\+\\sqrt\{\\beta\}\\bigr\)^\{2\},\(52\)\|T3\|\\displaystyle\|T\_\{3\}\|≤η21−η​\(‖𝒂‖\+β\)2\.\\displaystyle\\leq\\frac\{\\eta^\{2\}\}\{1\-\\eta\}\\bigl\(\\\|\\bm\{a\}\\\|\+\\sqrt\{\\beta\}\\bigr\)^\{2\}\.\(53\)Substituting into \([47](https://arxiv.org/html/2605.21798#A2.E47)\):

\|σGP2−σd2\|\\displaystyle\|\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\-\\sigma\_\{d\}^\{2\}\|=\|kT−Δ​I\|\\displaystyle=\|k\_\{T\}\-\\Delta I\|≤kT\+2​‖𝐚‖​β\+β\+η1−η​\(‖𝐚‖\+β\)2\.\\displaystyle\\leq k\_\{T\}\+2\\\|\\mathbf\{a\}\\\|\\sqrt\{\\beta\}\+\\beta\+\\frac\{\\eta\}\{1\-\\eta\}\\bigl\(\\\|\\mathbf\{a\}\\\|\+\\sqrt\{\\beta\}\\bigr\)^\{2\}\.\(54\)Substituting the bounds \([48](https://arxiv.org/html/2605.21798#A2.E48)\):

\|σGP2−σd2\|≤kT\+2​κk​kTσϵ2\+kTσϵ2\+η1−η⋅\(κk\+kT\)2σϵ2\.\|\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\-\\sigma\_\{d\}^\{2\}\|\\leq k\_\{T\}\+\\frac\{2\\sqrt\{\\kappa\_\{k\}\\,k\_\{T\}\}\}\{\\sigma\_\{\\epsilon\}^\{2\}\}\+\\frac\{k\_\{T\}\}\{\\sigma\_\{\\epsilon\}^\{2\}\}\+\\frac\{\\eta\}\{1\-\\eta\}\\cdot\\frac\{\(\\sqrt\{\\kappa\_\{k\}\}\+\\sqrt\{k\_\{T\}\}\)^\{2\}\}\{\\sigma\_\{\\epsilon\}^\{2\}\}\.\(55\)

#### Extracting the rate\.

Every term on the right of \([55](https://arxiv.org/html/2605.21798#A2.E55)\) isO​\(kT\)O\(\\sqrt\{k\_\{T\}\}\)asd→∞d\\to\\infty\(usingη≤‖KT‖op/σϵ2→0\\eta\\leq\\\|K\_\{T\}\\\|\_\{\\mathrm\{op\}\}/\\sigma\_\{\\epsilon\}^\{2\}\\to 0andkT→0k\_\{T\}\\to 0\)\. After squaring, the dominant contribution is:

\(2​κk​kTσϵ2\)2=4​κkσϵ4​kT,\\left\(\\frac\{2\\sqrt\{\\kappa\_\{k\}\\,k\_\{T\}\}\}\{\\sigma\_\{\\epsilon\}^\{2\}\}\\right\)^\{2\}=\\frac\{4\\kappa\_\{k\}\}\{\\sigma\_\{\\epsilon\}^\{4\}\}\\,k\_\{T\},\(56\)which isO​\(kT\)O\(k\_\{T\}\)\. All other squared terms areO​\(kT3/2\)O\(k\_\{T\}^\{3/2\}\)or smaller\. Specifically, expanding \([55](https://arxiv.org/html/2605.21798#A2.E55)\) squared and usinga​b≤12​\(a2\+b2\)ab\\leq\\tfrac\{1\}\{2\}\(a^\{2\}\+b^\{2\}\)to absorb cross terms:

\(σGP2−σd2\)2≤CS​kT​\(x∗\),\(\\sigma^\{2\}\_\{\\mathrm\{GP\}\}\-\\sigma\_\{d\}^\{2\}\)^\{2\}\\leq C\_\{S\}\\,k\_\{T\}\(x\_\{\*\}\),\(57\)where, in the regimeη≤1/2\\eta\\leq 1/2andkT≤κkk\_\{T\}\\leq\\kappa\_\{k\}:

CS=C​κkσϵ4C\_\{S\}=\\frac\{C\\,\\kappa\_\{k\}\}\{\\sigma\_\{\\epsilon\}^\{4\}\}\(58\)for a constantCCdepending only onκk/σϵ2\\kappa\_\{k\}/\\sigma\_\{\\epsilon\}^\{2\}\. Applying \([46](https://arxiv.org/html/2605.21798#A2.E46)\):

Sd​\(k,x∗\)≤CS​∑j\>dλj​ej​\(x∗\)2\.S\_\{d\}\(k,x\_\{\*\}\)\\leq C\_\{S\}\\sum\_\{j\>d\}\\lambda\_\{j\}\\,e\_\{j\}\(x\_\{\*\}\)^\{2\}\.\(59\)

#### Estimation errorRdR\_\{d\}\.

The estimation errorRd​\(k,n\)R\_\{d\}\(k,n\)arises from the variance ofϕ¯X\\bar\{\\phi\}\_\{X\}as an estimator of𝔼μ​\[ϕ​\(x\)\]\\mathbb\{E\}\_\{\\mu\}\[\\phi\(x\)\]and isO​\(d2/n\)O\(d^\{2\}/n\)by the central limit theorem applied to thedd\-dimensional i\.i\.d\. sum, plus the irreducible information lossRdinfoR\_\{d\}^\{\\mathrm\{info\}\}from mean aggregation being an insufficient statistic for the head kernel matrix\. We do not boundRdinfoR\_\{d\}^\{\\mathrm\{info\}\}in general as it depends on the relationship between the first and second moments of the eigenfunction evaluations underμ\\mu\. Under second\-order aggregation,ϕ¯X\\bar\{\\phi\}\_\{X\}is replaced by1n​∑iϕ​\(xi\)​ϕ​\(xi\)⊤\\frac\{1\}\{n\}\\sum\_\{i\}\\phi\(x\_\{i\}\)\\phi\(x\_\{i\}\)^\{\\top\}, which is a sufficient statistic for the head kernel matrix, soRdinfo=0R\_\{d\}^\{\\mathrm\{info\}\}=0and the estimation error reduces toO​\(d2/n\)O\(d^\{2\}/n\)\. ∎

### B\.3Proof of[Proposition1](https://arxiv.org/html/2605.21798#Thmproposition1)\(Scalar Amortization Gap\)

###### Proof\.

The posterior precision isΣp−1=1\+α​v^\\Sigma\_\{p\}^\{\-1\}=1\+\\alpha\\,\\hat\{v\}, wherev^=1n​∑i=1ne1​\(xi\)2\\hat\{v\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}e\_\{1\}\(x\_\{i\}\)^\{2\}is the empirical second moment andα=n​λ1/σd2\\alpha=n\\lambda\_\{1\}/\\sigma\_\{d\}^\{2\}\.

#### Uncorrelatedness ofv^\\hat\{v\}ande¯\\bar\{e\}\.

The mean representation ise¯=1n​∑i=1ne1​\(xi\)\\bar\{e\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}e\_\{1\}\(x\_\{i\}\)\. We claim thatv^\\hat\{v\}ande¯\\bar\{e\}are uncorrelated\. Sincexi​∼iid​μx\_\{i\}\\overset\{\\mathrm\{iid\}\}\{\\sim\}\\mu:

Cov​\(e1​\(x\)2,e1​\(x\)\)=𝔼μ​\[e1​\(x\)3\]−𝔼μ​\[e1​\(x\)2\]​𝔼μ​\[e1​\(x\)\]\.\\mathrm\{Cov\}\(e\_\{1\}\(x\)^\{2\},\\;e\_\{1\}\(x\)\)=\\mathbb\{E\}\_\{\\mu\}\[e\_\{1\}\(x\)^\{3\}\]\-\\mathbb\{E\}\_\{\\mu\}\[e\_\{1\}\(x\)^\{2\}\]\\,\\mathbb\{E\}\_\{\\mu\}\[e\_\{1\}\(x\)\]\.\(60\)Both terms vanish:𝔼μ​\[e1\]=0\\mathbb\{E\}\_\{\\mu\}\[e\_\{1\}\]=0by the zero\-mean property of eigenfunctions underμ\\mu, and𝔼μ​\[e13\]=0\\mathbb\{E\}\_\{\\mu\}\[e\_\{1\}^\{3\}\]=0by the symmetry ofe1e\_\{1\}underμ\\mu\(which holds for standard eigenfunction bases on symmetric domains, includinge1​\(x\)=2​cos⁡\(π​x\)e\_\{1\}\(x\)=\\sqrt\{2\}\\cos\(\\pi x\)on\[0,1\]\[0,1\]with the uniform measure\)\. Thereforev^\\hat\{v\}ande¯\\bar\{e\}are uncorrelated\. Since both are sample means of i\.i\.d\. random variables, by the multivariate CLT they are approximately jointly Gaussian, so uncorrelatedness implies approximate independence\.

#### Optimal estimator\.

The optimal estimator ofΣp\\Sigma\_\{p\}givene¯\\bar\{e\}therefore uses𝔼​\[v^∣e¯\]≈𝔼​\[v^\]=1\\mathbb\{E\}\[\\hat\{v\}\\mid\\bar\{e\}\]\\approx\\mathbb\{E\}\[\\hat\{v\}\]=1, yieldingΣz∗=\(1\+α\)−1\\Sigma\_\{z\}^\{\*\}=\(1\+\\alpha\)^\{\-1\}\.

#### KL expansion\.

The KL divergence between the two scalar Gaussians \(with zero means\) is:

KL=12​\(r−1−log⁡r\),r=Σz∗/Σp=1\+α​v^1\+α\.\\mathrm\{KL\}=\\tfrac\{1\}\{2\}\\bigl\(r\-1\-\\log r\\bigr\),\\qquad r=\\Sigma\_\{z\}^\{\*\}/\\Sigma\_\{p\}=\\frac\{1\+\\alpha\\,\\hat\{v\}\}\{1\+\\alpha\}\.\(61\)Writev^=1\+δ\\hat\{v\}=1\+\\deltawhere𝔼​\[δ\]=0\\mathbb\{E\}\[\\delta\]=0andVar​\[δ\]=1n​Varμ​\[e1​\(x\)2\]\\mathrm\{Var\}\[\\delta\]=\\frac\{1\}\{n\}\\mathrm\{Var\}\_\{\\mu\}\[e\_\{1\}\(x\)^\{2\}\]\. The kurtosis integral gives𝔼μ​\[e14\]=32\\mathbb\{E\}\_\{\\mu\}\[e\_\{1\}^\{4\}\]=\\frac\{3\}\{2\}\(fore1=2​cos⁡\(π​x\)e\_\{1\}=\\sqrt\{2\}\\cos\(\\pi x\)under the uniform measure\), soVarμ​\[e12\]=12\\mathrm\{Var\}\_\{\\mu\}\[e\_\{1\}^\{2\}\]=\\frac\{1\}\{2\}andVar​\[δ\]=12​n\\mathrm\{Var\}\[\\delta\]=\\frac\{1\}\{2n\}\.

Settingε=α​δ1\+α\\varepsilon=\\frac\{\\alpha\\,\\delta\}\{1\+\\alpha\}so thatr=1\+εr=1\+\\varepsilon, and expanding to second order:

𝔼​\[12​\(ε−log⁡\(1\+ε\)\)\]\\displaystyle\\mathbb\{E\}\\\!\\left\[\\tfrac\{1\}\{2\}\\bigl\(\\varepsilon\-\\log\(1\+\\varepsilon\)\\bigr\)\\right\]=14​𝔼​\[ε2\]\+O​\(𝔼​\[\|ε\|3\]\)\\displaystyle=\\tfrac\{1\}\{4\}\\,\\mathbb\{E\}\[\\varepsilon^\{2\}\]\+O\\\!\\left\(\\mathbb\{E\}\[\|\\varepsilon\|^\{3\}\]\\right\)\(62\)=α24​\(1\+α\)2⋅12​n\+O​\(n−3/2\)\.\\displaystyle=\\frac\{\\alpha^\{2\}\}\{4\(1\+\\alpha\)^\{2\}\}\\cdot\\frac\{1\}\{2n\}\+O\\\!\\left\(n^\{\-3/2\}\\right\)\.\(63\)This yields \([19](https://arxiv.org/html/2605.21798#S3.E19)\)\. ∎

## Appendix CNumerical Verification of the Amortization Gap

We verify[Proposition1](https://arxiv.org/html/2605.21798#Thmproposition1)and[Proposition2](https://arxiv.org/html/2605.21798#Thmproposition2)numerically\. All experiments use the squared\-exponential kernel on\[0,1\]\[0,1\]with lengthscaleℓ=0\.3\\ell=0\.3, eigenfunctionsej​\(x\)=2​cos⁡\(j​π​x\)e\_\{j\}\(x\)=\\sqrt\{2\}\\cos\(j\\pi x\), eigenvaluesλj=exp⁡\(−12​\(ℓ​j​π\)2\)\\lambda\_\{j\}=\\exp\(\-\\frac\{1\}\{2\}\(\\ell\\,j\\pi\)^\{2\}\), andσd2=1\\sigma\_\{d\}^\{2\}=1\.

### C\.1Scalar case Formula verification

Table[3](https://arxiv.org/html/2605.21798#A3.T3)compares the closed\-form prediction \([19](https://arxiv.org/html/2605.21798#S3.E19)\) with Monte Carlo estimates \(100,000 draws ofXXi\.i\.d\.Uniform​\[0,1\]n\\mathrm\{Uniform\}\[0,1\]^\{n\}per row\)\. The formula is accurate to within10%10\\%forn=5n=5and within1%1\\%forn≥50n\\geq 50\. The ratio converges to 1, confirming that theO​\(1/n\)O\(1/n\)remainder is genuine\.

Table 3:Scalar amortization gap formula vs\. Monte Carlo\.The asymptotic prediction𝔼​\[Aσ\]→1/\(8​n\)\\mathbb\{E\}\[A\_\{\\sigma\}\]\\to 1/\(8n\)is verified in the rightmost column:𝔼​\[Aσ\]/\(1/\(8​n\)\)\\mathbb\{E\}\[A\_\{\\sigma\}\]/\(1/\(8n\)\)approaches 1 monotonically from below \(0\.970 atn=100n=100, 0\.998 atn=2000n=2000\)\.

### C\.2Uncorrelatedness ofv^\\hat\{v\}ande¯\\bar\{e\}

The key structural claim isCov​\(e1​\(x\)2,e1​\(x\)\)=𝔼μ​\[e13\]=0\\mathrm\{Cov\}\(e\_\{1\}\(x\)^\{2\},e\_\{1\}\(x\)\)=\\mathbb\{E\}\_\{\\mu\}\[e\_\{1\}^\{3\}\]=0\. Table[4](https://arxiv.org/html/2605.21798#A3.T4)reports the empirical correlationρ^​\(v^,e¯\)\\hat\{\\rho\}\(\\hat\{v\},\\bar\{e\}\)over 50,000 Monte Carlo draws\.

Table 4:Empirical correlation betweenv^\\hat\{v\}ande¯\\bar\{e\}\.The correlation is indistinguishable from zero at all sample sizes, andVar​\[v^\]\\mathrm\{Var\}\[\\hat\{v\}\]matches1/\(2​n\)1/\(2n\)to four significant figures\.

### C\.3Example with identical representations different posteriors

Ford=1d=1, any pair\(x1,x2\)=\(a,1−a\)\(x\_\{1\},x\_\{2\}\)=\(a,1\-a\)yieldsϕ¯=0\\bar\{\\phi\}=0by the symmetrye1​\(a\)\+e1​\(1−a\)=2​\[cos⁡\(π​a\)\+cos⁡\(π​\(1−a\)\)\]=0e\_\{1\}\(a\)\+e\_\{1\}\(1\-a\)=\\sqrt\{2\}\[\\cos\(\\pi a\)\+\\cos\(\\pi\(1\-a\)\)\]=0\. Yet the posterior precisionΣp−1=1\+λ1σd2​\[e1​\(x1\)2\+e1​\(x2\)2\]\\Sigma\_\{p\}^\{\-1\}=1\+\\frac\{\\lambda\_\{1\}\}\{\\sigma\_\{d\}^\{2\}\}\[e\_\{1\}\(x\_\{1\}\)^\{2\}\+e\_\{1\}\(x\_\{2\}\)^\{2\}\]varies substantially\.

Table 5:Contexts with identical mean representationϕ¯=0\\bar\{\\phi\}=0but different posteriors\.The amortized encoder must map all five contexts to the same variational parameters, since they share the representationϕ¯=0\\bar\{\\phi\}=0\. Using the average covariance as a compromise incurs an average KL of 0\.104 between the extreme cases\. This illustrates the mechanism that mean aggregation cannot distinguish clustered context points \(near 0\.5, providing little information\) from spread out context points \(near 0 and 1, providing much more information\)\.

### C\.4Mean aggregation vs\. second\-order aggregation

Table[6](https://arxiv.org/html/2605.21798#A3.T6)compares the amortization gap under mean and second\-order aggregation ford=3d=3, estimated via linear regression ofΣp\\Sigma\_\{p\}on the representation \(2,000 Monte Carlo contexts per row\)\. The gap ratio grows linearly innn, confirming the scaling separation:O​\(1/n\)O\(1/n\)for mean aggregation vs\.O​\(1/n2\)O\(1/n^\{2\}\)for second\-order\.

Table 6:Amortization gap of mean vs\. second\-order aggregation \(d=3d=3\)\.Fitting power laws, the mean aggregation gap decays asn−0\.81n^\{\-0\.81\}\(consistent withO​\(1/n\)O\(1/n\)up to the linear\-predictor lower bound\), while the second\-order gap decays asn−1\.75n^\{\-1\.75\}\(consistent withO​\(1/n2\)O\(1/n^\{2\}\)\)\.

### C\.5Scaling with representation dimension

Table[7](https://arxiv.org/html/2605.21798#A3.T7)shows how the gap depends onddfor fixedn=50n=50\. The mean aggregation gap initially grows withdd\(more second\-order structure to miss\) then declines \(eigenvalue decay reduces the sensitivity\)\. The second\-order gap remains small throughout, confirming that it is a sufficient statistic for the head kernel matrix\.

Table 7:Amortization gap vs\. representation dimension \(n=50n=50\)\.Atd=8d=8the two methods converge, because the eigenvaluesλj\\lambda\_\{j\}forj\>3j\>3are small enough under the SE kernel \(ℓ=0\.3\\ell=0\.3\) that the corresponding second\-order terms contribute negligibly, and both representations capture the relevant structure equally well\. This is consistent with the effective dimension scaling predicted by[Proposition2](https://arxiv.org/html/2605.21798#Thmproposition2)\.

## Appendix DEmpirical Verification of[Theorem1](https://arxiv.org/html/2605.21798#Thmtheorem1)

We verify the two\-component decomposition of[Theorem1](https://arxiv.org/html/2605.21798#Thmtheorem1)on a trained LNP\. The theorem predicts

𝔼C​\[Var𝒚\|X​\[σLNP2​\(x∗;C\)\]\]≤LΣ2​Bw4​Bψ2​\(σϵ2n⏟noise​δϵ\+κk⏟signal​δf\),\\mathbb\{E\}\_\{C\}\\\!\\left\[\\mathrm\{Var\}\_\{\\bm\{y\}\|X\}\\\!\\left\[\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\(x\_\{\*\};C\)\\right\]\\right\]\\leq L\_\{\\Sigma\}^\{2\}B\_\{w\}^\{4\}B\_\{\\psi\}^\{2\}\\\!\\left\(\\underbrace\{\\frac\{\\sigma\_\{\\epsilon\}^\{2\}\}\{n\}\}\_\{\\text\{noise \}\\delta\_\{\\epsilon\}\}\+\\underbrace\{\\kappa\_\{k\}\}\_\{\\text\{signal \}\\delta\_\{f\}\}\\right\),\(64\)with the signal componentO​\(1\)O\(1\)and irreducible, and the noise componentO​\(1/n\)O\(1/n\)and vanishing\.

### D\.1Setup

We train a small LNP on samples from𝒢​𝒫​\(0,kSE\)\\mathcal\{GP\}\(0,k\_\{\\mathrm\{SE\}\}\)with lengthscaleℓ=0\.2\\ell=0\.2, signal variance11, and observation noiseσϵ2=0\.05\\sigma\_\{\\epsilon\}^\{2\}=0\.05on𝒳=\[0,1\]\\mathcal\{X\}=\[0,1\]\. The architecture follows[Definition1](https://arxiv.org/html/2605.21798#Thmdefinition1)but uses MLP components rather than the analytical idealizations of Assumptions[2](https://arxiv.org/html/2605.21798#Thmassumption2)and[4](https://arxiv.org/html/2605.21798#Thmassumption4):

- •Encoder:h​\(x,y\)=MLP​\(\[x,y\]\)→ℝ64h\(x,y\)=\\mathrm\{MLP\}\(\[x,y\]\)\\to\\mathbb\{R\}^\{64\}\(two hidden layers of width6464, ReLU\)\. Notably,hhis non\-affine inyy, so[2](https://arxiv.org/html/2605.21798#Thmassumption2)is violated\.
- •Aggregation: mean\.
- •Recognition:\(μz,log⁡σz2\)=\(Linear​\(rC\),Linear​\(rC\)\)\(\\mu\_\{z\},\\log\\sigma\_\{z\}^\{2\}\)=\(\\mathrm\{Linear\}\(r\_\{C\}\),\\mathrm\{Linear\}\(r\_\{C\}\)\)withz∈ℝ32z\\in\\mathbb\{R\}^\{32\}and diagonal covariance\.
- •Decoder:MLP​\(\[x∗,z\]\)→\(μ,softplus​\(ν\)\)\\mathrm\{MLP\}\(\[x\_\{\*\},z\]\)\\to\(\\mu,\\mathrm\{softplus\}\(\\nu\)\)withσd2=0\.05\\sigma\_\{d\}^\{2\}=0\.05floor on the output variance\.

Training uses the standard NP ELBO over context/target splits with sizes drawn uniformly from\{5,…,49\}\\\{5,\\dots,49\\\}, batch size1616, Adam at learning rate10−310^\{\-3\},40004000steps\. Using a non\-affine encoder is intentional: it tests whether the qualitative two\-component prediction of[Theorem1](https://arxiv.org/html/2605.21798#Thmtheorem1)survives the violation of[2](https://arxiv.org/html/2605.21798#Thmassumption2)\.

### D\.2Protocol

To isolate the two components of \([64](https://arxiv.org/html/2605.21798#A4.E64)\) we evaluate the trained model under two label\-resampling protocols at fixed context locationsXX:

Full \(δf\+δϵ\\delta\_\{f\}\+\\delta\_\{\\epsilon\}\)\.For each context location setXX, drawf∼𝒢​𝒫f\\sim\\mathcal\{GP\}andϵ∼𝒩​\(0,σϵ2​I\)\\epsilon\\sim\\mathcal\{N\}\(0,\\sigma\_\{\\epsilon\}^\{2\}I\)independently acrossR=400R=400resamples; sety\(r\)=f\(r\)​\(X\)\+ϵ\(r\)y^\{\(r\)\}=f^\{\(r\)\}\(X\)\+\\epsilon^\{\(r\)\}\. Both label\-dependent components contribute\.

Noise\-only \(δϵ\\delta\_\{\\epsilon\}\)\.Draw a single GP functionf0f\_\{0\}, fix it across resamples, and resample onlyϵ\(r\)\\epsilon^\{\(r\)\}; sety\(r\)=f0​\(X\)\+ϵ\(r\)y^\{\(r\)\}=f\_\{0\}\(X\)\+\\epsilon^\{\(r\)\}\. Only the noise component contributes\.

For each protocol we compute the empirical variance ofσLNP2​\(x∗=0\.5;X,y\(r\)\)\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}\(x\_\{\*\}=0\.5;X,y^\{\(r\)\}\)across theRRlabel resamples, then average over3030independent context location setsXX\(withxi∼Uniform​\[0,1\]x\_\{i\}\\sim\\mathrm\{Uniform\}\[0,1\]\)\. The marginal predictive variance is computed as𝔼z​\[Var​\(y∗\|z\)\]\+Varz​\[𝔼​\(y∗\|z\)\]\\mathbb\{E\}\_\{z\}\[\\mathrm\{Var\}\(y\_\{\*\}\|z\)\]\+\\mathrm\{Var\}\_\{z\}\[\\mathbb\{E\}\(y\_\{\*\}\|z\)\]using6464samples fromq​\(z∣C\)q\(z\\mid C\)\.

### D\.3Results

Table 8:Conditional variance of the LNP predictive variance under the two protocols\. The Full protocol measuresδf\+δϵ\\delta\_\{f\}\+\\delta\_\{\\epsilon\}; the Noise\-only protocol measuresδϵ\\delta\_\{\\epsilon\}alone\. The Floor/Noise ratio quantifies how much of the contamination is irreducible at eachnn\.![Refer to caption](https://arxiv.org/html/2605.21798v1/x1.png)Figure 2:Empirical verification of[Theorem1](https://arxiv.org/html/2605.21798#Thmtheorem1)\. The Full protocol \(red\) plateaus at anO​\(1\)O\(1\)floor across two orders of magnitude innn\. The Noise\-only protocol \(blue\) decays as1/n1/non the early\-nnside, consistent with theσϵ2/n\\sigma\_\{\\epsilon\}^\{2\}/nterm, before saturating at the latent\-zzMonte Carlo floor of the variance estimator\.Fitting power lawsVar∝nα\\mathrm\{Var\}\\propto n^\{\\alpha\}on the early\-nnrangen∈\{5,10,20,50,100\}n\\in\\\{5,10,20,50,100\\\}where the noise contribution dominates the latent\-zzMonte Carlo floor:

- •Noise\-only protocol:α=−0\.78\\alpha=\-0\.78\(theory:−1\-1\)\.
- •Full protocol:α=−0\.09\\alpha=\-0\.09\(theory:→0\\to 0\)\.

### D\.4Interpretation

#### Signal floor\.

The full protocol plateaus near0\.210\.21acrossn∈\[5,1000\]n\\in\[5,1000\], providing direct empirical evidence for theO​\(1\)O\(1\)signal floor predicted by[Theorem1](https://arxiv.org/html/2605.21798#Thmtheorem1)\. This contamination does not vanish with increasing context size\. Atn=1000n=1000, label dependence still injects variance of orderσLNP2\\sigma^\{2\}\_\{\\mathrm\{LNP\}\}itself \(which isO​\(10−1\)O\(10^\{\-1\}\)atx∗=0\.5x\_\{\*\}=0\.5on this trained model\)\.

#### Noise decay\.

The noise\-only protocol decays from1\.5×10−21\.5\\times 10^\{\-2\}atn=5n=5to∼8×10−4\\sim 8\\times 10^\{\-4\}atn=1000n=1000, consistent with the predictedσϵ2/n\\sigma\_\{\\epsilon\}^\{2\}/nscaling\. The fitted slope of−0\.78\-0\.78is shallower than the theoretical−1\-1, and this is consistent with a finite\-sample floor introduced by the latent\-zzMonte Carlo expectation in the marginal predictive \(computed here with6464samples per evaluation\), which lower\-bounds the measurable variance at roughly10−310^\{\-3\}on this network\. The early\-nnportion, where the signal\-vs\-MC ratio is largest, gives the cleanest test of the rate\.

#### Regime change\.

The Floor/Noise ratio grows from20×20\\timesatn=5n=5to256×256\\timesatn=1000n=1000, demonstrating exactly the regime change the theorem predicts: at smallnnboth components contribute non\-trivially, while at largennthe signal floor entirely dominates the contamination\.

#### Robustness to[2](https://arxiv.org/html/2605.21798#Thmassumption2)\.

The trained encoder is a non\-affine MLP, so[2](https://arxiv.org/html/2605.21798#Thmassumption2)is violated and the precise constants in[Theorem1](https://arxiv.org/html/2605.21798#Thmtheorem1)do not apply\. The qualitative two\-component decomposition nonetheless appears robustly\. This supports the interpretation of[SectionA\.3](https://arxiv.org/html/2605.21798#A1.SS3)\. The structural mechanism \(label contamination through any encoder that mixesxxandyyinputs into the variance pathway\) is general, even when the analytical decompositionrC=ϕ¯X\+δyr\_\{C\}=\\bar\{\\phi\}\_\{X\}\+\\delta\_\{y\}does not hold exactly\.

## Appendix EEmpirical Verification of[4](https://arxiv.org/html/2605.21798#Thmassumption4)

[4](https://arxiv.org/html/2605.21798#Thmassumption4)posits that the encoder learns the firstddkernel eigenfunctions, which is the optimaldd\-dimensional kernel approximation\. This is the strongest assumption in our analysis, as the standard NP training objective contains no explicit term encouraging Mercer alignment\. We test whether trained NP encoders approximately recover the Mercer eigenfunction subspace\.

### E\.1Setup

For each of three SE kernels on\[0,1\]\[0,1\]with lengthscalesℓ∈\{0\.1,0\.2,0\.5\}\\ell\\in\\\{0\.1,0\.2,0\.5\\\}\(signal variance11, noiseσϵ2=0\.05\\sigma\_\{\\epsilon\}^\{2\}=0\.05\), we train the same MLP\-based LNP architecture used in[AppendixD](https://arxiv.org/html/2605.21798#A4)\(encoder MLP→ℝ64\\to\\mathbb\{R\}^\{64\}, mean aggregation, latentz∈ℝ32z\\in\\mathbb\{R\}^\{32\}, MLP decoder\) via the standard ELBO for 4000 steps\.

#### Ground\-truth eigenfunctions\.

The top\-dmax=16d\_\{\\max\}=16Mercer eigenfunctions\{ej\}\\\{e\_\{j\}\\\}and eigenvalues\{λj\}\\\{\\lambda\_\{j\}\\\}of each kernel under the uniform measure on\[0,1\]\[0,1\]are computed via a Nyström decomposition on a dense grid ofN=2000N=2000equispaced points, with eigenfunctions normalized to unitL2L^\{2\}\-norm\.

#### Encoder features\.

The trained encoderh​\(x,y\):𝒳×𝒴→ℝ64h\(x,y\):\\mathcal\{X\}\\times\\mathcal\{Y\}\\to\\mathbb\{R\}^\{64\}produces a 64\-dimensional feature vector for each input pair\. To isolate thexx\-dependent subspace we evaluate the encoder in two modes:

y = 0:Sety=0y=0for all inputs, giving the direct readout of thexx\-pathway:h​\(x,0\)h\(x,0\)\.

Marginal:Averageh​\(x,y\)h\(x,y\)overy∼p​\(y∣x\)y\\sim p\(y\\mid x\)under the GP prior, approximated with 200 joint GP samples\. This is the theoretically cleaner “location\-only” representationϕ¯​\(x\)=𝔼y\|x​\[h​\(x,y\)\]\\bar\{\\phi\}\(x\)=\\mathbb\{E\}\_\{y\|x\}\[h\(x,y\)\]\.

#### Subspace alignment\.

For eachd∈\{1,…,16\}d\\in\\\{1,\\dots,16\\\}, we measure how well thedd\-dimensional Mercer eigenfunction subspacespan​\(e1,…,ed\)\\mathrm\{span\}\(e\_\{1\},\\dots,e\_\{d\}\)is contained in the 64\-dimensional encoder feature subspace\. Both subspaces are orthonormalized with respect to theL2L^\{2\}\-uniform inner product, and we compute the principal angles between them\. The minimum cosinecos⁡θmin\\cos\\theta\_\{\\min\}across theddprincipal angles equals11if and only if the Mercer subspace is exactly contained in the encoder subspace; values near11indicate near\-containment\.

#### Per\-axis recovery\.

For each eigenfunctioneje\_\{j\}individually, we computeRj2=‖PQ​ej‖2/‖ej‖2R^\{2\}\_\{j\}=\\\|P\_\{Q\}e\_\{j\}\\\|^\{2\}/\\\|e\_\{j\}\\\|^\{2\}, wherePQP\_\{Q\}is the orthogonal projector onto the encoder subspace\.Rj2=1R^\{2\}\_\{j\}=1meanseje\_\{j\}is exactly recoverable from the encoder features\.

### E\.2Results

Table 9:Per\-axisR2R^\{2\}for the top eigenfunction recovery by the trained encoder \(y=0y=0mode\) across three SE kernels\. Bold indicatesR2≥0\.95R^\{2\}\\geq 0\.95\.Table 10:Minimum cosine of principal angles between the top\-ddMercer subspace and the full encoder subspace \(y=0y=0mode\)\.cos⁡θmin=1\\cos\\theta\_\{\\min\}=1means exact containment\.
### E\.3Interpretation

#### Approximate Mercer alignment\.

The trained encoder recovers the top Mercer eigenfunctions with high fidelity withR2\>0\.95R^\{2\}\>0\.95for all eigenfunctions whose eigenvalues exceed approximately10−210^\{\-2\}\(Table[9](https://arxiv.org/html/2605.21798#A5.T9)\)\. The subspace containment analysis \(Table[10](https://arxiv.org/html/2605.21798#A5.T10)\) confirms this at the subspace level, withcos⁡θmin\>0\.9\\cos\\theta\_\{\\min\}\>0\.9ford≤6d\\leq 6\(ℓ=0\.2\\ell=0\.2\),d≤8d\\leq 8\(ℓ=0\.1\\ell=0\.1\), andd≤7d\\leq 7\(ℓ=0\.5\\ell=0\.5\)\.

#### Alignment tracks the spectrum\.

Recovery degrades where the eigenvalue spectrum becomes negligible relative to the observation noiseσϵ2=0\.05\\sigma\_\{\\epsilon\}^\{2\}=0\.05\. Forℓ=0\.5\\ell=0\.5, the spectrum drops below10−810^\{\-8\}atj=9j=9, andR2R^\{2\}simultaneously collapses from0\.790\.79to0\.020\.02\. Forℓ=0\.1\\ell=0\.1, the slower spectral decay means more eigenfunctions carry signal, and the encoder accordingly recovers more of them before alignment degrades\. This is consistent with the ELBO training objective implicitly encouraging the encoder to capture the directions of maximum variation under the GP prior which are the top Mercer eigenfunctions\.

#### The ELBO encourages Mercer alignment\.

No explicit spectral regularization is applied during training\. The alignment arises because the top eigenfunctions are the features that minimize prediction error under the GP prior\. GP samples vary primarily along the directionseje\_\{j\}with largeλj\\lambda\_\{j\}, so an encoder trained to predict well on GP samples recovers these directions\. This provides empirical justification for[4](https://arxiv.org/html/2605.21798#Thmassumption4)in the regime where the NP is trained on samples from a fixed GP prior, which is the regime our analysis targets\.

#### Implications for the bottleneck rates\.

Since the trained encoder approximately spans the top\-ddMercer subspace forddup to the effective spectral dimension of the kernel, the bottleneck rates in[Corollary1](https://arxiv.org/html/2605.21798#Thmcorollary1)are not properties of an idealized encoder but are approximately predictive of trained\-NP behavior\. The effective aligned dimension tracks the kernel lengthscale as the theory predicts\. Smoother kernels \(ℓ=0\.5\\ell=0\.5, faster eigenvalue decay\) require fewer encoder dimensions for high\-fidelity alignment than rougher kernels \(ℓ=0\.1\\ell=0\.1, slower decay\)\.