Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network

# Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network
Source: [https://arxiv.org/html/2609.01952](https://arxiv.org/html/2609.01952)
Tiyao LiuJinhao JingZeji LiKaijie ChenHarikrishna Kuttivelil\\correspondingKatia Obraczka\\corresponding

###### Abstract

Decentralized, serverless learning increasingly connects devices running different architectures, where the standard tool, decentralized SGD, is undefined as models with different parameter counts cannot be averaged\. Knowledge distillation \(KD\) exchanges soft predictions rather than weights and sidesteps this obstacle, yet convergence theory for fully decentralized, asynchronous peer\-to\-peer \(P2P\) KD is lacking\. We provide one, relocating consensus from parameter space to function \(output\) space: a KD event is a geometric contraction operator in logit space on the peers’ predictive distributions, which we analyse in the Hilbert space of predictions on a reference measure\. Under standard smoothness/variance assumptions and two realizability assumptions, one bridging parameter SGD to the functional step and one controlling restricted task/KD alignment, the time\-averaged functional stationarity and function\-space disagreement converge at rate𝒪⁡\(1/\(η​T\)\)\\mathcal\{O\}\(1/\(\\eta T\)\)to an𝒪⁡\(η\)\+𝒪⁡\(Bf2\)\+𝒪⁡\(ζf2\)\\mathcal\{O\}\(\\eta\)\+\\mathcal\{O\}\(B\_\{f\}^\{2\}\)\+\\mathcal\{O\}\(\\zeta\_\{f\}^\{2\}\)neighbourhood\. HereBfB\_\{f\}is the distance from the task optimum to the peers’ reachable classes andζf\\zeta\_\{f\}measures persistent local\-task heterogeneity\. Across homogeneous, width\-heterogeneous, and mixed\-family networks of the experiments, KD contracts function disagreement by4040–61×61\\times, while isolated training does not\. The sampled stationarity diagnostic has late\-transient exponents0\.990\.99–1\.901\.90on the shared\-skeleton main runs, and the four\-point step\-size sweep exhibits the predicted transient–neighbourhood tradeoff\.

1University of California, Santa Cruz, USA

2China University of Petroleum, China

3The Chinese University of Hong Kong, Shenzhen, China

4City University of Hong Kong, Hong Kong SAR, China

5University of Virginia, USA

## Introduction

Gossip learning removes the central server from distributed training\. Each peer keeps private data, performs local updates, and communicates with neighbours over a peer\-to\-peer \(P2P\) network without a global clock\. In decentralized SGD, the communication step is a weighted average of parameter vectors,xi←∑jWi​j​xjx\_\{i\}\\leftarrow\\sum\_\{j\}W\_\{ij\}x\_\{j\}\. This operation requires all peers to use the same parameter spaceℝd\\mathbb\{R\}^\{d\}\([Lian et al\. 2017](https://arxiv.org/html/2609.01952#bib.bib23);[Boyd et al\. 2006](https://arxiv.org/html/2609.01952#bib.bib6)\), so it is undefined when the network contains different architectures\.

Knowledge distillation \(KD\) provides a natural alternative because peers exchange soft predictions rather than parameters\. Predictions lie in the class simplex regardless of the network that produced them\. Federated distillation methods use this common output space, but the methods considered in this line of work rely on a coordinator or synchronized rounds\([Hinton, Vinyals, and Dean 2015](https://arxiv.org/html/2609.01952#bib.bib13);[Li and Wang 2019](https://arxiv.org/html/2609.01952#bib.bib20);[Lin et al\. 2020](https://arxiv.org/html/2609.01952#bib.bib24);[Chang et al\. 2019](https://arxiv.org/html/2609.01952#bib.bib7)\)and do not analyze the fully decentralized, asynchronous setting\. We explore whether an asynchronize P2P KD process admits a convergence theory when the peers have different reachable function classes\.

We measure consensus on a reference measureμ\\mu\. For peerii, letpi=softmax⁡\(zi/τ\)p\_\{i\}=\\mathrm\{softmax\}\(z\_\{i\}/\\tau\)denote the temperature\-scaled prediction\. The predictions belong to the common Hilbert spaceL2​\(μ,ℝC\)L^\{2\}\(\\mu;\\mathbb\{R\}^\{C\}\), where we measure disagreement byΨtf=1N​∑i𝔼​‖pi−p¯‖μ2\\Psi^\{f\}\_\{t\}=\\tfrac\{1\}\{N\}\\sum\_\{i\}\\mathbb\{E\}\\\|p\_\{i\}\-\\bar\{p\}\\\|\_\{\\mu\}^\{2\}\. A KD exchange has logit gradientτ⁡\(pi−pj\)\\tau\(p\_\{i\}\-p\_\{j\}\), so it acts as a function\-space mixing step\. Its contraction rate is proportional toρf=Θ⁡\(η​α​pmin​λ2​\(ℒ\)/\|E\|\)\\rho\_\{f\}=\\Theta\(\\eta\\alpha p\_\{\\min\}\\lambda\_\{2\}\(\\mathcal\{L\}\)/\|E\|\)\. We couple this contraction with descent of the task risk in a Lyapunov argument\. The proof uses A4 \(detailed and justified in Appendix[E](https://arxiv.org/html/2609.01952#A5), along with A10\) to relate a parameter update to its functional effect and A10 to control directions that are outside an individual peer’s reachable class\.

##### Contributions\.

We establish a convergence bound for asynchronous random\-edge P2P KD that controls both time\-averaged functional stationarity and function\-space disagreement within an𝒪⁡\(η\)\+𝒪⁡\(Bf2\)\+𝒪⁡\(ζf2\)\\mathcal\{O\}\(\\eta\)\+\\mathcal\{O\}\(B\_\{f\}^\{2\}\)\+\\mathcal\{O\}\(\\zeta\_\{f\}^\{2\}\)neighbourhood\. The three terms separate stochastic discretization, representational error, and persistent local\-task heterogeneity\. We characterize the function\-space communication operator that remains defined for heterogeneous peers and keep parameter averaging as a homogeneous\-model control\. The appendix gives a self\-contained proof and states the two modelling bridges explicitly, including sufficient verification models for A10\. In the experiments, we record the theorem’s observables and use isolated training and homogeneous D\-SGD only as mechanism controls\. The experiments do not make a task\-accuracy ranking claim or determineBfB\_\{f\}from architecture\.

### Related Work

##### Decentralized optimization and gossip SGD\.

Decentralized\-SGD theory covers non\-convex optimization with parameter averaging and has been extended to changing topology, directed graphs, data heterogeneity, and asynchronous updates\([Lian et al\. 2017](https://arxiv.org/html/2609.01952#bib.bib23);[Koloskova et al\. 2020](https://arxiv.org/html/2609.01952#bib.bib17);[Assran et al\. 2019](https://arxiv.org/html/2609.01952#bib.bib2)\)\. These analyses use local stochastic gradients in a shared parameter space\([Dandi et al\. 2022](https://arxiv.org/html/2609.01952#bib.bib8);[Sun, Daneshmand, and Scutari 2019](https://arxiv.org/html/2609.01952#bib.bib30)\)\. Thus, this mechanism is naturally undefined in a system with models that have different parameter size and architecture, while Knowledge Distillation offers a solution because it operates at output space\.

##### Knowledge distillation \(KD\) and federated KD\.

KD transfers a teacher’s soft predictions to a student\([Hinton, Vinyals, and Dean 2015](https://arxiv.org/html/2609.01952#bib.bib13)\)\. Because the exchanged object is a prediction, KD can connect models with different architectures\. FedMD distils on a shared public set\([Li and Wang 2019](https://arxiv.org/html/2609.01952#bib.bib20)\), FedDF uses server\-side ensemble distillation\([Lin et al\. 2020](https://arxiv.org/html/2609.01952#bib.bib24)\), and Cronus studies black\-box soft\-label exchange under malicious clients\([Chang et al\. 2019](https://arxiv.org/html/2609.01952#bib.bib7)\)\. These methods use a coordinator or a synchronized protocol\. Recent decentralized federated KD methods retain central coordination\([Taya et al\. 2022](https://arxiv.org/html/2609.01952#bib.bib31)\), including IMFL’s gradient aggregator\([Ying et al\. 2026](https://arxiv.org/html/2609.01952#bib.bib35)\)and TopoMoDistill’s synchronization server\([Wu et al\. 2026](https://arxiv.org/html/2609.01952#bib.bib34)\)\. Serverless gossip distillation\([Khowaja et al\. 2026](https://arxiv.org/html/2609.01952#bib.bib16)\)has so far been evaluated, but its general convergence behaviour is not established\. Our setting is the asynchronize serverless case, for which we provide the first convergence analysis in function space\. The more comprehensive related work is in Appendix[C](https://arxiv.org/html/2609.01952#A3)\.

## Preliminaries

### Important Notations

The full table can be found in Appendix[A](https://arxiv.org/html/2609.01952#A1)\.

### Core Assumptions

The standard assumptions listed in Appendix[D](https://arxiv.org/html/2609.01952#A4)follow those used in D\-SGD theory\([Zeng and Lei 2025](https://arxiv.org/html/2609.01952#bib.bib36);[Dandi et al\. 2022](https://arxiv.org/html/2609.01952#bib.bib8)\)\. For the assumptions specific to P2P KD, they are listed below\. For A4 and A10, though remaining hypotheses, they are well\-grounded assumptions that are strongly defended both mathematically and from related work that established on that in Appendix[E](https://arxiv.org/html/2609.01952#A5)\.

\(A3\) Representational capacity floor:A3 names the irreducible approximation error that appears when heterogeneous peers are asked to approach the same task\-optimal predictor\. Let𝒫i:=\{pθ:θ∈ℝdi\}⊆ℋ\\mathcal\{P\}\_\{i\}:=\\\{p\_\{\\theta\}:\\theta\\in\\mathbb\{R\}^\{d\_\{i\}\}\\\}\\subseteq\\mathcal\{H\}be the reachable prediction class of peeriion the reference measure, and letq⋆:=arg⁡minp∈𝒮⁡F⁡\(p\)q^\{\\star\}:=\\arg\\min\_\{p\\in\\mathcal\{S\}\}F\(p\)be the constrained task minimizer over𝒮:=\{p=softmax⁡\(z/τ\):‖z‖∞≤Gz\}\\mathcal\{S\}:=\\\{p=\\operatorname\{softmax\}\(z/\\tau\):\\\|z\\\|\_\{\\infty\}\\leq G\_\{z\}\\\}\. The oracle capacity floor is finite:

Bf2\\displaystyle B\_\{f\}^\{2\}:=1N​∑i=1Ndistμ⁡\(q⋆,𝒫i\)2,\\displaystyle:=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\operatorname\{dist\}\_\{\\mu\}\(q^\{\\star\},\\mathcal\{P\}\_\{i\}\)^\{2\},distμ⁡\(q,𝒫i\)\\displaystyle\\operatorname\{dist\}\_\{\\mu\}\(q,\\mathcal\{P\}\_\{i\}\):=infp∈𝒫i‖q−p‖μ\.\\displaystyle:=\\inf\_\{p\\in\\mathcal\{P\}\_\{i\}\}\\\|q\-p\\\|\_\{\\mu\}\.This floor bridges the gap between a common output\-space optimum and peer\-specific reachable function classes\. The proof usesBf2B\_\{f\}^\{2\}only as an approximation\-error parameter in the terminal neighbourhood\. Appendix[D](https://arxiv.org/html/2609.01952#A4)gives the constrained\-minimizer convention and explains whyBfB\_\{f\}is an oracle quantity\.

\(A4\) Parameter\-to\-logit realizability \(kernel geometry\):This states when a small parameter\-space SGD step can be interpreted as a controlled finite\-support step in logit and prediction space\. LetμM=M−1​∑r=1Mδxr\\mu\_\{M\}=M^\{\-1\}\\sum\_\{r=1\}^\{M\}\\delta\_\{x\_\{r\}\}be the empirical reference measure and let

ℋz:=\{Z∈ℝM×C:Zr,:𝟏=0\}\\mathcal\{H\}\_\{z\}:=\\\{Z\\in\\mathbb\{R\}^\{M\\times C\}:Z\_\{r,:\}\\mathbf\{1\}=0\\\}with theM−1M^\{\-1\}Frobenius inner product\. For peerii, define centered logitsZi​\(θi\)Z\_\{i\}\(\\theta\_\{i\}\), the logit JacobianJiz:=Dθi​Zi​\(θi\)J\_\{i\}^\{z\}:=D\_\{\\theta\_\{i\}\}Z\_\{i\}\(\\theta\_\{i\}\), the restricted logit kernelΘi:=Jiz​\(Jiz\)∗\\Theta\_\{i\}:=J\_\{i\}^\{z\}\(J\_\{i\}^\{z\}\)^\{\*\}, the tangent rangeTiz:=range⁡\(Jiz\)T\_\{i\}^\{z\}:=\\operatorname\{range\}\(J\_\{i\}^\{z\}\), and the orthogonal projectorΠiz:=ΠTiz\\Pi\_\{i\}^\{z\}:=\\Pi\_\{T\_\{i\}^\{z\}\}\. In the frozen finite\-support kernel region used by Theorem[2](https://arxiv.org/html/2609.01952#Thmtheorem2),

κ−​Πiz⪯Θi⪯κ\+​Πiz\.\\kappa\_\{\-\}\\Pi\_\{i\}^\{z\}\\preceq\\Theta\_\{i\}\\preceq\\kappa\_\{\+\}\\Pi\_\{i\}^\{z\}\.For the prediction JacobianJip:=Dθi​PiJ\_\{i\}^\{p\}:=D\_\{\\theta\_\{i\}\}P\_\{i\}and prediction kernelKi:=Jip​\(Jip\)∗K\_\{i\}:=J\_\{i\}^\{p\}\(J\_\{i\}^\{p\}\)^\{\*\}, with tangent rangeTipT\_\{i\}^\{p\}and projectorΠip\\Pi\_\{i\}^\{p\}, there are constants0<κ¯p≤κ¯p<∞0<\\underline\{\\kappa\}\_\{p\}\\leq\\overline\{\\kappa\}\_\{p\}<\\inftysuch that

κ¯p​Πip⪯Ki⪯κ¯p​Πip\.\\underline\{\\kappa\}\_\{p\}\\Pi\_\{i\}^\{p\}\\preceq K\_\{i\}\\preceq\\overline\{\\kappa\}\_\{p\}\\Pi\_\{i\}^\{p\}\.The parameter\-to\-function Taylor remainders and stochastic pushforward noise satisfy the conditional moment bounds used in Appendix[D](https://arxiv.org/html/2609.01952#A4)\. For the private\-batch to reference\-support task pushforward, write

Gitask\\displaystyle G\_\{i\}^\{\\rm task\}=Ki​git\+δit,\\displaystyle=K\_\{i\}g\_\{i\}^\{t\}\+\\delta\_\{i\}^\{t\},1N​∑i‖δit‖μ2\\displaystyle\\frac\{1\}\{N\}\\sum\_\{i\}\\\|\\delta\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}≤χG​1N​∑i‖git‖μ2\+χB​Bf2\+χζ​ζf2\.\\displaystyle\\leq\\chi\_\{G\}\\frac\{1\}\{N\}\\sum\_\{i\}\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\+\\chi\_\{B\}B\_\{f\}^\{2\}\+\\chi\_\{\\zeta\}\\zeta\_\{f\}^\{2\}\.Heregit=∇fFi​\(pit\)g\_\{i\}^\{t\}=\\nabla\_\{f\}F\_\{i\}\(p\_\{i\}^\{t\}\)\. This cross\-kernel fidelity clause is part of A4 and is not asserted by A6\. A4 bridges actual parameter SGD and the finite\-support functional recursion used in the proof\. It does not assert full\-space surjectivity and does not imply A10\. The chain\-rule identity, Taylor remainder, and kernel\-persistence details are deferred to Appendix[D](https://arxiv.org/html/2609.01952#A4)\(the assumption itself is further justified in Appendix[E](https://arxiv.org/html/2609.01952#A5)\) and Lemma[G1](https://arxiv.org/html/2609.01952#Thmlemma1a)\. \(A10\) Restricted task/KD alignment:This limits how much of the task gradient and directed KD pull lies outside the tangent directions that an individual peer can realize\. For every event timett, conditional onℱt\\mathcal\{F\}\_\{t\}, letgit:=∇fFi​\(pit\)g\_\{i\}^\{t\}:=\\nabla\_\{f\}F\_\{i\}\(p\_\{i\}^\{t\}\),ψtf:=N−1​∑i‖pit−p¯t‖μ2\\psi\_\{t\}^\{f\}:=N^\{\-1\}\\sum\_\{i\}\\\|p\_\{i\}^\{t\}\-\\bar\{p\}^\{t\}\\\|\_\{\\mu\}^\{2\}, andψtz:=N−1​∑i‖Zit−Z¯t‖μM2\\psi\_\{t\}^\{z\}:=N^\{\-1\}\\sum\_\{i\}\\\|Z\_\{i\}^\{t\}\-\\bar\{Z\}^\{t\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\. The task\-gradient normal component obeys

1N​∑i‖\(I−Πip,t\)​git‖μ2\\displaystyle\\frac\{1\}\{N\}\\sum\_\{i\}\\\|\(I\-\\Pi\_\{i\}^\{p,t\}\)g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}≤γtask​1N​∑i‖git‖μ2\+νtask,Ψ​ψtf\\displaystyle\\leq\\gamma\_\{\\mathrm\{task\}\}\\frac\{1\}\{N\}\\sum\_\{i\}\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\+\\nu\_\{\\mathrm\{task\},\\Psi\}\\psi\_\{t\}^\{f\}\+νtask,B​Bf2,0≤γtask<1\.\\displaystyle\+\\nu\_\{\\mathrm\{task\},B\}B\_\{f\}^\{2\},\\qquad 0\\leq\\gamma\_\{\\mathrm\{task\}\}<1\.For the KD clause, define the centered finite\-support pull

di​jz\\displaystyle d\_\{ij\}^\{z\}:=Ai​j​\(Zi−Zj\)=Pi−Pj,\\displaystyle:=A\_\{ij\}\(Z\_\{i\}\-Z\_\{j\}\)=P\_\{i\}\-P\_\{j\},Ai​j\\displaystyle A\_\{ij\}:=∫01DZ​softmax⁡\(Zj\+s⁡\(Zi−Zj\)τ\)​ds\.\\displaystyle:=\\int\_\{0\}^\{1\}D\_\{Z\}\\operatorname\{softmax\}\\left\(\\frac\{Z\_\{j\}\+s\(Z\_\{i\}\-Z\_\{j\}\)\}\{\\tau\}\\right\)\\,ds\.BothZi−ZjZ\_\{i\}\-Z\_\{j\}andPi−PjP\_\{i\}\-P\_\{j\}are represented in the class\-sum\-zero finite\-support ambient space, soΠiz\\Pi\_\{i\}^\{z\}acts ondi​jzd\_\{ij\}^\{z\}in that common coordinate system\. The directed KD\-pull normal components obey

1\|E→\|​∑\(i,j\)∈E→‖\(I−Πiz\)​di​jz‖μM2≤γKD​ψtz\+νKD,B​Bf2,E→:=\{\(i,j\),\(j,i\):\(i,j\)∈E\}\.\\begin\{split\}\\frac\{1\}\{\|\\vec\{E\}\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|\(I\-\\Pi\_\{i\}^\{z\}\)d\_\{ij\}^\{z\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\\leq\{\}&\\gamma\_\{\\mathrm\{KD\}\}\\psi\_\{t\}^\{z\}\+\\nu\_\{\\mathrm\{KD\},B\}B\_\{f\}^\{2\},\\\\ \\vec\{E\}:=\\\{\(i,j\),\(j,i\):\(i,j\)\\in E\\\}\.\\end\{split\}A10 bridges common output\-space KD pulls and peer\-specific tangent spaces\. The small\-gain conditions use only the transient coefficientsγtask\\gamma\_\{\\mathrm\{task\}\}andγKD\\gamma\_\{\\mathrm\{KD\}\}; the coefficients multiplyingBf2B\_\{f\}^\{2\}need only be finite\. Appendix Propositions[G1](https://arxiv.org/html/2609.01952#Thmproposition1a)–[G3](https://arxiv.org/html/2609.01952#Thmproposition3a)give sufficient cases and explain why the relative task\-gradient term is necessary \(the assumption itself is further justified in Appendix[E](https://arxiv.org/html/2609.01952#A5)\)\.

## System Architecture

We use the gossip\-distillation architecture shown in Figure[1](https://arxiv.org/html/2609.01952#Sx3.F1)\.

Figure 1:Asynchronous P2P knowledge\-distillation architecture\. Each peer alternates between the student and teacher roles as gossip pairs form stochastically\. No parameters are exchanged\.![Refer to caption](https://arxiv.org/html/2609.01952v1/pics/gossip_kd_architecture.png)### Network Topology and Heterogeneous Peers

The communication graph is an undirected connected graph𝒢=\(V,E\)\\mathcal\{G\}=\(V,E\)with peer setVVand edge setEE\. There areN=\|V\|N=\|V\|peers\. Peeriihas parametersθi∈ℝdi\\theta\_\{i\}\\in\\mathbb\{R\}^\{d\_\{i\}\}, and the dimensionsdid\_\{i\}may differ\. Parameter averaging is then undefined, so peers exchange output distributions\. Given an inputxx, peeriiproduces logitszi​\(x,θi\)∈ℝCz\_\{i\}\(x;\\theta\_\{i\}\)\\in\\mathbb\{R\}^\{C\}and the temperature\-scaled distributionpi​\(x,θi\)∈ΔC−1p\_\{i\}\(x;\\theta\_\{i\}\)\\in\\Delta^\{C\-1\}:

pi​\(x\)c=exp⁡\(zi​\(x\)c/τ\)∑k=1Cexp⁡\(zi​\(x\)k/τ\)p\_\{i\}\(x\)\_\{c\}=\\frac\{\\exp\(z\_\{i\}\(x\)\_\{c\}/\\tau\)\}\{\\sum\_\{k=1\}^\{C\}\\exp\(z\_\{i\}\(x\)\_\{k\}/\\tau\)\}\(1\)whereτ\>0\\tau\>0is the distillation temperature governing the softness of the distribution\.

### Random\-Edge Asynchrony and EMA Teachers

We use a random\-edge asynchrony model with no global clock\. At eventtt, edgee=\(i,j\)∈Ee=\(i,j\)\\in Eis activated and both endpoints act once as student and teacher\. The model captures uncoordinated activation but does not model concurrent writes\. A bounded number of such races can be represented by increasing the lag boundAAbelow\. Peerjjtransmits soft probabilities on a shared reference batch\. In the EMA variant, those probabilities come from an exponential moving average of its parameters, denoted bypjEMAp\_\{j\}^\{\\mathrm\{EMA\}\}\. The live\-network variant sets the EMA parameters equal to the current parameters\.

### The P2P\-KD Loss and Student Update

When edge\(i,j\)\(i,j\)is selected, peeriitakes an SGD step on a loss that combines the supervised task with a distillation pull toward peerjj:

ℒi​\(θi\)=\(1−α\)​ℓCE​\(pi​\(x,θi\),y\)\+ατ2DKL\(pjEMA\(x\)∥pi\(x;θi\)\),\\begin\{split\}\\mathcal\{L\}\_\{i\}\(\\theta\_\{i\}\)=\(1\-\\alpha\)\\,\\ell\_\{\\mathrm\{CE\}\}\\big\(p\_\{i\}\(x;\\theta\_\{i\}\),y\\big\)\+\\\\ \\alpha\\,\\tau^\{2\}\\,D\_\{\\mathrm\{KL\}\}\\big\(p\_\{j\}^\{\\mathrm\{EMA\}\}\(x\)\\parallel p\_\{i\}\(x;\\theta\_\{i\}\)\\big\),\\end\{split\}\(2\)HereℓCE\\ell\_\{\\mathrm\{CE\}\}is the cross\-entropy with labelyy,DKLD\_\{\\mathrm\{KL\}\}is the teacher\-to\-student divergence, andα∈\(0,1\)\\alpha\\in\(0,1\)controls their relative weight\.

Algorithm 1Asynchronous random\-edge P2P knowledge distillation0:peers

\{θi\}i=1N\\\{\\theta\_\{i\}\\\}\_\{i=1\}^\{N\}\(heterogeneous\), graph

𝒢=\(V,E\)\\mathcal\{G\}=\(V,E\), reference measure

μ\\mu, weight

α\\alpha, temperature

τ\\tau, step

η\\eta, EMA decay

β\\beta
1:initialise each

θi\\theta\_\{i\}\(local CE\-only pretraining sets

Ψ0f\\Psi^\{f\}\_\{0\}\), then set

θiEMA←θi\\theta\_\{i\}^\{\\mathrm\{EMA\}\}\\\!\\leftarrow\\\!\\theta\_\{i\}
2:forevent

t=0,1,2,…t=0,1,2,\\dotsdo

3:sample an active edge

\(i,j\)∼Unif⁡\(E\)\(i,j\)\\sim\\mathrm\{Unif\}\(E\)\(clock\)

4:draw a reference/training batch

xxand form teacher labels

pjT←softmax⁡\(zjEMA​\(x\)/τ\)p\_\{j\}^\{T\}\\\!\\leftarrow\\\!\\mathrm\{softmax\}\(z\_\{j\}^\{\\mathrm\{EMA\}\}\(x\)/\\tau\)\(live:

θjEMA=θj\\theta\_\{j\}^\{\\mathrm\{EMA\}\}\\\!=\\\!\\theta\_\{j\}\)

5:

θi←θi−η∇θi\[\(1−α\)ℓCE\(pi\(x\),y\)\+ατ2DKL\(pjT∥pi\(x\)\)\]\\theta\_\{i\}\\leftarrow\\theta\_\{i\}\-\\eta\\nabla\_\{\\theta\_\{i\}\}\\\!\\big\[\(1\-\\alpha\)\\ell\_\{\\mathrm\{CE\}\}\(p\_\{i\}\(x\),y\)\+\\alpha\\tau^\{2\}D\_\{\\mathrm\{KL\}\}\(p\_\{j\}^\{T\}\\\|p\_\{i\}\(x\)\)\\big\]
6:symmetrically update

θj\\theta\_\{j\}with teacher

piTp\_\{i\}^\{T\}\{both endpoints are student and teacher\}

7:

θiEMA←β​θiEMA\+\(1−β\)​θi\\theta\_\{i\}^\{\\mathrm\{EMA\}\}\\\!\\leftarrow\\\!\\beta\\theta\_\{i\}^\{\\mathrm\{EMA\}\}\+\(1\-\\beta\)\\theta\_\{i\}and update

jjlikewise

8:endfor

### Network Objective

Let𝒟i\\mathcal\{D\}\_\{i\}denote the private data distribution of peerii\. Its local risk is

Fi​\(p\):=𝔼\(x,y\)∼𝒟i​ℓCE​\(p⁡\(x\),y\),F\_\{i\}\(p\):=\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{i\}\}\\,\\ell\_\{\\mathrm\{CE\}\}\(p\(x\),y\),and the network objective is the average local risk:

F⁡\(p\):=1N∑i=1NFi​\(p\),F⋆:=infp∈ℋF⁡\(p\)\>−∞,ΔF:=F⁡\(p¯\(0\)\)−F⋆\\begin\{split\}F\(p\):=\\frac\{1\}\{N\}&\\sum\_\{i=1\}^\{N\}F\_\{i\}\(p\)\\ ,\\quad F^\{\\star\}:=\\inf\_\{p\\in\\mathcal\{H\}\}F\(p\)\>\-\\infty,\\\\ &\\Delta\_\{F\}:=F\(\\bar\{p\}^\{\(0\)\}\)\-F^\{\\star\}\\end\{split\}\(3\)The identity∇fF=1N​∑i∇fFi\\nabla\_\{\\\!f\}F=\\tfrac\{1\}\{N\}\\sum\_\{i\}\\nabla\_\{\\\!f\}F\_\{i\}follows from this definition\.

## Theoretical Convergence Analysis

We analyse the event process on the finite reference measureμM\\mu\_\{M\}and useμ:=μM\\mu:=\\mu\_\{M\}in all theorem norms below\. This is the space on which heterogeneous peers have a common state\. A4 separates the exact parameter\-to\-logit chain rule from the local frozen\-kernel condition, and A10 controls only the components that the peer\-specific tangent spaces cannot realize\. The proof uses the actual preconditioned KD update\.

### KD Geometry

###### Lemma 1\(KD logit gradient\)\.

ForPi=softmax⁡\(Zi/τ\)P\_\{i\}=\\operatorname\{softmax\}\(Z\_\{i\}/\\tau\)and a detached teacherPjTP\_\{j\}^\{T\},

∇Zi\[τ2DKL\(PjT∥Pi\)\]=τ\(Pi−PjT\)\.\\nabla\_\{Z\_\{i\}\}\\\!\\left\[\\tau^\{2\}D\_\{\\rm KL\}\(P\_\{j\}^\{T\}\\\|P\_\{i\}\)\\right\]=\\tau\(P\_\{i\}\-P\_\{j\}^\{T\}\)\.\(4\)Consequently, the first\-order logit update is−η​α​τ​Θi​\(Pi−PjT\)\-\\eta\\alpha\\tau\\Theta\_\{i\}\(P\_\{i\}\-P\_\{j\}^\{T\}\)\.

See Lemma[G1](https://arxiv.org/html/2609.01952#Thmlemma1a)for the chain\-rule and Taylor\-remainder proof\. The temperature factor in Equation[4](https://arxiv.org/html/2609.01952#Sx4.E4)is exact\.

###### Lemma 2\(Softmax geometry\)\.

Under A1, centered logits and probabilities satisfy

pminτ​‖Z−Z′‖μM≤‖P−P′‖μM≤12​τ​‖Z−Z′‖μM,\\frac\{p\_\{\\min\}\}\{\\tau\}\\\|Z\-Z^\{\\prime\}\\\|\_\{\\mu\_\{M\}\}\\leq\\\|P\-P^\{\\prime\}\\\|\_\{\\mu\_\{M\}\}\\leq\\frac\{1\}\{2\\tau\}\\\|Z\-Z^\{\\prime\}\\\|\_\{\\mu\_\{M\}\},\(5\)andτ2DKL\(P′∥P\)\\tau^\{2\}D\_\{\\rm KL\}\(P^\{\\prime\}\\\|P\)is bounded above and below by fixed multiples of‖Z−Z′‖μM2\\\|Z\-Z^\{\\prime\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\.

The constants and the induced equivalenceΨtf≍Ψtz\\Psi\_\{t\}^\{f\}\\asymp\\Psi\_\{t\}^\{z\}are proved in Lemma[G3](https://arxiv.org/html/2609.01952#Thmlemma3a)\.

### Global KD Contraction

For each frozen range\-restricted logit kernel define

Wi:=Θi†\+γ0​\(I−Πiz\),𝒱W​\(Z\):=minc∈ℋz⁡1N​∑i‖Zi−c‖Wi2,\\begin\{split\}W\_\{i\}&:=\\Theta\_\{i\}^\{\\dagger\}\+\\gamma\_\{0\}\(I\-\\Pi\_\{i\}^\{z\}\),\\\\ \\mathcal\{V\}\_\{W\}\(Z\)&:=\\min\_\{c\\in\\mathcal\{H\}\_\{z\}\}\\frac\{1\}\{N\}\\sum\_\{i\}\\\|Z\_\{i\}\-c\\\|\_\{W\_\{i\}\}^\{2\},\\end\{split\}\(6\)whereγ0\>0\\gamma\_\{0\}\>0\. The metric is positive definite and uniformly equivalent to arithmetic logit disagreement\. It also satisfiesWi​Θi=ΠizW\_\{i\}\\Theta\_\{i\}=\\Pi\_\{i\}^\{z\}\. We writeΨtz:=N−1​∑i‖Zit−Z¯t‖μM2\\Psi\_\{t\}^\{z\}:=N^\{\-1\}\\sum\_\{i\}\\\|Z\_\{i\}^\{t\}\-\\bar\{Z\}^\{t\}\\\|\_\{\\mu\_\{M\}\}^\{2\}for the unweighted logit disagreement\.

###### Theorem 1\(Random\-edge KD contraction with defect\)\.

LetGt=N−1​∑i‖git‖μ2G\_\{t\}=N^\{\-1\}\\sum\_\{i\}\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\. For a connected graph with no isolated peer, under A1, A4, and the KD clause of A10, there are constantsr0,CG,CB,Cζ,Cσ,CA\>0r\_\{0\},C\_\{G\},C\_\{B\},C\_\{\\zeta\},C\_\{\\sigma\},C\_\{A\}\>0, independent ofη\\etaandTT, such that

𝔼t​𝒱W​\(Zt\+1\)≤\(1−r0​η\)​𝒱W​\(Zt\)\+CG​η​Gt\+CB​η​Bf2\+Cζ​η​ζf2\+Cσ​η2\+CA​η​Atf,\\begin\{split\}\\mathbb\{E\}\_\{t\}\\mathcal\{V\}\_\{W\}\(Z^\{t\+1\}\)\\leq\{\}&\(1\-r\_\{0\}\\eta\)\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+C\_\{G\}\\eta G\_\{t\}\+C\_\{B\}\\eta B\_\{f\}^\{2\}\\\\ \+&C\_\{\\zeta\}\\eta\\zeta\_\{f\}^\{2\}\+C\_\{\\sigma\}\\eta^\{2\}\+C\_\{A\}\\eta A\_\{t\}^\{f\},\\end\{split\}\(7\)with coefficients:

CG\\displaystyle C\_\{G\}=RG​Kpre;CB=4​MW​dmax​νKD,B​α​τN⁡\(pmin/τ\)​λ2​mW\+RB​Kpre\\displaystyle=R\_\{G\}K\_\{\\rm pre\};\\quad C\_\{B\}=\\frac\{4M\_\{W\}d\_\{\\max\}\\nu\_\{\\rm KD,B\}\\alpha\\tau\}\{N\(p\_\{\\min\}/\\tau\)\\lambda\_\{2\}m\_\{W\}\}\+R\_\{B\}K\_\{\\rm pre\}Cζ\\displaystyle C\_\{\\zeta\}=Rζ​Kpre;Cη=R0​Kpre\+Σ0;CA=RA​α​τ​Kpre\\displaystyle=R\_\{\\zeta\}K\_\{\\rm pre\};\\quad C\_\{\\eta\}=R\_\{0\}K\_\{\\rm pre\}\+\\Sigma\_\{0\};\\quad C\_\{A\}=R\_\{A\}\\alpha\\tau K\_\{\\rm pre\}providedη≤ηKD\\eta\\leq\\eta\_\{\\rm KD\}andγKD\\gamma\_\{\\rm KD\}is below the explicit absorption threshold in Lemma[G6](https://arxiv.org/html/2609.01952#Thmlemma6)\. Moreover,

r0=α​pmin​λ2​\(ℒ\)2​MW​\|E\|,mW​I⪯Wi⪯MW​I\.r\_\{0\}=\\frac\{\\alpha p\_\{\\min\}\\lambda\_\{2\}\(\\mathcal\{L\}\)\}\{2M\_\{W\}\|E\|\},\\qquad m\_\{W\}I\\preceq W\_\{i\}\\preceq M\_\{W\}I\.\(8\)

The termCG​η​GtC\_\{G\}\\eta G\_\{t\}is critical\. A local CE step is a deterministic peer\-specific force and enters consensus at first order\. Under non\-IID data it cannot be relabelled asO⁡\(η2\)O\(\\eta^\{2\}\)\. The termCζ​ζf2C\_\{\\zeta\}\\zeta\_\{f\}^\{2\}in the final neighbourhood records this persistent task heterogeneity\.

### Task Descent and Staleness

Uniform edge sampling activates peeriiwith probabilityqi=di/\|E\|q\_\{i\}=d\_\{i\}/\|E\|\. Define the activation\-corrected task potential

ℛt:=1N​∑iqi−1​\(Fi​\(pit\)−Fi⋆\)\.\\mathcal\{R\}\_\{t\}:=\\frac\{1\}\{N\}\\sum\_\{i\}q\_\{i\}^\{\-1\}\\big\(F\_\{i\}\(p\_\{i\}^\{t\}\)\-F\_\{i\}^\{\\star\}\\big\)\.\(9\)The inverse activation weights make the conditional edge expectation a uniform peer average even when the graph is not regular\.

###### Lemma 3\(Functional task descent\)\.

Under A2, A4, A6, and the task clause of A10, there are constantsaF,bF,DB,Dζ,Dσ,DA\>0a\_\{F\},b\_\{F\},D\_\{B\},D\_\{\\zeta\},D\_\{\\sigma\},D\_\{A\}\>0such that

𝔼t​ℛt\+1≤ℛt−aF​η​Gt\+bF​η​𝒱W​\(Zt\)\+DB​η​Bf2\+Dζ​η​ζf2\+Dσ​η2\+DA​η​Atf,\\begin\{split\}\\mathbb\{E\}\_\{t\}\\mathcal\{R\}\_\{t\+1\}\\leq\{\}&\\mathcal\{R\}\_\{t\}\-a\_\{F\}\\eta G\_\{t\}\+b\_\{F\}\\eta\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+D\_\{B\}\\eta B\_\{f\}^\{2\}\\\\ &\+D\_\{\\zeta\}\\eta\\zeta\_\{f\}^\{2\}\+D\_\{\\sigma\}\\eta^\{2\}\+D\_\{A\}\\eta A\_\{t\}^\{f\},\\end\{split\}\(10\)
whereaF\>0a\_\{F\}\>0when the task margin, including the factor\(1−α\)\(1\-\\alpha\)and the cross\-kernel residual constants from A4, is positive; the theorem assumes0≤α<10\\leq\\alpha<1\. Furthermore,

‖∇fF​\(p¯t\)‖μ2≤2​Gt\+2​Lf2​Ψtf\.\\\|\\nabla\_\{f\}F\(\\bar\{p\}^\{t\}\)\\\|\_\{\\mu\}^\{2\}\\leq 2G\_\{t\}\+2L\_\{f\}^\{2\}\\Psi\_\{t\}^\{f\}\.\(11\)

Lemma[G8](https://arxiv.org/html/2609.01952#Thmlemma8)proves these statements directly through⟨gi,Ki​gi⟩\\langle g\_\{i\},K\_\{i\}g\_\{i\}\\rangle; noη\\eta\-independent conditioning error is absorbed into anO⁡\(η\)O\(\\eta\)remainder\.

###### Lemma 4\(Simplex\-bounded staleness\)\.

If the EMA teacher is at mostAAversions behind, then

Atf≤A¯f=cstep​\(min⁡\{A,11−β\}\)2​η2\.A\_\{t\}^\{f\}\\leq\\bar\{A\}^\{f\}=c\_\{\\rm step\}\\left\(\\min\\left\\\{A,\\frac\{1\}\{1\-\\beta\}\\right\\\}\\right\)^\{2\}\\eta^\{2\}\.\(12\)

The proof is Lemma[G5](https://arxiv.org/html/2609.01952#Thmlemma5a)\. It uses the simplex diameter and does not feed consensus back into the lag bound\.

### Coupled Bound

###### Lemma 5\(Strict coupled descent\)\.

Suppose there existsλ\>0\\lambda\>0such that

aF−λ​CG\>0,λ​r0−bF\>0\.a\_\{F\}\-\\lambda C\_\{G\}\>0,\\qquad\\lambda r\_\{0\}\-b\_\{F\}\>0\.\(13\)Forℒt:=ℛt\+λ​𝒱W​\(Zt\)\\mathcal\{L\}\_\{t\}:=\\mathcal\{R\}\_\{t\}\+\\lambda\\mathcal\{V\}\_\{W\}\(Z^\{t\}\), Equations[7](https://arxiv.org/html/2609.01952#Sx4.E7)and[10](https://arxiv.org/html/2609.01952#Sx4.E10)imply

𝔼t​ℒt\+1≤ℒt−cG​η​Gt−cV​η​𝒱W​\(Zt\)\+C​η​\(Bf2\+ζf2\)\+C′​η2\.\\mathbb\{E\}\_\{t\}\\mathcal\{L\}\_\{t\+1\}\\leq\\mathcal\{L\}\_\{t\}\-c\_\{G\}\\eta G\_\{t\}\-c\_\{V\}\\eta\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+C\\eta\(B\_\{f\}^\{2\}\+\\zeta\_\{f\}^\{2\}\)\+C^\{\\prime\}\\eta^\{2\}\.\(14\)for constantscG,cV\>0c\_\{G\},c\_\{V\}\>0\. The staleness contribution is absorbed intoC′​η2C^\{\\prime\}\\eta^\{2\}by Equation[12](https://arxiv.org/html/2609.01952#Sx4.E12)\.

See Lemma[G9](https://arxiv.org/html/2609.01952#Thmlemma9)\. Condition[13](https://arxiv.org/html/2609.01952#Sx4.E13)states the required balance between task drift and KD mixing; it is not implied by a small step size\.

###### Theorem 2\(Main convergence result\)\.

Under A1–A10,0<α<10<\\alpha<1, a connected graph with no isolated peers, the frozen\-kernel condition in A4, the alignment margins in A10, Condition[13](https://arxiv.org/html/2609.01952#Sx4.E13), and0<η≤ηmax0<\\eta\\leq\\eta\_\{\\max\}, there are constantsc,C0,C1,C2,C3\>0c,C\_\{0\},C\_\{1\},C\_\{2\},C\_\{3\}\>0such that

1T∑t=0T−1\[𝔼​‖∇fF​\(p¯t\)‖μ2\+c​Ψtf\]≤C0η​T\+C1​η\+C2​Bf2\+C3​ζf2withK∗=max⁡\{2cG,2​Lf2​Cf​z/mW\+c​Cf​z/mWcV\}C0=K∗ℒ0;C1=K∗Kη;C2=K∗KB;C3=K∗Kζ\.\\begin\{split\}\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}&\\left\[\\mathbb\{E\}\\\|\\nabla\_\{f\}F\(\\bar\{p\}^\{t\}\)\\\|\_\{\\mu\}^\{2\}\+c\\Psi\_\{t\}^\{f\}\\right\]\\leq\\frac\{C\_\{0\}\}\{\\eta T\}\+C\_\{1\}\\eta\+C\_\{2\}B\_\{f\}^\{2\}\\\\ &\\qquad\\qquad\\qquad\\qquad\\qquad\+C\_\{3\}\\zeta\_\{f\}^\{2\}\\\\ \\text\{with\}&\\quad K\_\{\*\}=\\max\\left\\\{\\frac\{2\}\{c\_\{G\}\},\\frac\{2L\_\{f\}^\{2\}C\_\{fz\}/m\_\{W\}\+cC\_\{fz\}/m\_\{W\}\}\{c\_\{V\}\}\\right\\\}\\\\ &C\_\{0\}=K\_\{\*\}\\mathcal\{L\}\_\{0\};\\quad C\_\{1\}=K\_\{\*\}K\_\{\\eta\};\\\\ &C\_\{2\}=K\_\{\*\}K\_\{B\};\\quad C\_\{3\}=K\_\{\*\}K\_\{\\zeta\}\.\\end\{split\}\(15\)

The transient constant contains the initial task potential and disagreement asC1C\_\{1\}contains stochastic\-gradient, curvature, and bounded\-lag constants\. A stable\-kernel extension holds only when its metric/projector drift coefficient is strictly smaller than the contraction margin\. The complete proof is Theorem[G1](https://arxiv.org/html/2609.01952#Thmtheorem1a)\. The additionalO⁡\(ζf2\)O\(\\zeta\_\{f\}^\{2\}\)term is unavoidable for the implemented non\-IID local CE updates: even withBf=0B\_\{f\}=0and zero stochastic noise, two peers with different quadratic objectives can have a nonzero consensus fixed point independent ofη\\eta\. The theorem therefore separates stochastic discretization, capacity, and deterministic task heterogeneity rather than assigning all three effects to the step size\.

## Experiments

The experiments test the observable implications of Theorem[2](https://arxiv.org/html/2609.01952#Thmtheorem2)\. They are not designed as a benchmark against specialized decentralized optimizers, so we are not trying to beat the benchmark or other models\. The completed campaign contains 25 trajectories and records function\-space consensus, a finite\-reference stationarity proxy, rate analysis, and task accuracy every 1,000 edge activations\.

We perform the experiments to provide insights into the consensus error convergence and error neighbourhood of gossip P2P KD under the real world setting\. This way, we can tell if the proposed theoretical convergence rate of𝒪⁡\(1/η​T\)\\mathcal\{O\}\(1/\\eta T\)is justified, and further, if the bound for convergence rate is tight\. In addition, does the stationary proxy1T​‖∇F‖2\\frac\{1\}\{T\}\|\|\\nabla F\|\|^\{2\}decreases during the event\-indexed transient\. Finally, will heterogeneous settings converge in the same qualitative sense even though a parameter\-average D\-SGD control is undefined when peer dimensions and architecture differs\.

Figure 2:Task accuracy for the width\-heterogeneous Setting B withη=0\.05\\eta=0\.05, seed00\.Table 1:Completed main KD runs\.TTis the actual stored horizon,b^Ψ\\widehat\{b\}\_\{\\Psi\}andb^G\\widehat\{b\}\_\{G\}are final\-60%1/T1/T\-regression intercepts\.G^T\\widehat\{G\}\_\{T\}is Equation[17](https://arxiv.org/html/2609.01952#Sx5.E17), andp~dyn\\widetilde\{p\}\_\{\\rm dyn\}is the late\-transient median\.### Protocol and Observables

We trainN=20N=20CIFAR\-10\([Krizhevsky, Nair, and Hinton 2009](https://arxiv.org/html/2609.01952#bib.bib18)\)peers whose private shards follow a Dirichlet label partition with concentrationγpart=0\.3\\gamma\_\{\\rm part\}=0\.3\. Each seed fixes a connectedER⁡\(0\.3\)\\operatorname\{ER\}\(0\.3\)graph\. Seeds 0 and 1 have 51 and 58 edges and spectral gaps 1\.232 and 1\.822, respectively\. Setting A contains 20 medium\-width ResNets\. Settings B and C use width rosters77s/77m/66l and77xs/77m/66xl for ResNets\. Setting D mixes ResNet\-18, MobileNetV2\([Dong et al\. 2020](https://arxiv.org/html/2609.01952#bib.bib10);[Sandler et al\. 2018](https://arxiv.org/html/2609.01952#bib.bib28)\), and ShuffleNetV2\([Ma et al\. 2018](https://arxiv.org/html/2609.01952#bib.bib25)\)\. Every model receives 1,500 local CE pretraining steps\. Gossip then runs for 70,000 events with a live teacher, KD weightα=0\.5\\alpha=0\.5, temperatureτ=4\\tau=4, batch size 64, and no gossip momentum\.

Main runs useη=0\.05\\eta=0\.05and two seeds; the seed\-0 sweep usesη∈\{0\.0125,0\.025,0\.05,0\.1\}\\eta\\in\\\{0\.0125,0\.025,0\.05,0\.1\\\}\. Appendix[H](https://arxiv.org/html/2609.01952#A8)gives the complete configuration and run inventory, including all of the results of the experiments alongside visualization of those experiments\. The complete code system, experiment log and README are in code and data supplement for reproduction\.

The reference measureμ\\muis the empirical measure on a fixed 2,000\-image CIFAR\-10 test subset\. Logits are clipped at 10 and parameter gradients at norm 5\. These controls instantiate the bounded finite\-support setting used by A1 and the moment bounds in A4, while the experiments measure the consequences of A4 and A10 rather than treating either assumption as an empirical conclusion\.

At eventtt, the consensus statistic is

Ψ^tf=1N​∑i=1N‖pit−p¯t‖μ2,p¯t=1N​∑i=1Npit\.\\widehat\{\\Psi\}\_\{t\}^\{f\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\\\|p\_\{i\}^\{t\}\-\\bar\{p\}^\{t\}\\right\\\|\_\{\\mu\}^\{2\},\\qquad\\bar\{p\}^\{t\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}p\_\{i\}^\{t\}\.\(16\)

### Rate Diagnostics

##### Decay \(Consensus\) Speed

Figure[3](https://arxiv.org/html/2609.01952#Sx5.F3)tests the transient term in Theorem[2](https://arxiv.org/html/2609.01952#Thmtheorem2)\. For a fixed step size \(learning rate\), the theorem predicts that the time\-averaged stationarity term approaches a nonzero neighbourhood at order𝒪⁡\(1/T\)\\mathcal\{O\}\(1/T\)\. We represent this behaviour by

G^T≈b^G\+a​T−p,\\widehat\{G\}\_\{T\}\\approx\\widehat\{b\}\_\{G\}\+aT^\{\-p\},\(17\)whereG^T\\widehat\{G\}\_\{T\}is the sampled prefix average of the stationarity proxy,b^G\\widehat\{b\}\_\{G\}is its fitted terminal neighbourhood, andG^T−b^G\\widehat\{G\}\_\{T\}\-\\widehat\{b\}\_\{G\}is the part that has not yet decayed\. The reference order isp=1p=1\. A valuep\>1p\>1describes faster decay and is also compatible with the upper bound\. Equation[17](https://arxiv.org/html/2609.01952#Sx5.E17)is a diagnostic model for the finite trajectory, not an equality asserted by the theorem\. As long asp\>1p\>1, the theoretical convergence bound is correct and the closerppis to11, the tighter the convergence rate bound is\.

ET\+E\_\{T\}^\{\+\}measures how far the running stationarity statistic remains above its terminal neighbourhood\.

ET\+=max⁡\{G^T−b^G,εb\},E\_\{T\}^\{\+\}=\\max\\\{\\widehat\{G\}\_\{T\}\-\\widehat\{b\}\_\{G\},\\varepsilon\_\{b\}\\\},\(18\)whereεb\\varepsilon\_\{b\}is the estimated tail\-noise level\. The maximum prevents random floor crossings from becoming undefined on a logarithmic axis\. If the excess followsa/Ta/T, its log–log curve is approximately a straight line with slope−1\-1, parallel to the dashed1/T1/Treference\. A shallower curve corresponds top<1p<1, while a steeper curve corresponds top\>1p\>1\. In Figure[3](https://arxiv.org/html/2609.01952#Sx5.F3), the curve is shallow during the early trajectory and steepens near the end\. The global fitp^=1\.26\\widehat\{p\}=1\.26consequently describes the whole trajectory perfectly as a1/T1/Tregime\.

##### A direct check of inverse\-time scaling\.

The second panel of Figure[3](https://arxiv.org/html/2609.01952#Sx5.F3)plots the signed scaled excess

QT=T⁡\(G^T−b^G\)\.Q\_\{T\}=T\(\\widehat\{G\}\_\{T\}\-\\widehat\{b\}\_\{G\}\)\.\(19\)IfG^T−b^G≈a/T\\widehat\{G\}\_\{T\}\-\\widehat\{b\}\_\{G\}\\approx a/T, thenQT≈aQ\_\{T\}\\approx aand the curve forms a plateau\. A rising curve indicates decay slower than1/T1/T, and a falling curve indicates decay faster than1/T1/T\. Keeping the statistic signed also shows when stochastic observations cross the fitted neighbourhood\. In Figure[3](https://arxiv.org/html/2609.01952#Sx5.F3),QTQ\_\{T\}rises during the early slow\-decay phase, reaches a maximum, and then turns downward\. The turnover marks the transition fromp<1p<1sharply towardp≥1p\\geq 1\.

##### The decay rate at each stage of training\.

The dynamic exponentp^dyn​\(T\)\\widehat\{p\}\_\{\\rm dyn\}\(T\)is the negative slope oflog⁡ET\+\\log E\_\{T\}^\{\+\}againstlog⁡T\\log Twithin a moving window of seven recorded snapshots\. For the displayed Setting\-C run, the late median is1\.261\.26and the interquartile range is\[0\.74,1\.77\]\[0\.74,1\.77\]\. The local exponent approaches and then exceeds one in the same region where Panel \(b\) stops rising and turns downward\. The rate is slow at the early stage, but it recovers to a rate faster than1/T1/Tin later rounds\. The more detailed experiment results and analysis for this part can be seen in Appendix[H](https://arxiv.org/html/2609.01952#A8)\.

Figure 3:Full\-horizon stationarity\-rate diagnostic for Setting B, seed 1\. The late resolvable transient has medianp~dyn=1\.26\\widetilde\{p\}\_\{\\rm dyn\}=1\.26with interquartile range\[0\.74,1\.77\]\[0\.74,1\.77\]\. The three panels show log–log excess decay, signedTT\-scaled excess, and the dynamic exponent defined in Equation[17](https://arxiv.org/html/2609.01952#Sx5.E17)\.![Refer to caption](https://arxiv.org/html/2609.01952v1/pics/gossip_kd_setB_eta0.05_live_pt1500_seed1_main_rate.png)

### Empirical Validation of the Bound

##### KD\-induced contraction\.

Across the eight main KD runs,Ψ^tf\\widehat\{\\Psi\}\_\{t\}^\{f\}contracts by40\.240\.2–61\.0×61\.0\\timesfrom the post\-pretraining state\. The terminal values are2\.81×10−42\.81\\times 10^\{\-4\}to4\.10×10−44\.10\\times 10^\{\-4\}\. This contraction holds for the homogeneous, width\-heterogeneous, and mixed\-family rosters, matching the common output\-space mechanism in Theorem[1](https://arxiv.org/html/2609.01952#Thmtheorem1)\. In the four isolated controls, disagreement remains at1\.36×10−21\.36\\times 10^\{\-2\}to1\.51×10−21\.51\\times 10^\{\-2\}\. At 70,000 events the seed\-0 KD values are therefore3636–48×48\\timessmaller\. This control isolates peer exchange as the source of consensus contraction; it is not an optimizer benchmark\.

Figure 4:Showcase for full event function\-space consensus trace for Setting C\. This corresponds to the two observables in Theorem[2](https://arxiv.org/html/2609.01952#Thmtheorem2)\.
##### Stationarity and rate\.

For the eight Setting A–D main runs,p~dyn\\widetilde\{p\}\_\{\\rm dyn\}ranges from 0\.88 to 1\.90, with four values in\[0\.88,1\.03\]\[0\.88,1\.03\], the two Set\-C values at 1\.21 and 1\.90, and two Set\-D values at 0\.88 and 0\.91\. These measurements support decay at the1/T1/Torder or faster over the resolvable transient predicted by Theorem[2](https://arxiv.org/html/2609.01952#Thmtheorem2)\.

Figure 5:Showcase for instantaneous and sampled\-prefix stationarity statistics for Setting B\. correspond to the two observables in Theorem[2](https://arxiv.org/html/2609.01952#Thmtheorem2)\.
##### Neighbourhood signatures\.

The four\-point seed\-0 sweep exhibits the finite\-horizon competition in the bound\. For all of the settings from A–D, the fitted consensus floor decreases from\(1\.10,1\.17,1\.25\)×10−3\(1\.10,1\.17,1\.25\)\\times 10^\{\-3\}atη=0\.0125\\eta=0\.0125to\(3\.87,3\.98,4\.08\)×10−4\(3\.87,3\.98,4\.08\)\\times 10^\{\-4\}atη=0\.05\\eta=0\.05, then rises slightly to\(4\.04,4\.20,4\.23\)×10−4\(4\.04,4\.20,4\.23\)\\times 10^\{\-4\}atη=0\.1\\eta=0\.1\. Smallη\\etaretains the1/\(η​T\)1/\(\\eta T\)transient at the fixed horizon, while largeη\\etaexposes the discretization neighbourhood\. At every swept step size the shared\-skeleton floor ordering is A<<B<<C, consistent with an architecture\-dependent additive term\. The sweep does not separately identify the oracle quantitiesBf2B\_\{f\}^\{2\}andζf2\\zeta\_\{f\}^\{2\}; doing so would require direct reachable\-class probes rather than a regression on training trajectories\.

##### Task quality and controls\.

The accuracy panel shows that consensus is not obtained by collapsing to a uniform predictor\. The main KD ensemble accuracies span 0\.535–0\.729 for seed 0 and remain well above chance in every roster\. We report D\-SGD only for Setting A, where parameter averaging is defined\. Its use of momentum 0\.9 differs from the analysed gossip phase, so it remains an appendix mechanism reference and is not used for a performance claim\. Lastly, the more detailed experiment results and analysis for this part can be seen in Appendix[H](https://arxiv.org/html/2609.01952#A8)\.

## Conclusion

We studied asynchronous random\-edge peer\-to\-peer knowledge distillation for serverless networks whose peers need not share a parameter space\. The analysis places all peers in the common prediction spaceL2​\(μ,ℝC\)L^\{2\}\(\\mu,\\mathbb\{R\}^\{C\}\), treats each KD exchange as a kernel\-weighted mixing step, and couples quotient\-space contraction with descent of the task objective\. Under the stated regularity and realizability conditions, Theorem[2](https://arxiv.org/html/2609.01952#Thmtheorem2)gives an𝒪⁡\(1/\(η​T\)\)\\mathcal\{O\}\(1/\(\\eta T\)\)transient plus an𝒪⁡\(η\)\+𝒪⁡\(Bf2\)\+𝒪⁡\(ζf2\)\\mathcal\{O\}\(\\eta\)\+\\mathcal\{O\}\(B\_\{f\}^\{2\}\)\+\\mathcal\{O\}\(\\zeta\_\{f\}^\{2\}\)neighbourhood, separating discretization, capacity, and persistent task heterogeneity\. The CIFAR\-10 diagnostics are consistent with this prediction as KD reduces function\-space disagreement by40\.240\.2–61\.0×61\.0\\timesacross all experiments, shared\-skeleton runs show late\-transient exponents near or above one, and the step\-size sweep exhibits the predicted transient–neighbourhood tradeoff\.

#### Limitations and Future Work

The theorem is conditional on stable kernels, bounded staleness, A4 for the parameter\-to\-logit bridge, A10 for restricted alignment\. These assumptions identify where the proof uses network geometry, but they are not directly automatic for arbitrary feature learning or aggressive step sizes\. The experiments also estimate a combined terminal neighbourhood, not the individual𝒪⁡\(η\)\\mathcal\{O\}\(\\eta\),Bf2B\_\{f\}^\{2\}, andζf2\\zeta\_\{f\}^\{2\}contributions, and they use a finite reference support for both KD and evaluation\. Future work should measure A4/A10 residuals during training, separate the KD and held\-out evaluation supports, as well as estimateBfB\_\{f\}through reachable\-class probes,

## References

- Allen\-Zhu, Li, and Song \(2019\)Allen\-Zhu, Z\.; Li, Y\.; and Song, Z\. 2019\.A Convergence Theory for Deep Learning via Over\-Parameterization\.In*International Conference on Machine Learning \(ICML\)*, volume 97, 242–252\.
- Assran et al\. \(2019\)Assran, M\.; Loizou, N\.; Ballas, N\.; and Rabbat, M\. 2019\.Stochastic Gradient Push for Distributed Deep Learning\.In*International Conference on Machine Learning \(ICML\)*\.
- Bai and Lee \(2020\)Bai, Y\.; and Lee, J\. D\. 2020\.Beyond Linearization: On Quadratic and Higher\-Order Approximation of Wide Neural Networks\.In*International Conference on Learning Representations \(ICLR\)*\.
- Banerjee et al\. \(2023\)Banerjee, A\.; Cisneros\-Velarde, P\.; Zhu, L\.; and Belkin, M\. 2023\.Neural Tangent Kernel at Initialization: Linear Width Suffices\.In*Uncertainty in Artificial Intelligence \(UAI\)*, volume 216, 110–118\.
- Bickel, Ritov, and Tsybakov \(2009\)Bickel, P\. J\.; Ritov, Y\.; and Tsybakov, A\. B\. 2009\.Simultaneous Analysis of Lasso and Dantzig Selector\.*The Annals of Statistics*, 37\(4\): 1705–1732\.
- Boyd et al\. \(2006\)Boyd, S\.; Ghosh, A\.; Prabhakar, B\.; and Shah, D\. 2006\.Randomized Gossip Algorithms\.*IEEE Transactions on Information Theory*, 52\(6\): 2508–2530\.
- Chang et al\. \(2019\)Chang, H\.; Shejwalkar, V\.; Shokri, R\.; and Houmansadr, A\. 2019\.Cronus: Robust and Heterogeneous Collaborative Learning with Black\-Box Knowledge Transfer\.In*arXiv preprint arXiv:1912\.11279*\.
- Dandi et al\. \(2022\)Dandi, Y\.; Koloskova, A\.; Jaggi, M\.; and Stich, S\. U\. 2022\.Data\-heterogeneity\-aware Mixing for Decentralized Learning\.In*International Conference on Learning Representations \(ICLR\)*\.
- Dembo, Eisenstat, and Steihaug \(1982\)Dembo, R\. S\.; Eisenstat, S\. C\.; and Steihaug, T\. 1982\.Inexact Newton Methods\.*SIAM Journal on Numerical Analysis*, 19\(2\): 400–408\.
- Dong et al\. \(2020\)Dong, K\.; Zhou, C\.; Ruan, Y\.; and Li, Y\. 2020\.MobileNetV2 model for image classification\.In*2020 2nd International Conference on Information Technology and Computer Application \(ITCA\)*, 476–480\. IEEE\.
- Du et al\. \(2019\)Du, S\. S\.; Zhai, X\.; Poczos, B\.; and Singh, A\. 2019\.Gradient descent provably optimizes over\-parameterized neural networks\.In*International Conference on Learning Representations*\.
- Eisenstat and Walker \(1996\)Eisenstat, S\. C\.; and Walker, H\. F\. 1996\.Choosing the Forcing Terms in an Inexact Newton Method\.*SIAM Journal on Scientific Computing*, 17\(1\): 16–32\.
- Hinton, Vinyals, and Dean \(2015\)Hinton, G\.; Vinyals, O\.; and Dean, J\. 2015\.Distilling the Knowledge in a Neural Network\.In*NeurIPS Deep Learning Workshop*\.ArXiv:1503\.02531\.
- Huang et al\. \(2024\)Huang, C\.\-Y\.; Srinivas, K\.; Zhang, X\.; and Li, X\. 2024\.Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, 20111–20133\. PMLR\.
- Jacot, Gabriel, and Hongler \(2018\)Jacot, A\.; Gabriel, F\.; and Hongler, C\. 2018\.Neural Tangent Kernel: Convergence and Generalization in Neural Networks\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Khowaja et al\. \(2026\)Khowaja, S\. A\.; Dev, K\.; Al Hamadi, H\.; Li, R\.; and Zhang, H\. 2026\.SplitGossip: Edge\-based Fine\-tuning of Distilled LLMs Using Split Gossip Learning and Gradient Compression\.In*2026 IEEE International Conference on Consumer Electronics \(ICCE\)*, 1–6\. IEEE\.
- Koloskova et al\. \(2020\)Koloskova, A\.; Loizou, N\.; Boreiri, S\.; Jaggi, M\.; and Stich, S\. U\. 2020\.A Unified Theory of Decentralized SGD with Changing Topology and Local Updates\.In*International Conference on Machine Learning \(ICML\)*\.
- Krizhevsky, Nair, and Hinton \(2009\)Krizhevsky, A\.; Nair, V\.; and Hinton, G\. 2009\.Learning multiple layers of features from tiny images\.Technical report, University of Toronto\.
- Lee et al\. \(2019\)Lee, J\.; Xiao, L\.; Schoenholz, S\. S\.; Bahri, Y\.; Novak, R\.; Sohl\-Dickstein, J\.; and Pennington, J\. 2019\.Wide Neural Networks of Any Depth Evolve as Linear Models under Gradient Descent\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Li and Wang \(2019\)Li, D\.; and Wang, J\. 2019\.FedMD: Heterogenous Federated Learning via Model Distillation\.In*NeurIPS Workshop on Federated Learning for Data Privacy and Confidentiality*\.
- Li et al\. \(2020\)Li, T\.; Sahu, A\. K\.; Talwalkar, A\.; and Smith, V\. 2020\.Federated Optimization in Heterogeneous Networks\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, volume 2, 429–450\.
- Li, Chen, and Lu \(2023\)Li, X\.; Chen, B\.; and Lu, W\. 2023\.FedDKD: Federated learning with decentralized knowledge distillation\.*Applied Intelligence*, 53\(15\): 18547–18563\.
- Lian et al\. \(2017\)Lian, X\.; Zhang, C\.; Zhang, H\.; Hsieh, C\.\-J\.; Zhang, W\.; and Liu, J\. 2017\.Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Lin et al\. \(2020\)Lin, T\.; Kong, L\.; Stich, S\. U\.; and Jaggi, M\. 2020\.Ensemble Distillation for Robust Model Fusion in Federated Learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*\.
- Ma et al\. \(2018\)Ma, N\.; Zhang, X\.; Zheng, H\.\-T\.; and Sun, J\. 2018\.Shufflenet v2: Practical guidelines for efficient cnn architecture design\.In*Proceedings of the European conference on computer vision \(ECCV\)*, 116–131\.
- Martínez Beltrán et al\. \(2026\)Martínez Beltrán, E\. T\.; Giryes, P\.; Bovet, G\.; Stiller, B\.; Martínez Pérez, G\.; and Huertas Celdrán, A\. 2026\.FedEnD: Communication\-efficient Federated Learning for non\-IID data via decentralized ensemble distillation\.*Future Generation Computer Systems*, 183: 108528\.
- Nocedal and Wright \(2006\)Nocedal, J\.; and Wright, S\. J\. 2006\.*Numerical Optimization*\.Springer, 2nd edition\.
- Sandler et al\. \(2018\)Sandler, M\.; Howard, A\.; Zhu, M\.; Zhmoginov, A\.; and Chen, L\.\-C\. 2018\.Mobilenetv2: Inverted residuals and linear bottlenecks\.In*Proceedings of the IEEE conference on computer vision and pattern recognition*, 4510–4520\.
- Shen et al\. \(2020\)Shen, T\.; Zhang, J\.; Jia, X\.; Zhang, F\.; Luo, G\.; Wu, F\.; Zhuang, Y\.; and Lin, X\. 2020\.Federated Mutual Learning\.ArXiv:2006\.16765\.
- Sun, Daneshmand, and Scutari \(2019\)Sun, Y\.; Daneshmand, A\.; and Scutari, G\. 2019\.Convergence Rate of Distributed Optimization Algorithms Based on Gradient Tracking\.ArXiv:1905\.02637\.
- Taya et al\. \(2022\)Taya, A\.; Nishio, T\.; Morikura, M\.; and Yamamoto, K\. 2022\.Decentralized and model\-free federated learning: Consensus\-based distillation in function space\.*IEEE Transactions on Signal and Information Processing over Networks*, 8: 799–814\.
- van de Geer and Bühlmann \(2009\)van de Geer, S\. A\.; and Bühlmann, P\. 2009\.On the Conditions Used to Prove Oracle Results for the Lasso\.*Electronic Journal of Statistics*, 3: 1360–1392\.
- Wu et al\. \(2022\)Wu, C\.; Wu, F\.; Lyu, L\.; Huang, Y\.; and Xie, X\. 2022\.Communication\-Efficient Federated Learning via Knowledge Distillation\.*Nature Communications*, 13: 2032\.
- Wu et al\. \(2026\)Wu, K\.; Qiu, J\.; Xue, Y\.; Yang, Z\.; Sun, J\.; and Wang, Y\. 2026\.Topology\-Aware Cross\-Modal Distillation for Communication\-Efficient Decentralized Multimodal Federated Learning\.In*2026 6th International Conference on Artificial Intelligence and Industrial Technology Applications \(AIITA\)*, 523–528\. IEEE\.
- Ying et al\. \(2026\)Ying, X\.; Yan, K\.; Gao, X\.; and Huang, J\. 2026\.IMFLKD: an incentive mechanism for decentralized federated learning based on knowledge distillation\.*Scientific Reports*\.
- Zeng and Lei \(2025\)Zeng, S\.; and Lei, Y\. 2025\.Stability and Generalization Analysis of Decentralized SGD: Sharper Bounds Beyond Lipschitzness and Smoothness\.In*Forty\-second International Conference on Machine Learning*\.
- Zhang et al\. \(2022\)Zhang, L\.; Shen, L\.; Ding, L\.; Tao, D\.; and Duan, L\.\-Y\. 2022\.Fine\-Tuning Global Model via Data\-Free Knowledge Distillation for Non\-IID Federated Learning\.In*IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*\.
- Zhu, Hong, and Zhou \(2021\)Zhu, Z\.; Hong, J\.; and Zhou, J\. 2021\.Data\-Free Knowledge Distillation for Heterogeneous Federated Learning\.In*International Conference on Machine Learning \(ICML\)*\.

Appendix

This appendix is self\-contained\. Appendix[D](https://arxiv.org/html/2609.01952#A4)states Assumptions A1–A10 in full\. The proof then proceeds as a sequence of named lemmas and propositions culminating in Theorem[G1](https://arxiv.org/html/2609.01952#Thmtheorem1a)\. In the end, it details about all of the experiments setups and results with comprehensive statistic analysis\.

## Appendix ANotations Clarification

This section details every mathematical notation, variable, operator, and constant used throughout the proof of convergence in the federated knowledge distillation appendix\.

### Network and Graph Topology

- •NN: Total number of peers \(nodes\) in the network\.
- •i,j,ki,j,k: Indices representing specific peers\.
- •EE: The set of undirected edges representing communication links in the graph\.
- •E→\\vec\{E\}: The set of directed edges \(both orientations for every undirected edge\)\.
- •e,ete,e\_\{t\}: A specific edge that is active at time steptt\.
- •did\_\{i\}: The degree \(number of neighbors\) of peerii\.
- •dmaxd\_\{\\max\}: The maximum degree across all peers in the graph\.
- •ℒ\\mathcal\{L\}: The unweighted graph Laplacian matrix, defined asℒ=D−A\\mathcal\{L\}=D\-A\(Degree matrix minus Adjacency matrix\)\.
- •λ1,λ2,…,λmax\\lambda\_\{1\},\\lambda\_\{2\},\\dots,\\lambda\_\{\\max\}: The eigenvalues of the Laplacianℒ\\mathcal\{L\}\. The algebraic connectivity \(λ2\\lambda\_\{2\}\) dictates the speed of information mixing across the graph\.
- •qiq\_\{i\}: The probability that peeriiis activated in a given event, defined under uniform edge sampling asdi/\|E\|d\_\{i\}/\|E\|\.

### Time and Optimization Parameters

- •tt: Discrete event time step \(counts individual edge activations\)\.
- •TT: Total number of events/steps in the horizon\.
- •η\\eta: The base parameter\-space step size \(learning rate\) for SGD\.
- •α\\alpha: The knowledge distillation \(KD\) balance parameter \(0<α<10<\\alpha<1\)\. Regulates the trade\-off between the local task loss and the KD loss\.
- •τ\\tau: The softmax temperature scalar used to scale logits before applying the softmax function\.
- •ww: The effective step size for the KD logit contraction, defined exactly asw=η​α​τw=\\eta\\alpha\\tau\.

### Data, Functions, and Prediction Spaces

- •μ\\mu: The reference measure\. A finite evaluation dataset over which all functional metrics \(logits, predictions\) are measured\.
- •CC: The number of classes in the classification task\.
- •θi\\theta\_\{i\}: The neural network parameter vector \(weights and biases\) for peerii\.
- •θi0\\theta\_\{i\}^\{0\}: The parameters of peeriiat initialization\.
- •Δ​θi\\Delta\\theta\_\{i\}: The parameter update applied to peeriiduring a training step\.
- •Zi,ZZ\_\{i\},Z: The logits \(pre\-softmax outputs\) evaluated over the reference measureμ\\mu\.
- •ZitZ\_\{i\}^\{t\}: Logits of peeriispecifically at timett\.
- •Z¯\\bar\{Z\}: The mean logits across the entire network,N−1​∑iZiN^\{\-1\}\\sum\_\{i\}Z\_\{i\}\.
- •Pi,piP\_\{i\},p\_\{i\}: The prediction probabilities, computed asPi=softmax⁡\(Zi/τ\)P\_\{i\}=\\operatorname\{softmax\}\(Z\_\{i\}/\\tau\)\.
- •PjT,pjEMAP\_\{j\}^\{T\},p\_\{j\}^\{\\mathrm\{EMA\}\}:T​h​e​t​e​a​c​h​e​r​t​a​r​g​e​t​p​r​e​d​i​c​t​i​o​n​s​f​r​o​m​p​e​e​r:Theteachertargetpredictionsfrompeerj \. Can be the live prediction or an Exponential Moving Average \(EMA\)\.
- •p¯t\\bar\{p\}^\{t\}: The mean prediction probability across all peers at timett\.
- •pminp\_\{\\min\}: A strictly positive lower bound on all class probabilities \(enforced by Assumption A1\)\.
- •q⋆q^\{\\star\}: The hypothetical optimal target prediction distribution for the global task\.

### Jacobians and Neural Tangent Kernels \(NTK\)

- •JizJ\_\{i\}^\{z\}: The Jacobian matrix of the logits with respect to the parameters,Dθi​Zi​\(θi\)D\_\{\\theta\_\{i\}\}Z\_\{i\}\(\\theta\_\{i\}\)\.
- •JipJ\_\{i\}^\{p\}: The Jacobian of the prediction probabilities with respect to the parameters\.
- •DZ​PD\_\{Z\}P: The Jacobian of the softmax mapping from logits to probabilities\.
- •Θi\\Theta\_\{i\}: The Neural Tangent Kernel \(NTK\) in*logit space*, defined asΘi=Jiz​\(Jiz\)∗\\Theta\_\{i\}=J\_\{i\}^\{z\}\(J\_\{i\}^\{z\}\)^\{\*\}\.
- •Θi0\\Theta\_\{i\}^\{0\}: The initial logit\-space NTK atθi0\\theta\_\{i\}^\{0\}\.
- •KiK\_\{i\}: The Neural Tangent Kernel in*prediction space*, defined asKi=Jip​\(Jip\)∗K\_\{i\}=J\_\{i\}^\{p\}\(J\_\{i\}^\{p\}\)^\{\*\}\.
- •HiH\_\{i\}: A cross\-Jacobian operator mapping the gradient from the prediction\-space KD loss into a parameter update,Hi=τ​Jip​\(Jiz\)∗H\_\{i\}=\\tau J\_\{i\}^\{p\}\(J\_\{i\}^\{z\}\)^\{\*\}\.

### Tangent Spaces and Projections

- •ℋz\\mathcal\{H\}\_\{z\}: The full ambient vector space of evaluated logits\.
- •TizT\_\{i\}^\{z\}: The*logit tangent space*\(the column space/range ofJizJ\_\{i\}^\{z\}orΘi\\Theta\_\{i\}\)\. Represents logit directions the model is physically capable of moving in\.
- •Πiz\\Pi\_\{i\}^\{z\}: The orthogonal projection matrix onto the logit tangent spaceTizT\_\{i\}^\{z\}\.
- •I−ΠizI\-\\Pi\_\{i\}^\{z\}: The orthogonal projection onto the*normal space*\(directions the model cannot move\)\.
- •Tip,ΠipT\_\{i\}^\{p\},\\Pi\_\{i\}^\{p\}: The*prediction tangent space*and its orthogonal projection matrix\.
- •M⁡\(P\)M\(P\): The softmax metric tensor defined asM⁡\(P\)=diag⁡\(P\)−P​P⊤M\(P\)=\\operatorname\{diag\}\(P\)\-PP^\{\\top\}\.

### Losses, Gradients, and Residuals

- •Fi​\(p\)F\_\{i\}\(p\): The local task risk \(e\.g\., Cross\-Entropy loss\) for peerii\.
- •F⁡\(p\)F\(p\): The global task risk, averaged across all peersN−1​∑Fi​\(p\)N^\{\-1\}\\sum F\_\{i\}\(p\)\.
- •Fi⋆,F⋆F\_\{i\}^\{\\star\},F^\{\\star\}: The infimum \(minimum achievable value\) of the local and global task risks\.
- •gitg\_\{i\}^\{t\}: The functional gradient of the local task risk with respect to the predictions,git=∇fFi​\(pit\)g\_\{i\}^\{t\}=\\nabla\_\{f\}F\_\{i\}\(p\_\{i\}^\{t\}\)\.
- •DKL\(P′∥P\)D\_\{\\mathrm\{KL\}\}\(P^\{\\prime\}\\\|P\): The Kullback\-Leibler divergence between two probability distributions\.
- •di​jd\_\{ij\}: The logit difference \(directed edge pull\) between peeriiand peerjj,di​j=Zi−Zjd\_\{ij\}=Z\_\{i\}\-Z\_\{j\}\.
- •δit\\delta\_\{i\}^\{t\}: The task residual defect\. The difference between the actual pushforward of the network and the ideal functional descent direction\.
- •riz,ripr\_\{i\}^\{z\},r\_\{i\}^\{p\}: The Taylor series remainder \(curvature\) vectors for logits and predictions\.
- •ξi,e,ξiz,ξip\\xi\_\{i,e\},\\xi\_\{i\}^\{z\},\\xi\_\{i\}^\{p\}: Martingale difference noise vectors representing the stochasticity introduced by random edge sampling\.

### Consensus and Error Metrics \(The Observables\)

- •Ψtf\\Psi\_\{t\}^\{f\}: Function\-space consensus error\. The variance of predictions across peers:1N​∑i𝔼​‖pit−p¯t‖μ2\\frac\{1\}\{N\}\\sum\_\{i\}\\mathbb\{E\}\\\|p\_\{i\}^\{t\}\-\\bar\{p\}^\{t\}\\\|\_\{\\mu\}^\{2\}\.
- •Ψtz\\Psi\_\{t\}^\{z\}: Logit\-space consensus error\. The variance of logits across peers:1N​∑i𝔼​‖Zit−Z¯t‖μ2\\frac\{1\}\{N\}\\sum\_\{i\}\\mathbb\{E\}\\\|Z\_\{i\}^\{t\}\-\\bar\{Z\}^\{t\}\\\|\_\{\\mu\}^\{2\}\.
- •𝒱W​\(Z\)\\mathcal\{V\}\_\{W\}\(Z\): The*quotient consensus energy*, the primary metric for analyzing contraction:minc⁡1N​∑i‖Zi−c‖Wi2\\min\_\{c\}\\frac\{1\}\{N\}\\sum\_\{i\}\\\|Z\_\{i\}\-c\\\|\_\{W\_\{i\}\}^\{2\}\.
- •cWc\_\{W\}: The weighted centroid of logits that minimizes the quotient energy𝒱W​\(Z\)\\mathcal\{V\}\_\{W\}\(Z\)\.
- •QiQ\_\{i\}: The logit deviation from the weighted centroid,Qi=Zi−cWQ\_\{i\}=Z\_\{i\}\-c\_\{W\}\.
- •GtG\_\{t\}: The average squared norm of the functional task gradients,1N​∑i‖git‖2\\frac\{1\}\{N\}\\sum\_\{i\}\\\|g\_\{i\}^\{t\}\\\|^\{2\}\.
- •Bf2B\_\{f\}^\{2\}: The*representational capacity floor*\. The irreducible minimum distance between the optimal functionq⋆q^\{\\star\}and the functions the network architectures can represent\.
- •ζf2\\zeta\_\{f\}^\{2\}: The data heterogeneity bound\. Measures how much the local dataset gradients vary from the global average\.
- •AtfA\_\{t\}^\{f\}: The teacher lag error\. Evaluates how “stale” the EMA teacher predictions are compared to live predictions\.
- •ℛt\\mathcal\{R\}\_\{t\}: The activation\-corrected task potential \(a weighted sum of local task sub\-optimalities\)\.
- •ℒt\\mathcal\{L\}\_\{t\}: The strict Lyapunov function combining task risk and consensus error,ℒt=ℛt\+λ​𝒱W​\(Zt\)\\mathcal\{L\}\_\{t\}=\\mathcal\{R\}\_\{t\}\+\\lambda\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\.

### Bounding Constants and Multipliers

- •κ−,κ\+\\kappa\_\{\-\},\\kappa\_\{\+\}: The strict lower and upper bounds on the non\-zero eigenvalues of the restricted logit NTKΘi\\Theta\_\{i\}\.
- •κ¯p,κ¯p\\underline\{\\kappa\}\_\{p\},\\overline\{\\kappa\}\_\{p\}: The lower and upper spectral bounds for the restricted prediction kernelKiK\_\{i\}\.
- •LJ,LKL\_\{J\},L\_\{K\}: Lipschitz continuity constants for the Jacobian matrices and the NTK\.
- •LfL\_\{f\}: Smoothness constant \(Lipschitz gradient\) for the functional task riskFiF\_\{i\}\.
- •HmaxH\_\{\\max\}: The absolute upper bound on the operator norm of the cross\-JacobianHiH\_\{i\}\.
- •γKD,νKD,B\\gamma\_\{\\mathrm\{KD\}\},\\nu\_\{\\mathrm\{KD,B\}\}: Constants bounding the non\-realizable \(normal\) component of the KD pull \(from Assumption A10\)\.
- •γtask,νtask,Ψ,νtask,B\\gamma\_\{\\mathrm\{task\}\},\\nu\_\{\\mathrm\{task,\\Psi\}\},\\nu\_\{\\mathrm\{task,B\}\}: Constants bounding the non\-realizable \(normal\) component of the task gradient\.
- •WiW\_\{i\}: The pseudo\-inverse metric matrix, used to precondition the quotient energy,Wi=Θi†\+γ0​\(I−Πiz\)W\_\{i\}=\\Theta\_\{i\}^\{\\dagger\}\+\\gamma\_\{0\}\(I\-\\Pi\_\{i\}^\{z\}\)\.
- •mW,MWm\_\{W\},M\_\{W\}: The absolute minimum and maximum eigenvalues ofWiW\_\{i\}\.
- •aW,bW,εWa\_\{W\},b\_\{W\},\\varepsilon\_\{W\}: Geometric contraction constants governing the speed of consensus\.
- •Db,t,Dξ,tD\_\{b,t\},D\_\{\\xi,t\}: Aggregate variance bounds for deterministic perturbations \(curvature, task, lag\) and stochastic noise\.
- •CG,CB,Cζ,Cη,CA,c0C\_\{G\},C\_\{B\},C\_\{\\zeta\},C\_\{\\eta\},C\_\{A\},c\_\{0\}: Aggregate algebraic coefficients governing the recursive consensus drift\.
- •aF,bF,DB,Dζ,Dη,DAa\_\{F\},b\_\{F\},D\_\{B\},D\_\{\\zeta\},D\_\{\\eta\},D\_\{A\}: Aggregate algebraic coefficients governing the task descent recursion\.
- •cG,cV,KB,Kζ,Kηc\_\{G\},c\_\{V\},K\_\{B\},K\_\{\\zeta\},K\_\{\\eta\}: The ultimate algebraic coefficients in the strict Lyapunov recursion representing the combined system drift\.
- •C0,C1,C2,C3C\_\{0\},C\_\{1\},C\_\{2\},C\_\{3\}: The final macroscopic constants in Theorem 1’s𝒪⁡\(1/T\)\\mathcal\{O\}\(1/T\)convergence bound\.C0C\_\{0\}scales the transient time rate,C1C\_\{1\}scales the step\-size error, andC2,C3C\_\{2\},C\_\{3\}scale the persistent capacity and data heterogeneity floors\.

## Appendix BExplicit Analytical Coefficients of the Convergence Proof

This section explicitly defines every composite coefficient used in the convergence proof \(Lemmas 7, 8, 9, and Theorem 1\), tracing their exact dependence on the structural network and optimization parameters\.

### Consensus Recursion Coefficients \(Theorem 1 in Main Paper, Lemma 7 in Appendix\)

The global contraction of the quotient energy𝒱W​\(Z\)\\mathcal\{V\}\_\{W\}\(Z\)follows the discrete\-time Lyapunov recursion:

𝔼t​𝒱W​\(Zt\+1\)≤\(1−c0​η\)​𝒱W​\(Zt\)\+CG​η​Gt\+CB​η​Bf2\+Cζ​η​ζf2\+Cη​η2\+CA​η​Atf\.\\begin\{split\}\\mathbb\{E\}\_\{t\}\\mathcal\{V\}\_\{W\}\(Z^\{t\+1\}\)\\leq\(1\-c\_\{0\}\\eta\)\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+C\_\{G\}\\eta G\_\{t\}\+C\_\{B\}\\eta B\_\{f\}^\{2\}\+\\\\ C\_\{\\zeta\}\\eta\\zeta\_\{f\}^\{2\}\+C\_\{\\eta\}\\eta^\{2\}\+C\_\{A\}\\eta A\_\{t\}^\{f\}\.\\end\{split\}The transient contraction ratec0c\_\{0\}is defined as:

c0=α​pmin​λ22​MW​\|E\|c\_\{0\}=\\frac\{\\alpha p\_\{\\min\}\\lambda\_\{2\}\}\{2M\_\{W\}\|E\|\}whereMW=max⁡\{κ−−1,γ0\}M\_\{W\}=\\max\\\{\\kappa\_\{\-\}^\{\-1\},\\gamma\_\{0\}\\\}\. The perturbation prefactorsRG,RB,Rζ,R0,RAR\_\{G\},R\_\{B\},R\_\{\\zeta\},R\_\{0\},R\_\{A\}are defined by bounding the task, curvature, and lag vectors using Young’s inequality:

RG\\displaystyle R\_\{G\}=9​MW​\(1−α\)2​\(κ¯p2\+χG\)\\displaystyle=9M\_\{W\}\(1\-\\alpha\)^\{2\}\(\\overline\{\\kappa\}\_\{p\}^\{2\}\+\\chi\_\{G\}\)RB\\displaystyle R\_\{B\}=9​MW​\(1−α\)2​χB\\displaystyle=9M\_\{W\}\(1\-\\alpha\)^\{2\}\\chi\_\{B\}Rζ\\displaystyle R\_\{\\zeta\}=9​MW​\(1−α\)2​χζ\\displaystyle=9M\_\{W\}\(1\-\\alpha\)^\{2\}\\chi\_\{\\zeta\}R0\\displaystyle R\_\{0\}=34​MW​LΘ2​Gθ,44\\displaystyle=\\frac\{3\}\{4\}M\_\{W\}L\_\{\\Theta\}^\{2\}G\_\{\\theta,4\}^\{4\}RA\\displaystyle R\_\{A\}=3​MW​κ\+​dmax\|E\|\\displaystyle=\\frac\{3M\_\{W\}\\kappa\_\{\+\}d\_\{\\max\}\}\{\|E\|\}LetKpre=\(8​dmaxaW​α​τ​\|E\|\+2\)K\_\{\\rm pre\}=\\left\(\\frac\{8d\_\{\\max\}\}\{a\_\{W\}\\alpha\\tau\|E\|\}\+2\\right\)\. By absorbing thew=η​α​τw=\\eta\\alpha\\taufactors into the stochastic edge distribution, the composite tracking coefficients are:

CG\\displaystyle C\_\{G\}=RG​Kpre\\displaystyle=R\_\{G\}K\_\{\\rm pre\}CB\\displaystyle C\_\{B\}=4​MW​dmax​νKD,B​α​τN⁡\(pmin/τ\)​λ2​mW\+RB​Kpre\\displaystyle=\\frac\{4M\_\{W\}d\_\{\\max\}\\nu\_\{\\rm KD,B\}\\alpha\\tau\}\{N\(p\_\{\\min\}/\\tau\)\\lambda\_\{2\}m\_\{W\}\}\+R\_\{B\}K\_\{\\rm pre\}Cζ\\displaystyle C\_\{\\zeta\}=Rζ​Kpre\\displaystyle=R\_\{\\zeta\}K\_\{\\rm pre\}Cη\\displaystyle C\_\{\\eta\}=R0​Kpre\+Σ0\\displaystyle=R\_\{0\}K\_\{\\rm pre\}\+\\Sigma\_\{0\}CA\\displaystyle C\_\{A\}=RA​α​τ​Kpre\\displaystyle=R\_\{A\}\\alpha\\tau K\_\{\\rm pre\}

### Task Descent Coefficients \(Lemma 8 in Appendix\)

The activation\-corrected task potentialℛt\\mathcal\{R\}\_\{t\}satisfies the descent inequality:

𝔼t​ℛt\+1≤ℛt−aF​η​Gt\+bF​η​𝒱W​\(Zt\)\+DB​η​Bf2\+Dζ​η​ζf2\+Dη​η2\+DA​η​Atf\.\\mathbb\{E\}\_\{t\}\\mathcal\{R\}\_\{t\+1\}\\leq\\mathcal\{R\}\_\{t\}\-a\_\{F\}\\eta G\_\{t\}\+b\_\{F\}\\eta\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+D\_\{B\}\\eta B\_\{f\}^\{2\}\+D\_\{\\zeta\}\\eta\\zeta\_\{f\}^\{2\}\+D\_\{\\eta\}\\eta^\{2\}\+D\_\{A\}\\eta A\_\{t\}^\{f\}\.The task drift explicitly counters the consensus drift\. The descent coefficientaFa\_\{F\}and the corresponding error coefficients are:

aF=\(1−α\)​\[κ¯p​\(1−γtask\)−εδ2\]−εKD−ηmax​Lf2​CΔ,GbF=\(1−α\)​\[κ¯p​νtask,Ψ\+χG2​εδ\]​Cf​zmW\+α​CKD,V\+ηmax​Lf2​CΔ,VDB=\(1−α\)​\[κ¯p​νtask,B\+χB2​εδ\]\+ηmax​Lf2​CΔ,BDζ=\(1−α\)​χζ2​εδDA=α​CKD,A\+ηmax​Lf2​CΔ,ADη=Lf2​\(4​Rp\+4​Σp\)\\begin\{split\}a\_\{F\}&=\(1\-\\alpha\)\\left\[\\underline\{\\kappa\}\_\{p\}\(1\-\\gamma\_\{\\rm task\}\)\-\\frac\{\\varepsilon\_\{\\delta\}\}\{2\}\\right\]\-\\varepsilon\_\{\\rm KD\}\-\\eta\_\{\\max\}\\frac\{L\_\{f\}\}\{2\}C\_\{\\Delta,G\}\\\\ b\_\{F\}&=\(1\-\\alpha\)\\left\[\\underline\{\\kappa\}\_\{p\}\\nu\_\{\\rm task,\\Psi\}\+\\frac\{\\chi\_\{G\}\}\{2\\varepsilon\_\{\\delta\}\}\\right\]\\frac\{C\_\{fz\}\}\{m\_\{W\}\}\+\\alpha C\_\{\\rm KD,V\}\+\\\\ &\\qquad\\qquad\\qquad\\qquad\\qquad\\quad\\qquad\\qquad\\quad\\eta\_\{\\max\}\\frac\{L\_\{f\}\}\{2\}C\_\{\\Delta,V\}\\\\ D\_\{B\}&=\(1\-\\alpha\)\\left\[\\underline\{\\kappa\}\_\{p\}\\nu\_\{\\rm task,B\}\+\\frac\{\\chi\_\{B\}\}\{2\\varepsilon\_\{\\delta\}\}\\right\]\+\\eta\_\{\\max\}\\frac\{L\_\{f\}\}\{2\}C\_\{\\Delta,B\}\\\\ D\_\{\\zeta\}&=\(1\-\\alpha\)\\frac\{\\chi\_\{\\zeta\}\}\{2\\varepsilon\_\{\\delta\}\}\\\\ D\_\{A\}&=\\alpha C\_\{\\rm KD,A\}\+\\eta\_\{\\max\}\\frac\{L\_\{f\}\}\{2\}C\_\{\\Delta,A\}\\\\ D\_\{\\eta\}&=\\frac\{L\_\{f\}\}\{2\}\(4R\_\{p\}\+4\\Sigma\_\{p\}\)\\end\{split\}

### Strict Lyapunov Coefficients \(Lemma 5 in Main Paper, Lemma 9 in Appendix\)

The strict Lyapunov function is defined asℒt=ℛt\+λ​𝒱W​\(Zt\)\\mathcal\{L\}\_\{t\}=\\mathcal\{R\}\_\{t\}\+\\lambda\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\. By taking a linear combination of the inequalities from Lemma 7 and Lemma 8, we obtain:

𝔼t​ℒt\+1≤ℒt−cG​η​Gt−cV​η​𝒱W​\(Zt\)\+KB​η​Bf2\+Kζ​η​ζf2\+Kη​η2\.\\mathbb\{E\}\_\{t\}\\mathcal\{L\}\_\{t\+1\}\\leq\\mathcal\{L\}\_\{t\}\-c\_\{G\}\\eta G\_\{t\}\-c\_\{V\}\\eta\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+K\_\{B\}\\eta B\_\{f\}^\{2\}\+K\_\{\\zeta\}\\eta\\zeta\_\{f\}^\{2\}\+K\_\{\\eta\}\\eta^\{2\}\.The necessary small\-gain condition for convergence isbF​CG<aF​c0b\_\{F\}C\_\{G\}<a\_\{F\}c\_\{0\}\. Assuming this holds, there existsλ\>0\\lambda\>0yielding strictly positive descent factors:

cG\\displaystyle c\_\{G\}=aF−λCG\>0\\displaystyle=a\_\{F\}\-\\lambda C\_\{G\}\\quad\>0cV\\displaystyle c\_\{V\}=λc0−bF\>0\\displaystyle=\\lambda c\_\{0\}\-b\_\{F\}\\quad\>0The remaining static error floors are linearly composed as follows:

KB\\displaystyle K\_\{B\}=DB\+λ​CB\\displaystyle=D\_\{B\}\+\\lambda C\_\{B\}Kζ\\displaystyle K\_\{\\zeta\}=Dζ\+λ​Cζ\\displaystyle=D\_\{\\zeta\}\+\\lambda C\_\{\\zeta\}Kη\\displaystyle K\_\{\\eta\}=Dη\+λ​Cη\+\(DA\+λ​CA\)​A¯f\\displaystyle=D\_\{\\eta\}\+\\lambda C\_\{\\eta\}\+\(D\_\{A\}\+\\lambda C\_\{A\}\)\\bar\{A\}^\{f\}whereA¯f\\bar\{A\}^\{f\}bounds the teacher staleness variance viaAtf≤A¯f​η2A\_\{t\}^\{f\}\\leq\\bar\{A\}^\{f\}\\eta^\{2\}\.

### Final Asymptotic Rates \(Theorem 1\)

The overall𝒪⁡\(1/T\)\\mathcal\{O\}\(1/T\)rate is derived by telescoping the strict Lyapunov recursion\. By isolating the time\-average of the functional gradient and consensus error, we bound:

1T​∑t=0T−1\[𝔼​‖∇fF​\(p¯t\)‖μ2\+c​Ψtf\]≤C0η​T\+C1​η\+C2​Bf2\+C3​ζf2\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\left\[\\mathbb\{E\}\\\|\\nabla\_\{f\}F\(\\bar\{p\}^\{t\}\)\\\|\_\{\\mu\}^\{2\}\+c\\Psi\_\{t\}^\{f\}\\right\]\\leq\\frac\{C\_\{0\}\}\{\\eta T\}\+C\_\{1\}\\eta\+C\_\{2\}B\_\{f\}^\{2\}\+C\_\{3\}\\zeta\_\{f\}^\{2\}\.The universal scaling factorK∗K\_\{\*\}balances the dual contraction criteria:

K∗=max⁡\{2cG,2​Lf2​Cf​z/mW\+c​Cf​z/mWcV\}\.K\_\{\*\}=\\max\\left\\\{\\frac\{2\}\{c\_\{G\}\},\\frac\{2L\_\{f\}^\{2\}C\_\{fz\}/m\_\{W\}\+cC\_\{fz\}/m\_\{W\}\}\{c\_\{V\}\}\\right\\\}\.Multiplying the Lyapunov errors by this global scalar yields the final macroscopic coefficients:

C0\\displaystyle C\_\{0\}=K∗​ℒ0\\displaystyle=K\_\{\*\}\\mathcal\{L\}\_\{0\}C1\\displaystyle C\_\{1\}=K∗​Kη\\displaystyle=K\_\{\*\}K\_\{\\eta\}C2\\displaystyle C\_\{2\}=K∗​KB\\displaystyle=K\_\{\*\}K\_\{B\}C3\\displaystyle C\_\{3\}=K∗​Kζ\\displaystyle=K\_\{\*\}K\_\{\\zeta\}

## Appendix CExpanded Related Work and Closest Comparisons

### Federated Distillation with a Coordinator

Knowledge distillation \(KD\) transfers information through soft predictions rather than through weights\([Hinton, Vinyals, and Dean 2015](https://arxiv.org/html/2609.01952#bib.bib13)\)\. This property makes KD a natural communication primitive for federated learning with heterogeneous architectures, since predictors share the class simplex even when their parameter vectors have different dimensions\. FedMD distils client models on a public dataset and alternates local private training with public\-set distillation\([Li and Wang 2019](https://arxiv.org/html/2609.01952#bib.bib20)\)\. FedDF fuses client models by server\-side ensemble distillation\([Lin et al\. 2020](https://arxiv.org/html/2609.01952#bib.bib24)\)\. Cronus studies heterogeneous collaborative learning through black\-box soft\-label transfer in a setting where robustness to malicious clients is central\([Chang et al\. 2019](https://arxiv.org/html/2609.01952#bib.bib7)\)\. Federated Mutual Learning, data\-free heterogeneous FL distillation, and DS\-FL further show that soft\-label or generated\-data transfer can reduce the need for direct parameter aggregation under non\-IID data and heterogeneous clients\([Shen et al\. 2020](https://arxiv.org/html/2609.01952#bib.bib29);[Zhu, Hong, and Zhou 2021](https://arxiv.org/html/2609.01952#bib.bib38);[Zhang et al\. 2022](https://arxiv.org/html/2609.01952#bib.bib37)\)\. These methods establish the practical relevance of prediction\-level communication\. Their protocols still retain a coordinator, a global aggregation step, a synchronized round structure, or a server\-side distillation stage\.

FedKD is a particularly relevant communication\-efficient representative of this line\([Wu et al\. 2022](https://arxiv.org/html/2609.01952#bib.bib33)\)\. Its mechanism keeps a local mentor model and a shared mentee model at each client, uses adaptive mutual distillation, and combines this with dynamic gradient compression\. The reported empirical result is a large reduction in communication cost, up to94\.89%94\.89\\%on the tasks studied by Wu et al\., without using auxiliary public data\. This differs from the present paper in the core protocol\. FedKD remains a server\-based FL method: the shared mentee participates in server aggregation, while our process samples one graph edge at a time and exchanges only peer predictions\.

### Decentralized and Serverless KD\-FL Systems

Recent work has moved KD closer to serverless or peer\-level learning\. FedDKD, which we use as the verified representative of the decentralized\-KD line, decentralizes knowledge transfer among clients inside a federated procedure\([Li, Chen, and Lu 2023](https://arxiv.org/html/2609.01952#bib.bib22)\)\. The method is close in communication object because clients exchange predictive information, but the analyzed protocol is not the same asynchronous random\-edge process considered here\.

DeSA addresses decentralized FL with both data and model heterogeneity by introducing synthetic anchors\([Huang et al\. 2024](https://arxiv.org/html/2609.01952#bib.bib14)\)\. Each client trains with a regularization term and a KD term induced by the anchors\. The ICML 2024 paper uses domain\-adaptation theory to motivate the anchor mechanism and reports improved intra\-domain and inter\-domain accuracy over decentralized baselines\. DeSA is directly relevant for a federated\-learning reviewer because it combines serverless decentralization, heterogeneity, and KD\. Its guarantee and algorithmic object are still different from ours\. DeSA uses synthetic anchors to create a shared alignment interface, while our theorem studies live random\-edge distillation on a fixed graph and proves contraction in a common prediction space\.

FedEnD is another close system because it is serverless and peer\-to\-peer\([Martínez Beltrán et al\. 2026](https://arxiv.org/html/2609.01952#bib.bib26)\)\. Its mechanism is a two\-stage specialist\-student protocol\. Clients first train specialists on private data, then broadcast model outputs once so that each student can distil from a class\-distribution\-weighted peer ensemble\. The reported results emphasize accuracy and communication: FedEnD improves macro\-F1 by up to5\.7%5\.7\\%in pathologically skewed non\-IID settings, reduces communication by68\.6%68\.6\\%relative to standard iterative averaging, and reports up to84%84\\%lower communication than SCAFFOLD\. FedEnD and this paper are complementary\. FedEnD targets one\-shot decentralized ensemble distillation without auxiliary public data\. Our paper studies an asynchronous stream of random\-edge KD events and derives a conditional convergence theorem for the resulting prediction\-space process\.

Other recent peer\-oriented KD\-FL systems reinforce the same motivation\. IMFLKD studies an incentive mechanism for decentralized FL based on KD and uses blockchain, smart contracts, label aggregation, contribution evaluation, and reputation incentives\([Ying et al\. 2026](https://arxiv.org/html/2609.01952#bib.bib35)\)\. TopoMoDistill uses topology\-aware neighbour aggregation and compressed class\-wise latent prototypes for decentralized multimodal FL\([Wu et al\. 2026](https://arxiv.org/html/2609.01952#bib.bib34)\)\. SplitGossip studies edge\-based fine\-tuning of distilled language models with split gossip learning and gradient compression\([Khowaja et al\. 2026](https://arxiv.org/html/2609.01952#bib.bib16)\)\. These works make peer\-level KD an active systems direction\. The gap addressed here is narrower: a serverless random\-edge P2P KD process with heterogeneous reachable prediction classes and a function\-space convergence bound\.

### Decentralized Optimization and the Type Mismatch

Classical decentralized optimization analyzes consensus and descent when every peer stores a vector in a shared parameter space\. Randomized gossip averaging and distributed subgradient methods characterize graph mixing in this common space\([Boyd et al\. 2006](https://arxiv.org/html/2609.01952#bib.bib6)\)\. Decentralized SGD and push\-sum variants extend the analysis to nonconvex stochastic optimization, directed communication, changing topology, local updates, compression, and data heterogeneity\([Lian et al\. 2017](https://arxiv.org/html/2609.01952#bib.bib23);[Assran et al\. 2019](https://arxiv.org/html/2609.01952#bib.bib2);[Koloskova et al\. 2020](https://arxiv.org/html/2609.01952#bib.bib17);[Dandi et al\. 2022](https://arxiv.org/html/2609.01952#bib.bib8)\)\. These results do not apply directly when peers use different architectures, because the parameter averagexi←∑jWi​j​xjx\_\{i\}\\leftarrow\\sum\_\{j\}W\_\{ij\}x\_\{j\}is not defined across different dimensions\. Our analysis preserves the graph viewpoint but changes the state variable from parameters to evaluated predictions\. A KD event then becomes a contraction step in a shared finite\-support output space\.

##### Implications for baselines\.

For a federated or distributed learning audience, isolated training and homogeneous D\-SGD are mechanism controls rather than a complete empirical positioning\. They test whether KD itself contracts prediction disagreement and whether ordinary parameter gossip is available only in a homogeneous roster\. They do not compare against the closest heterogeneous FL distillation systems\. If the paper makes a broad empirical competitiveness claim, the relevant baseline set should include DeSA, FedEnD, a FedDKD or decentralized\-KD representative, FedKD, and at least one recent heterogeneous FL distillation method\. If the paper keeps its current theory\-first claim, these methods should still appear in the closest\-work table, while the experiments should be described as diagnostics of the theorem’s mechanism rather than as a SOTA benchmark\.

## Appendix DDetailed Assumptions\.

##### A1 \(Bounded logits\)\.

‖zi\(t\)​\(x\)‖∞≤Gz\\\|z\_\{i\}^\{\(t\)\}\(x\)\\\|\_\{\\infty\}\\leq G\_\{z\}for alli,t,xi,t,x, implemented by weight decay \+ logit/gradient clipping\. Lemma[G2](https://arxiv.org/html/2609.01952#Thmlemma2a)proves that every class probability is bounded below,

pi​\(x\)c≥pmin:=e−2Gz/τC\>0\.p\_\{i\}\(x\)\_\{c\}\\ \\geq\\ p\_\{\\min\}:=\\frac\{e^\{\-2G\_\{z\}/\\tau\}\}\{C\}\\ \>\\ 0\.

##### A2 \(Bounded stochastic variance\)\.

\. The stochastic CE functional gradient has variance≤σf2\\leq\\sigma\_\{f\}^\{2\}per peer \(in theℋ\\mathcal\{H\}\-metric; used in Lemmas[3](https://arxiv.org/html/2609.01952#Thmlemma3)and[G7](https://arxiv.org/html/2609.01952#Thmlemma7)\)\.

##### A3 \(Representational capacity floor\)\.

Each peer’s reachable prediction set𝒫i:=\{pθ:θ∈ℝdi\}⊆ℋ\\mathcal\{P\}\_\{i\}:=\\\{\\,p\_\{\\theta\}:\\theta\\in\\mathbb\{R\}^\{d\_\{i\}\}\\,\\\}\\subseteq\\mathcal\{H\}is assumed closed, although it may be non\-convex\. Define the representational floor as the explicitL2​\(μ\)L^\{2\}\(\\mu\)quantity

Bf2:=1N​∑i=1Ndistμ​\(q⋆,𝒫i\)2,distμ​\(q,𝒫i\)=infp∈𝒫i‖q−p‖μ\\boxed\{B\_\{f\}^\{2\}:=\\ \\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathrm\{dist\}\_\{\\mu\}\\\!\\big\(q^\{\\star\},\\mathcal\{P\}\_\{i\}\\big\)^\{2\},\\mathrm\{dist\}\_\{\\mu\}\(q,\\mathcal\{P\}\_\{i\}\)=\\inf\_\{p\\in\\mathcal\{P\}\_\{i\}\}\\\|q\-p\\\|\_\{\\mu\}\}whereq⋆:=arg⁡minp∈𝒮⁡F⁡\(p\)q^\{\\star\}:=\\arg\\min\_\{p\\in\\mathcal\{S\}\}F\(p\)is the*constrained*task minimiser over the bounded\-logit set𝒮:=\{p=softmax⁡\(z/τ\):‖z‖∞≤Gz\}\\mathcal\{S\}:=\\\{p=\\mathrm\{softmax\}\(z/\\tau\):\\\|z\\\|\_\{\\infty\}\\leq G\_\{z\}\\\}\. The set𝒮\\mathcal\{S\}is compact andFFis assumed continuous on it, soq⋆q^\{\\star\}exists\. Constraining to𝒮\\mathcal\{S\}is essential for classification: the unconstrained CE minimiser drives logits to infinity \(one\-hot\), whereas on𝒮\\mathcal\{S\}the minimiser is finite and interior, and under label smoothing \(as used in our experiments\) its target is non\-one\-hot, makingq⋆q^\{\\star\}a genuine interior point consistent with A1\. Letπi:=Π𝒫i​\(q⋆\)\\pi\_\{i\}:=\\Pi\_\{\\mathcal\{P\}\_\{i\}\}\(q^\{\\star\}\)be the closest reachable point of peerii, so1N​∑i‖πi−q⋆‖μ2=Bf2\\tfrac\{1\}\{N\}\\sum\_\{i\}\\\|\\pi\_\{i\}\-q^\{\\star\}\\\|\_\{\\mu\}^\{2\}=B\_\{f\}^\{2\}\. We anchor to the fixedq⋆q^\{\\star\}rather than a freeinfq\\inf\_\{q\}\(the latter is attained by the trivial uniform predictor every peer can represent, giving the uselessBf=0B\_\{f\}=0\)\.Bf=0B\_\{f\}=0iff every peer can represent the required optimum\. This is a*capacity*floor \(approximation error toq⋆q^\{\\star\}\), not a pairwise heterogeneity measure\.BfB\_\{f\}is a theoretical*oracle*quantity depending on the unknownq⋆q^\{\\star\}and the non\-convex𝒫i\\mathcal\{P\}\_\{i\}\. Theorem[G1](https://arxiv.org/html/2609.01952#Thmtheorem1a)uses it only as an upper\-bound parameter\.

##### A4 \(Parameter\-to\-logit realizability, kernel geometry\)\.

LetμM=M−1​∑r=1Mδxr\\mu\_\{M\}=M^\{\-1\}\\sum\_\{r=1\}^\{M\}\\delta\_\{x\_\{r\}\}be the empirical reference measure andℋz=\{Z∈ℝM×C:Zr,:𝟏=0\}\\mathcal\{H\}\_\{z\}=\\\{Z\\in\\mathbb\{R\}^\{M\\times C\}:Z\_\{r,:\}\\mathbf\{1\}=0\\\}with theM−1M^\{\-1\}Frobenius inner product\. DefineZi​\(θi\)=𝖢​\[zi​\(xr,θi\)\]r=1MZ\_\{i\}\(\\theta\_\{i\}\)=\\mathsf\{C\}\[z\_\{i\}\(x\_\{r\};\\theta\_\{i\}\)\]\_\{r=1\}^\{M\},Jiz=Dθi​Zi​\(θi\)J\_\{i\}^\{z\}=D\_\{\\theta\_\{i\}\}Z\_\{i\}\(\\theta\_\{i\}\), andΘi=Jiz​\(Jiz\)∗\\Theta\_\{i\}=J\_\{i\}^\{z\}\(J\_\{i\}^\{z\}\)^\{\*\}\. LetTiz=range⁡\(Jiz\)T\_\{i\}^\{z\}=\\operatorname\{range\}\(J\_\{i\}^\{z\}\)andΠiz=ΠTiz\\Pi\_\{i\}^\{z\}=\\Pi\_\{T\_\{i\}^\{z\}\}\. The frozen restricted NTK satisfies

κ−​Πiz⪯Θi⪯κ\+​Πiz,‖D​Zi​\(θ′\)−D​Zi​\(θ\)‖≤LΘ​‖θ′−θ‖,\\kappa\_\{\-\}\\Pi\_\{i\}^\{z\}\\preceq\\Theta\_\{i\}\\preceq\\kappa\_\{\+\}\\Pi\_\{i\}^\{z\},\\qquad\\\|DZ\_\{i\}\(\\theta^\{\\prime\}\)\-DZ\_\{i\}\(\\theta\)\\\|\\leq L\_\{\\Theta\}\\\|\\theta^\{\\prime\}\-\\theta\\\|,soker⁡\(Θi\)=\(Tiz\)⟂\\ker\(\\Theta\_\{i\}\)=\(T\_\{i\}^\{z\}\)^\{\\perp\}\. This range\-restricted statement is compatible with capacity\-deficient peers and asserts no infinite\-dimensional coercivity\. LetJip=Dθi​piJ\_\{i\}^\{p\}=D\_\{\\theta\_\{i\}\}p\_\{i\}on the reference support,Tip=range⁡\(Jip\)T\_\{i\}^\{p\}=\\operatorname\{range\}\(J\_\{i\}^\{p\}\),Πip=ΠTip\\Pi\_\{i\}^\{p\}=\\Pi\_\{T\_\{i\}^\{p\}\}, andKi=Jip​\(Jip\)∗K\_\{i\}=J\_\{i\}^\{p\}\(J\_\{i\}^\{p\}\)^\{\*\}\. We require explicit prediction\-space constants0<κ¯p≤κ¯p<∞0<\\underline\{\\kappa\}\_\{p\}\\leq\\overline\{\\kappa\}\_\{p\}<\\inftysatisfying

κ¯p​Πip⪯Ki⪯κ¯p​Πip\.\\underline\{\\kappa\}\_\{p\}\\Pi\_\{i\}^\{p\}\\preceq K\_\{i\}\\preceq\\overline\{\\kappa\}\_\{p\}\\Pi\_\{i\}^\{p\}\.ForZi,ZjZ\_\{i\},Z\_\{j\}, define the averaged softmax Jacobian

Ai​j\\displaystyle A\_\{ij\}:=∫01DZ​softmax⁡\(Zj\+s⁡\(Zi−Zj\)τ\)​𝑑s,\\displaystyle:=\\int\_\{0\}^\{1\}D\_\{Z\}\\operatorname\{softmax\}\\left\(\\frac\{Z\_\{j\}\+s\(Z\_\{i\}\-Z\_\{j\}\)\}\{\\tau\}\\right\)ds,\(D1\)Pi−Pj\\displaystyle P\_\{i\}\-P\_\{j\}=Ai​j​\(Zi−Zj\)\.\\displaystyle=A\_\{ij\}\(Z\_\{i\}\-Z\_\{j\}\)\.\(D2\)
For a detached KD teacher,

∇θi\[τ2DKL\(Pj∥Pi\)\]=\(Jiz\)∗\[τ\(Pi−Pj\)\],\\nabla\_\{\\theta\_\{i\}\}\\\!\\left\[\\tau^\{2\}D\_\{\\mathrm\{KL\}\}\(P\_\{j\}\\\|P\_\{i\}\)\\right\]=\(J\_\{i\}^\{z\}\)^\{\*\}\\\!\\left\[\\tau\(P\_\{i\}\-P\_\{j\}\)\\right\],and therefore the random evaluated\-logit recursion is

Δ​Zi=−η⁡\[\(1−α\)​Gitask\+α​τ​Θi​\(Pi−Pj\)\]\+riz\+ξiz,\\displaystyle\\Delta Z\_\{i\}=\-\\eta\\Big\[\(1\-\\alpha\)G\_\{i\}^\{\\mathrm\{task\}\}\+\\alpha\\tau\\Theta\_\{i\}\(P\_\{i\}\-P\_\{j\}\)\\Big\]\+r\_\{i\}^\{z\}\+\\xi\_\{i\}^\{z\},\(D3\)𝔼t​ξiz=0\.\\displaystyle\\mathbb\{E\}\_\{t\}\\xi\_\{i\}^\{z\}=0\.\(D4\)
HereGitaskG\_\{i\}^\{\\mathrm\{task\}\}is the cross\-kernel pushforward from the private CE support to the reference support\. We require𝔼t​‖riz‖2≤Rz​η4\\mathbb\{E\}\_\{t\}\\\|r\_\{i\}^\{z\}\\\|^\{2\}\\leq R\_\{z\}\\eta^\{4\}and𝔼t​‖ξiz‖2≤σz2​η2\\mathbb\{E\}\_\{t\}\\\|\\xi\_\{i\}^\{z\}\\\|^\{2\}\\leq\\sigma\_\{z\}^\{2\}\\eta^\{2\}; these are the explicit second\-moment forms used by the proof\. For ReLU networks the local remainder condition requires a fixed activation pattern or a separately bounded activation\-crossing remainder\. Kernel linearization and empirical conditioning in the relevant local regimes are established, under their respective hypotheses, by Jacot et al\.\([Jacot, Gabriel, and Hongler 2018](https://arxiv.org/html/2609.01952#bib.bib15)\), Lee et al\.\([Lee et al\. 2019](https://arxiv.org/html/2609.01952#bib.bib19)\), Du et al\.\([Du et al\. 2019](https://arxiv.org/html/2609.01952#bib.bib11)\), Allen\-Zhu et al\.\([Allen\-Zhu, Li, and Song 2019](https://arxiv.org/html/2609.01952#bib.bib1)\), and Banerjee et al\.\([Banerjee et al\. 2023](https://arxiv.org/html/2609.01952#bib.bib4)\)\. These results motivate A4 but do not establish it for arbitrary narrow feature\-learning networks\.

##### A10 \(Restricted task/KD alignment\)\.

For everytt, almost surely conditional onℱt\\mathcal\{F\}\_\{t\}, letgit=∇fFi​\(pit\)g\_\{i\}^\{t\}=\\nabla\_\{f\}F\_\{i\}\(p\_\{i\}^\{t\}\)\. The task residual satisfies

1N​∑i‖\(I−Πip,t\)​git‖μ2≤γtask​1N​∑i‖git‖μ2\+νtask,Ψ​Ψtf\+νtask,B​Bf2,0≤γtask<1\.\\begin\{split\}\\frac\{1\}\{N\}\\sum\_\{i\}\\\|\(I\-\\Pi\_\{i\}^\{p,t\}\)g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\\leq\\gamma\_\{\\rm task\}\\frac\{1\}\{N\}\\sum\_\{i\}\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\\\\ \+\\nu\_\{\\rm task,\\Psi\}\\Psi\_\{t\}^\{f\}\+\\nu\_\{\\rm task,B\}B\_\{f\}^\{2\},\\qquad 0\\leq\\gamma\_\{\\rm task\}<1\.\\end\{split\}\(D5\)Fordi​jz=Pi−Pj=Ai​j​\(Zi−Zj\)d\_\{ij\}^\{z\}=P\_\{i\}\-P\_\{j\}=A\_\{ij\}\(Z\_\{i\}\-Z\_\{j\}\), the directed KD residual satisfies

1\|E→\|​∑\(i,j\)∈E→‖\(I−Πiz\)​di​jz‖μM2≤γKD​Ψtz\+νKD,B​Bf2\.\\frac\{1\}\{\|\\vec\{E\}\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|\(I\-\\Pi\_\{i\}^\{z\}\)d\_\{ij\}^\{z\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\\leq\\gamma\_\{\\rm KD\}\\Psi\_\{t\}^\{z\}\+\\nu\_\{\\rm KD,B\}B\_\{f\}^\{2\}\.\(D6\)The transient coefficients are subject to separate absorption margins; the capacity coefficients need only be finite\. This is a restricted\-angle/error condition\. It is not implied by A3 or by a PL inequality\. The appendix proves a graph\-angle sufficient condition, an exact affine\-class case, and the counterexample that necessitates the relative term in Equation[D5](https://arxiv.org/html/2609.01952#A4.E5)\. Bounded\-dissimilarity and relative\-error assumptions play analogous roles in heterogeneous and inexact optimization\([Li et al\. 2020](https://arxiv.org/html/2609.01952#bib.bib21)\), but do not prove A10 for neural prediction manifolds\. For the frozen\-kernel theorem, the projectors in A10 are the fixed projectors from A4; a time\-varying feature\-learning extension would require an additional projector\-drift term\.

##### A5 \(Softmax/KL geometry\)\.

\. On the bounded\-logit set,τ2DKL\(p′∥p\)\\tau^\{2\}D\_\{\\mathrm\{KL\}\}\(p^\{\\prime\}\\\|p\)is two\-sided comparable to the squared mean\-subtracted logit gap, with explicit constants in Lemma[G3](https://arxiv.org/html/2609.01952#Thmlemma3a)\. This lets us pass between KL\-,L2L^\{2\}\-probability\-, andL2L^\{2\}\-logit\-consensus\.

##### A6 \(Data heterogeneity\)\.

The per\-peer CE functional gradients disperse about their mean by at mostζf2\\zeta\_\{f\}^\{2\}:1N​∑i‖∇fFi​\(p\)−∇fF​\(p\)‖μ2≤ζf2\\tfrac\{1\}\{N\}\\sum\_\{i\}\\\|\\nabla\_\{\\\!f\}F\_\{i\}\(p\)\-\\nabla\_\{\\\!f\}F\(p\)\\\|\_\{\\mu\}^\{2\}\\leq\\zeta\_\{f\}^\{2\}\.

##### A7 \(Bounded soft\-target gradient\)\.

\. The KD functional gradientτ⁡\(pi−pj\)\\tau\(p\_\{i\}\-p\_\{j\}\)and the CE functional gradient are bounded \(predictions in the simplex, probabilities≥pmin\\geq p\_\{\\min\}by A1\)\. No separate bounded\-gradient assumption is needed; boundedness is a consequence of A1\.

##### A8 \(Bounded teacher lag\)\.

The teacher is either live, a finite\-memory EMA with support at mostAA, or an ordinary infinite\-memory EMA with decayβ\\beta\. DefineAeff=min⁡\{A,\(1−β\)−1\}A\_\{\\rm eff\}=\\min\\\{A,\(1\-\\beta\)^\{\-1\}\\\}for the finite\-memory case andAeff=\(1−β\)−1A\_\{\\rm eff\}=\(1\-\\beta\)^\{\-1\}for the ordinary infinite\-memory case\. The lag quantity isAtf=N−1​∑i𝔼​‖pit−piEMA,t‖μ2A\_\{t\}^\{f\}=N^\{\-1\}\\sum\_\{i\}\\mathbb\{E\}\\\|p\_\{i\}^\{t\}\-p\_\{i\}^\{\\rm EMA,t\}\\\|\_\{\\mu\}^\{2\}\.

##### A9 \(Objective lower bound\)\.

SimplyF⋆\>−∞F^\{\\star\}\>\-\\infty\.

## Appendix EAssumption Validity and Diagnostics

A4 and A10 are structural bridge assumptions\. A4 connects parameter updates to finite\-support logit or prediction updates through local kernel geometry\. A10 requires the task gradient and KD pull to have enough mass in the tangent directions that each peer can realize\. These assumptions are not universal facts about finite neural networks\. They are local, trajectory\-dependent conditions that make the random\-edge KD recursion analyzable\. The role of this section is to state why these assumptions are mathematically standard in form, give sufficient regimes where they hold, and define empirical diagnostics that can audit them on stored checkpoints\.

### Precedents for A4 and A10

A4 follows the same logic as finite\-support kernel analyses in the neural tangent and lazy\-training literature\. The neural tangent kernel was introduced to describe the training dynamics of wide neural networks through an evolving kernel\([Jacot, Gabriel, and Hongler 2018](https://arxiv.org/html/2609.01952#bib.bib15)\)\. Subsequent wide\-network results show that, under their own width, initialization, and step\-size hypotheses, network outputs evolve close to their linearization around initialization\([Lee et al\. 2019](https://arxiv.org/html/2609.01952#bib.bib19);[Allen\-Zhu, Li, and Song 2019](https://arxiv.org/html/2609.01952#bib.bib1)\)\. Banerjee et al\. study positive definiteness of the NTK at initialization under linear\-width scaling\([Banerjee et al\. 2023](https://arxiv.org/html/2609.01952#bib.bib4)\)\. Bai and Lee analyze quadratic and higher\-order terms, which is useful here because it marks the boundary of a purely linearized argument\([Bai and Lee 2020](https://arxiv.org/html/2609.01952#bib.bib3)\)\. These results support the use of local finite\-support kernels and restricted conditioning, but they do not prove A4 for arbitrary finite\-width feature\-learning networks\.

The range restriction in A4 also has a precedent outside neural networks\. In high\-dimensional statistics, restricted eigenvalue and compatibility conditions require curvature only on identifiable cones or structured subspaces, rather than on the full ambient space\([Bickel, Ritov, and Tsybakov 2009](https://arxiv.org/html/2609.01952#bib.bib5);[van de Geer and Bühlmann 2009](https://arxiv.org/html/2609.01952#bib.bib32)\)\. A4 uses the same principle in prediction space\. It does not require the finite neural tangent kernel to be coercive on all evaluated logits\. It requires coercivity only on the tangent range that the peer’s parameters can move\.

A10 has an analogous role to relative\-error and gradient\-related assumptions in optimization\. Inexact Newton methods allow an approximate linear solve when the residual is controlled relative to the target system\([Dembo, Eisenstat, and Steihaug 1982](https://arxiv.org/html/2609.01952#bib.bib9);[Eisenstat and Walker 1996](https://arxiv.org/html/2609.01952#bib.bib12)\)\. Classical descent analyses allow directions that are not exact gradients when they maintain a nonzero angle with a descent direction\([Nocedal and Wright 2006](https://arxiv.org/html/2609.01952#bib.bib27)\)\. In heterogeneous FL, FedProx uses bounded dissimilarity to control the mismatch between local and global objectives\([Li et al\. 2020](https://arxiv.org/html/2609.01952#bib.bib21)\)\. A10 is the function\-space analogue: it controls the part of the task or KD vector that lies outside the peer’s reachable tangent range, and it charges persistent mismatch to consensus, capacity, and heterogeneity terms\.

Table 1:Closest related settings\. “Reference data” denotes public data, synthetic anchors, generated data, proxy data, or the fixed reference support used for distillation or diagnostics\.WorkSettingServerlessModel heterogeneityReference dataAsynchronyConvergence theoryFedMD / public\-set KD\([Li and Wang 2019](https://arxiv.org/html/2609.01952#bib.bib20)\)Federated public\-set distillationNoYesPublic datasetSynchronousNo random\-edge P2P theoremFedDF / data\-free FL distillation\([Lin et al\. 2020](https://arxiv.org/html/2609.01952#bib.bib24);[Zhu, Hong, and Zhou 2021](https://arxiv.org/html/2609.01952#bib.bib38);[Zhang et al\. 2022](https://arxiv.org/html/2609.01952#bib.bib37)\)Server\-side ensemble or generated\-data distillationNoYesServer proxy or generated dataSynchronousNo serverless P2P theoremFedKD\([Wu et al\. 2022](https://arxiv.org/html/2609.01952#bib.bib33)\)Server\-based adaptive mutual KD with compressionNoPartly, through mentor/mentee structureNo auxiliary public dataRound\-basedNo asynchronous random\-edge theoremFedDKD / decentralized KD line\([Li, Chen, and Lu 2023](https://arxiv.org/html/2609.01952#bib.bib22)\)Client\-to\-client knowledge transfer inside FLPartlyYesProxy or public data depending on variantRound\-basedNo function\-space gossip contraction theoremDeSA\([Huang et al\. 2024](https://arxiv.org/html/2609.01952#bib.bib14)\)Decentralized FL with synthetic anchorsYesYesSynthetic anchorsPeer protocol, not our random\-edge event modelDomain\-adaptation analysis, not our convergence theoremFedEnD\([Martínez Beltrán et al\. 2026](https://arxiv.org/html/2609.01952#bib.bib26)\)Serverless P2P ensemble distillationYesSpecialist\-student heterogeneity in non\-IID dataNo auxiliary public dataOne\-shot broadcast, not live random\-edge gossipNo random\-edge contraction theoremRecent peer KD systems\([Ying et al\. 2026](https://arxiv.org/html/2609.01952#bib.bib35);[Wu et al\. 2026](https://arxiv.org/html/2609.01952#bib.bib34);[Khowaja et al\. 2026](https://arxiv.org/html/2609.01952#bib.bib16)\)Incentive, topology\-aware, multimodal, or edge\-LLM KDOftenOftenProtocol dependentProtocol dependentDifferent guarantee or empirical focusThis paperAsynchronous random\-edge P2P KDYesYesFixed reference supportYesConditional function\-space convergence theorem
### Local Sufficient Conditions for A4

###### Proposition E1\(Finite\-support Taylor bridge\)\.

Fix a peeriiand a finite reference support with centered evaluated logitsZi​\(θi\)∈ℋzZ\_\{i\}\(\\theta\_\{i\}\)\\in\\mathcal\{H\}\_\{z\}\. SupposeD​ZiDZ\_\{i\}isLJL\_\{J\}\-Lipschitz in a neighbourhood ofθit\\theta\_\{i\}^\{t\}, and letΔ​θit=θit\+1−θit\\Delta\\theta\_\{i\}^\{t\}=\\theta\_\{i\}^\{t\+1\}\-\\theta\_\{i\}^\{t\}satisfy𝔼t​‖Δ​θit‖4≤K4​ηt4\\mathbb\{E\}\_\{t\}\\\|\\Delta\\theta\_\{i\}^\{t\}\\\|^\{4\}\\leq K\_\{4\}\\eta\_\{t\}^\{4\}\. Then

Zi​\(θit\+Δ​θit\)−Zi​\(θit\)=Jiz​Δ​θit\+rit;𝔼t​‖rit‖μM2≤LJ2​K44​ηt4Z\_\{i\}\(\\theta\_\{i\}^\{t\}\+\\Delta\\theta\_\{i\}^\{t\}\)\-Z\_\{i\}\(\\theta\_\{i\}^\{t\}\)=J\_\{i\}^\{z\}\\Delta\\theta\_\{i\}^\{t\}\+r\_\{i\}^\{t\};\\quad\\mathbb\{E\}\_\{t\}\\\|r\_\{i\}^\{t\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\\leq\\frac\{L\_\{J\}^\{2\}K\_\{4\}\}\{4\}\\eta\_\{t\}^\{4\}
then, forKiz​p:=Jiz​\(Jip\)∗K\_\{i\}^\{zp\}:=J\_\{i\}^\{z\}\(J\_\{i\}^\{p\}\)^\{\*\}:

Δθit=−ηt\(Jip\)∗git\+νit,𝔼t∥Jizνit∥μM2≤σν,i2ηt2,𝔼t​‖−Zi​\(θit\+Δ​θit\)−Zi​\(θit\)ηt−Kiz​p​git‖μM2≤2​σν,i2\+LJ2​K42​ηt2\.\\begin\{split\}\\Delta\\theta\_\{i\}^\{t\}=\-\\eta\_\{t\}\(J\_\{i\}^\{p\}\)^\{\*\}g\_\{i\}^\{t\}\+\\nu\_\{i\}^\{t\},\\quad\\mathbb\{E\}\_\{t\}\\\|J\_\{i\}^\{z\}\\nu\_\{i\}^\{t\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\\leq\\sigma\_\{\\nu,i\}^\{2\}\\eta\_\{t\}^\{2\},\\\\ \\mathbb\{E\}\_\{t\}\\left\\\|\-\\frac\{Z\_\{i\}\(\\theta\_\{i\}^\{t\}\+\\Delta\\theta\_\{i\}^\{t\}\)\-Z\_\{i\}\(\\theta\_\{i\}^\{t\}\)\}\{\\eta\_\{t\}\}\-K\_\{i\}^\{zp\}g\_\{i\}^\{t\}\\right\\\|\_\{\\mu\_\{M\}\}^\{2\}\\\\ \\leq 2\\sigma\_\{\\nu,i\}^\{2\}\+\\frac\{L\_\{J\}^\{2\}K\_\{4\}\}\{2\}\\eta\_\{t\}^\{2\}\.\\end\{split\}

###### Proof\.

Define the pathφ⁡\(s\)=Zi​\(θit\+s​Δ​θit\)\\varphi\(s\)=Z\_\{i\}\(\\theta\_\{i\}^\{t\}\+s\\Delta\\theta\_\{i\}^\{t\}\)fors∈\[0,1\]s\\in\[0,1\]\. The fundamental theorem of calculus gives

Zi​\(θitCLOSE\\displaystyle Z\_\{i\}\(\\theta\_\{i\}^\{t\}OPEN\+Δ​θit\)−Zi​\(θit\)=φ⁡\(1\)−φ⁡\(0\)\\displaystyle\+\\Delta\\theta\_\{i\}^\{t\}\)\-Z\_\{i\}\(\\theta\_\{i\}^\{t\}\)=\\varphi\(1\)\-\\varphi\(0\)=∫01dd​s​φ​\(s\)​𝑑s\\displaystyle=\\int\_\{0\}^\{1\}\\frac\{d\}\{ds\}\\varphi\(s\)\\,ds=∫01D​Zi​\(θit\+s​Δ​θit\)​Δ​θit​𝑑s\\displaystyle=\\int\_\{0\}^\{1\}DZ\_\{i\}\(\\theta\_\{i\}^\{t\}\+s\\Delta\\theta\_\{i\}^\{t\}\)\\Delta\\theta\_\{i\}^\{t\}\\,ds=Jiz​Δ​θit\+∫01\[D​Zi​\(θit\+s​Δ​θit\)−D​Zi​\(θit\)\]​Δ​θit​𝑑s\\displaystyle=J\_\{i\}^\{z\}\\Delta\\theta\_\{i\}^\{t\}\+\\int\_\{0\}^\{1\}\\bigl\[DZ\_\{i\}\(\\theta\_\{i\}^\{t\}\+s\\Delta\\theta\_\{i\}^\{t\}\)\-DZ\_\{i\}\(\\theta\_\{i\}^\{t\}\)\\bigr\]\\Delta\\theta\_\{i\}^\{t\}\\,ds
The second term isritr\_\{i\}^\{t\}\. By the Lipschitz condition onD​ZiDZ\_\{i\},

‖rit‖μM\\displaystyle\\\|r\_\{i\}^\{t\}\\\|\_\{\\mu\_\{M\}\}≤∫01‖D​Zi​\(θit\+s​Δ​θit\)−D​Zi​\(θit\)‖​‖Δ​θit‖​𝑑s\\displaystyle\\leq\\int\_\{0\}^\{1\}\\\|DZ\_\{i\}\(\\theta\_\{i\}^\{t\}\+s\\Delta\\theta\_\{i\}^\{t\}\)\-DZ\_\{i\}\(\\theta\_\{i\}^\{t\}\)\\\|\\\|\\Delta\\theta\_\{i\}^\{t\}\\\|\\,ds≤∫01LJ​s​‖Δ​θit‖2​𝑑s\\displaystyle\\leq\\int\_\{0\}^\{1\}L\_\{J\}s\\\|\\Delta\\theta\_\{i\}^\{t\}\\\|^\{2\}\\,ds=LJ2​‖Δ​θit‖2\.\\displaystyle=\\frac\{L\_\{J\}\}\{2\}\\\|\\Delta\\theta\_\{i\}^\{t\}\\\|^\{2\}\.Squaring the last display and taking conditional expectation gives

𝔼t​‖rit‖μM2≤LJ24​𝔼t​‖Δ​θit‖4≤LJ2​K44​ηt4\.\\mathbb\{E\}\_\{t\}\\\|r\_\{i\}^\{t\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\\leq\\frac\{L\_\{J\}^\{2\}\}\{4\}\\mathbb\{E\}\_\{t\}\\\|\\Delta\\theta\_\{i\}^\{t\}\\\|^\{4\}\\leq\\frac\{L\_\{J\}^\{2\}K\_\{4\}\}\{4\}\\eta\_\{t\}^\{4\}\.Under the first\-order update decomposition,

Zi​\(θit\+Δ​θit\)−Zi​\(θit\)\\displaystyle Z\_\{i\}\(\\theta\_\{i\}^\{t\}\+\\Delta\\theta\_\{i\}^\{t\}\)\-Z\_\{i\}\(\\theta\_\{i\}^\{t\}\)=Jiz​\[−ηt​\(Jip\)∗​git\+νit\]\+rit\\displaystyle=J\_\{i\}^\{z\}\\bigl\[\-\\eta\_\{t\}\(J\_\{i\}^\{p\}\)^\{\*\}g\_\{i\}^\{t\}\+\\nu\_\{i\}^\{t\}\\bigr\]\+r\_\{i\}^\{t\}=−ηt​Kiz​p​git\+Jiz​νit\+rit\.\\displaystyle=\-\\eta\_\{t\}K\_\{i\}^\{zp\}g\_\{i\}^\{t\}\+J\_\{i\}^\{z\}\\nu\_\{i\}^\{t\}\+r\_\{i\}^\{t\}\.so we have:

−Zi​\(θit\+Δ​θit\)−Zi​\(θit\)ηt−Kiz​p​git=−Jiz​νitηt−ritηt\.\-\\frac\{Z\_\{i\}\(\\theta\_\{i\}^\{t\}\+\\Delta\\theta\_\{i\}^\{t\}\)\-Z\_\{i\}\(\\theta\_\{i\}^\{t\}\)\}\{\\eta\_\{t\}\}\-K\_\{i\}^\{zp\}g\_\{i\}^\{t\}=\-\\frac\{J\_\{i\}^\{z\}\\nu\_\{i\}^\{t\}\}\{\\eta\_\{t\}\}\-\\frac\{r\_\{i\}^\{t\}\}\{\\eta\_\{t\}\}\.For vectorsaaandbb, the inequality‖a\+b‖2≤2​‖a‖2\+2​‖b‖2\\\|a\+b\\\|^\{2\}\\leq 2\\\|a\\\|^\{2\}\+2\\\|b\\\|^\{2\}gives

𝔼t​‖−Zi​\(θit\+Δ​θit\)−Zi​\(θit\)ηt−Kiz​p​git‖μM2\\displaystyle\\mathbb\{E\}\_\{t\}\\left\\\|\-\\frac\{Z\_\{i\}\(\\theta\_\{i\}^\{t\}\+\\Delta\\theta\_\{i\}^\{t\}\)\-Z\_\{i\}\(\\theta\_\{i\}^\{t\}\)\}\{\\eta\_\{t\}\}\-K\_\{i\}^\{zp\}g\_\{i\}^\{t\}\\right\\\|\_\{\\mu\_\{M\}\}^\{2\}≤2ηt2​𝔼t​‖Jiz​νit‖μM2\+2ηt2​𝔼t​‖rit‖μM2\\displaystyle\\leq\\frac\{2\}\{\\eta\_\{t\}^\{2\}\}\\mathbb\{E\}\_\{t\}\\\|J\_\{i\}^\{z\}\\nu\_\{i\}^\{t\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\+\\frac\{2\}\{\\eta\_\{t\}^\{2\}\}\\mathbb\{E\}\_\{t\}\\\|r\_\{i\}^\{t\}\\\|\_\{\\mu\_\{M\}\}^\{2\}≤2​σν,i2\+LJ2​K42​ηt2\.\\displaystyle\\leq 2\\sigma\_\{\\nu,i\}^\{2\}\+\\frac\{L\_\{J\}^\{2\}K\_\{4\}\}\{2\}\\eta\_\{t\}^\{2\}\.∎

Proposition[E1](https://arxiv.org/html/2609.01952#Thmproposition1)proves local logit linearization and a unit\-consistent task\-pushforward residual, which is a partial proof or at least a support for A4 though it does not by itself prove the cross\-kernel fidelity clause in A4\. Thus, we state here that this assumption that we made in the main paper is neither arbitrary nor overly strong\. If the proof uses a prediction\-space kernelKi=Jip​\(Jip\)∗K\_\{i\}=J\_\{i\}^\{p\}\(J\_\{i\}^\{p\}\)^\{\*\}while the evaluated logit update containsKiz​p=Jiz​\(Jip\)∗K\_\{i\}^\{zp\}=J\_\{i\}^\{z\}\(J\_\{i\}^\{p\}\)^\{\*\}, one also needs a finite\-support range\-overlap condition\. A sufficient version is that the principal angle between the task\-induced parameter row space and the logit row space is bounded away fromπ/2\\pi/2on the observed trajectory\. Under such an angle bound, the part of\(Jip\)∗​git\(J\_\{i\}^\{p\}\)^\{\*\}g\_\{i\}^\{t\}invisible toJizJ\_\{i\}^\{z\}is controlled by the visible part, and the remaining operator difference can be charged to the A4 residual\. Without this overlap, two kernels can be individually well conditioned on their own ranges while still moving in almost orthogonal parameter directions\.

### Local Sufficient Conditions for A10

###### Proposition E2\(Graph\-angle condition implies the KD clause of A10\)\.

Under the undirected communication graph, and letE→\\vec\{E\}contain both orientations of every edge\. Letwi​j=wj​i≥0w\_\{ij\}=w\_\{ji\}\\geq 0be symmetric directed\-edge weights\. On centered finite\-support logits, definedi​jz=Pi−Pjd\_\{ij\}^\{z\}=P\_\{i\}\-P\_\{j\}\. Suppose there are constantsρKD≥0\\rho\_\{\\rm KD\}\\geq 0andβKD≥0\\beta\_\{\\rm KD\}\\geq 0such that

∑\(i,j\)∈E→wi​j​‖\(I−Πiz\)​di​jz‖μM2≤ρKD​∑\(i,j\)∈E→wi​j​‖di​jz‖μM2\\displaystyle\\sum\_\{\(i,j\)\\in\\vec\{E\}\}w\_\{ij\}\\\|\(I\-\\Pi\_\{i\}^\{z\}\)d\_\{ij\}^\{z\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\\leq\\rho\_\{\\rm KD\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}w\_\{ij\}\\\|d\_\{ij\}^\{z\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\(E7\)\+βKD∑iϵi2\.\\displaystyle\+\\beta\_\{\\rm KD\}\\sum\_\{i\}\\epsilon\_\{i\}^\{2\}\.\(E8\)
If the softmax map isLsmL\_\{\\rm sm\}\-Lipschitz from centered logits to probabilities on the bounded\-logit region andN−1​∑iϵi2≤cB​Bf2N^\{\-1\}\\sum\_\{i\}\\epsilon\_\{i\}^\{2\}\\leq c\_\{B\}B\_\{f\}^\{2\}, then

1\|E→\|​∑\(i,j\)∈E→‖\(I−Πiz\)​di​jz‖μM2≤γKD​Ψtz\+νKD,B​Bf2\\displaystyle\\frac\{1\}\{\|\\vec\{E\}\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|\(I\-\\Pi\_\{i\}^\{z\}\)d\_\{ij\}^\{z\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\\leq\\gamma\_\{\\rm KD\}\\Psi\_\{t\}^\{z\}\+\\nu\_\{\{\\rm KD\},B\}B\_\{f\}^\{2\}\(E9\)γKD=4​ρKD​Lsm2​dmax​N\|E→\|,νKD,B=βKD​N​cB\|E→\|\.\\displaystyle\\gamma\_\{\\rm KD\}=\\frac\{4\\rho\_\{\\rm KD\}L\_\{\\rm sm\}^\{2\}d\_\{\\max\}N\}\{\|\\vec\{E\}\|\},\\qquad\\nu\_\{\{\\rm KD\},B\}=\\frac\{\\beta\_\{\\rm KD\}Nc\_\{B\}\}\{\|\\vec\{E\}\|\}\.\(E10\)

###### Proof\.

For every oriented edge, the definitiondi​jz=Pi−Pjd\_\{ij\}^\{z\}=P\_\{i\}\-P\_\{j\}and the Lipschitz property of softmax give

‖di​jz‖μM2=‖Pi−Pj‖μM2≤Lsm2​‖Zi−Zj‖μM2\.\\displaystyle\\\|d\_\{ij\}^\{z\}\\\|\_\{\\mu\_\{M\}\}^\{2\}=\\\|P\_\{i\}\-P\_\{j\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\\leq L\_\{\\rm sm\}^\{2\}\\\|Z\_\{i\}\-Z\_\{j\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\.\(E11\)∑\(i,j\)∈E→‖Zi−Zj‖μM2=2​∑\{i,j\}∈E‖Zi−Zj‖μM2\.\\displaystyle\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|Z\_\{i\}\-Z\_\{j\}\\\|\_\{\\mu\_\{M\}\}^\{2\}=2\\sum\_\{\\\{i,j\\\}\\in E\}\\\|Z\_\{i\}\-Z\_\{j\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\.\(E12\)
LetZ¯=N−1​∑iZi\\bar\{Z\}=N^\{\-1\}\\sum\_\{i\}Z\_\{i\}\. For each undirected edge and Peter\-Paul Inequality, we sum over all undirected edges counts each peer at mostdmaxd\_\{\\max\}times,

Zi−Zj=\(Zi−Z¯\)−\(Zj−Z¯\)\.‖Zi−Zj‖μM2≤2​‖Zi−Z¯‖μM2\+2​‖Zj−Z¯‖μM2\.∑\{i,j\}∈E‖Zi−Zj‖μM2≤2​dmax​∑i‖Zi−Z¯‖μM2=2​dmax​N​Ψtz\.∑\(i,j\)∈E→‖di​jz‖μM2≤4​Lsm2​dmax​N​Ψtz\.\\begin\{split\}Z\_\{i\}\-Z\_\{j\}&=\(Z\_\{i\}\-\\bar\{Z\}\)\-\(Z\_\{j\}\-\\bar\{Z\}\)\.\\\\ \\\|Z\_\{i\}\-Z\_\{j\}\\\|\_\{\\mu\_\{M\}\}^\{2\}&\\leq 2\\\|Z\_\{i\}\-\\bar\{Z\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\+2\\\|Z\_\{j\}\-\\bar\{Z\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\.\\\\ \\sum\_\{\\\{i,j\\\}\\in E\}\\\|Z\_\{i\}\-Z\_\{j\}\\\|\_\{\\mu\_\{M\}\}^\{2\}&\\leq 2d\_\{\\max\}\\sum\_\{i\}\\\|Z\_\{i\}\-\\bar\{Z\}\\\|\_\{\\mu\_\{M\}\}^\{2\}\\\\ &=2d\_\{\\max\}N\\Psi\_\{t\}^\{z\}\.\\\\ \\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|d\_\{ij\}^\{z\}\\\|\_\{\\mu\_\{M\}\}^\{2\}&\\leq 4L\_\{\\rm sm\}^\{2\}d\_\{\\max\}N\\Psi\_\{t\}^\{z\}\.\\end\{split\}Insert this bound into the graph\-angle condition and use∑iϵi2≤N​cB​Bf2\\sum\_\{i\}\\epsilon\_\{i\}^\{2\}\\leq Nc\_\{B\}B\_\{f\}^\{2\}:

∑\(i,j\)∈E→‖\(I−Πiz\)​di​jz‖μM2\\displaystyle\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|\(I\-\\Pi\_\{i\}^\{z\}\)d\_\{ij\}^\{z\}\\\|\_\{\\mu\_\{M\}\}^\{2\}≤4​ρKD​Lsm2​dmax​N​Ψtz\+βKD​N​cB​Bf2\.\\displaystyle\\leq 4\\rho\_\{\\rm KD\}L\_\{\\rm sm\}^\{2\}d\_\{\\max\}N\\Psi\_\{t\}^\{z\}\+\\beta\_\{\\rm KD\}Nc\_\{B\}B\_\{f\}^\{2\}\.Dividing by\|E→\|\|\\vec\{E\}\|proves the displayed A10\-KD bound\. The weighted case is identical after replacingdmaxd\_\{\\max\}and\|E→\|\|\\vec\{E\}\|by the corresponding weighted degree and total directed weight\. ∎

Table 2:Mapping between proof objects and implementation\-level quantities\. The table is meant to make A4 and A10 auditable rather than implicit\.###### Proposition E3\(Local leakage condition implies the task clause of A10\)\.

For each peer, letΠip\\Pi\_\{i\}^\{p\}denote the tangent projector on the reference support\. Suppose the local task gradient admits the decomposition

‖\(I−Πip\)​git‖μ\\displaystyle\\\|\(I\-\\Pi\_\{i\}^\{p\}\)g\_\{i\}^\{t\}\\\|\_\{\\mu\}≤κi​‖Πip​git‖μ\+ai​‖pit−p¯t‖μ\\displaystyle\\leq\\kappa\_\{i\}\\\|\\Pi\_\{i\}^\{p\}g\_\{i\}^\{t\}\\\|\_\{\\mu\}\+a\_\{i\}\\\|p\_\{i\}^\{t\}\-\\bar\{p\}^\{t\}\\\|\_\{\\mu\}\+bi​‖πi−q⋆‖μ\+ci​‖∇fFi​\(q⋆\)−∇fF​\(q⋆\)‖μ,\\displaystyle\+b\_\{i\}\\\|\\pi\_\{i\}\-q^\{\\star\}\\\|\_\{\\mu\}\+c\_\{i\}\\\|\\nabla\_\{f\}F\_\{i\}\(q^\{\\star\}\)\-\\nabla\_\{f\}F\(q^\{\\star\}\)\\\|\_\{\\mu\},whereκi≥0\\kappa\_\{i\}\\geq 0measures local tangent\-normal leakage and\(1\+ε\)​κi2≤γ¯<1\(1\+\\varepsilon\)\\kappa\_\{i\}^\{2\}\\leq\\bar\{\\gamma\}<1for someε\>0\\varepsilon\>0and allii\. Then there are finite constantsνtask,Ψ\\nu\_\{\\rm task,\\Psi\},νtask,B\\nu\_\{\\rm task,B\}, andνtask,ζ\\nu\_\{\\rm task,\\zeta\}such that

1N​∑i‖\(I−Πip\)​git‖μ2≤γ¯​1N​∑i‖git‖μ2\+νtask,Ψ​Ψtf\\displaystyle\\frac\{1\}\{N\}\\sum\_\{i\}\\\|\(I\-\\Pi\_\{i\}^\{p\}\)g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\\leq\\bar\{\\gamma\}\\frac\{1\}\{N\}\\sum\_\{i\}\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\+\\nu\_\{\\rm task,\\Psi\}\\Psi\_\{t\}^\{f\}\(E13\)\+νtask,B​Bf2\+νtask,ζ​ζf2\.\\displaystyle\+\\nu\_\{\\rm task,B\}B\_\{f\}^\{2\}\+\\nu\_\{\\rm task,\\zeta\}\\zeta\_\{f\}^\{2\}\.\(E14\)

###### Proof\.

We define

ni=\(I−Πip\)​git,\\displaystyle n\_\{i\}=\(I\-\\Pi\_\{i\}^\{p\}\)g\_\{i\}^\{t\},\\qquadti=Πip​git,\\displaystyle t\_\{i\}=\\Pi\_\{i\}^\{p\}g\_\{i\}^\{t\},\(E15\)ri=ai​‖pit−p¯t‖μ\+bi​‖πi−q⋆‖μ\\displaystyle r\_\{i\}=a\_\{i\}\\\|p\_\{i\}^\{t\}\-\\bar\{p\}^\{t\}\\\|\_\{\\mu\}\+b\_\{i\}\\\|\\pi\_\{i\}\-q^\{\\star\}\\\|\_\{\\mu\}\+ci​‖∇fFi​\(q⋆\)−∇fF​\(q⋆\)‖μ\.\\displaystyle\+c\_\{i\}\\\|\\nabla\_\{f\}F\_\{i\}\(q^\{\\star\}\)\-\\nabla\_\{f\}F\(q^\{\\star\}\)\\\|\_\{\\mu\}\.\(E16\)
The assumed leakage condition gives

‖ni‖μ≤κi​‖ti‖μ\+ri\.\\\|n\_\{i\}\\\|\_\{\\mu\}\\leq\\kappa\_\{i\}\\\|t\_\{i\}\\\|\_\{\\mu\}\+r\_\{i\}\.For anyε\>0\\varepsilon\>0, Young’s inequality in the form\(x\+y\)2≤\(1\+ε\)​x2\+\(1\+ε−1\)​y2\(x\+y\)^\{2\}\\leq\(1\+\\varepsilon\)x^\{2\}\+\(1\+\\varepsilon^\{\-1\}\)y^\{2\}gives

‖ni‖μ2≤\(1\+ε\)​κi2​‖ti‖μ2\+\(1\+ε−1\)​ri2\.\\\|n\_\{i\}\\\|\_\{\\mu\}^\{2\}\\leq\(1\+\\varepsilon\)\\kappa\_\{i\}^\{2\}\\\|t\_\{i\}\\\|\_\{\\mu\}^\{2\}\+\(1\+\\varepsilon^\{\-1\}\)r\_\{i\}^\{2\}\.Sincetit\_\{i\}andnin\_\{i\}are orthogonal projections ofgitg\_\{i\}^\{t\}and\(1\+ε\)​κi2≤γ¯\(1\+\\varepsilon\)\\kappa\_\{i\}^\{2\}\\leq\\bar\{\\gamma\}, we have:

‖ti‖μ2≤‖ti‖μ2\+‖ni‖μ2=‖git‖μ2\.\\displaystyle\\\|t\_\{i\}\\\|\_\{\\mu\}^\{2\}\\leq\\\|t\_\{i\}\\\|\_\{\\mu\}^\{2\}\+\\\|n\_\{i\}\\\|\_\{\\mu\}^\{2\}=\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\.\(E17\)‖ni‖μ2≤γ¯​‖git‖μ2\+\(1\+ε−1\)​ri2\.\\displaystyle\\\|n\_\{i\}\\\|\_\{\\mu\}^\{2\}\\leq\\bar\{\\gamma\}\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\+\(1\+\\varepsilon^\{\-1\}\)r\_\{i\}^\{2\}\.\(E18\)
For three nonnegative termsuu,vv, andww,\(u\+v\+w\)2≤3​u2\+3​v2\+3​w2\(u\+v\+w\)^\{2\}\\leq 3u^\{2\}\+3v^\{2\}\+3w^\{2\}\. Applying this torir\_\{i\}yields

ri2≤3​ai2​‖pit−p¯t‖μ2\+3​bi2​‖πi−q⋆‖μ2\+\\displaystyle r\_\{i\}^\{2\}\\leq 3a\_\{i\}^\{2\}\\\|p\_\{i\}^\{t\}\-\\bar\{p\}^\{t\}\\\|\_\{\\mu\}^\{2\}\+3b\_\{i\}^\{2\}\\\|\\pi\_\{i\}\-q^\{\\star\}\\\|\_\{\\mu\}^\{2\}\+3​ci2​‖∇fFi​\(q⋆\)−∇fF​\(q⋆\)‖μ2\.\\displaystyle 3c\_\{i\}^\{2\}\\\|\\nabla\_\{f\}F\_\{i\}\(q^\{\\star\}\)\-\\nabla\_\{f\}F\(q^\{\\star\}\)\\\|\_\{\\mu\}^\{2\}\.Average the previous display over peers\. By the definitions ofΨtf\\Psi\_\{t\}^\{f\}andBf2B\_\{f\}^\{2\},

1N​∑i‖pit−p¯t‖μ2=Ψtf,1N​∑i‖πi−q⋆‖μ2=Bf2\.\\frac\{1\}\{N\}\\sum\_\{i\}\\\|p\_\{i\}^\{t\}\-\\bar\{p\}^\{t\}\\\|\_\{\\mu\}^\{2\}=\\Psi\_\{t\}^\{f\},\\qquad\\frac\{1\}\{N\}\\sum\_\{i\}\\\|\\pi\_\{i\}\-q^\{\\star\}\\\|\_\{\\mu\}^\{2\}=B\_\{f\}^\{2\}\.The final term is bounded by the heterogeneity constant atq⋆q^\{\\star\}, or by its local analogue if A6 is stated on the bounded\-logit neighbourhood:

1N​∑i‖∇fFi​\(q⋆\)−∇fF​\(q⋆\)‖μ2≤ζf2\.\\frac\{1\}\{N\}\\sum\_\{i\}\\\|\\nabla\_\{f\}F\_\{i\}\(q^\{\\star\}\)\-\\nabla\_\{f\}F\(q^\{\\star\}\)\\\|\_\{\\mu\}^\{2\}\\leq\\zeta\_\{f\}^\{2\}\.Collecting the constants gives the stated task\-alignment inequality\. ∎

Proposition[E3](https://arxiv.org/html/2609.01952#Thmproposition3)is deliberately local\. It holds in an affine prediction class with squared loss when the target decomposes into a reachable tangent component plus capacity and heterogeneity residuals\. For cross\-entropy it applies after local linearization in centered\-logit or prediction coordinates and after the leakage coefficient is measured or bounded\.

### Proof\-to\-Implementation Map

### Checkpoint\-Level Diagnostics

The strongest version of the empirical section would not only show consensus contraction\. It would also report whether the recorded trajectory lies in the A4/A10 regime\. The following diagnostics use the same finite reference support as the proof\. Letδ\>0\\delta\>0be a small numerical stabilizer\.

For A4, store logits immediately before and after a small task update and estimate

RA4​\(t\)=∑i‖−ΔZi\(t\)/ηt−Kiz​p\(t\)gi\(t\)‖μM2∑i‖Kiz​p​\(t\)​gi​\(t\)‖μM2\+δ\.R\_\{\\rm A4\}\(t\)=\\frac\{\\sum\_\{i\}\\left\\\|\-\\Delta Z\_\{i\}\(t\)/\\eta\_\{t\}\-K\_\{i\}^\{zp\}\(t\)g\_\{i\}\(t\)\\right\\\|\_\{\\mu\_\{M\}\}^\{2\}\}\{\\sum\_\{i\}\\\|K\_\{i\}^\{zp\}\(t\)g\_\{i\}\(t\)\\\|\_\{\\mu\_\{M\}\}^\{2\}\+\\delta\}\.\(E19\)Small values ofRA4​\(t\)R\_\{\\rm A4\}\(t\)indicate that the observed parameter step has the finite\-support functional effect assumed by A4\. This statistic should be computed from before/after update pairs\. A final checkpoint alone is not enough to reconstructΔ​Zi​\(t\)\\Delta Z\_\{i\}\(t\)for each event\.

For the KD part of A10, a single checkpoint and the graph are enough if the projectors are estimated independently:

RA10KD​\(t\)=∑\(i,j\)∈E→wi​j​‖\(I−Πiz\)​di​jz​\(t\)‖μM2∑\(i,j\)∈E→wi​j​‖di​jz​\(t\)‖μM2\+δ\.R\_\{\\rm A10\}^\{\\rm KD\}\(t\)=\\frac\{\\sum\_\{\(i,j\)\\in\\vec\{E\}\}w\_\{ij\}\\\|\(I\-\\Pi\_\{i\}^\{z\}\)d\_\{ij\}^\{z\}\(t\)\\\|\_\{\\mu\_\{M\}\}^\{2\}\}\{\\sum\_\{\(i,j\)\\in\\vec\{E\}\}w\_\{ij\}\\\|d\_\{ij\}^\{z\}\(t\)\\\|\_\{\\mu\_\{M\}\}^\{2\}\+\\delta\}\.\(E20\)This ratio measures how much of the directed KD pull falls outside each student’s logit tangent space\.

For the task part of A10, use

RA10task​\(t\)=∑i‖\(I−Πip\)​git‖μ2∑i‖git‖μ2\+δ\.R\_\{\\rm A10\}^\{\\rm task\}\(t\)=\\frac\{\\sum\_\{i\}\\\|\(I\-\\Pi\_\{i\}^\{p\}\)g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\}\{\\sum\_\{i\}\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\+\\delta\}\.\(E21\)This ratio measures the normal component of the task gradient\. It should be reported together withΨtf\\Psi\_\{t\}^\{f\}, because A10 permits the task normal component to increase with consensus error and capacity mismatch\.

If these ratios are not measured, the theorem statement and experiment interpretation should remain narrowed\. The correct claim is that the reported experiments validate observable consequences of the conditional theory, especially prediction\-space contraction and finite\-reference stationarity proxies, under the stated A4/A10 bridge assumptions\. A stronger claim that the neural CIFAR\-10 trajectories themselves satisfy A4 and A10 should be made only after reporting Equations[E19](https://arxiv.org/html/2609.01952#A5.E19)–[E21](https://arxiv.org/html/2609.01952#A5.E21)or comparable assumption\-margin diagnostics\.

## Appendix FConvergence Proof

The following results give the complete convergence proof\. All conditional expectations are taken with respect to the event historyℱt\\mathcal\{F\}\_\{t\}\.

###### Lemma G1\(Exact KD bridge and finite\-support remainder\)\.

LetPi=softmax⁡\(Zi/τ\)P\_\{i\}=\\operatorname\{softmax\}\(Z\_\{i\}/\\tau\)and letPjTP\_\{j\}^\{T\}be detached\. Then

∇Zi\[τ2DKL\(PjT∥Pi\)\]=τ\(Pi−PjT\)\.\\nabla\_\{Z\_\{i\}\}\\\!\\left\[\\tau^\{2\}D\_\{\\rm KL\}\(P\_\{j\}^\{T\}\\\|P\_\{i\}\)\\right\]=\\tau\(P\_\{i\}\-P\_\{j\}^\{T\}\)\.\(G1\)IfZi​\(θ\)Z\_\{i\}\(\\theta\)has anLJL\_\{J\}\-Lipschitz Jacobian on the segment joiningθi\\theta\_\{i\}andθi\+Δ​θi\\theta\_\{i\}\+\\Delta\\theta\_\{i\}, then

Zi​\(θi\+Δ​θi\)−Zi​\(θi\)=Jiz​Δ​θi\+riz,‖riz‖≤LJ2​‖Δ​θi‖2\.\\begin\{split\}Z\_\{i\}\(\\theta\_\{i\}\+\\Delta\\theta\_\{i\}\)\-Z\_\{i\}\(\\theta\_\{i\}\)=J\_\{i\}^\{z\}\\Delta\\theta\_\{i\}\+r\_\{i\}^\{z\},\\\\ \\\|r\_\{i\}^\{z\}\\\|\\leq\\frac\{L\_\{J\}\}\{2\}\\\|\\Delta\\theta\_\{i\}\\\|^\{2\}\.\\end\{split\}\(G2\)Consequently, a parameter KD step gives

Δ​ZiKD=−η​α​τ​Θi​\(Pi−PjT\)\+riz,Θi=Jiz​\(Jiz\)∗\.\\begin\{split\}\\Delta Z\_\{i\}^\{\\rm KD\}=\-\\eta\\alpha\\tau\\Theta\_\{i\}\(P\_\{i\}\-P\_\{j\}^\{T\}\)\+r\_\{i\}^\{z\},\\\\ \\Theta\_\{i\}=J\_\{i\}^\{z\}\(J\_\{i\}^\{z\}\)^\{\*\}\.\\end\{split\}\(G3\)

###### Proof\.

LetCCbe the number of classes and letμM=M−1​∑r=1Mδxr\\mu\_\{M\}=M^\{\-1\}\\sum\_\{r=1\}^\{M\}\\delta\_\{x\_\{r\}\}be the empirical reference measure\. For finite\-support arrays, use

⟨U,V⟩μM=1M​∑r=1M∑c=1CUr,c​Vr,c\\langle U,V\\rangle\_\{\\mu\_\{M\}\}=\\frac\{1\}\{M\}\\sum\_\{r=1\}^\{M\}\\sum\_\{c=1\}^\{C\}U\_\{r,c\}V\_\{r,c\}and its induced norm\. Equation[G1](https://arxiv.org/html/2609.01952#A7.E1)is the Riesz gradient under this inner product, so the empirical factorM−1M^\{\-1\}is part of the inner product\. For one reference point, setsi,c=Zi,c/τs\_\{i,c\}=Z\_\{i,c\}/\\tauand write

DKL\(PjT∥Pi\)=∑c=1CPj,cT\(logPj,cT−logPi,c\)\.D\_\{\\rm KL\}\(P\_\{j\}^\{T\}\\\|P\_\{i\}\)=\\sum\_\{c=1\}^\{C\}P\_\{j,c\}^\{T\}\\left\(\\log P\_\{j,c\}^\{T\}\-\\log P\_\{i,c\}\\right\)\.The teacher is detached, soPj,cTP\_\{j,c\}^\{T\}andlog⁡Pj,cT\\log P\_\{j,c\}^\{T\}are constants when differentiating with respect tosis\_\{i\}\. Since

log⁡Pi,c=si,c−log⁡\(∑a=1Cesi,a\),\\log P\_\{i,c\}=s\_\{i,c\}\-\\log\\\!\\left\(\\sum\_\{a=1\}^\{C\}e^\{s\_\{i,a\}\}\\right\),the component derivative for classbbis

∂DKL∂si,b\\displaystyle\\frac\{\\partial D\_\{\\rm KL\}\}\{\\partial s\_\{i,b\}\}=−∑c=1CPj,cT\(𝟏\{c=b\}−esi,b∑a=1Cesi,a\)\\displaystyle=\-\\sum\_\{c=1\}^\{C\}P\_\{j,c\}^\{T\}\\left\(\\mathbf\{1\}\\\{c=b\\\}\-\\frac\{e^\{s\_\{i,b\}\}\}\{\\sum\_\{a=1\}^\{C\}e^\{s\_\{i,a\}\}\}\\right\)=−Pj,bT\+Pi,b​∑c=1CPj,cT\\displaystyle=\-P\_\{j,b\}^\{T\}\+P\_\{i,b\}\\sum\_\{c=1\}^\{C\}P\_\{j,c\}^\{T\}=Pi,b−Pj,bT\.\\displaystyle=P\_\{i,b\}\-P\_\{j,b\}^\{T\}\.The second line separates the selected\-logit derivative from the log\-normalizer derivative\. The last line uses∑cPj,cT=1\\sum\_\{c\}P\_\{j,c\}^\{T\}=1\. Thus the fullsis\_\{i\}\-gradient is the column vectorPi−PjTP\_\{i\}\-P\_\{j\}^\{T\}\. The mapsi=Zi/τs\_\{i\}=Z\_\{i\}/\\tauhas JacobianDZi​si=τ−1​ID\_\{Z\_\{i\}\}s\_\{i\}=\\tau^\{\-1\}I, so the exact chain rule gives

∇Zi\[τ2DKL\(PjT∥Pi\)\]\\displaystyle\\nabla\_\{Z\_\{i\}\}\\left\[\\tau^\{2\}D\_\{\\rm KL\}\(P\_\{j\}^\{T\}\\\|P\_\{i\}\)\\right\]=τ2​\(DZi​si\)∗​\(Pi−PjT\)\\displaystyle=\\tau^\{2\}\(D\_\{Z\_\{i\}\}s\_\{i\}\)^\{\*\}\(P\_\{i\}\-P\_\{j\}^\{T\}\)=τ2​\(1τ​I\)⊤​\(Pi−PjT\)\\displaystyle=\\tau^\{2\}\\left\(\\frac\{1\}\{\\tau\}I\\right\)^\{\\top\}\(P\_\{i\}\-P\_\{j\}^\{T\}\)=τ⁡\(Pi−PjT\)\.\\displaystyle=\\tau\(P\_\{i\}\-P\_\{j\}^\{T\}\)\.Moreover,∑c\(Pi,c−Pj,cT\)=1−1=0\\sum\_\{c\}\(P\_\{i,c\}\-P\_\{j,c\}^\{T\}\)=1\-1=0, so the gradient belongs to the centered class subspace\. For the parameter gradient, define the finite\-support JacobianJiz=Dθi​Zi​\(θi\)J\_\{i\}^\{z\}=D\_\{\\theta\_\{i\}\}Z\_\{i\}\(\\theta\_\{i\}\)\. Thus, for parameter coordinateaa,

∂∂θi,a\[τ2DKL\(PjT∥Pi\)\]\\displaystyle\\frac\{\\partial\}\{\\partial\\theta\_\{i,a\}\}\\left\[\\tau^\{2\}D\_\{\\rm KL\}\(P\_\{j\}^\{T\}\\\|P\_\{i\}\)\\right\]=1M​∑r=1M∑c=1C∂Zi,r,c∂θi,a​τ​\(Pi,r,c−Pj,r,cT\)\\displaystyle=\\frac\{1\}\{M\}\\sum\_\{r=1\}^\{M\}\\sum\_\{c=1\}^\{C\}\\frac\{\\partial Z\_\{i,r,c\}\}\{\\partial\\theta\_\{i,a\}\}\\tau\(P\_\{i,r,c\}\-P\_\{j,r,c\}^\{T\}\)=\[\(Jiz\)∗​τ​\(Pi−PjT\)\]a\.\\displaystyle=\\left\[\(J\_\{i\}^\{z\}\)^\{\*\}\\tau\(P\_\{i\}\-P\_\{j\}^\{T\}\)\\right\]\_\{a\}\.The adjoint is defined by⟨Jiz​h,U⟩μM=⟨h,\(Jiz\)∗​U⟩\\langle J\_\{i\}^\{z\}h,U\\rangle\_\{\\mu\_\{M\}\}=\\langle h,\(J\_\{i\}^\{z\}\)^\{\*\}U\\ranglefor every parameter directionhh\.

Now defineϑ⁡\(s\)=θi\+s​Δ​θi\\vartheta\(s\)=\\theta\_\{i\}\+s\\Delta\\theta\_\{i\}fors∈\[0,1\]s\\in\[0,1\]\. For each support\-class coordinate\(r,c\)\(r,c\), the chain rule gives

dd​s​Zi,r,c​\(ϑ⁡\(s\)\)=D​Zi,r,c​\(ϑ⁡\(s\)\)​\[Δ​θi\]\.\\frac\{d\}\{ds\}Z\_\{i,r,c\}\(\\vartheta\(s\)\)=DZ\_\{i,r,c\}\(\\vartheta\(s\)\)\[\\Delta\\theta\_\{i\}\]\.Applying the fundamental theorem of calculus to every coordinate and collecting them gives

Zi​\(θi\+Δ​θi\)−Zi​\(θi\)\\displaystyle Z\_\{i\}\(\\theta\_\{i\}\+\\Delta\\theta\_\{i\}\)\-Z\_\{i\}\(\\theta\_\{i\}\)=∫01D​Zi​\(ϑ⁡\(s\)\)​Δ​θi​𝑑s\\displaystyle=\\int\_\{0\}^\{1\}DZ\_\{i\}\(\\vartheta\(s\)\)\\Delta\\theta\_\{i\}\\,ds=∫01Jiz​Δ​θi​𝑑s\\displaystyle=\\int\_\{0\}^\{1\}J\_\{i\}^\{z\}\\Delta\\theta\_\{i\}\\,ds\+∫01\[DZi\(ϑ\(s\)\)−DZi\(θi\)\]Δθids\\displaystyle\+\\int\_\{0\}^\{1\}\\left\[DZ\_\{i\}\(\\vartheta\(s\)\)\-DZ\_\{i\}\(\\theta\_\{i\}\)\\right\]\\Delta\\theta\_\{i\}\\,ds=Jiz​Δ​θi\+riz\.\\displaystyle=J\_\{i\}^\{z\}\\Delta\\theta\_\{i\}\+r\_\{i\}^\{z\}\.
The assumed Lipschitz condition and‖ϑ⁡\(s\)−θi‖=s​‖Δ​θi‖\\\|\\vartheta\(s\)\-\\theta\_\{i\}\\\|=s\\\|\\Delta\\theta\_\{i\}\\\|give, for everys∈\[0,1\]s\\in\[0,1\],

‖\[D​Zi​\(ϑ⁡\(s\)\)−D​Zi​\(θi\)\]​Δ​θi‖≤LJ​s​‖Δ​θi‖2\.\\left\\\|\\left\[DZ\_\{i\}\(\\vartheta\(s\)\)\-DZ\_\{i\}\(\\theta\_\{i\}\)\\right\]\\Delta\\theta\_\{i\}\\right\\\|\\leq L\_\{J\}s\\\|\\Delta\\theta\_\{i\}\\\|^\{2\}\.The triangle inequality for Bochner integrals gives

‖riz‖\\displaystyle\\\|r\_\{i\}^\{z\}\\\|=‖∫01\[D​Zi​\(ϑ⁡\(s\)\)−D​Zi​\(θi\)\]​Δ​θi​ds‖\\displaystyle=\\left\\\|\\int\_\{0\}^\{1\}\\left\[DZ\_\{i\}\(\\vartheta\(s\)\)\-DZ\_\{i\}\(\\theta\_\{i\}\)\\right\]\\Delta\\theta\_\{i\}\\,ds\\right\\\|≤∫01‖\[D​Zi​\(ϑ⁡\(s\)\)−D​Zi​\(θi\)\]​Δ​θi‖​𝑑s\\displaystyle\\leq\\int\_\{0\}^\{1\}\\left\\\|\\left\[DZ\_\{i\}\(\\vartheta\(s\)\)\-DZ\_\{i\}\(\\theta\_\{i\}\)\\right\]\\Delta\\theta\_\{i\}\\right\\\|\\,ds≤∫01LJ​s​‖Δ​θi‖2​𝑑s\\displaystyle\\leq\\int\_\{0\}^\{1\}L\_\{J\}s\\\|\\Delta\\theta\_\{i\}\\\|^\{2\}\\,ds=LJ​‖Δ​θi‖2​\[s22\]01\\displaystyle=L\_\{J\}\\\|\\Delta\\theta\_\{i\}\\\|^\{2\}\\left\[\\frac\{s^\{2\}\}\{2\}\\right\]\_\{0\}^\{1\}=LJ2​‖Δ​θi‖2\.\\displaystyle=\\frac\{L\_\{J\}\}\{2\}\\\|\\Delta\\theta\_\{i\}\\\|^\{2\}\.For a KD\-only parameter step,Δ​θi=−η​α​\(Jiz\)∗​τ​\(Pi−PjT\)\\Delta\\theta\_\{i\}=\-\\eta\\alpha\(J\_\{i\}^\{z\}\)^\{\*\}\\tau\(P\_\{i\}\-P\_\{j\}^\{T\}\)\. Multiplication byJizJ\_\{i\}^\{z\}gives

Jiz​Δ​θi\\displaystyle J\_\{i\}^\{z\}\\Delta\\theta\_\{i\}=Jiz​\[−η​α​\(Jiz\)∗​τ​\(Pi−PjT\)\]\\displaystyle=J\_\{i\}^\{z\}\\left\[\-\\eta\\alpha\(J\_\{i\}^\{z\}\)^\{\*\}\\tau\(P\_\{i\}\-P\_\{j\}^\{T\}\)\\right\]=−η​α​τ​Jiz​\(Jiz\)∗​\(Pi−PjT\)\\displaystyle=\-\\eta\\alpha\\tau J\_\{i\}^\{z\}\(J\_\{i\}^\{z\}\)^\{\*\}\(P\_\{i\}\-P\_\{j\}^\{T\}\)=−η​α​τ​Θi​\(Pi−PjT\)\.\\displaystyle=\-\\eta\\alpha\\tau\\Theta\_\{i\}\(P\_\{i\}\-P\_\{j\}^\{T\}\)\.whereΘi=Jiz​\(Jiz\)∗\\Theta\_\{i\}=J\_\{i\}^\{z\}\(J\_\{i\}^\{z\}\)^\{\*\}\. Combining this identity with the integral remainder proves Equation[G3](https://arxiv.org/html/2609.01952#A7.E3)\. ∎

###### Lemma G2\(Softmax spectrum\)\.

ForP=softmax⁡\(Z/τ\)P=\\operatorname\{softmax\}\(Z/\\tau\), letM⁡\(P\)=diag⁡\(P\)−P​P⊤M\(P\)=\\operatorname\{diag\}\(P\)\-PP^\{\\top\}\. On the centered class subspace,

pmin​I⪯M⁡\(P\)⪯12​I,pminτ​I⪯DZ​P⪯12​τ​I\.p\_\{\\min\}I\\preceq M\(P\)\\preceq\\frac\{1\}\{2\}I,\\qquad\\frac\{p\_\{\\min\}\}\{\\tau\}I\\preceq D\_\{Z\}P\\preceq\\frac\{1\}\{2\\tau\}I\.\(G4\)

###### Proof\.

Let𝟏∈ℝC\\mathbf\{1\}\\in\\mathbb\{R\}^\{C\}denote the all\-ones vector and letu∈ℝCu\\in\\mathbb\{R\}^\{C\}\. SincePc\>0P\_\{c\}\>0and∑c=1CPc=1\\sum\_\{c=1\}^\{C\}P\_\{c\}=1, expanding the matrix product gives

u⊤​M​\(P\)​u\\displaystyle u^\{\\top\}M\(P\)u=u⊤​diag⁡\(P\)​u−u⊤​P​P⊤​u\\displaystyle=u^\{\\top\}\\operatorname\{diag\}\(P\)u\-u^\{\\top\}PP^\{\\top\}u=∑c=1CPc​uc2−\(∑c=1CPc​uc\)2\\displaystyle=\\sum\_\{c=1\}^\{C\}P\_\{c\}u\_\{c\}^\{2\}\-\\left\(\\sum\_\{c=1\}^\{C\}P\_\{c\}u\_\{c\}\\right\)^\{2\}=∑c=1CPc​uc2−2​\(∑a=1CPa​ua\)​\(∑c=1CPc​uc\)\\displaystyle=\\sum\_\{c=1\}^\{C\}P\_\{c\}u\_\{c\}^\{2\}\-2\\left\(\\sum\_\{a=1\}^\{C\}P\_\{a\}u\_\{a\}\\right\)\\left\(\\sum\_\{c=1\}^\{C\}P\_\{c\}u\_\{c\}\\right\)\+\(∑a=1CPaua\)2∑c=1CPc\\displaystyle\+\\left\(\\sum\_\{a=1\}^\{C\}P\_\{a\}u\_\{a\}\\right\)^\{2\}\\sum\_\{c=1\}^\{C\}P\_\{c\}=∑c=1CPc​\(uc−∑a=1CPa​ua\)2≥0\.\\displaystyle=\\sum\_\{c=1\}^\{C\}P\_\{c\}\\left\(u\_\{c\}\-\\sum\_\{a=1\}^\{C\}P\_\{a\}u\_\{a\}\\right\)^\{2\}\\geq 0\.The second\-to\-last line displays the cross term explicitly\. WithmP=∑aPa​uam\_\{P\}=\\sum\_\{a\}P\_\{a\}u\_\{a\}, it is−2​mP2\-2m\_\{P\}^\{2\}, while the final constant term ismP2m\_\{P\}^\{2\}\. Equality holds exactly whenuc=mPu\_\{c\}=m\_\{P\}for everycc, which is equivalent tou∈span⁡\{𝟏\}u\\in\\operatorname\{span\}\\\{\\mathbf\{1\}\\\}\. Thusker⁡M⁡\(P\)=span⁡\{𝟏\}\\ker M\(P\)=\\operatorname\{span\}\\\{\\mathbf\{1\}\\\}\.

For the lower bound, writemP=∑aPa​uam\_\{P\}=\\sum\_\{a\}P\_\{a\}u\_\{a\}and expand around an arbitrary scalara∈ℝa\\in\\mathbb\{R\}:

∑cPc​\(uc−a\)2\\displaystyle\\sum\_\{c\}P\_\{c\}\(u\_\{c\}\-a\)^\{2\}=∑cPc​\(\(uc−mP\)\+\(mP−a\)\)2\\displaystyle=\\sum\_\{c\}P\_\{c\}\\big\(\(u\_\{c\}\-m\_\{P\}\)\+\(m\_\{P\}\-a\)\\big\)^\{2\}=∑cPc​\(uc−mP\)2\\displaystyle=\\sum\_\{c\}P\_\{c\}\(u\_\{c\}\-m\_\{P\}\)^\{2\}\+2\(mP−a\)∑cPc\(uc−mP\)\\displaystyle\+2\(m\_\{P\}\-a\)\\sum\_\{c\}P\_\{c\}\(u\_\{c\}\-m\_\{P\}\)\+\(mP−a\)2∑cPc\\displaystyle\+\(m\_\{P\}\-a\)^\{2\}\\sum\_\{c\}P\_\{c\}=∑cPc​\(uc−mP\)2\+\(a−mP\)2\.\\displaystyle=\\sum\_\{c\}P\_\{c\}\(u\_\{c\}\-m\_\{P\}\)^\{2\}\+\(a\-m\_\{P\}\)^\{2\}\.The cross term is zero because∑cPc​\(uc−mP\)=mP−mP​∑cPc=0\\sum\_\{c\}P\_\{c\}\(u\_\{c\}\-m\_\{P\}\)=m\_\{P\}\-m\_\{P\}\\sum\_\{c\}P\_\{c\}=0\. Hence

min⁡∑ca∈ℝ⁡Pc​\(uc−a\)2=∑cPc​\(uc−mP\)2=u⊤​M​\(P\)​u\.\\min\_\{a\\in\\mathbb\{R\}\}\\sum\_\{c\}P\_\{c\}\(u\_\{c\}\-a\)^\{2\}=\\sum\_\{c\}P\_\{c\}\(u\_\{c\}\-m\_\{P\}\)^\{2\}=u^\{\\top\}M\(P\)u\.Assumption A1 gives\|Zc−Za\|≤2​Gz\|Z\_\{c\}\-Z\_\{a\}\|\\leq 2G\_\{z\}for alla,ca,c\. Consequently,

Pc=eZc/τ∑aeZa/τ≥e−Gz/τC​eGz/τ=e−2Gz/τC=:pmin\.P\_\{c\}=\\frac\{e^\{Z\_\{c\}/\\tau\}\}\{\\sum\_\{a\}e^\{Z\_\{a\}/\\tau\}\}\\geq\\frac\{e^\{\-G\_\{z\}/\\tau\}\}\{Ce^\{G\_\{z\}/\\tau\}\}=\\frac\{e^\{\-2G\_\{z\}/\\tau\}\}\{C\}=:p\_\{\\min\}\.Using the weighted minimizermPm\_\{P\}in the lower probability bound gives

u⊤​M​\(P\)​u\\displaystyle u^\{\\top\}M\(P\)u=∑cPc​\(uc−mP\)2\\displaystyle=\\sum\_\{c\}P\_\{c\}\(u\_\{c\}\-m\_\{P\}\)^\{2\}≥pmin​∑c\(uc−mP\)2\\displaystyle\\geq p\_\{\\min\}\\sum\_\{c\}\(u\_\{c\}\-m\_\{P\}\)^\{2\}≥pmin​min⁡∑ca∈ℝ⁡\(uc−a\)2\.\\displaystyle\\geq p\_\{\\min\}\\min\_\{a\\in\\mathbb\{R\}\}\\sum\_\{c\}\(u\_\{c\}\-a\)^\{2\}\.Ifu⟂𝟏u\\perp\\mathbf\{1\}, then∑cuc=0\\sum\_\{c\}u\_\{c\}=0\. The unweighted square has the explicit expansion

∑c\(uc−a\)2\\displaystyle\\sum\_\{c\}\(u\_\{c\}\-a\)^\{2\}=∑cuc2−2​a​∑cuc\+C​a2\\displaystyle=\\sum\_\{c\}u\_\{c\}^\{2\}\-2a\\sum\_\{c\}u\_\{c\}\+Ca^\{2\}=‖u‖22\+C​a2≥‖u‖22\.\\displaystyle=\\\|u\\\|\_\{2\}^\{2\}\+Ca^\{2\}\\geq\\\|u\\\|\_\{2\}^\{2\}\.The minimum is attained ata=0a=0, so

min⁡∑ca∈ℝ⁡\(uc−a\)2=∑cuc2=‖u‖22\.\\min\_\{a\\in\\mathbb\{R\}\}\\sum\_\{c\}\(u\_\{c\}\-a\)^\{2\}=\\sum\_\{c\}u\_\{c\}^\{2\}=\\\|u\\\|\_\{2\}^\{2\}\.Thusu⊤​M​\(P\)​u≥pmin​‖u‖22u^\{\\top\}M\(P\)u\\geq p\_\{\\min\}\\\|u\\\|\_\{2\}^\{2\}on the centered class subspace\.

For the upper eigenvalue, the diagonal entry in rowccisMc​c=Pc​\(1−Pc\)M\_\{cc\}=P\_\{c\}\(1\-P\_\{c\}\)and the sum of absolute off\-diagonal entries in that row is∑a≠cPc​Pa=Pc​\(1−Pc\)\\sum\_\{a\\neq c\}P\_\{c\}P\_\{a\}=P\_\{c\}\(1\-P\_\{c\}\)\. Gershgorin’s theorem states that every eigenvalue lies in a disk centered atPc​\(1−Pc\)P\_\{c\}\(1\-P\_\{c\}\)with radiusPc​\(1−Pc\)P\_\{c\}\(1\-P\_\{c\}\)\. The right endpoint of that disk is2​Pc​\(1−Pc\)2P\_\{c\}\(1\-P\_\{c\}\)\. Since0≤Pc≤10\\leq P\_\{c\}\\leq 1impliesPc​\(1−Pc\)≤1/4P\_\{c\}\(1\-P\_\{c\}\)\\leq 1/4, every eigenvalue is at most1/21/2\.

Finally, differentiatingPc=eZc/τ/∑aeZa/τP\_\{c\}=e^\{Z\_\{c\}/\\tau\}/\\sum\_\{a\}e^\{Z\_\{a\}/\\tau\}componentwise gives

∂Pc∂Zb\\displaystyle\\frac\{\\partial P\_\{c\}\}\{\\partial Z\_\{b\}\}=1τ​𝟏\{c=b\}eZc/τ∑aeZa/τ−eZc/τeZb/τ\(∑aeZa/τ\)2\\displaystyle=\\frac\{1\}\{\\tau\}\\frac\{\\mathbf\{1\}\\\{c=b\\\}e^\{Z\_\{c\}/\\tau\}\\sum\_\{a\}e^\{Z\_\{a\}/\\tau\}\-e^\{Z\_\{c\}/\\tau\}e^\{Z\_\{b\}/\\tau\}\}\{\(\\sum\_\{a\}e^\{Z\_\{a\}/\\tau\}\)^\{2\}\}=1τPc\(𝟏\{c=b\}−Pb\)\.\\displaystyle=\\frac\{1\}\{\\tau\}P\_\{c\}\(\\mathbf\{1\}\\\{c=b\\\}\-P\_\{b\}\)\.soDZ​P=τ−1​\(diag⁡\(P\)−P​P⊤\)=M⁡\(P\)/τD\_\{Z\}P=\\tau^\{\-1\}\(\\operatorname\{diag\}\(P\)\-PP^\{\\top\}\)=M\(P\)/\\tau\. ∎

###### Lemma G3\(Consensus\-measure equivalence\)\.

For centered logitsZ,Z′Z,Z^\{\\prime\}satisfying A1 and their predictionsP,P′P,P^\{\\prime\},

pminτ​‖Z−Z′‖≤‖P−P′‖≤12​τ​‖Z−Z′‖,\\frac\{p\_\{\\min\}\}\{\\tau\}\\\|Z\-Z^\{\\prime\}\\\|\\leq\\\|P\-P^\{\\prime\}\\\|\\leq\\frac\{1\}\{2\\tau\}\\\|Z\-Z^\{\\prime\}\\\|,\(G5\)and

pmin2∥Z−Z′∥2≤τ2DKL\(P′∥P\)≤14∥Z−Z′∥2\.\\frac\{p\_\{\\min\}\}\{2\}\\\|Z\-Z^\{\\prime\}\\\|^\{2\}\\leq\\tau^\{2\}D\_\{\\rm KL\}\(P^\{\\prime\}\\\|P\)\\leq\\frac\{1\}\{4\}\\\|Z\-Z^\{\\prime\}\\\|^\{2\}\.\(G6\)Hence there are constantscf​z,Cf​z\>0c\_\{fz\},C\_\{fz\}\>0such thatcf​z​Ψz≤Ψf≤Cf​z​Ψzc\_\{fz\}\\Psi^\{z\}\\leq\\Psi^\{f\}\\leq C\_\{fz\}\\Psi^\{z\}\.

###### Proof\.

Leth=Z−Z′h=Z\-Z^\{\\prime\}\. Since both logits are centered,h∈𝟏⟂h\\in\\mathbf\{1\}^\{\\perp\}, and the segmentZs=Z′\+s​hZ\_\{s\}=Z^\{\\prime\}\+shremains in the centered subspace for everys∈\[0,1\]s\\in\[0,1\]\. For each classcc, apply the fundamental theorem of calculus tos↦Pc​\(Zs\)s\\mapsto P\_\{c\}\(Z\_\{s\}\):

Pc−Pc′=∫01∑b=1C∂Pc​\(Zs\)∂Zb​hb​𝑑s\.P\_\{c\}\-P^\{\\prime\}\_\{c\}=\\int\_\{0\}^\{1\}\\sum\_\{b=1\}^\{C\}\\frac\{\\partial P\_\{c\}\(Z\_\{s\}\)\}\{\\partial Z\_\{b\}\}h\_\{b\}\\,ds\.Collecting the class coordinates gives

P−P′=\(∫01DZ​P​\(Zs\)​𝑑s\)​h\.P\-P^\{\\prime\}=\\left\(\\int\_\{0\}^\{1\}D\_\{Z\}P\(Z\_\{s\}\)\\,ds\\right\)h\.SetAh=∫01DZ​P​\(Zs\)​𝑑sA\_\{h\}=\\int\_\{0\}^\{1\}D\_\{Z\}P\(Z\_\{s\}\)\\,ds\. The segment is covered by A1, so Lemma[G2](https://arxiv.org/html/2609.01952#Thmlemma2a)applies at everyss\. Integrating the quadratic\-form bounds gives

pminτ​‖h‖2≤h⊤​Ah​h≤12​τ​‖h‖2\.\\frac\{p\_\{\\min\}\}\{\\tau\}\\\|h\\\|^\{2\}\\leq h^\{\\top\}A\_\{h\}h\\leq\\frac\{1\}\{2\\tau\}\\\|h\\\|^\{2\}\.‖P−P′‖​‖h‖\\displaystyle\\\|P\-P^\{\\prime\}\\\|\\\|h\\\|≥h⊤​\(P−P′\)\\displaystyle\\geq h^\{\\top\}\(P\-P^\{\\prime\}\)=h⊤​\(∫01DZ​P​\(Zs\)​𝑑s\)​h\\displaystyle=h^\{\\top\}\\left\(\\int\_\{0\}^\{1\}D\_\{Z\}P\(Z\_\{s\}\)\\,ds\\right\)h≥pminτ​‖h‖2,\\displaystyle\\geq\\frac\{p\_\{\\min\}\}\{\\tau\}\\\|h\\\|^\{2\},where the first line is Cauchy–Schwarz\. Ifh≠0h\\neq 0, division by‖h‖\\\|h\\\|gives

‖P−P′‖≥pminτ​‖h‖\.\\\|P\-P^\{\\prime\}\\\|\\geq\\frac\{p\_\{\\min\}\}\{\\tau\}\\\|h\\\|\.Forh=0h=0, the same inequality holds with equality\. For the upper bound,‖Ah‖op≤1/\(2​τ\)\\\|A\_\{h\}\\\|\_\{\\rm op\}\\leq 1/\(2\\tau\), and hence

‖P−P′‖\\displaystyle\\\|P\-P^\{\\prime\}\\\|=‖Ah​h‖\\displaystyle=\\\|A\_\{h\}h\\\|≤‖Ah‖op​‖h‖\\displaystyle\\leq\\\|A\_\{h\}\\\|\_\{\\rm op\}\\\|h\\\|≤12​τ​‖h‖\.\\displaystyle\\leq\\frac\{1\}\{2\\tau\}\\\|h\\\|\.This proves Equation[G5](https://arxiv.org/html/2609.01952#A7.E5)\.

For the KL statement, define the log\-partition potential

ϕ⁡\(Z\)=τ2​log⁡\(∑c=1CeZc/τ\)\.\\phi\(Z\)=\\tau^\{2\}\\log\\left\(\\sum\_\{c=1\}^\{C\}e^\{Z\_\{c\}/\\tau\}\\right\)\.Its component derivatives are

∂ϕ∂Zc=τeZc/τ∑aeZa/τ=τPc,∂2ϕ∂Zc​∂Zb=Pc\(𝟏\{c=b\}−Pb\)\.\\frac\{\\partial\\phi\}\{\\partial Z\_\{c\}\}=\\tau\\frac\{e^\{Z\_\{c\}/\\tau\}\}\{\\sum\_\{a\}e^\{Z\_\{a\}/\\tau\}\}=\\tau P\_\{c\},\\qquad\\frac\{\\partial^\{2\}\\phi\}\{\\partial Z\_\{c\}\\partial Z\_\{b\}\}=P\_\{c\}\(\\mathbf\{1\}\\\{c=b\\\}\-P\_\{b\}\)\.Thus∇ϕ​\(Z\)=τ​P\\nabla\\phi\(Z\)=\\tau Pand∇2ϕ​\(Z\)=M​\(P\)\\nabla^\{2\}\\phi\(Z\)=M\(P\)\. UsinglogPc=Zc/τ−log∑aeZa/τ\\log P\_\{c\}=Z\_\{c\}/\\tau\-\\log\\sum\_\{a\}e^\{Z\_\{a\}/\\tau\}gives

τ2DKL\(P′∥P\)\\displaystyle\\tau^\{2\}D\_\{\\rm KL\}\(P^\{\\prime\}\\\|P\)=τ2​∑cPc′​log⁡Pc′Pc\\displaystyle=\\tau^\{2\}\\sum\_\{c\}P^\{\\prime\}\_\{c\}\\log\\frac\{P^\{\\prime\}\_\{c\}\}\{P\_\{c\}\}=τ2​∑cPc′​log⁡Pc′−τ​∑cPc′​Zc\+ϕ⁡\(Z\)\.\\displaystyle=\\tau^\{2\}\\sum\_\{c\}P^\{\\prime\}\_\{c\}\\log P^\{\\prime\}\_\{c\}\-\\tau\\sum\_\{c\}P^\{\\prime\}\_\{c\}Z\_\{c\}\+\\phi\(Z\)\.AtZ′Z^\{\\prime\}, the same identity givesτ2​∑cPc′​log⁡Pc′=τ​∑cPc′​Zc′−ϕ⁡\(Z′\)\\tau^\{2\}\\sum\_\{c\}P^\{\\prime\}\_\{c\}\\log P^\{\\prime\}\_\{c\}=\\tau\\sum\_\{c\}P^\{\\prime\}\_\{c\}Z^\{\\prime\}\_\{c\}\-\\phi\(Z^\{\\prime\}\)\. Hence

τ2DKL\(P′∥P\)\\displaystyle\\tau^\{2\}D\_\{\\rm KL\}\(P^\{\\prime\}\\\|P\)=ϕ⁡\(Z\)−ϕ⁡\(Z′\)−⟨∇ϕ​\(Z′\),Z−Z′⟩\.\\displaystyle=\\phi\(Z\)\-\\phi\(Z^\{\\prime\}\)\-\\langle\\nabla\\phi\(Z^\{\\prime\}\),Z\-Z^\{\\prime\}\\rangle\.
Setφ⁡\(s\)=ϕ⁡\(Z′\+s​h\)\\varphi\(s\)=\\phi\(Z^\{\\prime\}\+sh\)\. The chain rule gives

φ′​\(s\)=⟨∇ϕ​\(Z′\+s​h\),h⟩,φ′′​\(s\)=h⊤​∇2ϕ​\(Z′\+s​h\)​h\.\\varphi^\{\\prime\}\(s\)=\\langle\\nabla\\phi\(Z^\{\\prime\}\+sh\),h\\rangle,\\qquad\\varphi^\{\\prime\\prime\}\(s\)=h^\{\\top\}\\nabla^\{2\}\\phi\(Z^\{\\prime\}\+sh\)h\.Two applications of the fundamental theorem of calculus give

ϕ⁡\(Z\)−ϕ⁡\(Z′\)−\\displaystyle\\phi\(Z\)\-\\phi\(Z^\{\\prime\}\)\-⟨∇ϕ​\(Z′\),h⟩=φ⁡\(1\)−φ⁡\(0\)−φ′​\(0\)\\displaystyle\\langle\\nabla\\phi\(Z^\{\\prime\}\),h\\rangle=\\varphi\(1\)\-\\varphi\(0\)\-\\varphi^\{\\prime\}\(0\)=∫01\(φ′​\(s\)−φ′​\(0\)\)​𝑑s\\displaystyle=\\int\_\{0\}^\{1\}\\big\(\\varphi^\{\\prime\}\(s\)\-\\varphi^\{\\prime\}\(0\)\\big\)\\,ds=∫01∫0sφ′′​\(r\)​𝑑r​𝑑s\\displaystyle=\\int\_\{0\}^\{1\}\\int\_\{0\}^\{s\}\\varphi^\{\\prime\\prime\}\(r\)\\,dr\\,ds=∫01\(1−r\)​h⊤​∇2ϕ​\(Z′\+r​h\)​h​dr\.\\displaystyle=\\int\_\{0\}^\{1\}\(1\-r\)h^\{\\top\}\\nabla^\{2\}\\phi\(Z^\{\\prime\}\+rh\)h\\,dr\.The final line changes the order of integration over0≤r≤s≤10\\leq r\\leq s\\leq 1\. Since∫01\(1−r\)​𝑑r=1/2\\int\_\{0\}^\{1\}\(1\-r\)\\,dr=1/2, the bounds from Lemma[G2](https://arxiv.org/html/2609.01952#Thmlemma2a)give

τ2DKL\(P′∥P\)\\displaystyle\\tau^\{2\}D\_\{\\rm KL\}\(P^\{\\prime\}\\\|P\)≥∫01\(1−r\)​pmin​‖h‖2​dr=pmin2​‖h‖2,\\displaystyle\\geq\\int\_\{0\}^\{1\}\(1\-r\)p\_\{\\min\}\\\|h\\\|^\{2\}\\,dr=\\frac\{p\_\{\\min\}\}\{2\}\\\|h\\\|^\{2\},τ2DKL\(P′∥P\)\\displaystyle\\tau^\{2\}D\_\{\\rm KL\}\(P^\{\\prime\}\\\|P\)≤∫01\(1−r\)​12​‖h‖2​dr=14​‖h‖2\.\\displaystyle\\leq\\int\_\{0\}^\{1\}\(1\-r\)\\frac\{1\}\{2\}\\\|h\\\|^\{2\}\\,dr=\\frac\{1\}\{4\}\\\|h\\\|^\{2\}\.Finally, for vectorsx1,…,xNx\_\{1\},\\ldots,x\_\{N\}andx¯=N−1​∑ixi\\bar\{x\}=N^\{\-1\}\\sum\_\{i\}x\_\{i\}, expand

∑i,j‖xi−xj‖2\\displaystyle\\sum\_\{i,j\}\\\|x\_\{i\}\-x\_\{j\}\\\|^\{2\}=∑i,j\(‖xi−x¯‖2\+‖xj−x¯‖2CLOSE\\displaystyle=\\sum\_\{i,j\}\\big\(\\\|x\_\{i\}\-\\bar\{x\}\\\|^\{2\}\+\\\|x\_\{j\}\-\\bar\{x\}\\\|^\{2\}OPEN−2​⟨xi−x¯,xj−x¯⟩\)\\displaystyle\-2\\langle x\_\{i\}\-\\bar\{x\},x\_\{j\}\-\\bar\{x\}\\rangle\\big\)=N​∑i‖xi−x¯‖2\+N​∑j‖xj−x¯‖2\\displaystyle=N\\sum\_\{i\}\\\|x\_\{i\}\-\\bar\{x\}\\\|^\{2\}\+N\\sum\_\{j\}\\\|x\_\{j\}\-\\bar\{x\}\\\|^\{2\}−2​⟨∑i\(xi−x¯\),∑j\(xj−x¯\)⟩\\displaystyle\-2\\left\\langle\\sum\_\{i\}\(x\_\{i\}\-\\bar\{x\}\),\\sum\_\{j\}\(x\_\{j\}\-\\bar\{x\}\)\\right\\rangle=2​N​∑i‖xi−x¯‖2\.\\displaystyle=2N\\sum\_\{i\}\\\|x\_\{i\}\-\\bar\{x\}\\\|^\{2\}\.The cross term is zero because∑i\(xi−x¯\)=∑ixi−N​x¯=0\\sum\_\{i\}\(x\_\{i\}\-\\bar\{x\}\)=\\sum\_\{i\}x\_\{i\}\-N\\bar\{x\}=0\. Dividing by2​N22N^\{2\}gives the variance identity and transfers the pairwise probability and KL bounds toΨf\\Psi^\{f\}andΨz\\Psi^\{z\}\. Under the paper’s normalization, the prediction comparison permitscf​z=pmin2/τ2c\_\{fz\}=p\_\{\\min\}^\{2\}/\\tau^\{2\}andCf​z=1/\(4​τ2\)C\_\{fz\}=1/\(4\\tau^\{2\}\)\. ∎

###### Proposition G1\(Prediction\-space graph\-angle diagnostic\)\.

For a connected undirected graph,

1\|E→\|​∑\(i,j\)∈E→‖\(I−Πip\)​\(pi−pj\)‖2≤N​λmax​\(ℒ\)\|E\|​Ψf\.\\frac\{1\}\{\|\\vec\{E\}\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|\(I\-\\Pi\_\{i\}^\{p\}\)\(p\_\{i\}\-p\_\{j\}\)\\\|^\{2\}\\leq\\frac\{N\\lambda\_\{\\max\}\(\\mathcal\{L\}\)\}\{\|E\|\}\\Psi^\{f\}\.\(G7\)If every directed pull makes angle at mostϕ\\phiwithTipT\_\{i\}^\{p\}, the right side is multiplied bysin2⁡ϕ\\sin^\{2\}\\phi\. A squared normal bias whose directed\-edge average is at mostνB​Bf2\\nu\_\{B\}B\_\{f\}^\{2\}gives the analogous prediction\-space bound, with the factor22from\(a\+b\)2≤2​a2\+2​b2\(a\+b\)^\{2\}\\leq 2a^\{2\}\+2b^\{2\}absorbed into the constants\. This diagnostic does not by itself imply the logit\-space KD clause in A10; that implication requires a separately verified softmax/Jacobian transport condition\.

###### Proof\.

LetE→=\{\(i,j\),\(j,i\):\{i,j\}∈E\}\\vec\{E\}=\\\{\(i,j\),\(j,i\):\\\{i,j\\\}\\in E\\\}, so\|E→\|=2​\|E\|\|\\vec\{E\}\|=2\|E\|\. Letℋ\\mathcal\{H\}denote the prediction Hilbert space and definep¯=N−1​∑ipi\\bar\{p\}=N^\{\-1\}\\sum\_\{i\}p\_\{i\}andqi=pi−p¯q\_\{i\}=p\_\{i\}\-\\bar\{p\}\. For each directed edge, orthogonal projection is nonexpansive because

‖\(I−Πip\)​\(pi−pj\)‖2\\displaystyle\\\|\(I\-\\Pi\_\{i\}^\{p\}\)\(p\_\{i\}\-p\_\{j\}\)\\\|^\{2\}=‖pi−pj‖2−‖Πip​\(pi−pj\)‖2\\displaystyle=\\\|p\_\{i\}\-p\_\{j\}\\\|^\{2\}\-\\\|\\Pi\_\{i\}^\{p\}\(p\_\{i\}\-p\_\{j\}\)\\\|^\{2\}≤‖pi−pj‖2\.\\displaystyle\\leq\\\|p\_\{i\}\-p\_\{j\}\\\|^\{2\}\.Counting both orientations gives

1\|E→\|​∑\(i,j\)∈E→‖pi−pj‖2\\displaystyle\\frac\{1\}\{\|\\vec\{E\}\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|p\_\{i\}\-p\_\{j\}\\\|^\{2\}=12​\|E\|​∑\{i,j\}∈E\(‖pi−pj‖2\+‖pj−pi‖2\)\\displaystyle=\\frac\{1\}\{2\|E\|\}\\sum\_\{\\\{i,j\\\}\\in E\}\\left\(\\\|p\_\{i\}\-p\_\{j\}\\\|^\{2\}\+\\\|p\_\{j\}\-p\_\{i\}\\\|^\{2\}\\right\)=1\|E\|​∑\{i,j\}∈E‖pi−pj‖2\.\\displaystyle=\\frac\{1\}\{\|E\|\}\\sum\_\{\\\{i,j\\\}\\in E\}\\\|p\_\{i\}\-p\_\{j\}\\\|^\{2\}\.
Letℒ=D−A\\mathcal\{L\}=D\-Abe the unweighted graph Laplacian\. For a scalar coordinate vectoru=\(u1,…,uN\)⊤u=\(u\_\{1\},\\ldots,u\_\{N\}\)^\{\\top\}, direct expansion gives

u⊤​ℒ​u\\displaystyle u^\{\\top\}\\mathcal\{L\}u=∑idi​ui2−∑i,jAi​j​ui​uj\\displaystyle=\\sum\_\{i\}d\_\{i\}u\_\{i\}^\{2\}\-\\sum\_\{i,j\}A\_\{ij\}u\_\{i\}u\_\{j\}=∑\{i,j\}∈E\(ui2\+uj2−2​ui​uj\)\\displaystyle=\\sum\_\{\\\{i,j\\\}\\in E\}\(u\_\{i\}^\{2\}\+u\_\{j\}^\{2\}\-2u\_\{i\}u\_\{j\}\)=∑\{i,j\}∈E\(ui−uj\)2\.\\displaystyle=\\sum\_\{\\\{i,j\\\}\\in E\}\(u\_\{i\}\-u\_\{j\}\)^\{2\}\.The edge set in this identity is counted once\. The factor1/21/2would appear only in the equivalent ordered\-adjacency expression12​∑i,jAi​j​\(ui−uj\)2\\frac\{1\}\{2\}\\sum\_\{i,j\}A\_\{ij\}\(u\_\{i\}\-u\_\{j\}\)^\{2\}\.

Choose an orthonormal basis of the finite support representation ofℋ\\mathcal\{H\}and apply the scalar identity coordinatewise\. Parseval’s identity then gives the Hilbert\-valued graph energy

∑\{i,j\}∈E‖pi−pj‖2=⟨q,\(ℒ⊗Iℋ\)​q⟩ℋN\.\\sum\_\{\\\{i,j\\\}\\in E\}\\\|p\_\{i\}\-p\_\{j\}\\\|^\{2\}=\\langle q,\(\\mathcal\{L\}\\otimes I\_\{\\mathcal\{H\}\}\)q\\rangle\_\{\\mathcal\{H\}^\{N\}\}\.The centered vectorq=\(q1,…,qN\)q=\(q\_\{1\},\\ldots,q\_\{N\}\)is orthogonal to the consensus subspace because∑iqi=0\\sum\_\{i\}q\_\{i\}=0\. Expandingqqin an orthonormal eigenbasis ofℒ\\mathcal\{L\}and using that every nonconsensus eigenvalue is at mostλmax​\(ℒ\)\\lambda\_\{\\max\}\(\\mathcal\{L\}\)gives

∑\{i,j\}∈E‖pi−pj‖2\\displaystyle\\sum\_\{\\\{i,j\\\}\\in E\}\\\|p\_\{i\}\-p\_\{j\}\\\|^\{2\}=⟨q,\(ℒ⊗Iℋ\)​q⟩\\displaystyle=\\langle q,\(\\mathcal\{L\}\\otimes I\_\{\\mathcal\{H\}\}\)q\\rangle≤λmax​\(ℒ\)​∑i=1N‖qi‖2\\displaystyle\\leq\\lambda\_\{\\max\}\(\\mathcal\{L\}\)\\sum\_\{i=1\}^\{N\}\\\|q\_\{i\}\\\|^\{2\}=N​λmax​\(ℒ\)​Ψf\.\\displaystyle=N\\lambda\_\{\\max\}\(\\mathcal\{L\}\)\\Psi^\{f\}\.Consequently,

1\|E\|​∑\{i,j\}∈E‖pi−pj‖2≤λmax\|E\|​∑i=1N‖pi−p¯‖2=N​λmax\|E\|​Ψf\.\\frac\{1\}\{\|E\|\}\\sum\_\{\\\{i,j\\\}\\in E\}\\\|p\_\{i\}\-p\_\{j\}\\\|^\{2\}\\leq\\frac\{\\lambda\_\{\\max\}\}\{\|E\|\}\\sum\_\{i=1\}^\{N\}\\\|p\_\{i\}\-\\bar\{p\}\\\|^\{2\}=\\frac\{N\\lambda\_\{\\max\}\}\{\|E\|\}\\Psi^\{f\}\.For the angle statement, definesin⁡∠⁡\(v,T\)=‖\(I−ΠT\)​v‖/‖v‖\\sin\\angle\(v,T\)=\\\|\(I\-\\Pi\_\{T\}\)v\\\|/\\\|v\\\|forv≠0v\\neq 0\. If each directed pull has angle at mostϕ\\phi, then

‖\(I−Πip\)​\(pi−pj\)‖2=sin2⁡∠⁡\(pi−pj,Tip\)​‖pi−pj‖2≤sin2⁡ϕ​‖pi−pj‖2\.\\\|\(I\-\\Pi\_\{i\}^\{p\}\)\(p\_\{i\}\-p\_\{j\}\)\\\|^\{2\}=\\sin^\{2\}\\angle\(p\_\{i\}\-p\_\{j\},T\_\{i\}^\{p\}\)\\\|p\_\{i\}\-p\_\{j\}\\\|^\{2\}\\leq\\sin^\{2\}\\phi\\\|p\_\{i\}\-p\_\{j\}\\\|^\{2\}\.Averaging this inequality inserts the factorsin2⁡ϕ\\sin^\{2\}\\phiinto the graph bound\. For a normal bias, assume the decompositionpi−pj=vi​jtan\+bi​jp\_\{i\}\-p\_\{j\}=v\_\{ij\}^\{\\rm tan\}\+b\_\{ij\}and the directed average condition

1\|E→\|​∑\(i,j\)∈E→‖bi​j‖2≤νB​Bf2\.\\frac\{1\}\{\|\\vec\{E\}\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|b\_\{ij\}\\\|^\{2\}\\leq\\nu\_\{B\}B\_\{f\}^\{2\}\.Fora=\(I−Πip\)​vi​jtana=\(I\-\\Pi\_\{i\}^\{p\}\)v\_\{ij\}^\{\\rm tan\}andb=\(I−Πip\)​bi​jb=\(I\-\\Pi\_\{i\}^\{p\}\)b\_\{ij\}, nonnegativity of‖a−b‖2\\\|a\-b\\\|^\{2\}gives2​⟨a,b⟩≤‖a‖2\+‖b‖22\\langle a,b\\rangle\\leq\\\|a\\\|^\{2\}\+\\\|b\\\|^\{2\}\. Hence

‖\(I−Πip\)​\(vi​jtan\+bi​j\)‖2\\displaystyle\\\|\(I\-\\Pi\_\{i\}^\{p\}\)\(v\_\{ij\}^\{\\rm tan\}\+b\_\{ij\}\)\\\|^\{2\}=‖a‖2\+2​⟨a,b⟩\+‖b‖2\\displaystyle=\\\|a\\\|^\{2\}\+2\\langle a,b\\rangle\+\\\|b\\\|^\{2\}≤2​‖a‖2\+2​‖b‖2\\displaystyle\\leq 2\\\|a\\\|^\{2\}\+2\\\|b\\\|^\{2\}≤2​‖a‖2\+2​‖bi​j‖2\.\\displaystyle\\leq 2\\\|a\\\|^\{2\}\+2\\\|b\_\{ij\}\\\|^\{2\}\.Averaging overE→\\vec\{E\}gives the stated prediction\-space bias term\. No logit\-space transport is used in this argument, which is why the final clause of the proposition remains a separate requirement\. ∎

###### Proposition G2\(An exact affine verification model\)\.

Suppose𝒫i=ai\+Ui\\mathcal\{P\}\_\{i\}=a\_\{i\}\+U\_\{i\}is a closed affine prediction class,Fi​\(p\)=12​‖p−q⋆‖2F\_\{i\}\(p\)=\\tfrac\{1\}\{2\}\\\|p\-q^\{\\star\}\\\|^\{2\}, andΠi\\Pi\_\{i\}projects ontoUiU\_\{i\}\. Then everypi∈𝒫ip\_\{i\}\\in\\mathcal\{P\}\_\{i\}satisfies

∥\(I−Πi\)∇Fi\(pi\)∥2=dist\(q⋆,𝒫i\)2\.\\\|\(I\-\\Pi\_\{i\}\)\\nabla F\_\{i\}\(p\_\{i\}\)\\\|^\{2\}=\\operatorname\{dist\}\(q^\{\\star\},\\mathcal\{P\}\_\{i\}\)^\{2\}\.\(G8\)Thus the averaged task residual is exactlyBf2B\_\{f\}^\{2\}\.

###### Proof\.

Letℋ\\mathcal\{H\}be the prediction Hilbert space\. SinceUiU\_\{i\}is closed, the orthogonal projectorΠi:ℋ→Ui\\Pi\_\{i\}:\\mathcal\{H\}\\to U\_\{i\}exists and satisfiesΠi∗=Πi\\Pi\_\{i\}^\{\*\}=\\Pi\_\{i\},Πi2=Πi\\Pi\_\{i\}^\{2\}=\\Pi\_\{i\}, and⟨Πi​u,\(I−Πi\)​v⟩=0\\langle\\Pi\_\{i\}u,\(I\-\\Pi\_\{i\}\)v\\rangle=0for allu,v∈ℋu,v\\in\\mathcal\{H\}\.

For an arbitrary directionh∈ℋh\\in\\mathcal\{H\}, expand the risk increment:

Fi​\(pi\+ϵ​h\)−Fi​\(pi\)\\displaystyle F\_\{i\}\(p\_\{i\}\+\\epsilon h\)\-F\_\{i\}\(p\_\{i\}\)=12​‖pi−q⋆\+ϵ​h‖2−12​‖pi−q⋆‖2\\displaystyle=\\frac\{1\}\{2\}\\\|p\_\{i\}\-q^\{\\star\}\+\\epsilon h\\\|^\{2\}\-\\frac\{1\}\{2\}\\\|p\_\{i\}\-q^\{\\star\}\\\|^\{2\}=ϵ⁡⟨pi−q⋆,h⟩\+ϵ22​‖h‖2\.\\displaystyle=\\epsilon\\langle p\_\{i\}\-q^\{\\star\},h\\rangle\+\\frac\{\\epsilon^\{2\}\}\{2\}\\\|h\\\|^\{2\}\.Dividing byϵ\\epsilonand takingϵ→0\\epsilon\\to 0givesD​Fi​\(pi\)​\[h\]=⟨pi−q⋆,h⟩DF\_\{i\}\(p\_\{i\}\)\[h\]=\\langle p\_\{i\}\-q^\{\\star\},h\\rangle, so∇Fi​\(pi\)=pi−q⋆\\nabla F\_\{i\}\(p\_\{i\}\)=p\_\{i\}\-q^\{\\star\}\.

Writepi=ai\+uip\_\{i\}=a\_\{i\}\+u\_\{i\}withui∈Uiu\_\{i\}\\in U\_\{i\}\. SinceΠi​ui=ui\\Pi\_\{i\}u\_\{i\}=u\_\{i\},\(I−Πi\)​ui=0\(I\-\\Pi\_\{i\}\)u\_\{i\}=0, and

\(I−Πi\)∇Fi\(pi\)\\displaystyle\(I\-\\Pi\_\{i\}\)\\nabla F\_\{i\}\(p\_\{i\}\)=\(I−Πi\)​\(ai\+ui−q⋆\)\\displaystyle=\(I\-\\Pi\_\{i\}\)\(a\_\{i\}\+u\_\{i\}\-q^\{\\star\}\)=\(I−Πi\)​\(ai−q⋆\)\+\(I−Πi\)​ui\\displaystyle=\(I\-\\Pi\_\{i\}\)\(a\_\{i\}\-q^\{\\star\}\)\+\(I\-\\Pi\_\{i\}\)u\_\{i\}=\(I−Πi\)​\(ai−q⋆\)\.\\displaystyle=\(I\-\\Pi\_\{i\}\)\(a\_\{i\}\-q^\{\\star\}\)\.For anyv∈Uiv\\in U\_\{i\}, decomposeai−q⋆\+va\_\{i\}\-q^\{\\star\}\+vinto itsUiU\_\{i\}andUi⟂U\_\{i\}^\{\\perp\}components\. Expanding the squared norm gives

‖ai−q⋆\+v‖2\\displaystyle\\\|a\_\{i\}\-q^\{\\star\}\+v\\\|^\{2\}=‖Πi​\(ai−q⋆\)\+v‖2\\displaystyle=\\\|\\Pi\_\{i\}\(a\_\{i\}\-q^\{\\star\}\)\+v\\\|^\{2\}\+2​⟨Πi​\(ai−q⋆\)\+v,\(I−Πi\)​\(ai−q⋆\)⟩\\displaystyle\+2\\langle\\Pi\_\{i\}\(a\_\{i\}\-q^\{\\star\}\)\+v,\(I\-\\Pi\_\{i\}\)\(a\_\{i\}\-q^\{\\star\}\)\\rangle\+‖\(I−Πi\)​\(ai−q⋆\)‖2\\displaystyle\+\\\|\(I\-\\Pi\_\{i\}\)\(a\_\{i\}\-q^\{\\star\}\)\\\|^\{2\}=‖Πi​\(ai−q⋆\)\+v‖2\+‖\(I−Πi\)​\(ai−q⋆\)‖2\.\\displaystyle=\\\|\\Pi\_\{i\}\(a\_\{i\}\-q^\{\\star\}\)\+v\\\|^\{2\}\+\\\|\(I\-\\Pi\_\{i\}\)\(a\_\{i\}\-q^\{\\star\}\)\\\|^\{2\}\.The cross term is zero because its first argument lies inUiU\_\{i\}and its second lies inUi⟂U\_\{i\}^\{\\perp\}\. The vectorvi⋆=−Πi​\(ai−q⋆\)v\_\{i\}^\{\\star\}=\-\\Pi\_\{i\}\(a\_\{i\}\-q^\{\\star\}\)belongs toUiU\_\{i\}and makes the first square zero\. Since every squared norm is nonnegative,

dist⁡\(q⋆,ai\+Ui\)2\\displaystyle\\operatorname\{dist\}\(q^\{\\star\},a\_\{i\}\+U\_\{i\}\)^\{2\}=infv∈Ui‖ai−q⋆\+v‖2\\displaystyle=\\inf\_\{v\\in U\_\{i\}\}\\\|a\_\{i\}\-q^\{\\star\}\+v\\\|^\{2\}=‖\(I−Πi\)​\(ai−q⋆\)‖2\.\\displaystyle=\\\|\(I\-\\Pi\_\{i\}\)\(a\_\{i\}\-q^\{\\star\}\)\\\|^\{2\}\.Together with the normal\-gradient identity, this proves Equation[G8](https://arxiv.org/html/2609.01952#A7.E8)\. Averaging over peers gives

1N∑i=1N∥\(I−Πi\)∇Fi\(pi\)∥2=1N∑i=1Ndist\(q⋆,𝒫i\)2=Bf2\.\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\\|\(I\-\\Pi\_\{i\}\)\\nabla F\_\{i\}\(p\_\{i\}\)\\\|^\{2\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\operatorname\{dist\}\(q^\{\\star\},\\mathcal\{P\}\_\{i\}\)^\{2\}=B\_\{f\}^\{2\}\.∎

###### Proposition G3\(Why the relative task term is necessary\)\.

The old inequality‖\(I−Πi\)​gi‖2≤ν⁡\(Bf2\+Ψf\)\\\|\(I\-\\Pi\_\{i\}\)g\_\{i\}\\\|^\{2\}\\leq\\nu\(B\_\{f\}^\{2\}\+\\Psi^\{f\}\)does not follow from A3\. Let𝒫=\{\(x,x2\):x∈ℝ\}\\mathcal\{P\}=\\\{\(x,x^\{2\}\):x\\in\\mathbb\{R\}\\\},q⋆=\(0,0\)q^\{\\star\}=\(0,0\), andF⁡\(p\)=12​‖p−q⋆‖2F\(p\)=\\tfrac\{1\}\{2\}\\\|p\-q^\{\\star\}\\\|^\{2\}\. ThenBf=0B\_\{f\}=0and, for a single peer,Ψf=0\\Psi^\{f\}=0, while the gradient at\(x,x2\)\(x,x^\{2\}\)has a nonzero normal component wheneverx≠0x\\neq 0\.

###### Proof\.

This construction addresses the abstract Euclidean prediction\-space implication that A3 alone would have to provide; the parabola is not asserted to be a literal softmax image\. The set𝒫=\{\(u,v\)∈ℝ2:v−u2=0\}\\mathcal\{P\}=\\\{\(u,v\)\\in\\mathbb\{R\}^\{2\}:v\-u^\{2\}=0\\\}is closed because it is the zero set of the continuous map\(u,v\)↦v−u2\(u,v\)\\mapsto v\-u^\{2\}\. It is parametrized byr⁡\(x\)=\(x,x2\)r\(x\)=\(x,x^\{2\}\)\.

The tangent vector and one normal vector are

r′​\(x\)=\(1,2​x\),n⁡\(x\)=\(2​x,−1\),\\displaystyle r^\{\\prime\}\(x\)=\(1,2x\),\\qquad n\(x\)=\(2x,\-1\),\(G9\)⟨r′​\(x\),n⁡\(x\)⟩=2​x−2​x=0\.\\displaystyle\\langle r^\{\\prime\}\(x\),n\(x\)\\rangle=2x\-2x=0\.\(G10\)
Both vectors are nonzero, so they span the one\-dimensional tangent and normal spaces\. The quadratic directional expansion gives∇F​\(p\)=p−q⋆=p\\nabla F\(p\)=p\-q^\{\\star\}=p\. Henceg⁡\(x\)=\(x,x2\)g\(x\)=\(x,x^\{2\}\), and its orthogonal projection onto the normal line is

\(I−ΠTx\)​g​\(x\)=⟨g⁡\(x\),n⁡\(x\)⟩‖n⁡\(x\)‖2​n​\(x\)\.\(I\-\\Pi\_\{T\_\{x\}\}\)g\(x\)=\\frac\{\\langle g\(x\),n\(x\)\\rangle\}\{\\\|n\(x\)\\\|^\{2\}\}n\(x\)\.The numerator and denominator are

⟨g⁡\(x\),n⁡\(x\)⟩\\displaystyle\\langle g\(x\),n\(x\)\\rangle=x⁡\(2​x\)\+x2​\(−1\)=x2,\\displaystyle=x\(2x\)\+x^\{2\}\(\-1\)=x^\{2\},‖n⁡\(x\)‖2\\displaystyle\\\|n\(x\)\\\|^\{2\}=\(2​x\)2\+\(−1\)2=4​x2\+1\.\\displaystyle=\(2x\)^\{2\}\+\(\-1\)^\{2\}=4x^\{2\}\+1\.Therefore

‖\(I−ΠTx\)​g​\(x\)‖2\\displaystyle\\\|\(I\-\\Pi\_\{T\_\{x\}\}\)g\(x\)\\\|^\{2\}=‖x24​x2\+1​n​\(x\)‖2\\displaystyle=\\left\\\|\\frac\{x^\{2\}\}\{4x^\{2\}\+1\}n\(x\)\\right\\\|^\{2\}=x4\(4​x2\+1\)2​‖n⁡\(x\)‖2\\displaystyle=\\frac\{x^\{4\}\}\{\(4x^\{2\}\+1\)^\{2\}\}\\\|n\(x\)\\\|^\{2\}=x44​x2\+1\>0\(x≠0\)\.\\displaystyle=\\frac\{x^\{4\}\}\{4x^\{2\}\+1\}\>0\\qquad\(x\\neq 0\)\.For one peer, the peer mean equals its prediction, soΨf=‖p1−p¯‖2=0\\Psi^\{f\}=\\\|p\_\{1\}\-\\bar\{p\}\\\|^\{2\}=0\. Alsoq⋆=r⁡\(0\)∈𝒫q^\{\\star\}=r\(0\)\\in\\mathcal\{P\}, and

0≤Bf=infy∈𝒫‖q⋆−y‖≤‖q⋆−r⁡\(0\)‖=0\.0\\leq B\_\{f\}=\\inf\_\{y\\in\\mathcal\{P\}\}\\\|q^\{\\star\}\-y\\\|\\leq\\\|q^\{\\star\}\-r\(0\)\\\|=0\.ThusBf=0B\_\{f\}=0andν⁡\(Bf2\+Ψf\)=0\\nu\(B\_\{f\}^\{2\}\+\\Psi^\{f\}\)=0, whereas the normal\-gradient square is positive for everyx≠0x\\neq 0\. This contradicts the old inequality in the stated prediction\-space geometry\.

The same local geometry can be embedded in the interior of a probability simplex\. LetC=3C=3, letq⋆=\(1/3,1/3,1/3\)q^\{\\star\}=\(1/3,1/3,1/3\), and choose orthonormale1,e2∈𝟏⟂e\_\{1\},e\_\{2\}\\in\\mathbf\{1\}^\{\\perp\}\. For fixedρ\>0\\rho\>0, chooseϵ\>0\\epsilon\>0small enough that

ϵ​max\|x\|≤ρ​‖x​e1\+x2​e2‖∞<13\.\\epsilon\\max\_\{\|x\|\\leq\\rho\}\\\|xe\_\{1\}\+x^\{2\}e\_\{2\}\\\|\_\{\\infty\}<\\frac\{1\}\{3\}\.Then

r~​\(x\):=q⋆\+ϵ⁡\(x​e1\+x2​e2\),\|x\|≤ρ,\\widetilde\{r\}\(x\):=q^\{\\star\}\+\\epsilon\(xe\_\{1\}\+x^\{2\}e\_\{2\}\),\\qquad\|x\|\\leq\\rho,has positive coordinates and coordinate sum one\. It is a softmax predictor with centered logits

Z~​\(x\)=τ⁡\[log⁡r~​\(x\)−𝟏3​𝟏⊤​log⁡r~​\(x\)\]\.\\widetilde\{Z\}\(x\)=\\tau\\left\[\\log\\widetilde\{r\}\(x\)\-\\frac\{\\mathbf\{1\}\}\{3\}\\mathbf\{1\}^\{\\top\}\\log\\widetilde\{r\}\(x\)\\right\]\.The compact interval and the strict positivity above make these logits uniformly bounded\. The tangent isϵ⁡\(e1\+2​x​e2\)\\epsilon\(e\_\{1\}\+2xe\_\{2\}\), while2​x​e1−e22xe\_\{1\}\-e\_\{2\}is normal to that tangent\. ForF~​\(p\)=12​‖p−q⋆‖2\\widetilde\{F\}\(p\)=\\frac\{1\}\{2\}\\\|p\-q^\{\\star\}\\\|^\{2\},

⟨∇F~​\(r~​\(x\)\),2​x​e1−e2⟩\\displaystyle\\left\\langle\\nabla\\widetilde\{F\}\(\\widetilde\{r\}\(x\)\),2xe\_\{1\}\-e\_\{2\}\\right\\rangle=ϵ⁡⟨x​e1\+x2​e2,2​x​e1−e2⟩\\displaystyle=\\epsilon\\langle xe\_\{1\}\+x^\{2\}e\_\{2\},2xe\_\{1\}\-e\_\{2\}\\rangle=ϵ⁡\(2​x2−x2\)=ϵ​x2\.\\displaystyle=\\epsilon\(2x^\{2\}\-x^\{2\}\)=\\epsilon x^\{2\}\.This quantity is nonzero forx≠0x\\neq 0, whiler~​\(0\)=q⋆\\widetilde\{r\}\(0\)=q^\{\\star\}\. Thus the obstruction persists in a bounded\-logit softmax realization, withBf=0B\_\{f\}=0and one\-peer consensus error zero\. ∎

###### Lemma G4\(Kernel range and local persistence\)\.

For any finite matrix or finite\-rank operatorJJ,range⁡\(J​J∗\)=range⁡\(J\)\\operatorname\{range\}\(JJ^\{\*\}\)=\\operatorname\{range\}\(J\)\. Ifλmin\+​\(Θi0\)=λi,0\>0\\lambda\_\{\\min\}^\{\+\}\(\\Theta\_\{i\}^\{0\}\)=\\lambda\_\{i,0\}\>0and

‖Θi​\(θ\)−Θi0‖op≤LK​‖θ−θi0‖,\\\|\\Theta\_\{i\}\(\\theta\)\-\\Theta\_\{i\}^\{0\}\\\|\_\{\\rm op\}\\leq L\_\{K\}\\\|\\theta\-\\theta\_\{i\}^\{0\}\\\|,\(G11\)then every point with‖θ−θi0‖≤R\\\|\\theta\-\\theta\_\{i\}^\{0\}\\\|\\leq RandLK​R<λi,0L\_\{K\}R<\\lambda\_\{i,0\}satisfiesλmin\+​\(Θi​\(θ\)\)≥λi,0−LK​R\>0\\lambda\_\{\\min\}^\{\+\}\(\\Theta\_\{i\}\(\\theta\)\)\\geq\\lambda\_\{i,0\}\-L\_\{K\}R\>0, provided the rank is constant in the region\.

###### Proof\.

First prove the range identity\. For any vectorvvin the output space,

⟨v,J​J∗​v⟩\\displaystyle\\langle v,JJ^\{\*\}v\\rangle=⟨J∗​v,J∗​v⟩\\displaystyle=\\langle J^\{\*\}v,J^\{\*\}v\\rangle=‖J∗​v‖2\.\\displaystyle=\\\|J^\{\*\}v\\\|^\{2\}\.IfJ​J∗​v=0JJ^\{\*\}v=0, the left side is zero, so‖J∗​v‖2=0\\\|J^\{\*\}v\\\|^\{2\}=0andJ∗​v=0J^\{\*\}v=0\. Conversely,J∗​v=0J^\{\*\}v=0directly givesJ​J∗​v=0JJ^\{\*\}v=0\. Henceker⁡\(J​J∗\)=ker⁡\(J∗\)\\ker\(JJ^\{\*\}\)=\\ker\(J^\{\*\}\)\.

Every vector inrange⁡\(J​J∗\)\\operatorname\{range\}\(JJ^\{\*\}\)has the formJ​J∗​v=J⁡\(J∗​v\)JJ^\{\*\}v=J\(J^\{\*\}v\)\. Therefore

range⁡\(J​J∗\)⊆range⁡\(J\)\.\\operatorname\{range\}\(JJ^\{\*\}\)\\subseteq\\operatorname\{range\}\(J\)\.For the reverse inclusion, finite dimensionality or finite rank makes both ranges closed, and the orthogonal\-complement identities give

range⁡\(J​J∗\)\\displaystyle\\operatorname\{range\}\(JJ^\{\*\}\)=ker⁡\(\(J​J∗\)∗\)⟂\\displaystyle=\\ker\(\(JJ^\{\*\}\)^\{\*\}\)^\{\\perp\}=ker⁡\(J​J∗\)⟂\\displaystyle=\\ker\(JJ^\{\*\}\)^\{\\perp\}=ker⁡\(J∗\)⟂\\displaystyle=\\ker\(J^\{\*\}\)^\{\\perp\}=range⁡\(J\)\.\\displaystyle=\\operatorname\{range\}\(J\)\.This also proves the reverse inclusion\. IfΘ=J​J∗=∑ℓ=1rλℓ​uℓ​uℓ⊤\\Theta=JJ^\{\*\}=\\sum\_\{\\ell=1\}^\{r\}\\lambda\_\{\\ell\}u\_\{\\ell\}u\_\{\\ell\}^\{\\top\}withλℓ\>0\\lambda\_\{\\ell\}\>0, then

Θ†​Θ\\displaystyle\\Theta^\{\\dagger\}\\Theta=\(∑ℓ=1rλℓ−1​uℓ​uℓ⊤\)​\(∑k=1rλk​uk​uk⊤\)\\displaystyle=\\left\(\\sum\_\{\\ell=1\}^\{r\}\\lambda\_\{\\ell\}^\{\-1\}u\_\{\\ell\}u\_\{\\ell\}^\{\\top\}\\right\)\\left\(\\sum\_\{k=1\}^\{r\}\\lambda\_\{k\}u\_\{k\}u\_\{k\}^\{\\top\}\\right\)=∑ℓ=1ruℓ​uℓ⊤,\\displaystyle=\\sum\_\{\\ell=1\}^\{r\}u\_\{\\ell\}u\_\{\\ell\}^\{\\top\},becauseuℓ⊤uk=𝟏\{ℓ=k\}u\_\{\\ell\}^\{\\top\}u\_\{k\}=\\mathbf\{1\}\\\{\\ell=k\\\}\. ThusΘ†​Θ\\Theta^\{\\dagger\}\\Thetais the orthogonal projector ontorange⁡\(J\)\\operatorname\{range\}\(J\)\.

For persistence, let the output dimension bemmand let the constant rank ber≥1r\\geq 1\. Order eigenvalues increasingly:λ1​\(A\)≤⋯≤λm​\(A\)\\lambda\_\{1\}\(A\)\\leq\\cdots\\leq\\lambda\_\{m\}\(A\)\. A positive semidefinite rank\-rrmatrix hasm−rm\-rzero eigenvalues, so its smallest positive eigenvalue has indexk=m−r\+1k=m\-r\+1\. Weyl’s perturbation inequality states that for symmetricAAandBB,

\|λℓ\(A\)−λℓ\(B\)\|≤∥A−B∥op,ℓ=1,…,m\.\|\\lambda\_\{\\ell\}\(A\)\-\\lambda\_\{\\ell\}\(B\)\|\\leq\\\|A\-B\\\|\_\{\\rm op\},\\qquad\\ell=1,\\ldots,m\.Apply it withA=Θi​\(θ\)A=\\Theta\_\{i\}\(\\theta\),B=Θi0B=\\Theta\_\{i\}^\{0\}, andℓ=k\\ell=k\. Then

λk​\(Θi​\(θ\)\)\\displaystyle\\lambda\_\{k\}\(\\Theta\_\{i\}\(\\theta\)\)≥λk​\(Θi0\)−\|λk​\(Θi​\(θ\)\)−λk​\(Θi0\)\|\\displaystyle\\geq\\lambda\_\{k\}\(\\Theta\_\{i\}^\{0\}\)\-\|\\lambda\_\{k\}\(\\Theta\_\{i\}\(\\theta\)\)\-\\lambda\_\{k\}\(\\Theta\_\{i\}^\{0\}\)\|≥λk​\(Θi0\)−‖Θi​\(θ\)−Θi0‖op\\displaystyle\\geq\\lambda\_\{k\}\(\\Theta\_\{i\}^\{0\}\)\-\\\|\\Theta\_\{i\}\(\\theta\)\-\\Theta\_\{i\}^\{0\}\\\|\_\{\\rm op\}≥λi,0−LK​‖θ−θi0‖\\displaystyle\\geq\\lambda\_\{i,0\}\-L\_\{K\}\\\|\\theta\-\\theta\_\{i\}^\{0\}\\\|≥λi,0−LK​R\>0\.\\displaystyle\\geq\\lambda\_\{i,0\}\-L\_\{K\}R\>0\.Constant rank ensures thatkkremains the first nonzero spectral index\. Thereforeλmin\+​\(Θi​\(θ\)\)=λk​\(Θi​\(θ\)\)≥λi,0−LK​R\\lambda\_\{\\min\}^\{\+\}\(\\Theta\_\{i\}\(\\theta\)\)=\\lambda\_\{k\}\(\\Theta\_\{i\}\(\\theta\)\)\\geq\\lambda\_\{i,0\}\-L\_\{K\}R\. ∎

###### Lemma G5\(Simplex\-bounded staleness\)\.

Suppose a single active\-peer update satisfies𝔼​‖pit\+1−pit‖2≤cstep​η2\\mathbb\{E\}\\\|p\_\{i\}^\{t\+1\}\-p\_\{i\}^\{t\}\\\|^\{2\}\\leq c\_\{\\rm step\}\\eta^\{2\}\. If the EMA teacher has effective lag constantAeffA\_\{\\rm eff\}as defined in A8, then

Atf≤cstep​Aeff2​η2\.A\_\{t\}^\{f\}\\leq c\_\{\\rm step\}A\_\{\\rm eff\}^\{2\}\\eta^\{2\}\.\(G12\)

###### Proof\.

For a finite\-memory teacher, write

piEMA,t=∑s=1Aωs​pit−s,ωs≥0,\\displaystyle p\_\{i\}^\{\\rm EMA,t\}=\\sum\_\{s=1\}^\{A\}\\omega\_\{s\}p\_\{i\}^\{t\-s\},\\qquad\\omega\_\{s\}\\geq 0,\(G13\)∑s=1Aωs=1,∑s=1As​ωs≤Aeff\.\\displaystyle\\sum\_\{s=1\}^\{A\}\\omega\_\{s\}=1,\\qquad\\sum\_\{s=1\}^\{A\}s\\omega\_\{s\}\\leq A\_\{\\rm eff\}\.\(G14\)
For an ordinary EMA, the same notation usesωs=\(1−β\)​βs−1\\omega\_\{s\}=\(1\-\\beta\)\\beta^\{s\-1\}fors≥1s\\geq 1, with0≤β<10\\leq\\beta<1\. The geometric\-series identities

∑s=1∞βs−1=11−β,∑s=1∞s​βs−1=dd​β​β1−β=1\(1−β\)2\\sum\_\{s=1\}^\{\\infty\}\\beta^\{s\-1\}=\\frac\{1\}\{1\-\\beta\},\\qquad\\sum\_\{s=1\}^\{\\infty\}s\\beta^\{s\-1\}=\\frac\{d\}\{d\\beta\}\\frac\{\\beta\}\{1\-\\beta\}=\\frac\{1\}\{\(1\-\\beta\)^\{2\}\}give∑s≥1ωs=1\\sum\_\{s\\geq 1\}\\omega\_\{s\}=1and∑s≥1s​ωs=\(1−β\)−1\\sum\_\{s\\geq 1\}s\\omega\_\{s\}=\(1\-\\beta\)^\{\-1\}\. The nonnegative weights and this finite first lag moment make the series below convergent inL2L^\{2\}\.

DefineAtf=N−1​∑i𝔼​‖pit−piEMA,t‖2A\_\{t\}^\{f\}=N^\{\-1\}\\sum\_\{i\}\\mathbb\{E\}\\\|p\_\{i\}^\{t\}\-p\_\{i\}^\{\\rm EMA,t\}\\\|^\{2\}\. Since the weights sum to one,

pit−piEMA,t\\displaystyle p\_\{i\}^\{t\}\-p\_\{i\}^\{\\rm EMA,t\}=pit​∑sωs−∑sωs​pit−s\\displaystyle=p\_\{i\}^\{t\}\\sum\_\{s\}\\omega\_\{s\}\-\\sum\_\{s\}\\omega\_\{s\}p\_\{i\}^\{t\-s\}=∑sωs​\(pit−pit−s\)\.\\displaystyle=\\sum\_\{s\}\\omega\_\{s\}\(p\_\{i\}^\{t\}\-p\_\{i\}^\{t\-s\}\)\.For eachss, telescope the difference into increments:

pit−pit−s\\displaystyle p\_\{i\}^\{t\}\-p\_\{i\}^\{t\-s\}=\(pit−pit−1\)\+\(pit−1−pit−2\)\+⋯\\displaystyle=\(p\_\{i\}^\{t\}\-p\_\{i\}^\{t\-1\}\)\+\(p\_\{i\}^\{t\-1\}\-p\_\{i\}^\{t\-2\}\)\+\\cdots\+\(pit−s\+1−pit−s\)\\displaystyle\+\(p\_\{i\}^\{t\-s\+1\}\-p\_\{i\}^\{t\-s\}\)=∑r=0s−1\(pit−r−pit−r−1\),\\displaystyle=\\sum\_\{r=0\}^\{s\-1\}\(p\_\{i\}^\{t\-r\}\-p\_\{i\}^\{t\-r\-1\}\),where all intermediate terms cancel pairwise\. Minkowski’s inequality inL2L^\{2\}states‖∑rXr‖L2≤∑r‖Xr‖L2\\\|\\sum\_\{r\}X\_\{r\}\\\|\_\{L^\{2\}\}\\leq\\sum\_\{r\}\\\|X\_\{r\}\\\|\_\{L^\{2\}\}\. Apply it withXr=pit−r−pit−r−1X\_\{r\}=p\_\{i\}^\{t\-r\}\-p\_\{i\}^\{t\-r\-1\}\. Inactive\-peer increments are zero, and active increments satisfy the assumed bound, so

\(𝔼​‖pit−pit−s‖2\)1/2\\displaystyle\\left\(\\mathbb\{E\}\\\|p\_\{i\}^\{t\}\-p\_\{i\}^\{t\-s\}\\\|^\{2\}\\right\)^\{1/2\}≤∑r=0s−1\(𝔼​‖pit−r−pit−r−1‖2\)1/2\\displaystyle\\leq\\sum\_\{r=0\}^\{s\-1\}\\left\(\\mathbb\{E\}\\\|p\_\{i\}^\{t\-r\}\-p\_\{i\}^\{t\-r\-1\}\\\|^\{2\}\\right\)^\{1/2\}≤∑r=0s−1cstep​η\\displaystyle\\leq\\sum\_\{r=0\}^\{s\-1\}\\sqrt\{c\_\{\\rm step\}\}\\,\\eta=s​cstep​η\.\\displaystyle=s\\sqrt\{c\_\{\\rm step\}\}\\,\\eta\.Applying Minkowski once more to the EMA sum yields

\(𝔼​‖pit−piEMA,t‖2\)1/2\\displaystyle\\left\(\\mathbb\{E\}\\\|p\_\{i\}^\{t\}\-p\_\{i\}^\{\\rm EMA,t\}\\\|^\{2\}\\right\)^\{1/2\}≤∑sωs​\(𝔼​‖pit−pit−s‖2\)1/2\\displaystyle\\leq\\sum\_\{s\}\\omega\_\{s\}\\left\(\\mathbb\{E\}\\\|p\_\{i\}^\{t\}\-p\_\{i\}^\{t\-s\}\\\|^\{2\}\\right\)^\{1/2\}≤cstep​η​∑ss​ωs\\displaystyle\\leq\\sqrt\{c\_\{\\rm step\}\}\\,\\eta\\sum\_\{s\}s\\omega\_\{s\}≤cstep​η​Aeff\.\\displaystyle\\leq\\sqrt\{c\_\{\\rm step\}\}\\,\\eta A\_\{\\rm eff\}\.For the infinite\-memory case, every finite partial sum obeys the same bound\. ItsL2L^\{2\}limit preserves the inequality because\(1−β\)​∑s≥1s​βs−1=1/\(1−β\)<∞\(1\-\\beta\)\\sum\_\{s\\geq 1\}s\\beta^\{s\-1\}=1/\(1\-\\beta\)<\\infty\. Squaring gives

𝔼​‖pit−piEMA,t‖2≤cstep​Aeff2​η2\.\\mathbb\{E\}\\\|p\_\{i\}^\{t\}\-p\_\{i\}^\{\\rm EMA,t\}\\\|^\{2\}\\leq c\_\{\\rm step\}A\_\{\\rm eff\}^\{2\}\\eta^\{2\}\.Averaging this bound gives

Atf\\displaystyle A\_\{t\}^\{f\}=1N​∑i=1N𝔼​‖pit−piEMA,t‖2\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{E\}\\\|p\_\{i\}^\{t\}\-p\_\{i\}^\{\\rm EMA,t\}\\\|^\{2\}≤1N​∑i=1Ncstep​Aeff2​η2\\displaystyle\\leq\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}c\_\{\\rm step\}A\_\{\\rm eff\}^\{2\}\\eta^\{2\}=cstep​Aeff2​η2,\\displaystyle=c\_\{\\rm step\}A\_\{\\rm eff\}^\{2\}\\eta^\{2\},which proves Equation[G12](https://arxiv.org/html/2609.01952#A7.E12)\. ∎

###### Lemma G6\(Frozen\-kernel quotient contraction with restricted KD defect\)\.

Let𝒢=\(V,E\)\\mathcal\{G\}=\(V,E\)be connected, with maximum degreedmaxd\_\{\\max\}and Laplacian eigenvalues0=λ1<λ2≤λmax0=\\lambda\_\{1\}<\\lambda\_\{2\}\\leq\\lambda\_\{\\max\}\. For each peer, letΘi\\Theta\_\{i\}be frozen and satisfyκ−​Πiz⪯Θi⪯κ\+​Πiz\\kappa\_\{\-\}\\Pi\_\{i\}^\{z\}\\preceq\\Theta\_\{i\}\\preceq\\kappa\_\{\+\}\\Pi\_\{i\}^\{z\}\. Define

Wi=Θi†\+γ0\(I−Πiz\),mW=min\{κ\+−1,γ0\},MW=max⁡\{κ−−1,γ0\}\.\\begin\{split\}W\_\{i\}=\\Theta\_\{i\}^\{\\dagger\}\+\\gamma\_\{0\}\(I\-\\Pi\_\{i\}^\{z\}\),\\quad m\_\{W\}=\\min\\\{\\kappa\_\{\+\}^\{\-1\},\\gamma\_\{0\}\\\},\\\\ M\_\{W\}=\\max\\\{\\kappa\_\{\-\}^\{\-1\},\\gamma\_\{0\}\\\}\.\\end\{split\}\(G15\)For𝒱W\\mathcal\{V\}\_\{W\}in Equation[6](https://arxiv.org/html/2609.01952#Sx4.E6), set

aW=2​\(pmin/τ\)​λ2MW​\|E\|,bW=4​κ\+​\(1/\(2​τ\)\)2​λmaxmW​\|E\|,εW=\(pmin/τ\)​λ2​mW2​MW​dmax\.\\begin\{split\}a\_\{W\}=\\frac\{2\(p\_\{\\min\}/\\tau\)\\lambda\_\{2\}\}\{M\_\{W\}\|E\|\},\\qquad b\_\{W\}=\\frac\{4\\kappa\_\{\+\}\(1/\(2\\tau\)\)^\{2\}\\lambda\_\{\\max\}\}\{m\_\{W\}\|E\|\},\\\\ \\varepsilon\_\{W\}=\\frac\{\(p\_\{\\min\}/\\tau\)\\lambda\_\{2\}m\_\{W\}\}\{2M\_\{W\}d\_\{\\max\}\}\.\\end\{split\}\(G16\)Letw=η​α​τw=\\eta\\alpha\\tau\. Assume0<α<10<\\alpha<1, the graph is connected and has no isolated peers, and suppose A10 holds and

γKD≤N​\(pmin/τ\)2​λ22​mW28​MW2​dmax​\|E\|,0<w≤min⁡\{aW8​bW,2aW,1\}\.\\begin\{split\}\\gamma\_\{\\rm KD\}\\leq\\frac\{N\(p\_\{\\min\}/\\tau\)^\{2\}\\lambda\_\{2\}^\{2\}m\_\{W\}^\{2\}\}\{8M\_\{W\}^\{2\}d\_\{\\max\}\|E\|\},\\\\ 0<w\\leq\\min\\left\\\{\\frac\{a\_\{W\}\}\{8b\_\{W\}\},\\frac\{2\}\{a\_\{W\}\},1\\right\\\}\.\\end\{split\}\(G17\)For an active edgee=\{i,j\}e=\\\{i,j\\\}, write the full update as

Zi\+=Zi−w​Θi​di​j\+bi,e\+ξi,e,Zj\+=Zj\+w​Θj​di​j\+bj,e\+ξj,e,\\begin\{split\}Z\_\{i\}^\{\+\}=Z\_\{i\}\-w\\Theta\_\{i\}d\_\{ij\}\+b\_\{i,e\}\+\\xi\_\{i,e\},\\\\ Z\_\{j\}^\{\+\}=Z\_\{j\}\+w\\Theta\_\{j\}d\_\{ij\}\+b\_\{j,e\}\+\\xi\_\{j,e\},\\end\{split\}\(G18\)wherebbcontains deterministic task, curvature, and teacher\-lag perturbations and𝔼t​ξk,e=0\\mathbb\{E\}\_\{t\}\\xi\_\{k,e\}=0\. Define

Db,t=1N​𝔼e​∑k∈e‖bk,e‖Wk2,\\displaystyle D\_\{b,t\}=\\frac\{1\}\{N\}\\mathbb\{E\}\_\{e\}\\sum\_\{k\\in e\}\\\|b\_\{k,e\}\\\|\_\{W\_\{k\}\}^\{2\},\(G19\)Dξ,t=1N​𝔼e​𝔼t​∑k∈e‖ξk,e‖Wk2\.\\displaystyle D\_\{\\xi,t\}=\\frac\{1\}\{N\}\\mathbb\{E\}\_\{e\}\\mathbb\{E\}\_\{t\}\\sum\_\{k\\in e\}\\\|\\xi\_\{k,e\}\\\|\_\{W\_\{k\}\}^\{2\}\.\(G20\)
Then

𝔼t​𝒱W​\(Z\+\)≤\(1−aW​w4\)​𝒱W​\(Z\)\+CB,W​w​Bf2\+\(8​dmaxaW​w​\|E\|\+2\)​Db,t\+Dξ,t,\\begin\{split\}\\mathbb\{E\}\_\{t\}\\mathcal\{V\}\_\{W\}\(Z^\{\+\}\)\\leq\{\}&\\left\(1\-\\frac\{a\_\{W\}w\}\{4\}\\right\)\\mathcal\{V\}\_\{W\}\(Z\)\+C\_\{B,W\}wB\_\{f\}^\{2\}\\\\ &\+\\left\(\\frac\{8d\_\{\\max\}\}\{a\_\{W\}w\|E\|\}\+2\\right\)D\_\{b,t\}\+D\_\{\\xi,t\},\\end\{split\}\(G21\)whereCB,W=4​MW​dmax​νKD,B/\[N⁡\(pmin/τ\)​λ2​mW\]C\_\{B,W\}=4M\_\{W\}d\_\{\\max\}\\nu\_\{\\rm KD,B\}/\[N\(p\_\{\\min\}/\\tau\)\\lambda\_\{2\}m\_\{W\}\]\.

###### Proof\.

For‖u‖Wi2:=⟨u,Wi​u⟩\\\|u\\\|\_\{W\_\{i\}\}^\{2\}:=\\langle u,W\_\{i\}u\\rangle, the spectral bounds onΘi\\Theta\_\{i\}imply

κ\+−1​Πiz⪯Θi†⪯κ−−1​Πiz\.\\kappa\_\{\+\}^\{\-1\}\\Pi\_\{i\}^\{z\}\\preceq\\Theta\_\{i\}^\{\\dagger\}\\preceq\\kappa\_\{\-\}^\{\-1\}\\Pi\_\{i\}^\{z\}\.LetΘi=∑ℓ=1riλi,ℓ​ui,ℓ​ui,ℓ⊤\\Theta\_\{i\}=\\sum\_\{\\ell=1\}^\{r\_\{i\}\}\\lambda\_\{i,\\ell\}u\_\{i,\\ell\}u\_\{i,\\ell\}^\{\\top\}onTizT\_\{i\}^\{z\}, whereκ−≤λi,ℓ≤κ\+\\kappa\_\{\-\}\\leq\\lambda\_\{i,\\ell\}\\leq\\kappa\_\{\+\}\. Then

Θi†=∑ℓ=1riλi,ℓ−1​ui,ℓ​ui,ℓ⊤,κ\+−1≤λi,ℓ−1≤κ−−1\.\\Theta\_\{i\}^\{\\dagger\}=\\sum\_\{\\ell=1\}^\{r\_\{i\}\}\\lambda\_\{i,\\ell\}^\{\-1\}u\_\{i,\\ell\}u\_\{i,\\ell\}^\{\\top\},\\qquad\\kappa\_\{\+\}^\{\-1\}\\leq\\lambda\_\{i,\\ell\}^\{\-1\}\\leq\\kappa\_\{\-\}^\{\-1\}\.On\(Tiz\)⟂\(T\_\{i\}^\{z\}\)^\{\\perp\},Θi†\\Theta\_\{i\}^\{\\dagger\}is zero and the second summand inWiW\_\{i\}isγ0​I\\gamma\_\{0\}I\. Every eigenvalue ofWiW\_\{i\}therefore lies in\[mW,MW\]\[m\_\{W\},M\_\{W\}\], which givesmW​I⪯Wi⪯MW​Im\_\{W\}I\\preceq W\_\{i\}\\preceq M\_\{W\}I\.

The pseudoinverse andΘi\\Theta\_\{i\}are diagonal in the same eigenbasis\. Thus they commute, and

Wi​Θi\\displaystyle W\_\{i\}\\Theta\_\{i\}=Θi†​Θi\+γ0​\(I−Πiz\)​Θi\\displaystyle=\\Theta\_\{i\}^\{\\dagger\}\\Theta\_\{i\}\+\\gamma\_\{0\}\(I\-\\Pi\_\{i\}^\{z\}\)\\Theta\_\{i\}=∑ℓ=1riui,ℓ​ui,ℓ⊤\+0\\displaystyle=\\sum\_\{\\ell=1\}^\{r\_\{i\}\}u\_\{i,\\ell\}u\_\{i,\\ell\}^\{\\top\}\+0=Πiz\.\\displaystyle=\\Pi\_\{i\}^\{z\}\.
Defineq⁡\(c\)=N−1​∑i⟨Zi−c,Wi​\(Zi−c\)⟩q\(c\)=N^\{\-1\}\\sum\_\{i\}\\langle Z\_\{i\}\-c,W\_\{i\}\(Z\_\{i\}\-c\)\\rangle\. Its derivative in the coordinate directionhhis

D​q​\(c\)​\[h\]\\displaystyle Dq\(c\)\[h\]=dd​ϵ​q​\(c\+ϵ​h\)\|ϵ=0\\displaystyle=\\left\.\\frac\{d\}\{d\\epsilon\}q\(c\+\\epsilon h\)\\right\|\_\{\\epsilon=0\}=1N​∑idd​ϵ​⟨Zi−c−ϵ​h,Wi​\(Zi−c−ϵ​h\)⟩\|ϵ=0\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i\}\\left\.\\frac\{d\}\{d\\epsilon\}\\langle Z\_\{i\}\-c\-\\epsilon h,W\_\{i\}\(Z\_\{i\}\-c\-\\epsilon h\)\\rangle\\right\|\_\{\\epsilon=0\}=2N​∑i⟨Wi​\(c−Zi\),h⟩\.\\displaystyle=\\frac\{2\}\{N\}\\sum\_\{i\}\\langle W\_\{i\}\(c\-Z\_\{i\}\),h\\rangle\.Moreover,D2​q​\(c\)​\[h,h\]=2​N−1​∑i⟨h,Wi​h⟩≥2​mW​‖h‖2\>0D^\{2\}q\(c\)\[h,h\]=2N^\{\-1\}\\sum\_\{i\}\\langle h,W\_\{i\}h\\rangle\\geq 2m\_\{W\}\\\|h\\\|^\{2\}\>0forh≠0h\\neq 0, soqqis strictly convex\. The unique stationary point satisfies

0\\displaystyle 0=∑iWi​\(cW−Zi\),\\displaystyle=\\sum\_\{i\}W\_\{i\}\(c\_\{W\}\-Z\_\{i\}\),\(∑iWi\)​cW\\displaystyle\\left\(\\sum\_\{i\}W\_\{i\}\\right\)c\_\{W\}=∑iWi​Zi,\\displaystyle=\\sum\_\{i\}W\_\{i\}Z\_\{i\},cW\\displaystyle c\_\{W\}=\(∑iWi\)−1​∑iWi​Zi,\\displaystyle=\\left\(\\sum\_\{i\}W\_\{i\}\\right\)^\{\-1\}\\sum\_\{i\}W\_\{i\}Z\_\{i\},∑iWi​\(Zi−cW\)\\displaystyle\\sum\_\{i\}W\_\{i\}\(Z\_\{i\}\-c\_\{W\}\)=0\.\\displaystyle=0\.The inverse exists because∑iWi⪰N​mW​I\\sum\_\{i\}W\_\{i\}\\succeq Nm\_\{W\}I\. LetZ¯=N−1​∑iZi\\bar\{Z\}=N^\{\-1\}\\sum\_\{i\}Z\_\{i\}\. The lower metric comparison is

𝒱W​\(Z\)\\displaystyle\\mathcal\{V\}\_\{W\}\(Z\)=minc⁡1N​∑i‖Zi−c‖Wi2\\displaystyle=\\min\_\{c\}\\frac\{1\}\{N\}\\sum\_\{i\}\\\|Z\_\{i\}\-c\\\|\_\{W\_\{i\}\}^\{2\}≥mW​minc​1N​∑i‖Zi−c‖2\\displaystyle\\geq m\_\{W\}\\min\_\{c\}\\frac\{1\}\{N\}\\sum\_\{i\}\\\|Z\_\{i\}\-c\\\|^\{2\}=mW​Ψz​\(Z\)\.\\displaystyle=m\_\{W\}\\Psi^\{z\}\(Z\)\.The final equality uses

∑i‖Zi−c‖2\\displaystyle\\sum\_\{i\}\\\|Z\_\{i\}\-c\\\|^\{2\}=∑i‖\(Zi−Z¯\)\+\(Z¯−c\)‖2\\displaystyle=\\sum\_\{i\}\\\|\(Z\_\{i\}\-\\bar\{Z\}\)\+\(\\bar\{Z\}\-c\)\\\|^\{2\}=∑i‖Zi−Z¯‖2\+2​⟨∑i\(Zi−Z¯\),Z¯−c⟩\\displaystyle=\\sum\_\{i\}\\\|Z\_\{i\}\-\\bar\{Z\}\\\|^\{2\}\+2\\left\\langle\\sum\_\{i\}\(Z\_\{i\}\-\\bar\{Z\}\),\\bar\{Z\}\-c\\right\\rangle\+N​‖Z¯−c‖2\\displaystyle\+N\\\|\\bar\{Z\}\-c\\\|^\{2\}=∑i‖Zi−Z¯‖2\+N​‖Z¯−c‖2,\\displaystyle=\\sum\_\{i\}\\\|Z\_\{i\}\-\\bar\{Z\}\\\|^\{2\}\+N\\\|\\bar\{Z\}\-c\\\|^\{2\},where the cross term vanishes because∑i\(Zi−Z¯\)=0\\sum\_\{i\}\(Z\_\{i\}\-\\bar\{Z\}\)=0\. The minimum is attained atc=Z¯c=\\bar\{Z\}\. Evaluating the minimum atc=Z¯c=\\bar\{Z\}gives

𝒱W​\(Z\)≤1N​∑i‖Zi−Z¯‖Wi2≤MW​Ψz​\(Z\)\.\\mathcal\{V\}\_\{W\}\(Z\)\\leq\\frac\{1\}\{N\}\\sum\_\{i\}\\\|Z\_\{i\}\-\\bar\{Z\}\\\|\_\{W\_\{i\}\}^\{2\}\\leq M\_\{W\}\\Psi^\{z\}\(Z\)\.\(G22\)This proves Equation[G22](https://arxiv.org/html/2609.01952#A7.E22)\.

LetQi=Zi−cWQ\_\{i\}=Z\_\{i\}\-c\_\{W\}, definedj​i:=−di​jd\_\{ji\}:=\-d\_\{ij\}, and setri​j=\(I−Πiz\)​di​jr\_\{ij\}=\(I\-\\Pi\_\{i\}^\{z\}\)d\_\{ij\}for every directed edge\. Since the new quotient potential minimizes over its center, evaluating it at the old minimizer gives

N​𝒱W​\(Z\+\)≤∑k∉e‖Qk‖Wk2\+∑k∈e‖Zk\+−cW‖Wk2\.N\\mathcal\{V\}\_\{W\}\(Z^\{\+\}\)\\leq\\sum\_\{k\\notin e\}\\\|Q\_\{k\}\\\|\_\{W\_\{k\}\}^\{2\}\+\\sum\_\{k\\in e\}\\\|Z\_\{k\}^\{\+\}\-c\_\{W\}\\\|\_\{W\_\{k\}\}^\{2\}\.Only the two endpoint terms differ fromN​𝒱W​\(Z\)N\\mathcal\{V\}\_\{W\}\(Z\)\. Expanding both endpoint squares gives

‖Qi−w​Θi​di​j\+bi,e\+ξi,e‖Wi2\+‖Qj\+w​Θj​di​j\+bj,e\+ξj,e‖Wj2\\displaystyle\\\|Q\_\{i\}\-w\\Theta\_\{i\}d\_\{ij\}\+b\_\{i,e\}\+\\xi\_\{i,e\}\\\|\_\{W\_\{i\}\}^\{2\}\+\\\|Q\_\{j\}\+w\\Theta\_\{j\}d\_\{ij\}\+b\_\{j,e\}\+\\xi\_\{j,e\}\\\|\_\{W\_\{j\}\}^\{2\}=‖Qi‖Wi2\+‖Qj‖Wj2−2​w​⟨Qi,Πiz​di​j⟩\+2​w​⟨Qj,Πjz​di​j⟩\\displaystyle=\\\|Q\_\{i\}\\\|\_\{W\_\{i\}\}^\{2\}\+\\\|Q\_\{j\}\\\|\_\{W\_\{j\}\}^\{2\}\-2w\\langle Q\_\{i\},\\Pi\_\{i\}^\{z\}d\_\{ij\}\\rangle\+2w\\langle Q\_\{j\},\\Pi\_\{j\}^\{z\}d\_\{ij\}\\rangle\+w2​‖Θi​di​j‖Wi2\+w2​‖Θj​di​j‖Wj2\+2​∑k∈e⟨Qk,Wk​\(bk,e\+ξk,e\)⟩\\displaystyle\+w^\{2\}\\\|\\Theta\_\{i\}d\_\{ij\}\\\|\_\{W\_\{i\}\}^\{2\}\+w^\{2\}\\\|\\Theta\_\{j\}d\_\{ij\}\\\|\_\{W\_\{j\}\}^\{2\}\+2\\sum\_\{k\\in e\}\\langle Q\_\{k\},W\_\{k\}\(b\_\{k,e\}\+\\xi\_\{k,e\}\)\\rangle\+2​w​⟨−Θi​di​j,Wi​\(bi,e\+ξi,e\)⟩\+2​w​⟨Θj​di​j,Wj​\(bj,e\+ξj,e\)⟩\\displaystyle\+2w\\langle\-\\Theta\_\{i\}d\_\{ij\},W\_\{i\}\(b\_\{i,e\}\+\\xi\_\{i,e\}\)\\rangle\+2w\\langle\\Theta\_\{j\}d\_\{ij\},W\_\{j\}\(b\_\{j,e\}\+\\xi\_\{j,e\}\)\\rangle\+∑k∈e∥bk,e\+ξk,e∥Wk2\.\\displaystyle\+\\sum\_\{k\\in e\}\\\|b\_\{k,e\}\+\\xi\_\{k,e\}\\\|\_\{W\_\{k\}\}^\{2\}\.For example, the first endpoint uses‖Qi\+ui‖Wi2=‖Qi‖Wi2\+2​⟨Qi,Wi​ui⟩\+‖ui‖Wi2\\\|Q\_\{i\}\+u\_\{i\}\\\|\_\{W\_\{i\}\}^\{2\}=\\\|Q\_\{i\}\\\|\_\{W\_\{i\}\}^\{2\}\+2\\langle Q\_\{i\},W\_\{i\}u\_\{i\}\\rangle\+\\\|u\_\{i\}\\\|\_\{W\_\{i\}\}^\{2\}withui=−w​Θi​di​j\+bi,e\+ξi,eu\_\{i\}=\-w\\Theta\_\{i\}d\_\{ij\}\+b\_\{i,e\}\+\\xi\_\{i,e\}; expanding the last square produces the KD–perturbation cross term displayed above\.

Now expand the linear KD terms\. SinceΠiz​di​j=di​j−ri​j\\Pi\_\{i\}^\{z\}d\_\{ij\}=d\_\{ij\}\-r\_\{ij\}andΠjz​di​j=di​j\+rj​i\\Pi\_\{j\}^\{z\}d\_\{ij\}=d\_\{ij\}\+r\_\{ji\}, whererj​i=\(I−Πjz\)​dj​i=−\(I−Πjz\)​di​jr\_\{ji\}=\(I\-\\Pi\_\{j\}^\{z\}\)d\_\{ji\}=\-\(I\-\\Pi\_\{j\}^\{z\}\)d\_\{ij\}, one obtains

−2w⟨Qi,Πizdi​j⟩\+2w⟨Qj,Πjzdi​j⟩=−2​w​⟨Qi,di​j⟩\+2​w​⟨Qi,ri​j⟩\+2​w​⟨Qj,di​j⟩\+2​w​⟨Qj,rj​i⟩=−2​w​⟨Qi−Qj,di​j⟩\+2​w​⟨Qi,ri​j⟩\+2​w​⟨Qj,rj​i⟩=−2​w​⟨Zi−Zj,di​j⟩\+2​w​⟨Qi,ri​j⟩\+2​w​⟨Qj,rj​i⟩\.\\begin\{split\}\-2w\\langle Q\_\{i\},&\\Pi\_\{i\}^\{z\}d\_\{ij\}\\rangle\+2w\\langle Q\_\{j\},\\Pi\_\{j\}^\{z\}d\_\{ij\}\\rangle\\\\ =\{\}&\-2w\\langle Q\_\{i\},d\_\{ij\}\\rangle\+2w\\langle Q\_\{i\},r\_\{ij\}\\rangle\\\\ &\+2w\\langle Q\_\{j\},d\_\{ij\}\\rangle\+2w\\langle Q\_\{j\},r\_\{ji\}\\rangle\\\\ =\{\}&\-2w\\langle Q\_\{i\}\-Q\_\{j\},d\_\{ij\}\\rangle\\\\ &\+2w\\langle Q\_\{i\},r\_\{ij\}\\rangle\+2w\\langle Q\_\{j\},r\_\{ji\}\\rangle\\\\ =\{\}&\-2w\\langle Z\_\{i\}\-Z\_\{j\},d\_\{ij\}\\rangle\\\\ &\+2w\\langle Q\_\{i\},r\_\{ij\}\\rangle\+2w\\langle Q\_\{j\},r\_\{ji\}\\rangle\.\\end\{split\}\(G23\)Lethi​j=Zi−Zjh\_\{ij\}=Z\_\{i\}\-Z\_\{j\}andAi​j=∫01DZ​P​\(Zj\+s​hi​j\)​𝑑sA\_\{ij\}=\\int\_\{0\}^\{1\}D\_\{Z\}P\(Z\_\{j\}\+sh\_\{ij\}\)\\,ds\. The fundamental theorem of calculus givesdi​j=Pi−Pj=Ai​j​hi​jd\_\{ij\}=P\_\{i\}\-P\_\{j\}=A\_\{ij\}h\_\{ij\}\. Lemma[G2](https://arxiv.org/html/2609.01952#Thmlemma2a)gives

⟨Zi−Zj,di​j⟩\\displaystyle\\langle Z\_\{i\}\-Z\_\{j\},d\_\{ij\}\\rangle=∫01hi​j⊤​DZ​P​\(Zj\+s​hi​j\)​hi​j​𝑑s\\displaystyle=\\int\_\{0\}^\{1\}h\_\{ij\}^\{\\top\}D\_\{Z\}P\(Z\_\{j\}\+sh\_\{ij\}\)h\_\{ij\}\\,ds≥∫01pminτ​‖hi​j‖2​𝑑s\\displaystyle\\geq\\int\_\{0\}^\{1\}\\frac\{p\_\{\\min\}\}\{\\tau\}\\\|h\_\{ij\}\\\|^\{2\}\\,ds=pminτ​‖Zi−Zj‖2\.\\displaystyle=\\frac\{p\_\{\\min\}\}\{\\tau\}\\\|Z\_\{i\}\-Z\_\{j\}\\\|^\{2\}\.For the graph Poincare inequality, expand the centered peer vector in an eigenbasis ofℒ\\mathcal\{L\}\. Its consensus coefficient is zero, so only eigenvalues at leastλ2\\lambda\_\{2\}remain:

∑\{i,j\}∈E‖Zi−Zj‖2\\displaystyle\\sum\_\{\\\{i,j\\\}\\in E\}\\\|Z\_\{i\}\-Z\_\{j\}\\\|^\{2\}=⟨Z−Z¯,\(ℒ⊗I\)​\(Z−Z¯\)⟩\\displaystyle=\\langle Z\-\\bar\{Z\},\(\\mathcal\{L\}\\otimes I\)\(Z\-\\bar\{Z\}\)\\rangle≥λ2​∑i‖Zi−Z¯‖2\\displaystyle\\geq\\lambda\_\{2\}\\sum\_\{i\}\\\|Z\_\{i\}\-\\bar\{Z\}\\\|^\{2\}=N​λ2​Ψz​\(Z\)\.\\displaystyle=N\\lambda\_\{2\}\\Psi^\{z\}\(Z\)\.Uniform edge sampling therefore gives

−2​wN​\|E\|∑\{i,j\}∈E⟨Zi−Zj,di​j⟩\\displaystyle\-\\frac\{2w\}\{N\|E\|\}\\sum\_\{\\\{i,j\\\}\\in E\}\\langle Z\_\{i\}\-Z\_\{j\},d\_\{ij\}\\rangle≤−2​w​\(pmin/τ\)N​\|E\|∑\{i,j\}∈E∥Zi−Zj∥2\\displaystyle\\leq\-\\frac\{2w\(p\_\{\\min\}/\\tau\)\}\{N\|E\|\}\\sum\_\{\\\{i,j\\\}\\in E\}\\\|Z\_\{i\}\-Z\_\{j\}\\\|^\{2\}≤−2​w​\(pmin/τ\)​λ2\|E\|​Ψz​\(Z\)\\displaystyle\\leq\-\\frac\{2w\(p\_\{\\min\}/\\tau\)\\lambda\_\{2\}\}\{\|E\|\}\\Psi^\{z\}\(Z\)≤−2​w​\(pmin/τ\)​λ2MW​\|E\|​𝒱W​\(Z\)\\displaystyle\\leq\-\\frac\{2w\(p\_\{\\min\}/\\tau\)\\lambda\_\{2\}\}\{M\_\{W\}\|E\|\}\\mathcal\{V\}\_\{W\}\(Z\)=−aW​w​𝒱W​\(Z\)\.\\displaystyle=\-a\_\{W\}w\\mathcal\{V\}\_\{W\}\(Z\)\.
Use Young’s inequality in the form2​⟨a,b⟩≤ε​‖a‖2\+ε−1​‖b‖22\\langle a,b\\rangle\\leq\\varepsilon\\\|a\\\|^\{2\}\+\\varepsilon^\{\-1\}\\\|b\\\|^\{2\}witha=Qia=Q\_\{i\},b=ri​jb=r\_\{ij\}, andε=εW\\varepsilon=\\varepsilon\_\{W\}\. This inequality comes from0≤∥εa−ε−1/2b∥20\\leq\\\|\\sqrt\{\\varepsilon\}a\-\\varepsilon^\{\-1/2\}b\\\|^\{2\}\. Summing over directed edges gives

2​wN​\|E\|​∑\(i,j\)∈E→⟨Qi,ri​j⟩≤\\displaystyle\\frac\{2w\}\{N\|E\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\langle Q\_\{i\},r\_\{ij\}\\rangle\\leq\{\}w​εWN​\|E\|​∑\(i,j\)∈E→‖Qi‖2\\displaystyle\\frac\{w\\varepsilon\_\{W\}\}\{N\|E\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|Q\_\{i\}\\\|^\{2\}\+wN​\|E\|​εW∑\(i,j\)∈E→∥ri​j∥2\.\\displaystyle\+\\frac\{w\}\{N\|E\|\\varepsilon\_\{W\}\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|r\_\{ij\}\\\|^\{2\}\.The first directed sum counts peeriiexactlydid\_\{i\}times:

∑\(i,j\)∈E→‖Qi‖2\\displaystyle\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|Q\_\{i\}\\\|^\{2\}=∑idi​‖Qi‖2\\displaystyle=\\sum\_\{i\}d\_\{i\}\\\|Q\_\{i\}\\\|^\{2\}≤dmax​∑i‖Qi‖2\\displaystyle\\leq d\_\{\\max\}\\sum\_\{i\}\\\|Q\_\{i\}\\\|^\{2\}≤N​dmaxmW​𝒱W\.\\displaystyle\\leq\\frac\{Nd\_\{\\max\}\}\{m\_\{W\}\}\\mathcal\{V\}\_\{W\}\.A10 and\|E→\|=2​\|E\|\|\\vec\{E\}\|=2\|E\|give

∑\(i,j\)∈E→‖ri​j‖2\\displaystyle\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|r\_\{ij\}\\\|^\{2\}≤2​\|E\|​\(γKD​Ψz\+νKD,B​Bf2\)\\displaystyle\\leq 2\|E\|\\left\(\\gamma\_\{\\rm KD\}\\Psi^\{z\}\+\\nu\_\{\\rm KD,B\}B\_\{f\}^\{2\}\\right\)≤2​\|E\|​\(γKDmW​𝒱W\+νKD,B​Bf2\)\.\\displaystyle\\leq 2\|E\|\\left\(\\frac\{\\gamma\_\{\\rm KD\}\}\{m\_\{W\}\}\\mathcal\{V\}\_\{W\}\+\\nu\_\{\\rm KD,B\}B\_\{f\}^\{2\}\\right\)\.Combining these two bounds gives

2​wN​\|E\|​∑\(i,j\)∈E→⟨Qi,ri​j⟩≤w⁡\[εW​dmaxmW​\|E\|\+2​γKDN​εW​mW\]​𝒱W\+2​w​νKD,BN​εW​Bf2\.\\begin\{split\}\\frac\{2w\}\{N\|E\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\langle Q\_\{i\},r\_\{ij\}\\rangle\\leq\{\}&w\\left\[\\frac\{\\varepsilon\_\{W\}d\_\{\\max\}\}\{m\_\{W\}\|E\|\}\+\\frac\{2\\gamma\_\{\\rm KD\}\}\{N\\varepsilon\_\{W\}m\_\{W\}\}\\right\]\\mathcal\{V\}\_\{W\}\\\\ &\+\\frac\{2w\\nu\_\{\\rm KD,B\}\}\{N\\varepsilon\_\{W\}\}B\_\{f\}^\{2\}\.\\end\{split\}\(G24\)The two coefficients in the bracket are explicit:

εW​dmaxmW​\|E\|=\(pmin/τ\)​λ22​MW​\|E\|=aW4,\\frac\{\\varepsilon\_\{W\}d\_\{\\max\}\}\{m\_\{W\}\|E\|\}=\\frac\{\(p\_\{\\min\}/\\tau\)\\lambda\_\{2\}\}\{2M\_\{W\}\|E\|\}=\\frac\{a\_\{W\}\}\{4\},and the assumed defect threshold gives

2​γKDN​εW​mW\\displaystyle\\frac\{2\\gamma\_\{\\rm KD\}\}\{N\\varepsilon\_\{W\}m\_\{W\}\}≤2N​εW​mW​N​\(pmin/τ\)2​λ22​mW28​MW2​dmax​\|E\|\\displaystyle\\leq\\frac\{2\}\{N\\varepsilon\_\{W\}m\_\{W\}\}\\frac\{N\(p\_\{\\min\}/\\tau\)^\{2\}\\lambda\_\{2\}^\{2\}m\_\{W\}^\{2\}\}\{8M\_\{W\}^\{2\}d\_\{\\max\}\|E\|\}=\(pmin/τ\)​λ22​MW​\|E\|=aW4\.\\displaystyle=\\frac\{\(p\_\{\\min\}/\\tau\)\\lambda\_\{2\}\}\{2M\_\{W\}\|E\|\}=\\frac\{a\_\{W\}\}\{4\}\.Thus the bracket is at mostaW/2a\_\{W\}/2\. The capacity coefficient is

2​νKD,BN​εW=4​MW​dmax​νKD,BN⁡\(pmin/τ\)​λ2​mW=CB,W\.\\frac\{2\\nu\_\{\\rm KD,B\}\}\{N\\varepsilon\_\{W\}\}=\\frac\{4M\_\{W\}d\_\{\\max\}\\nu\_\{\\rm KD,B\}\}\{N\(p\_\{\\min\}/\\tau\)\\lambda\_\{2\}m\_\{W\}\}=C\_\{B,W\}\.
For the quadratic KD contribution, commutation ofWiW\_\{i\}andΘi\\Theta\_\{i\}gives

‖Θi​di​j‖Wi2\\displaystyle\\\|\\Theta\_\{i\}d\_\{ij\}\\\|\_\{W\_\{i\}\}^\{2\}=di​j⊤​Θi​Wi​Θi​di​j\\displaystyle=d\_\{ij\}^\{\\top\}\\Theta\_\{i\}W\_\{i\}\\Theta\_\{i\}d\_\{ij\}=di​j⊤​Θi​Πiz​di​j\\displaystyle=d\_\{ij\}^\{\\top\}\\Theta\_\{i\}\\Pi\_\{i\}^\{z\}d\_\{ij\}=di​j⊤​Θi​di​j\\displaystyle=d\_\{ij\}^\{\\top\}\\Theta\_\{i\}d\_\{ij\}≤κ\+​‖di​j‖2\\displaystyle\\leq\\kappa\_\{\+\}\\\|d\_\{ij\}\\\|^\{2\}≤κ\+4​τ2​‖Zi−Zj‖2\.\\displaystyle\\leq\\frac\{\\kappa\_\{\+\}\}\{4\\tau^\{2\}\}\\\|Z\_\{i\}\-Z\_\{j\}\\\|^\{2\}\.The final line uses‖di​j‖≤\(2​τ\)−1​‖Zi−Zj‖\\\|d\_\{ij\}\\\|\\leq\(2\\tau\)^\{\-1\}\\\|Z\_\{i\}\-Z\_\{j\}\\\|, obtained by integrating the softmax Jacobian bound\. Summing the two endpoint squares and averaging over the edge gives

QKD\\displaystyle Q\_\{\\rm KD\}:=w2N​\|E\|​∑\{i,j\}∈E\(‖Θi​di​j‖Wi2\+‖Θj​di​j‖Wj2\)\\displaystyle:=\\frac\{w^\{2\}\}\{N\|E\|\}\\sum\_\{\\\{i,j\\\}\\in E\}\\left\(\\\|\\Theta\_\{i\}d\_\{ij\}\\\|\_\{W\_\{i\}\}^\{2\}\+\\\|\\Theta\_\{j\}d\_\{ij\}\\\|\_\{W\_\{j\}\}^\{2\}\\right\)≤w2​κ\+2​τ2​N​\|E\|​∑\{i,j\}∈E‖Zi−Zj‖2\\displaystyle\\leq\\frac\{w^\{2\}\\kappa\_\{\+\}\}\{2\\tau^\{2\}N\|E\|\}\\sum\_\{\\\{i,j\\\}\\in E\}\\\|Z\_\{i\}\-Z\_\{j\}\\\|^\{2\}≤w2​κ\+​λmax2​τ2​\|E\|​Ψz\\displaystyle\\leq\\frac\{w^\{2\}\\kappa\_\{\+\}\\lambda\_\{\\max\}\}\{2\\tau^\{2\}\|E\|\}\\Psi^\{z\}≤bW​w22​𝒱W\.\\displaystyle\\leq\\frac\{b\_\{W\}w^\{2\}\}\{2\}\\mathcal\{V\}\_\{W\}\.The factor1/21/2in the last line is retained becausebWb\_\{W\}is twice the preceding coefficient afterΨz≤𝒱W/mW\\Psi^\{z\}\\leq\\mathcal\{V\}\_\{W\}/m\_\{W\}\.

For the deterministic KD–perturbation cross term, chooseu=w​Θk​di​ju=w\\Theta\_\{k\}d\_\{ij\}andv=bk,ev=b\_\{k,e\}in2​\|⟨u,v⟩Wk\|≤‖u‖Wk2\+‖v‖Wk22\|\\langle u,v\\rangle\_\{W\_\{k\}\}\|\\leq\\\|u\\\|\_\{W\_\{k\}\}^\{2\}\+\\\|v\\\|\_\{W\_\{k\}\}^\{2\}\. After endpoint and edge averaging, the two squares contributeQKD\+Db,tQ\_\{\\rm KD\}\+D\_\{b,t\}\. The original KD square and this extra copy satisfy

2​QKD≤bW​w2​𝒱W≤aW​w8​𝒱W,2Q\_\{\\rm KD\}\\leq b\_\{W\}w^\{2\}\\mathcal\{V\}\_\{W\}\\leq\\frac\{a\_\{W\}w\}\{8\}\\mathcal\{V\}\_\{W\},where the last inequality usesw≤aW/\(8​bW\)w\\leq a\_\{W\}/\(8b\_\{W\}\)\. The KD vectors are measurable conditional on\(ℱt,e\)\(\\mathcal\{F\}\_\{t\},e\), so their inner products with the centeredξk,e\\xi\_\{k,e\}have zero conditional expectation\.

For deterministic perturbations, apply

2​⟨Qk,Wk​bk,e⟩≤εb​‖Qk‖Wk2\+εb−1​‖bk,e‖Wk22\\langle Q\_\{k\},W\_\{k\}b\_\{k,e\}\\rangle\\leq\\varepsilon\_\{b\}\\\|Q\_\{k\}\\\|\_\{W\_\{k\}\}^\{2\}\+\\varepsilon\_\{b\}^\{\-1\}\\\|b\_\{k,e\}\\\|\_\{W\_\{k\}\}^\{2\}withεb=aW​w​\|E\|/\(8​dmax\)\\varepsilon\_\{b\}=a\_\{W\}w\|E\|/\(8d\_\{\\max\}\)\. The degree count in the edge average gives

2N​𝔼e​∑k∈e⟨Qk,Wk​bk,e⟩\\displaystyle\\frac\{2\}\{N\}\\mathbb\{E\}\_\{e\}\\sum\_\{k\\in e\}\\langle Q\_\{k\},W\_\{k\}b\_\{k,e\}\\rangle≤εbN​\|E\|​∑idi​‖Qi‖Wi2\+εb−1​Db,t\\displaystyle\\leq\\frac\{\\varepsilon\_\{b\}\}\{N\|E\|\}\\sum\_\{i\}d\_\{i\}\\\|Q\_\{i\}\\\|\_\{W\_\{i\}\}^\{2\}\+\\varepsilon\_\{b\}^\{\-1\}D\_\{b,t\}≤εb​dmax\|E\|​𝒱W\+εb−1​Db,t\\displaystyle\\leq\\frac\{\\varepsilon\_\{b\}d\_\{\\max\}\}\{\|E\|\}\\mathcal\{V\}\_\{W\}\+\\varepsilon\_\{b\}^\{\-1\}D\_\{b,t\}=aW​w8​𝒱W\+8​dmaxaW​w​\|E\|​Db,t\.\\displaystyle=\\frac\{a\_\{W\}w\}\{8\}\\mathcal\{V\}\_\{W\}\+\\frac\{8d\_\{\\max\}\}\{a\_\{W\}w\|E\|\}D\_\{b,t\}\.Conditional on\(ℱt,e\)\(\\mathcal\{F\}\_\{t\},e\),QkQ\_\{k\},bk,eb\_\{k,e\}, and the KD vectors are measurable, while𝔼t​ξk,e=0\\mathbb\{E\}\_\{t\}\\xi\_\{k,e\}=0\. Hence

𝔼t​⟨Qk,Wk​ξk,e⟩=0,𝔼t​⟨bk,e,Wk​ξk,e⟩=0\.\\mathbb\{E\}\_\{t\}\\langle Q\_\{k\},W\_\{k\}\\xi\_\{k,e\}\\rangle=0,\\qquad\\mathbb\{E\}\_\{t\}\\langle b\_\{k,e\},W\_\{k\}\\xi\_\{k,e\}\\rangle=0\.The perturbation square has the exact conditional expansion

𝔼t​‖bk,e\+ξk,e‖Wk2\\displaystyle\\mathbb\{E\}\_\{t\}\\\|b\_\{k,e\}\+\\xi\_\{k,e\}\\\|\_\{W\_\{k\}\}^\{2\}=‖bk,e‖Wk2\+2​𝔼t​⟨bk,e,Wk​ξk,e⟩\\displaystyle=\\\|b\_\{k,e\}\\\|\_\{W\_\{k\}\}^\{2\}\+2\\mathbb\{E\}\_\{t\}\\langle b\_\{k,e\},W\_\{k\}\\xi\_\{k,e\}\\rangle\+𝔼t​‖ξk,e‖Wk2\\displaystyle\+\\mathbb\{E\}\_\{t\}\\\|\\xi\_\{k,e\}\\\|\_\{W\_\{k\}\}^\{2\}=‖bk,e‖Wk2\+𝔼t​‖ξk,e‖Wk2\.\\displaystyle=\\\|b\_\{k,e\}\\\|\_\{W\_\{k\}\}^\{2\}\+\\mathbb\{E\}\_\{t\}\\\|\\xi\_\{k,e\}\\\|\_\{W\_\{k\}\}^\{2\}\.The KD–bbcross bound contributes oneDb,tD\_\{b,t\}and the perturbation square contributes another, giving2​Db,t\+Dξ,t2D\_\{b,t\}\+D\_\{\\xi,t\}\. The coefficient of𝒱W\\mathcal\{V\}\_\{W\}is

−aW​w\+aW​w2\+aW​w8\+aW​w8=−aW​w4\.\-a\_\{W\}w\+\\frac\{a\_\{W\}w\}\{2\}\+\\frac\{a\_\{W\}w\}\{8\}\+\\frac\{a\_\{W\}w\}\{8\}=\-\\frac\{a\_\{W\}w\}\{4\}\.Together with the capacity term and the inverse\-wwdeterministic perturbation term, this proves Equation[G21](https://arxiv.org/html/2609.01952#A7.E21)\. ∎

###### Lemma G7\(Full\-event consensus recursion\)\.

LetGt=N−1​∑i‖git‖2G\_\{t\}=N^\{\-1\}\\sum\_\{i\}\\\|g\_\{i\}^\{t\}\\\|^\{2\}\. Under A2, A4, A6, and A8, the perturbations in Lemma[G6](https://arxiv.org/html/2609.01952#Thmlemma6)satisfy

Db,t≤RG​η2​Gt\+RB​η2​Bf2\+Rζ​η2​ζf2\+R0​η4\+RA​w2​Atf,Dξ,t≤Σ0​η2\.\\begin\{split\}D\_\{b,t\}\\leq R\_\{G\}\\eta^\{2\}G\_\{t\}\+R\_\{B\}\\eta^\{2\}B\_\{f\}^\{2\}\+R\_\{\\zeta\}\\eta^\{2\}\\zeta\_\{f\}^\{2\}\+R\_\{0\}\\eta^\{4\}\+R\_\{A\}w^\{2\}A\_\{t\}^\{f\},\\\\ D\_\{\\xi,t\}\\leq\\Sigma\_\{0\}\\eta^\{2\}\.\\end\{split\}\(G25\)Consequently,

𝔼t​𝒱W​\(Zt\+1\)≤\(1−c0​η\)​𝒱W​\(Zt\)\+CG​η​Gt\+CB​η​Bf2\+Cζ​η​ζf2\+Cη​η2\+CA​η​Atf,\\begin\{split\}\\mathbb\{E\}\_\{t\}\\mathcal\{V\}\_\{W\}\(Z^\{t\+1\}\)\\leq\(1\-c\_\{0\}\\eta\)\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+C\_\{G\}\\eta G\_\{t\}\+C\_\{B\}\\eta B\_\{f\}^\{2\}\\\\ \+C\_\{\\zeta\}\\eta\\zeta\_\{f\}^\{2\}\+C\_\{\\eta\}\\eta^\{2\}\+C\_\{A\}\\eta A\_\{t\}^\{f\},\\end\{split\}\(G26\)wherec0=aW​α​τ/4=α​pmin​λ2/\(2​MW​\|E\|\)c\_\{0\}=a\_\{W\}\\alpha\\tau/4=\\alpha p\_\{\\min\}\\lambda\_\{2\}/\(2M\_\{W\}\|E\|\)\.

###### Proof\.

For an active directed endpoint\(i,j\)\(i,j\), define the three deterministic pieces of the perturbation in Equation[G18](https://arxiv.org/html/2609.01952#A7.E18)by

bi,etask\\displaystyle b\_\{i,e\}^\{\\rm task\}=−η⁡\(1−α\)​Gitask,\\displaystyle=\-\\eta\(1\-\\alpha\)G\_\{i\}^\{\\rm task\},bi,ecurv\\displaystyle b\_\{i,e\}^\{\\rm curv\}=riz,\\displaystyle=r\_\{i\}^\{z\},bi,elag\\displaystyle b\_\{i,e\}^\{\\rm lag\}=−w​Θi​\(PjEMA−Pj\),\\displaystyle=\-w\\Theta\_\{i\}\(P\_\{j\}^\{\\rm EMA\}\-P\_\{j\}\),bi,e\\displaystyle b\_\{i,e\}=bi,etask\+bi,ecurv\+bi,elag\.\\displaystyle=b\_\{i,e\}^\{\\rm task\}\+b\_\{i,e\}^\{\\rm curv\}\+b\_\{i,e\}^\{\\rm lag\}\.HereGitaskG\_\{i\}^\{\\rm task\}is the logit\-space private\-task pushforward defined in A4\. At first order it isJiz​\(Jip\)∗​gitJ\_\{i\}^\{z\}\(J\_\{i\}^\{p\}\)^\{\*\}g\_\{i\}^\{t\}plus the finite\-support fidelity residual\. This cross\-kernel map is distinct from the prediction kernelJip​\(Jip\)∗J\_\{i\}^\{p\}\(J\_\{i\}^\{p\}\)^\{\*\}used in Lemma[G8](https://arxiv.org/html/2609.01952#Thmlemma8)\. The exact A4 closure used here is that finite constantscz,G,cz,B,cz,ζc\_\{z,G\},c\_\{z,B\},c\_\{z,\\zeta\}satisfy

1N​∑i𝔼t​‖Gitask‖2≤cz,G​Gt\+cz,B​Bf2\+cz,ζ​ζf2\.\\frac\{1\}\{N\}\\sum\_\{i\}\\mathbb\{E\}\_\{t\}\\\|G\_\{i\}^\{\\rm task\}\\\|^\{2\}\\leq c\_\{z,G\}G\_\{t\}\+c\_\{z,B\}B\_\{f\}^\{2\}\+c\_\{z,\\zeta\}\\zeta\_\{f\}^\{2\}\.All three deterministic pieces are measurable conditional on\(ℱt,e\)\(\\mathcal\{F\}\_\{t\},e\)\.

The stochastic component is the centered pushforwardξi,e=ξiz−𝔼t\[ξiz∣ℱt,e\]\\xi\_\{i,e\}=\\xi\_\{i\}^\{z\}\-\\mathbb\{E\}\_\{t\}\[\\xi\_\{i\}^\{z\}\\mid\\mathcal\{F\}\_\{t\},e\]\. For any vectorsa,b,ca,b,cin a Hilbert space,

‖a\+b\+c‖2\\displaystyle\\\|a\+b\+c\\\|^\{2\}=‖a‖2\+‖b‖2\+‖c‖2\+2​⟨a,b⟩\+2​⟨a,c⟩\+2​⟨b,c⟩\\displaystyle=\\\|a\\\|^\{2\}\+\\\|b\\\|^\{2\}\+\\\|c\\\|^\{2\}\+2\\langle a,b\\rangle\+2\\langle a,c\\rangle\+2\\langle b,c\\rangle≤3​‖a‖2\+3​‖b‖2\+3​‖c‖2,\\displaystyle\\leq 3\\\|a\\\|^\{2\}\+3\\\|b\\\|^\{2\}\+3\\\|c\\\|^\{2\},where each cross term uses2​⟨u,v⟩≤‖u‖2\+‖v‖22\\langle u,v\\rangle\\leq\\\|u\\\|^\{2\}\+\\\|v\\\|^\{2\}\. Apply this inequality in theWiW\_\{i\}norm to the three perturbation pieces\. The task piece satisfies

‖bi,etask‖Wi2\\displaystyle\\\|b\_\{i,e\}^\{\\rm task\}\\\|\_\{W\_\{i\}\}^\{2\}=η2​\(1−α\)2​‖Gitask‖Wi2\\displaystyle=\\eta^\{2\}\(1\-\\alpha\)^\{2\}\\\|G\_\{i\}^\{\\rm task\}\\\|\_\{W\_\{i\}\}^\{2\}≤MW​η2​\(1−α\)2​‖Gitask‖2\.\\displaystyle\\leq M\_\{W\}\\eta^\{2\}\(1\-\\alpha\)^\{2\}\\\|G\_\{i\}^\{\\rm task\}\\\|^\{2\}\.For nonnegative endpoint quantitiesuiu\_\{i\}, uniform edge sampling gives

1N​𝔼e​∑i∈eui=1N​∑idi\|E\|​ui≤1N​∑iui\.\\frac\{1\}\{N\}\\mathbb\{E\}\_\{e\}\\sum\_\{i\\in e\}u\_\{i\}=\\frac\{1\}\{N\}\\sum\_\{i\}\\frac\{d\_\{i\}\}\{\|E\|\}u\_\{i\}\\leq\\frac\{1\}\{N\}\\sum\_\{i\}u\_\{i\}\.Apply this activation identity and the A4 task\-pushforward closure\. Before the outer factor33, the task contribution obeys

1N​∑i𝔼t​‖bi,etask‖Wi2\\displaystyle\\frac\{1\}\{N\}\\sum\_\{i\}\\mathbb\{E\}\_\{t\}\\\|b\_\{i,e\}^\{\\rm task\}\\\|\_\{W\_\{i\}\}^\{2\}≤MW​\(1−α\)2​η2​\(cz,G​Gt\+cz,B​Bf2CLOSE\\displaystyle\\leq M\_\{W\}\(1\-\\alpha\)^\{2\}\\eta^\{2\}\\left\(c\_\{z,G\}G\_\{t\}\+c\_\{z,B\}B\_\{f\}^\{2\}\\right\.\(G27\)OPEN\+cz,ζ​ζf2\)\.\\displaystyle\\quad\\left\.\+c\_\{z,\\zeta\}\\zeta\_\{f\}^\{2\}\\right\)\.\(G28\)
The outer three\-piece inequality multiplies these constants by33\. One valid set of constants for the finalDb,tD\_\{b,t\}bound is

RG\\displaystyle R\_\{G\}=3​MW​\(1−α\)2​cz,G,\\displaystyle=3M\_\{W\}\(1\-\\alpha\)^\{2\}c\_\{z,G\},\(G29\)RB\\displaystyle R\_\{B\}=3​MW​\(1−α\)2​cz,B,\\displaystyle=3M\_\{W\}\(1\-\\alpha\)^\{2\}c\_\{z,B\},\(G30\)Rζ\\displaystyle R\_\{\\zeta\}=3​MW​\(1−α\)2​cz,ζ\.\\displaystyle=3M\_\{W\}\(1\-\\alpha\)^\{2\}c\_\{z,\\zeta\}\.\(G31\)
The integral remainder bound from Lemma[G1](https://arxiv.org/html/2609.01952#Thmlemma1a)and the fourth\-moment assumption give the explicit calculation

𝔼t​‖bi,ecurv‖Wi2\\displaystyle\\mathbb\{E\}\_\{t\}\\\|b\_\{i,e\}^\{\\rm curv\}\\\|\_\{W\_\{i\}\}^\{2\}≤MW​𝔼t​‖riz‖2\\displaystyle\\leq M\_\{W\}\\mathbb\{E\}\_\{t\}\\\|r\_\{i\}^\{z\}\\\|^\{2\}\(G32\)≤MW​LΘ24​𝔼t​‖Δ​θi‖4\\displaystyle\\leq\\frac\{M\_\{W\}L\_\{\\Theta\}^\{2\}\}\{4\}\\mathbb\{E\}\_\{t\}\\\|\\Delta\\theta\_\{i\}\\\|^\{4\}\(G33\)≤MW​LΘ2​Gθ,444​η4\.\\displaystyle\\leq\\frac\{M\_\{W\}L\_\{\\Theta\}^\{2\}G\_\{\\theta,4\}^\{4\}\}\{4\}\\eta^\{4\}\.\(G34\)
The second line squares‖riz‖≤\(LΘ/2\)​‖Δ​θi‖2\\\|r\_\{i\}^\{z\}\\\|\\leq\(L\_\{\\Theta\}/2\)\\\|\\Delta\\theta\_\{i\}\\\|^\{2\}, and the third uses the A4 conditional fourth\-moment bound\. Thus one may takeR0=3​MW​LΘ2​Gθ,44/4R\_\{0\}=3M\_\{W\}L\_\{\\Theta\}^\{2\}G\_\{\\theta,4\}^\{4\}/4after the outer three\-piece inequality\.

For the teacher lag,Θi​Wi​Θi=Θi\\Theta\_\{i\}W\_\{i\}\\Theta\_\{i\}=\\Theta\_\{i\}andΘi⪯κ\+​Πiz\\Theta\_\{i\}\\preceq\\kappa\_\{\+\}\\Pi\_\{i\}^\{z\}imply

‖bi,elag‖Wi2\\displaystyle\\\|b\_\{i,e\}^\{\\rm lag\}\\\|\_\{W\_\{i\}\}^\{2\}=w2​\(PjEMA−Pj\)⊤​Θi​Wi​Θi​\(PjEMA−Pj\)\\displaystyle=w^\{2\}\(P\_\{j\}^\{\\rm EMA\}\-P\_\{j\}\)^\{\\top\}\\Theta\_\{i\}W\_\{i\}\\Theta\_\{i\}\(P\_\{j\}^\{\\rm EMA\}\-P\_\{j\}\)\(G35\)=w2​\(PjEMA−Pj\)⊤​Θi​\(PjEMA−Pj\)\\displaystyle=w^\{2\}\(P\_\{j\}^\{\\rm EMA\}\-P\_\{j\}\)^\{\\top\}\\Theta\_\{i\}\(P\_\{j\}^\{\\rm EMA\}\-P\_\{j\}\)\(G36\)≤w2​κ\+​‖PjEMA−Pj‖2\.\\displaystyle\\leq w^\{2\}\\kappa\_\{\+\}\\\|P\_\{j\}^\{\\rm EMA\}\-P\_\{j\}\\\|^\{2\}\.\(G37\)
The directed\-edge average obeys

1N​\|E\|​∑\(i,j\)∈E→‖PjEMA−Pj‖2\\displaystyle\\frac\{1\}\{N\|E\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}\\\|P\_\{j\}^\{\\rm EMA\}\-P\_\{j\}\\\|^\{2\}=1N​\|E\|​∑jdj​‖PjEMA−Pj‖2\\displaystyle=\\frac\{1\}\{N\|E\|\}\\sum\_\{j\}d\_\{j\}\\\|P\_\{j\}^\{\\rm EMA\}\-P\_\{j\}\\\|^\{2\}\(G38\)≤dmaxN​\|E\|​∑j‖PjEMA−Pj‖2\\displaystyle\\leq\\frac\{d\_\{\\max\}\}\{N\|E\|\}\\sum\_\{j\}\\\|P\_\{j\}^\{\\rm EMA\}\-P\_\{j\}\\\|^\{2\}\(G39\)≤dmax\|E\|​Atf\.\\displaystyle\\leq\\frac\{d\_\{\\max\}\}\{\|E\|\}A\_\{t\}^\{f\}\.\(G40\)
WithRA=3​κ\+​dmax/\|E\|R\_\{A\}=3\\kappa\_\{\+\}d\_\{\\max\}/\|E\|, the lag contribution is bounded byRA​w2​AtfR\_\{A\}w^\{2\}A\_\{t\}^\{f\}\. Finally, the assumed pushforward variance bound and conditional centering give

𝔼t​‖ξi,e‖2\\displaystyle\\mathbb\{E\}\_\{t\}\\\|\\xi\_\{i,e\}\\\|^\{2\}=𝔼t‖ξiz−𝔼t\[ξiz∣ℱt,e\]‖2\\displaystyle=\\mathbb\{E\}\_\{t\}\\left\\\|\\xi\_\{i\}^\{z\}\-\\mathbb\{E\}\_\{t\}\[\\xi\_\{i\}^\{z\}\\mid\\mathcal\{F\}\_\{t\},e\]\\right\\\|^\{2\}\(G41\)=𝔼t∥ξiz∥2−‖𝔼t\[ξiz∣ℱt,e\]‖2\\displaystyle=\\mathbb\{E\}\_\{t\}\\\|\\xi\_\{i\}^\{z\}\\\|^\{2\}\-\\left\\\|\\mathbb\{E\}\_\{t\}\[\\xi\_\{i\}^\{z\}\\mid\\mathcal\{F\}\_\{t\},e\]\\right\\\|^\{2\}\(G42\)≤σz2​η2\.\\displaystyle\\leq\\sigma\_\{z\}^\{2\}\\eta^\{2\}\.\(G43\)
Since‖u‖Wi2≤MW​‖u‖2\\\|u\\\|\_\{W\_\{i\}\}^\{2\}\\leq M\_\{W\}\\\|u\\\|^\{2\}and every activation probability is at most one,Dξ,t≤MW​σz2​η2=:Σ0​η2D\_\{\\xi,t\}\\leq M\_\{W\}\\sigma\_\{z\}^\{2\}\\eta^\{2\}=:\\Sigma\_\{0\}\\eta^\{2\}\. Thus the three\-piece inequality is

Db,t≤RG​η2​Gt\+RB​η2​Bf2\+Rζ​η2​ζf2\+R0​η4\+RA​w2​Atf\.D\_\{b,t\}\\leq R\_\{G\}\\eta^\{2\}G\_\{t\}\+R\_\{B\}\\eta^\{2\}B\_\{f\}^\{2\}\+R\_\{\\zeta\}\\eta^\{2\}\\zeta\_\{f\}^\{2\}\+R\_\{0\}\\eta^\{4\}\+R\_\{A\}w^\{2\}A\_\{t\}^\{f\}\.
Insert this inequality andDξ,t≤Σ0​η2D\_\{\\xi,t\}\\leq\\Sigma\_\{0\}\\eta^\{2\}into Equation[G21](https://arxiv.org/html/2609.01952#A7.E21)\. Define

H1:=8​dmaxaW​α​τ​\|E\|\.H\_\{1\}:=\\frac\{8d\_\{\\max\}\}\{a\_\{W\}\\alpha\\tau\|E\|\}\.Sincew=η​α​τw=\\eta\\alpha\\tau,

8​dmaxaW​w​\|E\|\+2=H1η\+2\.\\frac\{8d\_\{\\max\}\}\{a\_\{W\}w\|E\|\}\+2=\\frac\{H\_\{1\}\}\{\\eta\}\+2\.For the gradient contribution,

\(H1η\+2\)​RG​η2​Gt\\displaystyle\\left\(\\frac\{H\_\{1\}\}\{\\eta\}\+2\\right\)R\_\{G\}\\eta^\{2\}G\_\{t\}=RG​\(H1\+2​η\)​η​Gt\\displaystyle=R\_\{G\}\(H\_\{1\}\+2\\eta\)\\eta G\_\{t\}≤RG​\(H1\+2​ηmax\)​η​Gt\.\\displaystyle\\leq R\_\{G\}\(H\_\{1\}\+2\\eta\_\{\\max\}\)\\eta G\_\{t\}\.The same calculation applies to theBf2B\_\{f\}^\{2\}andζf2\\zeta\_\{f\}^\{2\}terms\. The capacity term already has the required scale becauseCB,W​w​Bf2=CB,W​α​τ​η​Bf2C\_\{B,W\}wB\_\{f\}^\{2\}=C\_\{B,W\}\\alpha\\tau\\,\\eta B\_\{f\}^\{2\}\.

For the curvature term, takingηmax≤1\\eta\_\{\\max\}\\leq 1gives

\(H1η\+2\)​R0​η4\\displaystyle\\left\(\\frac\{H\_\{1\}\}\{\\eta\}\+2\\right\)R\_\{0\}\\eta^\{4\}=R0​\(H1​η3\+2​η4\)\\displaystyle=R\_\{0\}\(H\_\{1\}\\eta^\{3\}\+2\\eta^\{4\}\)≤R0​\(H1\+2\)​η2\.\\displaystyle\\leq R\_\{0\}\(H\_\{1\}\+2\)\\eta^\{2\}\.For the lag term,w≤1w\\leq 1andw=η​α​τw=\\eta\\alpha\\taugive

\(H1η\+2\)​RA​w2​Atf\\displaystyle\\left\(\\frac\{H\_\{1\}\}\{\\eta\}\+2\\right\)R\_\{A\}w^\{2\}A\_\{t\}^\{f\}=RA​\(H1​η​α2​τ2\+2​η2​α2​τ2\)​Atf\\displaystyle=R\_\{A\}\\left\(H\_\{1\}\\eta\\alpha^\{2\}\\tau^\{2\}\+2\\eta^\{2\}\\alpha^\{2\}\\tau^\{2\}\\right\)A\_\{t\}^\{f\}≤RA​α2​τ2​\(H1\+2​ηmax\)​η​Atf\.\\displaystyle\\leq R\_\{A\}\\alpha^\{2\}\\tau^\{2\}\(H\_\{1\}\+2\\eta\_\{\\max\}\)\\eta A\_\{t\}^\{f\}\.Finally,Dξ,tD\_\{\\xi,t\}contributesΣ0​η2\\Sigma\_\{0\}\\eta^\{2\}\. Absorb the displayed finite coefficients intoCG,CB,Cζ,Cη,CAC\_\{G\},C\_\{B\},C\_\{\\zeta\},C\_\{\\eta\},C\_\{A\}\. The contraction term is

aW​w4=aW​α​τ4​η=c0​η\.\\frac\{a\_\{W\}w\}\{4\}=\\frac\{a\_\{W\}\\alpha\\tau\}\{4\}\\eta=c\_\{0\}\\eta\.These bounds give Equation[G26](https://arxiv.org/html/2609.01952#A7.E26)\. ∎

###### Lemma G8\(Activation\-corrected task descent\)\.

Letqi=∑e∋iPr⁡\(et=e\)=di/\|E\|q\_\{i\}=\\sum\_\{e\\ni i\}\\Pr\(e\_\{t\}=e\)=d\_\{i\}/\|E\|and

ℛt=1N​∑iqi−1​\(Fi​\(pit\)−Fi⋆\)\.\\mathcal\{R\}\_\{t\}=\\frac\{1\}\{N\}\\sum\_\{i\}q\_\{i\}^\{\-1\}\\big\(F\_\{i\}\(p\_\{i\}^\{t\}\)\-F\_\{i\}^\{\\star\}\\big\)\.\(G44\)Under A2, A4, A6, and the task clause of A10, there are constantsaF,bF,DB,Dζ,Dη,DA\>0a\_\{F\},b\_\{F\},D\_\{B\},D\_\{\\zeta\},D\_\{\\eta\},D\_\{A\}\>0such that

𝔼t​ℛt\+1≤ℛt−aF​η​Gt\+bF​η​𝒱W​\(Zt\)\+DB​η​Bf2\+Dζ​η​ζf2\+Dη​η2\+DA​η​Atf\.\\begin\{split\}\\mathbb\{E\}\_\{t\}\\mathcal\{R\}\_\{t\+1\}\\leq\\mathcal\{R\}\_\{t\}\-a\_\{F\}\\eta G\_\{t\}\+b\_\{F\}\\eta\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+D\_\{B\}\\eta B\_\{f\}^\{2\}\+D\_\{\\zeta\}\\eta\\zeta\_\{f\}^\{2\}\\\\ \+D\_\{\\eta\}\\eta^\{2\}\+D\_\{A\}\\eta A\_\{t\}^\{f\}\.\\end\{split\}\(G45\)‖∇fF​\(p¯t\)‖2≤2​Gt\+2​Lf2​ψtf\.\\\|\\nabla\_\{f\}F\(\\bar\{p\}^\{t\}\)\\\|^\{2\}\\leq 2G\_\{t\}\+2L\_\{f\}^\{2\}\\psi\_\{t\}^\{f\}\.\(G46\)

###### Proof\.

Let⟨⋅,⋅⟩μ\\langle\\cdot,\\cdot\\rangle\_\{\\mu\}and∥⋅∥μ\\\|\\cdot\\\|\_\{\\mu\}denote the prediction Hilbert\-space inner product and norm\. Setgit=∇fFi​\(pit\)g\_\{i\}^\{t\}=\\nabla\_\{f\}F\_\{i\}\(p\_\{i\}^\{t\}\)\. For an active endpointii, A4 decomposes the prediction update as

Δ​pi\\displaystyle\\Delta p\_\{i\}=Jip​Δ​θi\+rip\+ξip\\displaystyle=J\_\{i\}^\{p\}\\Delta\\theta\_\{i\}\+r\_\{i\}^\{p\}\+\\xi\_\{i\}^\{p\}=−η⁡\(1−α\)​\(Ki​git\+δit\)−η​α​Hi​\(Pi−PjT\)\+rip\+ξip\.\\displaystyle=\-\\eta\(1\-\\alpha\)\(K\_\{i\}g\_\{i\}^\{t\}\+\\delta\_\{i\}^\{t\}\)\-\\eta\\alpha H\_\{i\}\(P\_\{i\}\-P\_\{j\}^\{T\}\)\+r\_\{i\}^\{p\}\+\\xi\_\{i\}^\{p\}\.whereKi=Jip​\(Jip\)∗K\_\{i\}=J\_\{i\}^\{p\}\(J\_\{i\}^\{p\}\)^\{\*\},Hi=τ​Jip​\(Jiz\)∗H\_\{i\}=\\tau J\_\{i\}^\{p\}\(J\_\{i\}^\{z\}\)^\{\*\}, and‖Hi‖op≤Hmax\\\|H\_\{i\}\\\|\_\{\\rm op\}\\leq H\_\{\\max\}by A4\. The remainder and noise satisfy𝔼t​‖rip‖2≤Rp​η4\\mathbb\{E\}\_\{t\}\\\|r\_\{i\}^\{p\}\\\|^\{2\}\\leq R\_\{p\}\\eta^\{4\}and𝔼t​‖ξip‖2≤Σp​η2\\mathbb\{E\}\_\{t\}\\\|\\xi\_\{i\}^\{p\}\\\|^\{2\}\\leq\\Sigma\_\{p\}\\eta^\{2\}\. FunctionalLfL\_\{f\}\-smoothness means, for everyu,vu,v,

Fi​\(v\)≤Fi​\(u\)\+⟨∇fFi​\(u\),v−u⟩\+Lf2​‖v−u‖μ2\.F\_\{i\}\(v\)\\leq F\_\{i\}\(u\)\+\\langle\\nabla\_\{f\}F\_\{i\}\(u\),v\-u\\rangle\+\\frac\{L\_\{f\}\}\{2\}\\\|v\-u\\\|\_\{\\mu\}^\{2\}\.Takingu=pitu=p\_\{i\}^\{t\},v=pit\+Δ​piv=p\_\{i\}^\{t\}\+\\Delta p\_\{i\}gives the one\-event inequality

Fi​\(pit\+1\)−Fi​\(pit\)\\displaystyle F\_\{i\}\(p\_\{i\}^\{t\+1\}\)\-F\_\{i\}\(p\_\{i\}^\{t\}\)≤⟨git,Δ​pi⟩μ\+Lf2​‖Δ​pi‖μ2\.\\displaystyle\\leq\\langle g\_\{i\}^\{t\},\\Delta p\_\{i\}\\rangle\_\{\\mu\}\+\\frac\{L\_\{f\}\}\{2\}\\\|\\Delta p\_\{i\}\\\|\_\{\\mu\}^\{2\}\.
The task inner product is

−η⁡\(1−α\)​⟨git,Ki​git⟩μ−η⁡\(1−α\)​⟨git,δit⟩μ\.\-\\eta\(1\-\\alpha\)\\langle g\_\{i\}^\{t\},K\_\{i\}g\_\{i\}^\{t\}\\rangle\_\{\\mu\}\-\\eta\(1\-\\alpha\)\\langle g\_\{i\}^\{t\},\\delta\_\{i\}^\{t\}\\rangle\_\{\\mu\}\.BecauseKi=Πip​Ki​ΠipK\_\{i\}=\\Pi\_\{i\}^\{p\}K\_\{i\}\\Pi\_\{i\}^\{p\}andKi⪰κ¯p​ΠipK\_\{i\}\\succeq\\underline\{\\kappa\}\_\{p\}\\Pi\_\{i\}^\{p\},

⟨git,Ki​git⟩μ\\displaystyle\\langle g\_\{i\}^\{t\},K\_\{i\}g\_\{i\}^\{t\}\\rangle\_\{\\mu\}=⟨Πip​git,Ki​Πip​git⟩μ\\displaystyle=\\langle\\Pi\_\{i\}^\{p\}g\_\{i\}^\{t\},K\_\{i\}\\Pi\_\{i\}^\{p\}g\_\{i\}^\{t\}\\rangle\_\{\\mu\}≥κ¯p​‖Πip​git‖μ2\\displaystyle\\geq\\underline\{\\kappa\}\_\{p\}\\\|\\Pi\_\{i\}^\{p\}g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}=κ¯p​\(‖git‖μ2−‖\(I−Πip\)​git‖μ2\)\.\\displaystyle=\\underline\{\\kappa\}\_\{p\}\\left\(\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\-\\\|\(I\-\\Pi\_\{i\}^\{p\}\)g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\\right\)\.The last equality is the orthogonal decomposition

‖git‖μ2\\displaystyle\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}=‖Πip​git\+\(I−Πip\)​git‖μ2\\displaystyle=\\\|\\Pi\_\{i\}^\{p\}g\_\{i\}^\{t\}\+\(I\-\\Pi\_\{i\}^\{p\}\)g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}=‖Πip​git‖μ2\+2​⟨Πip​git,\(I−Πip\)​git⟩μ\\displaystyle=\\\|\\Pi\_\{i\}^\{p\}g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\+2\\langle\\Pi\_\{i\}^\{p\}g\_\{i\}^\{t\},\(I\-\\Pi\_\{i\}^\{p\}\)g\_\{i\}^\{t\}\\rangle\_\{\\mu\}\+‖\(I−Πip\)​git‖μ2\\displaystyle\+\\\|\(I\-\\Pi\_\{i\}^\{p\}\)g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}=‖Πip​git‖μ2\+‖\(I−Πip\)​git‖μ2,\\displaystyle=\\\|\\Pi\_\{i\}^\{p\}g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\+\\\|\(I\-\\Pi\_\{i\}^\{p\}\)g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\},where the cross term is zero because the two projector ranges are orthogonal\.

For the residual, Young’s inequality withεδ\>0\\varepsilon\_\{\\delta\}\>0gives

2​\|⟨git,δit⟩μ\|\\displaystyle 2\|\\langle g\_\{i\}^\{t\},\\delta\_\{i\}^\{t\}\\rangle\_\{\\mu\}\|≤εδ​‖git‖μ2\+εδ−1​‖δit‖μ2\.\\displaystyle\\leq\\varepsilon\_\{\\delta\}\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\+\\varepsilon\_\{\\delta\}^\{\-1\}\\\|\\delta\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\.The inequality is obtained by expanding∥εδgit∓εδ−1/2δit∥μ2≥0\\\|\\sqrt\{\\varepsilon\_\{\\delta\}\}g\_\{i\}^\{t\}\\mp\\varepsilon\_\{\\delta\}^\{\-1/2\}\\delta\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\\geq 0and choosing the sign that matches the inner product\. Define the task decrease before multiplying byη\\etaas

𝔇task:=1−αN​∑i\(⟨git,Ki​git⟩μ\+⟨git,δit⟩μ\)\.\\mathfrak\{D\}\_\{\\rm task\}:=\\frac\{1\-\\alpha\}\{N\}\\sum\_\{i\}\\left\(\\langle g\_\{i\}^\{t\},K\_\{i\}g\_\{i\}^\{t\}\\rangle\_\{\\mu\}\+\\langle g\_\{i\}^\{t\},\\delta\_\{i\}^\{t\}\\rangle\_\{\\mu\}\\right\)\.The preceding projection and Young bounds give

𝔇task≥\(1−α\)​\[κ¯p​Gt−κ¯pN​∑i‖\(I−Πip\)​git‖μ2−εδ2​Gt−12​εδ​N​∑i‖δit‖μ2\]\.\\displaystyle\\mathfrak\{D\}\_\{\\rm task\}\\geq\(1\-\\alpha\)\\left\[\\underline\{\\kappa\}\_\{p\}G\_\{t\}\-\\frac\{\\underline\{\\kappa\}\_\{p\}\}\{N\}\\sum\_\{i\}\\\|\(I\-\\Pi\_\{i\}^\{p\}\)g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\-\\frac\{\\varepsilon\_\{\\delta\}\}\{2\}G\_\{t\}\-\\frac\{1\}\{2\\varepsilon\_\{\\delta\}N\}\\sum\_\{i\}\\\|\\delta\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\\right\]\.\(G47\)
Insert the A10 task\-normal bound and the A4 fidelity boundN−1​∑i‖δit‖μ2≤χG​Gt\+χB​Bf2\+χζ​ζf2N^\{\-1\}\\sum\_\{i\}\\\|\\delta\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\\leq\\chi\_\{G\}G\_\{t\}\+\\chi\_\{B\}B\_\{f\}^\{2\}\+\\chi\_\{\\zeta\}\\zeta\_\{f\}^\{2\}\. The result is

𝔇task≥\(1−α\)​\[κ¯p​\(1−γtask\)−εδ2−χG2​εδ\]​Gt\\displaystyle\\mathfrak\{D\}\_\{\\rm task\}\\geq\(1\-\\alpha\)\\left\[\\underline\{\\kappa\}\_\{p\}\(1\-\\gamma\_\{\\rm task\}\)\-\\frac\{\\varepsilon\_\{\\delta\}\}\{2\}\-\\frac\{\\chi\_\{G\}\}\{2\\varepsilon\_\{\\delta\}\}\\right\]G\_\{t\}−\(1−α\)​κ¯p​νtask,Ψ​ψtf\\displaystyle\-\(1\-\\alpha\)\\underline\{\\kappa\}\_\{p\}\\nu\_\{\\rm task,\\Psi\}\\psi\_\{t\}^\{f\}−\(1−α\)​\[κ¯p​νtask,B\+χB2​εδ\]​Bf2−\(1−α\)​χζ2​εδ​ζf2\.\\displaystyle\-\(1\-\\alpha\)\\left\[\\underline\{\\kappa\}\_\{p\}\\nu\_\{\\rm task,B\}\+\\frac\{\\chi\_\{B\}\}\{2\\varepsilon\_\{\\delta\}\}\\right\]B\_\{f\}^\{2\}\-\(1\-\\alpha\)\\frac\{\\chi\_\{\\zeta\}\}\{2\\varepsilon\_\{\\delta\}\}\\zeta\_\{f\}^\{2\}\.The task\-margin requirement choosesεδ\>0\\varepsilon\_\{\\delta\}\>0so the coefficient ofGtG\_\{t\}in this display is positive\. Sinceψtf≤Cf​z​ψtz≤\(Cf​z/mW\)​𝒱W​\(Zt\)\\psi\_\{t\}^\{f\}\\leq C\_\{fz\}\\psi\_\{t\}^\{z\}\\leq\(C\_\{fz\}/m\_\{W\}\)\\mathcal\{V\}\_\{W\}\(Z^\{t\}\), its negative term becomes a finite positive coefficient multiplying𝒱W​\(Zt\)\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)in the upper descent bound\.

For the KD term, Young’s inequality withεKD\>0\\varepsilon\_\{\\rm KD\}\>0gives

2​\|⟨git,Hi​\(Pi−PjT\)⟩μ\|\\displaystyle 2\|\\langle g\_\{i\}^\{t\},H\_\{i\}\(P\_\{i\}\-P\_\{j\}^\{T\}\)\\rangle\_\{\\mu\}\|≤εKD​‖git‖μ2\+εKD−1​Hmax2​‖Pi−PjT‖μ2\.\\displaystyle\\leq\\varepsilon\_\{\\rm KD\}\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\+\\varepsilon\_\{\\rm KD\}^\{\-1\}H\_\{\\max\}^\{2\}\\\|P\_\{i\}\-P\_\{j\}^\{T\}\\\|\_\{\\mu\}^\{2\}\.The teacher decomposition isPi−PjT=\(Pi−Pj\)\+\(Pj−PjEMA\)P\_\{i\}\-P\_\{j\}^\{T\}=\(P\_\{i\}\-P\_\{j\}\)\+\(P\_\{j\}\-P\_\{j\}^\{\\rm EMA\}\)\. Hence

‖Pi−PjT‖μ2\\displaystyle\\\|P\_\{i\}\-P\_\{j\}^\{T\}\\\|\_\{\\mu\}^\{2\}=‖Pi−Pj‖μ2\+2​⟨Pi−Pj,Pj−PjEMA⟩μ\\displaystyle=\\\|P\_\{i\}\-P\_\{j\}\\\|\_\{\\mu\}^\{2\}\+2\\langle P\_\{i\}\-P\_\{j\},P\_\{j\}\-P\_\{j\}^\{\\rm EMA\}\\rangle\_\{\\mu\}\+‖Pj−PjEMA‖μ2\\displaystyle\+\\\|P\_\{j\}\-P\_\{j\}^\{\\rm EMA\}\\\|\_\{\\mu\}^\{2\}≤2​‖Pi−Pj‖μ2\+2​‖Pj−PjEMA‖μ2\.\\displaystyle\\leq 2\\\|P\_\{i\}\-P\_\{j\}\\\|\_\{\\mu\}^\{2\}\+2\\\|P\_\{j\}\-P\_\{j\}^\{\\rm EMA\}\\\|\_\{\\mu\}^\{2\}\.The final line uses2​⟨u,v⟩μ≤‖u‖μ2\+‖v‖μ22\\langle u,v\\rangle\_\{\\mu\}\\leq\\\|u\\\|\_\{\\mu\}^\{2\}\+\\\|v\\\|\_\{\\mu\}^\{2\}\. Letqmin=mini⁡qiq\_\{\\min\}=\\min\_\{i\}q\_\{i\}\. The graph upper inequality and Lemma[G3](https://arxiv.org/html/2609.01952#Thmlemma3a)give

1\|E\|​∑\{i,j\}∈E‖Pi−Pj‖μ2\\displaystyle\\frac\{1\}\{\|E\|\}\\sum\_\{\\\{i,j\\\}\\in E\}\\\|P\_\{i\}\-P\_\{j\}\\\|\_\{\\mu\}^\{2\}≤N​λmax\|E\|​ψtf\\displaystyle\\leq\\frac\{N\\lambda\_\{\\max\}\}\{\|E\|\}\\psi\_\{t\}^\{f\}≤N​λmax​Cf​z\|E\|​ψtz\\displaystyle\\leq\\frac\{N\\lambda\_\{\\max\}C\_\{fz\}\}\{\|E\|\}\\psi\_\{t\}^\{z\}≤N​λmax​Cf​z\|E\|​mW​𝒱W​\(Zt\)\.\\displaystyle\\leq\\frac\{N\\lambda\_\{\\max\}C\_\{fz\}\}\{\|E\|m\_\{W\}\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\.The factorNNappears because∑i‖Pi−P¯‖μ2=N​ψtf\\sum\_\{i\}\\\|P\_\{i\}\-\\bar\{P\}\\\|\_\{\\mu\}^\{2\}=N\\psi\_\{t\}^\{f\}\. In the activation\-corrected potential, the directed endpoint average is

1N​\|E\|​∑\(i,j\)∈E→qi−1​‖Pi−Pj‖μ2\\displaystyle\\frac\{1\}\{N\|E\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}q\_\{i\}^\{\-1\}\\\|P\_\{i\}\-P\_\{j\}\\\|\_\{\\mu\}^\{2\}≤2N​qmin​\|E\|​∑\{i,j\}∈E‖Pi−Pj‖μ2\\displaystyle\\leq\\frac\{2\}\{Nq\_\{\\min\}\|E\|\}\\sum\_\{\\\{i,j\\\}\\in E\}\\\|P\_\{i\}\-P\_\{j\}\\\|\_\{\\mu\}^\{2\}≤2​λmax​Cf​zqmin​\|E\|​mW​𝒱W​\(Zt\)\.\\displaystyle\\leq\\frac\{2\\lambda\_\{\\max\}C\_\{fz\}\}\{q\_\{\\min\}\|E\|m\_\{W\}\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\.Similarly,

1N​\|E\|​∑\(i,j\)∈E→qi−1​‖Pj−PjEMA‖μ2≤dmaxqmin​\|E\|​Atf\.\\frac\{1\}\{N\|E\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}q\_\{i\}^\{\-1\}\\\|P\_\{j\}\-P\_\{j\}^\{\\rm EMA\}\\\|\_\{\\mu\}^\{2\}\\leq\\frac\{d\_\{\\max\}\}\{q\_\{\\min\}\|E\|\}A\_\{t\}^\{f\}\.The constants may therefore be chosen as

CKD,V:=2​Hmax2​λmax​Cf​zεKD​qmin​\|E\|​mW,CKD,A:=2​Hmax2​dmaxεKD​qmin​\|E\|\.C\_\{\\rm KD,V\}:=\\frac\{2H\_\{\\max\}^\{2\}\\lambda\_\{\\max\}C\_\{fz\}\}\{\\varepsilon\_\{\\rm KD\}q\_\{\\min\}\|E\|m\_\{W\}\},\\qquad C\_\{\\rm KD,A\}:=\\frac\{2H\_\{\\max\}^\{2\}d\_\{\\max\}\}\{\\varepsilon\_\{\\rm KD\}q\_\{\\min\}\|E\|\}\.The first KD square contributesCKD,V​𝒱W​\(Zt\)C\_\{\\rm KD,V\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)and the teacher\-lag square contributesCKD,A​AtfC\_\{\\rm KD,A\}A\_\{t\}^\{f\}\. The factor22inCKD,AC\_\{\\rm KD,A\}is a harmless enlargement\. Define

d0\\displaystyle d\_\{0\}:=\(1−α\)​\[κ¯p​\(1−γtask\)−εδ2−χG2​εδ\],\\displaystyle:=\(1\-\\alpha\)\\left\[\\underline\{\\kappa\}\_\{p\}\(1\-\\gamma\_\{\\rm task\}\)\-\\frac\{\\varepsilon\_\{\\delta\}\}\{2\}\-\\frac\{\\chi\_\{G\}\}\{2\\varepsilon\_\{\\delta\}\}\\right\],dtask\\displaystyle d\_\{\\rm task\}:=d0−α​εKD2\.\\displaystyle:=d\_\{0\}\-\\frac\{\\alpha\\varepsilon\_\{\\rm KD\}\}\{2\}\.The first line is the task coefficient obtained above\. The factorα​εKD/2\\alpha\\varepsilon\_\{\\rm KD\}/2is the gradient square in the KD Young inequality after multiplying by the update weightη​α\\eta\\alpha\. ChooseεKD\>0\\varepsilon\_\{\\rm KD\}\>0so thatdtask\>0d\_\{\\rm task\}\>0\. The remaining KD squares then contribute

η​α​CKD,V​𝒱W​\(Zt\)\+η​α​CKD,A​Atf,\\eta\\alpha C\_\{\\rm KD,V\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+\\eta\\alpha C\_\{\\rm KD,A\}A\_\{t\}^\{f\},which are included inbF​η​𝒱W​\(Zt\)b\_\{F\}\\eta\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)andDA​η​AtfD\_\{A\}\\eta A\_\{t\}^\{f\}, respectively\.

The activation correction is exact\. If edgeeeis sampled uniformly, thenPr⁡\(i∈e\)=qi=di/\|E\|\\Pr\(i\\in e\)=q\_\{i\}=d\_\{i\}/\|E\|\. Consequently

𝔼e​\[1N​∑i∈eqi−1​hi\]\\displaystyle\\mathbb\{E\}\_\{e\}\\left\[\\frac\{1\}\{N\}\\sum\_\{i\\in e\}q\_\{i\}^\{\-1\}h\_\{i\}\\right\]=1N∑ePr\(e\)∑i=1N𝟏\{i∈e\}qi−1hi\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{e\}\\Pr\(e\)\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\\\{i\\in e\\\}q\_\{i\}^\{\-1\}h\_\{i\}=1N∑iqi−1hi∑ePr\(e\)𝟏\{i∈e\}\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i\}q\_\{i\}^\{\-1\}h\_\{i\}\\sum\_\{e\}\\Pr\(e\)\\mathbf\{1\}\\\{i\\in e\\\}=1N​∑iqi−1​Pr⁡\(i∈e\)​hi\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i\}q\_\{i\}^\{\-1\}\\Pr\(i\\in e\)h\_\{i\}=1N​∑ihi\.\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i\}h\_\{i\}\.For the smoothness square, denote the task and KD components by

uitask:=−η⁡\(1−α\)​\(Ki​git\+δit\),uiKD:=−η​α​Hi​\(Pi−PjT\)\.u\_\{i\}^\{\\rm task\}:=\-\\eta\(1\-\\alpha\)\(K\_\{i\}g\_\{i\}^\{t\}\+\\delta\_\{i\}^\{t\}\),\\qquad u\_\{i\}^\{\\rm KD\}:=\-\\eta\\alpha H\_\{i\}\(P\_\{i\}\-P\_\{j\}^\{T\}\)\.Cauchy–Schwarz inℝ4\\mathbb\{R\}^\{4\}gives

‖Δ​pi‖μ2\\displaystyle\\\|\\Delta p\_\{i\}\\\|\_\{\\mu\}^\{2\}=‖uitask\+uiKD\+rip\+ξip‖μ2\\displaystyle=\\\|u\_\{i\}^\{\\rm task\}\+u\_\{i\}^\{\\rm KD\}\+r\_\{i\}^\{p\}\+\\xi\_\{i\}^\{p\}\\\|\_\{\\mu\}^\{2\}≤\(‖uitask‖μ\+‖uiKD‖μ\+‖rip‖μ\+‖ξip‖μ\)2\\displaystyle\\leq\\left\(\\\|u\_\{i\}^\{\\rm task\}\\\|\_\{\\mu\}\+\\\|u\_\{i\}^\{\\rm KD\}\\\|\_\{\\mu\}\+\\\|r\_\{i\}^\{p\}\\\|\_\{\\mu\}\+\\\|\\xi\_\{i\}^\{p\}\\\|\_\{\\mu\}\\right\)^\{2\}≤4​\(‖uitask‖μ2\+‖uiKD‖μ2\+‖rip‖μ2\+‖ξip‖μ2\)\.\\displaystyle\\leq 4\\left\(\\\|u\_\{i\}^\{\\rm task\}\\\|\_\{\\mu\}^\{2\}\+\\\|u\_\{i\}^\{\\rm KD\}\\\|\_\{\\mu\}^\{2\}\+\\\|r\_\{i\}^\{p\}\\\|\_\{\\mu\}^\{2\}\+\\\|\\xi\_\{i\}^\{p\}\\\|\_\{\\mu\}^\{2\}\\right\)\.The activation identity gives an exact uniform average for the task square:

1N​𝔼e​∑i∈eqi−1​𝔼t​‖uitask‖μ2\\displaystyle\\frac\{1\}\{N\}\\mathbb\{E\}\_\{e\}\\sum\_\{i\\in e\}q\_\{i\}^\{\-1\}\\mathbb\{E\}\_\{t\}\\\|u\_\{i\}^\{\\rm task\}\\\|\_\{\\mu\}^\{2\}=1N​∑i𝔼t​‖uitask‖μ2\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i\}\\mathbb\{E\}\_\{t\}\\\|u\_\{i\}^\{\\rm task\}\\\|\_\{\\mu\}^\{2\}≤2​η2​\(1−α\)2​\[κ¯p2​Gt\+1N​∑i𝔼t​‖δit‖μ2\]\\displaystyle\\leq 2\\eta^\{2\}\(1\-\\alpha\)^\{2\}\\left\[\\overline\{\\kappa\}\_\{p\}^\{2\}G\_\{t\}\+\\frac\{1\}\{N\}\\sum\_\{i\}\\mathbb\{E\}\_\{t\}\\\|\\delta\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\\right\]≤2​η2​\(1−α\)2​\[\(κ¯p2\+χG\)​Gt\+χB​Bf2\+χζ​ζf2\]\.\\displaystyle\\leq 2\\eta^\{2\}\(1\-\\alpha\)^\{2\}\\left\[\(\\overline\{\\kappa\}\_\{p\}^\{2\}\+\\chi\_\{G\}\)G\_\{t\}\+\\chi\_\{B\}B\_\{f\}^\{2\}\+\\chi\_\{\\zeta\}\\zeta\_\{f\}^\{2\}\\right\]\.For the KD square, the teacher decomposition and the two activation\-corrected edge bounds above give finite constantsCsq,V,Csq,A\>0C\_\{\\rm sq,V\},C\_\{\\rm sq,A\}\>0such that

1N​𝔼e​∑i∈eqi−1​𝔼t​‖uiKD‖μ2\\displaystyle\\frac\{1\}\{N\}\\mathbb\{E\}\_\{e\}\\sum\_\{i\\in e\}q\_\{i\}^\{\-1\}\\mathbb\{E\}\_\{t\}\\\|u\_\{i\}^\{\\rm KD\}\\\|\_\{\\mu\}^\{2\}≤η2​α2​Hmax2​1N​\|E\|​∑\(i,j\)∈E→qi−1​𝔼t​‖Pi−PjT‖μ2\\displaystyle\\leq\\eta^\{2\}\\alpha^\{2\}H\_\{\\max\}^\{2\}\\frac\{1\}\{N\|E\|\}\\sum\_\{\(i,j\)\\in\\vec\{E\}\}q\_\{i\}^\{\-1\}\\mathbb\{E\}\_\{t\}\\\|P\_\{i\}\-P\_\{j\}^\{T\}\\\|\_\{\\mu\}^\{2\}≤η2​Csq,V​𝒱W​\(Zt\)\+η2​Csq,A​Atf\.\\displaystyle\\leq\\eta^\{2\}C\_\{\\rm sq,V\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+\\eta^\{2\}C\_\{\\rm sq,A\}A\_\{t\}^\{f\}\.The remainder and noise bounds, together with the same activation identity, give

1N​𝔼e​∑i∈eqi−1​𝔼t​‖rip‖μ2≤Rp​η4,\\displaystyle\\frac\{1\}\{N\}\\mathbb\{E\}\_\{e\}\\sum\_\{i\\in e\}q\_\{i\}^\{\-1\}\\mathbb\{E\}\_\{t\}\\\|r\_\{i\}^\{p\}\\\|\_\{\\mu\}^\{2\}\\leq R\_\{p\}\\eta^\{4\},\(G48\)1N​𝔼e​∑i∈eqi−1​𝔼t​‖ξip‖μ2≤Σp​η2\.\\displaystyle\\frac\{1\}\{N\}\\mathbb\{E\}\_\{e\}\\sum\_\{i\\in e\}q\_\{i\}^\{\-1\}\\mathbb\{E\}\_\{t\}\\\|\\xi\_\{i\}^\{p\}\\\|\_\{\\mu\}^\{2\}\\leq\\Sigma\_\{p\}\\eta^\{2\}\.\(G49\)
Collecting the four squares produces constantsCΔ,G,CΔ,V,CΔ,B,CΔ,ζ,CΔ,AC\_\{\\Delta,G\},C\_\{\\Delta,V\},C\_\{\\Delta,B\},C\_\{\\Delta,\\zeta\},C\_\{\\Delta,A\}such that

1N​𝔼e​∑i∈eqi−1​𝔼t​‖Δ​pi‖μ2\\displaystyle\\frac\{1\}\{N\}\\mathbb\{E\}\_\{e\}\\sum\_\{i\\in e\}q\_\{i\}^\{\-1\}\\mathbb\{E\}\_\{t\}\\\|\\Delta p\_\{i\}\\\|\_\{\\mu\}^\{2\}\(G50\)≤η2\[CΔ,GGt\+CΔ,V𝒱W\(Zt\)\\displaystyle\\leq\\eta^\{2\}\\big\[C\_\{\\Delta,G\}G\_\{t\}\+C\_\{\\Delta,V\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\(G51\)\+CΔ,BBf2\+CΔ,ζζf2\+CΔ,AAtf\]\\displaystyle\\qquad\+C\_\{\\Delta,B\}B\_\{f\}^\{2\}\+C\_\{\\Delta,\\zeta\}\\zeta\_\{f\}^\{2\}\+C\_\{\\Delta,A\}A\_\{t\}^\{f\}\\big\]\(G52\)\+4​Rp​η4\+4​Σp​η2\.\\displaystyle\\qquad\+4R\_\{p\}\\eta^\{4\}\+4\\Sigma\_\{p\}\\eta^\{2\}\.\(G53\)
Letdtask\>0d\_\{\\rm task\}\>0be the remaining coefficient ofGtG\_\{t\}after the first\-order task and KD inner\-product bounds\. Choose the admissible step\-size ceiling so thatηmax≤1\\eta\_\{\\max\}\\leq 1andLf​CΔ,G​ηmax/2≤dtask/2L\_\{f\}C\_\{\\Delta,G\}\\eta\_\{\\max\}/2\\leq d\_\{\\rm task\}/2\. Then

−dtask​η​Gt\+Lf2​CΔ,G​η2​Gt≤−dtask2​η​Gt\.\-d\_\{\\rm task\}\\eta G\_\{t\}\+\\frac\{L\_\{f\}\}\{2\}C\_\{\\Delta,G\}\\eta^\{2\}G\_\{t\}\\leq\-\\frac\{d\_\{\\rm task\}\}\{2\}\\eta G\_\{t\}\.SetaF=dtask/2a\_\{F\}=d\_\{\\rm task\}/2\. Each remaining smoothness term satisfiesη2​X≤ηmax​η​X\\eta^\{2\}X\\leq\\eta\_\{\\max\}\\eta X, while4​Rp​η4\+4​Σp​η2≤4​\(Rp\+Σp\)​η24R\_\{p\}\\eta^\{4\}\+4\\Sigma\_\{p\}\\eta^\{2\}\\leq 4\(R\_\{p\}\+\\Sigma\_\{p\}\)\\eta^\{2\}\. Absorbing these finite coefficients intobF,DB,Dζ,Dη,DAb\_\{F\},D\_\{B\},D\_\{\\zeta\},D\_\{\\eta\},D\_\{A\}proves Equation[G45](https://arxiv.org/html/2609.01952#A7.E45)\.

For the gradient transfer, defineF=N−1​∑iFiF=N^\{\-1\}\\sum\_\{i\}F\_\{i\}andp¯t=N−1​∑ipit\\bar\{p\}^\{t\}=N^\{\-1\}\\sum\_\{i\}p\_\{i\}^\{t\}\. Add and subtractN−1​∑i∇fFi​\(pit\)N^\{\-1\}\\sum\_\{i\}\\nabla\_\{f\}F\_\{i\}\(p\_\{i\}^\{t\}\):

∇fF​\(p¯t\)\\displaystyle\\nabla\_\{f\}F\(\\bar\{p\}^\{t\}\)=1N​∑i∇fFi​\(p¯t\)\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i\}\\nabla\_\{f\}F\_\{i\}\(\\bar\{p\}^\{t\}\)\(G54\)=1N​∑igit\+1N​∑i\[∇fFi​\(p¯t\)−∇fFi​\(pit\)\]\.\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i\}g\_\{i\}^\{t\}\+\\frac\{1\}\{N\}\\sum\_\{i\}\\left\[\\nabla\_\{f\}F\_\{i\}\(\\bar\{p\}^\{t\}\)\-\\nabla\_\{f\}F\_\{i\}\(p\_\{i\}^\{t\}\)\\right\]\.\(G55\)
For any vectorsviv\_\{i\}, the Jensen bound follows from

‖1N​∑ivi‖μ2=1N2​∑i,j⟨vi,vj⟩μ≤1N2​∑i,j‖vi‖μ2\+‖vj‖μ22\\displaystyle\\left\\\|\\frac\{1\}\{N\}\\sum\_\{i\}v\_\{i\}\\right\\\|\_\{\\mu\}^\{2\}=\\frac\{1\}\{N^\{2\}\}\\sum\_\{i,j\}\\langle v\_\{i\},v\_\{j\}\\rangle\_\{\\mu\}\\leq\\frac\{1\}\{N^\{2\}\}\\sum\_\{i,j\}\\frac\{\\\|v\_\{i\}\\\|\_\{\\mu\}^\{2\}\+\\\|v\_\{j\}\\\|\_\{\\mu\}^\{2\}\}\{2\}\(G56\)=1N​∑i‖vi‖μ2\.\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i\}\\\|v\_\{i\}\\\|\_\{\\mu\}^\{2\}\.\(G57\)
Apply this bound to both sums and use‖a\+b‖2≤2​‖a‖2\+2​‖b‖2\\\|a\+b\\\|^\{2\}\\leq 2\\\|a\\\|^\{2\}\+2\\\|b\\\|^\{2\}:

‖∇fF​\(p¯t\)‖μ2\\displaystyle\\\|\\nabla\_\{f\}F\(\\bar\{p\}^\{t\}\)\\\|\_\{\\mu\}^\{2\}≤2N​∑i‖git‖μ2\\displaystyle\\leq\\frac\{2\}\{N\}\\sum\_\{i\}\\\|g\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\(G58\)\+2N∑i∥∇fFi\(p¯t\)−∇fFi\(pit\)∥μ2\\displaystyle\\quad\+\\frac\{2\}\{N\}\\sum\_\{i\}\\\|\\nabla\_\{f\}F\_\{i\}\(\\bar\{p\}^\{t\}\)\-\\nabla\_\{f\}F\_\{i\}\(p\_\{i\}^\{t\}\)\\\|\_\{\\mu\}^\{2\}\(G59\)≤2​Gt\+2​Lf2N​∑i‖p¯t−pit‖μ2\\displaystyle\\leq 2G\_\{t\}\+\\frac\{2L\_\{f\}^\{2\}\}\{N\}\\sum\_\{i\}\\\|\\bar\{p\}^\{t\}\-p\_\{i\}^\{t\}\\\|\_\{\\mu\}^\{2\}\(G60\)=2​Gt\+2​Lf2​ψtf\.\\displaystyle=2G\_\{t\}\+2L\_\{f\}^\{2\}\\psi\_\{t\}^\{f\}\.\(G61\)
∎

###### Lemma G9\(Strict Lyapunov recursion\)\.

Assume there existsλ\>0\\lambda\>0such that

aF−λ​CG\>0,λ​c0−bF\>0\.a\_\{F\}\-\\lambda C\_\{G\}\>0,\\qquad\\lambda c\_\{0\}\-b\_\{F\}\>0\.\(G62\)Such aλ\\lambdaexists exactly whenbF​CG<aF​c0b\_\{F\}C\_\{G\}<a\_\{F\}c\_\{0\}; this is the explicit task–gossip small\-gain requirement\. Forℒt=ℛt\+λ​𝒱W​\(Zt\)\\mathcal\{L\}\_\{t\}=\\mathcal\{R\}\_\{t\}\+\\lambda\\mathcal\{V\}\_\{W\}\(Z^\{t\}\), there are constantscG,cV,KB,Kζ,Kη\>0c\_\{G\},c\_\{V\},K\_\{B\},K\_\{\\zeta\},K\_\{\\eta\}\>0such that

𝔼t​ℒt\+1≤ℒt−cG​η​Gt−cV​η​𝒱W​\(Zt\)\+KB​η​Bf2\+Kζ​η​ζf2\+Kη​η2\.\\begin\{split\}\\mathbb\{E\}\_\{t\}\\mathcal\{L\}\_\{t\+1\}\\leq\\mathcal\{L\}\_\{t\}\-c\_\{G\}\\eta G\_\{t\}\-c\_\{V\}\\eta\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+K\_\{B\}\\eta B\_\{f\}^\{2\}\\\\ \+K\_\{\\zeta\}\\eta\\zeta\_\{f\}^\{2\}\+K\_\{\\eta\}\\eta^\{2\}\.\\end\{split\}\(G63\)

###### Proof\.

The first inequality in Equation[G62](https://arxiv.org/html/2609.01952#A7.E62)is equivalent toλ<aF/CG\\lambda<a\_\{F\}/C\_\{G\}, while the second is equivalent toλ\>bF/c0\\lambda\>b\_\{F\}/c\_\{0\}\. Thus a validλ\\lambdaexists precisely when

bFc0\\displaystyle\\frac\{b\_\{F\}\}\{c\_\{0\}\}<aFCG⟺bFCG<aFc0\.\\displaystyle<\\frac\{a\_\{F\}\}\{C\_\{G\}\}\\quad\\Longleftrightarrow\\quad b\_\{F\}C\_\{G\}<a\_\{F\}c\_\{0\}\.When this strict inequality holds, everyλ∈\(bF/c0,aF/CG\)\\lambda\\in\(b\_\{F\}/c\_\{0\},a\_\{F\}/C\_\{G\}\)satisfies both margins\. Fix one suchλ\\lambda\. Multiply Equation[G26](https://arxiv.org/html/2609.01952#A7.E26)byλ\\lambdaand display each term:

λ​𝔼t​𝒱W\\displaystyle\\lambda\\mathbb\{E\}\_\{t\}\\mathcal\{V\}\_\{W\}\(Zt\+1\)≤λ​𝒱W​\(Zt\)−λ​c0​η​𝒱W​\(Zt\)\+λ​CG​η​Gt\\displaystyle\(Z^\{t\+1\}\)\\leq\\lambda\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\-\\lambda c\_\{0\}\\eta\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+\\lambda C\_\{G\}\\eta G\_\{t\}\+λ​CB​η​Bf2\+λ​Cζ​η​ζf2\+λ​Cη​η2\+λ​CA​η​Atf\.\\displaystyle\+\\lambda C\_\{B\}\\eta B\_\{f\}^\{2\}\+\\lambda C\_\{\\zeta\}\\eta\\zeta\_\{f\}^\{2\}\+\\lambda C\_\{\\eta\}\\eta^\{2\}\+\\lambda C\_\{A\}\\eta A\_\{t\}^\{f\}\.Equation[G45](https://arxiv.org/html/2609.01952#A7.E45)is

𝔼t​ℛt\+1\\displaystyle\\mathbb\{E\}\_\{t\}\\mathcal\{R\}\_\{t\+1\}≤ℛt−aF​η​Gt\+bF​η​𝒱W​\(Zt\)\+DB​η​Bf2\+Dζ​η​ζf2\\displaystyle\\leq\\mathcal\{R\}\_\{t\}\-a\_\{F\}\\eta G\_\{t\}\+b\_\{F\}\\eta\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+D\_\{B\}\\eta B\_\{f\}^\{2\}\+D\_\{\\zeta\}\\eta\\zeta\_\{f\}^\{2\}\+Dη​η2\+DA​η​Atf\.\\displaystyle\+D\_\{\\eta\}\\eta^\{2\}\+D\_\{A\}\\eta A\_\{t\}^\{f\}\.By definition,ℒt\+1=ℛt\+1\+λ​𝒱W​\(Zt\+1\)\\mathcal\{L\}\_\{t\+1\}=\\mathcal\{R\}\_\{t\+1\}\+\\lambda\\mathcal\{V\}\_\{W\}\(Z^\{t\+1\}\)\. Adding the two displays term by term gives

𝔼t​ℒt\+1\\displaystyle\\mathbb\{E\}\_\{t\}\\mathcal\{L\}\_\{t\+1\}≤ℒt−\(aF−λ​CG\)​η​Gt−\(λ​c0−bF\)​η​𝒱W​\(Zt\)\\displaystyle\\leq\\mathcal\{L\}\_\{t\}\-\(a\_\{F\}\-\\lambda C\_\{G\}\)\\eta G\_\{t\}\-\(\\lambda c\_\{0\}\-b\_\{F\}\)\\eta\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\+\(DB\+λ​CB\)​η​Bf2\+\(Dζ\+λ​Cζ\)​η​ζf2\\displaystyle\+\(D\_\{B\}\+\\lambda C\_\{B\}\)\\eta B\_\{f\}^\{2\}\+\(D\_\{\\zeta\}\+\\lambda C\_\{\\zeta\}\)\\eta\\zeta\_\{f\}^\{2\}\+\(Dη\+λ​Cη\)​η2\+\(DA\+λ​CA\)​η​Atf\.\\displaystyle\+\(D\_\{\\eta\}\+\\lambda C\_\{\\eta\}\)\\eta^\{2\}\+\(D\_\{A\}\+\\lambda C\_\{A\}\)\\eta A\_\{t\}^\{f\}\.Set

cG\\displaystyle c\_\{G\}:=aF−λ​CG\>0,\\displaystyle:=a\_\{F\}\-\\lambda C\_\{G\}\>0,cV\\displaystyle c\_\{V\}:=λ​c0−bF\>0,\\displaystyle:=\\lambda c\_\{0\}\-b\_\{F\}\>0,KB\\displaystyle K\_\{B\}:=DB\+λ​CB,\\displaystyle:=D\_\{B\}\+\\lambda C\_\{B\},Kζ\\displaystyle K\_\{\\zeta\}:=Dζ\+λ​Cζ,\\displaystyle:=D\_\{\\zeta\}\+\\lambda C\_\{\\zeta\},KA\\displaystyle K\_\{A\}:=DA\+λ​CA,\\displaystyle:=D\_\{A\}\+\\lambda C\_\{A\},Kη\(0\)\\displaystyle K\_\{\\eta\}^\{\(0\)\}:=Dη\+λ​Cη\.\\displaystyle:=D\_\{\\eta\}\+\\lambda C\_\{\\eta\}\.Lemma[G5](https://arxiv.org/html/2609.01952#Thmlemma5a)givesAtf≤A¯f​η2A\_\{t\}^\{f\}\\leq\\bar\{A\}^\{f\}\\eta^\{2\}withA¯f=cstep​Aeff2\\bar\{A\}^\{f\}=c\_\{\\rm step\}A\_\{\\rm eff\}^\{2\}\. Forη≤1\\eta\\leq 1,

KA​η​Atf≤KA​A¯f​η3≤KA​A¯f​η2\.K\_\{A\}\\eta A\_\{t\}^\{f\}\\leq K\_\{A\}\\bar\{A\}^\{f\}\\eta^\{3\}\\leq K\_\{A\}\\bar\{A\}^\{f\}\\eta^\{2\}\.DefiningKη:=Kη\(0\)\+KA​A¯fK\_\{\\eta\}:=K\_\{\\eta\}^\{\(0\)\}\+K\_\{A\}\\bar\{A\}^\{f\}proves Equation[G63](https://arxiv.org/html/2609.01952#A7.E63)\. ∎

###### Theorem G1\(Constant\-step convergence\)\.

Under A1–A10,0<α<10<\\alpha<1, a connected graph with no isolated peers, the frozen\-kernel condition, the thresholds in Equations[G17](https://arxiv.org/html/2609.01952#A7.E17)and[G62](https://arxiv.org/html/2609.01952#A7.E62), and0<η≤ηmax0<\\eta\\leq\\eta\_\{\\max\}, there are constantsc,C0,C1,C2,C3\>0c,C\_\{0\},C\_\{1\},C\_\{2\},C\_\{3\}\>0such that

1T​∑t=0T−1\[𝔼​‖∇fF​\(p¯t\)‖μ2\+c​Ψtf\]≤C0η​T\+C1​η\+C2​Bf2\+C3​ζf2\.\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\left\[\\mathbb\{E\}\\\|\\nabla\_\{f\}F\(\\bar\{p\}^\{t\}\)\\\|\_\{\\mu\}^\{2\}\+c\\Psi\_\{t\}^\{f\}\\right\]\\leq\\frac\{C\_\{0\}\}\{\\eta T\}\+C\_\{1\}\\eta\+C\_\{2\}B\_\{f\}^\{2\}\+C\_\{3\}\\zeta\_\{f\}^\{2\}\.\(G64\)

###### Proof\.

Take total expectation in Equation[G63](https://arxiv.org/html/2609.01952#A7.E63)\. The tower property𝔼⁡\[𝔼t​X\]=𝔼​X\\mathbb\{E\}\[\\mathbb\{E\}\_\{t\}X\]=\\mathbb\{E\}Xgives, for eachtt,

cG​η​𝔼​Gt\+cV​η​𝔼​𝒱W​\(Zt\)\\displaystyle c\_\{G\}\\eta\\mathbb\{E\}G\_\{t\}\+c\_\{V\}\\eta\\mathbb\{E\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)≤𝔼​ℒt−𝔼​ℒt\+1\\displaystyle\\leq\\mathbb\{E\}\\mathcal\{L\}\_\{t\}\-\\mathbb\{E\}\\mathcal\{L\}\_\{t\+1\}\+KB​η​Bf2\+Kζ​η​ζf2\+Kη​η2\.\\displaystyle\+K\_\{B\}\\eta B\_\{f\}^\{2\}\+K\_\{\\zeta\}\\eta\\zeta\_\{f\}^\{2\}\+K\_\{\\eta\}\\eta^\{2\}\.Summing fromt=0t=0toT−1T\-1telescopes the Lyapunov differences:

∑t=0T−1\(𝔼​ℒt−𝔼​ℒt\+1\)=𝔼​ℒ0−𝔼​ℒT\.\\sum\_\{t=0\}^\{T\-1\}\\left\(\\mathbb\{E\}\\mathcal\{L\}\_\{t\}\-\\mathbb\{E\}\\mathcal\{L\}\_\{t\+1\}\\right\)=\\mathbb\{E\}\\mathcal\{L\}\_\{0\}\-\\mathbb\{E\}\\mathcal\{L\}\_\{T\}\.Both terms inℒT=ℛT\+λ​𝒱W​\(ZT\)\\mathcal\{L\}\_\{T\}=\\mathcal\{R\}\_\{T\}\+\\lambda\\mathcal\{V\}\_\{W\}\(Z^\{T\}\)are nonnegative, becauseFi​\(piT\)−Fi⋆≥0F\_\{i\}\(p\_\{i\}^\{T\}\)\-F\_\{i\}^\{\\star\}\\geq 0and𝒱W≥0\\mathcal\{V\}\_\{W\}\\geq 0\. Thus

cG​η​∑t=0T−1𝔼​Gt\+cV​η​∑t=0T−1𝔼​𝒱W​\(Zt\)\\displaystyle c\_\{G\}\\eta\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}G\_\{t\}\+c\_\{V\}\\eta\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)≤𝔼​ℒ0\+T⁡\(KB​η​Bf2\+Kζ​η​ζf2\+Kη​η2\)\.\\displaystyle\\leq\\mathbb\{E\}\\mathcal\{L\}\_\{0\}\+T\(K\_\{B\}\\eta B\_\{f\}^\{2\}\+K\_\{\\zeta\}\\eta\\zeta\_\{f\}^\{2\}\+K\_\{\\eta\}\\eta^\{2\}\)\.For deterministic initialization,𝔼​ℒ0=ℒ0\\mathbb\{E\}\\mathcal\{L\}\_\{0\}=\\mathcal\{L\}\_\{0\}\.

Lemma[G3](https://arxiv.org/html/2609.01952#Thmlemma3a)and the metric comparison provide a finite constantCf​zC\_\{fz\}such thatψtf≤Cf​z​𝒱W​\(Zt\)/mW\\psi\_\{t\}^\{f\}\\leq C\_\{fz\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)/m\_\{W\}pathwise\. Taking expectation gives

Ψtf≤Cf​zmW​𝔼​𝒱W​\(Zt\)\.\\Psi\_\{t\}^\{f\}\\leq\\frac\{C\_\{fz\}\}\{m\_\{W\}\}\\mathbb\{E\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\.Taking expectation in Equation[G46](https://arxiv.org/html/2609.01952#A7.E46)gives

𝔼​‖∇fF​\(p¯t\)‖μ2\\displaystyle\\mathbb\{E\}\\\|\\nabla\_\{f\}F\(\\bar\{p\}^\{t\}\)\\\|\_\{\\mu\}^\{2\}≤2​𝔼​Gt\+2​Lf2​Ψtf\\displaystyle\\leq 2\\mathbb\{E\}G\_\{t\}\+2L\_\{f\}^\{2\}\\Psi\_\{t\}^\{f\}≤2​𝔼​Gt\+2​Lf2​Cf​zmW​𝔼​𝒱W​\(Zt\)\.\\displaystyle\\leq 2\\mathbb\{E\}G\_\{t\}\+\\frac\{2L\_\{f\}^\{2\}C\_\{fz\}\}\{m\_\{W\}\}\\mathbb\{E\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\.For any fixedc\>0c\>0, define

K∗:=max⁡\{2cG,2​Lf2​Cf​z/mW\+c​Cf​z/mWcV\}\.K\_\{\*\}:=\\max\\left\\\{\\frac\{2\}\{c\_\{G\}\},\\frac\{2L\_\{f\}^\{2\}C\_\{fz\}/m\_\{W\}\+cC\_\{fz\}/m\_\{W\}\}\{c\_\{V\}\}\\right\\\}\.The definition ofK∗K\_\{\*\}ensures, for eachtt,

𝔼​‖∇fF​\(p¯t\)‖μ2\+c​Ψtf\\displaystyle\\mathbb\{E\}\\\|\\nabla\_\{f\}F\(\\bar\{p\}^\{t\}\)\\\|\_\{\\mu\}^\{2\}\+c\\Psi\_\{t\}^\{f\}≤2​𝔼​Gt\+\(2​Lf2\+c\)​Cf​zmW​𝔼​𝒱W​\(Zt\)\\displaystyle\\leq 2\\mathbb\{E\}G\_\{t\}\+\\frac\{\(2L\_\{f\}^\{2\}\+c\)C\_\{fz\}\}\{m\_\{W\}\}\\mathbb\{E\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)≤K∗​\(cG​𝔼​Gt\+cV​𝔼​𝒱W​\(Zt\)\)\.\\displaystyle\\leq K\_\{\*\}\\left\(c\_\{G\}\\mathbb\{E\}G\_\{t\}\+c\_\{V\}\\mathbb\{E\}\\mathcal\{V\}\_\{W\}\(Z^\{t\}\)\\right\)\.Summing this inequality, applying the telescoped Lyapunov bound, and dividing byη​T\\eta Tgives

1T​∑t=0T−1\[𝔼​‖∇fF​\(p¯t\)‖μ2\+c​Ψtf\]\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\left\[\\mathbb\{E\}\\\|\\nabla\_\{f\}F\(\\bar\{p\}^\{t\}\)\\\|\_\{\\mu\}^\{2\}\+c\\Psi\_\{t\}^\{f\}\\right\]≤K∗​ℒ0η​T\+K∗​Kη​η\\displaystyle\\leq\\frac\{K\_\{\*\}\\mathcal\{L\}\_\{0\}\}\{\\eta T\}\+K\_\{\*\}K\_\{\\eta\}\\eta\+K∗​KB​Bf2\+K∗​Kζ​ζf2\.\\displaystyle\+K\_\{\*\}K\_\{B\}B\_\{f\}^\{2\}\+K\_\{\*\}K\_\{\\zeta\}\\zeta\_\{f\}^\{2\}\.SetC0=K∗​ℒ0C\_\{0\}=K\_\{\*\}\\mathcal\{L\}\_\{0\},C1=K∗​KηC\_\{1\}=K\_\{\*\}K\_\{\\eta\},C2=K∗​KBC\_\{2\}=K\_\{\*\}K\_\{B\}, andC3=K∗​KζC\_\{3\}=K\_\{\*\}K\_\{\\zeta\}\. This proves Equation[G64](https://arxiv.org/html/2609.01952#A7.E64)\. ∎

Theζf2\\zeta\_\{f\}^\{2\}term cannot generally be removed\. With two identical full\-rank peers,Bf=0B\_\{f\}=0, no noise, and local quadratic risks centered atbband−b\-b, the symmetric CE\+KD recursion has a nonzero disagreement fixed point independent ofη\\eta\. This example satisfies the kernel assumptions and has zero KD projection defect\. Deterministic non\-IID task drift is therefore a separate neighbourhood contribution rather than a discretization error\.

## Appendix HExperimental Protocol and Complete Results

This appendix gives the complete empirical validation of the convergence claims\. The campaign contains 25 stored trajectories: 20 random\-edge KD runs, four isolated local\-training controls, and one homogeneous D\-SGD control\. Twenty\-four histories end at 70,000 edge events\. The Setting\-C,η=0\.05\\eta=0\.05, seed\-1 history contains 101 snapshots through 100,000 events, although its copied configuration retains a 70,000\-event target\. All reported values use the actual event indices stored in the JSON history\. The figures and summary statistics in this appendix are regenerated byappendix\_analysis\.pyfrom the files inresults/full, but due to the limit of 50MB of data supplement, we only upload the code with the detailed execution procedures inREADME\.md\. For running the whole experiment, we used MacStudio M4 Max 128 GB and we ran the whole experiments set for≈100\\approx 100hours\.

### Experimental Setup and Theorem\-to\-Protocol Mapping

We simulateN=20N=20peers on CIFAR\-10\. Each peer receives a private shard from a Dirichlet label partition with concentrationγpart=0\.3\\gamma\_\{\\rm part\}=0\.3\. For each seed, the same connectedER⁡\(0\.3\)\\operatorname\{ER\}\(0\.3\)graph is used throughout the run\. Seed 0 has 51 edges and seed 1 has 58 edges, with graph spectral gaps1\.2321\.232and1\.8221\.822, respectively\. The reference measureμ\\muis the empirical measure on a fixed 2,000\-image CIFAR\-10 test subset\. The same subset is used for KD and evaluation, so all function\-space measurements are transductive onμ\\mu\.

Table 3:Model rosters used in the four settings\. The parameter counts are approximate and are included to describe capacity scale, not to estimateBfB\_\{f\}\.Every peer first receives 1,500 local cross\-entropy pretraining steps\. The gossip phase samples one undirected edge per event and updates both endpoints\. The teacher is live, so there is no stored\-version delay in these runs\. Each KD update uses the shared reference batch,α=0\.5\\alpha=0\.5,τ=4\\tau=4, constant step sizeη\\eta, batch size 64, weight decay5×10−45\\times 10^\{\-4\}, label smoothing0\.10\.1, logit clipping at 10, and parameter\-gradient clipping at 5\. The main runs useη=0\.05\\eta=0\.05and seeds 0 and 1\. The seed\-0 sensitivity sweep usesη∈\{0\.0125,0\.025,0\.05,0\.1\}\\eta\\in\\\{0\.0125,0\.025,0\.05,0\.1\\\}\. Evaluation is recorded every 1,000 events\. Table[4](https://arxiv.org/html/2609.01952#A8.T4)records the remaining settings\.

Table 4:Fixed protocol for the completed campaign\.The measured consensus statistic is

Ψ^tf=1N​∑i=1N‖pit−p¯t‖μ2,p¯t=1N​∑i=1Npit,\\widehat\{\\Psi\}\_\{t\}^\{f\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\\\|p\_\{i\}^\{t\}\-\\overline\{p\}^\{t\}\\right\\\|\_\{\\mu\}^\{2\},\\qquad\\overline\{p\}^\{t\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}p\_\{i\}^\{t\},\(H65\)wherepitp\_\{i\}^\{t\}is the temperature\-scaled prediction of peeriion the reference support\. For the stationarity channel, letS^t\\widehat\{S\}\_\{t\}denote the stored squared residual between the mean temperature\-one prediction and the label\-smoothed target onμ\\mu\. IfTk=1000​kT\_\{k\}=1000kdenotes a logged horizon, the theorem\-facing prefix statistic is

G^T\\displaystyle\\widehat\{G\}\_\{T\}=1K⁡\(T\)\+1​∑k=0K⁡\(T\)S^Tk,\\displaystyle=\\frac\{1\}\{K\(T\)\+1\}\\sum\_\{k=0\}^\{K\(T\)\}\\widehat\{S\}\_\{T\_\{k\}\},\(H66\)K⁡\(T\)\\displaystyle K\(T\)=max⁡\{k:Tk≤T\}\.\\displaystyle=\\max\\\{k:T\_\{k\}\\leq T\\\}\.Thus,G^T\\widehat\{G\}\_\{T\}is a sampled time average, whereas the final raw valueS^T\\widehat\{S\}\_\{T\}is an instantaneous terminal residual\.

The protocol provides operational controls corresponding to several analytical objects\. The finite reference support, logit clipping, and gradient clipping make the numerical observables bounded, which is the implementation analogue of the bounded\-support and moment controls used in the analysis\. The common prediction space makesΨ^tf\\widehat\{\\Psi\}\_\{t\}^\{f\}well\-defined for all four rosters\. The restricted task/KD alignment condition associated with A10 is probed by Settings B–D, especially the mixed\-family Setting D\. This appendix uses the experiments to test the theorem’s observable consequences\.

### Synthetic Verification of the Convergence\-Rate Diagnostic

Before interpreting neural\-network traces, we calibrated the rate estimator on known signals\. For a prescribed floorbband exponentpp, the calibration trace has the form

GT=b\+a​T−p\+εT,G\_\{T\}=b\+aT^\{\-p\}\+\\varepsilon\_\{T\},\(H67\)with deterministicp∈\{0\.5,1\.0,1\.5\}p\\in\\\{0\.5,1\.0,1\.5\\\}and small zero\-mean perturbations\. The diagnostic fits a line tolog⁡\(GT−b\)\\log\(G\_\{T\}\-b\)againstlog⁡T\\log Tover the positive\-excess region\. The recovered exponents are0\.5000\.500,0\.9880\.988, and1\.4281\.428, respectively\. Figure[6](https://arxiv.org/html/2609.01952#A8.F6)shows the traces, the fitted lines, and the prescribed reference slopes\.

Figure 6:Synthetic calibration of the inverse\-power rate diagnostic\. The estimator recovers the known exponentsp=0\.5p=0\.5,1\.01\.0, and1\.51\.5from finite noisy traces\. This validates the numerical diagnostic\. This is regression, so it does not serve as any approximation for the exponent used in the theory, it only shows the variation of the data towards different exponents\.First, the implementation can recover an inverse\-power exponent when the floor and transient are known\. Second, a neural trajectory must estimate its neighbourhood from the same finite data, which introduces floor\-estimation error and makes the early transient and terminal noise relevant\. The neural results below therefore report the global fit and the late dynamic exponent separately\.

### Stored Convergence Traces

Figure[7](https://arxiv.org/html/2609.01952#A8.F7)displays the seed\-0 main trajectories\. The large panels show the instantaneous function\-space disagreementΨ^tf\\widehat\{\\Psi\}\_\{t\}^\{f\}\. The inset in each panel shows the prefix stationarity statisticG^T\\widehat\{G\}\_\{T\}\. KD produces a rapid reduction in disagreement in all four rosters, including the mixed\-family Setting D where in the system, both architecture and the parameter sizes are different across models\. The stationarity trace is less uniform across rosters, which is expected because it also contains local\-task and representation effects\.

Figure 7:Seed\-0 main trajectories\. The large panels show the instantaneous function\-space disagreement in Equation[H65](https://arxiv.org/html/2609.01952#A8.E65); each inset shows the prefix stationarity statistic in Equation[H66](https://arxiv.org/html/2609.01952#A8.E66)\. The same output\-space observable is available for homogeneous, width\-heterogeneous, and mixed\-family peers\.
### Empirical Convergence\-Rate Diagnostics

For each stored history, the rate analysis estimates a terminal neighbourhoodb^G\\widehat\{b\}\_\{G\}by regressing the prefix statistic on1/T1/Tover the final 80 percent of positive horizons \(after pre\-training, because if the models start from nothing but noise parameters, it does not make sense\)\. It then defines the regularized excess

ET\+=max⁡\{G^T−b^G,εb\},E\_\{T\}^\{\+\}=\\max\\\{\\widehat\{G\}\_\{T\}\-\\widehat\{b\}\_\{G\},\\varepsilon\_\{b\}\\\},\(H68\)whereεb\\varepsilon\_\{b\}is a robust terminal\-noise estimate\. The dynamic exponent is the negative local slope oflog⁡ET\+\\log E\_\{T\}^\{\+\}againstlog⁡T\\log Tin a seven\-snapshot moving window\. The late valuep~dyn\\widetilde\{p\}\_\{\\rm dyn\}is the median over the latter half of the resolvable positive\-excess region\.

Table[5](https://arxiv.org/html/2609.01952#A8.T5)gives the main\-run diagnostics\. Settings A and B have late85%85\\%dynamic exponents close to1\.001\.00for both seeds\. Setting C reaches1\.211\.21for seed 0 and1\.901\.90for the extended seed\-1 history\. The mixed\-family Setting D also reaches the consensus neighbourhood\.

Table 5:Main KD trajectories atη=0\.05\\eta=0\.05\.b^Ψ\\widehat\{b\}\_\{\\Psi\}andb^G\\widehat\{b\}\_\{G\}are finite\-horizon final\-85%1/T1/T\-regression intercepts\.G^T\\widehat\{G\}\_\{T\}is the prefix statistic in Equation[H66](https://arxiv.org/html/2609.01952#A8.E66)\. A dash indicates that no positive stationarity excess remained above three terminal\-noise standard errors\.p~dyn\\widetilde\{p\}\_\{\\rm dyn\}is the best fit exponent for that specific experiment\.The global log\-log fit is intentionally not used as the primary rate claim\. It includes early transients and is sensitive to the fitted floor\. In the present data its exponent is below one for many runs, while the late dynamic exponent for A–D is near or above1\.001\.00\. This pattern is consistent with a finite trajectory entering the inverse\-time regime after an initial communication and optimization transient\. The synthetic calibration shows why the distinction is necessary: a correct estimator still returns a biased global slope if the fit includes a non\-asymptotic region or a poorly resolved floor\.

### Analysis and Verification of the Error Neighbourhood

The convergence bound has the schematic form

1T​∑t<T\(‖∇fF​\(p¯t\)‖μ2​cΨ​Ψtf\)≤C0η​T​C1​η​C2​Bf2​C3​ζf2\.\\frac\{1\}\{T\}\\sum\_\{t<T\}\\left\(\\\|\\nabla\_\{f\}F\(\\overline\{p\}^\{t\}\)\\\|\_\{\\mu\}^\{2\}c\_\{\\Psi\}\\Psi\_\{t\}^\{f\}\\right\)\\leq\\frac\{C\_\{0\}\}\{\\eta T\}C\_\{1\}\\eta C\_\{2\}B\_\{f\}^\{2\}C\_\{3\}\\zeta\_\{f\}^\{2\}\.\(H69\)The step\-size sweep tests the two finite\-horizon consequences that are identifiable in the present campaign\. At fixedTT, a smallerη\\etareduces the explicit discretization contribution but contracts more slowly because the communication rate is proportional toη\\eta\. A largerη\\etareaches the neighbourhood sooner but can leave a larger stochastic neighbourhood\. The measured intercepts are not used to identifyBf2B\_\{f\}^\{2\}andζf2\\zeta\_\{f\}^\{2\}separately\.

Figure[7](https://arxiv.org/html/2609.01952#A8.F7)reports the fitted consensus and stationarity intercepts for the complete seed\-0 sweep\. For A–C, the consensus intercept decreases substantially fromη=0\.0125\\eta=0\.0125toη=0\.05\\eta=0\.05and then changes little atη=0\.1\\eta=0\.1\. Setting D has a higher and less regular stationarity intercept, which is consistent with the additional representation and local task mismatch in the mixed\-model architecture roster\.

Figure 8:Finite\-horizon step\-size sensitivity\. Filled markers satisfy the stored convergence gate and open markers do not\. The fitted intercepts summarize the observed neighbourhood of the prefix statistic and are not identifications of individual terms in Equation[H69](https://arxiv.org/html/2609.01952#A8.E69)\.Table 6:Seed\-0 step\-size sweep\. The intercepts are descriptive fits to the stored histories\. The convergence flag is the simulator’s fixed tail\-stability criterion, not a theorem test\.The low\-step\-size rows should not be read as contradictions of the𝒪⁡\(1/\(η​T\)\)\\mathcal\{O\}\(1/\(\\eta T\)\)transient\. At the common event horizon, decreasingη\\etaalso decreases the amount of function\-space mixing completed by the experiment\. A longer horizon proportional to1/η1/\\etawould be required to compare the asymptotic neighbourhoods at equal effective communication time\. Likewise, the current data do not include an independent capacity probe that would estimateBfB\_\{f\}directly\. The results support the aggregate transient–neighbourhood decomposition in Equation[H69](https://arxiv.org/html/2609.01952#A8.E69), while leaving the separate constantsBfB\_\{f\}andζf\\zeta\_\{f\}unidentifiable\.

### Statistical Analysis of the Data Results

The main KD campaign has two graph and partition seeds per setting\. We report means and sample standard deviations over those two seeds, rather than treating the 20 peers within one run as independent repetitions\. Figure[9](https://arxiv.org/html/2609.01952#A8.F9)shows the resulting dispersion\. The consensus statistics are relatively stable: the seed standard deviations of terminalΨ^Tf\\widehat\{\\Psi\}\_\{T\}^\{f\}are1\.5×10−51\.5\\times 10^\{\-5\},3\.7×10−53\.7\\times 10^\{\-5\},8\.6×10−58\.6\\times 10^\{\-5\}, and4\.2×10−54\.2\\times 10^\{\-5\}for A–D, respectively\. The variation in the stationarity prefix is also small for A–C, with means0\.3630\.363,0\.3650\.365, and0\.3680\.368, while D has mean0\.5570\.557and a larger seed standard deviation of0\.1200\.120\. This separates the robust consensus effect from the more variable task\-stationarity channel\.

Figure 9:Descriptive mean and seed standard deviation for the two main runs in each setting\. The lower\-right panel omits Setting D because its stationarity excess is not resolvable for either seed\.The mechanism control compares the seed\-0 KD run atη=0\.05\\eta=0\.05with isolated local training using the same roster, data partition, and pretraining protocol\. The isolated runs finish with disagreement between1\.36×10−21\.36\\times 10^\{\-2\}and1\.51×10−21\.51\\times 10^\{\-2\}, whereas the KD runs finish between2\.88×10−42\.88\\times 10^\{\-4\}and3\.74×10−43\.74\\times 10^\{\-4\}\. The paired reductions are36\.3×36\.3\\times,40\.9×40\.9\\times,47\.5×47\.5\\times, and41\.0×41\.0\\timesfor A–D, with a mean reduction of41\.4×41\.4\\times\. An exploratory bootstrap interval over the four setting\-level ratios is\[37\.4,45\.8\]×\[37\.4,45\.8\]\\times\. Because the bootstrap has only four paired setting units and the runs are not independent samples from a target population, this interval is descriptive and is not used as a significance claim\.

Figure 10:Mechanism control atη=0\.05\\eta=0\.05, seed 0\. KD gossip reduces function\-space disagreement by 36–47 times relative to isolated local training across all rosters\. The accuracy panel is included to show that the consensus result is not being presented as a task\-accuracy ranking\.The accuracy values are not interpreted as evidence that KD dominates local training on task performance\. In fact, the isolated control has higher ensemble accuracy in these four runs\. This is compatible with our theoretical objective: the theorem concerns functional consensus and a stationarity proxy, while the task channel contains local\-data heterogeneity and representation effects\.

### Sensitivity to Learning Rate and Model Heterogeneity

The learning\-rate sweep and the four rosters jointly test whether the function\-space mechanism is restricted to the homogeneous reference case\. Settings A–C have shared ResNet skeletons with different width schedules\. Their late rate diagnostics remain close to one atη=0\.05\\eta=0\.05, and their terminal consensus values remain on the order of10−410^\{\-4\}across both main seeds\. Setting D replaces width changes with architecture\-family changes\. Its terminal consensus values are of the same order as A–C, but its stationarity residual is higher\. This is the expected distinction between a communication observable, which remains defined in prediction space, and a task\-stationarity observable, which depends on the reachable function classes and local objectives\.

The isolated and D\-SGD controls are interpreted as mechanism references\. D\-SGD is reported only for homogeneous Setting A because parameter averaging is not defined for B–D\. Its final disagreement is1\.34×10−31\.34\\times 10^\{\-3\}and its ensemble accuracy is0\.9050\.905, but its optimizer and momentum configuration differ from the KD phase\. It is therefore not an optimizer ranking baseline\. The isolated control is more directly aligned with the consensus question of local CE updates alone do not create the two\-order reduction inΨ^tf\\widehat\{\\Psi\}\_\{t\}^\{f\}observed after prediction exchange\.

### Scope of the Empirical Evidence

The experimental evidence supports our theoretical statements\. First, a random\-edge KD event contracts a common prediction\-space disagreement measure for homogeneous, width\-heterogeneous, and mixed\-family rosters\. Second, the synthetic calibration verifies the rate estimator, and the late dynamic exponents for the shared\-skeleton main runs are compatible with the inverse\-time transient in the theorem\. Third, the learning\-rate sweep exhibits the predicted finite\-horizon tradeoff between slow contraction at smallη\\etaand the observed terminal neighbourhood at largerη\\eta\. Fourth, the mechanism control shows that the consensus reduction requires peer exchange rather than local task training alone\.

The evidence does not identify the numerical values ofBfB\_\{f\},ζf\\zeta\_\{f\}, or the constants in Equation[H69](https://arxiv.org/html/2609.01952#A8.E69)\. It also does not prove the assumptions A4 and A10 as grounded truth\. Their role is to provide mathematical bridges in the proof\. We evaluate the observables produced by those bridges, check the rate\-diagnostic implementation on known signals, vary capacity and architecture family, and reports the finite\-horizon behaviour\.