TallyTrain: Communication-Efficient Federated Distillation

# TallyTrain: Communication-Efficient Federated Distillation
Source: [https://arxiv.org/html/2607.00173](https://arxiv.org/html/2607.00173)
\\jmlryear

2026\\jmlrworkshop

\\NameRadhakrishna Achanta\\Emailrachanta@cisco\.com \\NameWill Reed\\Emailwilreed@cisco\.com \\addrCisco Systems Inc

###### Abstract

Federated learning is bandwidth\-bound on two orthogonal axes: model size, which limits how often parameter\-averaging methods can afford to merge, and class count, which makes per\-probe soft\-label distillation prohibitive at large vocabularies\. Both ceilings tighten as modern systems scale\. We collapse the class\-count axis to⌈log2⁡C⌉\\lceil\\log\_\{2\}C\\rceilbits per probe by transmitting only each peer’sarg⁡max\\arg\\maxclass index, whereCCis the number of output classes\. The resulting protocol, TallyTrain, is not merely compressed: under non\-IID training it can be*preferable*to soft\-label distillation, because under\-trained peers are confidently wrong and majority voting filters this noise where soft\-label averaging amplifies it\. Across standard benchmarks, TallyTrain matches or beats soft\-label distillation at up to three orders of magnitude less communication\. We also relax the model\-size axis: we compose the cheap hard\-label consensus with sparse parameter merges to obtain a bandwidth\-bridge variant, which Pareto\-dominates every tested operating point of the standard FedAvg, FedProx and FedDF baselines\.

###### keywords:

federated learning; knowledge distillation; communication efficiency; hard\-label consensus; decentralized training; non\-IID

## 1Introduction

Federated learning \(FL\) trains a model from data shards held by many peers without centralizing the raw data\. The two dominant paradigms differ in*what*they communicate\.

Parameter\-space methodsexchange model parameters \(FedAvgmcmahan2017communication\) or per\-round weight deltas that play the role of an outer\-loop pseudo\-gradient \(DiLoCodouillard2023diloco\)\. Both have per\-round bandwidthΘ​\(\|W\|\)\\Theta\(\|W\|\), where\|W\|\|W\|is the number of model parameters, which is impractical for billion\-parameter models on edge or mobile substrates\.

Function\-space methods\(FedMDfedmd2019\) exchange per\-example predictions on a shared public probe set\. Bandwidth scales asΘ​\(C⋅\|𝒟pub\|\)\\Theta\(C\\\!\\cdot\\\!\|\\mathcal\{D\}\_\{\\mathrm\{pub\}\}\|\)per peer per round, whereCCis the number of output classes – independent of model size, but growing linearly withCC, which becomes prohibitive for large\-vocabulary tasks \(BPE\-tokenized language models routinely useC∈\[2,048,50,000\]C\\in\[2\{,\}048,50\{,\}000\]\)\.

These paradigms share a structural assumption: bandwidth is reduced along the*frequency*axis, by communicating less often, while the size of each message is treated as fixed \(full weights, or a fullCC\-dimensional softmax\)\. This paper takes the orthogonal*size*axis\. We keep communication frequent but make each message tiny: only thearg⁡max\\arg\\maxclass index per probe, one byte forC≤256C\\\!\\leq\\\!256, two bytes forC≤65,536C\\\!\\leq\\\!65\{,\}536\. Argmax voting acts as a noise filter – when peers agree, the consensus is reliable, and when they disagree, voting averages out individual error rather than amplifying it \(which is what the soft\-label expectation does when peers are simultaneously under\-trained and confident\)\. We call the resulting protocolTallyTrain– each round, peers*tally*their argmax votes on the public probe set into a single consensus histogram and then*train*against it – and show that, combined with sparse parameter merges, it dominates the bandwidth–accuracy Pareto frontier of federated learning across two modalities and three class counts\. Our contributions are as follows:

\(1\) A hard\-label communication primitive\.We introduce TallyTrain and argue that the voting histogram ofNNpeers’ argmax predictions over a public probe set is a valid – and surprisingly powerful – consensus distribution for distillation \(§[4\.1](https://arxiv.org/html/2607.00173#S4.SS1)\)\.

\(2\) Hard labels match or beat soft labels across modalities\.Thearg⁡max\\arg\\maxchannel carries essentially all the signal that soft labels do, at a per\-probe bandwidth ratio that grows linearly inCC\(§[4\.1](https://arxiv.org/html/2607.00173#S4.SS1), §[4\.4](https://arxiv.org/html/2607.00173#S4.SS4), §[4\.6](https://arxiv.org/html/2607.00173#S4.SS6)\)\.

\(3\) Two operational regimes\.The same primitive supports a*purely function\-space*mode \(α\>0\\alpha\\\!\>\\\!0when probes carry labels,α=0\\alpha\\\!=\\\!0with KL\-decay otherwise; §[4\.1](https://arxiv.org/html/2607.00173#S4.SS1), §[G](https://arxiv.org/html/2607.00173#A7)\) and a*function\-space stabilizer for parameter\-space averaging*– the bandwidth\-bridge variant TallyTrain\+faMMthat interleaves sparse FedAvg merges with the cheap hard\-label channel \(§[4\.2](https://arxiv.org/html/2607.00173#S4.SS2), §[4\.6](https://arxiv.org/html/2607.00173#S4.SS6)\)\. Against themcmahan2017communicationCIFAR default \(E≈5E\\\!\\approx\\\!5, fa≈400\\\!\\approx\\\!400\), the bridge halves the parameter\-channel bandwidth and*improves*accuracy; against fa=1\\\!=\\\!1it cuts it by two orders of magnitude\.

\(4\) A contractive theory of hard\-label distillation:a function\-space contraction lemma, a Condorcet bound on the voted top\-1, and a variance\-reduction bound on the distillation gradient \(§[3](https://arxiv.org/html/2607.00173#S3)\)\.

#### Scope:

Bandwidth is the binding resource\. We do not amortize training compute: each peer holds a full model and runs independent local SGD, so total FLOPs scale linearly inNN\. We evaluate under a full\-mesh topology withN≤10N\\\!\\leq\\\!10, which is trajectory\-equivalent to a centralized relay \(§[3](https://arxiv.org/html/2607.00173#S3)\); sparse\-gossip extensions that scale to thousands of peers are discussed but not evaluated here\.

## 2Related Work

Distributed and federated learning methodskairouz2019advancesdiffer not only in topology \(centralized versus peer\-to\-peer\), but also in the*object of communication*\. Classical decentralized federated learning \(DFL\) exchanges parameters, gradients, or compressed model deltas among neighboring clients\. Distillation\-based methods instead communicate*prediction\-space objects*such as logits, softened targets, prototypes, centroids, or function values, thereby decoupling collaboration from exact parameter alignment\. This distinction is especially important in heterogeneous and decentralized settings, where clients may differ in architecture, capacity, data distribution, availability, and communication budget\. Below, we review the most relevant literature through this lens, beginning with the distillation lineage and then positioning it relative to recent optimization\-centric DFL methods\.

### 2\.1Knowledge Distillation as a Collaboration Primitive

Knowledge distillation \(KD\) was introduced as a teacher\-student paradigm in which a student learns from a stronger teacher’s softened predictionshinton2015distilling\. The relevant implication for distributed learning is that collaboration can happen in*function space*: the shared object is a predictive distribution rather than a weight vector, which accommodates heterogeneous models and can in principle reduce communication if only compact predictive summaries are exchanged\.*Federated distillation*\(FD\)jeong2018communicationis the early form of this idea, exchanging averaged per\-class logits over the federation\.*FedMD*fedmd2019formalises the public\-probe\-set version: clients hold private data and possibly different architectures but share a small public reference set, repeatedly evaluate their models on it, and use the aggregated logits as distillation targets\.*FedDF*lin2020ensembleextends this to ensemble distillation: a server\-side student is trained on the average of clients’ soft predictions on an unlabelled public set, but the per\-probe payload is still a length\-CCfloat vector\.*Cronus*chang2019cronusis closer in spirit to our primitive: it pairs black\-box knowledge transfer with hard predictions over a public set, and identifies the resulting bandwidth and privacy advantages\.*FedKD*fedkd2021retains the KD viewpoint but compresses the communicated object to compact student\-model artifacts\. These methods are server\-coordinated; together they establish that distillation can bridge heterogeneity and reduce communication, but none gives a fully decentralized protocol whose per\-probe payload scales withlog2⁡C\\log\_\{2\}Crather thanCC\.

### 2\.2Decentralized Distillation for Heterogeneous Learning

Several lines of work transplant the FedMD intuition into peer\-to\-peer settings\.*DFML*dfml2024is serverless and avoids public auxiliary data: neighboring clients distill from each other directly with a cyclic supervised/distillation schedule, at the cost of sensitivity to neighborhood quality and local data imbalance\.*DeSA*desa2024reconstructs a shared alignment substrate through*synthetic*anchors, providing a common medium for distillation across heterogeneous models without real public data, while paying a synthesis cost and inheriting anchor\-quality risk\.*DFedCAD*dfedcad2025replaces full predictions with class\-centroid summaries, attractive when bandwidth or straggler synchronization dominates but coarser in the uncertainty information transmitted\. The modern decentralized\-distillation design space therefore decomposes by what is shared \(real public anchors, peer predictions, synthetic anchors, centroids, or function\-space agreement terms\), with each choice carrying different assumptions about auxiliary data, heterogeneity, topology, and communication\.

### 2\.3Optimization\-Centric Decentralized Federated Learning

In parallel with the distillation literature, a strong line of work has improved DFL through better optimization, topology design, personalization, and privacy, while remaining largely in parameter space\. DSpodFLdspodfl2025, DPFLdpfl2025, NTK\-DFLntkdfl2025, FedSPDfedspd2025, GFlatgflat2026, PaMEpame2026, and Spod\-GTspodgt2026respectively contribute sporadic\-communication convergence guarantees, learned collaboration graphs, NTK\-based stabilization, soft\-cluster personalization, flat\-minima generalization, partial\-message bandwidth reduction, and gradient tracking on directed graphs\. They are preferable to distillation when clients can share a common model family and the main challenge is systems efficiency, but their parameter\-space messages presuppose that the transmitted coordinates mean comparable things across clients, which limits cross\-architecture transfer\. A complementary strand expands the scope of DFL beyond accuracy and convergence:*f\-DP for DFL*fdpdfl2025on privacy\-utility trade\-offs,*Competitive Advantage Attacks*compadvdfl2025on strategic manipulation, and*Learning in Orbit*orbitdfl2026on satellite\-network topology\.

#### Positioning TallyTrain\.

Decentralized Hard\-Label Federated Distillation \(TallyTrain\) targets the engineering bottlenecks left open above\. Unlike FedAvg or DiLoCo it operates entirely in function space, sidestepping the weight\-merging barrier on non\-IID data\. Unlike FedMD/FedDF/Cronus it transmits a single 1\-bytearg⁡max\\arg\\maxindex per probe rather than length\-CClogits\. Unlike DFML and other peer\-to\-peer mutual\-distillation protocols, the inter\-peer payload is*only*the 1\-byte hard prediction; no soft signals, anchors, or parameters cross the wire\. This makes the protocol especially attractive whenever\|W\|\|W\|orCCis large enough that float\-vector or parameter exchange dominates the communication budget \(§[4\.6](https://arxiv.org/html/2607.00173#S4.SS6), §[4\.4](https://arxiv.org/html/2607.00173#S4.SS4)\)\.

## 3Method

FedAvgweightsΘ​\(\|W\|\)\\Theta\(\|W\|\)bytesDiLoCopseudo\-gradientsΘ​\(\|W\|\)\\Theta\(\|W\|\)bytesFedMDsoft labels4​C4\\,Cbytes/probeTallyTrainargmax votes11byte/probe

Figure 1:Federated learning paradigms ordered by per\-round payload\. Arrows colour\-code the exchanged object: weights \(black,Θ​\(\|W\|\)\\Theta\(\|W\|\)\), pseudo\-gradients \(red,Θ​\(\|W\|\)\\Theta\(\|W\|\)\), soft labels \(orange,4​C4CB/probe\), argmax votes \(green,11B/probe\)\. All four panels are drawn on the same star topology to make the visual comparison about payload only; topology is orthogonal to the payload choice and any of the four protocols runs on full\-mesh, ring or gossip without algorithmic change \(§[3](https://arxiv.org/html/2607.00173#S3)\)\.TallyTrain is built around a single design choice – argmax voting over a shared public probe set – with two operating variants that compose with it: a labelled\-public\-set hybrid that anchors peers to ground truth, and a bandwidth\-bridge variant that interleaves periodic FedAvg parameter merges\. Figure[1](https://arxiv.org/html/2607.00173#S3.F1)situates the primitive against existing federated learning paradigms by per\-round payload\. Under full\-mesh and a centralized relay the primitive is trajectory\-equivalent\.

### 3\.1Function\-space alignment

Let𝒟n\\mathcal\{D\}\_\{n\}denote peernn’s private non\-IID shard and𝒟pub\\mathcal\{D\}\_\{\\mathrm\{pub\}\}a public probe set shared by all peers, with each peer holdingfn​\(⋅;Wn\):𝒳→ΔC−1f\_\{n\}\(\\cdot;W\_\{n\}\):\\mathcal\{X\}\\\!\\to\\\!\\Delta^\{C\-1\}\. Weight\-averagingWg=1N​∑nWnW\_\{g\}\\\!=\\\!\\tfrac\{1\}\{N\}\\sum\_\{n\}W\_\{n\}inherits the loss\-surface mismatch endemic to non\-linear models on non\-IID data\. TallyTrain aligns the*functions*on𝒟pub\\mathcal\{D\}\_\{\\mathrm\{pub\}\}instead: two models that agree on allx∈𝒟pubx\\\!\\in\\\!\\mathcal\{D\}\_\{\\mathrm\{pub\}\}implement the same predictor on any test distribution dominated by it\.

### 3\.2The communication primitive: argmax voting

For eachx∈𝒟pubx\\in\\mathcal\{D\}\_\{\\mathrm\{pub\}\}, peernnbroadcasts only the top\-1 prediction

yn​\(x\)=arg⁡max⁡fn​\(x;Wn\)∈\{1,…,C\}\.y\_\{n\}\(x\)=\\arg\\max f\_\{n\}\(x;W\_\{n\}\)\\in\\\{1,\\dots,C\\\}\.\(1\)A class index requires⌈log2⁡C⌉\\lceil\\log\_\{2\}C\\rceilbits, transmitted byte\-aligned \(bC=1b\_\{C\}\\\!=\\\!1B forC≤256C\\\!\\leq\\\!256,22B forC≤65,536C\\\!\\leq\\\!65\{,\}536\)\. Against4​C4C\-byte 32\-bit soft labels the implementation ratio isρC=4​C/bC\\rho\_\{C\}\\\!=\\\!4C/b\_\{C\}\(ρ10=40,ρ100=400,ρ2048=4096\\rho\_\{10\}\\\!=\\\!40,\\rho\_\{100\}\\\!=\\\!400,\\rho\_\{2048\}\\\!=\\\!4096, Figure[5](https://arxiv.org/html/2607.00173#S4.F5)\); the bit\-packed lower bound isρC⋆=32​C/⌈log2⁡C⌉\\rho\_\{C\}^\{\\star\}\\\!=\\\!32C/\\lceil\\log\_\{2\}C\\rceil\. The saving grows linearly inCC\.

Across peers, the hard\-label predictions form the empirical voting histogram

H¯​\(x\)=1N​∑n=1Neyn​\(x\)∈ΔC−1,\\bar\{H\}\(x\)=\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}e\_\{y\_\{n\}\(x\)\}\\in\\Delta^\{C\-1\},\(2\)whereece\_\{c\}is thecc\-th basis vector\.H¯​\(x\)\\bar\{H\}\(x\)is a valid probability distribution and serves as the consensus target for distillation\.

#### Decentralized≡\\equivcentralized in trajectory\.

Under full\-mesh, each peer locally averages\{ym​\(x\)\}\\\{y\_\{m\}\(x\)\\\}to obtain the sameH¯​\(x\)\\bar\{H\}\(x\)a centralized relay would compute, so the per\-peer distillation gradient is identical and the two topologies share their optimization trajectory; they differ only in the wire pattern \(all\-to\-all vs\. star\)\. All numbers in §[4](https://arxiv.org/html/2607.00173#S4)are therefore simultaneously valid for both interpretations\. Per\-peer per\-probe bandwidth scales as2​\(N−1\)2\(N\\\!\-\\\!1\)bytes for decentralized hard labels vs\.8​C8Cbytes for a centralized soft\-label star, so hard labels are cheaper wheneverN≤4​C\+1N\\\!\\leq\\\!4C\\\!\+\\\!1\(e\.g\.,N≤41N\\\!\\leq\\\!41forC=10C\\\!=\\\!10,N≤4001N\\\!\\leq\\\!4001forC=1000C\\\!=\\\!1000\); beyond that, sparse\-gossip topologies are required \(§[5\.3](https://arxiv.org/html/2607.00173#S5.SS3)\)\.

### 3\.3Theoretical analysis

Three results explain TallyTrain’s behavior; full statements, assumption lists and proofs are in Appendix[B](https://arxiv.org/html/2607.00173#A2)\. Throughout, letH¯\(r\)​\(x\)=1N​∑nearg⁡max⁡fn\(r\)​\(x\)\\bar\{H\}^\{\(r\)\}\(x\)=\\frac\{1\}\{N\}\\sum\_\{n\}e\_\{\\arg\\max f\_\{n\}^\{\(r\)\}\(x\)\}be the voting histogram andDKL¯\(r\)\\overline\{D\_\{\\mathrm\{KL\}\}\}^\{\(r\)\}the mean pairwise function\-space disagreement1N​\(N−1\)​∑m≠n𝔼x​DKL​\(fn\(r\)∥fm\(r\)\)\\frac\{1\}\{N\(N\-1\)\}\\sum\_\{m\\neq n\}\\mathbb\{E\}\_\{x\}\\,D\_\{\\mathrm\{KL\}\}\(f\_\{n\}^\{\(r\)\}\\\|f\_\{m\}^\{\(r\)\}\)\.

#### Lemma 1 \(Function\-space contraction\)\.

Under standardLL\-smoothness, bounded SGD variance and non\-degenerate predictors, withη​L≤1/2\\eta L\\\!\\leq\\\!1/2,

𝔼​\[DKL¯\(r\+1\)\]≤\(1−\(1−α\)​η​μ\)​DKL¯\(r\)\+η2​σ2β0,\\mathbb\{E\}\\\!\\left\[\\,\\overline\{D\_\{\\mathrm\{KL\}\}\}^\{\(r\+1\)\}\\,\\right\]\\leq\\big\(1\-\(1\-\\alpha\)\\,\\eta\\,\\mu\\big\)\\,\\overline\{D\_\{\\mathrm\{KL\}\}\}^\{\(r\)\}\+\\tfrac\{\\eta^\{2\}\\sigma^\{2\}\}\{\\beta\_\{0\}\},\(3\)so peers contract toward agreement on𝒟pub\\mathcal\{D\}\_\{\\mathrm\{pub\}\}; predictors withDKL¯→0\\overline\{D\_\{\\mathrm\{KL\}\}\}\\\!\\to\\\!0are indistinguishable on any test distribution dominated by it\.

#### Proposition 1 \(Voting accuracy; Condorcet\)\.

If each peer’s top\-1 accuracy exceedsp¯\>1/2\\bar\{p\}\>1/2and the\{arg⁡max⁡fn​\(x\)\}\\\{\\arg\\max f\_\{n\}\(x\)\\\}are pairwise independent, the majority vote hits the true classccwith probability≥1−exp⁡\(−2​N​\(p¯−1/2\)2\)\\geq 1\-\\exp\(\-2N\(\\bar\{p\}\-1/2\)^\{2\}\)\. We usep¯\>1/2\\bar\{p\}\>1/2as the empirical trigger for activating the consensus channel after warm\-up\.

#### Proposition 2 \(Hard\-label variance reduction\)\.

For peer top\-1 margins≥γ\\geq\\gamma, the hard\-label distillation gradient satisfiesVar​\[ghard\]≤Var​\[gsoft\]\+ℬ​\(p¯,γ\)\\mathrm\{Var\}\[g\_\{\\mathrm\{hard\}\}\]\\leq\\mathrm\{Var\}\[g\_\{\\mathrm\{soft\}\}\]\+\\mathcal\{B\}\(\\bar\{p\},\\gamma\)withℬ→0\\mathcal\{B\}\\to 0asp¯,γ→1\\bar\{p\},\\gamma\\to 1\. Argmax truncates the high\-entropy tails of under\-trained peers’ soft outputs; the quantization costℬ\\mathcal\{B\}vanishes once top\-1 margins separate\.

### 3\.4Algorithm and operating axes

Algorithm[1](https://arxiv.org/html/2607.00173#algorithm1)is a single per\-peer procedure with two orthogonal operating axes that are selected by setting the KL weightα\\alpha\(with optional KL\-decay\) and the FedAvg cadenceMM\.

Input:private data

𝒟n\\mathcal\{D\}\_\{n\}, public probes

𝒟pub\\mathcal\{D\}\_\{\\mathrm\{pub\}\}, rounds

RR, local steps

KK, sample size

BsampleB\_\{\\mathrm\{sample\}\}, warm\-up

WW, KL weight

α\\alpha, optional KL decay window

TT, optional FedAvg cadence

MM
Initialize local weights

WnW\_\{n\}
for*roundr=1,…,Rr=1,\\dots,R*do

for*k=1,…,Kk=1,\\dots,K*do

Sample

\(x,y\)∼𝒟n\(x,y\)\\sim\\mathcal\{D\}\_\{n\}
//local SGD on private data

Wn←Wn−η​∇ℒCE​\(f​\(x;Wn\),y\)W\_\{n\}\\leftarrow W\_\{n\}\-\\eta\\,\\nabla\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(f\(x;W\_\{n\}\),y\)
end for

if*r\>Wr\>W*then

Sample

𝒮r⊂𝒟pub\\mathcal\{S\}\_\{r\}\\subset\\mathcal\{D\}\_\{\\mathrm\{pub\}\},

\|𝒮r\|=Bsample\|\\mathcal\{S\}\_\{r\}\|\\\!=\\\!B\_\{\\mathrm\{sample\}\}
//consensus channel

Broadcast

Yn=arg⁡max⁡f​\(𝒮r;Wn\)Y\_\{n\}=\\arg\\max f\(\\mathcal\{S\}\_\{r\};W\_\{n\}\); receive

\{Ym\}m≠n\\\{Y\_\{m\}\\\}\_\{m\\neq n\}
//1 byte/probe

H¯←1N​∑meYm\\bar\{H\}\\leftarrow\\tfrac\{1\}\{N\}\\sum\_\{m\}e\_\{Y\_\{m\}\}on

𝒮r\\mathcal\{S\}\_\{r\}
//voting histogram

λr←KL​\-​decay​\(r;W,T\)\\lambda\_\{r\}\\leftarrow\\mathrm\{KL\\text\{\-\}decay\}\(r;W,T\)
//

≡1\\equiv 1ifTTunset

ℒ=α​ℒCE​\(f​\(𝒮r;Wn\),ypub\)\+λr​\(1−α\)​DKL​\(H¯∥f​\(𝒮r;Wn\)\)\\mathcal\{L\}=\\alpha\\,\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\(f\(\\mathcal\{S\}\_\{r\};W\_\{n\}\),y\_\{\\mathrm\{pub\}\}\)\+\\lambda\_\{r\}\(1\\\!\-\\\!\\alpha\)\\,D\_\{\\mathrm\{KL\}\}\(\\bar\{H\}\\\|f\(\\mathcal\{S\}\_\{r\};W\_\{n\}\)\)
Wn←Wn−η​∇ℒW\_\{n\}\\leftarrow W\_\{n\}\-\\eta\\,\\nabla\\mathcal\{L\}
end if

if*M\>0M\>0andrmodM=0r\\bmod M=0*then

Wn←1N​∑m=1NWmW\_\{n\}\\leftarrow\\tfrac\{1\}\{N\}\\sum\_\{m=1\}^\{N\}W\_\{m\}
//Axis B: FedAvg merge

end if

end for

return

WnW\_\{n\}

Algorithm 1TallyTrain \(per\-peer, roundrr\)

#### Axis A – pure function\-space distillation \(M=0M\\\!=\\\!0\)\.

The protocol runs entirely on the hard\-label channel\. Two sub\-modes cover the public\-set assumption\.

*A\.1 Labelled probes \(α\>0\\alpha\\\!\>\\\!0, no decay\)\.*When𝒟pub\\mathcal\{D\}\_\{\\mathrm\{pub\}\}carries ground\-truth labels, the CE term anchors each peer above the Condorcet thresholdp¯\>1/2\\bar\{p\}\\\!\>\\\!1/2, so Lemma 1’s contraction has a non\-degenerate fixed point and runs converge stably\. Empiricallyα=0\.5\\alpha\\\!=\\\!0\.5transfers across CIFAR\-10/100 without retuning \(Tables[1](https://arxiv.org/html/2607.00173#S4.T1)&[2](https://arxiv.org/html/2607.00173#S4.T2), top TallyTrain row\)\.

*A\.2 Unlabelled probes \(α=0\\alpha\\\!=\\\!0, KL\-decay\)\.*When𝒟pub\\mathcal\{D\}\_\{\\mathrm\{pub\}\}has no useful labels \(e\.g\. drawn from a different distribution\), the CE anchor is absent and naïve pure\-KL training drifts into Condorcet\-class collapse\. A linear decayλr=max⁡\(0,1−\(r−W\)/T\)\\lambda\_\{r\}\\\!=\\\!\\max\(0,1\\\!\-\\\!\(r\\\!\-\\\!W\)/T\)forr\>Wr\\\!\>\\\!W, withλr=0\\lambda\_\{r\}\\\!=\\\!0thereafter, halts the drift while still letting the consensus channel transfer useful knowledge; the choice ofTTis robust across an order of magnitude \(Appendix[G](https://arxiv.org/html/2607.00173#A7)\)\.

#### Axis B – function\-space stabilizer for parameter\-space averaging \(M\>0M\\\!\>\\\!0\)\.

Pure function\-space distillation plateaus below FedAvg’s accuracy ceiling: hard\-label voting cannot transfer the rich feature representations parameter averaging does\. SettingM\>0M\\\!\>\\\!0interleaves a FedAvg merge everyMMrounds, lifting peers toward the parameter\-space ceiling while the hard\-label channel keeps them tightly synchronized between merges\. The combination has substantially lower cross\-peer drift than FedAvg at the sameMMand reaches strictly higher tail accuracy at the same bandwidth\. We write*faMM*for ”FedAvg everyMMrounds” and use it for both the bridge variant TallyTrain\+faMMand the parameter\-only baseline FedAvg\-faMM\(Tables[2](https://arxiv.org/html/2607.00173#S4.T2)&[3](https://arxiv.org/html/2607.00173#S4.T3); Pareto frontier in Figure[4](https://arxiv.org/html/2607.00173#S4.F4)\)\. On our CIFAR\-10 non\-IID setup,fa=500\\mathrm\{fa\}\\\!=\\\!500is the McMahan local\-epoch default \(E≈6\.4E\\\!\\approx\\\!6\.4\);fa=200\\mathrm\{fa\}\\\!=\\\!200is∼2\.5×\\sim\\\!2\.5\\timesmore frequent\. The fullM↔EM\\\!\\leftrightarrow\\\!Ecorrespondence is in Appendix[A](https://arxiv.org/html/2607.00173#A1)\.

The two axes are orthogonal: any A\-mode composes with anyMM\. Most deployments fall into one of three combinations – A\.1\+\+M=0M\\\!=\\\!0\(§[4\.1](https://arxiv.org/html/2607.00173#S4.SS1)\), A\.2\+\+M=0M\\\!=\\\!0\(§[4\.5](https://arxiv.org/html/2607.00173#S4.SS5)/Appendix[G](https://arxiv.org/html/2607.00173#A7)\), and A\.1\+\+M\>0M\\\!\>\\\!0\(§[4\.2](https://arxiv.org/html/2607.00173#S4.SS2), §[4\.6](https://arxiv.org/html/2607.00173#S4.SS6)\); the latter is our recommended deployment when absolute accuracy matters more than the lowest possible bandwidth\.

### 3\.5Design rules

Five hyperparameters drive deployment, with the following empirical defaults that we use throughout\. The distillation step operates on a stochastic subset of sizeBsampleB\_\{\\mathrm\{sample\}\}, so per\-peer per\-round egress is\(N−1\)​Bsample\(N\\\!\-\\\!1\)B\_\{\\mathrm\{sample\}\}bytes regardless of\|𝒟pub\|\|\\mathcal\{D\}\_\{\\mathrm\{pub\}\}\|; the consensus channel is disabled for the firstWWwarm\-up rounds so peers clear the Condorcet threshold before influencing each other \(W=300W\\\!=\\\!300in\-distribution,W=1500W\\\!=\\\!1500cross\-distribution\)\. Local stepsK=5K\\\!=\\\!5: weight\-exchange methods are forced to largeKKto amortizeΘ​\(\|W\|\)\\Theta\(\|W\|\)sync cost, but TallyTrain removes that constraint\(stich2018local;khaled2020tighter\)\. Public set\|𝒟pub\|=2,000\|\\mathcal\{D\}\_\{\\mathrm\{pub\}\}\|\\\!=\\\!2\{,\}000for CIFAR\-10 and10,00010\{,\}000for CIFAR\-100, sized so𝒟pub\\mathcal\{D\}\_\{\\mathrm\{pub\}\}ϵ\\epsilon\-covers the test manifold\. Sub\-sampleBsampleB\_\{\\mathrm\{sample\}\}is best read as a regularization knob: empirically the per\-class coverage rateρ=Bsample/C∈\[1,2\]\\rho\\\!=\\\!B\_\{\\mathrm\{sample\}\}/C\\in\[1,2\]is optimal, with the sweep in Appendix[F](https://arxiv.org/html/2607.00173#A6)\. Number of peersNN: Proposition 1 saturates atN≈1/\(2​\(p¯−1/2\)2\)N\\\!\\approx\\\!1/\(2\(\\bar\{p\}\\\!\-\\\!1/2\)^\{2\}\)\(≈20\\approx\\\!20forp¯≈0\.6\\bar\{p\}\\\!\\approx\\\!0\.6\); the hard\-label primitive itself remains cheaper than soft labels up to the crossoverN≤4​C\+1N\\\!\\leq\\\!4C\\\!\+\\\!1derived in §[3\.2](https://arxiv.org/html/2607.00173#S3.SS2)\.

## 4Experiments

We evaluate TallyTrain against soft\-label distillation \(FedMD\(fedmd2019\)\), centralized ensemble distillation \(FedDF\(lin2020ensemble\)\), parameter\-averaging baselines \(FedAvg\(mcmahan2017communication\), FedProx\(li2018federated\)\), and a no\-communication local\-only baseline on CIFAR\-10/100 with Dirichlet non\-IID splits \(N=10N\\\!=\\\!10peers, ResNet\-18, AdamWη=10−3\\eta\\\!=\\\!10^\{\-3\},K=5K\\\!=\\\!5,R=3000R\\\!=\\\!3000\)\. Each multi\-seed cell uses seeds\{0,1,2\}\\\{0,1,2\\\}and we report tail mean±\\pmstd over the last 100 logged rounds \(CIFAR\) or 600 rounds \(WikiText\-2, three full fa=200=200merge cycles\)\. Peers use distinct per\-peer init RNG so voting has a non\-degenerate signal at round zero\. Full hyperparameters in Appendix[A](https://arxiv.org/html/2607.00173#A1)\.

### 4\.1Hard labels match or beat soft labels atC×C\\\!\\timesless bandwidth

Table[1](https://arxiv.org/html/2607.00173#S4.T1)reports the labelled\-hybrid cell on CIFAR\-100 non\-IID \(α=0\.5\\alpha\\\!=\\\!0\.5,W=300W\\\!=\\\!300,Bsample=32B\_\{\\mathrm\{sample\}\}\\\!=\\\!32,\|𝒟pub\|=10,000\|\\mathcal\{D\}\_\{\\mathrm\{pub\}\}\|\\\!=\\\!10\{,\}000\)\. TallyTrain and FedMD share the labelled\-hybrid loss; theα=1\\alpha\\\!=\\\!1row is the public\-CE\-only baseline that isolates each method’s consensus contribution\.

Table 1:CIFAR\-100 non\-IID hybrid, 3 seeds each\. Theα=1\.0\\alpha=1\.0row is the public\-CE\-only baseline \(consensus weight is zero, so the result is identical regardless of how consensus is computed\)\. TallyTrain beats FedMD by 1\.35 pp at400×400\\timesless bandwidth\. Hard\-label consensus contributes\+1\.13\+1\.13pp over no\-consensus; soft\-label consensus contributes−0\.22\-0\.22pp \(slightly hurts\)\.We make three findings: \(i\) TallyTrain beats FedMD by1\.351\.35pp at400×400\\timesless bandwidth – about5×5\\timesthe per\-method seed std, with TallyTrain’s tail above FedMD’s peak\. \(ii\)*Soft\-label consensus contributes nothing on CIFAR\-100*: theα=1\\alpha\\\!=\\\!1no\-consensus baseline \(32\.03%32\.03\\%\) sits0\.220\.22pp*above*FedMD; the soft\-label average is dragged down by confidently\-wrong under\-trained peers, while hard\-label voting filters them and recovers\+1\.13\+1\.13pp\. \(iii\) Richer mechanisms \(disagreement\-weighted distillation, asymmetric consensus\) add nothing over uniform argmax voting \(Appendix[D](https://arxiv.org/html/2607.00173#A4)\); uniform voting is already at the function\-space ceiling\.

### 4\.2Bandwidth\-bridge variant: distillation as peer stabilizer

Table[2](https://arxiv.org/html/2607.00173#S4.T2)reports CIFAR\-10 non\-IID under the bandwidth\-bridge variant, which interleaves a FedAvg parameter merge everyMMrounds on top of the distillation channel\. We compare TallyTrain and FedMD \(withM=200M=200merges\) against pure FedAvg at four merge frequencies\.

Table 2:CIFAR\-10 non\-IID,N=10N\\\!=\\\!10, 3 seeds\. Cross\-peer std is the mean within\-run std of peer test accuracies over the last 100 rounds\. FedProx uses bestμ=0\.001\\mu\\\!=\\\!0\.001from a\{0\.001,0\.01,0\.1\}\\\{0\.001,0\.01,0\.1\\\}sweep \(Appendix[L](https://arxiv.org/html/2607.00173#A12)\); FedDF cited as\(lin2020ensemble\)\.†FedDF’s single server\-side student makes cross\-peer std formally zero\.‡fa500500is the McMahan CIFAR local\-epoch default \(E≈6\.4E\\\!\\approx\\\!6\.4, §[3\.4](https://arxiv.org/html/2607.00173#S3.SS4)\)\. TallyTrain\+faMM\(withM=200M\\\!=\\\!200\) beats every FedAvg/FedProx/FedDF cell \(\+21\.2\+21\.2pp over the McMahan default;\+19\.6\+19\.6pp over FedDF at214×214\\timesless bandwidth\) with the smallest seed std \(σ=0\.38\\sigma\\\!=\\\!0\.38\) and the lowest peer drift\.Figure[2](https://arxiv.org/html/2607.00173#S4.F2)visualizes the difference\. The mean accuracy curves for TallyTrain\+fa200 and FedMD\+fa200 ride together along the top, with thin shaded bands \(±1\\pm 1std across peers\)\. The FedAvg\-only curves are lower and surrounded by much wider bands; the distinctive “spike\-and\-recover” shape of FedAvg\-only at every merge is exactly the inter\-merge drift that distillation prevents\.

![Refer to caption](https://arxiv.org/html/2607.00173v1/x1.png)Figure 2:Peer stabilization on CIFAR\-10 non\-IID, 3 seeds\. Lines are peer means, bands are±\\pm1 cross\-peer std\. TallyTrain\+faMMand FedMD\+fa200 maintain thin bands throughout training; FedAvg\-fa200 at the same merge cadence oscillates between merges\. The same hard\-label channel that delivers the headline bandwidth saving of §[4\.1](https://arxiv.org/html/2607.00173#S4.SS1)doubles as a peer\-stabilization mechanism\.Distillation between merges anchors peers in function space, so a single merge can fully reconcile them\. The bridge is not a workaround for insufficient probe data: a separate sweep across in\-distribution, out\-of\-distribution and OOD\+\+anchor probe pools \(Figure[3](https://arxiv.org/html/2607.00173#S4.F3), full breakdown in Appendix[K](https://arxiv.org/html/2607.00173#A11)\) shows that pure TallyTrain saturates well below the bridge ceiling regardless of probe\-pool size\. The parameter channel is the missing channel\.

![Refer to caption](https://arxiv.org/html/2607.00173v1/x2.png)Figure 3:Pure TallyTrain on CIFAR\-10 non\-IID with three families of probe pools \(data from Tables[8](https://arxiv.org/html/2607.00173#A11.T8)–[10](https://arxiv.org/html/2607.00173#A11.T10)in Appendix[K](https://arxiv.org/html/2607.00173#A11), 3 seeds each\)\. In\-distribution probes saturate around67\.567\.5%; OOD probes alone plateau near4848%; OOD probes with a small in\-distribution anchor recover only to∼57\\sim\\\!57%\. The bridge variant TallyTrain\+faMM, even at\|𝒟pub\|=2\|\\mathcal\{D\}\_\{\\mathrm\{pub\}\}\|\\\!=\\\!2k, sits at71\.9271\.92% \(red dashed\); FedAvg\-fa50 at79\.679\.6% \(gray dotted\)\. Public\-data abundance is not what closes the gap to parameter averaging\.
### 4\.3Pareto frontier

Figure[4](https://arxiv.org/html/2607.00173#S4.F4)plots tail accuracy vs\. total per\-peer bandwidth \(log scale\)\. Pure TallyTrain anchors the low\-bandwidth end \(65\.0%65\.0\\%at0\.390\.39MB\); TallyTrain\+faMMand FedMD\+fa200 cluster at the top\-right and dominate every FedAvg/FedProx point at∼1\.2\\sim\\\!1\.2GB\. FedProx \(Appendix[L](https://arxiv.org/html/2607.00173#A12)\) is strictly dominated by FedAvg here\.

![Refer to caption](https://arxiv.org/html/2607.00173v1/x3.png)Figure 4:Bandwidth\-accuracy Pareto frontier, CIFAR\-10 non\-IID, 3 seeds \(error bars omitted when smaller than the marker\)\. Pure TallyTrain anchors the low\-bandwidth end; TallyTrain\+faMMdominates the upper\-right\.
### 4\.4Scaling with the number of output classesCC

The byte\-aligned per\-probe bandwidth ratio isρC=4​C/bC\\rho\_\{C\}\\\!=\\\!4C/b\_\{C\}withbC∈\{1,2\}b\_\{C\}\\\!\\in\\\!\\\{1,2\\\}, givingρ10=40\\rho\_\{10\}\\\!=\\\!40,ρ100=400\\rho\_\{100\}\\\!=\\\!400,ρ2048=4096\\rho\_\{2048\}\\\!=\\\!4096\(Figure[5](https://arxiv.org/html/2607.00173#S4.F5)\); the empirical Payload ratios in Tables[1](https://arxiv.org/html/2607.00173#S4.T1)&[3](https://arxiv.org/html/2607.00173#S4.T3)match exactly\. The bit\-packed lower boundρC⋆=32​C/⌈log2⁡C⌉\\rho\_\{C\}^\{\\star\}\\\!=\\\!32C/\\lceil\\log\_\{2\}C\\rceilgivesρ2048⋆≈5957\\rho\_\{2048\}^\{\\star\}\\\!\\approx\\\!5957\.

![Refer to caption](https://arxiv.org/html/2607.00173v1/x4.png)Figure 5:Per\-probe bandwidth ratioρC=4​C/bC\\rho\_\{C\}\\\!=\\\!4C/b\_\{C\}between 32\-bit soft labels and byte\-aligned hard\-label indices \(bC∈\{1,2\}b\_\{C\}\\in\\\{1,2\\\}\), with empirical anchors atC=10C\\\!=\\\!10\(CIFAR\-10\),C=100C\\\!=\\\!100\(CIFAR\-100\) andC=2048C\\\!=\\\!2048\(WikiText\-2 BPE vocabulary\)\. The dashed curve is the bit\-packed structural lower boundρC⋆=32​C/⌈log2⁡C⌉\\rho\_\{C\}^\{\\star\}\\\!=\\\!32C/\\lceil\\log\_\{2\}C\\rceil\.
### 4\.5Three secondary findings \(details in supplementary\)

#### Cross\-distribution KL\-decay \(§[G](https://arxiv.org/html/2607.00173#A7)\)\.

With CIFAR\-100 probes for a CIFAR\-10 task \(α=0\\alpha\\\!=\\\!0, no useful public labels\), TallyTrain with a linear KL\-decay window ofT=200T\\\!=\\\!200rounds reaches48\.35±2\.10%48\.35\\pm 2\.10\\%tail accuracy versus FedMD’s49\.45%49\.45\\%at40×40\\timesless bandwidth;T∈\[50,400\]T\\in\[50,400\]all hold within seed noise\. Without decay, both methods collapse to near\-random as the consensus drifts into a self\-reinforcing fixed point\.

#### Per\-class coverage rule forBsampleB\_\{\\mathrm\{sample\}\}\(Appendix[F](https://arxiv.org/html/2607.00173#A6)\)\.

The relevant scale isρ=Bsample/C\\rho\\\!=\\\!B\_\{\\mathrm\{sample\}\}/C, withρ∈\[1,2\]\\rho\\\!\\in\\\!\[1,2\]optimal across both CIFAR tasks:ρ<0\.5\\rho\\\!<\\\!0\.5starves the channel \(CIFAR\-10 tail65→37%65\\\!\\to\\\!37\\%atρ=0\.4\\rho\\\!=\\\!0\.4\), andρ\>5\\rho\\\!\>\\\!5*actively degrades*accuracy on low\-CCtasks because the distillation gradient over\-weights public\-set fit\.

#### Rank information beyond top\-1 helps negligibly \(Appendix[E](https://arxiv.org/html/2607.00173#A5)\)\.

RC\-FD withtop​\-​k∈\{1,3,5\}\\mathrm\{top\}\\text\{\-\}k\\\!\\in\\\!\\\{1,3,5\\\}on CIFAR\-10 hybrid gives tail accuracies64\.82,65\.20,65\.29%64\.82,65\.20,65\.29\\%: 1 byte captures nearly all information transmitted by the full softmax in the labelled regime\.

### 4\.6Cross\-modality validation: federated language modeling atC=2048C=2048

To test whether the size\-axis advantage transfers, we replicate the headline cells on next\-token prediction over WikiText\-2merity2016pointer:N=8N\\\!=\\\!8peers, a tiny GPT \(6 layers,d=256d\\\!=\\\!256,∼5\.84\\sim\\\!5\.84M parameters\) trained from scratch on disjoint chunks of the BPE\-2048 corpus, with5%5\\%of train held out as the public probe set\. Each peer transmits one prediction per token position;Bsample=32B\_\{\\mathrm\{sample\}\}\\\!=\\\!32length\-256256sequences gives81928192probe positions per round\. Full setup in Appendix[I](https://arxiv.org/html/2607.00173#A9); we report mean top\-1 accuracy and perplexity on the validation split\.

Table 3:WikiText\-2, BPE\-2048,N=8N\\\!=\\\!8, 3 seeds\. Tail metrics averaged over the last 600 rounds\. Pure\-distillation rows collapse in PPL \(tail44k–1010k\) but match within0\.330\.33pp on accuracy at the4096×4096\\timesbandwidth ratio\. TallyTrain\+\+fa200 beats FedAvg\-fa200 by\+1\.45\+1\.45pp at∼6%\\sim\\\!6\\%extra bandwidth – the same delta as on CIFAR\-10 \(Table[2](https://arxiv.org/html/2607.00173#S4.T2)\)\. All five cells haveσ≤0\.16\\sigma\\\!\\leq\\\!0\.16pp; the bridge hasσ=0\.03\\sigma\\\!=\\\!0\.03\.Three findings reproduce\. \(LM\-1\)*Hard≈\\approxsoft atC=2048C\\\!=\\\!2048*: TallyTrain\-pure \(15\.45±0\.03%15\.45\\pm 0\.03\\%\) matches FedMD\-pure \(15\.78±0\.02%15\.78\\pm 0\.02\\%\) within0\.330\.33pp while FedMD spent4096×4096\\timesmore bandwidth \(1\.271\.27TB vs0\.310\.31GB\), confirming the structural ratio32​V/⌈log2⁡V⌉32V/\\lceil\\log\_\{2\}V\\rceilon a new modality atCCtwo orders of magnitude larger than CIFAR\-10\. \(LM\-2\)*Pure distillation collapses on high\-entropy targets*: both methods peak near round 450 \(accuracy18\.4→15\.4%18\.4\\\!\\to\\\!15\.4\\%, perplexity127→10,087127\\\!\\to\\\!10\{,\}087for TallyTrain;113→3,622113\\\!\\to\\\!3\{,\}622for FedMD\); same Condorcet drift as the cross\-distribution image case \(§[G](https://arxiv.org/html/2607.00173#A7)\), now in\-distribution because rare\-token positions have intrinsically high entropy\. \(LM\-3\)*The bandwidth\-bridge variant transfers*: TallyTrain\+\+fa200 beats FedAvg\-fa200 by\+1\.45\+1\.45pp at∼6%\\sim\\\!6\\%extra bandwidth – the same delta as on CIFAR\-10\. Pure TallyTrain does not catch FedAvg\-fa1 in absolute accuracy on this task; the Pareto\-optimal low\-bandwidth point is TallyTrain\+\+fa200, not pure TallyTrain\.

## 5Discussion

### 5\.1Pure\-KL distillation has a single, identifiable failure mode – and the bridge fixes it

The failure mode is Condorcet\-class collapse\.Pure\-KL distillation collapses whenever peer agreement on the public probes is too weak for the contraction of Lemma 1 to converge to the truth: peers initially agree on accurate predictions, then drift into a self\-reinforcing fixed point unrelated to the test distribution\. We observe two flavours\. \(i\)*Cross\-distribution*\(§[4\.5](https://arxiv.org/html/2607.00173#S4.SS5), Appendix[G](https://arxiv.org/html/2607.00173#A7)\): the public set has no useful labels, the CE anchor is absent, and accuracy peaks around roundW\+T/2W\\\!\+\\\!T/2before collapsing\. \(ii\)*Same\-distribution but high\-entropy targets*\(§[4\.6](https://arxiv.org/html/2607.00173#S4.SS6)\): on WikiText\-2 BPE\-2048, peers genuinely disagree on rare\-token positions even with a CE term, so the consensus is partly noise and KL pressure pushes peers to confidently match wrong consensus – both TallyTrain and FedMD pure distillation peak around round 450 before perplexity explodes by 1–2 orders of magnitude\.The bridge variant fixes both cases without retuning\.Introducing a more reliable anchor – KL\-decay for cross\-distribution images, periodic FedAvg merges for the LM \(the bandwidth\-bridge variant of §[4\.2](https://arxiv.org/html/2607.00173#S4.SS2)\) – restores stability and yields the TallyTrain\+faMMcells that dominate Tables[2](https://arxiv.org/html/2607.00173#S4.T2)and[3](https://arxiv.org/html/2607.00173#S4.T3)\.

### 5\.2The bridge dominates parameter averaging on every operating point we tested

The bridge closes the absolute\-accuracy gap\.Pure TallyTrain plateaus∼15\\sim\\\!15pp below the parameter\-space ceiling on both modalities \(65%65\\%vs\. FedAvg\-fa5079\.6%79\.6\\%on CIFAR\-10;15\.4%15\.4\\%vs\. FedAvg\-fa129\.2%29\.2\\%on WikiText\-2\): function\-space distillation alone does not catch parameter exchange in absolute accuracy\. TallyTrain\+faMMcloses the gap and dominates every FedAvg/FedProx/FedDF cell we ran on both modalities \(Tables[2](https://arxiv.org/html/2607.00173#S4.T2),[3](https://arxiv.org/html/2607.00173#S4.T3)\); the WikiText\-2 win is the same\+1\.45\+1\.45pp at∼6%\\sim\\\!6\\%extra bandwidth that we see on CIFAR\-10\.

The bridge is also the most reproducible cell\.Across all 23 multi\-seed cells, the bandwidth\-bridge has the smallest seed variance:σ=0\.38\\sigma\\\!=\\\!0\.38pp on CIFAR\-10 \(≥1\.52\\geq\\\!1\.52for any FedAvg,≥4\.79\\geq\\\!4\.79for FedDF\) andσ=0\.03\\sigma\\\!=\\\!0\.03pp on WikiText\-2 – essentially deterministic\. Plurality voting is invariant to small logit perturbations and the merge step further contracts parameter spread\.

FedProx is dominated by vanilla FedAvg in this low\-bandwidth regime\.With its best proximal coefficient \(μ=0\.001\\mu\\\!=\\\!0\.001from a\{0\.001,0\.01,0\.1\}\\\{0\.001,0\.01,0\.1\\\}sweep\), FedProx is consistently1\.71\.7–2\.92\.9pp below FedAvg at every fa we tested; canonical FedProx with merge\-every\-round is outside our low\-bandwidth scope\. The mechanism is that the proximal anchor is refreshed only on merge rounds, so atfa≫1\\mathrm\{fa\}\\\!\\gg\\\!1it pulls peers toward an increasingly stale consensus over thefa⋅K\\mathrm\{fa\}\\\!\\cdot\\\!Kinter\-merge steps\. CIFAR\-100 replicates \(Appendix[L](https://arxiv.org/html/2607.00173#A12)\); the bridge beats the best FedProx cell by\+13\.1\+13\.1pp\.

FedDF’s tail collapses at this bandwidth budget\.FedDF\(lin2020ensemble\)reaches a respectable peak \(66\.0±1\.6%66\.0\\\!\\pm\\\!1\.6\\%\) but its tail collapses \(52\.3±4\.852\.3\\\!\\pm\\\!4\.8\) at the same bandwidth as FedAvg\-fa1\. Two failure modes interact: the server’s own distillation step is a function\-space objective on the same high\-entropy targets that crash pure distillation in our LM cell, and the centralized aggregator removes the cross\-peer voting filter TallyTrain relies on\. The bridge dominates FedDF by19\.619\.6pp tail accuracy at214×214\\timesless bandwidth\.

The hard\-label channel turns FedAvg’s known\-but\-unused frequency\-axis slack into a Pareto\-dominant operating point\.Ourfaaxis is the same local\-SGD averaging interval the originalmcmahan2017communicationFedAvg exposes throughEE, its number of local epochs per merge: on our CIFAR\-10 non\-IID setup,fa=500\\mathrm\{fa\}\\\!=\\\!500matches the McMahan CIFAR default \(E≈5E\\\!\\approx\\\!5\)\. The multi\-seed sweep in Table[2](https://arxiv.org/html/2607.00173#S4.T2)traces a smooth degradation curve \(FedAvg\-fa5060\.5±1\.560\.5\\\!\\pm\\\!1\.5, fa20054\.6±1\.854\.6\\\!\\pm\\\!1\.8, fa50050\.7±1\.9%50\.7\\\!\\pm\\\!1\.9\\%\), consistent with local\-SGD bounds\(stich2018local;khaled2020tighter\)\. TallyTrain\+faMM\(withM=200M\\\!=\\\!200\) at71\.92±0\.38%71\.92\\\!\\pm\\\!0\.38\\%and1\.251\.25GB beats every cell:\+11\.4\+11\.4pp over FedAvg\-fa50 \(4×4\\timesless bandwidth\),\+17\.3\+17\.3pp at the same bandwidth as FedAvg\-fa200, and\+21\.2\+21\.2pp over the McMahan\-default FedAvg\-fa500\. The contribution is not that low\-frequency FedAvg works – McMahan’s own ablations and local\-SGD theory predict that – but that the hard\-label channel cleanly fills the inter\-merge gap that low\-frequency FedAvg leaves open\.

### 5\.3Limitations and future work

Scope of this evaluation\.Headline cells useN≤10N\\\!\\leq\\\!10homogeneous\-architecture peers under full\-mesh averaging on Dirichlet\-0\.50\.5non\-IID splits – the harder and more practically relevant regime; we expect the IID case to be at least as favourable, but do not measure it here\. We do not validateN≫10N\\\!\\gg\\\!10, sparse\-gossip topologies, or heterogeneous architectures\.

Four extensions\.\(i\)*Sparse gossip*: the full\-mesh topology used here scales asΘ​\(N2\)\\Theta\(N^\{2\}\); sparse gossip would scale asΘ​\(N​log⁡N\)\\Theta\(N\\log N\)and make decentralization strictly bandwidth\-favorable, but we have not validated the hard\-label primitive under restricted neighbourhoods empirically\. \(ii\)*Adaptive consensus weighting*: an online estimator of voting margin could replace the fixed KL\-decay of Appendix[G](https://arxiv.org/html/2607.00173#A7)and unify the labelled, cross\-distribution and LM regimes\. \(iii\)*Heterogeneous architectures*: the hard\-label primitive is architecture\-free by construction, but the contraction lemma’s behaviour under model\-class mismatch is open\. \(iv\)*Production\-scale LMs*: scaling ourV=2048V\\\!=\\\!2048tiny\-GPT anchor toV=32,000V\\\!=\\\!32\{,\}000–50,00050\{,\}000vocabularies and billion\-parameter models would pushρV\\rho\_\{V\}past60,000×60\{,\}000\\timeson the distillation channel; we expect the TallyTrain\+faMMvs\. FedAvg\-faMMranking to persist, but demonstrating it at scale is left to future work\.

## 6Conclusion

TallyTrain transmits only each peer’sarg⁡max\\arg\\maxclass index, matching or beating soft\-label distillation at one to two orders of magnitude less bandwidth across CIFAR\-10/100 and WikiText\-2, and – composed with sparse FedAvg merges – Pareto\-dominates every parameter\-averaging operating point we tested\. The accuracy gain comes from the noise\-filtering property of majority voting under under\-trained peers \(Proposition 2;\+1\.13\+1\.13vs−0\.22\-0\.22pp consensus contributions on CIFAR\-100\); the bandwidth gain scales asρC=4​C/bC\\rho\_\{C\}\\\!=\\\!4C/b\_\{C\}, making the primitive especially attractive for large\-vocabulary settings that current soft\-label methods cannot reach\.

## References

## Appendix AHyperparameter settings

#### Common settings\.

All federated runs in §[4](https://arxiv.org/html/2607.00173#S4)use ResNet\-18 \(no pretraining\), AdamW \(η=10−3\\eta\\\!=\\\!10^\{\-3\}, weight decay5⋅10−45\\\!\\cdot\\\!10^\{\-4\}\),N=10N\\\!=\\\!10peers under full\-mesh averaging,K=5K\\\!=\\\!5local steps per round, local batchB=32B\\\!=\\\!32,R=3000R\\\!=\\\!3000rounds, Dirichlet non\-IID splitαDir=0\.5\\alpha\_\{\\mathrm\{Dir\}\}\\\!=\\\!0\.5, warm\-upW=300W\\\!=\\\!300\(labelled hybrid\) orW=1500W\\\!=\\\!1500\(cross\-distribution\), and a public probe set of\|𝒟pub\|=2,000\|\\mathcal\{D\}\_\{\\mathrm\{pub\}\}\|\\\!=\\\!2\{,\}000\(CIFAR\-10\) or10,00010\{,\}000\(CIFAR\-100\)\. Multi\-seed cells use seeds\{0,1,2\}\\\{0,1,2\\\}\.\|𝒟pub\|\|\\mathcal\{D\}\_\{\\mathrm\{pub\}\}\|matches prior FedMD/FedDF\(fedmd2019;lin2020ensemble\); see Appendix[K](https://arxiv.org/html/2607.00173#A11)for a probe\-count sweep\.

#### Data, BN, andvmap\.

Private partitions use the standard CIFAR recipe \(RandomCrop\(32,padding=4\),RandomHorizontalFlip, per\-channel normalisation\); the public probe set uses no stochastic augmentation, so peers see identical probes round\-to\-round and the voting histogram is well\-defined on a fixed grid\. ResNet\-18’s BN running statistics are stripped viatorch\.func’sreplace\_all\_batch\_norm\_modules\_helper becausevmapcannot safely batch in\-place buffer updates; the cost is<1<\\\!1pp uniformly across all methods, so relative comparisons are unaffected\. The TinyGPT model used on WikiText\-2 uses LayerNorm and is unaffected\.

#### Local\-epoch correspondence \(faM↔EM\\\!\\leftrightarrow\\\!E\)\.

On the CIFAR\-10 non\-IID partition withK=5K\\\!=\\\!5local steps per round and∼390\\sim\\\!390batches per local epoch per peer \(N=8N\\\!=\\\!8,Blocal=16B\_\{\\mathrm\{local\}\}\\\!=\\\!16\), one merge perMMrounds equals one merge perK⋅MK\\\!\\cdot\\\!Mlocal SGD steps; this givesfa=200↔E≈2\.5\\mathrm\{fa\}\\\!=\\\!200\\\!\\leftrightarrow\\\!E\\\!\\approx\\\!2\.5,fa=500↔E≈6\.4\\mathrm\{fa\}\\\!=\\\!500\\\!\\leftrightarrow\\\!E\\\!\\approx\\\!6\.4\(McMahan CIFAR default\), andfa=1000↔E≈13\\mathrm\{fa\}\\\!=\\\!1000\\\!\\leftrightarrow\\\!E\\\!\\approx\\\!13\.

## Appendix BProofs

### Proof sketch of Lemma 1 \(Function\-space contraction\)\.

The KL gradient with respect to the predictor is bounded below by the strong\-convexity constantμ\\muof KL on theβ0\\beta\_\{0\}\-interior of the simplex\. One round of distillation moves each peer’s predictor along the negative KL gradient with step size\(1−α\)​η\(1\-\\alpha\)\\eta, yielding the contraction factor1−\(1−α\)​η​μ1\-\(1\-\\alpha\)\\eta\\mu\. The variance termη2​σ2/β0\\eta^\{2\}\\sigma^\{2\}/\\beta\_\{0\}comes from the bounded\-variance assumption \(A2\) and the lower boundβ0\\beta\_\{0\}on simplex coordinates\. Full proof deferred; the argument adapts standard stochastic\-approximation contraction reasoning \(in the spirit of local\-SGD analyses such asstich2018local;khaled2020tighter\) to the function\-space KL\-distillation operator considered here\.

### Proof of Proposition 1 \(Condorcet\)\.

Under \(B1\) and \(B2\), Hoeffding’s inequality bounds the probability that fewer thanN/2N/2ofNNi\.i\.d\. Bernoulli\(pn≥p¯p\_\{n\}\\geq\\bar\{p\}\) trials succeed byexp⁡\(−2​N​\(p¯−1/2\)2\)\\exp\(\-2N\(\\bar\{p\}\-1/2\)^\{2\}\)\. Replacing “majority correct” with “mode correct” is a tightening, so the same bound applies\.

### Proof sketch of Proposition 2 \(Variance reduction\)\.

The soft consensusP¯​\(x\)∈ΔC−1\\bar\{P\}\(x\)\\in\\Delta^\{C\-1\}has variance proportional to the per\-peer softmax entropy onxx; this entropy isΘ​\(1\)\\Theta\(1\)for under\-trained peers\. The hard consensusH¯​\(x\)\\bar\{H\}\(x\)is a one\-hot delta in the mode position, with varianceΘ​\(1/N\)\\Theta\(1/N\)from peer disagreement only\. The hard target’s gradient is therefore lower variance, modulo a biasℬ​\(p¯,γ\)\\mathcal\{B\}\(\\bar\{p\},\\gamma\)that captures the information lost by quantization\. Asp¯,γ→1\\bar\{p\},\\gamma\\to 1this bias vanishes\.

## Appendix CLearning\-rate sweep on CIFAR\-100

A natural counter\-hypothesis is that the inner\-loop learning rateη\\etashould scale with1/K1/Kto keep per\-round weight movement constant, by analogy with the large\-minibatch linear\-scaling heuristic ofgoyal2017accurate\. We sweepη\\etaover a2×32\\times 3grid of\(K,η\)\(K,\\eta\)on CIFAR\-100 non\-IID, holdingS=K⋅R=15,000S=K\\cdot R=15\{,\}000constant\.

Table 4:Final test accuracy of TallyTrain on CIFAR\-100 non\-IID across the\(K,η\)\(K,\\eta\)grid\. Higherη\\etadegrades accuracy at everyKK, contradicting the linear\-scaling hypothesis\. The likely cause is AdamW’s adaptive preconditioner, which is approximately gradient\-scale\-invariant\.
## Appendix DMechanism ablations

We tested two richer\-than\-mean aggregation mechanisms on CIFAR\-100 hybrid:

#### Disagreement\-weighted distillation\.

Scale the KL term per example by1\+βdis​\(1−maxc⁡H¯c​\(x\)\)1\+\\beta\_\{\\mathrm\{dis\}\}\(1\-\\max\_\{c\}\\bar\{H\}\_\{c\}\(x\)\), boosting the influence of examples where peers disagree\. Withβdis=2\\beta\_\{\\mathrm\{dis\}\}=2: peak 33\.11, tail 32\.74 \(vs uniform baseline 33\.24/32\.94 – worse than baseline\)\.

#### Asymmetric peer weighting\.

Blend uniform consensus with inverse\-agreement weights, giving more influence to peers whose votes disagree with the plurality, with mixing coefficientβasy=0\.5\\beta\_\{\\mathrm\{asy\}\}=0\.5: peak 33\.06, tail 32\.85 \(also worse than baseline\)\.

#### Combined\.

βdis=2,βasy=0\.5\\beta\_\{\\mathrm\{dis\}\}=2,\\beta\_\{\\mathrm\{asy\}\}=0\.5: peak 33\.33, tail 32\.94 \(within seed noise of baseline\)\.

We conclude that uniform argmax voting is already at the function\-space ceiling for our setting; the negative result is reported for completeness\.

## Appendix ERC\-FD rate\-distortion sweep

To characterize the cost of discarding rank, we ran RC\-FD with top\-k∈\{1,3,5\}k\\in\\\{1,3,5\\\}on CIFAR\-10 hybrid;k=1k=1recovers TallyTrain, and ask→Ck\\to Cthe method approaches full\-softmax FedMD\.

Table 5:RC\-FD on CIFAR\-10 non\-IID hybrid \(1 seed each\)\. Top\-kksendskkindices pluskk4\-byte probabilities\. Tail accuracy is essentially flat: rank information beyond top\-1 contributes∼0\.5\\sim 0\.5pp\.
## Appendix FTeacher subset sampling sweep

Table 6:BsampleB\_\{\\mathrm\{sample\}\}sweep on CIFAR\-10 and CIFAR\-100, hybrid setting, 1 seed each\. Per\-class coverage rateρ=Bsample/C\\rho=B\_\{\\mathrm\{sample\}\}/Cis the relevant scale\. CIFAR\-10 peaks atρ≈1\.6\\rho\\approx 1\.6; further increases hurt\. CIFAR\-100 is still climbing atρ≈1\.3\\rho\\approx 1\.3\.![Refer to caption](https://arxiv.org/html/2607.00173v1/x5.png)Figure 6:BsampleB\_\{\\mathrm\{sample\}\}sweep on CIFAR\-10 and CIFAR\-100\. Thexx\-axis is the per\-class coverage rateρ\\rho\.The sweep is referenced from the body of §[F](https://arxiv.org/html/2607.00173#A6)and substantiates the regularization\-not\-budget reading ofBsampleB\_\{\\mathrm\{sample\}\}: belowρ≈1\\rho\\approx 1the consensus signal is too sparse and CIFAR\-10 tail collapses to 37\.57% \(cross\-peer std 11\.81\); aboveρ≈5\\rho\\approx 5the distillation gradient overweights public\-set fit and CIFAR\-10 tail drops from64\.9864\.98% to62\.8662\.86% asBsampleB\_\{\\mathrm\{sample\}\}grows from 16 to 256\.

## Appendix GCross\-distribution variant: KL\-decay sweep

In the cross\-distribution regime \(CIFAR\-10 task, CIFAR\-100 probes,α=0\\alpha\\\!=\\\!0,W=1500W\\\!=\\\!1500\), the public set carries no useful labels for the task and the consensus channel is the only inter\-peer information flow\. We use a linear KL\-decay scheduleλr=max⁡\(0,1−\(r−W\)/T\)\\lambda\_\{r\}=\\max\(0,1\-\(r\\\!\-\\\!W\)/T\)forr\>Wr\\\!\>\\\!Wto suppress distillation before the consensus drifts into the self\-reinforcing fixed point characterized in §[H](https://arxiv.org/html/2607.00173#A8)\. The sweep below variesTTby an order of magnitude\.

Table 7:CIFAR\-10 task with CIFAR\-100 probes,α=0\\alpha=0,W=1500W=1500\. The decay window is robust across an order of magnitude inTT\.![Refer to caption](https://arxiv.org/html/2607.00173v1/x6.png)Figure 7:KL\-decay sensitivity on CIFAR\-10 cross\-distribution\. 3\-seed mean and std plotted atT=200T=200; single\-seed points at otherTT\.
## Appendix HCross\-distribution without decay \(no\-decay collapse\)

To demonstrate that the KL\-decay schedule of §[G](https://arxiv.org/html/2607.00173#A7)is necessary, we ran TallyTrain and FedMD in the cross\-distribution regime \(α=0\\alpha=0, CIFAR\-10 task, CIFAR\-100 probes,W=1500W=1500\) without any decay\. Both methods peak around round∼2000\\sim 2000at≈50%\\approx 50\\%test accuracy, then collapse over the remaining 1000 rounds to≈25%\\approx 25\\%\. The collapse pattern is identical for both label formats, ruling out label fidelity as the cause: the issue is that the consensus contraction has no useful fixed point in this regime\.

## Appendix ILanguage\-model experiment details

#### Dataset and tokenizer\.

WikiText\-2*raw*variant \(wikitext\-2\-raw\-v1\), 2\.05M training tokens after BPE tokenization\. We train a Byte\-Pair Encoding tokenizer with vocabulary sizeV=2048V=2048on the train split \(one\-shot, cached\)\. After tokenizing, we hold out the last5%5\\%of training tokens as the public probe set \(177,689 tokens\) and partition the remainder intoN=8N=8contiguous chunks of∼422\\sim 422K tokens each as the peer\-private slices\. Validation and test splits are used unchanged\.

#### Model\.

A 6\-layer causal transformer with hidden dimensiond=256d=256, 8 attention heads, MLP multiplier 4, context length 256, and non\-tied embedding/output heads\. Total∼5\.84\\sim 5\.84M parameters per peer\. We deliberately leave the embedding and output projections*untied*because PyTorch’sstack\_module\_statededuplicates shared tensors, which undervmapwould cause the output head to fall back to its un\-stacked initialization\.

#### Training\.

AdamW withη=3×10−4\\eta=3\\times 10^\{\-4\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999, weight decay0\.010\.01\. Per\-peer gradient norm clipping at1\.01\.0with NaN scrubbing \(transformer training at this scale diverges without clipping\)\.K=5K=5local steps per round, batch sizeBlocal=16B\_\{\\mathrm\{local\}\}=16sequences of length 256,R=3000R=3000rounds\. Distillation:Bsample=32B\_\{\\mathrm\{sample\}\}=32public sequences per round \(yielding32×256=819232\\times 256=8192probe positions per round, orρpositions≈4\.0\\rho\_\{\\mathrm\{positions\}\}\\approx 4\.0againstV=2048V=2048\),αteacher=0\.5\\alpha\_\{\\mathrm\{teacher\}\}=0\.5, warm\-upW=300W=300rounds\.

#### Bandwidth accounting\.

A “probe” for the LM experiment is a single token position in a length\-TTcontext window\. One forward pass over a length\-TTsequence yieldsTTnext\-token predictions; we distill at every position\. TallyTrain transmits a 2\-byte argmax index per position \(sinceV\>256V\>256\)\. FedMD transmits a4​V=81924V=8192\-byte float softmax vector per position\. The byte\-aligned bandwidth ratio atV=2048V=2048is thereforeρV=4V/2=4096×\\rho\_\{V\}=4V/2=4096\\times, which matches the empirical*Total comm*columns in Table[3](https://arxiv.org/html/2607.00173#S4.T3)to within the per\-round protocol overhead\.

#### Bandwidth\-bridge variant\.

The TallyTrain\+fa200 and FedAvg\-fa200 runs add a single FedAvg merge everyM=200M=200rounds after the warm\-up window\. Each merge transmits\(N−1\)×\|W\|×4\(N\-1\)\\times\|W\|\\times 4bytes per peer, where\|W\|=5,839,360\|W\|=5\{,\}839\{,\}360parameters; forN=8N=8this is∼163\\sim 163MB per merge per peer\. Over 3000 rounds the merges contribute∼13\\sim 13merges×\\times163 MB=∼2\.1=\\sim 2\.1GB per peer, which dominates the distillation channel \(309309MB\) by an order of magnitude\.

## Appendix JPrivacy and threat model

We do not claim formal privacy guarantees, but the hard\-label primitive sits at a strictly tighter point on the leakage–utility curve than soft\-label distillation\. We sketch the threat model and the per\-channel information bounds, and identify the regimes in which the hard\-label channel is preferable\.

#### Threat model\.

We consider an honest\-but\-curious adversary that may \(i\) eavesdrop on the inter\-peer channel, \(ii\) operate as a peer in the federation, or \(iii\) collude with a strict minority of peers\. The public probe set𝒟pub\\mathcal\{D\}\_\{\\mathrm\{pub\}\}is assumed known to all parties \(this is the cost of using a public reference set; it is shared by FedMD, FedDF, and FD\)\. Private partitions𝒟n\\mathcal\{D\}\_\{n\}are not transmitted\.

#### Per\-probe information leakage\.

Each public probe carries at most the entropy of what is transmitted\. With byte\-aligned soft labels the per\-probe payload is32​C32Cbits and the payload entropy upper\-bounds the leakage aboutfnf\_\{n\}\. With hard labels the payload is⌈log2⁡C⌉\\lceil\\log\_\{2\}C\\rceilbits per probe and so the per\-probe leakage is at mostlog2⁡C\\log\_\{2\}Cbits regardless of how many bits the model actually computed\. ForC=2048C=2048this is1111bits/probe versus65,53665\{,\}536bits/probe for soft labels: a5957×5957\\timesreduction in the information\-theoretic upper bound on what an eavesdropper can learn aboutfnf\_\{n\}from one probe\.

This matters in two attacks:

- •Membership inference\(shokri2017membership\)on a public probe\. The attacker observes peernn’s output on probexxand asks whetherx∈𝒟nx\\in\\mathcal\{D\}\_\{n\}\. With soft labels the confidence vector reveals proximity\-to\-decision\- boundary, which is the canonical membership signal; with hard labels only the argmax index is observable, removing this signal\. Cronus\(chang2019cronus\)exploits the same logic\.
- •Model inversionvia gradient or logit access\. Soft\-label distillation channels carry approximately full differential information aboutfnf\_\{n\}on𝒟pub\\mathcal\{D\}\_\{\\mathrm\{pub\}\}; hard\-label channels carry only the level sets of the argmax, which are coarser byΘ​\(log2⁡C\)\\Theta\(\\log\_\{2\}C\)bits\.

#### Byzantine tolerance\.

Plurality voting tolerates an adversary controlling fewer than⌊N/2⌋\\lfloor N/2\\rfloorpeers without loss of consensus correctness on probes where the honest majority agrees\. Soft\-label averaging has no equivalent guarantee: a single adversarial peer broadcasting a confidently\-wrong soft label of sufficient magnitude can shift the mean\. Robust aggregation methods such as trimmed\-mean or median aggregation\(yin2018byzantine\)can recover Byzantine resilience for the soft\-label channel, but they introduce an additional robust\-aggregation step that plurality voting already absorbs into its definition\. We note thatyin2018byzantinetreat the parameter\-space case; we apply the same intuition informally here to the function\-space setting\.

#### Caveats and what hard labels do*not*buy\.

- •Aggregate leakage over many probes is not bounded by the per\-probe analysis above\. A sequence of hard labels reveals the argmax map offnf\_\{n\}over𝒟pub\\mathcal\{D\}\_\{\\mathrm\{pub\}\}; for sufficiently large public sets this map can be near\-uniquely identifying\.
- •Hard labels do not protect𝒟n\\mathcal\{D\}\_\{n\}from gradient inversion attacks against the local training step itself; that surface is unchanged from any local\-SGD federated method\.
- •The bandwidth\-bridge variant transmits parameters at every merge, so its privacy posture during merge rounds is that of FedAvg, not TallyTrain\. Operators worried about parameter leakage should compose the bridge with secure aggregation\(bonawitz2017practical\)on the merge channel\.

#### Summary\.

The hard\-label primitive provides a strictly tighter per\-probe information\-theoretic leakage bound than soft labels \(linear inlog2⁡C\\log\_\{2\}CvsCC\), and gracefully composes with plurality\-voting Byzantine tolerance\. We do not claim differential privacy and do not study aggregate leakage empirically; both are natural follow\-ups\.

## Appendix KPublic\-data scaling: probe coverage cannot replace parameter averaging

The bandwidth\-bridge story rests on the empirical claim that*pure TallyTrain’s gap to FedAvg is not closable by giving the distillation channel more public probes*\. We test this directly on CIFAR\-10 non\-IID with three orthogonal axes: \(A\) more in\-distribution probes; \(B\) much more out\-of\-distribution unlabeled probes; \(C\) OOD probes mixed with a small in\-distribution labeled anchor pool\. Across3333runs \(3 seeds per cell\) the ceiling is consistent: pure TallyTrain plateaus around6767% in the in\-distribution case and below6060% with OOD probes, never approaching the bridge variant’s71\.92±0\.3871\.92\\pm 0\.38% \(Table[2](https://arxiv.org/html/2607.00173#S4.T2)\) – much less FedAvg\-fa50’s79\.679\.6%\.

#### Arm A: in\-distribution probe\-size sweep\.

Pure TallyTrain withα=0\.5\\alpha\\\!=\\\!0\.5,W=300W\\\!=\\\!300, nofa\\mathrm\{fa\}merges, varying\|𝒟pub\|\|\\mathcal\{D\}\_\{\\mathrm\{pub\}\}\|over the in\-distribution CIFAR\-10 train split\.

Table 8:Arm A: in\-distribution\|𝒟pub\|\|\\mathcal\{D\}\_\{\\mathrm\{pub\}\}\|sweep, pure TallyTrain on CIFAR\-10 non\-IID, 3 seeds\. Saturation around67\.567\.5% from55k onward;2020k regresses slightly\. The bridge variant TallyTrain\+\+fa200 at\|𝒟pub\|=2\|\\mathcal\{D\}\_\{\\mathrm\{pub\}\}\|\\\!=\\\!2k already reaches71\.92±0\.3871\.92\\\!\\pm\\\!0\.38%\.
#### Arm B: OOD unlabeled probes\.

Pure TallyTrain withα=0\\alpha\\\!=\\\!0\(no labels on STL\-10\) and KL\-decayT=200T\\\!=\\\!200\(matching §[G](https://arxiv.org/html/2607.00173#A7)’s recipe\), main probe pool drawn from STL\-10’s unlabeled split \(resized96→3296\\\!\\to\\\!32with CIFAR\-10 normalization\)\.

Table 9:Arm B: STL\-10 unlabeled probes, no anchor, 3 seeds\. Tail accuracy is essentially constant in\|𝒟pub\|\|\\mathcal\{D\}\_\{\\mathrm\{pub\}\}\|– ten times more OOD probes adds nothing measurable, consistent with the cross\-distribution Condorcet drift of §[G](https://arxiv.org/html/2607.00173#A7)\.
#### Arm C: OOD probes with in\-distribution anchor\.

Same as Arm B but with a stratified mixture: every distillation batch draws44examples from a small CIFAR\-10*labeled*anchor pool \(carved from the end of the CIFAR\-10 train split, no overlap with the private partitions\) and the remainder from STL\-10\. The anchor provides the CE signal that pure TallyTrain’s OOD branch lacks;α=0\.3\\alpha\\\!=\\\!0\.3, no KL\-decay\.

Table 10:Arm C: STL\-10 main pool\+\+small CIFAR\-10 anchor with stratified44\-of\-1616probe sampling, 3 seeds\. Anchor doubling \(11k→2\\to\\\!2k\) adds∼2\\sim\\\!2–33pp; main pool doubling \(5050k→100\\to\\\!100k\) is within seed noise\. Best operating point \(\|𝒟pub\|=100\|\\mathcal\{D\}\_\{\\mathrm\{pub\}\}\|\\\!=\\\!100k, anchor=2\\\!=\\\!2k\) reaches57\.3457\.34%, still14\.614\.6pp below the bridge variant\.
#### Summary\.

The strongest public\-data configuration we tested \(100100k STL\-10 probes\+\+22k CIFAR\-10 anchor\) reaches57\.3457\.34%, leaving a14\.614\.6pp gap to TallyTrain\+\+fa200 \(71\.9271\.92%\) and a22\.322\.3pp gap to FedAvg\-fa50 \(79\.6079\.60%\)\. Probe coverage – in\-distribution or out\-of\-distribution, with or without anchors – plateaus before reaching the parameter\-space accuracy ceiling\. The bandwidth\-bridge variant is therefore not a workaround for insufficient probe data; it is the missing channel that function\-space distillation cannot reach by itself\.

## Appendix LFedProx baseline: full\(μ,fa\)\(\\mu,\\mathrm\{fa\}\)grid

We sweep FedProx\(li2018federated\)overμ∈\{0\.001,0\.01,0\.1\}\\mu\\in\\\{0\.001,0\.01,0\.1\\\}at our two FedAvg operating points \(fa∈\{50,200\}\\mathrm\{fa\}\\\!\\in\\\!\\\{50,200\\\}\) on both CIFAR\-10 and CIFAR\-100 non\-IID,33seeds per cell,3636runs total\. The proximal termμ/2⋅‖θ−θanchor‖2\\mu/2\\cdot\\\|\\theta\-\\theta\_\{\\mathrm\{anchor\}\}\\\|^\{2\}is added to the local CE loss;θanchor\\theta\_\{\\mathrm\{anchor\}\}is refreshed at every FedAvg merge round \(everyfa\\mathrm\{fa\}rounds\), matching canonical FedProx semantics withE=fa⋅KE\\\!=\\\!\\mathrm\{fa\}\\cdot Klocal SGD steps between merges\. Bandwidth is identical to vanilla FedAvg at the samefa\\mathrm\{fa\}\.

Table 11:FedProx full grid,33seeds per cell\. Bestμ\\muon every row is the smallest \(μ=0\.001\\mu\\\!=\\\!0\.001\); largerμ\\mudegrades monotonically because the proximal anchor is held fixed acrossfa⋅K\\mathrm\{fa\}\\\!\\cdot\\\!Klocal SGD steps between merges, so the regularizer becomes increasingly stale and over\-pulls peers toward an obsolete consensus\. For comparison vanilla FedAvg\-fa50 reaches peak79\.60±0\.5579\.60\\\!\\pm\\\!0\.55% / tail60\.51±1\.5260\.51\\\!\\pm\\\!1\.52% on CIFAR\-10 \(Table[2](https://arxiv.org/html/2607.00173#S4.T2)\); FedProx is0\.90\.9–2\.12\.1pp below FedAvg in peak and1\.71\.7–2\.92\.9pp below in tail at the bestμ\\mu\. The bridge variant TallyTrain\+\+fa200 \(71\.92±0\.3871\.92\\\!\\pm\\\!0\.38% on CIFAR\-10\) beats every cell of this grid by≥13\\geq\\\!13pp\.#### Why the bestμ\\muis the smallest\.

In our parallel implementation the per\-peer optimizer is AdamW; the proximal gradientμ​\(θ−θanchor\)\\mu\(\\theta\-\\theta\_\{\\mathrm\{anchor\}\}\)enters through the same Adam moments as the CE gradient, where Adam’s second\-moment normalization rescales it by its own running RMS\. Combined with the stale\-anchor effect atfa\>1\\mathrm\{fa\}\\\!\>\\\!1, this makes the effective regularization much stronger than the literalμ\\muwould suggest, so evenμ=0\.01\\mu\\\!=\\\!0\.01over\-pulls\. Canonical FedProx with SGD\-momentum atfa=1\\mathrm\{fa\}\\\!=\\\!1is outside our low\-bandwidth scope;li2018federatedthemselves report typicalμ∈\{0\.001,0\.01\}\\mu\\in\\\{0\.001,0\.01\\\}as best\.