CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning

# CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning
Source: [https://arxiv.org/html/2609.00446](https://arxiv.org/html/2609.00446)
Mengchen FanAffiliation:Dept\. of Computer ScienceUniversity of Alabama at Birmingham, AL, USABaocheng GengAffiliation:Dept\. of Computer ScienceUniversity of Alabama at Birmingham, AL, USAQing TianAffiliation:Dept\. of Computer ScienceUniversity of Alabama at Birmingham, AL, USA

###### Abstract

Conventional federated learning \(FL\) relies on parameter averaging, which forces clients to be doubly homogeneous: it demands an identical architecture and degrades under non\-IID data\. Real\-world deployments usually break both assumptions\. We sidestep both by building a decentralized knowledge distillation framework in which each client evaluates its peers’ model snapshots on its own local data and distills from the resulting soft predictions\. Because knowledge is transferred through the shared class posterior, clients are free to run different architectures; and because every teacher is evaluated on the student’s own device, raw data never leaves the client, with no central server or public dataset required\. Within this setting, we identify and address an under\-examined problem: how to combine the peer teacher predictions\. Existing methods, like uniform averaging, ignore how knowledge reliability varies across teachers and classes\. We propose Class\-wise Reliability\-Aware Distillation \(CRAD\), which, per class, first discards teachers that disagree with the peer consensus and then takes a weighted average of the rest, weighting each teacher by its per\-class reliability \(precision, or inverse variance\)\. Since the variance of an accuracy fromnnsamples scales as1/n1/n, support enters automatically: among the teachers that survive filtering, a teacher is trusted for a class to the degree that it is both accurate and well\-evidenced for it\. On three image\-classification benchmarks \(CIFAR\-10, CIFAR\-100, and PathMNIST colon pathology\), across heterogeneous architectures under severe non\-IID skew, CRAD consistently outperforms competing methods in global accuracy\.

11footnotetext:These authors contributed equally\.22footnotetext:Corresponding author:qtian@uab\.edu## 1Introduction

Federated learning \(FL\) has enabled collaborative training of machine learning models across distributed clients while keeping raw data local\[[27](https://arxiv.org/html/2609.00446#bib.bib27),[26](https://arxiv.org/html/2609.00446#bib.bib26)\]\. However, a fundamental limitation in practical deployments is the architectural homogeneity imposed by parameter averaging\[[8](https://arxiv.org/html/2609.00446#bib.bib8),[28](https://arxiv.org/html/2609.00446#bib.bib28),[18](https://arxiv.org/html/2609.00446#bib.bib18)\]\. This assumption is rarely satisfied in real\-world FL systems\[[18](https://arxiv.org/html/2609.00446#bib.bib18),[30](https://arxiv.org/html/2609.00446#bib.bib30),[38](https://arxiv.org/html/2609.00446#bib.bib38)\]\. Consider a network of hospitals collaborating to train a diagnostic imaging model\. Each operates under distinct hardware and data availability: a large urban hospital may run a full\-scale network on a GPU server, while a rural clinic can only afford a compact model on a low\-power device\. Because these institutions cannot share patient data across jurisdictions, FL is the natural solution\[[15](https://arxiv.org/html/2609.00446#bib.bib15),[1](https://arxiv.org/html/2609.00446#bib.bib1)\]\. Yet standard FL fails here: averaging across different architectures is undefined, since the weight tensors differ in shape and semantics\[[8](https://arxiv.org/html/2609.00446#bib.bib8),[22](https://arxiv.org/html/2609.00446#bib.bib22)\]\. This incompatibility bites three ways: no single architecture suits all institutions without over\-provisioning weak devices or under\-using capable ones\[[30](https://arxiv.org/html/2609.00446#bib.bib30),[38](https://arxiv.org/html/2609.00446#bib.bib38)\]; knowledge held in incompatible weight spaces cannot be meaningfully combined\[[8](https://arxiv.org/html/2609.00446#bib.bib8),[22](https://arxiv.org/html/2609.00446#bib.bib22)\]; and even under a shared architecture, averaging models that have each specialized to their own skewed data can yield one worse than any single specialist\[[39](https://arxiv.org/html/2609.00446#bib.bib39),[2](https://arxiv.org/html/2609.00446#bib.bib2),[14](https://arxiv.org/html/2609.00446#bib.bib14)\]\.

Existing FL methods fall into two groups, neither of which fits this setting\. Parameter\-space consensus methods\[[21](https://arxiv.org/html/2609.00446#bib.bib21),[16](https://arxiv.org/html/2609.00446#bib.bib16),[34](https://arxiv.org/html/2609.00446#bib.bib34)\]curb client drift under non\-IID data but still combine weights within a shared model space, so they remain inapplicable across incompatible architectures\[[28](https://arxiv.org/html/2609.00446#bib.bib28),[18](https://arxiv.org/html/2609.00446#bib.bib18)\]\. Knowledge\-distillation methods\[[22](https://arxiv.org/html/2609.00446#bib.bib22),[12](https://arxiv.org/html/2609.00446#bib.bib12),[20](https://arxiv.org/html/2609.00446#bib.bib20)\]instead exchange architecture\-agnostic soft predictions, escaping the homogeneity constraint, but typically reintroduce a shared public dataset or a central server\[[28](https://arxiv.org/html/2609.00446#bib.bib28),[3](https://arxiv.org/html/2609.00446#bib.bib3)\], the very privacy and centralization costs FL is meant to avoid\[[25](https://arxiv.org/html/2609.00446#bib.bib25),[29](https://arxiv.org/html/2609.00446#bib.bib29)\]\. These observations motivate our approach: a fully decentralized, peer\-to\-peer knowledge distillation \(KD\) framework in which clients collaborate without parameter averaging\. Each client shares model snapshots, instantiates peers as local teachers, evaluates them on its own data, and distills from their combined soft predictions, with no public transfer set and no central coordinator\. Decentralized, public\-data\-free distillation of this kind is attractive because it is natively compatible with heterogeneous architectures, and it lets each client learn the classes it sees rarely or not at all\[[18](https://arxiv.org/html/2609.00446#bib.bib18)\]\. But making this setting work surfaces a problem the FL distillation literature has largely left implicit: once a client holds a pool of peer teacher predictions, how should those predictions be combined to guide the student?

The default answer, inherited from ensemble distillation\[[22](https://arxiv.org/html/2609.00446#bib.bib22)\], is a uniform average over all peers\. This is a poor choice under heterogeneity, where teacher reliability is uneven and class\-dependent\. The central question of our work is therefore how a client should weight its peer teachers, per class, accounting for both how accurate and how well\-supported each is \(a class accuracy from few samples being itself unreliable\), without a server, public data, or expensive tuning\.

We answer with Class\-wise Reliability\-Aware Distillation \(CRAD\)\. Alongside its model snapshot, each client shares a compact class\-wise statistics vector summarizing, for every class, how many validation samples it holds and how often it classifies them correctly\. On receiving these statistics from its peers, a client works class by class: it first discards the teachers that hold too few samples of the class or whose predictions deviate most from the per\-class peer consensus, then combines the survivors by inverse\-variance \(precision\) weighting, so that each teacher counts in proportion to the statistical evidence behind its class\-accuracy estimate\. Because the variance of an accuracy estimated fromnnsamples scales as1/n1/n, sample support enters the weight automatically: a teacher that survives the filter and is both accurate and well\-supported for a class receives a large weight, while a teacher guessing from a few samples is discounted\. Our contributions are as follows:

- •We present a decentralized, public\-data\-free FL framework that handles model and data heterogeneity at once: clients of differing architectures collaborate purely through peer distillation over the sharedCC\-dimensional class posterior\. Within it we surface an under\-examined problem of how to combine peer teacher predictions, and show that, since teacher reliability is uneven across peers and classes, the usual uniform averaging is a poor choice\.
- •As our central methodological contribution, we propose CRAD, which, per class, filters out teachers that disagree with the peer consensus and then weights the survivors by the inverse variance \(precision\) of their class\-accuracy estimates, which, up to smoothing constants, is the Fisher information of each estimate: the amount of statistical evidence behind the teacher’s competence for that class\. It rewards both accuracy and sample support, requires few tuned hyperparameters, and adds only a compact per\-class statistics vector to communication\.
- •On the CIFAR\-10, CIFAR\-100, and PathMNIST image\-classification benchmarks, under heterogeneous architectures and non\-IID skew, CRAD attains the best global accuracy of all compared methods while maintaining strong local accuracy\.

![Refer to caption](https://arxiv.org/html/2609.00446v1/figures/CRAD_overview_drawio.png)Figure 1:Overview of the proposed CRAD framework\.\(1\) Decentralized heterogeneous setting:clients running different architectures collaborate over non\-IID private data, with no central server and no public dataset; the only shared interface between any two clients is theCC\-dimensional class posterior\.\(2\) Peer knowledge transfer:a student client receives its peers’ model snapshots together with a compact per\-class statistics vector, and runs each peer on its own local batch to obtain soft predictionsqjq\_\{j\}, so raw \(and often private\) data never leaves the client\.\(3\) Class\-wise reliability\-aware aggregation:for each class, CRAD first discards the teachers that disagree with the peer consensus \(an adaptive per\-class median filter\), then combines the survivors by precision \(inverse\-variance\) weighting of their class\-accuracy estimates, so that decisive, well\-supported teachers dominate the aggregated distillation targetq¯\\overline\{q\}\. The student then updates its model by minimizing a combination of cross\-entropy on its local labels and a distillation loss toward this target\.
## 2Related Work

### 2\.1Data Heterogeneity in Federated Learning

FL degrades under data heterogeneity, the non\-IID data problem\[[39](https://arxiv.org/html/2609.00446#bib.bib39)\]\. Previous efforts to mitigate this degradation have focused on parameter\-space regularization to bound client drift, such as FedProx\[[21](https://arxiv.org/html/2609.00446#bib.bib21)\], or variance reduction via control variates, such as SCAFFOLD\[[16](https://arxiv.org/html/2609.00446#bib.bib16)\], to stabilize convergence\. Another series of works emphasizes model personalization, allowing clients to adapt to local distributions through partial parameter sharing, meta\-learning, or personalized objective functions, such as pFedMe and APFL\[[6](https://arxiv.org/html/2609.00446#bib.bib6),[5](https://arxiv.org/html/2609.00446#bib.bib5)\]\. All of these, however, operate within a shared parameter space, whether by global, proximal, or personalized aggregation\[[8](https://arxiv.org/html/2609.00446#bib.bib8)\], and so remain incompatible with architecturally distinct models\[[8](https://arxiv.org/html/2609.00446#bib.bib8),[30](https://arxiv.org/html/2609.00446#bib.bib30)\], a setting common in practice\.

### 2\.2Knowledge Distillation in Federated Learning

KD sidesteps the homogeneity constraint by transferring knowledge through soft predictions rather than by averaging weight vectors; predictions stay comparable across architectures over a shared label space\[[28](https://arxiv.org/html/2609.00446#bib.bib28),[31](https://arxiv.org/html/2609.00446#bib.bib31)\]\. To fuse those predictions into a common target, KD\-based federated methods rely on auxiliary infrastructure: a shared public dataset, a central server, or both\. The first group aligns clients on a shared public proxy dataset, distilling each client toward a consensus formed on the common inputs, as in FedMD\[[20](https://arxiv.org/html/2609.00446#bib.bib20)\]\. This transfers knowledge across architectures but requires a task\-relevant public dataset, raising availability and privacy concerns\[[3](https://arxiv.org/html/2609.00446#bib.bib3)\]\. The second group routes fusion through a central server that distills clients’ models or predictions into a global consensus and broadcasts it back, as in ensemble distillation\[[22](https://arxiv.org/html/2609.00446#bib.bib22)\]\. This typically runs on the server’s own public or generated data, relocating the dependency rather than removing it, and reintroduces a central coordinator, the bottleneck and single point of failure FL aims to avoid\[[18](https://arxiv.org/html/2609.00446#bib.bib18),[29](https://arxiv.org/html/2609.00446#bib.bib29)\]\. Recent server\-centric variants follow the same pattern: the multi\-teacher FedMKD of Sun et al\.\[[32](https://arxiv.org/html/2609.00446#bib.bib32)\]distills client models into a server\-side student with one confidence scalar per teacher, and the personalized FedMKD of Lin et al\.\[[23](https://arxiv.org/html/2609.00446#bib.bib23)\]initializes each client by blending the global model with the client’s own previous\-round model in parameter space\. Both require a central server and architecturally identical clients\. A related self\-distillation line\[[19](https://arxiv.org/html/2609.00446#bib.bib19),[10](https://arxiv.org/html/2609.00446#bib.bib10),[9](https://arxiv.org/html/2609.00446#bib.bib9)\]selectively distills the global model into the local one, filtering or reweighting its knowledge class by class \(and, in\[[9](https://arxiv.org/html/2609.00446#bib.bib9)\], sample by sample\), but operates within FedAvg, with a single server\-aggregated teacher, identical client architectures, and in\[[10](https://arxiv.org/html/2609.00446#bib.bib10)\]a server\-side auxiliary set for scoring the global model\. The third group needs neither: clients distill from one another peer\-to\-peer over their own data, with no server and no shared set, as in DFML\[[18](https://arxiv.org/html/2609.00446#bib.bib18),[13](https://arxiv.org/html/2609.00446#bib.bib13)\], the regime in which our framework operates\.

### 2\.3Combining Teacher Predictions

Across nearly all KD\-based federated methods, peer predictions are fused by a uniform average over the available teachers, or a fixed data\-volume weighting inherited from FedAvg\[[22](https://arxiv.org/html/2609.00446#bib.bib22),[20](https://arxiv.org/html/2609.00446#bib.bib20)\]\. This treats every teacher as equally trustworthy for every class, which is unjustified when teachers differ in capacity and are trained on sharply non\-IID partitions, so that a given peer may be reliable on its well\-represented classes and badly miscalibrated elsewhere\. A natural alternative, used in centralized multi\-teacher distillation\[[36](https://arxiv.org/html/2609.00446#bib.bib36)\], is to weight teachers by prediction confidence or entropy, but confidence is a poor proxy for correctness under distribution shift, and such schemes ignore how much data actually supports each teacher’s belief\. Reliability\-based weighting normally assumes held\-out validation data on a trusted server or access to a shared transfer set, neither of which exists in our setting\. How to weight teachers by reliability in a fully decentralized, public\-data\-free, non\-IID setting, accounting for the statistical uncertainty of reliability estimated from finite local data, remains largely open\. Our work targets exactly this gap: we keep the decentralized, public\-data\-free KD backbone and present a per\-class aggregation that first filters teachers by agreement and then weights the survivors by the precision of each teacher’s class\-accuracy estimate\.

## 3Method

### 3\.1Setting and Overview

We considerNNclients, each holding a private local dataset𝒟i\\mathcal\{D\}\_\{i\}of input\-label pairs\(x,y\)∈𝒳×𝒴\(x,y\)\\in\\mathcal\{X\}\\times\\mathcal\{Y\}drawn from a client\-specific distributionPiP\_\{i\}, with a label space𝒴=\{1,…,C\}\\mathcal\{Y\}=\\\{1,\\ldots,C\\\}shared across clients\. The federation is heterogeneous in two ways at once\. The data are non\-IID: the class proportionsPi​\(y\)P\_\{i\}\(y\)differ sharply across clients, and under severe skew a client may hold few or even no samples of some classes \(withPi​\(y\)=0P\_\{i\}\(y\)=0in the extreme\), so its local label set can be a strict subset of𝒴\\mathcal\{Y\}\. The models are heterogeneous: each clientiiruns a networkfif\_\{i\}of its own architecture, scaled to its hardware\. These are exactly the conditions under which standard Federated Averaging breaks down: weight averaging is undefined across heterogeneous architectures, and even within one, averaging classifiers trained on disjoint classes degrades the global model below a local specialist\[[39](https://arxiv.org/html/2609.00446#bib.bib39),[7](https://arxiv.org/html/2609.00446#bib.bib7),[21](https://arxiv.org/html/2609.00446#bib.bib21)\]\.

We therefore discard parameter averaging in favor of fully decentralized, peer\-to\-peer knowledge distillation, and within it focus on the step that determines what each student actually learns: the fusion of peer teacher predictions into a distillation target\. The backbone departs from parameter averaging across three coupled dimensions\. Rather than averaging in parameter space, knowledge is transferred through soft output distributions produced by teacher models evaluated locally on each client’s own data\. Rather than requiring architectural uniformity, each client maintains a structurally distinct model\. The only interface between teacher and student is the semantically uniformCC\-dimensional class posterior\. Rather than relying on a central server, we operate over a fully connected peer\-to\-peer topology in which every client communicates directly with all others, with no global coordinator at any stage of training\. On top of this backbone, CRAD forms the distillation target through a per\-class procedure in two stages: it first discards teachers that disagree with the per\-class peer consensus, then combines the survivors with each teacher’s contribution scaled by the statistical precision of its class\-accuracy estimate, so that the consensus the student aligns with is dominated, for every class, by the teachers that are both in agreement and well\-evidenced for it\.

### 3\.2Architecture\-Agnostic Knowledge Transfer

At communication roundrr, each clientiimaintains its own modelfif\_\{i\}and receives a set of model snapshots\{\(θjr−1,archj,𝐬jr−1\)\}j∈𝒩i\\\{\(\\theta\_\{j\}^\{r\-1\},\\mathrm\{arch\}\_\{j\},\\mathbf\{s\}\_\{j\}^\{r\-1\}\)\\\}\_\{j\\in\\mathcal\{N\}\_\{i\}\}from its peers, where𝒩i=\{1,…,N\}∖\{i\}\\mathcal\{N\}\_\{i\}=\\\{1,\\ldots,N\\\}\\setminus\\\{i\\\}denotes the set of all other clients,archj\\mathrm\{arch\}\_\{j\}denotes the architecture identifier for clientjj, and𝐬jr−1\\mathbf\{s\}\_\{j\}^\{r\-1\}is a compact class\-wise statistics vector defined in Section[3\.3](https://arxiv.org/html/2609.00446#S3.SS3)\. Critically, all clients in roundrrdistill from roundr−1r\-1snapshots\. This pattern prevents intra\-round race conditions, where clients training earlier in a round distill from neighbors already updated within the same round, which would break the synchronous\-round semantics required for stable convergence\[[40](https://arxiv.org/html/2609.00446#bib.bib40)\]\. For each training batch\(x\(b\),y\(b\)\)\(x^\{\(b\)\},y^\{\(b\)\}\)sampled from𝒟i\\mathcal\{D\}\_\{i\}, clientiiinstantiates each neighbor’s architecturefθjr−1f\_\{\\theta\_\{j\}^\{r\-1\}\}locally and computes the teacher logits:

zj\(b\)=fθjr−1​\(x\(b\)\),j∈𝒩i,z\_\{j\}^\{\(b\)\}=f\_\{\\theta\_\{j\}^\{r\-1\}\}\(x^\{\(b\)\}\),\\quad j\\in\\mathcal\{N\}\_\{i\},\(1\)followed by a temperature\-scaled softmax, with distillation temperatureT\>0T\>0that softens the distribution to expose the inter\-class similarities the student learns from\[[11](https://arxiv.org/html/2609.00446#bib.bib11)\], to obtain the soft target vectorqj\(b\)q\_\{j\}^\{\(b\)\}:

qj\(b\)=softmax⁡\(zj\(b\)/T\)∈ΔC−1,q\_\{j\}^\{\(b\)\}=\\mathrm\{softmax\}\(z\_\{j\}^\{\(b\)\}/T\)\\in\\Delta^\{C\-1\},\(2\)whereΔC−1\\Delta^\{C\-1\}denotes the probability simplex overCCclasses\. The soft targetqj\(b\)q\_\{j\}^\{\(b\)\}encodes what neighborjjbelieves about the class structure of the student’s own local samplex\(b\)x^\{\(b\)\}, including the relative likelihoods it assigns to classes the student may have never encountered\[[11](https://arxiv.org/html/2609.00446#bib.bib11)\]\. Crucially, teacher inference is performed entirely on the student’s local data: the student never transmits its raw samples to any peer\. This is the mechanism by which knowledge of unseen classes propagates across the network without a shared dataset or central coordinator\[[3](https://arxiv.org/html/2609.00446#bib.bib3)\]\. The architectural heterogeneity of the teacher pool is handled naturally: because each teacher functions as a black\-box mappingx→ℝCx\\rightarrow\\mathbb\{R\}^\{C\}, any two models sharing the same label space𝒴\\mathcal\{Y\}are mutually compatible for knowledge transfer regardless of their internal architecture\.

### 3\.3Class\-wise Reliability\-Aware Aggregation

Given the per\-teacher soft targets\{qj\(b\)\}j∈𝒩i\\\{q\_\{j\}^\{\(b\)\}\\\}\_\{j\\in\\mathcal\{N\}\_\{i\}\}of Eq\.[2](https://arxiv.org/html/2609.00446#S3.E2), the remaining question is how to combine them into a single distillation target\. The standard choice is the uniform mean over all peers\[[22](https://arxiv.org/html/2609.00446#bib.bib22)\],

q¯unif\(b\)=1\|𝒩i\|​∑j∈𝒩iqj\(b\),\\overline\{q\}^\{\(b\)\}\_\{\\mathrm\{unif\}\}=\\frac\{1\}\{\|\\mathcal\{N\}\_\{i\}\|\}\\sum\_\{j\\in\\mathcal\{N\}\_\{i\}\}q\_\{j\}^\{\(b\)\},\(3\)which weights every teacher equally for every class\. This is wasteful under heterogeneity: a teacher’s competence is class\-dependent, likely high for classes it has seen often and low for classes it has seen rarely or never\. We instead operate separately for each class in two stages: we first discard teachers that lack support for the class or disagree with the per\-class peer consensus, then aggregate the survivors weighted by an estimate of their reliability for that class\.

#### Class\-wise agreement filtering\.

Before weighting, we first drop, for each class, uninformed teachers with insufficient support for the class, and write𝒩ic⊆𝒩i\\mathcal\{N\}\_\{i\}^\{c\}\\subseteq\\mathcal\{N\}\_\{i\}for the teachers that survive this support filter for classcc\. Among these remaining teachers, we then discard those whose predictions deviate most from the peer consensus on that class\. We form a preliminary per\-class consensus and measure each teacher’s deviation from it,

q~c=1\|𝒩ic\|​∑j∈𝒩icqjc,djc=\|qjc−q~c\|,\\tilde\{q\}^\{c\}=\\frac\{1\}\{\|\\mathcal\{N\}\_\{i\}^\{c\}\|\}\\sum\_\{j\\in\\mathcal\{N\}\_\{i\}^\{c\}\}q\_\{j\}^\{c\},\\qquad d\_\{j\}^\{c\}=\\left\|\\,q\_\{j\}^\{c\}\-\\tilde\{q\}^\{c\}\\,\\right\|,\(4\)and retain for each class the teachers whose deviation is at most the median deviation for that class,

τc=medianj∈𝒩icdjc,𝒮ic=\{j∈𝒩ic:djc≤τc\}\.\\tau\_\{c\}=\\operatorname\*\{median\}\_\{j\\in\\mathcal\{N\}\_\{i\}^\{c\}\}d\_\{j\}^\{c\},\\qquad\\mathcal\{S\}\_\{i\}^\{c\}=\\left\\\{\\,j\\in\\mathcal\{N\}\_\{i\}^\{c\}\\;:\\;d\_\{j\}^\{c\}\\leq\\tau\_\{c\}\\,\\right\\\}\.\(5\)This adaptive, per\-class median keeps the half of the supported teachers most in agreement with the consensus and introduces no threshold to tune\. The class\-wise aggregation below then ranges over𝒮ic\\mathcal\{S\}\_\{i\}^\{c\}rather than the full peer set, so different classes may draw on different teacher subsets\.

#### Class\-wise reliability statistics\.

Alongside its snapshot, each clientjjshares a compact statistics vector𝐬j=\{\(njc,ajc\)\}c=1C\\mathbf\{s\}\_\{j\}=\\\{\(n\_\{j\}^\{c\},a\_\{j\}^\{c\}\)\\\}\_\{c=1\}^\{C\}recording, for each classcc, its class\-ccaccuracyajca\_\{j\}^\{c\}, evaluated with its current model on a held\-out validation split of its own shard, and the numbernjcn\_\{j\}^\{c\}of class\-ccexamples in that split, on which the accuracy is computed\. This is2​C2Cscalars per client per round, negligible beside the model weights, and exposes no raw samples\.

#### Reliability as a statistical estimate\.

The accuracyajca\_\{j\}^\{c\}is a noisy signal: the same value is more credible when measured on many class\-ccexamples than on a handful\. We therefore treatajca\_\{j\}^\{c\}as a Bernoulli proportion over thenjcn\_\{j\}^\{c\}examples it was measured on, applying the Agresti–Coull correction to stabilize small\-sample estimates and taking its associated variance,

a~jc=ajc​njc\+2njc\+4,σj,c2=a~jc​\(1−a~jc\)njc\+4\.\\tilde\{a\}\_\{j\}^\{c\}=\\frac\{a\_\{j\}^\{c\}\\,n\_\{j\}^\{c\}\+2\}\{n\_\{j\}^\{c\}\+4\},\\qquad\\sigma\_\{j,c\}^\{2\}=\\frac\{\\tilde\{a\}\_\{j\}^\{c\}\\,\(1\-\\tilde\{a\}\_\{j\}^\{c\}\)\}\{n\_\{j\}^\{c\}\+4\}\.\(6\)

#### Inverse\-variance \(precision\) weighting\.

We weight each surviving teacher by the precision of its class\-accuracy estimate, the inverse of its variance, an evidence\-based weighting rule\. The weight rises with both the decisiveness of the estimated accuracy and the amount of evidence behind it \(up to the Agresti–Coull smoothing constants, the precision equals the Fisher information of the estimate\), and inverse variance is the standard rule for combining estimates of differing reliability\[[4](https://arxiv.org/html/2609.00446#bib.bib4),[17](https://arxiv.org/html/2609.00446#bib.bib17),[24](https://arxiv.org/html/2609.00446#bib.bib24)\]\. Precision is also the scale on which evidence is additive: an accuracy evaluated on additional samples accrues precision linearly, so a teacher’s influence grows in proportion to its evidence\. Weights not proportional to1/σ21/\\sigma^\{2\}break this proportionality, and every alternative we tested performs worse \(Tab\.[4](https://arxiv.org/html/2609.00446#S4.T4)\)\. We thus set the class\-wise weight

wjc=1σj,c2\+ϵ,w\_\{j\}^\{c\}=\\frac\{1\}\{\\sigma\_\{j,c\}^\{2\}\+\\epsilon\},\(7\)whereϵ\>0\\epsilon\>0only guards against division by zero, and aggregate the surviving teachers class by class as a precision\-weighted average:

q¯c=∑j∈𝒮icwjc​qjc∑j∈𝒮icwjc\+ϵ,q¯←q¯∑c=1Cq¯c,\\overline\{q\}^\{c\}=\\frac\{\\sum\_\{j\\in\\mathcal\{S\}\_\{i\}^\{c\}\}w\_\{j\}^\{c\}\\,q\_\{j\}^\{c\}\}\{\\sum\_\{j\\in\\mathcal\{S\}\_\{i\}^\{c\}\}w\_\{j\}^\{c\}\+\\epsilon\},\\qquad\\overline\{q\}\\leftarrow\\frac\{\\overline\{q\}\}\{\\sum\_\{c=1\}^\{C\}\\overline\{q\}^\{c\}\},\(8\)where𝒮ic\\mathcal\{S\}\_\{i\}^\{c\}is the agreement\-filtered teacher set of Eq\.[5](https://arxiv.org/html/2609.00446#S3.E5)and the renormalization restoresq¯\\overline\{q\}to the probability simplex \(we omit the batch indexbbonq¯c\\overline\{q\}^\{c\},qjcq\_\{j\}^\{c\}for brevity\)\. Becausenjcn\_\{j\}^\{c\}enters the variance as the effective sample size, with variance scaling as1/njc1/n\_\{j\}^\{c\}, a teacher whose class\-ccaccuracy rests on few examples has a large estimated variance \(Eq\.[6](https://arxiv.org/html/2609.00446#S3.E6)\) and hence a small weight\. The weight is thus a per\-class trust score on the teacher’s predictions\. Precision alone rewards decisiveness rather than correctness\. The weighting is therefore applied only after the class\-wise filter removes teachers with insufficient support or predictions that substantially deviate from the peer consensus, and under the standard assumption that unreliable or outlying teachers constitute a minority, this filtering reduces the risk that a confidently inaccurate teacher receives a high weight\. A teacher is therefore trusted for a class to the degree that it is confident, well\-evidenced, and in consensus\. Repeated mutual distillation couples the teachers across rounds, so their predictions are not independent\. We therefore use the weight as a trust score rather than as a minimum\-variance combiner, for which independence would be required\.

Given the aggregated targetq¯\\overline\{q\}, the student aligns its prediction with this precision\-weighted consensus through a convex combination of the standard cross\-entropy supervision on the local labels,ℒCE\\mathcal\{L\}\_\{\\mathrm\{CE\}\}, and the distillation loss\[[11](https://arxiv.org/html/2609.00446#bib.bib11)\]:

ℒ=\(1−λ\)ℒCE\+λT2⋅KL\(q¯∥softmax\(zs/T\)\),\\mathcal\{L\}=\(1\-\\lambda\)\\,\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\+\\lambda\\,T^\{2\}\\cdot\\mathrm\{KL\}\\left\(\\overline\{q\}\\;\\\|\\;\\mathrm\{softmax\}\(z\_\{s\}/T\)\\right\),\(9\)wherezs=fi​\(x\(b\)\)z\_\{s\}=f\_\{i\}\(x^\{\(b\)\}\)is the student’s own logit vector andλ∈\[0,1\]\\lambda\\in\[0,1\]is the distillation weight, balancing reliance on the peer consensus against the local ground\-truth labels\. The temperature\-scaled KL term with itsT2T^\{2\}factor is the standard distillation objective\[[11](https://arxiv.org/html/2609.00446#bib.bib11)\], here applied to the aggregated targetq¯\\overline\{q\}rather than to a single teacher\. The precision weightswjcw\_\{j\}^\{c\}depend onnjcn\_\{j\}^\{c\}andajca\_\{j\}^\{c\}alone \(Eqs\.[6](https://arxiv.org/html/2609.00446#S3.E6)–[7](https://arxiv.org/html/2609.00446#S3.E7)\), never on the teachers’ parameters, so the aggregation preserves the backbone’s architecture\-agnostic property\. Algorithm[1](https://arxiv.org/html/2609.00446#algorithm1)summarizes the complete training procedure\.

Algorithm 1Class\-wise Reliability\-Aware Distillation \(CRAD\)Input :

NNclients with local datasets

\{𝒟i\}i=1N\\\{\\mathcal\{D\}\_\{i\}\\\}\_\{i=1\}^\{N\}, temperature

TT, loss weight

λ\\lambda, smoothing constant

ϵ\\epsilon
Output :Updated client models

\{fθi\}i=1N\\\{f\_\{\\theta\_\{i\}\}\\\}\_\{i=1\}^\{N\}
Initialize model

fθif\_\{\\theta\_\{i\}\}for all clients

i∈\{1,…,N\}i\\in\\\{1,\\ldots,N\\\};

for*each communication roundr=0r=0toR−1R\-1*do

Freeze current models as snapshots:

θprev,i←θi\\theta\_\{\\mathrm\{prev\},i\}\\leftarrow\\theta\_\{i\}for all

ii;

Each client computes class statistics

𝐬i=\{\(nic,aic\)\}c=1C\\mathbf\{s\}\_\{i\}=\\\{\(n\_\{i\}^\{c\},a\_\{i\}^\{c\}\)\\\}\_\{c=1\}^\{C\}\(validation count

nicn\_\{i\}^\{c\}, validation accuracy

aica\_\{i\}^\{c\}\);

for*each clienti=1i=1toNN*do

Define peer set

𝒩i=\{1,…,N\}∖\{i\}\\mathcal\{N\}\_\{i\}=\\\{1,\\ldots,N\\\}\\setminus\\\{i\\\}and, per class, the support\-filtered subset

𝒩ic⊆𝒩i\\mathcal\{N\}\_\{i\}^\{c\}\\subseteq\\mathcal\{N\}\_\{i\}of teachers with sufficient class\-

ccsamples;

Compute precision weights from peer statistics:

a~jc=ajc​njc\+2njc\+4\\tilde\{a\}\_\{j\}^\{c\}=\\frac\{a\_\{j\}^\{c\}n\_\{j\}^\{c\}\+2\}\{n\_\{j\}^\{c\}\+4\},

σj,c2=a~jc​\(1−a~jc\)njc\+4\\sigma\_\{j,c\}^\{2\}=\\frac\{\\tilde\{a\}\_\{j\}^\{c\}\(1\-\\tilde\{a\}\_\{j\}^\{c\}\)\}\{n\_\{j\}^\{c\}\+4\},

wjc=1σj,c2\+ϵw\_\{j\}^\{c\}=\\frac\{1\}\{\\sigma\_\{j,c\}^\{2\}\+\\epsilon\},

∀j∈𝒩i,c\\forall j\\in\\mathcal\{N\}\_\{i\},c;

for*each batch\(x,y\)∈𝒟i\(x,y\)\\in\\mathcal\{D\}\_\{i\}*do

Student logits and cross\-entropy loss:

zs=fθi​\(x\)z\_\{s\}=f\_\{\\theta\_\{i\}\}\(x\),

ℒCE=CE⁡\(zs,y\)\\mathcal\{L\}\_\{\\mathrm\{CE\}\}=\\mathrm\{CE\}\(z\_\{s\},y\);

Compute teacher soft targets \(no gradients\):

qj=Softmax⁡\(fθprev,j​\(x\)/T\),∀j∈𝒩iq\_\{j\}=\\mathrm\{Softmax\}\(f\_\{\\theta\_\{\\mathrm\{prev\},j\}\}\(x\)/T\),\\hskip 8\.50012pt\\forall j\\in\\mathcal\{N\}\_\{i\};

Class\-wise agreement filter \(adaptive per\-class median\), over

𝒩ic\\mathcal\{N\}\_\{i\}^\{c\}:

q~c=1\|𝒩ic\|​∑j∈𝒩icqjc\\tilde\{q\}^\{c\}=\\frac\{1\}\{\|\\mathcal\{N\}\_\{i\}^\{c\}\|\}\\sum\_\{j\\in\\mathcal\{N\}\_\{i\}^\{c\}\}q\_\{j\}^\{c\},

τc=medianj∈𝒩ic⁡\|qjc−q~c\|\\tau^\{c\}=\\operatorname\{median\}\_\{j\\in\\mathcal\{N\}\_\{i\}^\{c\}\}\|q\_\{j\}^\{c\}\-\\tilde\{q\}^\{c\}\|,

𝒮c=\{j∈𝒩ic:\|qjc−q~c\|≤τc\}\\mathcal\{S\}^\{c\}=\\\{j\\in\\mathcal\{N\}\_\{i\}^\{c\}:\|q\_\{j\}^\{c\}\-\\tilde\{q\}^\{c\}\|\\leq\\tau^\{c\}\\\};

Filtered precision\-weighted class\-wise aggregate:

q¯c=∑j∈𝒮cwjc​qjc∑j∈𝒮cwjc\+ϵ\\overline\{q\}^\{c\}=\\frac\{\\sum\_\{j\\in\\mathcal\{S\}^\{c\}\}w\_\{j\}^\{c\}q\_\{j\}^\{c\}\}\{\\sum\_\{j\\in\\mathcal\{S\}^\{c\}\}w\_\{j\}^\{c\}\+\\epsilon\}, then normalize

q¯←q¯/∑cq¯c\\overline\{q\}\\leftarrow\\overline\{q\}/\\sum\_\{c\}\\overline\{q\}^\{c\};

KD and total loss:

ℒKD=T2⋅KL\(q¯∥Softmax\(zs/T\)\)\\mathcal\{L\}\_\{\\mathrm\{KD\}\}=T^\{2\}\\cdot\\mathrm\{KL\}\(\\overline\{q\}\\,\\\|\\,\\mathrm\{Softmax\}\(z\_\{s\}/T\)\),

ℒ=\(1−λ\)​ℒCE\+λ​ℒKD\\mathcal\{L\}=\(1\-\\lambda\)\\mathcal\{L\}\_\{\\mathrm\{CE\}\}\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{KD\}\};

Update

θi\\theta\_\{i\}via backpropagation \(with gradient clipping\);

## 4Experiments

### 4\.1Experimental Setup

#### Datasets and Partitioning\.

We evaluate the proposed framework on three benchmarks from distinct domains: the CIFAR\-10 and CIFAR\-100 natural\-image classification benchmarks, and PathMNIST, a colon\-pathology histology benchmark of nine tissue classes from the MedMNIST collection\[[35](https://arxiv.org/html/2609.00446#bib.bib35)\], which tests a realistic, privacy\-sensitive medical setting\. Each dataset is partitioned acrossN=10N=10clients \(by default\) under a Dirichlet distribution with concentration parameterα=0\.3\\alpha=0\.3\. This induces severe label skew\. These public benchmarks serve only to construct the simulated federation: each client trains on its own private shard alone, and the method uses no shared or public data\.

#### Architecture Pool\.

To validate architectural heterogeneity, the framework uses three structurally distinct models: ResNet\-18 \(≈11\\approx 11M parameters\), ResNet\-18\-Half \(≈2\.8\\approx 2\.8M parameters,0\.5×0\.5\\timeschannel scaling\), and CNN\-6 \(≈0\.8\\approx 0\.8M parameters\)\. This pool evaluates both width heterogeneity, through the channel scaling between the ResNet variants, and family heterogeneity, through the absence of residual connections in CNN\-6\. Architectures are assigned deterministically across the 10 clients via a round\-robin schedule, denoted asarchi=POOL⁡\[imod3\]\\mathrm\{arch\}\_\{i\}=\\mathrm\{POOL\}\[i\\bmod 3\]\. This capacity heterogeneity, combined with the non\-IID skew, makes teacher reliability uneven and class\-dependent, reflecting common real\-world deployment scenarios\.

#### Compared Methods\.

All methods are evaluated under the same heterogeneous architecture pool, non\-IID partition, and training schedule\. We compare against five representative heterogeneous\-FL distillation baselines that span the auxiliary\-component taxonomy of Section[2](https://arxiv.org/html/2609.00446#S2)\. FedMD\[[20](https://arxiv.org/html/2609.00446#bib.bib20)\]aligns clients on a shared public proxy dataset \(size500500\) before local adaptation, directly testing the value of our public\-data\-free design\. FedGD\[[37](https://arxiv.org/html/2609.00446#bib.bib37)\]replaces the public proxy with a synthetic transfer set \(size500500\) generated rather than collected, a data\-free variant of the same public\-data paradigm\. MSFKD\[[33](https://arxiv.org/html/2609.00446#bib.bib33)\]is a multi\-teacher selective federated distillation method\. DFML\[[18](https://arxiv.org/html/2609.00446#bib.bib18)\]is the closest baseline to our backbone: a fully decentralized, public\-data\-free, peer\-to\-peer mutual distillation method that, like ours, exchanges no raw data and uses no server, but fuses peers without class\-wise reliability weighting\. FedMKD\[[23](https://arxiv.org/html/2609.00446#bib.bib23)\]is a recent multi\-teacher method that reuses each client’s own previous\-round model as an additional teacher\. No implementation is public, so we adapt it to our decentralized, architecture\-heterogeneous setting\. Within our framework, the central comparison isolates the effect of teacher aggregation\. We compare CRAD against two aggregation variants: Uniform KD, the conventional uniform average \(Eq\.[3](https://arxiv.org/html/2609.00446#S3.E3)\); and Uncertainty KD, which weights teachers by the negative entropy of their predictions\.

#### Training Protocol\.

The network is trained forR=300R=300communication rounds, one local epoch per round, with the distillation weightλ=0\.7\\lambda=0\.7and a random seed of 1024\. Each client’s held\-out partition is split evenly into a validation half and a test half\. The validation half supplies the reliability statistics of Section[3\.3](https://arxiv.org/html/2609.00446#S3.SS3)and selects the checkpoint\. The test half is touched only for final reporting\. We report two metrics: global accuracy on each benchmark’s standard test set, never used during training or model selection \(our primary metric, measuring generalization\), and local accuracy on each client’s held\-out test half \(measuring local adaptation\)\. Because a client can score high local accuracy simply by overfitting the few classes it holds, local accuracy serves mainly as a guardrail, confirming that global\-accuracy gains do not come at a large cost in local adaptation\. For every method alike, both metrics are reported at the round with the highest average local validation accuracy\.

### 4\.2Experimental Results

Table[1](https://arxiv.org/html/2609.00446#S4.T1)reports all methods on all three benchmarks under identical conditions\.

#### CIFAR\-10\.

The three aggregation variants are comparable on local accuracy, consistent with its guardrail role\. They differ chiefly on global accuracy, which draws on borrowed knowledge of rarely\-seen classes\. There CRAD leads at78\.60%78\.60\\%, ahead of Uncertainty KD \(77\.41%77\.41\\%\) and the uniform average \(77\.19%77\.19\\%\)\. Uncertainty KD barely improves on uniform averaging because confidence is a weak proxy for correctness under distribution shift, whereas CRAD’s agreement filter keeps only teachers that align with the peer consensus, and its precision weight then favors those whose class beliefs are both decisive and well\-supported\. All competing methods trail our framework in global accuracy\. FedMKD\[[23](https://arxiv.org/html/2609.00446#bib.bib23)\]is the strongest at74\.54%74\.54\\%, still four points below CRAD\. The public\-data methods FedMD, MSFKD and FedGD are limited by the quality of their proxy or synthetic transfer set, a poor stand\-in for the clients’ own data\. DFML \(61\.99%61\.99\\%\), the decentralized public\-data\-free method closest to our backbone, trails here, although it is the strongest competitor on CIFAR\-100\. That every aggregation variant in our framework exceeds every competing method isolates the contribution of the decentralized backbone, while the gap between CRAD and Uniform KD isolates that of reliability\-aware aggregation\.

Table 1:Teacher\-aggregation comparison on CIFAR\-10, CIFAR\-100 and PathMNIST \(N=10N=10, non\-IID Dirichlet,R=300R=300\)\. All decentralized\-KD methods share the identical backbone and distillation weightλ=0\.7\\lambda=0\.7, differing only in how peer teachers are combined; the checkpoint is selected on each client’s validation split\. Accuracies are percentages, with global accuracy the primary metric\. Best aggregation result in each column inbold\.
#### Scaling to More Classes \(CIFAR\-100\)\.

The CIFAR\-100 columns of Table[1](https://arxiv.org/html/2609.00446#S4.T1)cover a 100\-class benchmark in which each client holds far fewer samples per class, stressing the reliability estimates that drive CRAD\. On this harder benchmark, CRAD again ranks first, at48\.22%48\.22\\%global accuracy, ahead of the strongest competitor DFML \(47\.29%47\.29\\%\) and well above the uniform average \(41\.53%41\.53\\%\) and Uncertainty KD \(42\.06%42\.06\\%\)\. Two patterns stand out: the margin over uniform averaging widens sharply relative to CIFAR\-10, from about one point to nearly seven, and DFML now overtakes the uniform and uncertainty\-based variants that outscored it on CIFAR\-10\. Both follow from the same cause\. With100100classes and far fewer samples per class, per\-class beliefs are noisier and locally\-rare classes more numerous, precisely the regime in which tying a teacher’s weight to its reliability pays off\.

#### Consistency on a Medical Benchmark\.

PathMNIST instantiates the clinical scenario of Section[1](https://arxiv.org/html/2609.00446#S1): sensitive slides that sites cannot share, held at institutions running widely differing hardware\. Its columns in Table[1](https://arxiv.org/html/2609.00446#S4.T1)show CRAD again attaining the best global accuracy \(89\.44%89\.44\\%, against87\.59%87\.59\\%for the uniform average\) and the best local accuracy \(88\.14%88\.14\\%\)\. That the gain transfers from natural images to histopathology indicates that reliability\-aware aggregation is not specific to one data domain\.

Table 2:Cost on CIFAR\-10 \(N=10N=10\), measured under the setup of Table[1](https://arxiv.org/html/2609.00446#S4.T1)on identical hardware\. Payload is what one client transmits per round and GPU its peak resident memory; Time is the wall\-clock duration of a full communication round\. “Proxy” marks methods additionally requiring public or synthetic data\.

### 4\.3Analyses

#### Per\-Class Gains\.

![Refer to caption](https://arxiv.org/html/2609.00446v1/figures/cifar-10_per_class_subplots_r1.png)Figure 2:Per\-class global accuracy on CIFAR\-10 with classes ordered by local training support \(shown in the parentheses, abundant to rare\), for all compared methods\. The methods agree more on well\-supported classes and separate on the rare ones, where CRAD degrades the least\.Figure[2](https://arxiv.org/html/2609.00446#S4.F2)breaks global accuracy down by class, ordered from abundant to rare local support\. On a class that a client holds few samples of, the uniform average dilutes the few competent teachers among a majority of uninformed ones, and a teacher can be confidently wrong on a class it has never seen; only tying the weight to sample support suppresses them\. The figure bears this out: the methods are nearly indistinguishable where support is plentiful and separate as it dwindles\. On the rarest class \(30 local samples\), CRAD retains about56%56\\%accuracy against about47%47\\%for the other aggregation variants, and it stays ahead of every competing method at every support level\. The Stage\-1 ablation, CRAD’s agreement filter without the precision weight, tracks the uniform average on these rare classes, so the lift comes mainly from precision\-weighting the survivors, not from the filter alone\.

#### Convergence and Stability\.

Figure[3](https://arxiv.org/html/2609.00446#S4.F3)plots global accuracy over communication rounds\. CRAD sits above every baseline, including the Stage\-1 ablation, from the early rounds onward and converges smoothly, whereas DFML’s curve swings violently round to round, so the round selected on validation sits far above its typical accuracy\.

![Refer to caption](https://arxiv.org/html/2609.00446v1/figures/cifar-10_learning_curves_r1.png)Figure 3:Global accuracy vs\. communication round for all compared methods on CIFAR\-10\. CRAD \(red\) stays above every other method from the early rounds onward and converges stably, whereas DFML swings sharply from round to round\.
#### Communication, Compute, and Memory Cost\.

Table[2](https://arxiv.org/html/2609.00446#S4.T2)shows that cost is set by the regime, not by the aggregation rule: proxy\-based methods communicate only predictions over the transfer set and are cheap, but require that set to exist, whereas every snapshot\-based decentralized method exchanges full snapshots\. Within that regime CRAD’s own overhead is the2​C2C\-scalar statistics vector, under 1 KB, and it completes a round faster and at lower peak memory than every competing method\. Both are dominated by holding and running theN−1N\{\-\}1peer teachers, hence identical across our three aggregation variants; the higher payload is the price of requiring neither public data nor a server\.

#### Client\-Number Ablation\.

We further evaluate whether CRAD remains effective as the candidate client pool grows\. Specifically, we set the pool size toN∈\{20,50,100\}N\\in\\\{20,50,100\\\}clients, of which 10 are selected to participate in each round\. As shown in Table[3](https://arxiv.org/html/2609.00446#S4.T3), CRAD improves both global and local accuracy over Uniform KD at every pool size, and its relative advantage widens as the pool grows, from a factor of1\.21\.2on global accuracy atN=20N\{=\}20to2\.02\.0atN=100N\{=\}100\(22\.19%22\.19\\%vs\.11\.01%11\.01\\%\)\. This suggests that CRAD is helpful under sparse participation, where teacher quality becomes more heterogeneous\.

Table 3:Client\-number ablation on CIFAR\-10 with partial participation\. Each run usesR=300R=300rounds and activates 10 clients per round\. Global accuracy is averaged over all clients\.Table 4:Ablation of teacher\-weighting rules on CIFAR\-100\. All variants use the same CRAD framework and differ only in the aggregation weight\. Acc\.: accuracy\-only; Sup\.: support\-only; SM:softmax⁡\(−σ2\)\\mathrm\{softmax\}\(\-\\sigma^\{2\}\); LCB:max⁡\(0,a~−σ\)\\max\(0,\\tilde\{a\}\-\\sigma\);CE\\mathrm\{CE\}: cross\-entropy\.
#### Weighting Rule\.

Table[4](https://arxiv.org/html/2609.00446#S4.T4)swaps the precision weight for six alternatives on CIFAR\-100, holding the rest of CRAD fixed\. Precision weighting is best on both metrics: accuracy alone and support alone each capture part of the gain, but combining them through the variance beats either\. Weighting by1/σ1/\\sigmarather than1/σ21/\\sigma^\{2\}is the weakest variant on local accuracy, consistent with precision being the scale on which evidence adds \(Sec\.[3\.3](https://arxiv.org/html/2609.00446#S3.SS3)\)\.

## 5Limitations and Future Work

Our evaluation is confined to image classification\. Extending CRAD to other modalities and vision tasks is future work\. The shared statistics vector reveals each client’s validation label proportions through the countsnjcn\_\{j\}^\{c\}\. These are fixed across rounds, so a single noised release suffices and its privacy cost does not compose over rounds; the counts haveL1L\_\{1\}\-sensitivity one under record add/remove adjacency, so Laplace noise of scale1/ε1/\\varepsilongivesε\\varepsilon\-differential privacy\. Atε=1\\varepsilon=1this costs0\.820\.82points of global accuracy on CIFAR\-100 \(48\.22%48\.22\\%vs\.47\.40%47\.40\\%\), still ahead of every competing method in Table[1](https://arxiv.org/html/2609.00446#S4.T1)\. End\-to\-end differential privacy covering the snapshots remains open\.

## 6Conclusion

We presented a decentralized, server\-free, public\-data\-free framework in which clients of differing architectures collaborate by distilling from their peers’ predictions, and within it identified an under\-examined problem: how to combine those predictions when teacher reliability is uneven, class\-dependent, and itself uncertain given finite local data\. CRAD addresses this per class, discarding teachers that disagree with the peer consensus and weighting the rest by the precision of their class\-accuracy estimates, with few tuned hyperparameters\. Across CIFAR\-10, CIFAR\-100, and PathMNIST under heterogeneous architectures and severe non\-IID skew, CRAD attains the best global accuracy\.

## References

- \[1\]Farzam Asad, Junaid Saif Khan, Maria Tariq, Sundus Munir, and Muhammad Adnan Khan\.Federated proximal optimization for privacy\-preserving heart disease prediction: A controlled simulation study on non\-iid clinical data\.*arXiv preprint arXiv:2601\.17183*, 2026\.
- \[2\]Christopher Briggs, Zhong Fan, and Peter Andras\.Federated learning with hierarchical clustering of local updates to improve training on non\-iid data\.In*Proceedings of the International Joint Conference on Neural Networks \(IJCNN\)*, pages 1–9, 2020\.
- \[3\]Wei Chen, Shuai Pu, and Dong Guo\.Parameterized data\-free knowledge distillation for heterogeneous federated learning\.*Knowledge\-Based Systems*, 318:113502, 2025\.
- \[4\]William G\. Cochran\.The combination of estimates from different experiments\.*Biometrics*, 10\(1\):101–129, 1954\.
- \[5\]Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi\.Adaptive personalized federated learning\.*arXiv preprint arXiv:2003\.13461*, 2020\.
- \[6\]Canh T\. Dinh, Nguyen H\. Tran, and Tuan D\. Nguyen\.Personalized federated learning with moreau envelopes\.*arXiv preprint arXiv:2006\.08848*, 2020\.
- \[7\]Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar\.Personalized federated learning with theoretical guarantees: A model\-agnostic meta\-learning approach\.In*Proceedings of the Conference on Neural Information Processing Systems \(NeurIPS\)*, pages 3557–3568, 2020\.
- \[8\]Boyu Fan, Siyang Jiang, Xiang Su, Sasu Tarkoma, and Pan Hui\.A survey on model\-heterogeneous federated learning: Problems, methods, and prospects\.*arXiv preprint arXiv:2312\.12091*, 2023\.
- \[9\]Yuting He, Yiqiang Chen, Xiaodong Yang, Hanchao Yu, Yi\-Hua Huang, and Yang Gu\.Learning critically: Selective self\-distillation in federated learning on non\-iid data\.*IEEE Transactions on Big Data*, 2022a\.
- \[10\]Yuting He, Yiqiang Chen, Xiaodong Yang, Yingwei Zhang, and Bixiao Zeng\.Class\-wise adaptive self distillation for heterogeneous federated learning\.In*AAAI\-22 International Workshop on Trustable, Verifiable and Auditable Federated Learning*, 2022b\.
- \[11\]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean\.Distilling the knowledge in a neural network\.*arXiv preprint arXiv:1503\.02531*, 2015\.
- \[12\]Sohei Itahara, Takayuki Nishio, Yusuke Koda, Masahiro Morikura, and Koji Yamamoto\.Distillation\-based semi\-supervised federated learning for communication\-efficient collaborative training with non\-iid private data\.*IEEE Transactions on Mobile Computing*, 22\(1\):191–205, 2023\.
- \[13\]Eunjeong Jeong and Marios Kountouris\.Personalized decentralized federated learning with knowledge distillation\.*arXiv preprint arXiv:2302\.12156*, 2023\.
- \[14\]Daniel M\. Jimenez\-Gutierrez, Mehrdad Hassanzadeh, Aris Anagnostopoulos, Ioannis Chatzigiannakis, and Andrea Vitaletti\.A thorough assessment of the non\-iid data impact in federated learning\.*arXiv preprint arXiv:2503\.17070*, 2025\.
- \[15\]Peter Kairouz, H\. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G\. L\. D’Oliveira, Salim El Rouayheb, David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B\. Gibbons, Marco Gruteser, and Zaïd Harchaoui\.Advances and open problems in federated learning\.*Foundations and Trends in Machine Learning*, 14\(1–2\):1–210, 2021\.
- \[16\]Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J\. Reddi, Sebastian U\. Stich, and Ananda Theertha Suresh\.SCAFFOLD: Stochastic controlled averaging for federated learning\.In*Proceedings of the International Conference on Machine Learning \(ICML\)*, pages 5132–5143, 2020\.
- \[17\]Alex Kendall, Yarin Gal, and Roberto Cipolla\.Multi\-task learning using uncertainty to weigh losses for scene geometry and semantics\.In*Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 7482–7491, 2018\.
- \[18\]Yasser H\. Khalil, Amir H\. Estiri, Mahdi Beitollahi, Nader Asadi, Sobhan Hemati, Xu Li, Guojun Zhang, and Xi Chen\.DFML: Decentralized federated mutual learning\.arXiv preprint arXiv:2402\.01863, 2024\.
- \[19\]Gihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae, and Se\-Young Yun\.Preservation of the global knowledge by not\-true distillation in federated learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- \[20\]Daliang Li and Junpu Wang\.FedMD: Heterogeneous federated learning via model distillation\.*arXiv preprint arXiv:1910\.03581*, 2019\.
- \[21\]Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith\.Federated optimization in heterogeneous networks\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, pages 429–450, 2020\.
- \[22\]Tao Lin, Lingjing Kong, Sebastian U\. Stich, and Martin Jaggi\.Ensemble distillation for robust model fusion in federated learning\.In*Proceedings of the Conference on Neural Information Processing Systems \(NeurIPS\)*, pages 2351–2363, 2020\.
- \[23\]Tianjia Lin, Zhou Tan, and Ximeng Liu\.FedMKD: Personalized federated learning with memory knowledge distillation\.In*IEEE International Conference on Communications \(ICC\)*, pages 4945–4950, 2025\.
- \[24\]Vincent Mai, Kaustubh Mani, and Liam Paull\.Sample efficient deep reinforcement learning via uncertainty estimation\.*arXiv preprint arXiv:2201\.01666*, 2022\.
- \[25\]Grigory Malinovsky, Samuel Horváth, Konstantin Burlachenko, and Peter Richtárik\.Federated learning with regularized client participation\.*arXiv preprint arXiv:2302\.03662*, 2023\.
- \[26\]H\. Brendan McMahan and Daniel Ramage\.Federated learning: Collaborative machine learning without centralized training data, 2017\.Google AI Blog\.
- \[27\]H\. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas\.Communication\-efficient learning of deep networks from decentralized data\.In*Proceedings of the 20th International Conference on Artificial Intelligence and Statistics \(AISTATS\)*, pages 1273–1282, 2017\.
- \[28\]Alessio Mora, Irene Tenison, Paolo Bellavista, and Irina Rish\.Knowledge distillation in federated learning: A practical guide\.In*Proceedings of the Thirty\-Third International Joint Conference on Artificial Intelligence \(IJCAI\)*, pages 8188–8196, 2024\.
- \[29\]Tuc Van Nguyen, Mark A\. Dakka, Sonya M\. Diakiw, Matthew D\. VerMilyea, Michelle Perugini, Jonathan M\. M\. Hall, and Don Perugini\.A novel decentralized federated learning approach to train on globally distributed, poor quality, and protected private medical data\.*Scientific Reports*, 12\(1\):8888, 2022\.
- \[30\]Kilian Pfeiffer, Martin Rapp, Ramin Khalili, and Jörg Henkel\.Federated learning for computationally\-constrained heterogeneous devices: A survey\.*ACM Computing Surveys*, 55\(14s\):1–27, 2023\.
- \[31\]Hassan Salman, Chamseddine Zaki, Nour Charara, Sonia Guehis, Jean\-François Pradat\-Peyre, and Abbass Nasser\.Knowledge distillation in federated learning: A comprehensive survey\.*Discover Computing*, 28\(1\):145, 2025\.
- \[32\]Huifang Sun, Jiaming Pei, Xiaoqing Xu, Rubing Xue, Lingyun Zhao, Qihang Sun, Longfei Li, and Lukun Wang\.FedMKD: Multi\-teacher knowledge distillation for communication\-efficient federated learning\.*Cluster Computing*, 28:638, 2025\.
- \[33\]Dong Wang, Naifu Zhang, Meixia Tao, and Xu Chen\.Knowledge selection and local updating optimization for federated knowledge distillation with heterogeneous models\.*IEEE Journal of Selected Topics in Signal Processing*, 17\(1\):82–97, 2023\.
- \[34\]Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H\. Vincent Poor\.Tackling the objective inconsistency problem in heterogeneous federated optimization\.In*Proceedings of the Conference on Neural Information Processing Systems \(NeurIPS\)*, pages 7325–7337, 2020\.
- \[35\]Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni\.MedMNIST v2 – a large\-scale lightweight benchmark for 2D and 3D biomedical image classification\.*Scientific Data*, 10\(1\):41, 2023\.
- \[36\]Hailin Zhang, Defang Chen, and Can Wang\.Confidence\-aware multi\-teacher knowledge distillation\.In*Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\)*, pages 4498–4502, 2022\.
- \[37\]Jie Zhang, Chen Chen, Weiming Zhuang, and Lingjuan Lyu\.Target: Federated class\-continual learning via exemplar\-free distillation\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 4782–4793, 2023\.
- \[38\]Yingqi Zhang, Hui Xia, Shuo Xu, Xiangxiang Wang, and Lijuan Xu\.AdaptFL: Adaptive federated learning framework for heterogeneous devices\.*Future Generation Computer Systems*, 165:107610, 2025\.
- \[39\]Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra\.Federated learning with non\-iid data\.*arXiv preprint arXiv:1806\.00582*, 2018\.
- \[40\]Andrey Zhmoginov, Mark Sandler, Nolan Miller, Gus Kristiansen, and Max Vladymyrov\.Decentralized learning with multi\-headed distillation\.In*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\)*, pages 8053–8063, 2023\.

## Appendix AQualitative Mechanism on Individual Images

This appendix adds a qualitative view of how CRAD forms its per\-class distillation target on individual test images, including a case where the mechanism fails\. Every number below comes from the same CIFAR\-10 run as Table[1](https://arxiv.org/html/2609.00446#S4.T1), read at the same best\-validation checkpoint\. The per\-class curves of Fig\.[2](https://arxiv.org/html/2609.00446#S4.F2)show which classes reliability\-aware aggregation helps on, averaged over clients and test images\. They do not show how the two stages act on any single prediction\. Figure[5](https://arxiv.org/html/2609.00446#A1.F5)resolves one success case and one failure case down to the individual teacher\.

![Refer to caption](https://arxiv.org/html/2609.00446v1/figures/cifar10_panel_a_val_support.png)

![Refer to caption](https://arxiv.org/html/2609.00446v1/figures/cifar10_panel_b_test_acc.png)

Figure 4:Context that drives the weighting, on CIFAR\-10\.\(a\)Per\-client validation supportnjcn\_\{j\}^\{c\}under the Dirichlet \(α=0\.3\\alpha\{=\}0\.3\) partition, the count that enters the variance of Eq\.[6](https://arxiv.org/html/2609.00446#S3.E6)through Agresti–Coull smoothing\.\(b\)Per\-client accuracy on the balanced global test set, by class\. The two are related but not identical, which is the reason CRAD weights on both accuracy and the evidence behind it rather than on either alone\.![Refer to caption](https://arxiv.org/html/2609.00446v1/figures/cifar10_panel_c_success_bird.png)

![Refer to caption](https://arxiv.org/html/2609.00446v1/figures/cifar10_panel_d_failure_dog.png)

Figure 5:CRAD’s two\-stage aggregation traced on two individual CIFAR\-10 test images\.WWis the assigned distillation weight after per\-class normalization, and the parenthesized value is the client’s probability on the true class\.\(c\)Success case, a bird image: teachers with a single sample of the class are dropped as uninformed, four more fall outside the acceptance bandq~c±τc\\tilde\{q\}^\{c\}\\pm\\tau\_\{c\}and are dropped as consensus outliers, and the surviving well\-supported specialist \(C1,n=180n=180\) receives59\.5%59\.5\\%of the weight, raising the aggregated target on the true class to79\.4%79\.4\\%from a preliminary consensus of68\.3%68\.3\\%\.\(d\)Failure case, a dog image: the best\-evidenced teacher \(C7,n=258n=258, and correct at63\.7%63\.7\\%\) deviates from a low consensus formed by less informed peers and is removed by Stage 1, so its knowledge never reaches Stage 2 and the target stays at27\.2%27\.2\\%\.#### Setup\.

Figure[4](https://arxiv.org/html/2609.00446#A1.F4)gives the context that drives the weights\. Panel \(a\) is the per\-client validation supportnjcn\_\{j\}^\{c\}under the Dirichlet \(α=0\.3\\alpha\{=\}0\.3\) partition, the count that enters the variance of Eq\.[6](https://arxiv.org/html/2609.00446#S3.E6)through Agresti–Coull smoothing\. Panel \(b\) is the per\-client accuracy on the balanced global test set, broken down by class\. The two are related but not identical, which is the reason CRAD weights on both accuracy and the evidence behind it rather than on either alone\.

The two panels of Fig\.[5](https://arxiv.org/html/2609.00446#A1.F5)trace one test image each\. For every client we report the assigned distillation weightWW, that is the precision weight of Eq\.[7](https://arxiv.org/html/2609.00446#S3.E7)after per\-class normalization, and in parentheses the probability that client assigns to the true class\. Clients are marked in one of three states: retained and weighted, dropped for insufficient class support, or dropped as a consensus outlier\. The acceptance band printed above each panel isq~c±τc\\tilde\{q\}^\{c\}\\pm\\tau\_\{c\}, whereq~c\\tilde\{q\}^\{c\}is the preliminary consensus andτc\\tau\_\{c\}the median absolute deviation from it, so by construction roughly half of the supported teachers survive Stage 1\.

#### Success case \(Fig\.[5](https://arxiv.org/html/2609.00446#A1.F5)c\)\.

On a bird image, two clients hold a single validation sample of the class and are dropped as uninformed before any weighting occurs\. Among the remaining eight, four fall outside the acceptance band and are discarded as consensus outliers\. Of the four survivors, the client holding180180bird samples receives59\.5%59\.5\\%of the total weight and assigns80\.1%80\.1\\%to the correct class, while the three lower\-support survivors split the remainder\. The aggregated target is therefore dominated by the one teacher that is both well\-evidenced and confident, and it rises to79\.4%79\.4\\%on the true class, well above the preliminary consensus of68\.3%68\.3\\%that a uniform average would have produced\.

#### Failure case \(Fig\.[5](https://arxiv.org/html/2609.00446#A1.F5)d\)\.

The same mechanism can suppress the teacher it should trust\. On a dog image the peer consensus is low \(27\.5%27\.5\\%\) and the median deviation is correspondingly tight \(τc=6\.8%\\tau\_\{c\}=6\.8\\%\)\. Client 7 holds258258validation samples of the class, by a wide margin the best\-evidenced teacher, and assigns63\.7%63\.7\\%to the correct class\. It is also the most accurate teacher on that class, at90\.5%90\.5\\%in panel \(b\), so the filter is discarding the one teacher the weighting was designed to favor\. Because that estimate lies far outside a consensus formed by less informed peers, Stage 1 removes it before Stage 2 can weight it\. The surviving teachers all sit near the low consensus value, and the aggregated target ends at27\.2%27\.2\\%, essentially unchanged from the consensus it started at\.

This is the cost of consensus filtering under the assumption stated in Sec\.[3\.3](https://arxiv.org/html/2609.00446#S3.SS3), that unreliable teachers are in the minority for a class\. Where that assumption holds the filter is protective, and where a well\-supported specialist disagrees with a less informed majority it can remove useful signal\. Letting support temper the consensus filter, rather than applying the two steps in sequence, is a natural refinement that we leave to future work\.