FedImp: Enhancing Federated Learning Convergence with Impurity-Based Weighting

# FedImp: Enhancing Federated Learning Convergence with Impurity-Based Weighting
Source: [https://arxiv.org/html/2608.14654](https://arxiv.org/html/2608.14654)
Cuong TaTruong X\. Tran[![[Uncaptioned image]](https://arxiv.org/html/2608.14654v1/ORCIDiD_icon64x64.png)](https://orcid.org/0000-0002-3214-010X)\\IEEEmembershipSenior Member, IEEEHai\-Anh Tran and Cuong Ta are with School of Information and Communication Technology \(SOICT\), Hanoi University of Science and Technology \(HUST\), Vietnam\. E\-mail: anhth@soict\.hust\.edu\.vn, tavietcuong2707@gmail\.comTruong X\. Tran is with School of Science, Engineering and Technology, Penn State Harrisburg, The Pennsylvania State University, USA\. Email: truong\.tran@psu\.eduCorresponding author: Truong X\. Tran; Email: truong\.tran@psu\.edu©2025 IEEE\. Personal use of this material is permitted\. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works\.

###### Abstract

Federated Learning \(FL\) is a collaborative paradigm that enables multiple devices to train a global model while preserving local data privacy\. A major challenge in FL is the non\-Independent and Identically Distributed \(non\-IID\) nature of data across devices, which hinders training efficiency and slows convergence\. To tackle this, we propose Federated Impurity Weighting \(FedImp\), a novel algorithm that quantifies each device’s contribution based on the informational content of its local data\. These contributions are normalized to compute distinct aggregation weights for the global model update\. Extensive experiments on EMNIST and CIFAR\-10 datasets show that FedImp significantly improves convergence speed, reducing communication rounds by up to 64\.4%, 27\.8%, and 66\.7% on EMNIST, and 44\.2%, 44%, and 25\.6% on CIFAR\-10 compared to FedAvg, FedProx, and FedAdp, respectively\. Under highly imbalanced data distributions, FedImp outperforms all baselines and achieves the highest accuracy\. Overall, FedImp offers an effective solution to enhance FL efficiency in non\-IID settings\.

\\IEEEpeerreviewmaketitle

\{IEEEImpStatement\}

Federated Learning \(FL\) plays a crucial role in training models without compromising data privacy\. This work paves the way for future advancements in adaptive federated learning techniques, ensuring that AI models can be trained more efficiently and equitably across diverse and decentralized data sources\. By ensuring faster and more reliable convergence, FedImp enhances the feasibility of FL in large\-scale deployments, particularly in resource\-constrained environments such as mobile networks, IoT systems, and medical diagnostics\.

\{IEEEkeywords\}

Federated Learning, Fast Convergence Rate, Model Aggregation, Collaborative Machine Learning

## 1Introduction

Artificial Intelligence and Machine learning power many data\-driven applications, but traditional centralized training raises privacy and security concerns\. Federated learning \(FL\) addresses these by enabling decentralized model training across devices without sharing raw data\[[10](https://arxiv.org/html/2608.14654#bib.bib1)\]\.

This approach preserves privacy, reduces data transfer, and is ideal for industries with strict confidentiality requirements\. The FL process unfolds through a sequence of iterative model updates\[[6](https://arxiv.org/html/2608.14654#bib.bib2)\]\. Each local node \(client node, device\) computes a local model update based on its unique data, capturing local patterns\. These updates are then aggregated \(e\.g\., via weighted averaging\) into a unified global model, effectively learning from diverse data across all participating devices\.

One of the crucial challenges faced by FL algorithms is the presence of non\-Independently and Identically Distributed \(non\-IID\) data across decentralized devices\[[16](https://arxiv.org/html/2608.14654#bib.bib3)\]\. In FL environment, data is often non\-IID due to factors like location or user behavior, making it harder to aggregate local models effectively, leading to slow convergence and poor generalization\. To mitigate Non\-IID data effects, several strategies are employed\. Weighted aggregation allows devices with higher quality or more relevant data to have a greater impact on the global model\. Data augmentation artificially expands the diversity of local datasets, improving the model’s ability to generalize\. In addition, adaptive learning rates allow the model to adjust its learning process based on the unique characteristics of each device’s non\-IID data\. Several algorithms aggregate local models into a global model for optimal results\[[9](https://arxiv.org/html/2608.14654#bib.bib4)\]\. Among them, FedAvg\[[11](https://arxiv.org/html/2608.14654#bib.bib5)\]is widely used, averaging model parameters from participating nodes\. However, it often suffers from slow convergence and reduced accuracy in non\-IID settings, where node contributions can vary significantly\. To improve this, FedAdp\[[14](https://arxiv.org/html/2608.14654#bib.bib6)\]assigns weights based on the angle between local and global gradients\. Yet, when many nodes are non\-IID, the global gradient can become misleading, failing to reflect the true descent direction of the global objective\. In such cases, local gradients may lead the aggregated gradients to point away from the global minimum, slowing learning or causing instability\. This motivates the need for an alternative that better captures the informativeness and diversity of local data, such as the approach used in FedImp\.

This research aims to address the highlighted limitations associated with two FL algorithms, FedAvg and FedAdp, by introducing a novel algorithm termed Federated Impurity Weighting \(FedImp\)\. Our observation highlights the varying contributions of nodes during global model aggregation\. We measure a node’s contribution by assessing the informational impurity in its data using entropy\. These contributions are then normalized into distinct weights for aggregation\. The proposed strategy enhances generalization across diverse data distributions and mitigates issues where high weights are assigned to nodes with unrepresentative data\. FedImp is designed to prevent such biases, ensuring a more balanced and effective global model aggregation in FL\.

This paper encompasses the following contributions:

- •Determining traditional FL algorithm limitations: The initial contribution involves a critical examination of the FedAvg and FedAdp algorithms and highlighting their limitations in certain scenarios of non\-IID data\.
- •Novel Aggregation Strategy: Introducing FedImp, a new FL algorithm that quantifies the informational richness \(impurity\) of each client’s data using entropy\.
- •Enhanced FL Convergence and Accuracy: By addressing the challenges posed by non\-IID data distributions, the proposed method can improve the convergence rate and overall accuracy of the global model\.

We have implemented the proposed algorithm and conducted experiments to evaluate its performance under various FL scenarios using the EMNIST and CIFAR\-10 datasets\. The results show that FedImp achieves significantly faster convergence than FedAvg\[[9](https://arxiv.org/html/2608.14654#bib.bib4)\], FedProx\[[8](https://arxiv.org/html/2608.14654#bib.bib17)\], and FedAdp\[[11](https://arxiv.org/html/2608.14654#bib.bib5)\]in FL with non\-IID data\. Experiments show reductions in communication rounds by up to 64\.4%, 27\.8%, and 66\.7% compared to FedAvg, FedProx, and FedAdp, respectively, on the EMNIST dataset\. For the tests with the CIFAR\-10 dataset, FedImp can also reduce the number of communication rounds, ranging up to 44\.2%, 44%, and 25\.6% compared to FedAvg, FedProx, and FedAdp, respectively\. In the experiment with highly local imbalanced data nodes, FedImp is particularly advantageous and often becomes the only algorithm to reach target accuracy\.

The rest of this paper is organized as follows: Section[2](https://arxiv.org/html/2608.14654#S2)discusses the related works\. Section[3](https://arxiv.org/html/2608.14654#S3)provides the foundation of FL and algorithms: FedAvg, FedProx, and FedAdp\. In Section[4](https://arxiv.org/html/2608.14654#S4), the proposed algorithm is presented\. Experimental implementation and results are shown in Section[5](https://arxiv.org/html/2608.14654#S5), and the conclusion is presented in Section[6](https://arxiv.org/html/2608.14654#S6)\.

## 2Related Work

Numerous methods have been suggested for consolidating local models into a single global model in order to attain desirable outcomes\. McMahan et al\.\[[11](https://arxiv.org/html/2608.14654#bib.bib5)\]introduced the Federated Averaging \(FedAvg\) algorithm, in which clients perform multiple epochs of SGD on their local datasets and send their models to the server, which averages them to form a new global model\. However, the presence of non\-IID data at each client can negatively affect the performance of the FedAvg algorithm, including both a slow convergence rate and poor accuracy\. Addressing this challenge is crucial for improving the effectiveness of the FL approach, and various solutions have been proposed\. Li et al\.\[[8](https://arxiv.org/html/2608.14654#bib.bib17)\]introduce FedProx, an extension of FedAvg designed to handle heterogeneity in federated networks\. Though its modifications are minor, they have significant effects\. Experiments show that FedProx achieves better convergence than FedAvg across real\-world datasets, with a 22% average improvement in test accuracy, especially in highly heterogeneous environments\.

SGD with momentum has demonstrated excellent success in speeding up network training in a centralized machine learning approach by accumulating the gradient history over time in order to dampen oscillations\. Utilizing this idea, Hsu et al\.\[[5](https://arxiv.org/html/2608.14654#bib.bib7)\]proposed the Federated Averaging with Server Momentum \(FedAvgM\) algorithm\. This is especially suitable for FL, where the participating parties may only hold a small subset of labels and a sparse distribution of data\. Experiments on CIFAR\-10 demonstrate improved classification performance for FedAvgM over FedAvg over a range of non\-identicalness, with classification accuracy improved from 30\.1% to 76\.9% in the most skewed settings\.

Besides, Yeganeh et al\.\[[15](https://arxiv.org/html/2608.14654#bib.bib8)\]proposed IDA \(Inverse Distance Aggregation\), a novel adaptive weighting approach for clients based on meta\-information, which handles unbalanced and non\-iid data\. The IDA method uses model parameter distances to minimize outlier effects and improve convergence\. Results show that IDA outperforms FedAvg in classification accuracy in non\-IID scenarios and is resilient to low\-quality or harmful data from client nodes\. Unlike FedAvg, which assumes clients with more data have better distributions, IDA allows aligned clients to exclude out\-of\-distribution models\.

Minimizing the local loss function doesn’t guarantee minimizing the global loss\. To address this in non\-IID data, Acar et al\. introduced FedDyn\[[1](https://arxiv.org/html/2608.14654#bib.bib9)\], which dynamically regularizes each client’s loss to align with the global loss\. This makes FedDyn robust to varying heterogeneity levels, allowing full minimization per client\. It achieves a convergence rate ofO​\(1T\)O\(\\frac\{1\}\{T\}\)in convex and non\-convex settings and a linear rate in strongly convex settings, remaining agnostic to device heterogeneity and resilient to large\-scale, unbalanced, and partially participating devices\.

Realizing the personalization of the global model becomes crucial in handling the challenges that arise with non\-IID data, Vahidian et al\.\[[13](https://arxiv.org/html/2608.14654#bib.bib10)\]introduced Personalized FL by Pruning \(Sub\-FedAvg\)\. Sub\-FedAvg enhances efficiency by identifying a small subnetwork per client using hybrid pruning \(structured and unstructured\)\. Instead of averaging all parameters like FedAvg, it averages only the remaining parameters in each client’s subnetwork\.

In scenarios involving non\-IID data, the contributions of participating nodes to the training process are unequal\. To address this, Wu et al\. introduced the Federated Adaptive Weighting \(FedAdp\) algorithm in their work\[[14](https://arxiv.org/html/2608.14654#bib.bib6)\]\. This algorithm assigns dynamic weights to update the global model based on each node’s contribution per training round\. The authors have shown that FL with FedAdp can reduce the number of communication rounds by up to 54\.1% on the MNIST dataset and up to 45\.4% on the FashionMNIST dataset, as compared to the FedAvg algorithm\.

While FedAdp offers advancements, it struggles with global gradient misalignment in non\-IID settings\. When nodes lack label diversity, local gradients deviate, leading to suboptimal global updates\. Overweighting nodes with similar gradients can further degrade performance\.

In recent years, privacy\-preserving techniques have been extensively explored to enhance the security of FL\. Fotohi et al\.\[[3](https://arxiv.org/html/2608.14654#bib.bib18)\]propose a lightweight framework using differential privacy to protect client updates from adversarial reconstruction\. Blockchain integration is another approach that provides decentralized trust and security\. A blockchain\-enabled FL model can use smart contracts and cryptographic verification to prevent adversarial tampering\[[4](https://arxiv.org/html/2608.14654#bib.bib19)\]\. While these privacy\-oriented methods are beyond the primary scope of FedImp, which focuses on convergence under non\-IID conditions, we mention them here to highlight the broader context and ongoing advancements in FL security\. We will further discuss potential privacy considerations and future enhancements of our proposed method in the discussion section of the experimental results\.

In summary, recent FL research has seen substantial progress through adaptive weighting mechanisms \(e\.g\., FedAdp, IDA\), personalization techniques \(e\.g\., Sub\-FedAvg\), and regularization\-based convergence strategies \(e\.g\., FedDyn\)\. While these methods each address aspects of heterogeneity, they often rely on gradient similarity, parameter distance, or client\-specific architectures\. In contrast, FedImp introduces a fundamentally different perspective by quantifying the informational richness of local datasets through entropy, allowing data\-driven weighting that is independent of model\-specific signals\. This positions FedImp as a general and lightweight approach to improve convergence in non\-IID settings without requiring modifications to model structure or optimization methods\.

## 3Preliminaries of Federated Learning Algorithms

This section introduces the fundamental concepts of FL and discusses the principles of the FedAvg, FedProx, and FedAdp algorithms\. Additionally, it examines the limitations of FedAdp and FedAvg, providing the motivation for proposing a more comprehensive algorithm\.

### 3\.1Overview of Federated Learning Algorithms

FL is a distributed learning approach where clients train a shared model without sharing raw data\. It aims to optimize a global loss function,F​\(𝐰\)F\(\\mathbf\{w\}\), which aggregates individual client losses in multiple communication rounds\. At each communication roundtt, a subset ofKKnodes is selected, and the global model𝐰​\(t−1\)\\mathbf\{w\}\(t\-1\)from the previous communication round is transmitted to the chosen nodes\. Subsequently, each participating nodeiiexecutes stochastic gradient descent \(SGD\) training to minimize its local lossFi​\(𝐰\)F\_\{i\}\(\\mathbf\{w\}\):

𝐰i​\(t\)=𝐰​\(t−1\)−η​∇Fi​\(𝐰​\(t−1\)\)\\mathbf\{w\}\_\{i\}\(t\)=\\mathbf\{w\}\(t\-1\)\-\\eta\\nabla F\_\{i\}\(\\mathbf\{w\}\(t\-1\)\)\(1\)whereη\\etadenotes the learning rate, and∇Fi​\(𝐰​\(t−1\)\)\\nabla F\_\{i\}\(\\mathbf\{w\}\(t\-1\)\)represents the gradient at nodeii\. FL algorithms then seek to update the global model parameters𝐰\\mathbf\{w\}by aggregating the local model updates \(either parameters or gradients\) from each node:

𝐰​\(t\)=∑i=1Kψi​𝐰i​\(t\)\\mathbf\{w\}\(t\)=\\sum^\{K\}\_\{i=1\}\\psi\_\{i\}\\mathbf\{w\}\_\{i\}\(t\)\(2\)Here,ψi\\psi\_\{i\}is the weighting factor of nodeiiwhen calculating the global parameters\. Different algorithms use various weighting strategies\. The following part explains node weight determination for FedAvg and FedAdp and the loss function modification in FedProx\.

Federated Averaging \(FedAvg\) Algorithm: The FedAvg\[[11](https://arxiv.org/html/2608.14654#bib.bib5)\]algorithm aggregates local model updates from participating nodes based on their dataset sizes\. While it is often described as treating node contributions equally in terms of participation, the actual aggregation weights are proportional to the number of training samples each client possesses\. Specifically, the weighting factor of each node in the global aggregation is given by:

ψi​\(t\)=Di∑i′=1KDi′\\psi\_\{i\}\(t\)=\\frac\{D\_\{i\}\}\{\\sum\_\{i^\{\\prime\}=1\}^\{K\}D\_\{i^\{\\prime\}\}\}\(3\)whereDiD\_\{i\}is the number of local training samples on clientii\. When allDiD\_\{i\}are identical, this reduces to equal weighting across clients\.

FedProx Algorithm: To tackle statistical heterogeneity, Li et al\.\[[8](https://arxiv.org/html/2608.14654#bib.bib17)\]introduce a proximal term in the local subproblem, keeping local updates closer to the global model\. In particular, instead of just minimizing the local functionFi​\(𝐰\)F\_\{i\}\(\\mathbf\{w\}\), nodeiiminimizes the following objectivehih\_\{i\}:

min𝐰⁡hi​\(𝐰\)=Fi​\(𝐰\)\+μ2​‖𝐰−𝐰​\(t−1\)‖2\\min\_\{\\mathbf\{w\}\}h\_\{i\}\(\\mathbf\{w\}\)=F\_\{i\}\(\\mathbf\{w\}\)\+\\frac\{\\mu\}\{2\}\\\|\\mathbf\{w\}\-\\mathbf\{w\}\(t\-1\)\\\|^\{2\}\(4\)where𝐰​\(t−1\)\\mathbf\{w\}\(t\-1\)is the global parameters at previous rounds andμ\\muis the weight of the proximal term used\. Whenμ\\muequals 0, this strategy is equivalent to FedAvg\. A smallμ\\mumay not make any difference when compared to FedAvg, while a largeμ\\mumay potentially slow the convergence by forcing the local parameters to be close to the global parameters\.

Federated Adaptive Weighting \(FedAdp\) Algorithm: The FedAdp algorithm\[[14](https://arxiv.org/html/2608.14654#bib.bib6)\]dynamically assigns weights based on the correlation between local and global gradients\. Specifically, it measures node contributions using the angle between local and global gradient vectors and applies a non\-linear mapping function to determine their weighting\. This approach can make a more adaptive and effective global model update\.

![Refer to caption](https://arxiv.org/html/2608.14654v1/x1.png)\(a\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x2.png)\(b\)

Figure 1:Illustration of the preliminary test accuracy of FedAvg and FedAdp over communication rounds under different non\-IID cases\. Demonstrating these algorithms’ limitations on identical, imbalanced, and highly skewed data scenarios\.![Refer to caption](https://arxiv.org/html/2608.14654v1/x3.png)Figure 2:Illustration of FedAdp weighting factor distribution across clients showcasing the evidence that extreme and evenly distributed imbalance reduces weight discrimination, potentially impacting performance\.
### 3\.2Limitation of FedAvg and FedAdp

The limitation of FedAvg is that it has low accuracy and slow convergence on non\-IID data\. FedAdp faces issues when the global gradient is misaligned with the expected direction, often due to nodes lacking full label representation\. When many nodes exhibit this, averaging their local gradients can lead to suboptimal performance, especially if high weights are assigned to misaligned nodes\.

To further explore these limitations, we re\-implement experiments from Wu et al\.\[[14](https://arxiv.org/html/2608.14654#bib.bib6)\], analyzing FedAvg and FedAdp convergence on a ten\-agent CNN model with MNIST under varying non\-IID distributions\. We also introduce a specific non\-IID scenario where FedAdp is expected to underperform\.

Figure[1](https://arxiv.org/html/2608.14654#S3.F1)shows the test accuracy over communication rounds of FedAvg and FedAdp on various non\-IID data scenarios\. FedAvg performs well when there are ten balanced data local nodes, but its performance gradually deteriorates with an increasing number of imbalanced data nodes\. On the other hand, FedAdp’s performance remains quite stable\. However, in 7 imbalanced data nodes with identical data distributions, FedAdp’s performance experiences a sudden decline\.

Figure[2](https://arxiv.org/html/2608.14654#S3.F2)illustrates the weighting factors assigned by FedAdp over communication rounds\. With 10 nodes and balanced data, weights remain stable at around 0\.1\. In the case of 3 imbalanced nodes, each with samples from only 2 classes, FedAdp assigns higher weights to balanced nodes and lower weights to imbalanced ones\. The gap is wider in early rounds but narrows over time\. A similar trend appears with 5 and 7 imbalanced nodes, where balanced nodes receive higher weights\. However, with 7 imbalanced nodes of identical distributions, FedAdp performs poorly as the weight differences between diverse and less diverse nodes diminish\. Although it might seem intuitive that identically distributed clients should receive equal weights, this becomes problematic when those distributions are heavily skewed\. In such cases, FedAdp assigns similar weights to a majority of clients that possess highly imbalanced and non\-representative data \(e\.g\., only a small subset of classes\), which collectively dominate the aggregation process\. This uniform weighting over biased nodes reinforces local data skew and leads the global model to overfit to those limited class patterns, ultimately hindering generalization and slowing convergence\.

Overall, FedAvg exhibits suboptimal performance in non\-IID scenarios\. While FedAdp successfully addresses FedAvg’s limitations in many instances, it may still encounter challenges in more complex distribution scenarios\.

## 4Methodology: Federated Impurity Weighting

This section describes our methodology for improving FL convergence and performance\. We introduce a method for quantifying and weighting each node’s data contribution and then present our FedImp algorithm\.

### 4\.1Impurity Weighting Updating Factor

We use entropy to represent the informational richness of each node’s data\. EntropySiS\_\{i\}quantifies the impurity or information content within nodeii’s dataset and is defined as:

Si=−∑j=1Cpj​logC⁡pjS\_\{i\}=\-\\sum\_\{j=1\}^\{C\}p\_\{j\}\\log\_\{C\}p\_\{j\}\(5\)whereCCdenotes the total number of class labels in the federated learning system, andpjp\_\{j\}represents the proportion of samples belonging to classjjat nodeii\.pjp\_\{j\}is computed as:

pj=\|Di​j\|\|Di\|p\_\{j\}=\\frac\{\|D\_\{ij\}\|\}\{\|D\_\{i\}\|\}\(6\)where\|Di​j\|\|D\_\{ij\}\|is the number of local samples at nodeiithat belong to classjj, and\|Di\|\|D\_\{i\}\|is the total number of training samples at nodeii\. This definition ensures that the entropy scoreSiS\_\{i\}is computed consistently across nodes with varying local class distributions\. Note thatlogC\\log\_\{C\}denotes the logarithm to the baseCC, which ensures thatSiS\_\{i\}is normalized between 0 and 1\. Higher entropy reflects greater data diversity or uncertainty, while lower entropy indicates more homogeneous or certain data\.

The value ofSiS\_\{i\}lies within the inclusive interval from 0 to 1\. The closerSiS\_\{i\}is to 0, the more concentrated the dataset is around a single class, while the closerSiS\_\{i\}is to 1, the more evenly distributed the dataset is among different classes, indicating higher uncertainty or impurity\. Expressly,SiS\_\{i\}assumes a value of 0 when the examples within the dataset exclusively pertain to a single class, reflecting a state of utmost certainty\. On the contrary,SiS\_\{i\}attains a value of 1 when the distribution of examples across various classes is perfectly equal, illustrating a scenario of maximum uncertainty\.

Weighting Factor: To weight nodes during global model aggregation, we use a Softmax function \(with temperatureτ\>0\\tau\>0\) applied to the impurity values calculated for each node:

ψi​\(t\)=Di​eSiτ∑i′=1KDi′​eSi′τ\\psi\_\{i\}\(t\)=\\frac\{D\_\{i\}e^\{\\frac\{S\_\{i\}\}\{\\tau\}\}\}\{\\sum^\{K\}\_\{i^\{\\prime\}=1\}D\_\{i^\{\\prime\}\}e^\{\\frac\{S\_\{i^\{\\prime\}\}\}\{\\tau\}\}\}\(7\)where:

- •ψi​\(t\)\\psi\_\{i\}\(t\)represents the aggregation weight assigned to clientiiat communication roundtt\.
- •DiD\_\{i\}denotes the number of local training data at clientii\.
- •SiS\_\{i\}is the entropy\-based impurity score of clientii, computed in equation \([5](https://arxiv.org/html/2608.14654#S4.E5)\)\.
- •τ\\tauis the temperature parameter, which controls the sharpness of the weighting distribution\. A lowerτ\\tauincreases the contrast between high\- and low\-entropy clients\.
- •KKis the number of clients participated in each round\.

Choosing an appropriate value for the temperature parameterτ\\tauis crucial\. Ifτ\\tauis set too low, the softmax function becomes sharp, resulting in overly aggressive weighting that may exaggerate small differences in entropy and disproportionately favor a few nodes\. Conversely, a high value ofτ\\tauleads to flatter weighting, making the aggregation resemble uniform averaging and diminishing the benefit of impurity\-aware differentiation\. A suitableτ\\taubalances these effects to guide meaningful contributions from diverse clients while avoiding dominance or dilution\. A detailed empirical study on the sensitivity ofτ\\tauis provided later in Section[5](https://arxiv.org/html/2608.14654#S5)\.

After forming weights for the participating nodes, the global model parameters𝐰\\mathbf\{w\}are updated as in equation \([2](https://arxiv.org/html/2608.14654#S3.E2)\)\.

The temperature parameterτ\\tauserves as a parameter controlling the difference between the weights of nodes\. A smallτ\\tauemphasizes the contribution of nodes with more information while diminishing the influence of nodes with less information\. However, it is crucial to note that using a small value ofτ\\tauis not universally effective\. In a hypothetical scenario where nodes with less information possess samples belonging to classes not present in nodes with more information, it would not be reasonable to diminish the influence of the less informative nodes significantly\. In such cases, maintaining a certain level of influence from these nodes becomes essential, as they contribute unique information that is not available in nodes with more comprehensive data\. Striking a balance in the FL process is crucial to ensure that all relevant information is considered and integrated, even from nodes with less data, to avoid losing valuable insights or potential improvements for the entire model\. The subsequent section empirically verifies the effect of differentτ\\tauvalues\.

### 4\.2FedImp Algorithm

Algorithm 1Federated Impurity Weighting Algorithm1:

TT: the number of communication rounds,

KK: the number of participating nodes in each round,

BB: the local minibatch size,

EE: the number of local epochs,

η\\eta: the learning rate,

τ\\tau: the control parameter

2:Server execute:

3:Initialize

𝐰​\(𝟎\)\\mathbf\{w\(0\)\}\.

4:for

t=1,2,…,Tt=1,2,\\ldots,Tdo

5:Choose random set of

KKnodes\.

6:foreach node

i∈Ki\\in Kin paralleldo

7:

𝐰𝐢​\(𝐭\)←LocalUpdate​\(i,𝐰​\(t−1\)\)\\mathbf\{w\_\{i\}\(t\)\}\\leftarrow\\text\{LocalUpdate\}\(i,\\mathbf\{w\}\(t\-1\)\)
8:endfor

9:Calculate

ψi\\psi\_\{i\}\(equations \([5](https://arxiv.org/html/2608.14654#S4.E5)\), \([7](https://arxiv.org/html/2608.14654#S4.E7)\)\)\.

10:

𝐰​\(t\)=∑i=1Kψi​𝐰i​\(t\)\\mathbf\{w\}\(t\)=\\sum^\{K\}\_\{i=1\}\\psi\_\{i\}\\mathbf\{w\}\_\{i\}\(t\)\.

11:endfor

12:Local execute:

13:FunctionLocalUpdate\(

i,𝐰i,\\mathbf\{w\}\):

14:for

e=1,2,…,Ee=1,2,\\ldots,Edo

15:for

b=1,2,…,⌈DiB⌉b=1,2,\\ldots,\\lceil\\frac\{D\_\{i\}\}\{B\}\\rceildo

16:

𝐰←𝐰−η​∇F​\(𝐰\)\\mathbf\{w\}\\leftarrow\\mathbf\{w\}\-\\eta\\nabla F\(\\mathbf\{w\}\)
17:endfor

18:endfor

19:return𝐰\\mathbf\{w\}

Algorithm[1](https://arxiv.org/html/2608.14654#alg1)outlines the FedImp algorithm, which iterates overTTcommunication rounds\. In each round, a randomly selected subset of \(K\) nodes participates\. Key algorithm parameters include the local minibatch sizeBB, the number of local epochsEE, the learning rateη\\eta, and the control parameterτ\\tau\. A detailed step\-by\-step description follows\.

- •Line 2: Initialize the global model parameters𝐰​\(𝟎\)\\mathbf\{w\(0\)\}
- •Lines 3\-10: Present main loop for communication roundsttfrom 1 toTT, item Line 4: Select a random set ofKKnodes for the current round
- •Lines 5, 6: Loop through each nodeiiin parallel and update its local model using the LocalUpdate function, which is further described later in the algorithm
- •Line 8: Calculateψi\\psi\_\{i\}using equations \([5](https://arxiv.org/html/2608.14654#S4.E5)\) and \([7](https://arxiv.org/html/2608.14654#S4.E7)\)
- •Line 9: Aggregate local models based on the calculated weightsψi\\psi\_\{i\}to update the global model𝐰​\(t\)\\mathbf\{w\}\(t\)\.
- •Lines 12\-18: Present the definition of the LocalUpdate function, which runs on each nodeii\.
- •Line 13\-17: Loop for local epochsee
- •Lines 14\-16: Loop for local mini\-batchesbb, updating the local model parameters𝐰\\mathbf\{w\}using stochastic gradient descent with step sizeη\\eta
- •Line 18: Return the updated local model parameters\.

FedImp, like FedAvg and FedProx, aggregates local model parameters𝐰i​\(t\)\\mathbf\{w\}\_\{i\}\(t\)instead of raw gradients\. This choice supports multiple local epochs between rounds, letting models adapt more to local data before syncing\. It avoids coordinating learning rates or batch stats across clients and ensures stable convergence in non\-IID settings\. For FedImp, parameter aggregation also aligns well with entropy\-based weighting, without requiring extra gradient tracking\.

Derived from equation \([7](https://arxiv.org/html/2608.14654#S4.E7)\), when all participating nodes possess an identical quantity of data samples, the FedImp algorithm will allocate weights exclusively predicated on their respective impurity levels\. Conversely, suppose there are variations in the sizes of data samples across participating nodes\. In that case, FedImp will assign weights by considering both the impurity and the size of the data at each node\.

### 4\.3Complexity and Convergence Analysis

#### 4\.3\.1Computational Cost Per Client

Each client performs local training using mini\-batch stochastic gradient descent \(SGD\) before sending the updated model to the server\. The computational complexity per client primarily depends on the number of local epochsEE, the number of samples per clientDiD\_\{i\}, and the model size \(total number of parameters\)PP\.

For FedAvg and FedProx, the computational cost per client per communication round can be expressed as:

𝒪​\(E​Di​P\)\.\\mathcal\{O\}\(ED\_\{i\}P\)\.\(8\)
In FedImp, there is an additional step for computing the entropy\-based impurity weightψi\\psi\_\{i\}\(equation \([7](https://arxiv.org/html/2608.14654#S4.E7)\)\), which is derived from the entropy calculation forDiD\_\{i\}equation \([5](https://arxiv.org/html/2608.14654#S4.E5)\)\)\. Since the entropy computation requires iterating over all local samples to compute class proportions, its complexity is𝒪​\(Di\)\\mathcal\{O\}\(D\_\{i\}\)\. Thus, the total complexity per client in FedImp is:

𝒪​\(E​Di​P\+Di\)\\mathcal\{O\}\(ED\_\{i\}P\+D\_\{i\}\)\(9\)
In typical deep learning scenarios where the model sizePPis significantly larger than the number of classesCC, the𝒪​\(Di\)\\mathcal\{O\}\(D\_\{i\}\)term becomes negligible compared to𝒪​\(E​Di​P\)\\mathcal\{O\}\(ED\_\{i\}P\)\. As such, the overall computational complexity of FedImp remains𝒪​\(E​Di​P\)\\mathcal\{O\}\(ED\_\{i\}P\)\.

In practical implementations, the impurity computation per client involves a single pass over local labels to count class frequencies, followed by a computation of entropy using these counts\. This can be implemented efficiently using a histogram\-based class counter, requiring negligible memory and minimal additional computation\. Even on resource\-constrained devices such as IoT sensors or mobile phones, this overhead is expected to be minimal, especially when compared to the cost of local model training \(i\.e\., forward and backward passes in deep networks\)\. Thus, FedImp remains feasible for deployment in low\-resource federated environments\.

#### 4\.3\.2Communication Overhead Per Round

The communication overhead in FL is primarily determined by the number of parameters transmitted between clients and the central server\. In each round, every client sends its model update𝐰i\\mathbf\{w\}\_\{i\}\(a vector of sizePP\) to the server, which aggregates updates and transmits the global model𝐰\\mathbf\{w\}back to the clients\.

For FedAvg and FedProx, each client transmits and receives𝐰\\mathbf\{w\}of sizePP, leading to a communication cost per client per round of𝒪​\(P\)\\mathcal\{O\}\(P\)\.

For FedImp, an additional scalar valueψi\\psi\_\{i\}must be transmitted for weighting\. Sinceψi\\psi\_\{i\}is a single floating\-point value, its communication overhead is negligible compared to the model parameters\. Thus, the total communication cost for FedImp remains𝒪​\(P\)\\mathcal\{O\}\(P\)\.

In summary, FedImp introduces an additional entropy calculation step, but this does not change its overall computational complexity compared to FedAvg and FedProx\. Additionally, the communication overhead remains unchanged since the weighting factorψi\\psi\_\{i\}is a negligible scalar value\.

#### 4\.3\.3Convergence of FedImp

The convergence of FedImp is driven by its impurity\-based weighting mechanism, which ensures that clients with more diverse and representative data contribute more significantly to the global model\. Unlike FedAvg, which weights updates solely based on dataset size, FedImp prioritizes clients with higher entropy \(diverse class distributions\), promoting balanced and informative aggregation\.

In particular, FedImp tends to exhibit improved convergence behavior under the following conditions:

- •The number of classesCCis moderate relative to dataset size, allowing entropy to effectively reflect class diversity\.
- •There exists a mix of balanced and imbalanced clients, where entropy weighting can help differentiate and amplify the impact of balanced clients\.
- •The local training process follows standard SGD or its variants with bounded gradients and smooth loss functions\.

These conditions align with typical non\-IID federated settings where some nodes possess skewed or low\-diversity data\. By emphasizing updates from more representative clients, FedImp reduces bias and accelerates convergence toward a global optimum\. While this study focuses on empirical validation of convergence improvements, a formal theoretical convergence bound for FedImp remains an important direction for future work\.

## 5Experiments and Results

This section evaluates our proposed FedImp algorithm and compares its performance with FedAvg, FedProx, and FedAdp\.

### 5\.1Dataset, Federated Learning Models, and Testing Scenario

#### 5\.1\.1Creation of Non\-IID Data

The experiments are carried out on two datasets:

- •EMNIST\[[2](https://arxiv.org/html/2608.14654#bib.bib14)\]: A 28x28 pixel handwritten character dataset, we use the Balanced split \(47 characters, equal samples per class\) with 112,800 training and 18,800 testing samples\.
- •CIFAR\-10\[[7](https://arxiv.org/html/2608.14654#bib.bib11)\]: 60,000 32x32 color images across 10 classes \(6,000 per class\), with 50,000 training and 10,000 testing images\.

These datasets provide a standardized, well\-understood benchmark, allowing direct comparisons with existing FL methods while ensuring the experiments remain computationally efficient\. Future work can extend FedImp’s evaluation to larger and more complex datasets\.

To simulate a diverse population across client nodes, the same approach as in\[[12](https://arxiv.org/html/2608.14654#bib.bib15)\]is used with some modifications\. Each client has an associated multinomial distribution over classes, which is drawn from a symmetric Dirichlet distribution,q∼Dir​\(θ\)q\\sim\\text\{Dir\}\(\\theta\)\. The positive numberθ\>0\\theta\>0serves as a concentration parameter controlling the level of balance among classes\. A sufficiently largeθ\\thetacontributes to the creation of a balanced dataset, whereas a sufficiently smallθ\\thetaresults in an imbalanced dataset\. Whenθ\\thetaincreases to infinity, all clients exhibit identical distributions\. Conversely, asθ\\thetaapproaches 0, each client exclusively possesses examples from a single class chosen at random\.

Algorithm 2Data partition1:

NbN\_\{b\},

Ni​m​bN\_\{imb\},

θb\\theta\_\{b\},

θi​m​b\\theta\_\{imb\},

MM
2:for

i=1,2,…,Nb\+Ni​m​bi=1,2,\\ldots,N\_\{b\}\+N\_\{imb\}do

3:if

i≤Nbi\\leq N\_\{b\}then

4:Sample

q∼Dir​\(θb,C\)q\\sim\\text\{Dir\}\(\\theta\_\{b\},C\)
5:else

6:Sample

q∼Dir​\(θi​m​b,C\)q\\sim\\text\{Dir\}\(\\theta\_\{imb\},C\)
7:endif

8:

Di=ØD\_\{i\}=\\text\{\\O \}
9:for

j=1,2,…,Mj=1,2,\\ldots,Mdo

10:Sample

y∈C​with probability​qy\\in C\\text\{ with probability \}q
11:Sample randomly

x∈Syx\\in S\_\{y\}
12:

Di=Di∪\(x,y\)D\_\{i\}=D\_\{i\}\\cup\{\(x,y\)\}
13:

Sy=Sy∖\(x,y\)S\_\{y\}=S\_\{y\}\\setminus\{\(x,y\)\}
14:if

\|Sy\|=0\|S\_\{y\}\|=0then

15:

C∖yC\\setminus y
16:

q←ReNormalize​\(q,y\)q\\leftarrow\\text\{ReNormalize\}\(q,y\)
17:endif

18:endfor

19:endfor

20:Support function:

21:FunctionReNormalize\(

q=\(p1,p2,…,pC\),yq=\(p\_\{1\},p\_\{2\},\.\.\.,p\_\{C\}\),y\):

22:

py=0p\_\{y\}=0
23:

a=∑i=1Cpia=\\sum\_\{i=1\}^\{C\}p\_\{i\}
24:

q=q/aq=q/a
25:return

qq

Algorithm[2](https://arxiv.org/html/2608.14654#alg2)outlines the process of partitioning the training set with a specified number of balancedNbN\_\{b\}and imbalancedNi​m​bN\_\{imb\}data nodes\. Each node hasMMsamples\. The distribution of data involves sampling class probabilities from a Dirichlet distribution with concentration parametersθb\\theta\_\{b\}andθi​m​b\\theta\_\{imb\}for balanced and imbalanced nodes, respectively\. The procedures of Algorithm[2](https://arxiv.org/html/2608.14654#alg2)are summarized below\.

- •Lines 1 to 18: Loop over each nodeiifrom 1 to \(Nb\+N​i​m​bN\_\{b\}\+N\{imb\}\) to generate its data sampling\.
- •Lines 2 to 5: Ifiiis less than or equal toNbN\_\{b\}, the associated multinomial distributionqqis obtained from a Dirichlet distribution with concentration parameterθ1\\theta\_\{1\}andCCcategories\. Otherwise, otherwise, it is sampled with concentration parameterθ2\\theta\_\{2\}\.
- •Lines 8 to 17: For each nodeii, loop throughMMsamples with the following steps:
- •Line 9 to 12: Sample a class labelyybased on the probability distributionqq, randomly select a sample\(x,y\)\(x,y\)from the remaining samples of classyyin the datasetSS, add the sample\(x,y\)\(x,y\)to the datasetDiD\_\{i\}, remove the selected sample from the datasetSyS\_\{y\}of classyy\.
- •Lines 13 to 16: If there are no more samples for classyy, removeyyfrom the set of classesCC, and re\-normalize the class probabilitiesqq\. TheR​e​N​o​r​m​a​l​i​z​e​\(q,y\)ReNormalize\(q,y\)function ensures that the class probabilitiesqqare normalized after removing a class\. This partitioning method guarantees all samples from the original dataset are sampled and prevents any duplication of samples within nodes\.

For the experiments, we setθb=100\\theta\_\{b\}=100andθi​m​b=0\.01\\theta\_\{i\}mb=0\.01\. The entire dataset is used, resulting inM=SNb\+N​i​m​bM=\\frac\{S\}\{N\_\{b\}\+N\{imb\}\}samples allocated to each node\. To evaluate the performance of the global model, we follow standard federated learning evaluation practice by using a centralized test set\. Specifically, we use the entire original test set of each dataset \(EMNIST and CIFAR\-10\) as a global benchmark, and after each communication round, the updated global model is evaluated on this fixed test set\. This ensures consistency and comparability across different algorithms and data distribution scenarios\.

![Refer to caption](https://arxiv.org/html/2608.14654v1/x4.png)\(a\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x5.png)\(b\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x6.png)\(c\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x7.png)\(d\)

Figure 3:Models architectures: \(a\) MLP and \(b\) 2\-layer CNN for EMNIST; \(c\) 2\-layer CNN and \(d\) 4\-layer CNN for CIFAR10
#### 5\.1\.2Structures of the Federated Learning Models

The experiments involve the FL training of four artificial neural networks \(ANN\) models\. The first two are an MLP model and a 2\-layer CNN model for EMNIST data\. For the CIFAR10 data, we test 2\-layer CNN and 4\-layer CNN models\. Their architectures \(see Figure[3](https://arxiv.org/html/2608.14654#S5.F3)\) are as follows:

- •MLP for EMNIST: This model has two fully connected hidden layers of 200 units using ReLU activation\. The input layer is set to fit the EMNIST image size of 28x28, while the output layer contains 47 neurons with Softmax activation for EMNIST characters classification\.
- •2\-layer CNN for EMNIST: This model has two 5x5 convolutional layers \(the first with 32 channels, the second with 64, each followed with 2x2 max pooling\) and a fully connected layer with 512 units using ReLu activation\. The input and output layers have the same format as in the MLP model for the EMNIST data\.
- •2\-layer CNN for CIFAR10: The main components of this model are the same as in the 2\-layer CNN for EMNIST, except that the input layer has more neurons to match the CIFAR10 input image size\. The output layer of this model is changed to contain 10 neurons for classifying CIFAR10 images\.
- •4\-layer CNN for CIFAR10: This model has four convolutional layers \(32, 32, 64, and 64 channels, 3x3 kernels, ReLU activation\), with max pooling after the second and fourth layers\. Two fully connected layers \(512 and 128 units, 40% dropout, ReLU\) follow\. Input and output layers match the 2\-layer CNN model for CIFAR10\.

#### 5\.1\.3Testing scenarios

To compare the FedImp algorithm with FedAvg, FedProx, and FedAdp algorithms, the testing scenarios are devised with changes in the number of nodes with balanced data and nodes with imbalanced data\. For each of the two datasets, EMNIST and CIFAR10, four testing scenarios are generated for evaluation as follows:

- •7 balanced nodes \+ 3 imbalanced nodes \(non\-IID\)
- •5 balanced nodes \+ 5 imbalanced nodes \(non\-IID\)
- •3 balanced nodes \+ 7 imbalanced nodes \(non\-IID\)
- •3 balanced nodes \+ 7 identically distributed and imbalanced nodes \(extreme skew non\-IID\)

The data in the first three non\-IID scenarios is partitioned using Algorithm[2](https://arxiv.org/html/2608.14654#alg2)withθb=100\\theta\_\{b\}=100,θi​m​b=0\.01\\theta\_\{imb\}=0\.01\.

The last scenario represents the case of FL with 3 balanced nodes \+ 7 identically distributed and imbalanced nodes\. In this case, the training data of the imbalanced nodes are randomly selected from just a few classes of data instead of from all classes as in the other case\. This is an extreme non\-IID scenario where the data is highly skewed to just a few classes\. In each of the 3 balanced nodes,MMsamples will be randomly sampled from the original training set, while in the 7 identically distributed and imbalanced nodes,MMsamples will be randomly sampled from a subset of only 2 classes for CIFAR and only 10 digits for EMNIST \(omitting letters\)\. Specifically, for the EMNIST dataset, the 10 digits \(classes 0–9\) were used as the fixed subset across all runs to ensure consistency\. For the CIFAR dataset, a random subset of 2 classes was chosen for each run\. In this scenario, the sampling process adheres to a mechanism of sampling with replacement, which ensures a sufficient number of samples per node\.

#### 5\.1\.4Pre\-processing and Hyperparameter Configuration

Input Image Pre\-processing: Several image transformations are executed before training the FL models\. For the EMNIST images, the normalization is applied to the pixel arrays to have a mean of 0\.5 and a standard deviation of 0\.5\. The CIFAR\-10 images are processed with random cropping of size 32 with padding of 4 pixels, random horizontal flipping, and normalization of the pixels arrays to have a mean of 0\.5 and a standard deviation of 0\.5\.

Hyperparameter: The experiments employ all available nodes for both training at each round \(i\.e\., a utilization fraction of 1\.0\)\. During each round, client nodes train their local models individually for one epoch with a batch size of 100\. The local model training employs the Stochastic Gradient Descent \(SGD\) optimization algorithm to minimize cross\-entropy loss\. The learning rate is set to 0\.1, and a decay rate of 0\.995 is applied after each communication round\. Additionally, the weightμ\\muof the proximal term in FedProx is set to 0\.1, the constantα\\alphain the non\-linear mapping function of FedAdp is set to 5, the temperature parameterτ\\tauin the weight forming function of FedImp is set to 0\.7\. After each round, the global model’s performance is assessed using the test sets from all participating nodes\. The values for key hyperparameters such asμ=0\.1\\mu=0\.1for FedProx,α=5\\alpha=5for FedAdp, andτ=0\.7\\tau=0\.7for FedImp were selected based on preliminary experiments and empirical tuning on a small validation split derived from the training data\. The goal was to ensure that each method operated under reasonably optimized settings\. For FedProx and FedAdp, we also consulted values recommended in their original papers\. We fixed these values across all scenarios to ensure fairness and reproducibility in our comparisons\.

### 5\.2Results and Discussion

To compare the convergence rates of FedImp, FedAvg, FedProx, and FedAdp algorithms, we measure the number of communication rounds needed to reach convergence\.

![Refer to caption](https://arxiv.org/html/2608.14654v1/x8.png)\(a\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x9.png)\(b\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x10.png)\(c\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x11.png)\(d\)

Figure 4:MLPmodel forEMNISTresults of test accuracy over communication rounds on FedAvg, FedProx, FedAdp, and FedImp algorithms with different levels of heterogeneous data distribution over participating nodes\.Table 1:The number of communication rounds for each FL algorithm to reach over target test accuracy with theMLPmodel forEMNISTdata\. N/A indicates that the algorithm cannot achieve the target accuracy\.#### 5\.2\.1Results on EMNIST data

MLP model for EMNIST results:Figure[4](https://arxiv.org/html/2608.14654#S5.F4)illustrates the performance of FedAvg, FedProx, FedAdp, and FedImp algorithms in terms of test accuracy across communication rounds under various heterogeneous data distribution scenarios among participating nodes, employing the MLP model\. Table[1](https://arxiv.org/html/2608.14654#S5.T1)summarizes the number of communication rounds required to reach a certain accuracy level, chosen based on the maximum accuracy of all four algorithms on the EMNIST dataset in each data distribution scenario\. Across all scenarios, the proposed FedImp consistently displays a fast convergence rate compared to other algorithms\. The observed results are as follows:

- •In the scenario featuring 7 balanced nodes and 3 imbalanced nodes, FedImp closely aligns with FedProx; they take 35 and 33 rounds, respectively, to surpass 83% accuracy\. Both FedImp and FedProx outperform FedAvg \(79 rounds\) and FedAdp \(71 rounds\)\.
- •In the FL with 5 balanced nodes and 5 imbalanced nodes, FedImp demonstrates a notable advancement, which takes only 47 communication rounds to achieve 83% or higher accuracy\. FedProx, FedAdp, and FedAvg require 56, 94, and 132 rounds, respectively\. In this case, FedImp has shown large reductions in communication rounds of approximately 16\.1%, 50%, and 64\.4% less than FedProx, FedAdp, and FedAvg\.
- •In the scenario of 3 balanced nodes and 7 imbalanced nodes, FedProx and FedAdp require 108 and 234 rounds to reach the required 83% accuracy\. FedAvg cannot reach target accuracy in this case\. It can approach nearly 83% after more than 300 rounds of communication\. The proposed FedImp algorithm exhibits clear superiority over all other algorithms, as it takes just 78 rounds to reach the target accuracy\. FedImp has required fewer rounds by approximately 27\.8%, 66\.7% than FedProx and FedAdp\.
- •In the last case of 3 balanced nodes \+ 7 identically distributed and imbalanced nodes, FedImp has clearly outperformed all other algorithms, as it is the only one that can converge at 79% accuracy\. All three other algorithms \(FedProx, FedAdp, and FedAvg\) only obtained an accuracy of around 77%\.

![Refer to caption](https://arxiv.org/html/2608.14654v1/x12.png)\(a\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x13.png)\(b\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x14.png)\(c\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x15.png)\(d\)

Figure 5:2\-layer CNNmodel forEMNISTresults on test accuracy over communication rounds of FedAvg, FedProx, FedAdp, and FedImp with different levels of heterogeneous data distribution over participating nodes\.Table 2:The number of communication rounds for FL algorithms to reach over target test accuracy with the2\-layer CNNmodel forEMNISTdata\. N/A indicates that the algorithm cannot achieve the target accuracy\.Testing scenarioACCFedAvgFedProxFedAdpFedImp7 bal \+ 3 imbal87%261727185 bal \+ 5 imbal87%412135193 bal \+ 7 imbals87%8150N/A483 bal \+ 7 iden\. dist\. imbal83%N/AN/A2011152\-Layer CNN for EMNIST results:Figure[5](https://arxiv.org/html/2608.14654#S5.F5)illustrates test accuracy over communication rounds for all four algorithms, considering diverse scenarios of heterogeneous data distribution among participating nodes and utilizing the 2\-layer CNN model\. Table[2](https://arxiv.org/html/2608.14654#S5.T2)presents the communication rounds required for each algorithm to achieve an accuracy level determined by the best accuracy of all algorithms for the data distribution scenario\. The results have shown the advancement of FedImp over other algorithms\.

- •In the scenario of 7 balanced nodes and 3 imbalanced nodes, FedImp and FedProx reach the target accuracy \(87%\) after 18 and 17 rounds, respectively, whereas FedAvg and FedAdp require 26 and 29 rounds\.
- •In the scenario of 5 balanced nodes and 5 imbalanced nodes, FedImp achieves the target accuracy \(87%\) in 19 rounds, indicating a slight improvement over FedProx \(needs 21 rounds\), and demonstrating superiority over FedAvg \(needs 41 rounds\), and FedAdp \(needs 35 rounds\), with reductions of 53\.7% and 45\.7% in that order\.
- •The results in the case of FL with 3 balanced nodes and 7 imbalanced nodes further highlight the efficiency of FedImp\. It achieves convergence in 48 rounds and demonstrates a remarkable reduction in the number of rounds, with an efficiency gain of 4% over FedProx and an even more substantial improvement of about 40\.7% over FedAvg\. In this scenario, FedAdp fails to reach the 87% accuracy target\.
- •For the case of 3 balanced nodes \+ 7 identically distributed and imbalanced nodes, the target accuracy is 83%\. Only FedImp \(115 rounds\) and FedAdp \(201 rounds\) can achieve this accuracy level\. However, FedAdp later converged to just around 82% accuracy, while FedImp still maintained its higher performance\.

![Refer to caption](https://arxiv.org/html/2608.14654v1/x16.png)\(a\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x17.png)\(b\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x18.png)\(c\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x19.png)\(d\)

Figure 6:2\-layer CNNmodel forCIFAR\-10results on test accuracy over communication rounds of FedAvg, FedProx, FedAdp, and FedImp with different levels of heterogeneous data distribution over participating nodes\.Table 3:The number of communication rounds for FL algorithms to reach over target test accuracy with the2\-layer CNNmodel forCIFAR\-10data\. N/A indicates that the algorithm cannot achieve the target accuracy\.Testing scenarioACCFedAvgFedProxFedAdpFedImp7 bal \+ 3 imbal80%2472372312355 bal \+ 5 imbal79%4523632552523 bal \+ 7 imbal76%2702502202113 bal \+ 7 iden\. dist\. imbal72%N/AN/AN/A408
#### 5\.2\.2Results on CIFAR\-10 Data

2\-Layer CNN for CIFAR\-10 results:Figure[6](https://arxiv.org/html/2608.14654#S5.F6)demonstrates the evolving test accuracy across communication rounds for FedAvg, FedProx, FedAdp, and FedImp, taking into account diverse scenarios of heterogeneous data distribution among participating nodes and employing the 2\-layer CNN model\. Table[3](https://arxiv.org/html/2608.14654#S5.T3)presents the results for the number of communication rounds required by FedAdp, FedAvg, and FedImp algorithms to achieve the target accuracy\.

- •In the scenario with 7 balanced nodes and 3 imbalanced nodes, FedImp takes 235 rounds to surpass 80%, while FedAvg and FedProx require a slightly higher number of rounds, namely 247 and 237 in the order given\. Nonetheless, FedImp is slightly slower than FedAdp, which takes 231 rounds\.
- •For the case with 5 balanced nodes and 5 imbalanced nodes, FedImp outperforms FedAvg, FedProx, and FedAdp by converging at 79% in 252 rounds, while FedAvg, FedProx, and FedAdp require 452, 363, and 255 rounds, respectively\. This demonstrates a reduction of rounds by about 44\.2%, 30\.6%, and 1\.2% for FedImp compared to FedAvg, FedProx, and FedAdp, respectively\.
- •In the scenario of 3 balanced nodes and 7 imbalanced nodes, FedImp again showcases its efficiency by achieving convergence to 76% accuracy in 211 rounds, while FedAvg, FedProx, and FedAdp require 270, 250, and 220 rounds, respectively\. This reflects a reduction of rounds by about 21\.9%, 15\.6%, and 4\.1% when comparing FedImp to FedAvg, FedProx, and FedAdp in that order\.
- •In the scenario of 3 balanced nodes \+ 7 identically distributed and imbalanced nodes, FedImp is the only algorithm that can converge at 72%\. All three other algorithms can only obtain an accuracy of around 65%\.

4\-Layer CNN for CIFAR\-10 results:Figure[7](https://arxiv.org/html/2608.14654#S5.F7)illustrates test accuracy across communication rounds for FedAvg, FedProx, FedAdp, and FedImp, considering various scenarios of heterogeneous data distribution among nodes using the 4\-layer CNN model\. In scenarios involving only 3 imbalanced data nodes, there is not a notable disparity in algorithm performance\. However, in scenarios with a greater number of imbalanced data nodes, both FedImp and FedAdp demonstrate a superior convergence rate compared to FedAvg and FedProx\. Table[4](https://arxiv.org/html/2608.14654#S5.T4)displays the number of communication rounds required to attain over the target accuracy on the test set using the 4\-layer CNN model in different testing scenarios\.

- •In a setting featuring 7 balanced nodes and 3 imbalanced nodes, FedImp achieves an over 83% accuracy after 533 rounds, while FedAvg and FedProx achieve similar target accuracy slower, at 577 and 677 rounds respectively\. However, FedImp is slightly slower than FedAdp, which requires 528 rounds\.
- •Transitioning to the case of 5 balanced nodes and 5 imbalanced nodes, FedImp requires 296 rounds to surpass the 80% accuracy target\. FedImp has demonstrated a substantially lower number of communication rounds as opposed to 398 rounds for FedAdp, 526 rounds for FedAvg, and 529 rounds for FedProx\. This signifies a decrease in the number of rounds by around 43\.7%, 44%, and 25\.6% when comparing FedImp to FedAvg, FedProx, and FedAdp in the order given\.
- •In the scenario with 3 balanced nodes and 7 imbalanced nodes, while FedAvg and FedProx cannot reach the target accuracy of 80%, FedImp demonstrates a slower convergence, reaching convergence in 599 rounds, whereas FedAdp achieves the same in 424 rounds\.
- •In the scenario of 3 balanced nodes \+ 7 identically distributed and imbalanced nodes, FedImp is still the only algorithm that can obtain 72% or higher accuracy level\. In contrast, FedProx, FedAdp, and FedImp converge at lower accuracy of around 60% to 65%\.

![Refer to caption](https://arxiv.org/html/2608.14654v1/x20.png)\(a\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x21.png)\(b\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x22.png)\(c\)
![Refer to caption](https://arxiv.org/html/2608.14654v1/x23.png)\(d\)

Figure 7:4\-layer CNNmodel forCIFAR\-10results on test accuracy over communication rounds of FedAvg, FedProx, FedAdp, and FedImp with different levels of heterogeneous data distribution over participating nodes\.Table 4:The number of communication rounds for FL algorithms to reach over target test accuracy with the4\-layer CNNmodel forCIFAR\-10data\. N/A indicates that the algorithm cannot achieve the target accuracy\.Table 5:Communication Overhead Comparison for Different FL Algorithms\. The reported per\-round overhead for FedImp \(25\.5 MB\) includes the minor additional cost of transmitting the scalar weighting factorψi\\psi\_\{i\}\.To assess the feasibility of FedImp in real\-world FL deployments, we conducted additional experiments to measure its communication overhead in comparison to FedAvg and FedProx\. Since communication efficiency is a critical factor in FL, particularly in IoT, mobile computing, and healthcare applications, understanding the bandwidth requirements of FedImp is essential\. The objective was to analyze the following three aspects:

- •The number of communication rounds required to reach 80% accuracy\.
- •The communication overhead per round \(i\.e\., the size of model updates exchanged between clients and the server in MB\)\.
- •The total communication cost until convergence, computed as the product of per\-round overhead and the number of rounds\.

Each client transmits and receives the full model update in every round\. The base model size used in this experiment was 2\.5 MB, consistent across all algorithms\. Since FedImp requires an additional impurity\-based weighting step, there was a minor increase in communication overhead due to the transmission of the weighting factorψi\\psi\_\{i\}, though this is negligible compared to the model parameters\. Table[5](https://arxiv.org/html/2608.14654#S5.T5)presents the experimental results comparing FedAvg, FedProx, and FedImp in terms of communication efficiency\. The slight increase in FedImp’s per\-round communication cost \(25\.5 MB\) compared to FedAvg and FedProx \(25\.0 MB\) accounts for the additional transmission of the scalar weighting factorψi\\psi\_\{i\}\. Although negligible in size, it is included in the reported overhead for completeness\.

As seen in the results, FedImp consistently requires fewer communication rounds to reach the same target accuracy compared to FedAvg and FedProx\. Specifically, FedImp reduces the number of rounds by approximately 44\.2% compared to FedAvg and 30\.6% compared to FedProx\. This results in a significant reduction in the total communication cost, with FedImp reducing bandwidth consumption by 43\.1% compared to FedAvg and 29\.2% compared to FedProx\. Although FedImp introduces a slight increase in per\-round communication overhead due to the transmission of impurity\-based weights, this additional cost is negligible\. The efficiency gain from faster convergence significantly outweighs this minor overhead, making FedImp a more communication\-efficient choice in federated learning deployments\.

#### 5\.2\.3Discussion

Based on the obtained results, we can observe that the proposed FedImp algorithm can achieve comparable or better test accuracy using considerably fewer communication rounds than FedAvg, FedProx, and FedAdp\. This is particularly evident in cases with a higher number of imbalanced data nodes\. These results demonstrate that FedImp offers a significant advantage in federated learning scenarios with non\-IID data distributions\. By requiring fewer communication rounds to achieve target accuracy, FedImp reduces communication overhead and improves the efficiency of the FL process\. This is particularly important for large\-scale deployments where communication costs can be substantial\.

##### Why FedImp Outperforms FedAvg, FedProx, and FedAdp

The superior performance of FedImp in reducing communication rounds stems from its entropy\-based impurity weighting mechanism, which dynamically assigns aggregation weights based on the informational richness of each client’s local dataset\. In contrast, traditional FL aggregation methods rely on either static or gradient\-based weighting, which may not sufficiently address data heterogeneity\. Below, we provide a detailed analysis of why FedImp achieves faster convergence\.

Adaptive Weighting for Improved Model Aggregation: FedAvg applies uniform weighting based on dataset size, assuming that clients contribute equally to the global model\. However, in non\-IID settings, larger datasets may not necessarily contain diverse data, leading to slower generalization\. FedImp, in contrast, assigns weights based on entropy\. Higher entropy values indicate more diverse datasets, allowing FedImp to prioritize updates that contribute to better global model generalization\.

Mitigating the Impact of Skewed Data Contributions: FedProx improves upon FedAvg by introducing a proximal term to constrain local updates\. However, it does not differentiate between clients with high\- and low\-quality data distributions, which can still lead to inefficient aggregation\. FedImp explicitly mitigates this issue by down\-weighting nodes with highly imbalanced local datasets, ensuring that the global model is not disproportionately influenced by a small subset of labels\.

Faster Convergence via Balanced Information Contribution:FedAdp attempts to improve convergence by assigning weights based on the local and global gradients\. However, this approach struggles in extreme non\-IID cases where clients lack overlapping label distributions, leading to misleading global gradient directions\. FedImp instead leverages local dataset diversity rather than gradient similarity, enabling faster model stabilization even in highly heterogeneous scenarios\.

Empirical Evidence Supporting FedImp’s Faster Convergence:The experimental results confirm that FedImp requires significantly fewer communication rounds to reach the same accuracy levels as FedAvg, FedProx, and FedAdp\. By adaptively adjusting aggregation weights based on data impurity, FedImp consistently ensures that each communication round contributes maximally to generalization, thereby accelerating convergence while maintaining model robustness\.

##### Impact of Temperature Parameterτ\\tauin FedImp

Choosing the right temperatureτ\\taufor FedImp is crucial\. Lowerτ\\tauvalues emphasize information\-rich nodes while downplaying others, but smallτ\\tauvalues do not guarantee optimal performance\. Figure[8](https://arxiv.org/html/2608.14654#S5.F8)provides a visualization of test accuracy over communication rounds of FedImp with differentτ\\tauvalues in the scenario employing the EMNIST dataset and the MLP model with a data distribution setting of 5 balanced nodes and 5 imbalanced nodes\. In this experimental setup, the most effectiveτ\\tauis 0\.7\.

The value ofτ\\tauaffects the weighting of the local models during aggregation, and choosing an inappropriate value may lead to suboptimal results\. Further research is needed to develop automated methods or heuristics for selecting the optimalτ\\tauvalue based on the characteristics of the dataset and the participating nodes\. Despite this limitation, FedImp shows promising results in improving the efficiency and effectiveness of federated learning in scenarios with imbalanced data distributions\.

![Refer to caption](https://arxiv.org/html/2608.14654v1/x24.png)Figure 8:Test accuracy over communication rounds of FedImp with differentτ\\tau\. The data distribution setting is 5 balanced nodes \+ 5 imbalanced nodes with the EMNIST dataset and the MLP model\.
##### Privacy and Security Considerations

FedImp operates within the standard federated learning framework, where raw data remains on the client device, thereby preserving privacy by design\. However, since aggregation weights are derived from local data statistics \(entropy\), it is important to consider potential privacy implications in adversarial environments\.

Fortunately, the impurity scoreSiS\_\{i\}and weighting factorψi\\psi\_\{i\}can be computed entirely on the client side and do not require sharing any intermediate statistics or raw data with the server\. This makes FedImp compatible with common privacy\-enhancing mechanisms such as Differential Privacy \(DP\) and Secure Aggregation \(SA\)\. For example, DP can be applied to local model updates before transmission, and SA protocols can be used to securely aggregate the weighted updates without revealing individual contributions\.

Future work may explore combining FedImp with formal DP guarantees or integrating it into secure federated protocols, particularly in high\-stakes domains like healthcare or finance where both convergence efficiency and privacy are paramount\.

##### Toward Real\-World Federated Settings

While our experiments use controlled simulations with benchmark datasets \(EMNIST and CIFAR\-10\) and synthetic non\-IID partitions, we acknowledge the importance of validating FedImp in real\-world federated learning deployments\. Such environments introduce additional factors like device availability, communication delays, hardware heterogeneity, and natural data drift\. Evaluating FedImp under these conditions—such as with real mobile users or healthcare data—remains a promising direction for future research to further assess its robustness and generalizability\.

##### Potential Limitation of Entropy\-Based Weighting

While FedImp demonstrates strong performance across various non\-IID scenarios, it has a limitation in the scenario where the entropy\-based weighting is less effective\. An exceptional scenario where clients are highly specialized \(low entropy\) and have mutually exclusive classes poses a serious challenge to FL in general as well as to FedImp\. In this context, each client is often limited to only one or two classes, while there is no overlapping classes among the clients at all\. Such a scenario will lead to a uniform aggregation weighting factor of the clients and reduce the effectiveness of the proposed algorithm\. In such a case, combining entropy with other signals, such as inter\-client label coverage or global class frequency awareness, could be a promising solution to address the challenge\.

## 6Conclusion

This paper presents FedImp, a novel FL algorithm for addressing the challenge of non\-IID data\. Our method entails assessing the contribution of each participating node by analyzing the informational entropy of its data\. These contributions are subsequently standardized to produce unique weights for aggregating the global model\. Through extensive experimental evaluations, FedImp showcases its superior convergence rate compared to FedAvg, FedProx, and FedAdp in various scenarios with non\-IID data\. Experimental findings demonstrate decreases of communication round up to 64\.4%, 27\.8%, and 66\.7% on the EMNIST dataset, relative to FedAvg, FedProx, and FedAdp, respectively\. Moreover, on CIFAR\-10 dataset, FedImp achieves great reductions, ranging up to 44\.2%, 44%, and 25\.6% compared to FedAvg, FedProx, and FedAdp, respectively\. In scenarios with a high number of imbalanced data nodes, such as the case of 3 balanced nodes and 7 identically distributed and imbalanced nodes, FedImp often emerges as the only algorithm among those evaluated that can reach the target convergence accuracy\.

While FedImp offers advantages, practical FL deployment requires considering factors like device computational constraints and privacy implications\. In future work, we plan to evaluate FedImp’s performance across diverse models and datasets, and integrate it with privacy\-preserving techniques, such as Differential Privacy and Secure Aggregation, to mitigate these risks\. Robust aggregation or adversarial defenses could further strengthen FedImp, making it more suitable for real\-world applications where both efficiency and privacy are essential\. Another promising direction is to explore alternative impurity or diversity measures beyond Shannon entropy, such as Gini impurity, Simpson’s index, or kernel\-based diversity metrics\. These alternatives may capture different aspects of client data heterogeneity and could offer improved sensitivity or robustness in weighting decisions under specific distributional scenarios\.

## References

- \[1\]D\. A\. E\. Acar, Y\. Zhao, R\. M\. Navarro, M\. Mattina, P\. N\. Whatmough, and V\. Saligrama\(2021\)Federated learning based on dynamic regularization\.arXiv preprint arXiv:2111\.04263\.Cited by:[§2](https://arxiv.org/html/2608.14654#S2.p4.1)\.
- \[2\]L\. Deng\(2012\)The mnist database of handwritten digit images for machine learning research \[best of the web\]\.IEEE signal processing magazine29\(6\),pp\. 141–142\.Cited by:[1st item](https://arxiv.org/html/2608.14654#S5.I1.i1.p1.1)\.
- \[3\]R\. Fotohi, F\. S\. Aliee, and B\. Farahani\(2024\)A lightweight and secure deep learning model for privacy\-preserving federated learning in intelligent enterprises\.IEEE Internet of Things Journal\.Cited by:[§2](https://arxiv.org/html/2608.14654#S2.p8.1)\.
- \[4\]R\. Fotohi, F\. S\. Aliee, and B\. Farahani\(2024\)Decentralized and robust privacy\-preserving model using blockchain\-enabled federated deep learning in intelligent enterprises\.Applied Soft Computing161,pp\. 111764\.Cited by:[§2](https://arxiv.org/html/2608.14654#S2.p8.1)\.
- \[5\]T\. H\. Hsu, H\. Qi, and M\. Brown\(2019\)Measuring the effects of non\-identical data distribution for federated visual classification\.arXiv preprint arXiv:1909\.06335\.Cited by:[§2](https://arxiv.org/html/2608.14654#S2.p2.1)\.
- \[6\]P\. Kairouz, H\. B\. McMahan, B\. Avent, A\. Bellet, M\. Bennis, A\. N\. Bhagoji, K\. Bonawitz, Z\. Charles, G\. Cormode, R\. Cummings,et al\.\(2021\)Advances and open problems in federated learning\.Foundations and Trends® in Machine Learning14\(1–2\),pp\. 1–210\.Cited by:[§1](https://arxiv.org/html/2608.14654#S1.p2.1)\.
- \[7\]A\. Krizhevsky\(2009\)Learning multiple layers of features from tiny images\.Technical report\.Cited by:[2nd item](https://arxiv.org/html/2608.14654#S5.I1.i2.p1.1)\.
- \[8\]T\. Li, A\. K\. Sahu, M\. Zaheer, M\. Sanjabi, A\. Talwalkar, and V\. Smith\(2020\)Federated optimization in heterogeneous networks\.Proceedings of Machine learning and systems2,pp\. 429–450\.Cited by:[§1](https://arxiv.org/html/2608.14654#S1.p7.1),[§2](https://arxiv.org/html/2608.14654#S2.p1.1),[§3\.1](https://arxiv.org/html/2608.14654#S3.SS1.p5.3)\.
- \[9\]Y\. Liu, L\. Zhang, N\. Ge, and G\. Li\(2020\)A systematic literature review on federated learning: from a model quality perspective\.arXiv preprint arXiv:2012\.01973\.Cited by:[§1](https://arxiv.org/html/2608.14654#S1.p3.1),[§1](https://arxiv.org/html/2608.14654#S1.p7.1)\.
- \[10\]P\. M\. Mammen\(2021\)Federated learning: opportunities and challenges\.arXiv preprint arXiv:2101\.05428\.Cited by:[§1](https://arxiv.org/html/2608.14654#S1.p1.1)\.
- \[11\]B\. McMahan, E\. Moore, D\. Ramage, S\. Hampson, and B\. A\. y Arcas\(2017\)Communication\-efficient learning of deep networks from decentralized data\.InArtificial intelligence and statistics,pp\. 1273–1282\.Cited by:[§1](https://arxiv.org/html/2608.14654#S1.p3.1),[§1](https://arxiv.org/html/2608.14654#S1.p7.1),[§2](https://arxiv.org/html/2608.14654#S2.p1.1),[§3\.1](https://arxiv.org/html/2608.14654#S3.SS1.p4.4)\.
- \[12\]S\. Reddi, Z\. Charles, M\. Zaheer, Z\. Garrett, K\. Rush, J\. Konečnỳ, S\. Kumar, and H\. B\. McMahan\(2020\)Adaptive federated optimization\.arXiv preprint arXiv:2003\.00295\.Cited by:[§5\.1\.1](https://arxiv.org/html/2608.14654#S5.SS1.SSS1.p3.6)\.
- \[13\]S\. Vahidian, M\. Morafah, and B\. Lin\(2021\)Personalized federated learning by structured and unstructured pruning under data heterogeneity\.In2021 IEEE 41st international conference on distributed computing systems workshops \(ICDCSW\),pp\. 27–34\.Cited by:[§2](https://arxiv.org/html/2608.14654#S2.p5.1)\.
- \[14\]H\. Wu and P\. Wang\(2021\)Fast\-convergent federated learning with adaptive weighting\.IEEE Transactions on Cognitive Communications and Networking7\(4\),pp\. 1078–1088\.Cited by:[§1](https://arxiv.org/html/2608.14654#S1.p3.1),[§2](https://arxiv.org/html/2608.14654#S2.p6.1),[§3\.1](https://arxiv.org/html/2608.14654#S3.SS1.p6.1),[§3\.2](https://arxiv.org/html/2608.14654#S3.SS2.p2.1)\.
- \[15\]Y\. Yeganeh, A\. Farshad, N\. Navab, and S\. Albarqouni\(2020\)Inverse distance aggregation for federated learning with non\-iid data\.InDomain Adaptation and Representation Transfer, and Distributed and Collaborative Learning: Second MICCAI Workshop, DART 2020, and First MICCAI Workshop, DCL 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4–8, 2020, Proceedings 2,pp\. 150–159\.Cited by:[§2](https://arxiv.org/html/2608.14654#S2.p3.1)\.
- \[16\]H\. Zhu, J\. Xu, S\. Liu, and Y\. Jin\(2021\)Federated learning on non\-iid data: a survey\.Neurocomputing465,pp\. 371–390\.Cited by:[§1](https://arxiv.org/html/2608.14654#S1.p3.1)\.