FedFIbOS: Fisher Importance based Optimal Submodelling for Heterogeneous Federated Learning

# FedFIbOS: Fisher Importance based Optimal Submodelling for Heterogeneous Federated Learning
Source: [https://arxiv.org/html/2609.19559](https://arxiv.org/html/2609.19559)
###### Abstract

Heterogeneous federated learning requires clients with diverse computational capacities to collaboratively train a global model, where each client trains a capacity\-constrained submodel\. Existing methods select submodel parameters using heuristic importance measures—most prominently parameter magnitude—without theoretical justification for why these measures support convergence\. We identify a fundamental gap: existing parameter selection criteria lack theoretical grounding in the convergence framework, partial client participation introduces additional estimation effects in the Fisher scores\. We proposeFedFIbOS: Fisher Importance\-based Optimal Submodelling for heterogeneous federated learning, using Fisher Information in a principled criterion derived from minimizing submodel masking error\. We theoretically formulate submodel selection through a Fisher\-weighted quadratic masking surrogate and show that the raw Fisher top\-kkrule implemented by FedFIbOS solves this surrogate under a Fisher\-dominant ranking condition\. The resulting method retains the convergence structure of the underlying masked federated optimization bound\. Fisher scores are efficiently estimated from empirical diagonal Fisher information using squared gradients, enabling stable and adaptive parameter selection without additional optimization overhead\. Experiments on CIFAR\-10, CIFAR\-100, and AGNews under pathological and Dirichlet non\-IID settings show FedFIbOS achieves≈10%\{\\approx\}10\\%higher accuracy than the state of the art, with improvements becoming more pronounced under stronger heterogeneity\.

1School of Computing, University of Otago, New Zealand

2School of Computer Science and Engineering, University of New South Wales, Australia

afzya097@student\.otago\.ac\.nz, jeremiah\.deng@otago\.ac\.nz, haibo\.zhang@unsw\.edu\.au

## Introduction

Federated learning \(FL\) enables collaborative model training across distributed clients without sharing private data\([McMahan et al\. 2017](https://arxiv.org/html/2609.19559#bib.bib24)\)\. A central challenge in practical deployments is*model heterogeneity*: clients span diverse hardware with vastly different compute, memory, and communication budgets\([Fang et al\. 2025](https://arxiv.org/html/2609.19559#bib.bib20);[Pfeiffer et al\. 2023](https://arxiv.org/html/2609.19559#bib.bib21);[Chen et al\. 2023](https://arxiv.org/html/2609.19559#bib.bib6)\)\. Model\-heterogeneous federated learning \(MHFL\) addresses this by assigning each client a submodel derived from the global model and tailored to its resource constraints\. The quality of the extracted submodels can directly affect convergence speed and final accuracy\([Liao et al\. 2023](https://arxiv.org/html/2609.19559#bib.bib8);[Yi et al\. 2024](https://arxiv.org/html/2609.19559#bib.bib7)\)\.

Existing MHFL methods primarily construct client submodels using three strategies: static slicing\([Diao et al\. 2021](https://arxiv.org/html/2609.19559#bib.bib23)\), dynamic rolling\-based extraction\([Alam et al\. 2022](https://arxiv.org/html/2609.19559#bib.bib18);[Liao et al\. 2025](https://arxiv.org/html/2609.19559#bib.bib19)\), and importance\-based selection\([Wu et al\. 2024](https://arxiv.org/html/2609.19559#bib.bib22);[Chen et al\. 2023](https://arxiv.org/html/2609.19559#bib.bib6);[Liao et al\. 2023](https://arxiv.org/html/2609.19559#bib.bib8);[Yi et al\. 2024](https://arxiv.org/html/2609.19559#bib.bib7)\)\. Static and dynamic strategies determine submodels according to predefined architectural rules rather than optimization relevance, which can lead to suboptimal parameter allocation, larger gradient mismatch, and increased client drift under heterogeneous data distributions\. Importance\-based methods address this partially, but existing approaches rank parameters by magnitude, a criterion independent of each client’s local data distribution, thereby providing no theoretical guarantee of minimising the masking approximation error that directly governs convergence\([Wu et al\. 2024](https://arxiv.org/html/2609.19559#bib.bib22)\)\.

To address these limitations, we proposeFedFIbOS\(Fisher\-Importance\-basedOptimalSubmodel Extraction for Heterogeneous Federated Learning\), which constructs client\-specific submodels by selecting parameters based on their importance estimated using Fisher information\. Instead of selecting parameters based on their position within the network, FedFIbOS retains the most optimization\-relevant parameters for each client while respecting its computational capacity\. As illustrated in Figure\.[1](https://arxiv.org/html/2609.19559#Sx1.F1), the colored connections represent the relative Fisher importance of the selected parameters\. Red edges correspond to the highest\-ranked parameters, which have the greatest estimated impact on the loss and are therefore prioritised for inclusion in the submodel\. Yellow edges indicate parameters of moderate importance, which are included when model capacity is available\. Green edges represent lower\-ranked parameters that are assigned only to high\-capacity clients with sufficient computational resources\. Consequently, low\-capacity clients receive only the most important \(red\) parameters, medium\-capacity clients receive both red and yellow parameters, while high\-capacity clients receive parameters in all three groups\.

#### Main Contributions

Our main contributions are summarized as follows\.

- •Fisher Guided submodel extraction\.We propose a Fisher Importance based Optimal Submodelling method that employs empirical diagonal Fisher information to identify the most informative parameters for submodel extraction in heterogeneous federated learning\.
- •Theoretical foundation for Fisher\-guided selection\.We formulate the Submodel Selection Problem \(SSP\), identify the conditions under which different importance criteria are equivalent, establish the optimality of raw Fisher selection under the SSP formulation under the Fisher\-dominant ranking condition\.
- •Convergence analysis\.We prove that Fisher\-guided selection preserves the standard𝒪⁡\(1/T\)\\mathcal\{O\}\(1/\\sqrt\{T\}\)convergence rate, where T is the number of communication rounds while yielding a no\-larger selection\-dependent contribution to the convergence bound than magnitude\-based selection, with strict improvement when the corresponding masking\-error coefficients differ\.
- •Extensive experimental validation\.Experiments on CIFAR\-10 , CIFAR\-100 , and AGnews under diverse model and data heterogeneity settings demonstrate consistent improvements over state\-of\-the\-art MHFL methods and validate the theoretical analysis\.

![Refer to caption](https://arxiv.org/html/2609.19559v1/FIbASE.png)Figure 1:Overview of Submodel Extraction for model training in FedFIbOS\.

## Prior work and its limitation\.

#### Static submodel extraction\.

Early heterogeneous\-FL methods such as HeteroFL\([Diao et al\. 2021](https://arxiv.org/html/2609.19559#bib.bib23)\)and related approaches\([Horváth et al\. 2021](https://arxiv.org/html/2609.19559#bib.bib12);[Li et al\. 2021](https://arxiv.org/html/2609.19559#bib.bib10);[Li et al\. 2020](https://arxiv.org/html/2609.19559#bib.bib11)\)assign clients subnetworks or masks with predefined or client\-specific structures, reducing computation and communication but limiting adaptation of the extracted submodels to evolving training dynamics\.

#### Dynamic submodel extraction\.

FedRolex\([Alam et al\. 2022](https://arxiv.org/html/2609.19559#bib.bib18)\)rolls the submodel selection channel\-wise across rounds, ensuring equal training opportunity for each channel\. However, on square\-channel tensors the rolling reduces to diagonal cycling, leaving large parameter regions permanently untrained regardless of the number of rounds\. FedBRB\([Liao et al\. 2025](https://arxiv.org/html/2609.19559#bib.bib19)\)addresses this by partitioning the global model into blocks and rolling block\-wise, with weighted broadcast to accelerate information sharing across submodels, achieving full parameter coverage that FedRolex cannot guarantee\. Despite this improvement, both methods remain data\-agnostic: neither asks which parameters matter most for the loss under each client’s local data distribution\. A capacity\-constrained client trains whichever slice the rolling schedule assigns, not the slice most important for convergence\.\([Wen et al\. 2022](https://arxiv.org/html/2609.19559#bib.bib9);[Liao et al\. 2023](https://arxiv.org/html/2609.19559#bib.bib8)\)

#### Importance\-aware submodel extraction\.

FIARSE\([Wu et al\. 2024](https://arxiv.org/html/2609.19559#bib.bib22)\)advances importance\-aware submodel extraction by selecting parameters based on magnitude and establishing the first convergence guarantee for model\-heterogeneous FL under partial participation\. However, its convergence analysis treats the masking error as a bounded black\-box term, without analysing its structure or the optimality of the underlying selection criterion\. FedLAGC\([Hu et al\. 2026](https://arxiv.org/html/2609.19559#bib.bib17)\)addresses a related but distinct limitation by introducing layer\-adaptive importance guided by gradient norms, yet its selection criterion remains a heuristic with no connection to the quantity that governs convergence\. This connection between the selection criterion and the quantity governing convergence has remained unestablished across all existing approaches, leaving submodel selection without a principled theoretical foundation\.

#### Fisher Information for model compression\.

Classical pruning methods use curvature to quantify parameter importance\. Optimal Brain Damage\([LeCun et al\. 1989](https://arxiv.org/html/2609.19559#bib.bib15)\)uses a diagonal Hessian approximation, while Optimal Brain Surgeon\([Hassibi and Stork 1992](https://arxiv.org/html/2609.19559#bib.bib16)\)accounts for parameter interactions through the inverse Hessian\. Under standard regularity conditions, Fisher Information provides a positive\-semidefinite, gradient\-based surrogate for the Hessian, motivating its use for efficient importance estimation\([Molchanov et al\. 2019](https://arxiv.org/html/2609.19559#bib.bib1)\)\. Recent work extends Fisher\-based importance to model sparsification, including FisherLAS\([Sun et al\. 2026](https://arxiv.org/html/2609.19559#bib.bib5)\)for LLMs\. In federated learning, Fed\-HeLLo\([Zhang et al\. 2025](https://arxiv.org/html/2609.19559#bib.bib4)\)uses layer\-wise Fisher information for heterogeneous LoRA allocation, while other works explore Fisher\-guided pruning, parameter selection, and privacy\-aware optimization\([Chen et al\. 2024](https://arxiv.org/html/2609.19559#bib.bib3);[Liu et al\. 2024](https://arxiv.org/html/2609.19559#bib.bib2)\)\. FedFIbOS instead uses Fisher information for submodel selection under non\-IID heterogeneity and partial participation, with formal convergence guarantees\.

## Problem Formulation

Consider a FL system withNNclients, which each local dataset for clientnnis drawn from distributions\{Pn\}n=1N\\\{P\_\{n\}\\\}\_\{n=1\}^\{N\}\. At each roundtt, a subset𝒜⊆\[N\]\\mathcal\{A\}\\subseteq\[N\]ofAAclients is sampled to participate training\. Each clientnnhas a compute capacityγn∈\(0,1\]\\gamma\_\{n\}\\in\(0,1\]and is assigned a submodel containingkn=⌈γn​d⌉k\_\{n\}=\\lceil\\gamma\_\{n\}d\\rceilparameters from the global model𝐱~∈ℝd\\tilde\{\\mathbf\{x\}\}\\in\\mathbb\{R\}^\{d\}\. A binary maskM\(n\)∈\{0,1\}dM^\{\(n\)\}\\in\\\{0,1\\\}^\{d\}, where‖M\(n\)‖0=kn\\\|M^\{\(n\)\}\\\|\_\{0\}=k\_\{n\}, specifies the parameters included in the submodel\. The global learning objective is to minimise

ℱ⁡\(𝐱~\)=1N​∑n=1N𝔼\(𝐱,y\)∼Pn​\[ℒ⁡\(𝐱~,𝐱,y\)\]\.\\mathcal\{F\}\(\\tilde\{\\mathbf\{x\}\}\)=\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\mathbb\{E\}\_\{\(\\mathbf\{x\},y\)\\sim P\_\{n\}\}\[\\mathcal\{L\}\(\\tilde\{\\mathbf\{x\}\};\\mathbf\{x\},y\)\]\.\(1\)
When clientnntrains on𝐱~t⊙M\(n\)\\tilde\{\\mathbf\{x\}\}\_\{t\}\\odot M^\{\(n\)\}rather than the full model, local gradient computation incurs a*masking approximation error*:

et\(n\)≜‖∇Fn​\(𝐱~t\)⊙M\(n\)−∇𝐱~tFn​\(𝐱~t⊙M\(n\)\)‖22\.e\_\{t\}^\{\(n\)\}\\;\\triangleq\\;\\bigl\\\|\\nabla F\_\{n\}\(\\tilde\{\\mathbf\{x\}\}\_\{t\}\)\\odot M^\{\(n\)\}\-\\nabla\_\{\\tilde\{\\mathbf\{x\}\}\_\{t\}\}F\_\{n\}\(\\tilde\{\\mathbf\{x\}\}\_\{t\}\\odot M^\{\(n\)\}\)\\bigr\\\|\_\{2\}^\{2\}\.Following\([Wu et al\. 2024](https://arxiv.org/html/2609.19559#bib.bib22)\), we bound this error as:

et\(n\)≤δt2​‖𝐱~t‖22,e\_\{t\}^\{\(n\)\}\\;\\leq\\;\\delta\_\{t\}^\{2\}\\,\\\|\\tilde\{\\mathbf\{x\}\}\_\{t\}\\\|\_\{2\}^\{2\},\(2\)whereδt2≥0\\delta\_\{t\}^\{2\}\\geq 0is the smallest constant for which Inequality \([2](https://arxiv.org/html/2609.19559#Sx3.E2)\) holds\. The parameter subsetM\(n\)M^\{\(n\)\}directly determineset\(n\)e\_\{t\}^\{\(n\)\}\. Fisher\-guided selection includes the parameters whose exclusion would induce the largest gradient discrepancy, thereby minimisinget\(n\)e\_\{t\}^\{\(n\)\}and tightening the convergence bound\. The selection criterion determinesδt2\\delta\_\{t\}^\{2\}: a smallerδt2\\delta\_\{t\}^\{2\}means the bound is tighter and local optimisation is more faithful to the full model\.

This error directly governs convergence\. The bound for masked heterogeneous FL\([Wu et al\. 2024](https://arxiv.org/html/2609.19559#bib.bib22)\)takes the form:

mint∈\[T\]⁡‖∇F​\(𝐱~t\)‖22≤C1​\(T\)⏟→0\+C2⏟heterogeneity\+32​NT​∑tδt2​‖𝐱~t‖22⏟masking error \(Term 3\),\\min\_\{t\\in\[T\]\}\\\|\\nabla F\(\\tilde\{\\mathbf\{x\}\}\_\{t\}\)\\\|\_\{2\}^\{2\}\\leq\\underbrace\{C\_\{1\}\(T\)\}\_\{\\to\\,0\}\+\\underbrace\{C\_\{2\}\}\_\{\\text\{heterogeneity\}\}\+\\underbrace\{\\frac\{32N\}\{T\}\\sum\_\{t\}\\delta\_\{t\}^\{2\}\\\|\\tilde\{\\mathbf\{x\}\}\_\{t\}\\\|\_\{2\}^\{2\}\}\_\{\\text\{masking error \(Term 3\)\}\},\(3\)whereC1​\(T\)→0C\_\{1\}\(T\)\\to 0andC2C\_\{2\}depends only on data heterogeneity\. Term 3 is the only term governed by the choice ofSS: minimisingδt2\\delta\_\{t\}^\{2\}directly tightens the convergence neighbourhood\.

To obtain a tractable selection criterion, letΔt​\(S\)=x~t⊙MS−x~t\\Delta\_\{t\}\(S\)=\\tilde\{x\}\_\{t\}\\odot M\_\{S\}\-\\tilde\{x\}\_\{t\}\. A first\-order expansion of the local gradient gives

∇Fn​\(x~t⊙MS\)≈∇Fn​\(x~t\)\+Hn​\(x~t\)​Δt​\(S\)\.\\nabla F\_\{n\}\(\\tilde\{x\}\_\{t\}\\odot M\_\{S\}\)\\approx\\nabla F\_\{n\}\(\\tilde\{x\}\_\{t\}\)\+H\_\{n\}\(\\tilde\{x\}\_\{t\}\)\\Delta\_\{t\}\(S\)\.Near a stationary point, the residual term∇Fn​\(x~t\)⊙\(1−MS\)\\nabla F\_\{n\}\(\\tilde\{x\}\_\{t\}\)\\odot\(1\-M\_\{S\}\)is negligible, so the selection\-dependent masking discrepancy is locally approximated by

‖Hn​\(x~t\)​Δt​\(S\)‖22\.\\\|H\_\{n\}\(\\tilde\{x\}\_\{t\}\)\\Delta\_\{t\}\(S\)\\\|\_\{2\}^\{2\}\.Under a diagonal\-Hessian approximation and the local Fisher–Hessian correspondence, this yields the quadratic surrogate

ϵt\(n\)​\(S\)≜∑j∉S\(Ft,j\(n\)\)2​x~t,j2\.\\epsilon\_\{t\}^\{\(n\)\}\(S\)\\triangleq\\sum\_\{j\\notin S\}\\left\(F^\{\(n\)\}\_\{t,j\}\\right\)^\{2\}\\tilde\{x\}\_\{t,j\}^\{2\}\.\(4\)
This surrogate provides a tractable approximation to the selection\-dependent masking error\.

The resulting Submodel Selection Problem \(SSP\) is:

S⋆\(n,t\)=arg⁡minS⊆\[d\],\|S\|=kn​ϵt\(n\)​\(S\)\.S^\{\\star\(n,t\)\}=\\arg\\min\_\{S\\subseteq\[d\],\\,\|S\|=k\_\{n\}\}\\epsilon\_\{t\}^\{\(n\)\}\(S\)\.\(5\)Its exact solution is Fisher\-weighted top\-kkselection:

S⋆\(n,t\)=TopKkn⁡\(\(Ft,j\(n\)\)2​x~t,j2\)\.S^\{\\star\(n,t\)\}=\\operatorname\{TopK\}\_\{k\_\{n\}\}\\left\(\\left\(F\_\{t,j\}^\{\(n\)\}\\right\)^\{2\}\\tilde\{x\}\_\{t,j\}^\{2\}\\right\)\.\(6\)Existing magnitude\-based methods\([Wu et al\. 2024](https://arxiv.org/html/2609.19559#bib.bib22)\)instead minimise∑j∉Sx~t,j2\\sum\_\{j\\notin S\}\\tilde\{x\}\_\{t,j\}^\{2\}, which omits the Fisher weights\(Fj\(n\)\)2\(F\_\{j\}^\{\(n\)\}\)^\{2\}and provides no guarantee of solving \([5](https://arxiv.org/html/2609.19559#Sx3.E5)\)\. This surrogate is particularly unreliable under non\-IID data:\|𝐱~j\|\|\\tilde\{\\mathbf\{x\}\}\_\{j\}\|is a property of the shared global model identical across all clients, whereasFj\(n\)F\_\{j\}^\{\(n\)\}captures the sensitivity ofFnF\_\{n\}to parameterjjunder clientnn’s local distributionPnP\_\{n\}\. SinceFt,j\(n\)F\_\{t,j\}^\{\(n\)\}contributes directly to the Fisher\-weighted importance score in \(4\), it provides the basis for the raw\-Fisher criterion used by FedFIbOS; under Assumption 4, this criterion induces the same ordering as the exact SSP objective\.

We further useCVF\\mathrm\{CV\}\_\{F\}to quantify heterogeneity in Fisher importance scores and empirically examine its relationship with selection divergence, whereCVF=std⁡\(Fj\(n\)\)/mean⁡\(Fj\(n\)\)\\mathrm\{CV\}\_\{F\}=\\mathrm\{std\}\(F\_\{j\}^\{\(n\)\}\)/\\mathrm\{mean\}\(F\_\{j\}^\{\(n\)\}\)\.

## FedFIbOS

We propose FedFIbOS, a principled importance\-aware submodel extraction framework for model\-heterogeneous federated learning\. Although it retains the optimization framework of FIARSE, what makes the difference is that FedFIbOS measures parameter importance using Fisher Information, which directly quantifies the sensitivity of the local objective to parameter perturbations\.

#### Fisher Information\.

The diagonal Fisher Information score for parameterjjat clientnn:

Fj\(n\)=𝔼\(x,y\)∼Pn​\[\(∂log⁡p⁡\(y\|x,θ\)∂θj\)2\]=𝔼Pn​\[\(∂ℒ∂θj\)2\]\.F\_\{j\}^\{\(n\)\}=\\mathbb\{E\}\_\{\(x,y\)\\sim P\_\{n\}\}\\\!\\\!\\left\[\\left\(\\frac\{\\partial\\log p\(y\|x,\\theta\)\}\{\\partial\\theta\_\{j\}\}\\right\)^\{\\\!2\}\\right\]=\\mathbb\{E\}\_\{P\_\{n\}\}\\\!\\left\[\\left\(\\frac\{\\partial\\mathcal\{L\}\}\{\\partial\\theta\_\{j\}\}\\right\)^\{\\\!2\}\\right\]\.\(7\)We useFt,j\(n\)F\_\{t,j\}^\{\(n\)\}to denote the Fisher score evaluated at the round\-ttmodelx~t\\tilde\{x\}\_\{t\}; when the round index is unambiguous, we writeFj\(n\)F\_\{j\}^\{\(n\)\}for brevity\.

f^j\(n\)=1K​B​∑k=0K−1∑b=1B\(∂ℒ⁡\(xt,k\(n\),ξb\)∂θj\)2\.\\hat\{f\}\_\{j\}^\{\(n\)\}=\\frac\{1\}\{KB\}\\sum\_\{k=0\}^\{K\-1\}\\sum\_\{b=1\}^\{B\}\\left\(\\frac\{\\partial\\mathcal\{L\}\(x\_\{t,k\}^\{\(n\)\},\\xi\_\{b\}\)\}\{\\partial\\theta\_\{j\}\}\\right\)^\{\\\!2\}\.\(8\)The empirical diagonal Fisher Information is computed from the squared gradients already available during local optimization, thereby avoiding any additional gradient evaluations\. To reduce the variance of the estimated importance scores, the server maintains an exponential moving average \(EMA\) of the aggregated Fisher estimates over communication rounds\. The server maintains an EMA across rounds:

f¯j\(n,t\+1\)=α​f¯j\(n,t\)\+\(1−α\)​f^j\(n\)\.\\bar\{f\}\_\{j\}^\{\(n,t\+1\)\}=\\alpha\\,\\bar\{f\}\_\{j\}^\{\(n,t\)\}\+\(1\{\-\}\\alpha\)\\,\\hat\{f\}\_\{j\}^\{\(n\)\}\.\(9\)

### Fisher Importance based Submodel Extraction

The quadratic SSP is minimized by Fisher\-weighted scoresIt,j\(n\)=\(Ft,j\(n\)\)2​x~t,j2I\_\{t,j\}^\{\(n\)\}=\(F\_\{t,j\}^\{\(n\)\}\)^\{2\}\\tilde\{x\}\_\{t,j\}^\{2\}\. FedFIbOS instead implements raw Fisher top\-kkselection using the maintained score

MF\(n\)​\(x~\)=\{j:f¯j\(n,t\)∈TopK⌈γn​d⌉⁡\(f¯\(n,t\)\)\}\.M\_\{F\}^\{\(n\)\}\(\\tilde\{x\}\)=\\left\\\{j:\\bar\{f\}^\{\(n,t\)\}\_\{j\}\\in\\operatorname\{TopK\}\_\{\\lceil\\gamma\_\{n\}d\\rceil\}\(\\bar\{f\}^\{\(n,t\)\}\)\\right\\\}\.\(10\)
For each participating client, FedFIbOS constructs a client\-specific submodel using the maintained Fisher importance estimates\. Given the target model ratioγn\\gamma\_\{n\}, the server identifies the⌈γn​d⌉\\lceil\\gamma\_\{n\}d\\rceilmost important parameters and generates a binary mask that determines the subset of the global model transmitted for local training\. Under Assumption 4, introduced in the theoretical analysis, the Fisher and Fisher\-weighted rankings coincide; hence this implemented raw\-Fisher rule solves the quadratic SSP under that condition\.

Algorithm 1FedFIbOS:Fisher\-Importance\-based Optimal Submodel Extraction1:Input:

x~0∈ℝd\\tilde\{x\}\_\{0\}\\in\\mathbb\{R\}^\{d\},

\{γn\}n=1N\\\{\\gamma\_\{n\}\\\}\_\{n=1\}^\{N\}, rounds

TT, local steps

KK, batch size

BB, rates

ηl,ηs\\eta\_\{l\},\\eta\_\{s\}, EMA rate

α∈\(0,1\)\\alpha\\in\(0,1\)
2:Server Initialization:

f¯\(n,0\)←\|𝐱~0\|\\bar\{f\}^\{\(n,0\)\}\\leftarrow\|\\tilde\{\\mathbf\{x\}\}\_\{0\}\|,

∀n∈\[N\]\\;\\forall n\\in\[N\]
3:for

t=0,1,…,T−1t=0,1,\\ldots,T\{\-\}1do

4:Sample

𝒜⊆\[N\]\\mathcal\{A\}\\subseteq\[N\],

\|𝒜\|=A\|\\mathcal\{A\}\|\{=\}Auniformly without replacement

5:foreach

n∈𝒜n\\in\\mathcal\{A\}do

6:

MF\(n\)←\{j∈\[d\]:f¯j\(n,t\)∈TopKγn​\(f¯\(n,t\)\)\}M\_\{F\}^\{\(n\)\}\\leftarrow\\bigl\\\{j\\in\[d\]:\\bar\{f\}\_\{j\}^\{\(n,t\)\}\\in\\mathrm\{TopK\}\_\{\\gamma\_\{n\}\}\\\!\(\\bar\{f\}^\{\(n,t\)\}\)\\bigr\\\},

7:Send

𝐱~t⊙MF\(n\)\\tilde\{\\mathbf\{x\}\}\_\{t\}\\odot M\_\{F\}^\{\(n\)\}to client

nn
8:endfor

9:foreach

n∈𝒜n\\in\\mathcal\{A\}in paralleldo

10:

𝐱t,0\(n\)←𝐱~t⊙MF\(n\)\\mathbf\{x\}\_\{t,0\}^\{\(n\)\}\\leftarrow\\tilde\{\\mathbf\{x\}\}\_\{t\}\\odot M\_\{F\}^\{\(n\)\}
11:for

k=0,…,K−1k=0,\\ldots,K\{\-\}1do

12:

𝐠t,k\+1\(n\)←∇𝐱t,k\(n\)Fn​\(𝐱t,k\(n\)⊙MF\(n\)​\(𝐱t,k\(n\)\)\)\\mathbf\{g\}\_\{t,k\+1\}^\{\(n\)\}\\leftarrow\\nabla\_\{\\mathbf\{x\}\_\{t,k\}^\{\(n\)\}\}F\_\{n\}\\\!\\left\(\\mathbf\{x\}\_\{t,k\}^\{\(n\)\}\\odot M\_\{F\}^\{\(n\)\}\\\!\\left\(\\mathbf\{x\}\_\{t,k\}^\{\(n\)\}\\right\)\\right\)
13:

𝐱t,k\+1\(n\)←𝐱t,k\(n\)−ηl⋅𝐠t,k\+1\(n\)\\mathbf\{x\}\_\{t,k\+1\}^\{\(n\)\}\\leftarrow\\mathbf\{x\}\_\{t,k\}^\{\(n\)\}\-\\eta\_\{l\}\\cdot\\mathbf\{g\}\_\{t,k\+1\}^\{\(n\)\}
14:

f^j\(n\)\+=1K​B​\(𝐠t,k\+1,j\(n\)\)2\\hat\{f\}\_\{j\}^\{\(n\)\}\\mathrel\{\+\}=\\frac\{1\}\{KB\}\\left\(\\mathbf\{g\}\_\{t,k\+1,j\}^\{\(n\)\}\\right\)^\{2\}
15:endfor

16:

Δ​xt\(n\)←𝐱~t−𝐱t,K\(n\)\\Delta x\_\{t\}^\{\(n\)\}\\leftarrow\\tilde\{\\mathbf\{x\}\}\_\{t\}\-\\mathbf\{x\}\_\{t,K\}^\{\(n\)\}
17:Send

Δ​𝐱t\(n\)\\Delta\\mathbf\{x\}\_\{t\}^\{\(n\)\}and

f^\(n\)\\hat\{f\}^\{\(n\)\}to server

18:endfor

19:

𝐱~t\+1←𝐱~t−ηsA​∑n∈𝒜Δ​𝐱t\(n\)\\tilde\{\\mathbf\{x\}\}\_\{t\+1\}\\leftarrow\\tilde\{\\mathbf\{x\}\}\_\{t\}\-\\frac\{\\eta\_\{s\}\}\{A\}\\sum\_\{n\\in\\mathcal\{A\}\}\\Delta\\mathbf\{x\}\_\{t\}^\{\(n\)\}
20:foreach

n∈𝒜n\\in\\mathcal\{A\}do

21:

f¯\(n,t\+1\)←α​f¯\(n,t\)\+\(1−α\)​f^\(n\)\\bar\{f\}^\{\(n,t\+1\)\}\\leftarrow\\alpha\\,\\bar\{f\}^\{\(n,t\)\}\+\(1\{\-\}\\alpha\)\\,\\hat\{f\}^\{\(n\)\}
22:endfor

23:

f¯\(n,t\+1\)←f¯\(n,t\)\\bar\{f\}^\{\(n,t\+1\)\}\\leftarrow\\bar\{f\}^\{\(n,t\)\},

∀n∉𝒜\\quad\\forall n\\notin\\mathcal\{A\}
24:endfor

#### Algorithm

At each round, the server samples a set of participating clients and construct a client specific Fisher\-guided maskMF\(n\)M\_\{F\}^\{\(n\)\}from the EMAf¯\(n,t\)\\bar\{f\}^\{\(n,t\)\}\(Algorithm[1](https://arxiv.org/html/2609.19559#alg1), line 3\-6\)\. The mask retains the⌈γn​d⌉\\lceil\\gamma\_\{n\}d\\rceilparameters with the largest estimated Fisher importance, and the corresponding masked global submodel is transmitted to clientnn\. Each client performsKKlocal SGD steps on its masked submodel while simultaneously accumulating squared gradients to estimate the empirical Fisher information \(line 14\)\. The server aggregates the received updates to form the next global model \(line 19\), updates the EMA Fisher scores of participating clients \(lines 21–23\), and keeps the EMA scores of non\-participating clients unchanged \(line 23\)\. Proposition[2](https://arxiv.org/html/2609.19559#Thmproposition2)characterizes the decay of initialization bias in the EMA Fisher estimates under partial participation\.

## Experiments

#### Datasets and Models\.

We evaluate the proposed FedFIbOS on three benchmark datasets spanning both computer vision and natural language processing tasks: CIFAR\-10, CIFAR\-100\([Krizhevsky et al\. 2009](https://arxiv.org/html/2609.19559#bib.bib14)\), and AGNews\([Zhang et al\. 2015](https://arxiv.org/html/2609.19559#bib.bib13)\)\. For the computer vision datasets, we adopt ResNet\-18 as the backbone model, while for AGNews we fine\-tune a pretrained RoBERTa\-Base model\. This selection enables evaluation across diverse modalities and model architectures\.

#### Data Heterogeneity\.

Experiments are conducted under two widely adopted non\-IID partitioning strategies: pathological partitioning and Dirichlet partitioning\. For pathological partitioning, we consider two levels of heterogeneity\.*Low heterogeneity*assigns 5 and 50 classes per client for CIFAR\-10 and CIFAR\-100, respectively, whereas*high heterogeneity*assigns only 2 and 20 classes per client\. Since AGNews contains only four classes, pathological partitioning is not sufficiently representative; therefore, AGNews is evaluated only under Dirichlet partitioning\. For Dirichlet partitioning, the concentration parameter is set toα=0\.3\\alpha=0\.3for CIFAR\-10 and CIFAR\-100\([Krizhevsky et al\. 2009](https://arxiv.org/html/2609.19559#bib.bib14)\), andα=1\.0\\alpha=1\.0for AGNews\([Zhang et al\. 2015](https://arxiv.org/html/2609.19559#bib.bib13)\)\.

#### Model Heterogeneity\.

Following the model\-heterogeneous federated learning setting, each client is assigned one of five model capacities corresponding to model ratios1\.0,0\.5,0\.25,0\.125,0\.0625\{1\.0,0\.5,0\.25,0\.125,0\.0625\}, where a ratio of1\.01\.0denotes the complete global model and smaller ratios correspond to progressively reduced submodels\. We evaluate two client\-capacity distributions\. The*uniform*distribution assigns 20% of clients to each model size \(a20\-b20\-c20\-d20\-e20\)\. The*non\-uniform*distribution \(a10\-b10\-c30\-d30\-e20\) better reflects practical deployments by assigning 80% of clients to resource\-constrained devices \(model ratios 0\.25–0\.0625\) and the remaining 20% to higher\-capacity devices \(model ratios 0\.5 and 1\.0\)\. For CIFAR\-10 and CIFAR\-100, experiments are conducted with 100 clients and a client participation rate of 10% per round, and training for800800communication rounds\. For AGNews, we use 200 clients with the same participation rate of 10%, and training for300300communication rounds\.

Figure 2:Global test accuracy comparison of FIARSE, FedBRB, HeteroFL, FedRolex, and FedFIbOS on CIFAR\-10\(first row\) and CIFAR\-100\(second row\) under four heterogeneous settings\. Curves show the moving average of global test accuracy over communication rounds\.
#### Structural divergence and resource sensitivity\.

Figure[3](https://arxiv.org/html/2609.19559#Sx5.F3)shows a consistent two\-phase behaviour across all capacity\-constrained model sizes\. In early rounds, Fisher and magnitude selections largely agree because the Fisher EMA is warm\-started from parameter magnitude \(Algorithm[1](https://arxiv.org/html/2609.19559#alg1), line 2\), before gradient information is available\. As local gradient estimates accumulate and the EMA converges toward the true Fisher distribution, the two selected parameter sets diverge rapidly and stabilise at a persistently low Jaccard plateau\. The observed persistent low plateau is an empirical finding; Lemma[1](https://arxiv.org/html/2609.19559#Thmremark1)provides a theoretical condition under which such selection divergence can occur\.

The divergence intensifies under tighter capacity constraints \- Jaccard stabilises at≈0\.42\{\\approx\}0\.42atγ=0\.5\\gamma\{=\}0\.5and falls to≈0\.22\{\\approx\}0\.22atγ=0\.0625\\gamma\{=\}0\.0625\- because a sharper selection boundary amplifies the consequence of choosing the wrong importance criterion\.

We use Fisher Coefficient of Variation \(CVF\\mathrm\{CV\}\_\{F\}\) to measure Fisher score divergence, whereCVF=std⁡\(Fj\(n\)\)/mean⁡\(Fj\(n\)\)\\mathrm\{CV\}\_\{F\}=\\mathrm\{std\}\(F\_\{j\}^\{\(n\)\}\)/\\mathrm\{mean\}\(F\_\{j\}^\{\(n\)\}\)\.CVF=0\\mathrm\{CV\}\_\{F\}=0iff all Fisher scores are equal; otherwiseCVF\>0\\mathrm\{CV\}\_\{F\}\>0\.Figure[4](https://arxiv.org/html/2609.19559#Sx5.F4)shows substantially greater relative dispersion in Fisher scores than in parameter magnitudes after the warm\-up period, withC​VFCV\_\{F\}reaching approximately 17\.5 in the evaluated setting\. This indicates substantial dispersion in Fisher importance and is consistent with the Fisher\-dominant ranking regime assumed in Assumption 4, althoughC​VFCV\_\{F\}alone does not establish the required rank correspondence\.

Figures[2](https://arxiv.org/html/2609.19559#Sx5.F2)and Tables[1](https://arxiv.org/html/2609.19559#Sx5.T1)\-[2](https://arxiv.org/html/2609.19559#Sx5.T2)summarize the experimental evaluation of FedFIbOS across CIFAR\-10 and CIFAR\-100 under diverse model and statistical heterogeneity settings\. The learning curves consistently show that FedFIbOS converges faster during the early communication rounds and attains higher final test accuracy than the competing baselines\. The performance gains become more pronounced under severe model heterogeneity and non\-IID data distributions\. Table[3](https://arxiv.org/html/2609.19559#Sx5.T3)shows that FedFIbOS consistently achieves the highest classification accuracy across both uniform and non\-uniform settings, demonstrating the effectiveness of Fisher\-guided submodel selection under heterogeneous federated learning\. The quantitative results reported in the tables further validate these observations, demonstrating that FedFIbOS achieves consistent improvements over FIARSE, FedBRB, FedRolex, and HeteroFL across all evaluated experimental settings\.

![Refer to caption](https://arxiv.org/html/2609.19559v1/jaccardIndex.png)Figure 3:Average Jaccard similarity between Fisher\-guided and magnitude\-guided parameter selection on CIFAR\-10 \(non\-IID,α=0\.3\\alpha\{=\}0\.3\) for different model sizes\.Figure 4:Fisher score heterogeneityCVF\\mathrm\{CV\}\_\{F\}\(FedFIbOS\) versus magnitude score heterogeneityCVmag\\mathrm\{CV\}\_\{\\mathrm\{mag\}\}on CIFAR\-10 \(non\-IID,α=0\.3\\alpha\{=\}0\.3\)\.Table 1:Comparison of classification accuracy \(%\) achieved by different federated learning algorithms under non\-IID data and non\-uniform model distribution a10\-b10\-c30\-d30\-e20\. The best result is shown inboldand the second\-best result isunderlined\.Table 2:Comparison of classification accuracy \(%\) achieved by different federated learning algorithms under non\-IID data and uniform model distribution a20\-b20\-c20\-d20\-e20\. The best result is shown inboldand the second\-best result isunderlined\.Table 3:Comparison of classification accuracy \(%\) under uniform \(U\) and non\-uniform \(NU\) model distributions\. CIFAR\-10 and CIFAR\-100 use Dirichletα=0\.3\\alpha=0\.3, while AGNews usesα=1\.0\\alpha=1\.0\. The best result is shown inbold\.

## Theoretical Analysis

Fisher top\-k solves SSP under a Fisher\-dominant ranking condition\. We use five working assumptions: Assumptions 1–3 are standard regularity conditions in heterogeneous FL \(Wu et al\. 2024\), Assumption 4 connects the Fisher\-weighted SSP solution to Fisher top\-kk, and Assumption 5 connects the surrogate to the masking\-error coefficient used in the convergence bound\.

###### Assumption 1\.

\(LL\-smoothness\)\. For alln∈\[N\]n\\in\[N\]andw,v∈ℝdw,v\\in\\mathbb\{R\}^\{d\}:‖∇wFn​\(w⊙MF\(n\)\)−∇vFn​\(v⊙MF\(n\)\)‖2≤L​‖w−v‖2\\\|\\nabla\_\{w\}F\_\{n\}\(w\\odot M\_\{F\}^\{\(n\)\}\)\-\\nabla\_\{v\}F\_\{n\}\(v\\odot M\_\{F\}^\{\(n\)\}\)\\\|\_\{2\}\\leq L\\\|w\-v\\\|\_\{2\}\.

###### Assumption 2\.

\(Bounded variance\)\.1N​∑n=1N‖∇Fn​\(w\)−∇F​\(w\)‖22≤σ2\\frac\{1\}\{N\}\\sum\_\{n=1\}^\{N\}\\\|\\nabla F\_\{n\}\(w\)\-\\nabla F\(w\)\\\|\_\{2\}^\{2\}\\leq\\sigma^\{2\}for allww\. This depends only on\{Pn\}\\\{P\_\{n\}\\\}, not the selection criterion\.

###### Assumption 3\.

\(Masking error bound\)\. For each roundtt, there existsδtF,2∈\[0,1\)\\delta\_\{t\}^\{F,2\}\\in\[0,1\)such that:

‖∇x~tFn​\(x~t⊙MF\(n\)\)‖22≤δF,n,t2​‖x~t‖22\.\\left\\\|\\nabla\_\{\\tilde\{x\}\_\{t\}\}F\_\{n\}\\left\(\\tilde\{x\}\_\{t\}\\odot M\_\{F\}^\{\(n\)\}\\right\)\\right\\\|\_\{2\}^\{2\}\\leq\\delta\_\{F,n,t\}^\{2\}\\left\\\|\\tilde\{x\}\_\{t\}\\right\\\|\_\{2\}^\{2\}\.\(11\)
Let

δF,t2:=maxn∈\[N\]⁡δF,n,t2,δM,t2:=maxn∈\[N\]⁡δM,n,t2\.\\delta\_\{F,t\}^\{2\}:=\\max\_\{n\\in\[N\]\}\\delta\_\{F,n,t\}^\{2\},\\qquad\\delta\_\{M,t\}^\{2\}:=\\max\_\{n\\in\[N\]\}\\delta\_\{M,n,t\}^\{2\}\.\(12\)
whereδM,n,t2\\delta\_\{M,n,t\}^\{2\}is the corresponding client\-specific coefficient under magnitude selection\. Thus,δF,t2\\delta\_\{F,t\}^\{2\}andδM,t2\\delta\_\{M,t\}^\{2\}are the global worst\-client masking\-error coefficients used in the convergence bound\.

###### Assumption 4\.

\(Fisher\-dominant ranking\)\. For each clientnn, roundtt, and any pair of parametersj,kj,k,

Ft,j\(n\)\>Ft,k\(n\)⟹It,j\(n\)\>It,k\(n\),It,j\(n\)=\(Ft,j\(n\)\)2​x~t,j2\.F\_\{t,j\}^\{\(n\)\}\>F\_\{t,k\}^\{\(n\)\}\\Longrightarrow I\_\{t,j\}^\{\(n\)\}\>I\_\{t,k\}^\{\(n\)\},\\qquad I\_\{t,j\}^\{\(n\)\}=\(F\_\{t,j\}^\{\(n\)\}\)^\{2\}\\tilde\{x\}\_\{t,j\}^\{2\}\.\(13\)
Thus, the ordering induced by raw Fisher scores agrees with the ordering of the Fisher\-weighted surrogate scores\. This condition is precisely what makes the raw Fisher top\-kkrule used by FedFIbOS coincide with the exact solution of the quadratic SSP\. It is a ranking assumption and is not implied by non\-IID data alone\.

###### Assumption 5\(Monotonicity of masking error\)\.

For each clientnnand roundtt, the masking\-error coefficientδS,n,t2\\delta\_\{S,n,t\}^\{2\}is monotone with respect to the Fisher\-weighted quadratic masking\-error surrogate\. Specifically, for any two feasible selectionsS1S\_\{1\}andS2S\_\{2\},

ϵt\(n\)​\(S1\)≤ϵt\(n\)​\(S2\)⟹δS1,n,t2≤δS2,n,t2\.\\epsilon\_\{t\}^\{\(n\)\}\(S\_\{1\}\)\\leq\\epsilon\_\{t\}^\{\(n\)\}\(S\_\{2\}\)\\quad\\Longrightarrow\\quad\\delta\_\{S\_\{1\},n,t\}^\{2\}\\leq\\delta\_\{S\_\{2\},n,t\}^\{2\}\.\(14\)
This order\-preservation condition is sufficient to transfer the surrogate ordering established by Proposition 1 to the selection\-dependent term in the convergence bound\.

### Magnitude Selection Suboptimal Under Non\-IID

###### Lemma 1\(Conditions for Selection Divergence\)\.

Suppose that at roundt\+1t\+1there exists a pair of parameters\(j,k\)\(j,k\)straddling the top\-knk\_\{n\}selection boundary such thatFt\+1,j\(n\)\>Ft\+1,k\(n\)F\_\{t\+1,j\}^\{\(n\)\}\>F\_\{t\+1,k\}^\{\(n\)\}, withjjselected andkkexcluded by the Fisher ranking, and suppose their magnitudes at the preceding round satisfy\|x~t,j\|≈\|x~t,k\|\|\\tilde\{x\}\_\{t,j\}\|\\approx\|\\tilde\{x\}\_\{t,k\}\|\. If the stochastic update has a nonzero probability of producing

\|x~t\+1,k\|\>\|x~t\+1,j\|,\|\\tilde\{x\}\_\{t\+1,k\}\|\>\|\\tilde\{x\}\_\{t\+1,j\}\|,then

Pr⁡\(SM\(n,t\+1\)≠SF\(n,t\+1\)\)\>0\.\\Pr\\\!\\left\(S\_\{M\}^\{\(n,t\+1\)\}\\neq S\_\{F\}^\{\(n,t\+1\)\}\\right\)\>0\.\(15\)

###### Proof\.

Under non\-IID data, the client\-specific Fisher scores need not be uniform across parameters\. Consider the stated boundary pair\(j,k\)\(j,k\), withFt\+1,j\(n\)\>Ft\+1,k\(n\)F\_\{t\+1,j\}^\{\(n\)\}\>F\_\{t\+1,k\}^\{\(n\)\}and approximately equal magnitudes at roundtt\. After the stochastic local updatex~​t\+1,j=x~​t,j−ηl​gt,j\\tilde\{x\}\{t\+1,j\}=\\tilde\{x\}\{t,j\}\-\\eta\_\{l\}g\_\{t,j\}, and analogously forkk, suppose the update has positive probability of producing\|x~t\+1,k\|\>\|x~t\+1,j\|\|\\tilde\{x\}\{t\+1,k\}\|\>\|\\tilde\{x\}\{t\+1,j\}\|\. On this event, the magnitude ranking placeskkahead ofjj, whereas the Fisher ranking retainsjjahead ofkkwheneverFt\+1,j\(n\)\>Ft\+1,k\(n\)F\_\{t\+1,j\}^\{\(n\)\}\>F\_\{t\+1,k\}^\{\(n\)\}\. Thus the two rankings differ at the boundary and consequently their top\-knk\_\{n\}selections differ\. Since the magnitude\-order reversal occurs with positive probability by assumption, the selection divergence in Equation \([15](https://arxiv.org/html/2609.19559#Sx6.E15)\) follows\. ∎

### Fisher Optimality and Masking Error Reduction

Theorem 1 is stated for the ideal Fisher scoresFt,j\(n\)F\_\{t,j\}^\{\(n\)\}\. In the practical algorithm, these scores are approximated by the client\-wise EMAf¯j\(n,t\)\\bar\{f\}\_\{j\}^\{\(n,t\)\}; the selection rule in Eq\. \(10\) therefore implements an empirical\-Fisher approximation to the ideal raw Fisher top\-kkrule\.

###### Theorem 1\(Optimality of Fisher Selection\)\.

For fixed clientnn, roundtt, and cardinalitykn=⌈γn​d⌉k\_\{n\}=\\lceil\\gamma\_\{n\}d\\rceil, the solution of SSP \(5\) is

S⋆\(n,t\)=TopKkn⁡\(It,j\(n\)\),It,j\(n\)=\(Ft,j\(n\)\)2​x~t,j2\.S^\{\\star\(n,t\)\}=\\operatorname\{TopK\}\_\{k\_\{n\}\}\\left\(I\_\{t,j\}^\{\(n\)\}\\right\),\\qquad I\_\{t,j\}^\{\(n\)\}=\(F\_\{t,j\}^\{\(n\)\}\)^\{2\}\\tilde\{x\}\_\{t,j\}^\{2\}\.\(16\)
Under Assumption 4,

S⋆\(n,t\)=TopKkn⁡\(Ft,j\(n\)\)=SF\(n,t\)\.S^\{\\star\(n,t\)\}=\\operatorname\{TopK\}\_\{k\_\{n\}\}\\left\(F\_\{t,j\}^\{\(n\)\}\\right\)=S\_\{F\}^\{\(n,t\)\}\.Hence, under Assumption 4, the raw Fisher top\-kkrule implemented by FedFIbOS solves the quadratic SSP\. The solution is unique when the relevant importance scores are distinct\.

###### Proof\.

Since

ϵt\(n\)​\(S\)=∑j∉SIt,j\(n\)=∑j=1dIt,j\(n\)−∑j∈SIt,j\(n\),\\epsilon\_\{t\}^\{\(n\)\}\(S\)=\\sum\_\{j\\notin S\}I\_\{t,j\}^\{\(n\)\}=\\sum\_\{j=1\}^\{d\}I\_\{t,j\}^\{\(n\)\}\-\\sum\_\{j\\in S\}I\_\{t,j\}^\{\(n\)\},minimizingϵt\(n\)​\(S\)\\epsilon\_\{t\}^\{\(n\)\}\(S\)over all sets of sizeknk\_\{n\}is equivalent to selecting theknk\_\{n\}largestIt,j\(n\)I\_\{t,j\}^\{\(n\)\}, giving

S⋆\(n,t\)=TopKkn⁡\(It,j\(n\)\)\.S^\{\\star\(n,t\)\}=\\operatorname\{TopK\}\_\{k\_\{n\}\}\(I\_\{t,j\}^\{\(n\)\}\)\.By Assumption 4, the ordering ofIt,j\(n\)I\_\{t,j\}^\{\(n\)\}agrees with that ofFt,j\(n\)F\_\{t,j\}^\{\(n\)\}\. Therefore,

TopKkn⁡\(It,j\(n\)\)=TopKkn⁡\(Ft,j\(n\)\),\\operatorname\{TopK\}\_\{k\_\{n\}\}\(I\_\{t,j\}^\{\(n\)\}\)=\\operatorname\{TopK\}\_\{k\_\{n\}\}\(F\_\{t,j\}^\{\(n\)\}\),which is exactly the raw Fisher top\-kkrule used by FedFIbOS\. Distinct scores imply uniqueness\. ∎

###### Proposition 1\(Quadratic Masking\-Error Improvement\)\.

LetSF\(n,t\)S\_\{F\}^\{\(n,t\)\}andSM\(n,t\)S\_\{M\}^\{\(n,t\)\}denote the raw Fisher and magnitude\-based selections, respectively\. Under Assumption 4,

ϵt\(n\)​\(SF\(n,t\)\)≤ϵt\(n\)​\(SM\(n,t\)\)\.\\epsilon\_\{t\}^\{\(n\)\}\(S\_\{F\}^\{\(n,t\)\}\)\\leq\\epsilon\_\{t\}^\{\(n\)\}\(S\_\{M\}^\{\(n,t\)\}\)\.\(17\)
If the two selected sets differ and the relevant importance scores are distinct, the inequality is strict\.

###### Proof\.

Under the stated coordinate\-activity condition, the EMA for parameterjjreceives one unbiased Fisher observation whenever clientnnparticipates\. Conditioning onRn​\(t\)=rR\_\{n\}\(t\)=rtherefore gives the standardrr\-step EMA recursion, yielding \([21](https://arxiv.org/html/2609.19559#Sx6.E21)\)\. Under uniform participation,Rn​\(t\)∼Binomial⁡\(t,p\)R\_\{n\}\(t\)\\sim\\mathrm\{Binomial\}\(t,p\), which gives \([22](https://arxiv.org/html/2609.19559#Sx6.E22)\)\. ∎

### Convergence Guarantee

The convergence bound for masked heterogeneous FL takes the form

mint∈\[T\]⁡‖∇F​\(x~t\)‖22≤C1​\(T\)\+C2\+32​NT​∑t=0T−1δt2​‖x~t‖22,\\min\_\{t\\in\[T\]\}\\\|\\nabla F\(\\tilde\{x\}\_\{t\}\)\\\|\_\{2\}^\{2\}\\leq C\_\{1\}\(T\)\+C\_\{2\}\+\\frac\{32N\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\delta\_\{t\}^\{2\}\\\|\\tilde\{x\}\_\{t\}\\\|\_\{2\}^\{2\},\(18\)whereC1​\(T\)→0C\_\{1\}\(T\)\\to 0asT→∞T\\to\\inftyandC2C\_\{2\}depends on data heterogeneity \(Wu et al\. 2024\)\. The selection criterion affects the bound through the masking\-error term\.

###### Theorem 2\(\(Convergence under Fisher\-guided selection\)\)\.

Under Assumptions 1–5 , the Fisher\-guided masked federated optimization procedure satisfies

mint∈\[T\]⁡‖∇F​\(x~t\)‖22≤C1​\(T\)\+C2\+32​NT​∑t=0T−1δF,t2​‖x~t‖22\.\\min\_\{t\\in\[T\]\}\\left\\\|\\nabla F\(\\tilde\{x\}\_\{t\}\)\\right\\\|\_\{2\}^\{2\}\\leq C\_\{1\}\(T\)\+C\_\{2\}\+\\frac\{32N\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\delta\_\{F,t\}^\{2\}\\left\\\|\\tilde\{x\}\_\{t\}\\right\\\|\_\{2\}^\{2\}\.\(19\)whereC1​\(T\)→0C\_\{1\}\(T\)\\to 0asT→∞T\\to\\infty\. Moreover, under Assumption 4, the ideal raw Fisher selectionSF\(n,t\)=TopKkn⁡\(Ft\(n\)\)S\_\{F\}^\{\(n,t\)\}=\\operatorname\{TopK\}\_\{k\_\{n\}\}\(F\_\{t\}^\{\(n\)\}\)minimizes the surrogate among all size\-knk\_\{n\}selections\. The implemented selectionTopKkn⁡\(f¯\(n,t\)\)\\operatorname\{TopK\}\_\{k\_\{n\}\}\(\\bar\{f\}^\{\(n,t\)\}\)is an EMA\-based approximation to this ideal rule; Proposition 2 characterizes the decay of its estimation bias under the stated unbiasedness and fixed\-target assumptions\. By Assumption 5, the ordering of the masking\-error surrogate is preserved by the corresponding masking\-error coefficients\.

δF,t2≤δM,t2\.\\delta\_\{F,t\}^\{2\}\\leq\\delta\_\{M,t\}^\{2\}\.\(20\)
Thus, the selection\-dependent contribution of the convergence bound is no larger for FedFIbOS than for magnitude\-based selection, with strict improvement whenever the corresponding masking\-error coefficients differ

###### Proof\.

The stated bound follows from the masked heterogeneous federated optimization analysis under Assumptions 1–3 for the ideal raw\-Fisher selection\. Under Assumption 4, Theorem 1 establishes thatSF\(n,t\)S\_\{F\}^\{\(n,t\)\}minimizesϵt\(n\)​\(S\)\\epsilon\_\{t\}^\{\(n\)\}\(S\)among all size\-knk\_\{n\}selections, and Proposition 1 therefore gives \([17](https://arxiv.org/html/2609.19559#Sx6.E17)\)\. Thus, by \([20](https://arxiv.org/html/2609.19559#Sx6.E20)\), the selection\-dependent term in the convergence bound is no larger for FedFIbOS than for magnitude\-based selection\. ∎

###### Proposition 2\(Fisher EMA under Partial Participation\)\.

LetRn​\(t\)R\_\{n\}\(t\)denote the number of rounds in which clientnnhas participated before roundtt\. If, whenever clientnnparticipates and parameterjjis active in its selected submodel, the resulting Fisher estimate is unbiased, then, for a fixed target Fisher valueFj\(n\)F\_\{j\}^\{\(n\)\},

𝔼⁡\[f¯j\(n,t\)∣Rn​\(t\)=r\]=αr​f¯j\(n,0\)\+\(1−αr\)​Fj\(n\)\.\\mathbb\{E\}\\left\[\\bar\{f\}\_\{j\}^\{\(n,t\)\}\\mid R\_\{n\}\(t\)=r\\right\]=\\alpha^\{r\}\\bar\{f\}\_\{j\}^\{\(n,0\)\}\+\(1\-\\alpha^\{r\}\)F\_\{j\}^\{\(n\)\}\.\(21\)Thus, the initialization bias decays geometrically with the number of participating rounds\. Under uniform participation probabilityp=A/Np=A/N,

𝔼⁡\[αRn​\(t\)\]=\(1−p\+p​α\)t\.\\mathbb\{E\}\\\!\\left\[\\alpha^\{R\_\{n\}\(t\)\}\\right\]=\(1\-p\+p\\alpha\)^\{t\}\.\(22\)

###### Proof\.

The EMA of clientnnis updated only when the client participates\. Conditioning onRn​\(t\)=rR\_\{n\}\(t\)=rtherefore gives the standardrr\-step EMA recursion, yielding[21](https://arxiv.org/html/2609.19559#Sx6.E21)\. Under uniform participation,Rn​\(t\)∼Binomial⁡\(t,p\)R\_\{n\}\(t\)\\sim\\operatorname\{Binomial\}\(t,p\), which gives[22](https://arxiv.org/html/2609.19559#Sx6.E22)\. ∎

## Conclusion

We presented FedFIbOS, a Fisher Information\-guided submodel extraction framework for model\-heterogeneous federated learning\. By leveraging client\-specific Fisher information, FedFIbOS constructs submodels that better preserve locally important parameters under heterogeneous data distributions\. Theoretical analysis characterizes the Fisher\-weighted quadratic submodel\-selection objective and establishes that the raw Fisher top\-kkrule used by FedFIbOS is optimal under the stated Fisher\-dominant ranking condition\. Under the corresponding monotonicity assumption, this yields a no\-larger selection\-dependent term in the convergence bound\. The proposed method incurs no additional computational overhead, as Fisher information is estimated directly from the squared gradients computed during local optimization and maintained using a server\-side exponential moving average\. Extensive experiments on CIFAR\-10, CIFAR\-100, and AGNews demonstrated consistent improvements over state\-of\-the\-art heterogeneous federated learning methods, including FIARSE, FedBRB, FedRolex, and HeteroFL, particularly under data and resource heterogeneity\.

## References

- Alamet al\.\(2022\)S\. Alam, L\. Liu, M\. Yan, and M\. ZhangFedRolex: model\-heterogeneous federated learning with rolling sub\-model extraction\.Advances in neural information processing systems35,pp\. 29677–29690\.Cited by:[Introduction](https://arxiv.org/html/2609.19559#Sx1.p2.1),[Dynamic submodel extraction\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2023\)D\. Chen, L\. Yao, D\. Gao, B\. Ding, and Y\. LiEfficient personalized federated learning via sparse model\-adaptation\.InInternational conference on machine learning,pp\. 5234–5256\.Cited by:[Introduction](https://arxiv.org/html/2609.19559#Sx1.p1.1),[Introduction](https://arxiv.org/html/2609.19559#Sx1.p2.1)\.
- Chenet al\.\(2024\)R\. Chen, X\. Chen, and J\. ZhaoSparsified federated learning with differential privacy for intrusion detection in vanets based on fisher information matrix\.Plos one19\(4\),pp\. e0301897\.Cited by:[Fisher Information for model compression\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px4.p1.1)\.
- Diaoet al\.\(2021\)E\. Diao, J\. Ding, and V\. TarokhHetero\{FL\}: computation and communication efficient federated learning for heterogeneous clients\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TNkPBBYFkXg)Cited by:[Introduction](https://arxiv.org/html/2609.19559#Sx1.p2.1),[Static submodel extraction\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px1.p1.1)\.
- Fanget al\.\(2025\)Z\. Fang, Z\. Lin, Z\. Chen, X\. Chen, Y\. Gao, and Y\. FangAutomated federated pipeline for parameter\-efficient fine\-tuning of large language models\.IEEE Transactions on Mobile Computing25,pp\. 8782 – 8797\.External Links:[Document](https://dx.doi.org/10.1109/TMC.2025.3649881)Cited by:[Introduction](https://arxiv.org/html/2609.19559#Sx1.p1.1)\.
- Hassibi and Stork \(1992\)B\. Hassibi and D\. StorkSecond order derivatives for network pruning: optimal brain surgeon\.Advances in neural information processing systems5\.Cited by:[Fisher Information for model compression\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px4.p1.1)\.
- Horváthet al\.\(2021\)S\. Horváth, S\. Laskaridis, M\. Almeida, I\. Leontiadis, S\. I\. Venieris, and N\. D\. LaneFjORD: fair and accurate federated learning under heterogeneous targets with ordered dropout\.CoRRabs/2102\.13451\.External Links:[Link](https://arxiv.org/abs/2102.13451),2102\.13451Cited by:[Static submodel extraction\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2026\)Q\. Hu, T\. Liao, S\. Wu, L\. Yang, and C\. ChenFedLAGC: towards high performance system\-heterogeneous federated learning via layer\-adaptive submodel extraction and gradient correction\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 21858–21866\.Cited by:[Importance\-aware submodel extraction\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px3.p1.1)\.
- Krizhevskyet al\.\(2009\)A\. Krizhevsky G\. Hintonet al\.Learning multiple layers of features from tiny images\.Cited by:[Datasets and Models\.](https://arxiv.org/html/2609.19559#Sx5.SSx1.SSS0.Px1.p1.1),[Data Heterogeneity\.](https://arxiv.org/html/2609.19559#Sx5.SSx1.SSS0.Px2.p1.1)\.
- LeCunet al\.\(1989\)Y\. LeCun, J\. S\. Denker, and S\. A\. SollaOptimal brain damage\.2,pp\. 598–605\.External Links:[Link](https://proceedings.neurips.cc/paper/1989/hash/6c9882bbac1c7093fe9a0c3d9a4f2b68-Abstract.html)Cited by:[Fisher Information for model compression\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px4.p1.1)\.
- Liet al\.\(2020\)A\. Li, J\. Sun, B\. Wang, L\. Duan, S\. Li, Y\. Chen, and H\. LiLotteryfl: personalized and communication\-efficient federated learning with lottery ticket hypothesis on non\-iid datasets\.arXiv preprint arXiv:2008\.03371\.Cited by:[Static submodel extraction\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2021\)A\. Li, J\. Sun, X\. Zeng, M\. Zhang, H\. Li, and Y\. ChenFedmask: joint computation and communication\-efficient personalized federated learning via heterogeneous masking\.InProceedings of the 19th ACM conference on embedded networked sensor systems,pp\. 42–55\.Cited by:[Static submodel extraction\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px1.p1.1)\.
- Liaoet al\.\(2023\)D\. Liao, X\. Gao, Y\. Zhao, and C\. XuAdaptive channel sparsity for federated learning under system heterogeneity\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 20432–20441\.Cited by:[Introduction](https://arxiv.org/html/2609.19559#Sx1.p1.1),[Introduction](https://arxiv.org/html/2609.19559#Sx1.p2.1),[Dynamic submodel extraction\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px2.p1.1)\.
- Liaoet al\.\(2025\)T\. Liao, Z\. Xu, Q\. Hu, H\. Dai, H\. Huang, Z\. Zheng, and C\. ChenFedBRB: a solution to the small\-to\-large scenario in device\-heterogeneity federated learning\.IEEE Transactions on Mobile Computing25,pp\. 3036 – 3050\.External Links:[Document](https://dx.doi.org/10.1109/TMC.2025.3610985)Cited by:[Introduction](https://arxiv.org/html/2609.19559#Sx1.p2.1),[Dynamic submodel extraction\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2024\)J\. Liu, J\. Ren, R\. Jin, Z\. Zhang, Y\. Zhou, P\. Valduriez, and D\. DouFisher information\-based efficient curriculum federated learning with large language models\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 10497–10523\.Cited by:[Fisher Information for model compression\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px4.p1.1)\.
- McMahanet al\.\(2017\)B\. McMahan, E\. Moore, D\. Ramage, S\. Hampson, and B\. A\. y ArcasCommunication\-efficient learning of deep networks from decentralized data\.InArtificial intelligence and statistics,pp\. 1273–1282\.Cited by:[Introduction](https://arxiv.org/html/2609.19559#Sx1.p1.1)\.
- Molchanovet al\.\(2019\)P\. Molchanov, A\. Mallya, S\. Tyree, I\. Frosio, and J\. KautzImportance estimation for neural network pruning\.In2019 IEEE/CVF conference on computer vision and pattern recognition \(CVPR\),pp\. 11256–11264\.Cited by:[Fisher Information for model compression\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px4.p1.1)\.
- Pfeifferet al\.\(2023\)K\. Pfeiffer, M\. Rapp, R\. Khalili, and J\. HenkelFederated learning for computationally constrained heterogeneous devices: a survey\.ACM Computing Surveys55\(14s\),pp\. 1–27\.Cited by:[Introduction](https://arxiv.org/html/2609.19559#Sx1.p1.1)\.
- Sunet al\.\(2026\)C\. Sun, H\. Yu, and X\. LiFisher\-based layer\-wise adaptive sparsification for efficient pruning of large language models\.IEEE Transactions on Mobile Computing25\(9\),pp\. 14930 – 14942\.External Links:[Document](https://dx.doi.org/10.1109/TMC.2026.3689307)Cited by:[Fisher Information for model compression\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px4.p1.1)\.
- Wenet al\.\(2022\)D\. Wen, K\. Jeon, and K\. HuangFederated dropout—a simple approach for enabling federated learning on resource constrained devices\.IEEE wireless communications letters11\(5\),pp\. 923–927\.Cited by:[Dynamic submodel extraction\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px2.p1.1)\.
- Wuet al\.\(2024\)F\. Wu, X\. Wang, Y\. Wang, T\. Liu, L\. Su, and J\. GaoFIARSE: model\-heterogeneous federated learning via importance\-aware submodel extraction\.Advances in Neural Information Processing Systems37,pp\. 115615–115651\.Cited by:[Introduction](https://arxiv.org/html/2609.19559#Sx1.p2.1),[Importance\-aware submodel extraction\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px3.p1.1),[Problem Formulation](https://arxiv.org/html/2609.19559#Sx3.p2.2),[Problem Formulation](https://arxiv.org/html/2609.19559#Sx3.p3.1),[Problem Formulation](https://arxiv.org/html/2609.19559#Sx3.p6.3),[Remark 4](https://arxiv.org/html/2609.19559#Thmremark4.p1.1.1)\.
- Yiet al\.\(2024\)K\. Yi, N\. Gazagnadou, P\. Richtárik, and L\. LyuFedp3: federated personalized and privacy\-friendly network pruning under model heterogeneity\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 46587–46619\.Cited by:[Introduction](https://arxiv.org/html/2609.19559#Sx1.p1.1),[Introduction](https://arxiv.org/html/2609.19559#Sx1.p2.1)\.
- Zhanget al\.\(2015\)X\. Zhang, J\. Zhao, and Y\. LeCunCharacter\-level convolutional networks for text classification\.pp\. 649–657\.Cited by:[Datasets and Models\.](https://arxiv.org/html/2609.19559#Sx5.SSx1.SSS0.Px1.p1.1),[Data Heterogeneity\.](https://arxiv.org/html/2609.19559#Sx5.SSx1.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025\)Z\. Zhang, P\. Liu, J\. Xu, and R\. HuFed\-hello: efficient federated foundation model fine\-tuning with heterogeneous lora allocation\.IEEE Transactions on Neural Networks and Learning Systems36\(10\),pp\. 17556–17569\.Cited by:[Fisher Information for model compression\.](https://arxiv.org/html/2609.19559#Sx2.SS0.SSS0.Px4.p1.1)\.