Federated Lightweight Fine-Tuning

# Federated Lightweight Fine-Tuning
Source: [https://arxiv.org/html/2607.18343](https://arxiv.org/html/2607.18343)
11institutetext:Cisco Systems Inc\., USA
11email:\{rachanta,wilreed\}@cisco\.com###### Abstract

Federated fine\-tuning is bottlenecked by communication: FedAvg and pseudo\-gradient schemes transmit a payload that scales with the model, and gradient compression shrinks it by only a constant factor\. We take a different lever\. Mapping networks generate a network’s weights from a small trainable latent through a frozen affine projection; because the map is shared and affine, averaging latents is*exactly*averaging the generated weights\. We turn this into a practical low\-bandwidth federated channel with two changes: a*low\-rank, seed\-regenerable*factorisation of the projection \(cutting generator memory from∼\\sim8080GB to∼\\sim1010MB\), and a*delta*formulationθ=θpre\+U​V⊤​z\\theta=\\theta^\{\\mathrm\{pre\}\}\+UV^\{\\top\}zthat learns an additive correction around a shared centrally\-pretrained base — federated fine\-tuning, which is what makes the method work at scale\. A frozen orthogonal classifier head further removes the head from the payload while improving accuracy\. On CIFAR\-100 with ResNet\-18\+GroupNorm, our method \(FLITE, Federated Low\-rank Iterative Training Engine\) communicates1,2801\{,\}280floats \(≈5\\approx 5KB\) per client per round — an8718×8718\\timesreduction — and reaches74\.67%74\.67\\%, within≈0\.5\\approx 0\.5pp of full\-weight FedAvg\. The averaging identity holds to floating\-point precision \(6×10−86\{\\times\}10^\{\-8\}\); the method sits one to two orders of magnitude below PowerSGD and top\-kkon the bandwidth–accuracy Pareto; it matches or exceeds full\-weight FedAvg under strong non\-IID skew\. int4 latents reach648648bytes per round at unchanged accuracy, whereas int4 full\-weight FedAvg collapses to chance\.

## 1Introduction

Federated and distributed training let many workers improve a shared model without pooling their data\[[21](https://arxiv.org/html/2607.18343#bib.bib16),[9](https://arxiv.org/html/2607.18343#bib.bib26)\], but they pay a steep communication cost\. The dominant paradigm exchanges model state directly: FedAvg\[[21](https://arxiv.org/html/2607.18343#bib.bib16)\]transmits the model itself, and pseudo\-gradient schemes such as DiLoCo\[[5](https://arxiv.org/html/2607.18343#bib.bib17)\]transmit a full\-size weight delta acting as an outer\-loop update\. Both have per\-round bandwidthΘ​\(\|W\|\)\\Theta\(\|W\|\), the number of model parameters, so for a modern network the message — tens of megabytes for even a small ResNet — dominates the cost of federation\.

A structural assumption underlies these methods: bandwidth is reduced along the*frequency*axis — communicating less often — while the size of each message is treated as fixed\. Even dedicated gradient\-compression methods \(low\-rank projection\[[29](https://arxiv.org/html/2607.18343#bib.bib48)\], top\-kksparsification\[[1](https://arxiv.org/html/2607.18343#bib.bib49)\], quantization\[[2](https://arxiv.org/html/2607.18343#bib.bib50)\]\) only shrink the fixed message by a constant factor, still transmitting an object of size proportional to\|W\|\|W\|, and lowering synchronisation frequency trades amortised cost for client drift, which is most damaging precisely when data are heterogeneous\. This paper takes the orthogonal*size*axis: we keep communication frequent but make each message a tiny, model\-independent latent, while preserving the exact FedAvg averaging semantics in weight space\.

#### Mapping networks: a different lever\.

A recent and very different idea,*mapping networks*, trains a small*latent*vectorzzthat, through a fixed \(frozen\) projection, generates the weights of a much larger network\. The latent is a compact carrier of model state\. Crucially, when the generator is affine and shared,*averaging latents is exactly averaging the generated weights*— so a federated client could communicate only the small latent while the server aggregation remains ordinary FedAvg\. The idea is attractive but, as proposed, has two blockers: \(i\) the projection matrix is itself huge \(on the order of tens of gigabytes for a ResNet\-18\), and \(ii\) the latent cannot drive a large network when trained from scratch \(it collapses to near\-chance accuracy\)\.

#### Our approach\.

FLITE\(Federated Low\-rank Iterative Training Engine\) turns the mapping\-network latent into a practical low\-bandwidth communication channel by addressing both blockers and re\-targeting the method to where it is strong\. First, we replace the dense projection with a*low\-rank, seed\-regenerable*factorisation, cutting generator memory from≈80\\approx 80GB to≈2\\approx 2GB \(or≈10\\approx 10MB if regenerated from a seed\) and the regeneration compute correspondingly\. Second, rather than asking the latent to encode a whole network, we use it to encode an*additive delta around a shared, centrally\-pretrained frozen base*— federated*fine\-tuning*rather than from\-scratch training\. This keeps the exact averaging identity, sidesteps the from\-scratch failure, and exploits the fact that the correction a well\-trained model needs is empirically low\-dimensional\. Finally, we freeze an*orthogonally initialised*classifier head, which removes it from the communicated payload and, as an ablation shows, improves accuracy\.

#### Results\.

On CIFAR\-100 with ResNet\-18\+GroupNorm, our latent communicates1,2801\{,\}280floats \(≈5\\approx 5KB\) per client per round — an8718×8718\\timesreduction — and reaches74\.67%74\.67\\%, within≈0\.5\\approx 0\.5pp of full\-weight FedAvg and statistically at the centralized baseline\. The dimension traces a controllable bandwidth–accuracy frontier; the method sits one to two orders of magnitude below PowerSGD and top\-kkon the Pareto; it matches or exceeds full\-weight FedAvg under strong non\-IID skew and is stable across client counts\. The frozen orthogonal head adds\+0\.54\+0\.54pp at zero communication\. Being low\-dimensional, the latent is highly quantization\-robust: int4 latents reach648648bytes per round at unchanged accuracy, while int4 full\-weight FedAvg collapses to chance\.

#### Contributions\.

- •Adelta low\-rank mappingθ=θpre\+U​V⊤​z\\theta=\\theta^\{\\mathrm\{pre\}\}\+UV^\{\\top\}zthat preserves the*exact*FedAvg averaging identity \(verified to6×10−86\{\\times\}10^\{\-8\}\) while reducing generator memory by∼\\sim40×40\\times\(further to∼\\sim1010MB via seed regeneration\)\.
- •Alow\-bandwidth federated fine\-tuning protocolthat transmits a∼\\sim55KB latent per round at parity accuracy, with a favourable bandwidth–accuracy Pareto against PowerSGD and top\-kk, and improved robustness under heterogeneity and small client counts\.
- •Thefrozen orthogonal classifieras a free, composable design choice that removes the head from the payload and improves accuracy;int4 quantization\-robustnessof the latent channel at648648bytes per round, where matched\-bit full\-weight FedAvg collapses\.
- •Negative resultsshowing that mapping networks as a stand\-alone training method collapse at ResNet\-18 scale regardless of dimension, rank, or warm start, identifying federated fine\-tuning as the regime in which the idea does pay off \(§[5\.7](https://arxiv.org/html/2607.18343#S5.SS7)\)\.

The remainder of the paper presents the method \(§[3](https://arxiv.org/html/2607.18343#S3)\), the setup \(§[4](https://arxiv.org/html/2607.18343#S4)\), results \(§[5](https://arxiv.org/html/2607.18343#S5)\), and concluding remarks on scope and limitations \(§[6](https://arxiv.org/html/2607.18343#S6)\)\.

## 2Related Work

#### Federated averaging and its variants\.

FedAvg\[[21](https://arxiv.org/html/2607.18343#bib.bib16)\]established the dominant paradigm: clients train locally and the server averages their models\. Subsequent work improves robustness to heterogeneity and client drift — FedProx\[[16](https://arxiv.org/html/2607.18343#bib.bib28)\]adds a proximal term, SCAFFOLD\[[10](https://arxiv.org/html/2607.18343#bib.bib29)\]uses control variates, Matched Averaging\[[30](https://arxiv.org/html/2607.18343#bib.bib31)\]aligns neurons before averaging, and adaptive server optimisation\[[25](https://arxiv.org/html/2607.18343#bib.bib30)\]stabilises aggregation\. All of these communicate an object of sizeΘ​\(\|W\|\)\\Theta\(\|W\|\)each round; they change*how*weights are aggregated, not the*size*of the message\. Our method is complementary: it preserves the FedAvg averaging rule exactly but shrinks the message to a latent\. We further note a setting distinction: FedAvg, FedProx, and DiLoCo are typically run as*from\-scratch*federated training, with the server initialisingθ0\\theta\_\{0\}and the federation itself producing the trained model\. Our scheme operates in the*federated fine\-tuning*regime, in which every client — ours and the FedAvg baseline alike — starts from the same shared pretrained checkpoint\. All bandwidth and accuracy comparisons in §[5](https://arxiv.org/html/2607.18343#S5)use this matched setting, so the contrast is between two fine\-tuning protocols at different bandwidths, not between a fine\-tuning method and a from\-scratch one\.

#### Low\-frequency and pseudo\-gradient methods\.

Local SGD\[[27](https://arxiv.org/html/2607.18343#bib.bib18),[12](https://arxiv.org/html/2607.18343#bib.bib19)\]and DiLoCo\[[5](https://arxiv.org/html/2607.18343#bib.bib17)\]reduce communication by synchronising less often, performing many local steps between merges\. This lowers amortised cost but transmits a full\-size update at each merge and risks client drift, particularly under heterogeneity\. Our latent is small enough that frequent synchronisation is affordable, which we show converts into a robustness advantage at a fixed byte budget \(§[5\.5](https://arxiv.org/html/2607.18343#S5.SS5)\)\.

#### Communication compression\.

A large literature compresses the transmitted gradient/update: low\-rank projection \(PowerSGD\[[29](https://arxiv.org/html/2607.18343#bib.bib48)\]\), magnitude sparsification \(top\-kk\[[1](https://arxiv.org/html/2607.18343#bib.bib49)\], deep gradient compression\[[20](https://arxiv.org/html/2607.18343#bib.bib52)\]\), and quantization \(QSGD\[[2](https://arxiv.org/html/2607.18343#bib.bib50)\], signSGD\[[3](https://arxiv.org/html/2607.18343#bib.bib51)\]\), often with error feedback to preserve convergence\[[11](https://arxiv.org/html/2607.18343#bib.bib53)\]\. These methods reduce the payload by a constant factor but the compressed object is still derived from, and scales with,\|W\|\|W\|\. We compare directly against PowerSGD and top\-kkand show our latent is one to two orders of magnitude cheaper at matched accuracy \(§[5\.3](https://arxiv.org/html/2607.18343#S5.SS3)\); the two approaches are moreover composable \(a latent can itself be quantized\)\.

#### Subspace training, low\-rank adaptation, and mapping networks\.

Training in a low\-dimensional random subspace is known to suffice for many objectives, an observation formalised by intrinsic\-dimension studies\[[14](https://arxiv.org/html/2607.18343#bib.bib56)\]and exploited for parameter\-efficient fine\-tuning by LoRA\[[8](https://arxiv.org/html/2607.18343#bib.bib57)\]\. Mapping networks\[[26](https://arxiv.org/html/2607.18343#bib.bib58)\]take this further, generating a network’s full weights from a low\-dimensional latent through a frozen generator whose core is an affine projection\. We adopt the*affine special case*of the mapping\-network generator and differ in three ways: \(i\) we factor the projection into a low\-rank, seed\-regenerable form \(the original materialises a dense projection\), removing its prohibitive memory and compute; \(ii\) we operate in delta mode on a shared pretrained base, which is what makes the method work at scale \(from\-scratch mapping fails, §[5\.7](https://arxiv.org/html/2607.18343#S5.SS7)\); and \(iii\) we use it as a federated*communication*channel, exploiting the exact averaging identity that the affine map admits\. Unlike LoRA, our factors are frozen and shared so that averaging latents equals averaging weights — the property federated aggregation requires\.

#### Optimization\-centric decentralized FL and function\-space methods\.

A parallel line improves decentralized FL through optimisation, topology, and personalisation while remaining in parameter space\[[32](https://arxiv.org/html/2607.18343#bib.bib6),[13](https://arxiv.org/html/2607.18343#bib.bib7),[28](https://arxiv.org/html/2607.18343#bib.bib8),[18](https://arxiv.org/html/2607.18343#bib.bib9),[4](https://arxiv.org/html/2607.18343#bib.bib12)\]; their messages still scale with\|W\|\|W\|and our latent channel is complementary\. A distinct line collaborates in*function space*, exchanging predictions on a shared probe set \(FedMD\[[15](https://arxiv.org/html/2607.18343#bib.bib2)\], FedDF\[[19](https://arxiv.org/html/2607.18343#bib.bib22)\]\)\. These decouple the message from\|W\|\|W\|and accommodate model heterogeneity, but optimise a different \(distillation\) objective and are not directly comparable to exact parameter\-space averaging\. Our channel is weight averaging, carried out exactly in a low\-dimensional latent\.

#### Frozen classifiers and normalization\.

Fixing the classifier head is known to be largely harmless\[[6](https://arxiv.org/html/2607.18343#bib.bib59)\]and can be beneficial with well\-conditioned geometries such as regular\-polytope/ETF classifiers\[[24](https://arxiv.org/html/2607.18343#bib.bib60)\]motivated by neural collapse\[[23](https://arxiv.org/html/2607.18343#bib.bib62)\]; FedBABU\[[22](https://arxiv.org/html/2607.18343#bib.bib61)\]freezes the head during federation\. We use a frozen orthogonal head as a composable component that removes the classifier from the payload and improves accuracy \(§[5\.6](https://arxiv.org/html/2607.18343#S5.SS6)\)\. BatchNorm statistics are not naturally averageable across heterogeneous clients, motivating GroupNorm\[[31](https://arxiv.org/html/2607.18343#bib.bib54)\]or local BatchNorm\[[17](https://arxiv.org/html/2607.18343#bib.bib63)\]in federated settings\[[7](https://arxiv.org/html/2607.18343#bib.bib55)\]; we adopt GroupNorm and accept absolute accuracy below BatchNorm SOTA\.

## 3Method

We first recall the mapping\-network generator and the property that makes it attractive for federated averaging \(§[3\.1](https://arxiv.org/html/2607.18343#S3.SS1)\)\. We then identify the two obstacles that prevent it from being used directly at the scale of a modern network \(§[3\.2](https://arxiv.org/html/2607.18343#S3.SS2)\), and present our delta low\-rank parameterisation that removes them \(§[3\.3](https://arxiv.org/html/2607.18343#S3.SS3)\)\. Finally we describe an optional seed\-regenerated projection \(§[3\.4](https://arxiv.org/html/2607.18343#S3.SS4)\), the frozen orthogonal classifier \(§[3\.5](https://arxiv.org/html/2607.18343#S3.SS5)\), and the full federated protocol \(§[3\.6](https://arxiv.org/html/2607.18343#S3.SS6)\)\.

### 3\.1Mapping networks and the averaging identity

Mapping networks\[[26](https://arxiv.org/html/2607.18343#bib.bib58)\]generate the weights of a target layer from a small trainable*latent*vectorzzthrough a fixed \(frozen\) generator\. The original proposal augments this map with modulation and a nonlinearity; we study its affine special case, which is the piece that admits an exact averaging identity\. For a layer withPPparameters andz∈ℝdz\\in\\mathbb\{R\}^\{d\},d≪Pd\\ll P,

θ=b\+Wm​z,\\theta\\;=\\;b\\;\+\\;W\_\{m\}\\,z,\(1\)whereb∈ℝPb\\in\\mathbb\{R\}^\{P\}is a fixed base vector andWm∈ℝP×dW\_\{m\}\\in\\mathbb\{R\}^\{P\\times d\}is a fixed projection; onlyzzis trained\. In the original formulationWmW\_\{m\}is a*dense*random matrix, materialised in full\. Because the mapz↦θz\\mapsto\\thetais affine andb,Wmb,W\_\{m\}are*shared*across all clients, averaging latents is identical to averaging the generated weights:

1K​∑k=1K\(b\+Wm​zk\)=b\+Wm​\(1K​∑k=1Kzk\)\.\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\bigl\(b\+W\_\{m\}\\,z\_\{k\}\\bigr\)\\;=\\;b\+W\_\{m\}\\\!\\left\(\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}z\_\{k\}\\right\)\.\(2\)*This is exactly the FedAvg update, carried out in thedd\-dimensional latent space instead of thePP\-dimensional weight space\.*A client therefore needs to transmit onlyzz\(ddfloats\) rather thanθ\\theta\(PPfloats\), and the server aggregation is unchanged\. Dropping the modulation term and any post\-map nonlinearity is thus a deliberate design choice: it preserves Eq\. \([2](https://arxiv.org/html/2607.18343#S3.E2)\) by construction\.

### 3\.2Two obstacles at scale

#### Memory and compute\.

In the original mapping network the projectionWmW\_\{m\}is a dense matrix of shapeP×dP\\times d*per layer*\. For a ResNet\-18 \(P≈11\.2P\\approx 11\.2M aggregated over layers\) even a modestddmakesWmW\_\{m\}enormous: materialising the dense projections requires on the order of8080GB infp32, far exceeding a single accelerator, and every weight regeneration is a denseP×dP\\times dmatrix–vector product\. The latent is small, but the dense generator that expands it is not\.

#### From\-scratch training\.

Whenbbis a random initialisation and the latent must encode the*entire*network, optimisation collapses at ResNet\-18 scale: training the latent from a Kaiming\-initialised base reaches only≈2\.5%\\approx 2\.5\\%test accuracy on CIFAR\-100 \(§[5](https://arxiv.org/html/2607.18343#S5)\)\. The latent does not have the capacity to drive a large network from scratch\.

### 3\.3Delta low\-rank mapping

We address both obstacles with a single change of parameterisation\. Rather than generating each layer’s weights from scratch with a dense projection, we \(i\) generate an additive*delta*on top of a frozen, centrally\-pretrained backbone, and \(ii\) replace the denseWmW\_\{m\}with a*low\-rank*factorisationWm=U​V⊤W\_\{m\}=UV^\{\\top\}of rankr≪dr\\ll d\. The low\-rank factorisation is*not*part of the original mapping network — which materialisesWmW\_\{m\}densely — but is introduced here specifically to cut both the storage of the generator and, by consequence, the cost of every weight\-regeneration matrix–vector product \(§[3\.2](https://arxiv.org/html/2607.18343#S3.SS2)\); it is also what makes the small\-ddregime that drives the per\-round payload down practical\. For each mapped layerll,

θl=θlpre⏟frozen base\+Ul​Vl⊤​zl,Ul∈ℝPl×r,Vl∈ℝdl×r,\\theta\_\{l\}\\;=\\;\\underbrace\{\\theta^\{\\mathrm\{pre\}\}\_\{l\}\}\_\{\\text\{frozen base\}\}\\;\+\\;U\_\{l\}\\,V\_\{l\}^\{\\top\}\\,z\_\{l\},\\qquad U\_\{l\}\\in\\mathbb\{R\}^\{P\_\{l\}\\times r\},\\;V\_\{l\}\\in\\mathbb\{R\}^\{d\_\{l\}\\times r\},\(3\)whereθlpre\\theta^\{\\mathrm\{pre\}\}\_\{l\}is the \(flattened\) pretrained weight of layerll,Ul,VlU\_\{l\},V\_\{l\}are*frozen*factors with orthonormal columns, and only the per\-layer latentzl∈ℝdlz\_\{l\}\\in\\mathbb\{R\}^\{d\_\{l\}\}is trained\. The effective projection isWm\(l\)=Ul​Vl⊤W\_\{m\}^\{\(l\)\}=U\_\{l\}V\_\{l\}^\{\\top\}of rank at mostrr\. We initialise the latent at zero \(zl=0z\_\{l\}=0\), so the network at the start of mapping is*exactly*the pretrained model; the latent only learns refinements\.

#### The averaging identity still holds exactly\.

Equation \([3](https://arxiv.org/html/2607.18343#S3.E3)\) is affine inzlz\_\{l\}with a shared frozen offsetθlpre\\theta^\{\\mathrm\{pre\}\}\_\{l\}and shared frozen factorsUl,VlU\_\{l\},V\_\{l\}\. The same algebra as Eq\. \([2](https://arxiv.org/html/2607.18343#S3.E2)\) applies per layer, and the pretrained offset cancels in the mean:

1K​∑k\(θlpre\+Ul​Vl⊤​zl,k\)=θlpre\+Ul​Vl⊤​\(1K​∑kzl,k\)\.\\frac\{1\}\{K\}\\sum\_\{k\}\\bigl\(\\theta^\{\\mathrm\{pre\}\}\_\{l\}\+U\_\{l\}V\_\{l\}^\{\\top\}z\_\{l,k\}\\bigr\)=\\theta^\{\\mathrm\{pre\}\}\_\{l\}\+U\_\{l\}V\_\{l\}^\{\\top\}\\\!\\Bigl\(\\tfrac\{1\}\{K\}\\textstyle\\sum\_\{k\}z\_\{l,k\}\\Bigr\)\.\(4\)We verify this numerically end\-to\-end: the maximum discrepancy between averaging latents and averaging the generated weights is5\.96×10−85\.96\\times 10^\{\-8\}, i\.e\. floating\-point round\-off\.

#### Memory and compute\.

The factored projection costsr​\(Pl\+dl\)r\\,\(P\_\{l\}\+d\_\{l\}\)parameters per layer instead ofPl​dlP\_\{l\}\\,d\_\{l\}, and the regeneration becomes two small matrix–vector products \(Vl⊤​zlV\_\{l\}^\{\\top\}z\_\{l\}thenUl​\(⋅\)U\_\{l\}\(\\cdot\)\) costing𝒪​\(r​\(Pl\+dl\)\)\\mathcal\{O\}\\bigl\(r\(P\_\{l\}\+d\_\{l\}\)\\bigr\)rather than𝒪​\(Pl​dl\)\\mathcal\{O\}\(P\_\{l\}d\_\{l\}\)flops\. Aggregated over ResNet\-18 atr=32r=32the storage is≈2\\approx 2GB rather than≈80\\approx 80GB — a reduction of more than an order of magnitude that brings the generator onto a single device\. Crucially, the low rank does*not*cost accuracy in delta mode: varyingr∈\{8,32,128,256\}r\\in\\\{8,32,128,256\\\}changes test accuracy by<0\.2<0\.2pp \(§[5](https://arxiv.org/html/2607.18343#S5)\), because the delta a well\-pretrained model needs is itself low\-dimensional\.

#### Why delta mode resolves the from\-scratch failure\.

The latent no longer has to represent the whole network, only the small correction around a strong base\. This both makes optimisation tractable and explains why a tiny latent suffices: withdl=64d\_\{l\}=64,r=32r=32the entire ResNet\-18 is steered by1,2801\{,\}280trainable floats \(20 mapped layers×64\\times\\,64\), an8718×8718\\timesreduction relative to the11\.211\.2M parameters, while recovering the pretrained accuracy \(§[5](https://arxiv.org/html/2607.18343#S5)\)\.

### 3\.4Seed\-regenerated projection \(optional\)

The factorsUl,VlU\_\{l\},V\_\{l\}are deterministic functions of a shared random seed \(Gaussian draws orthonormalised by QR\): a single6464\-bit seed regenerates them identically on every client, preserving Eq\. \([4](https://arxiv.org/html/2607.18343#S3.E4)\) exactly\. This trades recompute for a further memory reduction, from≈2\\approx 2GB of stored factors to≈10\\approx 10MB of peak working memory\.

### 3\.5Frozen orthogonal classifier

We initialise the final classifier \(the fully\-connected head\) with an orthogonal weight matrix and*freeze*it\. This removes the classifier from the set of mapped/communicated parameters entirely, and — as an ablation confirms \(§[5](https://arxiv.org/html/2607.18343#S5)\) — is accuracy\-positive rather than merely neutral: on CIFAR\-100 the frozen orthogonal head reaches73\.53%73\.53\\%versus72\.99%72\.99\\%for a trainable head and72\.95%72\.95\\%for a frozen Kaiming head\. An orthonormal, fixed set of class prototypes gives a well\-conditioned target geometry for the backbone to align to, while contributing zero communication\.

### 3\.6Federated protocol

The complete scheme is a drop\-in replacement for FedAvg in which the per\-round payload is the latent rather than the model\. The server broadcasts the frozen pretrained baseθpre\\theta^\{\\mathrm\{pre\}\}and the seed*once*; thereafter each round transmits onlyzz\(Algorithm[1](https://arxiv.org/html/2607.18343#alg1)\)\. WithKKclients andD=∑ldlD=\\sum\_\{l\}d\_\{l\}latents, one round transfers2​K​D⋅42KD\\cdot 4bytes:≈5\\approx 5KB per client per direction for ResNet\-18 withD=1,280D=1\{,\}280, against≈45\\approx 45MB for the full11\.211\.2M\-parameter model\.

Algorithm 1FLITEfederated round with optional latent quantization\. All clients cacheθpre\\theta^\{\\mathrm\{pre\}\}and factors\{Ul,Vl\}\\\{U\_\{l\},V\_\{l\}\\\}\(or the shared seed that regenerates them, §[3\.4](https://arxiv.org/html/2607.18343#S3.SS4)\); factors live on the mapped weight tensors only\. Non\-mapped parameters \(FC head, biases, normalization\) remain frozen\. The generatorθl​\(zk\)=θlpre\+Ul​Vl⊤​zk,l\\theta\_\{l\}\(z\_\{k\}\)=\\theta^\{\\mathrm\{pre\}\}\_\{l\}\+U\_\{l\}V\_\{l\}^\{\\top\}z\_\{k,l\}is re\-evaluated on every forward/backward pass; initialisez¯←0\\bar\{z\}\\leftarrow 0\. Aggregation is the unweighted mean; sample\-weightedpk∝nkp\_\{k\}\\propto n\_\{k\}preserves the identity too\.1:Serverbroadcasts

z¯\\bar\{z\}to clients

k∈𝒮k\\in\\mathcal\{S\}\.

2:foreach client

k∈𝒮k\\in\\mathcal\{S\}in paralleldo

3:Set

zk←z¯z\_\{k\}\\leftarrow\\bar\{z\}\.

4:Train

zkz\_\{k\}for

EElocal epochs on client data

𝒟k\\mathcal\{D\}\_\{k\}\.

5:*\(Optional\)*Quantize

zkz\_\{k\}to

qq\-bit integers \(uniform\-affine, per\-tensor scale\)\.

6:Upload

zkz\_\{k\}\(

4​D4Dbytes at fp32,

q​D/8qD/8at

qq\-bit;

DDis the total latent size\)\.

7:endfor

8:Servertakes the unweighted mean of the \(dequantized\) uploads\.

## 4Experimental Setup

#### Models and datasets\.

Our primary setting isResNet\-18 with GroupNorm\(≈11\.2\\approx 11\.2M parameters\) onCIFAR\-100\. We additionally evaluate ResNet\-18\+GN onCIFAR\-10and, to test scaling in model and label space,ResNet\-34 with GroupNorm\(≈21\.3\\approx 21\.3M parameters\) onTinyImageNet\(200 classes\)\. To probe a non\-vision modality we additionally run a compactTinyGPT\(66layers,384384\-dim embeddings,≈11\.5\\approx 11\.5M trainable parameters\) onWikiText\-2language modelling\. We use GroupNorm rather than BatchNorm throughout because BatchNorm running statistics are not naturally averageable across clients in federated learning, whereas GroupNorm carries no such buffers\[[31](https://arxiv.org/html/2607.18343#bib.bib54),[7](https://arxiv.org/html/2607.18343#bib.bib55)\]\. We note up front that GroupNorm places the*absolute*accuracies somewhat below BatchNorm SOTA; our claims concern*relative*parity at greatly reduced bandwidth, not absolute state of the art\.

#### Centrally\-pretrained base\.

All methods share a single centrally pretrained backbone, trained with SGD \(momentum0\.90\.9, learning rate0\.050\.05with a33\-epoch warmup and cosine decay, weight decay5×10−45\\times 10^\{\-4\}, batch size128128\) for100100epochs, with the orthogonal classifier head frozen from initialisation \(§[3\.5](https://arxiv.org/html/2607.18343#S3.SS5)\)\. The base reaches94\.4%94\.4\\%on CIFAR\-10,74\.6%74\.6\\%on CIFAR\-100, and62\.5%62\.5\\%on TinyImageNet\. The TinyGPT base reaches validation perplexity66\.166\.1on WikiText\-2\. This base is broadcast once at the start of federation; thereafter only latents \(ours\) or the corresponding per\-method payload \(baselines\) are communicated\.

#### Federated configuration\.

We simulateKKclients \(K∈\{4,8,16,32\}K\\in\\\{4,8,16,32\\\}, defaultK=8K=8\) overRRcommunication rounds withEElocal epochs per round\. Client data is partitioned eitherIID\(uniform random\) ornon\-IIDvia a Dirichlet split with concentrationα∈\{0\.5,0\.1\}\\alpha\\in\\\{0\.5,0\.1\\\}\(smallerα\\alpha= more heterogeneous\)\. Each round, clients start from the global state, train locally, and upload their payload for averaging \(Algorithm[1](https://arxiv.org/html/2607.18343#alg1)\)\. The averaging identity is re\-checked at every run\.

#### Methods compared\.

On a single bandwidth–accuracy axis we compare:

- •Latent \(ours\):delta low\-rank mapping, default latent dimensiond=64d=64per layer and rankr=32r=32on CIFAR\-100 \(D=1,280D=1\{,\}280latents total\), with ad/rd/rsweep for the Pareto curve\. Trained with Adam \(lr10−210^\{\-2\}, weight decay10−410^\{\-4\}, gradient clip1\.01\.0\) on the latents only\.
- •Full\-weight FedAvg\[[21](https://arxiv.org/html/2607.18343#bib.bib16)\]: the standard baseline, communicating all trainable weights\.
- •PowerSGD\[[29](https://arxiv.org/html/2607.18343#bib.bib48)\]: low\-rank gradient compression with rank∈\{2,8\}\\in\\\{2,8\\\}and error feedback\.
- •Top\-kk\[[1](https://arxiv.org/html/2607.18343#bib.bib49)\]: magnitude sparsification of the update at densities∈\{1%,10%\}\\in\\\{1\\%,10\\%\\\}with error feedback\.

Except for the head\-to\-head study in §[5\.2](https://arxiv.org/html/2607.18343#S5.SS2), all baselines share the same frozen pretrained base and frozen orthogonal head, so differences reflect the communication scheme alone; the head\-to\-head study additionally reports FedAvg with a trainable Kaiming head to isolate the head’s contribution\.

#### Communication accounting\.

We reportcumulative bytes per client, counting both upload and broadcast directions \(fp32=44bytes\)\. For top\-kkwe count transmitted values*and*their integer indices; for PowerSGD we count both transmitted factors\. The per\-round latent payload isD⋅4D\\cdot 4bytes \(≈5\\approx 5KB forD=1,280D=1\{,\}280\); the per\-round full\-weight payload is≈45\\approx 45MB\.111We report bytes, not wall\-clock time: the experiments are a faithful federated simulation with exact byte accounting, but we make no latency or system\-throughput claims\.

#### Ablations and protocol\.

The frozen\-classifier ablation \(§[3\.5](https://arxiv.org/html/2607.18343#S3.SS5)\) compares a trainable head, a frozen Kaiming\-initialised head, and our frozen orthogonal head, each over33seeds\. Unless stated otherwise, all federated results are reported over33seeds \(mean±\\pmstandard deviation\)\. Pretraining and mapping both checkpoint per epoch and are resumable, and every run logs the latent\-averaging identity error as a correctness check\.

## 5Results

We organise the results around the three claims of §[1](https://arxiv.org/html/2607.18343#S1): massive bandwidth reduction at parity accuracy \(§[5\.1](https://arxiv.org/html/2607.18343#S5.SS1)–[5\.3](https://arxiv.org/html/2607.18343#S5.SS3)\), robustness under heterogeneity and client count \(§[5\.4](https://arxiv.org/html/2607.18343#S5.SS4)–[5\.5](https://arxiv.org/html/2607.18343#S5.SS5)\), and the supporting design choices \(§[5\.6](https://arxiv.org/html/2607.18343#S5.SS6)–[5\.7](https://arxiv.org/html/2607.18343#S5.SS7)\)\. All federated numbers are over33seeds \(mean±\\pmstd\) unless noted\. Additional plots \(fixed\-byte budget,KK\-sweep, quantization Pareto, WikiText\-2 language modelling\) and the full appendix appear in Appendix[0\.A](https://arxiv.org/html/2607.18343#Pt0.A1)–[0\.G](https://arxiv.org/html/2607.18343#Pt0.A7)\.

### 5\.1Federated fidelity at extreme compression

Table[1](https://arxiv.org/html/2607.18343#S5.T1)reports the headline federated result on CIFAR\-100 \(ResNet\-18\+GN,K=8K=8, IID\)\. Communicating only the latent —1,2801\{,\}280floats \(≈5\\approx 5KB\) per client per round, an8718×8718\\timesreduction relative to the11\.211\.2M\-parameter model — our method reaches74\.67±0\.01%74\.67\\pm 0\.01\\%, within≈0\.5\\approx 0\.5pp of full\-weight FedAvg \(75\.16±0\.15%75\.16\\pm 0\.15\\%\) and statistically indistinguishable from the centrally pretrained ceiling \(74\.6%74\.6\\%\)\. The latent carries essentially all of the federated signal at a per\-round payload four orders of magnitude smaller than the model\.

Table 1:Federated CIFAR\-100 \(ResNet\-18\+GN,K=8K=8, IID,33seeds\)\. Our latent exchange matches full\-weight FedAvg within seed noise at8718×8718\\timesfewer transmitted parameters per round\.
### 5\.2Head\-to\-head: ours vs textbook FedAvg

Table[2](https://arxiv.org/html/2607.18343#S5.T2)and Fig\.[1](https://arxiv.org/html/2607.18343#S5.F1)summarise the central comparison across both datasets and partitions: our full scheme \(frozen orthogonal head\+\+latent\) against textbook FedAvg \(trainable Kaiming head, full\-weight averaging; the only experiment where the FedAvg baseline uses a trainable Kaiming head — elsewhere the head is matched to isolate the communication channel\)\. Under IID data our method is within seed noise of FedAvg \(−0\.25\-0\.25pp on CIFAR\-10,−0\.23\-0\.23pp on CIFAR\-100\) at540540–8700×8700\\timeslower per\-round bandwidth\. Under strong heterogeneity \(α=0\.1\\alpha\{=\}0\.1\) the comparison*reverses*: we are\+0\.39\+0\.39pp ahead on CIFAR\-10 and\+0\.97\+0\.97pp ahead on CIFAR\-100\.

Table 2:Ours \(frozen orthogonal FC head \+ latent communication\) vs\. textbook FedAvg \(trainable Kaiming FC, full\-weight averaging\)\. Best test accuracy \(%\) \(mean over 3 seeds\);Δ=\\Delta=ours−\-FedAvg \(positive favours ours\)\.![Refer to caption](https://arxiv.org/html/2607.18343v1/x1.png)Figure 1:Our scheme \(frozen\-orthogonal head\+\+latent\) vs textbook FedAvg \(trainable head, full\-weight\)\. Parity under IID, a clear advantage under non\-IID \(α=0\.1\\alpha\{=\}0\.1\), at a fraction of the bandwidth\.
### 5\.3Bandwidth–accuracy Pareto

Figure[2](https://arxiv.org/html/2607.18343#S5.F2)plots test accuracy against cumulative communication\. On CIFAR\-100 our latent points occupy the bottom\-left of the plot — the entire trajectory lives below≈2\\approx 2MB cumulative, while full\-weight FedAvg requires10310^\{3\}–10410^\{4\}MB to reach comparable accuracy\. The latent dimensionddtraces a controllable frontier \(Table[3](https://arxiv.org/html/2607.18343#S5.T3)\): fromd=64d=64\(8718×8718\\times,74\.67%74\.67\\%\) up tod=1024d=1024\(545×545\\times,74\.66%74\.66\\%\), accuracy is essentially flat, confirming that the correction a pretrained model needs is genuinely small\. On CIFAR\-10 the same picture holds against the stronger compression baselines: PowerSGD and top\-kkreduce full\-weight cost by4040–140×140\\times, but our latent is a further11–22orders of magnitude cheaper at matched accuracy\.

![Refer to caption](https://arxiv.org/html/2607.18343v1/x2.png)

![Refer to caption](https://arxiv.org/html/2607.18343v1/x3.png)

Figure 2:Bandwidth–accuracy Pareto\. Left: CIFAR\-100, latent \(ours\) vs full\-weight FedAvg\. Right: CIFAR\-10, vs full\-weight, PowerSGD \(r∈\{2,8\}r\\in\\\{2,8\\\}\) and top\-kk\(\{1%,10%\}\\\{1\\%,10\\%\\\}\)\. Our latent occupies the low\-bandwidth frontier\.Table 3:Latent\-dimension sweep, federated CIFAR\-100 \(K=8K=8, IID\)\. Accuracy is flat across two orders of magnitude of per\-round payload\.
### 5\.4Robustness to non\-IID data

Figure[3](https://arxiv.org/html/2607.18343#S5.F3)compares our method with full\-weight FedAvg on CIFAR\-10 across IID and Dirichletα∈\{0\.5,0\.1\}\\alpha\\in\\\{0\.5,0\.1\\\}splits\. The gap to full\-weight, already small under IID \(−0\.2\-0\.2pp\), vanishes atα=0\.5\\alpha=0\.5and*reverses*under the strongest skew: atα=0\.1\\alpha=0\.1our latent reaches94\.07±0\.07%94\.07\\pm 0\.07\\%versus full\-weight93\.79±0\.18%93\.79\\pm 0\.18\\%\(\+0\.28\+0\.28pp\)\. Because the latent moves in a low\-dimensional shared subspace anchored at the pretrained base, client updates are constrained and average more stably than full\-weight updates — exactly the regime where full\-weight FedAvg suffers most from client drift\.

![Refer to caption](https://arxiv.org/html/2607.18343v1/x4.png)Figure 3:Non\-IID robustness on CIFAR\-10\. Our latent matches full\-weight FedAvg atα=0\.5\\alpha=0\.5and exceeds it under strong heterogeneity \(α=0\.1\\alpha=0\.1\), at≈545×\\approx 545\\timeslower per\-round bandwidth\.
### 5\.5Fixed\-byte budget and client\-count sensitivity

Two further axes support the same picture \(details in Appendix[0\.A](https://arxiv.org/html/2607.18343#Pt0.A1)\): under a fixed200200MB per\-client budget on CIFAR\-10 \(α=0\.5\\alpha\{=\}0\.5\), our method completes∼\\sim150150latent rounds and reaches94\.21%94\.21\\%while full\-weight FedAvg can afford one round and attains93\.48%93\.48\\%\(\+0\.73\+0\.73pp for ours\)\. SweepingK∈\{4,8,16,32\}K\\in\\\{4,8,16,32\\\}on CIFAR\-100, our latent accuracy is essentially flat inKK, whereas full\-weight underα=0\.5\\alpha\{=\}0\.5degrades to74\.2%74\.2\\%atK=4K\{=\}4before recovering at largeKK— a further symptom of drift that our low\-dimensional channel avoids\.

### 5\.6Design ablations

#### Frozen orthogonal classifier\.

On CIFAR\-100 \(3 seeds\), freezing a Kaiming\-initialised head is accuracy\-neutral \(72\.95%72\.95\\%vs trainable72\.99%72\.99\\%\), but freezing an*orthogonal*head improves accuracy to73\.53%73\.53\\%\(\+0\.54\+0\.54pp\) while removing it from the payload\. A fixed, well\-conditioned set of class prototypes gives the backbone a clean target geometry to align to at zero communication cost\.

#### Rank irrelevance under delta mode\.

Consistent with §[3\.3](https://arxiv.org/html/2607.18343#S3.SS3), the projection rankrrhas little effect on accuracy in delta mode: varyingr∈\{8,32,128,256\}r\\in\\\{8,32,128,256\\\}atd=256d\{=\}256on CIFAR\-100 moves test accuracy by<0\.2<0\.2pp\. Rank therefore trades only generator memory and compute, not accuracy — justifying the smallr=32r\{=\}32used throughout\.

### 5\.7Negative result: mapping networks do not scale as stand\-alone training

The original mapping\-network proposal targets stand\-alone training: a tiny trainable latent, generated through a frozen random projection, replaces direct weight optimisation\. We find this does not extend beyond small architectures\. At ResNet\-18 scale on CIFAR\-100, training the latent from a randomly initialised base collapses to≈2\.5%\\approx 2\.5\\%test accuracy, against74\.7%74\.7\\%that direct SGD reaches and that our delta\-mode latent recovers\. The collapse is robust: it persists across latent dimensionsd∈\{1024,4096,16384\}d\\in\\\{1024,4096,16384\\\}, projection ranksr∈\{8,32,128,256\}r\\in\\\{8,32,128,256\\\}, and warm\-start variants including PCANet\-initialised bases, short SGD warm\-ups, and least\-squares latent initialisation from a converged teacher\. The latent has ample capacity to encode a*correction*around a strong base, but not to drive a large network from scratch\.

This is the negative result that shapes our framing: rather than competing with direct training, mapping networks are useful precisely as a low\-bandwidth*communication channel*between models that have already been trained\.

### 5\.8Scaling to TinyImageNet

To test whether the trade\-off holds at larger scale, we federate aResNet\-34\+GNbackbone \(≈21\.3\\approx 21\.3M parameters,200200classes\) onTinyImageNetwithd=1024d\{=\}1024\(K=8K\{=\}8,R=20R\{=\}20,22local epochs\)\. Under IID partitioning, the latent channel matches full\-weight FedAvg within0\.20\.2pp \(62\.14±0\.08%62\.14\\pm 0\.08\\%vs62\.34±0\.11%62\.34\\pm 0\.11\\%\) while transmitting≈144\\approx 144KB per client per round vs≈81\\approx 81MB \(≈577×\\approx 577\\timesless\)\. Under non\-IID data \(α=0\.5\\alpha\{=\}0\.5\), the latent path is*more*accurate \(62\.22±0\.10%62\.22\\pm 0\.10\\%vs60\.79±0\.19%60\.79\\pm 0\.19\\%,\+1\.43\+1\.43pp\), echoing the CIFAR head\-to\-head trend at a larger scale \(Table[4](https://arxiv.org/html/2607.18343#S5.T4)\)\.

Table 4:TinyImageNet \(ResNet\-34\+GN,d=1024d\{=\}1024,K=8K\{=\}8,R=20R\{=\}20\): latent vs full\-weight FedAvg\. Best test accuracy \(%\) \(mean over 3 seeds\);Δ=\\Delta=latent−\-full\-weight\. Per\-round payload: latent 144 KB vs full\-weight 81\.2 MB\.
### 5\.9Integer\-quantized latents

The latent is a short vector, so it composes directly with standard payload compression\. We apply uniform\-affine integer quantization to the uploaded latent and compare against quantizing the full\-weight update at*matched*bit\-widths \(Table[5](https://arxiv.org/html/2607.18343#S5.T5); see Fig\.[7](https://arxiv.org/html/2607.18343#Pt0.A1.F7)in the appendix for the accuracy\-vs\-precision plot\)\. On CIFAR\-100, quantizing our latent to int8 or*int4*leaves accuracy essentially unchanged \(74\.73±0\.02%74\.73\\pm 0\.02\\%at int4 vs74\.72±0\.03%74\.72\\pm 0\.03\\%at fp32,33seeds\), shrinking the payload to648648bytes per client per round — a68,943×68\{,\}943\\timesreduction relative to the fp32 full\-weight model\. By contrast, textbook full\-weight FedAvg tolerates int8 but*collapses*at int4 \(to1\.11±0\.02%1\.11\\pm 0\.02\\%, i\.e\. chance\), because44\-bit quantization of11\.211\.2M weights injects far more error than averaging can absorb\. Quantization and our latent channel are complementary and together push the per\-round payload below11KB at full accuracy\.

Table 5:CIFAR\-100 \(d=64d\{=\}64\) matched\-bit\-width communication ladder\. Payload is per client per round; best test accuracy \(%\) \(mean over 3 seeds\)\. Our latent scheme is unaffected by 8/4\-bit quantization, while textbook full\-weight FedAvg collapses at int4 \(†\\dagger\)\.

## 6Conclusion

#### Why it helps under heterogeneity\.

Latent updates live in a low\-dimensional subspace anchored at the shared base, so client updates are implicitly constrained and average more stably than full\-weight updates\. The gap to full\-weight FedAvg thus closes and then reverses as data become more non\-IID \(§[5\.4](https://arxiv.org/html/2607.18343#S5.SS4)\); because each round is cheap, the method can also synchronise often within a fixed byte budget, directly attacking drift \(§[5\.5](https://arxiv.org/html/2607.18343#S5.SS5)\)\. The empirical flatness of accuracy in bothdd\(Table[3](https://arxiv.org/html/2607.18343#S5.T3)\) andrr\(§[5\.6](https://arxiv.org/html/2607.18343#S5.SS6)\) is consistent with adapting a well\-pretrained network being intrinsically low\-dimensional\[[14](https://arxiv.org/html/2607.18343#bib.bib56)\]: the latent encodes only a correction around a strong base — which is also why the from\-scratch variant fails \(§[5\.7](https://arxiv.org/html/2607.18343#S5.SS7)\)\.

#### Composability\.

The frozen orthogonal head, seed regeneration, and latent are modular: any can be dropped without breaking the others, and the latent itself composes with standard payload compression — integer quantization pushes the payload to648648B/round at parity, where matched\-bit full\-weight FedAvg collapses \(§[5\.9](https://arxiv.org/html/2607.18343#S5.SS9)\)\. The shared centrally\-pretrained base fits current federated practice, in that many practical deployments adapt off public checkpoints rather than train from scratch\.

#### Limitations\.

\(i\) GroupNorm places*absolute*accuracy below BatchNorm SOTA; our claims are relative parity at massive compression, not absolute state of the art\. \(ii\) The method is federated*fine\-tuning*: it requires a shared centrally\-pretrained base and a one\-time broadcast of that base and the seed\. \(iii\) Our evaluation is a federated simulation with exact byte accounting; we make no wall\-clock claims\. A preliminary autoregressive language\-modelling extension is in Appendix[0\.B](https://arxiv.org/html/2607.18343#Pt0.A2)\.

#### Code\.

The code will be released publicly upon acceptance of the paper\.

## References

- \[1\]A\. F\. Aji and K\. Heafield\(2017\)Sparse communication for distributed gradient descent\.InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 440–445\.Cited by:[§1](https://arxiv.org/html/2607.18343#S1.p2.2),[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px3.p1.3),[4th item](https://arxiv.org/html/2607.18343#S4.I1.i4.p1.2)\.
- \[2\]D\. Alistarh, D\. Grubic, J\. Li, R\. Tomioka, and M\. Vojnovic\(2017\)QSGD: communication\-efficient SGD via gradient quantization and encoding\.InAdvances in Neural Information Processing Systems 30 \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2607.18343#S1.p2.2),[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px3.p1.3)\.
- \[3\]J\. Bernstein, Y\. Wang, K\. Azizzadenesheli, and A\. Anandkumar\(2018\)signSGD: compressed optimisation for non\-convex problems\.InProceedings of the 35th International Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px3.p1.3)\.
- \[4\]S\. Choudhary, S\. A\. Aketi, and K\. Roy\(2026\)Achieving global flatness in decentralized learning with heterogeneous data\.Transactions on Machine Learning Research\.Note:Accepted by TMLRCited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px5.p1.2)\.
- \[5\]A\. Douillard, Q\. Feng, A\. A\. Rusu, R\. Chhaparia, Y\. Donchev, A\. Kuncoro, M\. Ranzato, A\. Szlam, and J\. Shen\(2023\)DiLoCo: distributed low\-communication training of language models\.arXiv preprint arXiv:2311\.08105\.Cited by:[§1](https://arxiv.org/html/2607.18343#S1.p1.1),[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px2.p1.1)\.
- \[6\]E\. Hoffer, I\. Hubara, and D\. Soudry\(2018\)Fix your classifier: the marginal value of training the last weight layer\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px6.p1.1)\.
- \[7\]K\. Hsieh, A\. Phanishayee, O\. Mutlu, and P\. B\. Gibbons\(2020\)The non\-IID data quagmire of decentralized machine learning\.InProceedings of the 37th International Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px6.p1.1),[§4](https://arxiv.org/html/2607.18343#S4.SS0.SSS0.Px1.p1.5)\.
- \[8\]E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen\(2022\)LoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px4.p1.1)\.
- \[9\]P\. Kairouz, H\. B\. McMahan, B\. Avent, A\. Bellet, M\. Bennis, A\. N\. Bhagoji, K\. Bonawitz, Z\. Charles, G\. Cormode, R\. Cummings,et al\.\(2019\)Advances and open problems in federated learning\.arXiv preprint arXiv:1912\.04977\.Cited by:[§1](https://arxiv.org/html/2607.18343#S1.p1.1)\.
- \[10\]S\. P\. Karimireddy, S\. Kale, M\. Mohri, S\. J\. Reddi, S\. U\. Stich, and A\. T\. Suresh\(2019\)SCAFFOLD: stochastic controlled averaging for on\-device federated learning\.arXiv preprint arXiv:1910\.06378\.Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px1.p1.2)\.
- \[11\]S\. P\. Karimireddy, Q\. Rebjock, S\. U\. Stich, and M\. Jaggi\(2019\)Error feedback fixes SignSGD and other gradient compression schemes\.InProceedings of the 36th International Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px3.p1.3)\.
- \[12\]A\. Khaled, K\. Mishchenko, and P\. Richtárik\(2020\)Tighter theory for local SGD on identical and heterogeneous data\.InProceedings of the 23rd International Conference on Artificial Intelligence and Statistics \(AISTATS\),Proceedings of Machine Learning Research, Vol\.108,pp\. 4519–4529\.Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px2.p1.1)\.
- \[13\]S\. Kharrat, M\. Canini, and S\. Horváth\(2025\)DPFL: decentralized personalized federated learning\.InProceedings of The 28th International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.258,pp\. 5086–5094\.Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px5.p1.2)\.
- \[14\]C\. Li, H\. Farkhoor, R\. Liu, and J\. Yosinski\(2018\)Measuring the intrinsic dimension of objective landscapes\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2607.18343#S6.SS0.SSS0.Px1.p1.2)\.
- \[15\]D\. Li and J\. Wang\(2019\)FedMD: heterogeneous federated learning via model distillation\.arXiv preprint arXiv:1910\.03581\.Note:NeurIPS 2019 Workshop on Federated Learning for Data Privacy and ConfidentialityCited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px5.p1.2)\.
- \[16\]T\. Li, A\. K\. Sahu, M\. Zaheer, M\. Sanjabi, A\. Talwalkar, and V\. Smith\(2018\)Federated optimization in heterogeneous networks\.arXiv preprint arXiv:1812\.06127\.Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px1.p1.2)\.
- \[17\]X\. Li, M\. Jiang, X\. Zhang, M\. Kamp, and Q\. Dou\(2021\)FedBN: federated learning on non\-IID features via local batch normalization\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px6.p1.1)\.
- \[18\]I\. Lin, O\. Yagan, and C\. Joe\-Wong\(2025\)FedSPD: a soft\-clustering approach for personalized decentralized federated learning\.InProceedings of the Forty\-first Conference on Uncertainty in Artificial Intelligence,Proceedings of Machine Learning Research, Vol\.286,pp\. 2618–2641\.Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px5.p1.2)\.
- \[19\]T\. Lin, L\. Kong, S\. U\. Stich, and M\. Jaggi\(2020\)Ensemble distillation for robust model fusion in federated learning\.arXiv preprint arXiv:2006\.07242\.Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px5.p1.2)\.
- \[20\]Y\. Lin, S\. Han, H\. Mao, Y\. Wang, and W\. J\. Dally\(2018\)Deep gradient compression: reducing the communication bandwidth for distributed training\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px3.p1.3)\.
- \[21\]H\. B\. McMahan, E\. Moore, D\. Ramage, S\. Hampson, and B\. Agüera y Arcas\(2017\)Communication\-efficient learning of deep networks from decentralized data\.InProceedings of the 20th International Conference on Artificial Intelligence and Statistics \(AISTATS\),Proceedings of Machine Learning Research, Vol\.54,pp\. 1273–1282\.Cited by:[§1](https://arxiv.org/html/2607.18343#S1.p1.1),[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px1.p1.2),[2nd item](https://arxiv.org/html/2607.18343#S4.I1.i2.p1.1)\.
- \[22\]J\. Oh, S\. Kim, and S\. Yun\(2022\)FedBABU: toward enhanced representation for federated image classification\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px6.p1.1)\.
- \[23\]V\. Papyan, X\. Y\. Han, and D\. L\. Donoho\(2020\)Prevalence of neural collapse during the terminal phase of deep learning training\.Proceedings of the National Academy of Sciences117\(40\),pp\. 24652–24663\.Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px6.p1.1)\.
- \[24\]F\. Pernici, M\. Bruni, C\. Baecchi, and A\. Del Bimbo\(2021\)Regular polytope networks\.IEEE Transactions on Neural Networks and Learning Systems\.Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px6.p1.1)\.
- \[25\]S\. Reddi, Z\. Charles, M\. Zaheer, Z\. Garrett, K\. Rush, J\. Konečný, S\. Kumar, and H\. B\. McMahan\(2020\)Adaptive federated optimization\.arXiv preprint arXiv:2003\.00295\.Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px1.p1.2)\.
- \[26\]L\. Sen and S\. Mukherjee\(2026\)Mapping networks: generating network weights from low\-dimensional latents\.arXiv preprint arXiv:2602\.19134\.Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px4.p1.1),[§3\.1](https://arxiv.org/html/2607.18343#S3.SS1.p1.4)\.
- \[27\]S\. U\. Stich\(2019\)Local SGD converges fast and communicates little\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px2.p1.1)\.
- \[28\]G\. Thompson, K\. Yue, C\. Wong, and H\. Dai\(2025\)NTK\-dfl: enhancing decentralized federated learning in heterogeneous settings via neural tangent kernel\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 59470–59491\.Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px5.p1.2)\.
- \[29\]T\. Vogels, S\. P\. Karimireddy, and M\. Jaggi\(2019\)PowerSGD: practical low\-rank gradient compression for distributed optimization\.InAdvances in Neural Information Processing Systems 32 \(NeurIPS\),Cited by:[§1](https://arxiv.org/html/2607.18343#S1.p2.2),[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px3.p1.3),[3rd item](https://arxiv.org/html/2607.18343#S4.I1.i3.p1.1)\.
- \[30\]H\. Wang, M\. Yurochkin, Y\. Sun, D\. Papailiopoulos, and Y\. Khazaeni\(2020\)Federated learning with matched averaging\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px1.p1.2)\.
- \[31\]Y\. Wu and K\. He\(2018\)Group normalization\.InProceedings of the European Conference on Computer Vision \(ECCV\),Cited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px6.p1.1),[§4](https://arxiv.org/html/2607.18343#S4.SS0.SSS0.Px1.p1.5)\.
- \[32\]S\. Zehtabi, D\. Han, R\. Parasnis, S\. Hosseinalipour, and C\. Brinton\(2025\)Decentralized sporadic federated learning: a unified algorithmic framework with convergence guarantees\.InInternational Conference on Learning Representations,Note:SpotlightCited by:[§2](https://arxiv.org/html/2607.18343#S2.SS0.SSS0.Px5.p1.2)\.

## Appendix 0\.AAdditional experimental figures

This appendix collects extra figures referenced by the main results \(§[5](https://arxiv.org/html/2607.18343#S5)\) but not shown there for length reasons\.

### 0\.A\.1Fixed\-byte budget

Under a fixed200200MB per\-client budget on CIFAR\-10 \(α=0\.5\\alpha\{=\}0\.5\), our latent completes∼\\sim150150rounds and reaches94\.21%94\.21\\%while full\-weight FedAvg can afford one round and attains93\.48%93\.48\\%\(\+0\.73\+0\.73pp for ours, Fig\.[4](https://arxiv.org/html/2607.18343#Pt0.A1.F4)\)\. This is the fixed\-budget view of §[5\.5](https://arxiv.org/html/2607.18343#S5.SS5)\.

![Refer to caption](https://arxiv.org/html/2607.18343v1/x5.png)Figure 4:Fixed200200MB/client budget \(CIFAR\-10,α=0\.5\\alpha\{=\}0\.5\): many cheap latent rounds vs one expensive full\-weight round\.
### 0\.A\.2Client\-count sweep

Figure[5](https://arxiv.org/html/2607.18343#Pt0.A1.F5)sweepsK∈\{4,8,16,32\}K\\in\\\{4,8,16,32\\\}on CIFAR\-100\. Latent accuracy is essentially flat inKK\(and the per\-round compression ratio isKK\-invariant by construction\)\. Full\-weight FedAvg underα=0\.5\\alpha\{=\}0\.5degrades at smallKK\(down to74\.2%74\.2\\%atK=4K\{=\}4\) and only recovers asKKgrows, while our method is stable across the range\. This complements the client\-count summary in §[5\.5](https://arxiv.org/html/2607.18343#S5.SS5)\.

![Refer to caption](https://arxiv.org/html/2607.18343v1/x6.png)Figure 5:Scaling in client countKK\(CIFAR\-100\)\. Latent accuracy is stable inKK; full\-weight is more sensitive at smallKK\.
### 0\.A\.3Frozen orthogonal head figure

Figure[6](https://arxiv.org/html/2607.18343#Pt0.A1.F6)visualises the classifier\-head ablation cited in §[5\.6](https://arxiv.org/html/2607.18343#S5.SS6)\. The frozen orthogonal head improves accuracy over both a trainable and a frozen Kaiming head, at zero communication cost\.

![Refer to caption](https://arxiv.org/html/2607.18343v1/x7.png)Figure 6:Classifier\-head ablation on CIFAR\-100 \(frozen orthogonal vs\. trainable vs\. frozen Kaiming\)\.
### 0\.A\.4Quantization precision plot

Figure[7](https://arxiv.org/html/2607.18343#Pt0.A1.F7)shows accuracy vs\. communicated precision on CIFAR\-100 \(d=64d\{=\}64\), the plot complement of Table[5](https://arxiv.org/html/2607.18343#S5.T5)in §[5\.9](https://arxiv.org/html/2607.18343#S5.SS9)\. Our latent is flat across fp32/int8/int4; full\-weight FedAvg collapses at int4\.

![Refer to caption](https://arxiv.org/html/2607.18343v1/x8.png)Figure 7:Latent quantization \(CIFAR\-100,d=64d\{=\}64\): accuracy vs communicated precision\.
### 0\.A\.5Quantization–bandwidth Pareto

Combining the latent channel with integer quantization pushes the per\-round payload below11KB at parity accuracy \(Fig\.[8](https://arxiv.org/html/2607.18343#Pt0.A1.F8)\), extending the Pareto discussion of §[5\.3](https://arxiv.org/html/2607.18343#S5.SS3)\.

![Refer to caption](https://arxiv.org/html/2607.18343v1/x9.png)Figure 8:Latent\-quantization bandwidth–accuracy points: int4 latent reaches648648B/round at parity accuracy\.

## Appendix 0\.BLanguage modelling on WikiText\-2

Beyond vision, we applyFLITEto federated language\-model fine\-tuning on WikiText\-2 with the TinyGPT backbone \(d=1024d\{=\}1024,K=8K\{=\}8,R=20R\{=\}20,11local epoch, IID split;33seeds\)\. Table[6](https://arxiv.org/html/2607.18343#Pt0.A2.T6)and Fig\.[9](https://arxiv.org/html/2607.18343#Pt0.A2.F9)report validation perplexity \(lower is better\)\.

The headline result here iscommunication: our latent channel transmits≈144\\approx 144KB per client per round versus≈44\\approx 44MB for full\-weight FedAvg — a≈313×\\approx 313\\timesreduction — while holding validation perplexity essentially at the shared pretrained base \(66\.0966\.09throughout federation\)\. By contrast, full\-weight FedAvg can briefly improve perplexity in early rounds \(61\.71±0\.0361\.71\\pm 0\.03best\) but then*diverges*as heterogeneous local updates are averaged \(81\.06±0\.0981\.06\\pm 0\.09final\)\. We therefore treat WikiText\-2 primarily as evidence that the kilobyte\-scale payload transfers to autoregressive LMs; accuracy parity on this task is left to future tuning of local steps and learning rates\.

Table 6:WikiText\-2 \(TinyGPT,d=1024d\{=\}1024,K=8K\{=\}8,R=20R\{=\}20, IID\)\. Per\-round payload and validation perplexity \(mean over 3 seeds\)\. Latent payload is313×313\\timessmaller\. Lower perplexity is better\.![Refer to caption](https://arxiv.org/html/2607.18343v1/x10.png)Figure 9:WikiText\-2 \(TinyGPT\): validation perplexity vs\. communication round\. Shaded bands are±1\\pm 1std over33seeds\. Latent federation is stable at the pretrained base; full\-weight FedAvg improves briefly then drifts\.
## Appendix 0\.CHyperparameters and training recipes

#### Centralized pretraining \(shared base\)\.

SGD with momentum0\.90\.9, initial learning rate0\.050\.05,33\-epoch linear warmup then cosine decay to zero, weight decay5×10−45\\times 10^\{\-4\}, batch size128128,100100epochs, standard crop\+flip augmentation\. GroupNorm with3232groups\. The classifier head is orthogonally initialised and frozen from the first step\. Per\-epoch checkpointing enables resume\.

#### Latent \(mapping\) training\.

Adam, learning rate10−210^\{\-2\}, weight decay10−410^\{\-4\}, gradient clipping at1\.01\.0, latents initialised at zero \(so the model starts exactly at the pretrained base\)\. Defaultd=64d=64,r=32r=32on CIFAR\-100\. Backbone, projection factorsU,VU,V, and the classifier head are all frozen; only the per\-layer latentszlz\_\{l\}are trained\.

#### Federated configuration\.

KKclients \(K∈\{4,8,16,32\}K\\in\\\{4,8,16,32\\\}, default88\),EElocal epochs per round, IID or Dirichlet\(α\\alpha\) partition withα∈\{0\.5,0\.1\}\\alpha\\in\\\{0\.5,0\.1\\\}\. Server averaging is the unweighted mean of client latents \(ours\) or client payloads \(baselines\)\. Baselines: full\-weight FedAvg; PowerSGD rank∈\{2,8\}\\in\\\{2,8\\\}with error feedback; top\-kkdensity∈\{1%,10%\}\\in\\\{1\\%,10\\%\\\}with error feedback\. Except for the head\-to\-head study in the main paper, all methods share the same frozen pretrained base and the same frozen orthogonal head\.

## Appendix 0\.DGenerator memory analysis

The original mapping\-network generator stores a dense projectionWmW\_\{m\}of shapePl×dlP\_\{l\}\\times d\_\{l\}for each mapped layer\. Aggregated over ResNet\-18 \(∑lPl≈11\.2\\sum\_\{l\}P\_\{l\}\\approx 11\.2M\), even a modest per\-layerdld\_\{l\}makes the dense projections require on the order of8080GB infp32, exceeding a single accelerator\. Our low\-rank factorisationWm=U​V⊤W\_\{m\}=UV^\{\\top\}storesr​\(Pl\+dl\)r\\,\(P\_\{l\}\+d\_\{l\}\)values per layer instead ofPl​dlP\_\{l\}\\,d\_\{l\}; atr=32r=32this is≈2\\approx 2GB\. Seed regeneration stores neither factor — a single6464\-bit seed deterministically regeneratesUl,VlU\_\{l\},V\_\{l\}\(Gaussian draw \+ QR\) identically on every client — so only≈10\\approx 10MB of peak working memory is needed during regeneration\. Figure[10](https://arxiv.org/html/2607.18343#Pt0.A4.F10)compares these storage options\. The exact averaging identity is preserved in all variants becauseUl,VlU\_\{l\},V\_\{l\}are identical across clients \(same seed⇒\\Rightarrowsame factors\)\.

![Refer to caption](https://arxiv.org/html/2607.18343v1/x11.png)Figure 10:Generator storage for ResNet\-18: dense \(≈80\\approx 80GB\) vs low\-rank \(≈2\\approx 2GB\) vs seed\-regenerated \(≈10\\approx 10MB working memory\)\.
## Appendix 0\.EFrom\-scratch failure at scale

Training the latent from a randomly \(Kaiming\) initialised base, with no pretrained offset, collapses at ResNet\-18 scale: on CIFAR\-100 it reaches only≈2\.5%\\approx 2\.5\\%test accuracy, against74\.7%74\.7\\%in delta mode under otherwise identical settings\. The latent has the capacity to encode a correction around a strong base, but not to drive a large network from scratch; this motivates the delta\-on\-pretrained framing and our positioning of the method as low\-bandwidth federated*fine\-tuning*\.

## Appendix 0\.FRank sweep \(rank irrelevance under delta mode\)

At fixedd=256d=256on CIFAR\-100, varying the projection rankr∈\{8,32,128,256\}r\\in\\\{8,32,128,256\\\}changes test accuracy by<0\.2<0\.2pp\. The rank therefore trades only generator memory and regeneration compute, not accuracy, consistent with the delta a well\-pretrained model needs being itself low\-dimensional\. We user=32r=32throughout\.

## Appendix 0\.GReproducibility

Every run logs the latent\-averaging identity error \(maximum discrepancy between averaging latents and averaging the generated weights\), which is at the level of floating\-point round\-off \(≤6×10−8\\leq 6\\times 10^\{\-8\}\) in all experiments\. Pretraining and mapping checkpoint per epoch and are resumable\. Communication is accounted in bytes \(both directions,fp32\), counting transmitted values and indices for top\-kkand both factors for PowerSGD\. Code and configuration files will be released\.