PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs

# PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs
Source: [https://arxiv.org/html/2607.12111](https://arxiv.org/html/2607.12111)
Jing Liu[ID](https://orcid.org/0000-0002-2819-0200), , Kun Yang[ID](https://orcid.org/0000-0002-9956-2200), Yan Wang[ID](https://orcid.org/0000-0002-4953-2660), Dingkang Yang[ID](https://orcid.org/0000-0003-1829-5671), Xiaoshuai Hao[ID](https://orcid.org/0009-0007-4209-6695), Wei Zhang[ID](https://orcid.org/0009-0002-3238-7898), Yang Liu[ID](https://orcid.org/0000-0002-1312-0146), , and Wei Zhou[ID](https://orcid.org/0000-0003-3641-1429)Jing Liu is with the College of Future Information Technology, Fudan University, Shanghai 200433, China, also with the Division of Natural and Applied Sciences, Duke Kunshan University, Suzhou 215316, China, and also with the Department of Electrical and Computer Engineering, The University of British Columbia, BC V6T 1Z4, Canada \(e\-mail: jing\.liu@ieee\.org\)\. Kun Yang is with the Ant Group, also with the College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou 310013, China \(e\-mail: kunyang20@zju\.edu\.cn\)\. Yan Wang is with the School of Data Science and Engineering, East China Normal University, Shanghai 200062, China \(e\-mail: yanwang@dase\.ecnu\.edu\.cn\)\. Dingkang Yang is with the College of Intelligent Robotics and Advanced Manufacturing, Fudan University & Fysics AI, Shanghai 200433, China \(e\-mail: dkyang20@fudan\.edu\.cn\)\. Xiaoshuai Hao is with Xiaomi EV, Xiaomi Campus, Anningzhuang Road, Haidian District, 100085, Beijing, China \(e\-mail: haoxiaoshuai@xiaomi\.com\)\. Wei Zhang is with the Information and Communications Technology Cluster, Singapore Institute of Technology \(SIT\), Singapore 828608 \(e\-mail: wei\.zhang@singaporetech\.edu\.sg\)\. Yang Liu is with the College of Electronic and Information Engineering, Tongji University, Shanghai 201804, China \(e\-mail: yang\_liu@ieee\.org\)\. Wei Zhou is with the School of Computer Science and Informatics, Cardiff University, CF24 4AG Cardiff, U\.K\. \(e\-mail: zhouw26@cardiff\.ac\.uk\)\.

###### Abstract

Agentic AI systems are reshaping communications and networking by deploying autonomous intelligent agents capable of collaborative learning while maintaining data privacy at network edges\. Within distributed network environments, Multimodal Large Language Models \(MLLMs\) serve as cognitive engines for edge devices, yet federated fine\-tuning faces substantial challenges in balancing global knowledge aggregation with local adaptation under heterogeneous network conditions\. Conventional federated protocols typically rely on uniform parameter aggregation, which conflates domain\-invariant features with client\-specific nuances, thereby resulting in suboptimal personalization and excessive communication overhead\. To address these challenges, we proposePFAdapter, a communication\-efficient framework introducing hierarchical LoRA decomposition to explicitly separate adapter parameters into global\-shared and local\-private components\. Query and key projections are assigned to global synchronization for capturing universal multimodal semantics across the network, while value and output projections remain localized for edge\-specific adaptation\. Additionally, orthogonality regularization based on the Frobenius norm enforces strict separation between these components, preventing redundant feature learning\. Selective aggregation protocols synchronize only global\-shared components across the federated network, preserving local expertise and reducing communication costs by nearly 50%\. Experiments on medical VQA and social multimodal benchmarks show thatPFAdapterconsistently improves over matched federated LoRA baselines while nearly halving synchronized adapter traffic\. These results indicate that projection\-level decomposition offers a practical path toward communication\-efficient agentic MLLM deployment in resource\-constrained edge networks\.

###### Index Terms:

Federated Learning, Edge Intelligence, Multimodal Large Language Models, Communication Efficiency, Agentic AI, Personalization, Parameter\-Efficient Fine\-Tuning

## IIntroduction

Agentic AI represents a transformative paradigm for modern communication networks, where autonomous agents collaborate to process multimodal information at network edges while keeping raw training data local to each client\[baltruvsaitis2018multimodal,radford2021learning\]\. Multimodal Large Language Models \(MLLMs\) serve as cognitive cores for intelligent agents deployed across distributed networking systems, enabling sophisticated reasoning over heterogeneous data modalities including text, images, and sensor data\[liu2025storageaware,mao2025optimizing,liu2026semantic\]\. Urgent deployment scenarios emerge in manufacturing anomaly detection requiring rapid inference under constrained budgets, autonomous driving processing multi\-sensor data with stringent latency, and medical diagnostics handling privacy\-sensitive imaging across distributed facilities\[yang2019federated,gao2025learner\], where Federated Learning \(FL\) enables collaborative model training without centralized data aggregation\[mcmahan2017communicationefficient,cheng2025snowball\]\. Integrating FL with MLLMs becomes critical for realizing scalable data\-local agentic systems across next\-generation communication networks\[wu2025survey,he2025dualcirculation,liu2026enhancing\]\. We use this wording deliberately: FL keeps raw samples on device, but it does not by itself guarantee resistance to update inversion, gradient leakage, or membership inference\[liu2025differentially,deng2023federated\]\. The privacy scope ofPFAdapteris therefore limited to decentralized training without centralized raw\-data pooling, and the stronger attack\-model discussion is deferred to[Sec\.˜V\-C](https://arxiv.org/html/2607.12111#S5.SS3)\.

Deploying federated MLLMs across heterogeneous edge networks introduces substantial challenges arising from extreme variations in local data distributions and network conditions\. Medical edge devices often process specialized imaging data that varies significantly across equipment manufacturers and patient demographics\[lau2018dataset,liu2021slake,liu2025multimodala\], while IoT nodes in social sensing networks must handle diverse cultural contexts and evolving linguistic patterns\[kiela2020hateful,alam2018crisismmd\]\. Standard global models frequently fail to achieve optimal performance on specialized edge tasks due to uniform aggregation strategies that ignore local data characteristics\. Consequently, developing personalized federated learning approaches becomes imperative for edge intelligence systems, enabling adaptation to local nuances while benefiting from collective knowledge\[fallah2020personalized,t\.dinh2020personalized,zhang2025personalized\]\.

Fundamental technical barriers hinder effective MLLM personalization within distributed communication networks\. Balancing generalizable multimodal representations against edge\-specific task expertise poses a primary challenge for network\-deployed agents\[wilson2020survey,he2025hivefl\], where uniform parameter aggregation induces catastrophic weight washing that dilutes critical client\-specific patterns through global averaging\[cheng2025federated,jia2025joint\]\. In manufacturing anomaly detection, extended local training capturing equipment\-specific fault signatures paradoxically degrades targeted performance after aggregation while fundamentally undermining model reliability by erasing calibrations essential for safety\-critical autonomous systems\[liu2025networking,liu2025anomaly\]\. Although earlier FL approaches attempted mitigation through proximal regularization\[li2020federated\], multi\-task learning\[zhang2021survey\], and meta\-learning\[fallah2020personalized\], high\-dimensional MLLM parameter spaces frequently lead to insufficient personalization or catastrophic forgetting across the federated network\[yao2025federatedllm\]\. Communication overhead represents a critical bottleneck for federated MLLM deployment across bandwidth\-constrained edge networks\. Even when employing Parameter\-Efficient Fine\-Tuning \(PEFT\) techniques like Low\-Rank Adaptation \(LoRA\)\[hu2021lora\], MLLM backbones impose prohibitive synchronization costs for resource\-limited edge devices\[liu2025storageaware\]\. Under typical industrial edge deployments with limited bandwidth \(e\.g\., 5\-20 Mbps in manufacturing facilities or vehicular networks\), transmitting hundreds of megabytes of adapter parameters per round incurs substantial delays that accumulate across training iterations, proving operationally unacceptable for time\-sensitive applications like autonomous vehicle fleets or manufacturing anomaly detection\[lim2020federated,liu2024generalized\]\. Existing solutions frequently overlook redundancy within adapter parameters by transmitting all modules uniformly despite certain components capturing universal features while others remain highly task\-specific\[fallah2020personalized,wu2025survey\], consequently wasting scarce network resources\.

![Refer to caption](https://arxiv.org/html/2607.12111v1/x1.png)Figure 1:Illustration of challenges in federated MLLM fine\-tuning and our proposed solution\. \(a\) Existing FL methods uniformly aggregate all model parameters, leading to high communication costs, loss of client\-specific knowledge, and poor personalization under heterogeneous data distributions\. \(b\)PFAdapterintroduces hierarchical LoRA decomposition with selective aggregation, where only global\-shared adapters are synchronized, while local\-private adapters remain on\-device\.Recent advances in federated MLLM fine\-tuning have introduced various strategies to mitigate communication challenges\.FedAvg\[mcmahan2017communicationefficient\]establishes the baseline for collaborative learning across distributed networks, whereasFedProx\[li2020federated\]incorporates proximal regularization for handling system and statistical heterogeneity\. More recently, frameworks includingFlexLoRA\[bai2024federated\]andFedMLLM\[xu2025fedmllm\]have explored LoRA\-based adaptation to reduce trainable parameters\. Systematic analysis reveals that uniform aggregation of all LoRA modules conflates domain\-invariant features with client\-specific nuances\[yi2024pfedlora,wang2024flora\]\. Additionally, methods includingFedPer\[arivazhagan2019federated\]andFedRep\[collins2021exploiting\]attempt model splitting into global and local layers, yet their application to complex MLLM architectures remains suboptimal\. Observed limitations highlight critical gaps in achieving precise knowledge disentanglement within the adapter space\.

To address communication and personalization challenges in edge networks, we proposePFAdapter, a resource\-efficient federated learning framework for agentic MLLM deployment via hierarchical adapter decomposition\. As illustrated in[Fig\.˜1](https://arxiv.org/html/2607.12111#S1.F1), our framework explicitly separates adapter parameters into global\-shared and local\-private sets based on functional roles of self\-attention projections, where query and key projections \(qp,kpq\_\{p\},k\_\{p\}\) capture universal multimodal semantics for global synchronization while value and output projections \(vp,opv\_\{p\},o\_\{p\}\) remain localized for edge\-specific adaptation\. Orthogonality regularization enforces strict separation by minimizing inner products between global and local parameter matrices, thereby reducing interference in respective feature subspaces while maximizing complementary representation capacity and preventing redundant feature learning\. The main contributions of this work are summarized as follows:

- •We introduce a novel architectural decomposition for MLLM adapters, categorizing projection modules into global and local components based on their functional roles in capturing multimodal semantics versus edge\-specific features\.
- •We propose Frobenius\-norm based orthogonality regularization to minimize correlation between global and local adapter weights, ensuring precise knowledge separation for distributed agents\.
- •We develop a communication\-efficient synchronization strategy that transmits only global\-shared parameters, reducing network traffic by nearly 50% while preserving local edge expertise\.
- •We conduct evaluations across medical VQA and social multimodal benchmarks, showing consistent accuracy, robustness, and communication\-efficiency gains under matched federated LoRA protocols\.

The remainder of this paper is organized as follows\.[Sec\.˜II](https://arxiv.org/html/2607.12111#S2)reviews related work in federated learning and MLLM fine\-tuning\.[Sec\.˜III](https://arxiv.org/html/2607.12111#S3)provides the necessary preliminaries and problem formulation\.[Sec\.˜IV](https://arxiv.org/html/2607.12111#S4)details the proposedPFAdapterframework and its technical components\.[Sec\.˜V](https://arxiv.org/html/2607.12111#S5)describes the experimental setup and analyzes the results\. Finally,[Sec\.˜VI](https://arxiv.org/html/2607.12111#S6)concludes the paper\. Additional theoretical analysis, implementation details, and extended experimental results are provided in the supplementary material,LABEL:supp:sec:supp\_theory,LABEL:supp:sec:supp\_protocol\_resultsandLABEL:supp:sec:supp\_extended\_interpretation\.

## IIRelated Work

### II\-AFederated learning for large language models

Federated fine\-tuning of Large Language Models \(LLMs\) and MLLMs has emerged as a critical research direction due to the growing demand for collaborative learning across institutions that cannot pool raw multimodal data centrally\[yao2025federatedllm,liu2026collaborative\]\. Early federated learning frameworks such asFedAvg\[mcmahan2017communicationefficient\]established the foundation for decentralized model training, yet they were designed for relatively small\-scale neural networks and homogeneous data distributions\. The emergence of parameter\-efficient fine\-tuning techniques, particularly LoRA\[hu2021lora\], has enabled practical federated adaptation of billion\-scale models by reducing the number of trainable parameters from billions to millions\. Recent frameworks such asFlexLoRA\[bai2024federated\]and split\-learning approaches\[li2025energyefficient\]have demonstrated the potential of LoRA\-based federated fine\-tuning for heterogeneous tasks and resource\-constrained environments\. However, recent work has identified significant challenges in applying standard FL protocols to MLLMs, including catastrophic forgetting of global knowledge during local updates and the "weight washing" phenomenon where client\-specific adaptations are diluted through uniform aggregation\[cheng2025federated\]\. Several approaches have attempted to address these limitations through regularization techniques, multi\-stage training protocols\[yang2025surveya\], and hybrid architectures that separate feature extractors from task\-specific heads\. Nevertheless, existing methods treat adapter parameters as a monolithic entity without considering the functional heterogeneity within different projection modules of the self\-attention mechanism\. In contrast,PFAdapterexplicitly recognizes that query and key projections tend to capture structural multimodal relationships that are more amenable to global aggregation, whereas value and output projections are inherently more task\-specific and should remain personalized\.

### II\-BPersonalized federated learning

Personalization in federated learning aims to balance the acquisition of global knowledge with the preservation of local expertise, particularly important for non\-IID data distributions\[wu2025survey\]\. Among pioneering approaches,FedPer\[arivazhagan2019federated\]introduced the concept of maintaining personalized layers at each client while aggregating only the base model parameters\. Building upon this foundation,FedRep\[collins2021exploiting\]proposed a representation learning approach where feature extractors are synchronized globally and classifiers remain local\. Recent work has explored more sophisticated personalization strategies including prototype\-based learning\[tan2022fedproto\], meta\-learning frameworks\[fallah2020personalized\], and dual\-prompt optimization\[zhang2025personalized,bai2024federated\]\. Furthermore,FedBABU\[oh2021fedbabu\]demonstrated that keeping the model head frozen during server aggregation can significantly improve personalization performance\. For multimodal scenarios,\[wang2024notall\]introduced task\-similarity\-aware model aggregation for heterogeneous multi\-modal clients\. However, these methods primarily focus on architectural separation at the layer level rather than parameter\-level decomposition within individual modules\. Moreover, layer\-wise splitting strategies become inefficient for deep transformer architectures where task\-specific knowledge is distributed across multiple layers\.PFAdapteraddresses this limitation by performing fine\-grained decomposition at the projection module level, enabling more nuanced control over which aspects of the model are shared versus personalized\. Compared withFedPerandFedRep\[arivazhagan2019federated,collins2021exploiting\], which personalize entire layers or heads,PFAdaptermoves the personalization boundary inside each attention block and explicitly separates globally aggregated query/key projections from locally retained value/output projections\. Accordingly, the projection\-level split is tailored to MLLM adapters, where structural cross\-modal alignment must be shared across clients but semantic realization remains strongly client\-dependent under non\-IID data\.

### II\-CParameter\-efficient fine\-tuning in federated settings

PEFT has become indispensable for adapting large\-scale models in resource\-constrained environments\[kim2025missing\]\. Beyond LoRA, several PEFT variants have been developed including adapter layers\[houlsby2019parameter\], prompt tuning\[lester2021power\], prefix tuning\[li2021prefix\], and Hadamard adapters\[chen2023hadamard\]\. Recent work has explored the integration of these techniques into federated learning frameworks to reduce communication overhead and computational burden\[wu2025survey\]\. Methods such asFedAdapter\[yan2024federa\]andpFedLoRA\[yi2024pfedlora\]have demonstrated the effectiveness of adapter\-based personalization for language models\. Building upon these foundations,FloRA\[wang2024flora\]introduced heterogeneous low\-rank adaptations for federated LLM fine\-tuning, while recent work on adaptive LoRA experts and differentially private federated LoRA\[liu2025differentially\]have further advanced the field\. For multimodal applications, approaches likeFedDLP\[nguyen2025federated\]have employed dual adapters with selective pruning to balance local specialization and global knowledge sharing, while visual\-language enhancement systems motivate similarly compact adaptation under edge visual workloads\[wu2025clipae\]\. However, these approaches typically apply uniform aggregation to all adapter parameters, failing to distinguish between domain\-invariant and domain\-specific components\. TheFedMLLMframework\[xu2025fedmllm\]represents the state\-of\-the\-art in federated MLLM fine\-tuning, yet it relies on full adapter synchronization that conflates global and local knowledge\. Recent theoretical analysis has shown that the optimal aggregation strategy should vary across different parameter subsets based on their sensitivity to local data distributions\[fallah2020personalized\]\. Motivated by this insight,PFAdapterintroduces selective aggregation that only synchronizes the global\-shared adapter components, reducing communication costs while preserving local expertise\.LABEL:supp:tab:supp\_pfadapter\_method\_comparisonprovides the full structured comparison of representative personalized FL and federated LoRA methods\[yi2024pfedlora,wang2024flora,nguyen2025federated\]\. In addition, it contrastsPFAdapterwith the closely relatedFedMLLMsetting\[xu2025fedmllm\]\.

## IIIPreliminaries

### III\-AProblem Formulation

Consider a federated learning system consisting of a central server and a set ofKKheterogeneous clients, where each clientk∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\}possesses a local multimodal dataset𝒟k\\mathcal\{D\}\_\{k\}\. Given the input space𝒳\\mathcal\{X\}comprising image\-text pairs and the output space𝒴\\mathcal\{Y\}representing textual responses, the local dataset is defined as𝒟k=\{\(𝐱k,i,yk,i\)\}i=1nk\\mathcal\{D\}\_\{k\}=\\\{\(\\mathbf\{x\}\_\{k,i\},y\_\{k,i\}\)\\\}\_\{i=1\}^\{n\_\{k\}\}, wherenkn\_\{k\}denotes the number of local samples\. Letfθ:𝒳→𝒴f\_\{\\theta\}:\\mathcal\{X\}\\rightarrow\\mathcal\{Y\}represent an MLLM parameterized byθ∈ℝd\\theta\\in\\mathbb\{R\}^\{d\}\. The primary objective in personalized federated learning is to find a set of parameters\{θ1,…,θK\}\\\{\\theta\_\{1\},\\dots,\\theta\_\{K\}\\\}that minimize the aggregate empirical risk:

min\{θk\}k=1K​∑k=1KnkN​𝔼\(𝐱,y\)∼𝒟k​\[ℓ​\(fθk​\(𝐱\),y\)\],\\min\_\{\\\{\\theta\_\{k\}\\\}\_\{k=1\}^\{K\}\}\\sum\_\{k=1\}^\{K\}\\frac\{n\_\{k\}\}\{N\}\\mathbb\{E\}\_\{\(\\mathbf\{x\},y\)\\sim\\mathcal\{D\}\_\{k\}\}\[\\ell\(f\_\{\\theta\_\{k\}\}\(\\mathbf\{x\}\),y\)\],\(1\)whereN=∑knkN=\\sum\_\{k\}n\_\{k\}is the total number of samples across all clients andℓ​\(⋅,⋅\)\\ell\(\\cdot,\\cdot\)denotes the cross\-entropy loss function\. Unlike standard federated learning which seeks a single global modelθ∗\\theta^\{\*\}, personalized approaches allow for client\-specific variations to account for statistical heterogeneity\[fallah2020personalized,t\.dinh2020personalized\]\.

### III\-BLow\-Rank Adaptation

LoRA serves as the foundational parameter\-efficient fine\-tuning technique for large\-scale models\. For a pre\-trained weight matrixW0∈ℝd×kW\_\{0\}\\in\\mathbb\{R\}^\{d\\times k\}, LoRA represents the weight updateΔ​W\\Delta Was the product of two low\-rank matricesB∈ℝd×rB\\in\\mathbb\{R\}^\{d\\times r\}andA∈ℝr×kA\\in\\mathbb\{R\}^\{r\\times k\}, where the rankr≪min⁡\(d,k\)r\\ll\\min\(d,k\)\. The forward pass of a modified linear layer is expressed as:

𝐡=W0​𝐱\+Δ​W​𝐱=W0​𝐱\+B​A​𝐱\.\\mathbf\{h\}=W\_\{0\}\\mathbf\{x\}\+\\Delta W\\mathbf\{x\}=W\_\{0\}\\mathbf\{x\}\+BA\\mathbf\{x\}\.\(2\)During the fine\-tuning process,W0W\_\{0\}remains frozen while onlyAAandBBare updated\. In the context of MLLMs, LoRA is typically applied to the projection matrices within the self\-attention mechanism, specifically the query \(WqW\_\{q\}\), key \(WkW\_\{k\}\), value \(WvW\_\{v\}\), and output \(WoW\_\{o\}\) projections\[chen2023hadamard\]\.

### III\-CFederated Optimization

Standard federated optimization often employs theFedAvgprotocol to synchronize model updates across clients\. In each communication roundtt, the server selects a subset of clients𝒮t\\mathcal\{S\}\_\{t\}and transmits the current global parametersθt\\theta^\{t\}\. Each selected client performsEEepochs of local stochastic gradient descent \(SGD\) to obtain updated parametersθkt\+1\\theta\_\{k\}^\{t\+1\}\. The server then aggregates these updates using a weighted average:

θt\+1=∑k∈𝒮tnk∑j∈𝒮tnj​θkt\+1\.\\theta^\{t\+1\}=\\sum\_\{k\\in\\mathcal\{S\}\_\{t\}\}\\frac\{n\_\{k\}\}\{\\sum\_\{j\\in\\mathcal\{S\}\_\{t\}\}n\_\{j\}\}\\theta\_\{k\}^\{t\+1\}\.\(3\)However, applying this uniform aggregation to all LoRA parameters in MLLMs often leads to the dilution of local task\-specific knowledge, necessitating a more granular approach to parameter management\[yi2024pfedlora,wang2024flora\]\.

## IVMethod

### IV\-AMethod Overview

Personalized federated learning for MLLMs requires a delicate balance between global knowledge acquisition and local task adaptation\. To achieve this balance,PFAdapterintroduces a structured decomposition of the adapter parameter space that explicitly separates domain\-invariant features from client\-specific nuances\. Our design philosophy rests on the observation that different projection modules within the self\-attention mechanism exhibit varying degrees of task\-specificity\. Specifically, the framework partitions the set of LoRA modules into global\-shared and local\-private subsets based on their functional roles in processing multimodal information\.[Fig\.˜2](https://arxiv.org/html/2607.12111#S4.F2)illustrates the overall architecture, where query and key projections are synchronized globally while value and output projections are maintained locally\. The overall optimization objective for the system is formulated as: This design differs from layer\-wise personalized FL and fully synchronized federated LoRA\[arivazhagan2019federated,collins2021exploiting,xu2025fedmllm\]because it introduces an intermediate projection\-level control point: attention\-map formation is shared through query/key adapters, while client\-specific representation realization is preserved through value/output adapters\. The supplementary structured comparison summarizes this distinction before the formal derivation\. The closest architectural contrast is withFedDLP\[nguyen2025federated\]: both methods separate shared and personalized adaptation paths, butFedDLPdecides what to transmit by pruning an auxiliary shared branch, whereasPFAdapterfixes the communication boundary at the projection type itself and aggregates only the query/key adapters\. Consequently, the distinction matters in multimodal non\-IID settings because it ties the synchronized subspace directly to attention\-relation formation rather than to a sparsified duplicate branch\.

minΘG,\{ΘkL\}k=1K​∑k=1KnkN​\[ℒt​\(𝒟k;ΘG,ΘkL\)\+λo​ℒo​\(ΘG,ΘkL\)\]\\displaystyle\\min\_\{\\Theta^\{G\},\\\{\\Theta\_\{k\}^\{L\}\\\}\_\{k=1\}^\{K\}\}\\sum\_\{k=1\}^\{K\}\\frac\{n\_\{k\}\}\{N\}\\left\[\\mathcal\{L\}\_\{t\}\(\\mathcal\{D\}\_\{k\};\\Theta^\{G\},\\Theta\_\{k\}^\{L\}\)\+\\lambda\_\{o\}\\mathcal\{L\}\_\{o\}\(\\Theta^\{G\},\\Theta\_\{k\}^\{L\}\)\\right\]

\(4\)whereΘG\\Theta^\{G\}represents the global\-shared parameters andΘkL\\Theta\_\{k\}^\{L\}denotes the local\-private parameters for clientkk\. The task lossℒt\\mathcal\{L\}\_\{t\}is typically defined as the negative log\-likelihood of the target sequence given the multimodal input:

ℒt​\(𝒟k;Θ\)=−1nk​∑i=1nk∑j=1\|yk,i\|log⁡P​\(yk,i,j∣yk,i,<j,𝐱k,i;Θ\)\\displaystyle\\mathcal\{L\}\_\{t\}\(\\mathcal\{D\}\_\{k\};\\Theta\)=\-\\frac\{1\}\{n\_\{k\}\}\\sum\_\{i=1\}^\{n\_\{k\}\}\\sum\_\{j=1\}^\{\|y\_\{k,i\}\|\}\\log P\(y\_\{k,i,j\}\\mid y\_\{k,i,<j\},\\mathbf\{x\}\_\{k,i\};\\Theta\)

\(5\)
![Refer to caption](https://arxiv.org/html/2607.12111v1/x2.png)Figure 2:Architecture ofPFAdapter\. The framework decomposes LoRA adapters into global\-shared components \(qpq\_\{p\},kpk\_\{p\}\) and local\-private components \(vpv\_\{p\},opo\_\{p\}\)\. Orthogonality regularization enforces knowledge disentanglement between these sets\.
### IV\-BHierarchical LoRA Decomposition

Hierarchical decomposition inPFAdapterformally splits the total set of trainable LoRA parametersΘ\\Thetainto two disjoint sets:ΘG\\Theta^\{G\}andΘL\\Theta^\{L\}\. Letℳ=\{q,k,v,o\}\\mathcal\{M\}=\\\{q,k,v,o\\\}denote the set of projection types in the transformer layers\. The global parameter setΘG\\Theta^\{G\}is defined as the union of LoRA weights for query and key projections:

ΘG=⋃l=1L\{Al,m,Bl,m∣m∈\{q,k\}\},\\Theta^\{G\}=\\bigcup\_\{l=1\}^\{L\}\\\{A\_\{l,m\},B\_\{l,m\}\\mid m\\in\\\{q,k\\\}\\\},\(6\)whereLLis the number of transformer layers\. Conversely, the local parameter setΘL\\Theta^\{L\}contains the weights for value and output projections:

ΘL=⋃l=1L\{Al,m,Bl,m∣m∈\{v,o\}\}\.\\Theta^\{L\}=\\bigcup\_\{l=1\}^\{L\}\\\{A\_\{l,m\},B\_\{l,m\}\\mid m\\in\\\{v,o\\\}\\\}\.\(7\)The intuition behind this specific split arises from the functional roles of self\-attention modules\. Query and key projections encode theattention patternsthat determine which tokens attend to each other, capturing structural relationships between multimodal tokens that reflect domain\-invariant correspondence patterns\[vaswani2017attention,liu2026projecting\]\. Recent work on adapter decomposition has shown that attention patterns exhibit higher cross\-domain transferability than value representations\[hu2021lora,houlsby2019parameter\]\. The attention score matrixSSis computed as:

S=\(Wq\+Δ​WqG\)​𝐱​\(\(Wk\+Δ​WkG\)​𝐱\)⊤dk\.S=\\frac\{\(W\_\{q\}\+\\Delta W\_\{q\}^\{G\}\)\\mathbf\{x\}\(\(W\_\{k\}\+\\Delta W\_\{k\}^\{G\}\)\\mathbf\{x\}\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\.\(8\)In contrast, value and output projections encode thesemantic contentthat is transformed into the final representation, making them inherently more task\-specific and susceptible to local data variations\[mickus2024role\]\. The final attended representationV′V^\{\\prime\}is obtained via:

V′=Softmax​\(S\)​\(Wv\+Δ​WvL\)​𝐱\.V^\{\\prime\}=\\text\{Softmax\}\(S\)\(W\_\{v\}\+\\Delta W\_\{v\}^\{L\}\)\\mathbf\{x\}\.\(9\)Consistent with this mechanism, the design aligns with empirical findings that value layers capture more task\-specific information than query\-key pairs in multi\-task learning scenarios\[zhang2021survey,koo2025loraa2\]\. Under non\-IID multimodal federation\[li2020federated,hsu2019measuring,liu2026diffusionguided\], the server needs to preserve client\-agnostic alignment cues while avoiding over\-averaging client\-specific semantics\.[Eqs\.˜8](https://arxiv.org/html/2607.12111#S4.E8)and[9](https://arxiv.org/html/2607.12111#S4.E9)make this separation explicit: query/key adapters perturb the attention logits that determine*which*visual\-textual tokens interact, whereas value/output adapters govern*what*task\-specific content is injected after the shared attention map is formed\[vaswani2017attention,mickus2024role\]\. We therefore assignΘG\\Theta^\{G\}in[Eq\.˜6](https://arxiv.org/html/2607.12111#S4.E6)to the transferable relation\-encoding subspace and keepΘL\\Theta^\{L\}in[Eq\.˜7](https://arxiv.org/html/2607.12111#S4.E7)client\-resident to absorb label skew, vocabulary bias, and modality imbalance without washing out local semantics during aggregation\. To keep the notation consistent throughout the remainder of the paper,ΘL\\Theta^\{L\}in[Eq\.˜7](https://arxiv.org/html/2607.12111#S4.E7)denotes the*structural*set of local\-private slots, while the actual parameters owned by clientkkare written asΘkL\\Theta\_\{k\}^\{L\}\. Accordingly, all optimization objectives, algorithmic updates, and convergence statements below use the pair\(ΘG,ΘkL\)\(\\Theta^\{G\},\\Theta\_\{k\}^\{L\}\)when referring to a concrete client state, consistent with personalized FL notation where local client states remain distinct from the shared server\-side model\[arivazhagan2019federated,collins2021exploiting\]\. For a given input𝐱\\mathbf\{x\}, the forward pass of a decomposed self\-attention layer is expressed as:

Attn​\(𝐱\)=Softmax​\(\(Wq\+Δ​WqG\)​𝐱​\(\(Wk\+Δ​WkG\)​𝐱\)⊤dk\)×\(Wv\+Δ​WvL\)​𝐱\\displaystyle\\begin\{aligned\} \\text\{Attn\}\(\\mathbf\{x\}\)&=\\text\{Softmax\}\\left\(\\frac\{\(W\_\{q\}\+\\Delta W\_\{q\}^\{G\}\)\\mathbf\{x\}\(\(W\_\{k\}\+\\Delta W\_\{k\}^\{G\}\)\\mathbf\{x\}\)^\{\\top\}\}\{\\sqrt\{d\_\{k\}\}\}\\right\)\\\\ &\\quad\\times\(W\_\{v\}\+\\Delta W\_\{v\}^\{L\}\)\\mathbf\{x\}\\end\{aligned\}

\(10\)whereΔ​Wm=Bm​Am\\Delta W\_\{m\}=B\_\{m\}A\_\{m\}represents the low\-rank update for modulemm\. Given the resulting partition, the local model for clientkkis represented as the combinationf​\(⋅;ΘG,ΘkL\)f\(\\cdot;\\Theta^\{G\},\\Theta\_\{k\}^\{L\}\)\. During the training process, onlyΘG\\Theta^\{G\}is subject to federated aggregation, whileΘkL\\Theta\_\{k\}^\{L\}remains resident on the client device to preserve personalized features\. Moreover, the rankrrfor both global and local adapters is kept consistent to maintain architectural symmetry\. Consequently, the total number of trainable parameters per client remains identical to standard LoRA\-based federated learning, ensuring no additional memory overhead during local training\.

### IV\-COrthogonality\-Driven Knowledge Disentanglement

Effective knowledge disentanglement requires that the global and local components learn non\-redundant features\. To enforce this separation,PFAdapterincorporates an orthogonality regularization term into the local optimization objective\. LetWl,m=Bl,m​Al,mW\_\{l,m\}=B\_\{l,m\}A\_\{l,m\}denote the effective weight update for layerlland modulemm\. The orthogonality lossℒo\\mathcal\{L\}\_\{o\}is formulated using the Frobenius norm of the product between global and local weight matrices:

ℒo=∑l=1L∑mg∈\{q,k\}∑ml∈\{v,o\}‖Wl,mg⊤​Wl,ml‖F2\.\\mathcal\{L\}\_\{o\}=\\sum\_\{l=1\}^\{L\}\\sum\_\{m\_\{g\}\\in\\\{q,k\\\}\}\\sum\_\{m\_\{l\}\\in\\\{v,o\\\}\}\\\|W\_\{l,m\_\{g\}\}^\{\\top\}W\_\{l,m\_\{l\}\}\\\|\_\{F\}^\{2\}\.\(11\)Minimizing this term encourages the column spaces of global and local adapters to be orthogonal, thereby preventing the local modules from re\-learning information already captured by the global components\. The Frobenius norm∥⋅∥F\\\|\\cdot\\\|\_\{F\}for a matrixA∈ℝm×nA\\in\\mathbb\{R\}^\{m\\times n\}is defined as:

‖A‖F=∑i=1m∑j=1nai​j2\.\\\|A\\\|\_\{F\}=\\sqrt\{\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^\{n\}a\_\{ij\}^\{2\}\}\.\(12\)Differentiability of the Frobenius norm allows for efficient gradient\-based optimization\. During local backpropagation, the gradients of the orthogonality loss with respect to the global and local parameters are computed as:

∇ΘGℒo=2​∑l,mg,mlWl,ml​Wl,ml⊤​Wl,mg,\\nabla\_\{\\Theta^\{G\}\}\\mathcal\{L\}\_\{o\}=2\\sum\_\{l,m\_\{g\},m\_\{l\}\}W\_\{l,m\_\{l\}\}W\_\{l,m\_\{l\}\}^\{\\top\}W\_\{l,m\_\{g\}\},\(13\)∇ΘLℒo=2​∑l,mg,mlWl,mg​Wl,mg⊤​Wl,ml\.\\nabla\_\{\\Theta^\{L\}\}\\mathcal\{L\}\_\{o\}=2\\sum\_\{l,m\_\{g\},m\_\{l\}\}W\_\{l,m\_\{g\}\}W\_\{l,m\_\{g\}\}^\{\\top\}W\_\{l,m\_\{l\}\}\.\(14\)Consequently, the total local loss function for clientkkbecomes:

ℒt​o​t​a​l=ℒt​\(𝒟k;ΘG,ΘkL\)\+λo​ℒo,\\mathcal\{L\}\_\{total\}=\\mathcal\{L\}\_\{t\}\(\\mathcal\{D\}\_\{k\};\\Theta^\{G\},\\Theta\_\{k\}^\{L\}\)\+\\lambda\_\{o\}\\mathcal\{L\}\_\{o\},\(15\)whereλo\\lambda\_\{o\}is a hyperparameter controlling the strength of the disentanglement constraint\. Regularization via orthogonality ensures that the local adapters focus exclusively on domain\-specific nuances that cannot be captured by the global model\. Moreover, the orthogonality constraint facilitates more stable federated aggregation by reducing the variance of local updates in the global parameter space\. This regularizer also sharpens the global\-local interpretation behind the Q/K versus V/O split: when query/key updates already explain a shared attention relation, the penalty discourages value/output adapters from redundantly encoding the same direction, forcing them to capture residual client\-specific semantics instead\[wu2023orthogonal\]\. In gradient terms,[Eqs\.˜13](https://arxiv.org/html/2607.12111#S4.E13)and[14](https://arxiv.org/html/2607.12111#S4.E14)project each update away from directions already occupied by its counterpart, thereby reducing the cosine overlap between global\-shared and local\-private descent steps before aggregation\. As a result, the selective aggregation step operates on a subspace whose cross\-client bias is controlled, which is the quantity explicitly bounded in our convergence analysis below\.

### IV\-DSelective Aggregation and Communication Efficiency

Selective aggregation protocols inPFAdaptersignificantly reduce communication overhead while maintaining high performance\. In each roundtt, the server only collects and averages the global\-shared parametersΘkG,t\+1\\Theta\_\{k\}^\{G,t\+1\}from the participating clients\. The aggregation rule is defined as:

ΘG,t\+1=∑k∈𝒮tnk∑j∈𝒮tnj​ΘkG,t\+1\.\\Theta^\{G,t\+1\}=\\sum\_\{k\\in\\mathcal\{S\}\_\{t\}\}\\frac\{n\_\{k\}\}\{\\sum\_\{j\\in\\mathcal\{S\}\_\{t\}\}n\_\{j\}\}\\Theta\_\{k\}^\{G,t\+1\}\.\(16\)Meanwhile, the local parametersΘkL\\Theta\_\{k\}^\{L\}are updated locally and never transmitted to the server\. LetCt​o​t​a​lC\_\{total\}denote the communication cost of standard federated LoRA tuning, where all adapter parameters are synchronized\. The communication cost ofPFAdapter, denoted asCP​F​e​dC\_\{PFed\}, is given by:

Pall=∑l=1L∑m∈\{q,k,v,o\}r​\(dl,min\+dl,mout\),PPF=∑l=1L∑m∈\{q,k\}r​\(dl,min\+dl,mout\),CP​F​e​d\(round\)=2​b​PPF=PPFPall​Ct​o​t​a​l\(round\)≈24​Ct​o​t​a​l\(round\),\\displaystyle\\begin\{aligned\} P\_\{\\mathrm\{all\}\}&=\\sum\_\{l=1\}^\{L\}\\sum\_\{m\\in\\\{q,k,v,o\\\}\}r\(d\_\{l,m\}^\{\\mathrm\{in\}\}\+d\_\{l,m\}^\{\\mathrm\{out\}\}\),\\\\ P\_\{\\mathrm\{PF\}\}&=\\sum\_\{l=1\}^\{L\}\\sum\_\{m\\in\\\{q,k\\\}\}r\(d\_\{l,m\}^\{\\mathrm\{in\}\}\+d\_\{l,m\}^\{\\mathrm\{out\}\}\),\\\\ C\_\{PFed\}^\{\(\\mathrm\{round\}\)\}&=2bP\_\{\\mathrm\{PF\}\}=\\frac\{P\_\{\\mathrm\{PF\}\}\}\{P\_\{\\mathrm\{all\}\}\}C\_\{total\}^\{\(\\mathrm\{round\}\)\}\\approx\\frac\{2\}\{4\}C\_\{total\}^\{\(\\mathrm\{round\}\)\},\\end\{aligned\}

\(17\)Hererris the LoRA rank\[hu2021lora\],dl,mind\_\{l,m\}^\{\\mathrm\{in\}\}anddl,moutd\_\{l,m\}^\{\\mathrm\{out\}\}are the input/output dimensions of projectionmmat layerll, andbbis the transmitted bytes per parameter\. Because the four self\-attention projections in the deployed MLLM use the same LoRA rank and matched hidden dimensions, synchronizing only\{q,k\}\\\{q,k\\\}yields the exact parameter\-count ratioPPF/Pall=2/4=0\.5P\_\{\\mathrm\{PF\}\}/P\_\{\\mathrm\{all\}\}=2/4=0\.5\. The measured traffic in our implementation is therefore reduced from 617 MB/round to 315 MB/round, corresponding to 30\.85 GB versus 15\.75 GB over 50 rounds, with the small deviation from an ideal 50\.0% explained by serialization and packet rounding overhead\. Furthermore, the preservation ofΘkL\\Theta\_\{k\}^\{L\}ensures that the model retains its personalized expertise across communication rounds, mitigating the negative effects of weight washing\. Given the massive scale of MLLM backbones, such reductions in communication traffic are critical for deployment in resource\-constrained environments\. Moreover, the selective aggregation strategy prevents the global model from being corrupted by highly specialized local features that do not generalize across the client population\.

### IV\-EAlgorithm Description

The complete training procedure forPFAdapteris detailed in[Alg\.˜1](https://arxiv.org/html/2607.12111#algorithm1)\. Initial steps involve the initialization of global parametersΘG,0\\Theta^\{G,0\}and local parametersΘkL,0\\Theta\_\{k\}^\{L,0\}for each client\. In each communication round, selected clients perform local updates using the combined loss function defined in[Eq\.˜15](https://arxiv.org/html/2607.12111#S4.E15)\. Following local training, only the global components are synchronized\. Iterative synchronization continues until convergence or for a fixed number of roundsTT\. The algorithm ensures that local expertise is preserved while global knowledge is shared efficiently\. For consistency with[Eqs\.˜6](https://arxiv.org/html/2607.12111#S4.E6)and[7](https://arxiv.org/html/2607.12111#S4.E7), we useΘG\\Theta^\{G\}for the server\-synchronized query/key adapters andΘkL\\Theta\_\{k\}^\{L\}for the client\-specific value/output adapters throughout the pseudocode\. A client\-side gradient step therefore takes the form:

\(ΘkG,t\+1,ΘkL,t\+1\)\\displaystyle\(\\Theta\_\{k\}^\{G,t\+1\},\\Theta\_\{k\}^\{L,t\+1\}\)=\(ΘkG,t,ΘkL,t\)\\displaystyle=\(\\Theta\_\{k\}^\{G,t\},\\Theta\_\{k\}^\{L,t\}\)\(18\)−η​∇\(ΘG,ΘkL\)ℒt​o​t​a​l​\(ΘkG,t,ΘkL,t\)\.\\displaystyle\\quad\-\\eta\\nabla\_\{\(\\Theta^\{G\},\\Theta\_\{k\}^\{L\}\)\}\\mathcal\{L\}\_\{total\}\(\\Theta\_\{k\}^\{G,t\},\\Theta\_\{k\}^\{L,t\}\)\.For cold\-start personalization, a newly joined clientknewk\_\{\\mathrm\{new\}\}does not participate in the federated rounds used to learnΘG,T\\Theta^\{G,T\}\. After server\-side training finishes, the final global query/key adaptersΘG,T\\Theta^\{G,T\}are broadcast toknewk\_\{\\mathrm\{new\}\}, while the value/output adapters remain client\-private and are initialized locally as in the standard training phase\. The client is then evaluated at step 0 \(zero\-shot transfer with no local updates\) and after a small number of local adaptation rounds using only its private data, matching the protocol visualized inLABEL:supp:fig:supp\_sensitivity\(d\) and the personalization setting considered in federated adaptation work\[yi2024pfedlora,xu2025fedmllm\]\.[Alg\.˜1](https://arxiv.org/html/2607.12111#algorithm1)also has four stages explicitly: server initialization, client\-side local update, upload of only the global branch, and server aggregation\. As a result, the synchronization boundary becomes visually explicit and the earlier ambiguity about whether private value/output adapters are ever transmitted is removed, matching the communication\-accounting motivation of LoRA\-based federated tuning\[wang2024flora\]\.

Input::Local datasets

\{𝒟k\}k=1K\\\{\\mathcal\{D\}\_\{k\}\\\}\_\{k=1\}^\{K\}, rounds

TT, epochs

EE, rate

η\\eta, weight

λo\\lambda\_\{o\}
Output::Personalized parameters

\{ΘkL\}k=1K\\\{\\Theta\_\{k\}^\{L\}\\\}\_\{k=1\}^\{K\}and global

ΘG\\Theta^\{G\}
Server initialization:initialize shared query/key adapters

ΘG,0\\Theta^\{G,0\}and each client’s private value/output adapters

ΘkL,0\\Theta\_\{k\}^\{L,0\}
for*roundt=0,1,…,T−1t=0,1,\\dots,T\-1*do

Server broadcast:select participating clients

𝒮t\\mathcal\{S\}\_\{t\}and transmit

ΘG,t\\Theta^\{G,t\}to all

k∈𝒮tk\\in\\mathcal\{S\}\_\{t\}
for*each clientk∈𝒮tk\\in\\mathcal\{S\}\_\{t\}in parallel*do

Clientkklocal update:set

ΘkG,t,0←ΘG,t\\Theta\_\{k\}^\{G,t,0\}\\leftarrow\\Theta^\{G,t\}and keep

ΘkL,t,0\\Theta\_\{k\}^\{L,t,0\}private on device

for*epoche=1,…,Ee=1,\\dots,E*do

Sample batch

ℬ∼𝒟k\\mathcal\{B\}\\sim\\mathcal\{D\}\_\{k\}
Compute

ℒt​\(ℬ;ΘkG,t,e−1,ΘkL,t,e−1\)\\mathcal\{L\}\_\{t\}\(\\mathcal\{B\};\\Theta\_\{k\}^\{G,t,e\-1\},\\Theta\_\{k\}^\{L,t,e\-1\}\)
Compute

ℒo\\mathcal\{L\}\_\{o\}via[Eq\.˜11](https://arxiv.org/html/2607.12111#S4.E11)

ΘkG,t,e←ΘkG,t,e−1−η​∇ΘG\(ℒt\+λo​ℒo\)\\Theta\_\{k\}^\{G,t,e\}\\leftarrow\\Theta\_\{k\}^\{G,t,e\-1\}\-\\eta\\nabla\_\{\\Theta^\{G\}\}\(\\mathcal\{L\}\_\{t\}\+\\lambda\_\{o\}\\mathcal\{L\}\_\{o\}\)
ΘkL,t,e←ΘkL,t,e−1−η​∇ΘL\(ℒt\+λo​ℒo\)\\Theta\_\{k\}^\{L,t,e\}\\leftarrow\\Theta\_\{k\}^\{L,t,e\-1\}\-\\eta\\nabla\_\{\\Theta^\{L\}\}\(\\mathcal\{L\}\_\{t\}\+\\lambda\_\{o\}\\mathcal\{L\}\_\{o\}\)
end for

ΘkG,t\+1←ΘkG,t,E\\Theta\_\{k\}^\{G,t\+1\}\\leftarrow\\Theta\_\{k\}^\{G,t,E\}
ΘkL,t\+1←ΘkL,t,E\\Theta\_\{k\}^\{L,t\+1\}\\leftarrow\\Theta\_\{k\}^\{L,t,E\}
Upload:client

kksends only

ΘkG,t\+1\\Theta\_\{k\}^\{G,t\+1\}to the server;

ΘkL,t\+1\\Theta\_\{k\}^\{L,t\+1\}is never uploaded

end for

Server aggregation:update

ΘG,t\+1←∑k∈𝒮tnk∑j∈𝒮tnj​ΘkG,t\+1\\Theta^\{G,t\+1\}\\leftarrow\\sum\_\{k\\in\\mathcal\{S\}\_\{t\}\}\\frac\{n\_\{k\}\}\{\\sum\_\{j\\in\\mathcal\{S\}\_\{t\}\}n\_\{j\}\}\\Theta\_\{k\}^\{G,t\+1\}

end for

Termination:return the final shared

ΘG,T\\Theta^\{G,T\}and personalized

\{ΘkL,T\}k=1K\\\{\\Theta\_\{k\}^\{L,T\}\\\}\_\{k=1\}^\{K\}

Algorithm 1PFAdapterTraining Protocol
### IV\-FTheoretical Analysis

Convergence analysis ofPFAdaptercan be established under standard assumptions of smoothness and bounded variance\. LetF​\(ΘG,\{ΘkL\}\)F\(\\Theta^\{G\},\\\{\\Theta\_\{k\}^\{L\}\\\}\)denote the global objective function\. Given that the orthogonality regularization is a smooth function of the parameters, the local updates follow a descent direction for the regularized objective\. Furthermore, the selective aggregation ofΘG\\Theta^\{G\}can be viewed as a block\-coordinate descent step in the parameter space\. We make the non\-IID setting explicit through four assumptions: \(A1\) each local objectiveFkF\_\{k\}isLFL\_\{F\}\-smooth; \(A2\) stochastic gradients satisfy𝔼​‖gkt−∇Fk‖2≤σ2\\mathbb\{E\}\\\|g\_\{k\}^\{t\}\-\\nabla F\_\{k\}\\\|^\{2\}\\leq\\sigma^\{2\}and‖∇Fk‖≤G\\\|\\nabla F\_\{k\}\\\|\\leq G; \(A3\) client heterogeneity is bounded by1K​∑k=1K‖∇Fk​\(ΘG,ΘkL\)−∇F​\(ΘG,\{ΘjL\}\)‖2≤δ2\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\\|\\nabla F\_\{k\}\(\\Theta^\{G\},\\Theta\_\{k\}^\{L\}\)\-\\nabla F\(\\Theta^\{G\},\\\{\\Theta\_\{j\}^\{L\}\\\}\)\\\|^\{2\}\\leq\\delta^\{2\}; and \(A4\) the selective\-aggregation bias and orthogonality gradient are bounded as‖bt‖≤β\\\|b\_\{t\}\\\|\\leq\\betaand‖∇ℒo‖≤Ho\\\|\\nabla\\mathcal\{L\}\_\{o\}\\\|\\leq H\_\{o\}\. Assumption \(A3\) does not require IID data; it only requires the cross\-client drift induced by non\-IID partitions to remain bounded, which is the regime probed by the Dirichlet\-α\\alphaexperiments in[Secs\.˜V\-A](https://arxiv.org/html/2607.12111#S5.SS1)and[V\-B](https://arxiv.org/html/2607.12111#S5.SS2)\[li2020federated,hsu2019measuring\]\. Theoretical results indicate that the framework achieves a convergence rate of𝒪​\(1/T\)\\mathcal\{O\}\(1/\\sqrt\{T\}\)for non\-convex objectives, matching the performance of standard federated learning while providing superior personalization guarantees\.

###### Theorem 1\(Convergence ofPFAdapter\)\.

Assume \(A1\)–\(A4\) above and chooseηt=c/T\\eta\_\{t\}=c/\\sqrt\{T\}with0<c≤1/LF0<c\\leq 1/L\_\{F\}\. For the iterates generated by[Alg\.˜1](https://arxiv.org/html/2607.12111#algorithm1), the averaged stationarity measure satisfies:

1T​∑t=0T−1𝔼​\[‖∇F​\(ΘG,t,\{ΘkL,t\}\)‖2\]\\displaystyle\\frac\{1\}\{T\}\\sum\_\{t=0\}^\{T\-1\}\\mathbb\{E\}\\\!\\left\[\\left\\\|\\nabla F\(\\Theta^\{G,t\},\\\{\\Theta\_\{k\}^\{L,t\}\\\}\)\\right\\\|^\{2\}\\right\]≤C1T\+C2​δ2\+C3​σ2\|𝒮t\|\\displaystyle\\leq\\frac\{C\_\{1\}\}\{\\sqrt\{T\}\}\+C\_\{2\}\\delta^\{2\}\+C\_\{3\}\\frac\{\\sigma^\{2\}\}\{\|\\mathcal\{S\}\_\{t\}\|\}\(19\)\+C4​λo2​Ho2\+C5​β2,\\displaystyle\\quad\+C\_\{4\}\\lambda\_\{o\}^\{2\}H\_\{o\}^\{2\}\+C\_\{5\}\\beta^\{2\},whereC1=2​\(F0−F⋆\)/cC\_\{1\}=2\(F^\{0\}\-F^\{\\star\}\)/c,C2=c​LFC\_\{2\}=cL\_\{F\},C3=c​LFC\_\{3\}=cL\_\{F\},C4=2C\_\{4\}=2, andC5=2C\_\{5\}=2\. Consequently,min0≤t<T⁡𝔼​\[‖∇F​\(ΘG,t,\{ΘkL,t\}\)‖2\]=𝒪​\(T−1/2\)\\min\_\{0\\leq t<T\}\\mathbb\{E\}\[\\\|\\nabla F\(\\Theta^\{G,t\},\\\{\\Theta\_\{k\}^\{L,t\}\\\}\)\\\|^\{2\}\]=\\mathcal\{O\}\(T^\{\-1/2\}\)whenever the non\-IID driftδ2\\delta^\{2\}, orthogonality\-gradient magnitudeHoH\_\{o\}, and selective\-aggregation biasβ\\betaremain bounded\.

A detailed proof sketch, including the descent inequality and telescoping derivation, is provided in the supplementary material,LABEL:supp:sec:supp\_theoryandLABEL:supp:sec:supp\_revision\_1\. The assumptions and theorem statement are placed next to the method definition so that the convergence guarantee remains visible where the selective\-aggregation mechanism is introduced, following standard non\-IID FL analyses that separate stochastic variance from client\-drift terms\[li2020federated,hsu2019measuring\]\.

## VExperiments

### V\-AExperimental Setup

Datasets and evaluation protocols\.We evaluatePFAdapteron four diverse multimodal benchmarks to assess its effectiveness across medical imaging and social media domains\. VQA\-RAD\[lau2018dataset\]comprises 3,515 question\-answer pairs across 315 radiology images, emphasizing specialized clinical reasoning and medical domain knowledge\. SLAKE\[liu2021slake\]provides 14,028 samples with 642 images in a bilingual medical VQA setting, offering more complex semantic structures for evaluation\. Hateful Memes\[kiela2020hateful\]contains 10,000 multimodal entries requiring joint text\-image processing for hate speech detection in social media contexts\. CrisisMMD\[alam2018crisismmd\]consists of 16,080 image\-text pairs from disaster scenarios, categorized into humanitarian assistance tasks including damage severity assessment and resource needs identification\. Performance metrics included Accuracy and F1\-score for all datasets, with Area Under the ROC Curve \(AUC\) additionally reported for the binary classification task on Hateful Memes\. Weighted F1\-scores were employed to account for class imbalance inherent in medical datasets\. All experiments followed the Aligned modal scenario with Dirichlet concentration parameterα=0\.5\\alpha=0\.5to simulate moderate data heterogeneity, matching the evaluation protocol established in prior federated MLLM work\[xu2025fedmllm\]\. Detailed preprocessing, partition reuse, and local\-test evaluation protocol notes are provided in the supplementary material,LABEL:supp:sec:supp\_revision\_2, for VQA\-RAD\[lau2018dataset\], SLAKE\[liu2021slake\], Hateful Memes\[kiela2020hateful\], and CrisisMMD\[alam2018crisismmd\]\.

Baseline methods and implementation configuration\.To benchmarkPFAdapteragainst established federated optimization strategies, we select five representative methods spanning adaptive learning rates and momentum\-based aggregation\. Zero\-shot performance of the pre\-trained base model served as the lower bound, while Local\-only training provided an upper bound for client\-specific personalization without any knowledge sharing\.FedYogi\[xu2025fedmllm\]implemented adaptive moment\-based federated averaging with per\-coordinate learning rates, representing the strongest baseline in prior work\.FedAdam\[reddi2020adaptive\]employed the Adam optimizer\[kingma2015adam\]with server\-side momentum accumulation, whileFedAvgM\[hsu2019measuring\]combined momentum acceleration with standard FedAvg updates\[mcmahan2017communicationefficient\]\.FedAdagrad\[reddi2020adaptive\]utilized Adagrad’s adaptive learning rate strategy for federated optimization\. The base architecture employedMiniCPM\-V\-2\_6\-int4, a quantized multimodal large language model with Qwen2 backbone\. LoRA\[hu2021lora\]was applied to self\-attention projection matrices with rankr=8r=8and scaling factorαLoRA=16\\alpha\_\{\\text\{LoRA\}\}=16\. Local training utilizedAdamWoptimizer with learning rate2×10−52\\times 10^\{\-5\}and cosine annealing over 50 communication rounds\. Each client executed one local epoch with batch size 1 and gradient accumulation steps of 16\. Federated training sampled 2 clients per round from a total population of 10\. Orthogonality regularization weight was set toλo=0\.1\\lambda\_\{\\text\{o\}\}=0\.1based on validation performance\. Experiments were conducted on a single Nvidia L60 GPU with 48GB VRAM, utilizing 8\-bit quantization and gradient checkpointing for memory efficiency\.

### V\-BPerformance Evaluation

Main results on aligned modal scenario\.Quantitative comparisons betweenPFAdapterand state\-of\-the\-art federated learning baselines are presented in[Table˜I](https://arxiv.org/html/2607.12111#S5.T1)\. The proposed method consistently achieves superior performance across all four multimodal datasets while simultaneously reducing communication overhead by nearly 50%\. On the medical VQA\-RAD dataset,PFAdapterattains 62\.83% overall accuracy, outperforming the strongest baselineFedYogiby 2\.30% and demonstrating the effectiveness of hierarchical decomposition for clinical reasoning tasks\. Performance gains are more pronounced on SLAKE, wherePFAdapterachieves 60\.08% accuracy compared to 58\.67% forFedYogi, representing a 1\.41% improvement\. For social media multimodal classification,PFAdapterobtains 75\.63% AUC on Hateful Memes, surpassingFedYogi\(72\.48%\) by 3\.15%, and achieves 62\.49% accuracy on CrisisMMD, outperformingFedYogi\(60\.82%\) by 1\.67%\. Communication analysis reveals thatPFAdaptertransmits only 315 MB per round compared to 617 MB for baseline methods, achieving a 48\.9% reduction in bandwidth requirements through selective aggregation of query and key projection modules only\.[Table˜I](https://arxiv.org/html/2607.12111#S5.T1)reports macro\-averaged client\-local test scores in the aligned setting together with mean±\\pmstandard deviation over three seeds for the learned federated baselines\. Larger\-client scaling, fairness, and claim\-scope details are provided in the supplementary material,LABEL:supp:sec:supp\_revision\_3andLABEL:supp:sec:supp\_revision\_4\.

TABLE I:Performance comparison across datasets on the aligned\-modal scenario \(α=0\.5\\alpha=0\.5\)\. Learned federated baselines are reported as mean±\\pmstandard deviation over three seeds, and best results are highlighted inbold\.MethodVQA\-RADSLAKEHateful MemesCrisisMMDComm\.Acc \(%\)↑\\uparrowF1 \(%\)↑\\uparrowAcc \(%\)↑\\uparrowF1 \(%\)↑\\uparrowAcc \(%\)↑\\uparrowAUC \(%\)↑\\uparrowAcc \(%\)↑\\uparrowF1 \(%\)↑\\uparrow\(MB/R\)Zero\-shot56\.9852\.464\.9561\.266\.5765\.8924\.2022\.8\-Local59\.6456\.861\.6358\.966\.3967\.1247\.3445\.2\-FedYogi\[xu2025fedmllm\]60\.53±0\.8860\.53\{\\pm\}0\.8857\.9±0\.6157\.9\{\\pm\}0\.6158\.67±0\.5558\.67\{\\pm\}0\.5555\.4±0\.3355\.4\{\\pm\}0\.3371\.41±0\.8271\.41\{\\pm\}0\.8272\.48±1\.4672\.48\{\\pm\}1\.4660\.82±0\.9560\.82\{\\pm\}0\.9558\.6±0\.5858\.6\{\\pm\}0\.58617FedAdam\[reddi2020adaptive\]60\.31±1\.4260\.31\{\\pm\}1\.4257\.5±0\.7557\.5\{\\pm\}0\.7556\.74±0\.5256\.74\{\\pm\}0\.5253\.8±0\.4153\.8\{\\pm\}0\.4172\.56±1\.2372\.56\{\\pm\}1\.2373\.24±2\.4373\.24\{\\pm\}2\.4359\.12±0\.5659\.12\{\\pm\}0\.5657\.1±0\.4157\.1\{\\pm\}0\.41617FedAvgM\[hsu2019measuring\]58\.98±1\.3558\.98\{\\pm\}1\.3555\.8±0\.6755\.8\{\\pm\}0\.6758\.47±1\.0658\.47\{\\pm\}1\.0655\.1±0\.4755\.1\{\\pm\}0\.4772\.18±1\.7172\.18\{\\pm\}1\.7172\.94±1\.6172\.94\{\\pm\}1\.6156\.87±0\.4256\.87\{\\pm\}0\.4254\.8±0\.5354\.8\{\\pm\}0\.53617FedAdagrad\[reddi2020adaptive\]60\.54±0\.5560\.54\{\\pm\}0\.5557\.8±0\.3157\.8\{\\pm\}0\.3155\.83±0\.3255\.83\{\\pm\}0\.3252\.9±0\.6052\.9\{\\pm\}0\.6073\.76±0\.8973\.76\{\\pm\}0\.8973\.34±1\.9973\.34\{\\pm\}1\.9960\.43±0\.6060\.43\{\\pm\}0\.6058\.3±0\.2358\.3\{\\pm\}0\.23617pFedLoRA\[yi2024pfedlora\]61\.74±0\.7461\.74\{\\pm\}0\.7459\.1±0\.5559\.1\{\\pm\}0\.5559\.12±0\.4359\.12\{\\pm\}0\.4356\.8±0\.3756\.8\{\\pm\}0\.3774\.38±0\.6874\.38\{\\pm\}0\.6874\.92±0\.7274\.92\{\\pm\}0\.7261\.48±0\.5861\.48\{\\pm\}0\.5859\.4±0\.4959\.4\{\\pm\}0\.49617PFAdapter62\.83±0\.62\\mathbf\{62\.83\{\\pm\}0\.62\}60\.1±0\.44\\mathbf\{60\.1\{\\pm\}0\.44\}60\.08±0\.38\\mathbf\{60\.08\{\\pm\}0\.38\}57\.6±0\.35\\mathbf\{57\.6\{\\pm\}0\.35\}75\.23±0\.51\\mathbf\{75\.23\{\\pm\}0\.51\}75\.63±0\.63\\mathbf\{75\.63\{\\pm\}0\.63\}62\.49±0\.46\\mathbf\{62\.49\{\\pm\}0\.46\}60\.3±0\.40\\mathbf\{60\.3\{\\pm\}0\.40\}315

TABLE II:Accuracy \(%\) and communication cost \(MB/R\) of ablation study on component contribution\.ConfigurationVQA\-RADSLAKEHatefulComm\.w/o Hierarchical Split60\.5358\.6772\.50617w/o Orthogonality Loss61\.358\.674\.3315w/o Selective Aggregation60\.958\.273\.8617PFAdapter\(Full\)62\.860\.175\.2315

Ablation study on component contributions\.To assess the contribution of individual components, we systematically removed each technical module and measured the resulting performance degradation\. Removing the orthogonality regularization loss results in accuracy decreases of 1\.5% on VQA\-RAD, 1\.5% on SLAKE, and 0\.9% on Hateful Memes, validating that knowledge disentanglement between global and local adapters is crucial for effective personalization\. Disabling selective aggregation while maintaining orthogonality constraints leads to performance drops of 1\.9%, 1\.9%, and 1\.4% across the three datasets, respectively, while simultaneously doubling communication overhead to 617 MB per round\. Most significantly, eliminating the hierarchical split entirely \(equivalent toFedYogi\) causes the largest degradation, with accuracy decreases of 2\.27% on VQA\-RAD, 1\.43% on SLAKE, and 2\.7% on Hateful Memes\. Experimental results confirm that: i\) explicit disentanglement prevents local adapters from redundantly learning global knowledge, ii\) selective aggregation preserves client\-specific expertise through private value and output projections, and iii\) hierarchical decomposition enables more nuanced control over knowledge sharing compared to monolithic adapter synchronization\. Detailed weight\-washing diagnostics and cross\-client attention\-map visualizations are provided in the supplementary material,LABEL:supp:sec:supp\_revision\_4\.

Decomposition strategy analysis\.Different module assignment strategies lead to varying performance outcomes depending on which projections are designated for global versus local adaptation\. Assigning query \(qpq\_\{p\}\) and key \(kpk\_\{p\}\) projections to the global set while keeping value \(vpv\_\{p\}\) and output \(opo\_\{p\}\) projections local yields the optimal configuration, achieving 62\.8% accuracy on VQA\-RAD and 60\.1% on SLAKE, as detailed in[Table˜III](https://arxiv.org/html/2607.12111#S5.T3)\. Alternative decomposition strategies result in varying degrees of performance degradation\. Specifically, assigningqpq\_\{p\}andvpv\_\{p\}to global aggregation reduces accuracy by 1\.3% on VQA\-RAD, suggesting that value projections are inherently more task\-specific and should remain personalized\. Restricting global synchronization to onlyqpq\_\{p\}leads to a more substantial 2\.6% accuracy decrease, indicating that key projections also capture essential cross\-client structural information\. Conversely, assigning three modules \(qp,kp,vpq\_\{p\},k\_\{p\},v\_\{p\}\) to the global set reduces communication less substantially and sacrifices 3\.1% accuracy, demonstrating the diminishing returns of excessive global synchronization\. The projection\-level evidence directly matches the mechanism in[Eqs\.˜8](https://arxiv.org/html/2607.12111#S4.E8)and[9](https://arxiv.org/html/2607.12111#S4.E9): removingkpk\_\{p\}from the global set degrades cross\-client attention alignment, while promotingvpv\_\{p\}to the global set erodes the local semantic capacity needed under non\-IID supervision\[vaswani2017attention,mickus2024role\]\. The best Q/K\-global and V/O\-local split therefore emerges not as a heuristic partition, but as the configuration that best preserves relation sharing and client\-specific reconstruction simultaneously\.

TABLE III:Accuracy \(%\) and communication cost \(MB/R\) of different global\-local decomposition strategies on VQA\-RAD\.GlobalLocalVQA\-RADComm\.qpq\_\{p\}onlykp,vp,opk\_\{p\},v\_\{p\},o\_\{p\}60\.2155qp,kp,vpq\_\{p\},k\_\{p\},v\_\{p\}opo\_\{p\}only59\.7469qp,vpq\_\{p\},v\_\{p\}kp,opk\_\{p\},o\_\{p\}61\.5315qp,kpq\_\{p\},k\_\{p\}vp,opv\_\{p\},o\_\{p\}62\.8315

Parameter sensitivity and robustness analysis\.Comprehensive sensitivity analysis across three critical hyperparameters reveals optimal configuration ranges and robustness characteristics\.LABEL:supp:fig:supp\_sensitivity\(a\) demonstrates that orthogonality weightλo=0\.1\\lambda\_\{\\text\{o\}\}=0\.1yields optimal performance for VQA\-RAD \(62\.8%\) and Hateful Memes \(75\.6% AUC\), while SLAKE achieves peak accuracy atλo=0\.05\\lambda\_\{\\text\{o\}\}=0\.05\(60\.1%\)\. Increasingλo\\lambda\_\{\\text\{o\}\}beyond these optimal values to 0\.5 or 1\.0 causes gradual degradation, as excessive orthogonality constraints restrict local adapters from capturing client\-specific knowledge\.LABEL:supp:fig:supp\_sensitivity\(b\) examines the trade\-off between accuracy and communication efficiency across different global\-local decomposition ratios, where the 50:50 configuration achieves optimal balance with 62\.8% accuracy at 50% communication cost\.LABEL:supp:fig:supp\_sensitivity\(c\) evaluates robustness under varying data heterogeneity levels, measured by Dirichlet parameterα\\alpharanging from 0\.1 \(extreme non\-IID\) to 5\.0 \(nearly IID\)\. Under high heterogeneity \(α=0\.1\\alpha=0\.1\),PFAdapterachieves 61\.2% accuracy compared to 57\.8% forFedYogi, representing a 3\.4% improvement that narrows to 0\.9% under low heterogeneity conditions, confirming that hierarchical decomposition provides greater benefits when client distributions diverge more significantly\.LABEL:supp:fig:supp\_sensitivity\(d\) demonstrates cold\-start adaptation capability for newly joined edge devices, whereinPFAdapterachieves 59\.8% zero\-shot accuracy compared to 56\.98% forFedYogi, and reaches 64\.8% after only 5 local tuning rounds through effective knowledge transfer from pre\-aggregated global components\. Detailed heterogeneity\-theory interpretation, cold\-start protocol, fairness statistics, rank ablation, and local\-epoch discussion are provided in the supplementary material,LABEL:supp:sec:supp\_revision\_4\. The full sensitivity visualization is provided in the supplementary material,LABEL:supp:fig:supp\_sensitivity\.

![Refer to caption](https://arxiv.org/html/2607.12111v1/x3.png)Figure 3:Convergence analysis on VQA\-RAD\. \(Left\) Accuracy vs\. communication rounds\. \(Right\) Training loss reduction\.Convergence behavior and training dynamics\.Orthogonality\-driven decomposition accelerates training convergence by reducing parameter conflicts during federated aggregation\.PFAdapterexhibits significantly faster convergence over 50 communication rounds on VQA\-RAD, reaching 61\.2% accuracy by round 20 compared to 59\.1% forFedYogiand 58\.2% forFedAvgM, as illustrated in[Fig\.˜3](https://arxiv.org/html/2607.12111#S5.F3)\(a\)\. Faster convergence can be attributed to the orthogonality constraint, which reduces parameter conflicts between local and global updates and facilitates more stable aggregation\. In particular, the gradients in[Eqs\.˜13](https://arxiv.org/html/2607.12111#S4.E13)and[14](https://arxiv.org/html/2607.12111#S4.E14)penalize overlap between the global and local update subspaces, so the server aggregates less mutually contradictory information from different clients at each round\.[Fig\.˜3](https://arxiv.org/html/2607.12111#S5.F3)\(b\) demonstrates thatPFAdapterachieves consistently lower training loss throughout the optimization process, with final loss of 0\.35 compared to 0\.82 forFedYogiand 1\.15 forFedAvgM, indicating a better\-optimized loss landscape and more efficient utilization of the parameter budget\.

Computational and communication efficiency\.Selective aggregation substantially reduces network resource requirements while maintaining computational efficiency comparable to baseline LoRA methods\. Compared to full model fine\-tuning requiring 45\.2 minutes per round and 42\.5 GB peak VRAM, LoRA\-based methods reduce training time by approximately 75% and memory usage by 63%, as summarized in[Table˜IV](https://arxiv.org/html/2607.12111#S5.T4)\.PFAdapterincurs a marginal 5% increase in training time \(10\.8 vs\. 10\.3 min/round forFedYogi\) due to the additional orthogonality loss computation, but achieves a 48\.9% reduction in total communication volume over 50 rounds \(15\.75 GB vs\. 30\.85 GB forFedYogi\)\. Communication savings are achieved by transmitting only the global\-shared query and key projection adapters \(2 of 4 LoRA modules\), while preserving client\-specific value and output projections locally\. Peak VRAM consumption of 15\.8 GB remains comparable to baseline LoRA methods, makingPFAdaptersuitable for deployment on single\-GPU systems without requiring specialized distributed computing infrastructure\.

TABLE IV:Comparison of train time \(min/round\), GPU\-hours per round, peak VRAM \(GB\), inference time, and total communication \(GB\) efficiency\.MethodTrain TimeGPU\-h/RInf\. TimePeak VRAMTotal Comm\.Full Tuning45\.20\.753125\.442\.5125\.4FedYogi\[xu2025fedmllm\]10\.30\.17214\.715\.730\.85FedAvgM\[hsu2019measuring\]10\.60\.17714\.215\.530\.85PFAdapter10\.80\.18014\.815\.815\.75

Comprehensive heterogeneity analysis\.Hierarchical decomposition demonstrates particularly strong advantages when edge devices exhibit severe data distribution mismatches\. Under high label skew \(α=0\.1\\alpha=0\.1\),PFAdapterachieves a substantial \+3\.4% improvement over FedYogi, while the margin decreases to \+0\.9% under near\-IID conditions \(α=5\.0\\alpha=5\.0\), as shown inLABEL:supp:tab:supp\_heterogeneity, thereby validating that hierarchical decomposition provides greatest benefits precisely when heterogeneity poses the most significant challenges\. Missing modal scenarios with higher missing rates \(β=50%\\beta=50\\%\) yield \+2\.9% improvement compared to \+2\.3% atβ=30%\\beta=30\\%, demonstrating that local adaptation throughvpv\_\{p\}andopo\_\{p\}parameters effectively compensates for modality\-specific distribution shifts\. Cross\-modal and hybrid scenarios maintain consistent advantages \(\+1\.57% and \+1\.26%, respectively\), confirming that the decomposition strategy generalizes across diverse heterogeneity types\. The controlled split\-policy note and the full heterogeneity table are provided in the supplementary material,LABEL:supp:tab:supp\_heterogeneity\.

Robustness across multimodal heterogeneity scenarios\.Selective parameter aggregation enablesPFAdapterto maintain consistent performance advantages across diverse challenging scenarios\. In the Missing Modal scenario where 50% of clients lack either image or text modalities,PFAdapterachieves 76\.8% AUC on Hateful Memes and 56\.5% F1 on CrisisMMD, outperforming FedYogi by 1\.68% and 2\.72%, respectively, as visualized inLABEL:supp:fig:supp\_robustness\(a\)\. Subsequently,LABEL:supp:fig:supp\_robustness\(b\) evaluates generalization when image\-dominant and text\-dominant clients coexist \(I\-5:T\-5 split\), withPFAdaptermaintaining superior performance across all three metrics\. Moreover,LABEL:supp:fig:supp\_robustness\(c\) combines aligned \(p=70%p=70\\%\) and missing modal conditions, where the local adaptation capability ofvpv\_\{p\}andopo\_\{p\}parameters proves particularly beneficial\. Finally,LABEL:supp:fig:supp\_robustness\(d\) demonstrates graceful degradation under increasing Gaussian noise levels, withPFAdaptermaintaining 54\.6% accuracy at 20% noise compared to 48\.5% for FedYogi, achieving a 6\.1% absolute advantage\. Collectively, experimental results validate that selective aggregation and hierarchical decomposition provide inherent robustness to diverse real\-world data distribution challenges\.

### V\-CDiscussion and Limitations

Superior performance ofPFAdapterstems from three interrelated factors: i\)Structural decompositionrecognizes functional heterogeneity within self\-attention mechanisms, wherein query and key projections encode cross\-attention patterns amenable to global sharing across the network, whereas value and output projections modulate edge\-specific representations; ii\)Orthogonality regularizationprevents redundant learning by enforcing local adapters to capture orthogonal directions in parameter space relative to global knowledge, leading to more efficient parameter budget utilization across distributed agents; and iii\)Selective aggregationreduces the weight washing effect common in federated MLLM fine\-tuning, wherein client\-specific adaptations become diluted through uniform parameter mixing\. Nevertheless, several limitations merit discussion regarding deployment in heterogeneous edge networks: i\) Optimal decomposition ratios may vary depending on the degree of local\-global divergence across network nodes, with 50:50 split providing the best balance for moderate heterogeneity \(α=0\.5\\alpha=0\.5\) yet potentially requiring adaptation for extreme distribution shifts; ii\) Current framework applies uniform decomposition across all transformer layers, whereas layer\-specific ratios based on sensitivity analysis could further enhance performance for edge devices with varying computational capabilities; and iii\) Although orthogonality regularization effectively disentangles knowledge, Frobenius norm constraints may not fully capture complex non\-linear dependencies between global and local parameters in highly dynamic network environments\. Future work could explore learnable decomposition ratios, layer\-adaptive splitting strategies tailored to network topology, and more sophisticated disentanglement metrics based on information\-theoretic measures suitable for agentic AI systems\.

## VIConclusion

In this paper, we presentedPFAdapter, a communication\-efficient personalized federated learning framework for deploying multimodal large language models as intelligent agents across heterogeneous edge networks\. Hierarchical LoRA decomposition was introduced to explicitly separate adapter parameters into global\-shared and local\-private components based on functional roles of self\-attention modules\. Query and key projections are assigned to global synchronization across the federated network, whereas value and output projections remain localized for edge\-specific adaptation\. Orthogonality regularization enforces effective knowledge disentanglement between network\-synchronized and edge\-retained parameters\. Selective aggregation protocols transmit only global\-shared components, reducing communication overhead by nearly 50% while preserving edge\-specific expertise\. Future research directions include exploring dynamic decomposition ratios adapted to network conditions, layer\-adaptive splitting strategies for heterogeneous edge devices, and integration with emerging 6G network to further enhance agentic AI deployment across next\-generation communication systems\.

## References