Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future Directions

# A Unified Framework, Empirical Analysis, and Future Directions
Source: [https://arxiv.org/html/2608.13844](https://arxiv.org/html/2608.13844)
, ,

## Federated Prompt Learning: A Unified Framework, Empirical Analysis, and Future DirectionsJournal:CSURVolume:1111DOI:[xxx](https://doi.org/xxx)Journal:JACMVolume:008CCS:Computing methodologies Natural language processingCCS:Computing methodologies Machine learningCCS:Security and privacy Privacy\-preserving protocolsCCS:Computer systems organization Distributed architectures

Qinglin YangOrcID:[0000\-0002\-7263\-8914](https://orcid.org/0000-0002-7263-8914)email:[yangqinglin@gzhu\.edu\.cn](mailto:[email protected])Affiliation:Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Laboratory of Industrial Control System Security; Huangpu Research School of Guangzhou University,Guangzhou,Guangdong,ChinaChen Qiuemail:[c\-qiu@ibrc\.or\.jp](mailto:[email protected])Affiliation:Department of Horticultural Science,Iwate Biotechnology Research Center,Iwate,Japan,Hongyuan Zhangemail:[b23110420@njupt\.edu\.cn](mailto:[email protected])Affiliation:Computer Science and Technology,Nanjing University of Posts and Telecommunications,Nanjing,Jiangsu,China,Pengdeng LiAffiliation:Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Laboratory of Industrial Control System Security; Huangpu Research School of Guangzhou University,Guangzhou,Guangdong,China,Yuan LiuNote:The Corresponding author\.Affiliation:Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Laboratory of Industrial Control System Security; Huangpu Research School of Guangzhou University,Guangzhou,Guangdong,ChinaandZhihong Tianemail:[\{pdli, yuanliu,tianzhihong\}@gzhu\.edu\.cn](mailto:{pdli,%20yuanliu,tianzhihong}@gzhu.edu.cn)Affiliation:Cyberspace Institute of Advanced Technology, Guangzhou University; Guangdong Key Laboratory of Industrial Control System Security; Huangpu Research School of Guangzhou University,Guangzhou,Guangdong,China

2026© acmlicensed;

###### Abstract\.

Large language models \(LLMs\) have become core components of cloud\-based intelligent services in academia and industry, yet their training and deployment are hindered by high computational costs, data centralization, and privacy concerns\. Federated learning \(FL\) offers a decentralized training paradigm that enables clients to collaboratively train a learning model without sharing raw data, making it a promising solution for privacy\-preserving LLM training and reasoning\. This paper presents a comprehensive survey of federated prompt learning \(FPL\) to review recent advances in integrating the federated learning paradigm and large language models, answering the following research questions: RQ1: The fundamental motivations, characteristics, and enabling technologies of FPL, and how it differs from conventional FL and full\-model federated fine\-tuning; RQ2: The trade\-offs FPL approaches exhibit in performance, communication efficiency, computational overhead, scalability, personalization, and heterogeneity handling; RQ3: The remaining security, privacy, robustness, and system challenges, along with key future research directions\. To this end, we systematically examine existing FPL methods across the full model lifecycle: pre\-training, fine\-tuning, and practical applications, while discussing security, privacy, and robustness issues and summarizing existing defense mechanisms\. Finally, we highlight open challenges and future directions, aiming to help readers understand how the insights drive research in FPL\.

###### Keywords:

Large Language Models, Collaborative Learning, Prompt Learning, Federated Fine\-tuning, Parameter\-Efficient Fine\-tuning, Security

## 1\.Introduction

Large language models \(LLMs\) have become the cornerstone of modern artificial intelligence, demonstrating remarkable capabilities in natural language understanding, reasoning, and generation across a wide range of domains, including healthcare, finance, software engineering, and education\(chang2024survey\)\. However, the rapid scaling of LLMs has exposed fundamental limitations of the prevailing centralized training and deployment paradigm\. For training, the reliance on massive centralized datasets raises severe privacy and regulatory concerns, while the growing scarcity of high\-quality public data and the prohibitive computational and communication costs increasingly constrain further model scaling\. These challenges motivate the exploration of new collaborative and privacy\-preserving learning paradigms for the development and adaptation of LLMs\.

FL has emerged as a promising decentralized framework that enables multiple data owners to collaboratively train machine learning models without sharing raw data\(zhang2021survey\)\. When FL meets LLMs, it raises some diverse challenges compared with conventional FL patterns\. \(i\) The distinctive significance of FL for LLMs is the stronger demand for multi\-source heterogeneous data\. \(ii\) Conventional LLMs usually rely on centralized training, in which massive amounts of data are aggregated onto a unified platform before pre\-training or fine\-tuning is conducted\. However, in real\-world scenarios, many forms of high\-value data cannot be centralized, such as internal enterprise documents, medical data, financial data, legal data, personal device data, and cross\-institutional business data\. \(iii\) In terms of model architecture, FL promotes the evolution of LLMs from a single unified model toward a structure in which shared capabilities and personalized capabilities coexist\.

Therefore, the integration of LLMs with FL faces greater technical challenges than traditional deep learning\. LLMs usually have enormous parameter scales, often reaching billions or even hundreds of billions of parameters, which is several orders of magnitude larger than conventional deep learning models \(e\.g\., GoogLeNet, AlexNet, VGG, and ResNet\)\. If the full model parameters are directly transmitted, both communication costs and storage pressure become extremely high\. Hence, the core issue is no longer merely how to aggregate models from multiple clients, but how to transmit and update only the most critical, minimal, and effective parameters under extremely large model scales\.

To this end, prompt learning and parameter\-efficient fine\-tuning \(PEFT\) have gained significant attention as lightweight alternatives to full\-model fine\-tuning\. By freezing the backbone of a pretrained foundation model and optimizing only a small number of task\-specific parameters—such as soft prompts, prefixes, or low\-rank adapters \(LoRA\)\(hu2022lora\), training cost is drastically reduced while preserving the generalization power of large models\. Importantly, the compactness and modularity of prompt\-based updates make them inherently well\-suited to federated environments\. This observation has led to a new paradigm, federated prompt learning \(FPL\), which leverages prompt learning as the primary interface between LLMs and federated optimization\.

FPL fundamentally reshapes how LLMs are trained, adapted, and deployed in distributed settings\. Instead of federating the entire model or large subsets of parameters, clients collaboratively optimize lightweight prompt or adapter modules on top of a shared frozen foundation model\. This design dramatically reduces communication costs, mitigates client\-side resource constraints, and enhances robustness to non\-IID data, while maintaining strong privacy guarantees\. In consequence, federated prompt learning has rapidly evolved from early proof\-of\-concept studies to a rich ecosystem of methods spanning prompt\-based fine\-tuning, personalization, split and off\-site architectures, multimodal learning, and real\-world applications\.

![Side-by-side workflow comparison. Conventional federated learning trains local models and sends model updates to a parameter server. Federated prompt learning keeps the foundation model fixed, trains only small prompt adapters on private client data, and aggregates prompt updates to improve the global model.](https://arxiv.org/html/2608.13844v1/FLComparison.png)Figure 1\.The Difference between conventional FL and federated prompt learning\.Side\-by\-side workflow comparison\. Conventional federated learning trains local models and sends model updates to a parameter server\. Federated prompt learning keeps the foundation model fixed, trains only small prompt adapters on private client data, and aggregates prompt updates to improve the global model\.### 1\.1\.Motivations and Contributions

Despite this rapid progress, the existing literature remains fragmented\. Prior surveys\(villalobos2024position;fan2023fate;thakur2025analyzing;yao2024survey;yang2025synergizing;yan2025federated\)have primarily focused on either federated learning in general, federated fine\-tuning of LLMs, or the security and privacy of LLMs\. Despite the rapid growth of this field, its literature remains fragmented across machine learning, distributed systems, cloud–edge computing, multimodal learning, and cybersecurity\. Existing surveys typically examine general FedLLM architectures, federated fine\-tuning, IoT\-oriented deployments, or LLM security independently\. Prompt\- and adapter\-based methods are often treated as auxiliary implementation techniques rather than as a distinct distributed optimization and service\-delivery paradigm\. Moreover, previous reviews rarely connect the complete model\-service lifecycle, including training, personalization, inference, deployment, application, and protection, or systematically compare methods in terms of computational overhead, communication cost, scalability, dependability, and privacy\.

A systematic and unified treatment that centers on federated prompt learning as a distinct paradigm, examines its role across the full LLM lifecycle, and synthesizes insights from algorithmic, system, application, and security perspectives is still lacking\. Without such a comprehensive view, it is difficult for researchers and practitioners to understand the design trade\-offs, identify open challenges, and navigate future research directions in this emerging field\. To address this gap, this paper presents a comprehensive survey of FPL for LLMs that develops a unified, lifecycle\-oriented understanding of federated prompt learning from both algorithmic and cloud\-system perspectives\. It organizes existing approaches by their technical mechanisms and deployment roles, and examines how prompt\-based adaptation interacts with cloud coordination, heterogeneous edge resources, distributed inference, multimodal applications, and adversarial environments\. Particular attention is given to performance, communication efficiency, client\-side computation and memory, scalability, personalization, privacy, robustness, and practical deployability\. It is critically important to answer the following research questions\(RQs\) comprehensively:

1. \(1\)RQ1: What are the fundamental motivations, characteristics, and enabling technologies of federated prompt learning, and how does it differ from conventional federated learning and full\-model federated fine\-tuning?
2. \(2\)RQ2: What trade\-offs do federated prompt learning approaches exhibit in performance, communication efficiency, computational overhead, scalability, personalization, and heterogeneity handling?
3. \(3\)RQ3: What security, privacy, robustness, and system challenges remain, and what future research directions are most important?

Our contributions are summarized as follows:

- •The core contribution of this survey lies in the experimental validation of representative federated prompt learning models and frameworks, through which we derive key empirical findings and practical insights\. These results are intended to provide guidance and inspiration for researchers to advance research in this emerging area\.
- •We analyze the security, privacy, and robustness challenges that arise when prompt learning meets federated optimization, and summarize existing attack models and defense mechanisms tailored to federated LLMs\.
- •We distill key lessons from current research and outline open challenges and promising future directions to guide subsequent studies in federated prompt learning\.

Through this survey, we aim to provide a structured and coherent reference for researchers and practitioners seeking to understand how prompt learning can effectively bridge large language models and federated learning, enabling scalable, efficient, and privacy\-preserving collaboration in the era of foundation models\.

### 1\.2\.Methods

This review combines a systematic literature review with taxonomy\-based comparative analysis\. Relevant studies are retrieved from databases, such as “Web of Science", “Google Scholar", and “dblp", using predefined search terms related to federated learning and prompt learning\. Explicit inclusion and exclusion criteria are applied according to publication period, document type, research relevance, and methodological completeness\. After duplicate removal, title and abstract screening, and full\-text assessment, 52 high\-quality studies were retained for subsequent analysis\. The selected studies are then systematically coded in terms of research objectives, core methods, model architectures, datasets, evaluation metrics, empirical results, and reported limitations\. Based on their underlying technical mechanisms, the reviewed approaches are organized into a structured taxonomy and compared with respect to performance, efficiency, communication cost, computational overhead, privacy protection, scalability, and application scenarios\. Finally, the review synthesizes the major advances, unresolved challenges, and potential directions for future research\.

## 2\.Related Work and Background

### 2\.1\.Related work

Recent research has increasingly investigated the integration of LLMs and FL as a response to the escalating challenges of data scarcity, privacy preservation, communication cost, and the prohibitive cost of centralized model training\. Nikolaou*et al\.*\(nikolaou2025language\)establish a fundamental theoretical property of decoder\-only Transformer language models, showing that they are almost surely injective and hence exactly invertible with respect to their input prompts\. Contrary to the prevailing intuition that nonlinearities, normalization, and attention mechanisms inevitably lead to information loss, the authors prove that distinct input prompts map to distinct hidden representations with probability one under standard architectures, continuous parameter initialization, and finite\-step gradient\-based training\. Leveraging tools from real analysis, they show that collisions can only occur on measure\-zero parameter sets and that common training procedures cannot reach such pathological configurations\. Beyond theory, Nikolaou*et al\.*introduce SIPIT, the first provably correct and efficient algorithm that reconstructs the exact input prompt from internal hidden states in linear time, exploiting the causal structure of Transformers\.

Cheng et al\.\(cheng2024towards\)provide a comprehensive, structured overview of federated large language models \(FedLLMs\), elucidating motivations, methodologies, and future directions at the intersection of FL and LLMs\. They highlight FedLLMs as a promising paradigm for addressing fundamental LLM challenges, including exhaustion of high\-quality public data, stringent privacy requirements, continuous model updating, and prohibitive computational and communication costs\. By systematically reviewing the full FedLLM lifecycle, from pre\-training and fine\-tuning to deployment and application, the paper categorizes existing approaches such as parameter\-efficient fine\-tuning, prompt\-based and split learning methods, personalized federated LLMs, and backpropagation\-free techniques, analyzing their performance trade\-offs under data, system, and model heterogeneity\. The survey further identifies critical challenges, including communication bottlenecks, synchronization and straggler issues, non\-IID data, and amplified security and privacy threats such as poisoning, backdoors, and inference attacks, and summarizes current defense mechanisms and their limitations\.

For data scarcity, Villalobos et al\.\(villalobos2024position\)quantitatively demonstrate that the supply of high\-quality public human\-generated text is finite and likely to be exhausted by frontier LLMs between 2026 and 2032 under current scaling trends\. This finding underscores a fundamental limitation of centralized data\-driven scaling and highlights the necessity of leveraging decentralized, privately held data sources, thereby motivating privacy\-preserving collaborative learning paradigms \(e\.g\., FL\)\.

On the systems side, Fan*et al\.*\(fan2023fate\)propose FATE\-LLM, an industrial\-grade FedLLM framework that shows how parameter\-efficient fine\-tuning \(PEFT\), off\-site tuning, and knowledge distillation can substantially reduce communication overhead while approaching centralized fine\-tuning performance\. These core techniques of the work demonstrate the practical feasibility of federated LLM training in enterprise environments, but primarily focus on system implementation rather than a unified methodological or conceptual abstraction\. Chen*et al\.*\(chen2024integration\)systematically review the emerging integration of LLMs and federated learning, highlighting their complementarity in addressing data scarcity, privacy constraints, and scalability\. It organizes prior work into a unified framework spanning: \(i\) incorporating LLM sub\-technologies, pre\-training and prompt engineering, into FL to improve convergence speed, robustness to non\-IID data, personalization, and domain generalization; \(ii\) applying FL sub\-technologies, distributed computing and privacy\-preserving mechanisms, to mitigate LLMs’ high computational cost, limited data coverage, and privacy risks; and \(iii\) holistic federated LLM \(FedLLM\) systems that jointly optimize training, instruction tuning, and alignment under decentralized settings\.

Several surveys have attempted to organize the rapidly growing literature at the intersection of LLMs and FL\. For example, Thakur et al\.\(thakur2025analyzing\)provide a broad unifying analysis of FL–LLM fusion, categorizing prior work into paradigms that either use LLM capabilities to enhance FL, apply FL to improve LLM adaptation, or jointly design holistic FedLLM systems\. Nevertheless, this review adopts a macro\-level perspective and does not systematically analyze prompt\- or adapter\-based learning as a first\-class federated optimization interface\. Yao et al\.\(yao2024survey\)focus on security and privacy issues of LLMs, offering an extensive taxonomy of risks and defenses, but largely decouple these concerns from federated training dynamics\. Yang et al\.\(yang2025synergizing\)further expand the scope by reviewing the joint integration of IoT, LLMs, and FL, emphasizing architectural synergies in edge systems, yet their analysis is application\-driven and not centered on LLM adaptation mechanisms\. At a finer granularity, Yan et al\.\(yan2025federated\)conduct a comparative study of federated fine\-tuning paradigms for LLMs, including FedLLMs, KD\-FedLLMs, and Split\-FedLLMs\. Their results reveal inherent trade\-offs between model accuracy, communication cost, and client\-side computation, illustrating that no single federated framework is universally optimal\.

Despite these substantial advances, existing studies and surveys exhibit a notable gap: the lack of a dedicated, systematic treatment of FPL as a distinct and unifying paradigm for LLM–FL integration\. Prompt\- and adapter\-based methods have emerged as the dominant practical mechanism for adapting large language models under resource, communication, and privacy constraints, yet they are often discussed only as auxiliary techniques within broader FedLLM frameworks or system implementations\.

Different from the existing surveys, our review places FPL at the center of analysis, systematically reviewing how prompt\- and adapter\-based techniques enable efficient, scalable, and privacy\-preserving collaboration of large language models under federated settings\. By organizing existing methods along the full LLM lifecycle and jointly examining algorithmic, system\-level, application\-driven, and security\-related dimensions, this survey constructs a unified framework that provides answers to the questions:Why do LLMs need FL?,How does FPL work?,What can FL and LLMs do for each other?,What kinds of challenges can FPL meet?, andWhat are the potential research directions of FPL?

### 2\.2\.Background and Fundamentals

This subsection introduces the fundamental concepts underlying federated prompt learning\. We first describe large language models and their adaptation requirements, followed by conventional federated learning, prompt learning, and parameter\-efficient fine\-tuning\. We then formalize federated prompt learning as a lightweight interface for adapting foundation models using decentralized private data\.

#### 2\.2\.1\.Large Language Models and Model Adaptation

LLMs are typically built on transformer\(han2021transformer\)architectures and pretrained on large\-scale text corpora using self\-supervised objectives\. Through large\-scale pretraining, LLMs acquire general\-purpose representations that support a broad range of downstream capabilities, including language understanding, text generation, reasoning, question answering, and instruction following\(chang2024survey\)\. These pretrained models can subsequently be adapted to specific tasks, domains, organizations, or users\.

LLMs containing billions of parameters impose substantial computation, memory, communication, and storage requirements\. In distributed settings, repeatedly transmitting full\-model parameters or gradients further creates a major communication bottleneck\. Full fine\-tuning may also require each participant to possess sufficient hardware resources to store and optimize the complete model\. The adaptation of LLMs is further complicated by the decentralized nature of high\-value data\. Enterprise documents, medical records, financial information, source code, personal\-device data, and institution\-specific knowledge are frequently distributed across independent data owners\. Privacy requirements, regulatory restrictions, and commercial confidentiality often prevent these data from being transferred to a centralized training platform\. These requirements motivate the combination of federated learning with lightweight model\-adaptation techniques\.

#### 2\.2\.2\.Conventional Federated Learning

As illustrated in Fig\.[1](https://arxiv.org/html/2608.13844#acmlabel1), a typical FL system consists of a central server and a set of participating clients\. Each client maintains a local dataset and performs model optimization within its own trusted environment\. The server coordinates training by distributing model parameters, collecting local updates, and aggregating them into a global model\. A conventional FL round generally contains the following three steps: global model broadcast, local model training and transmission, and model aggregation\(li2020federated\)\.

LetKKdenote the number of clients,DkD\_\{k\}the private dataset of clientkk, andnk=\|Dk\|n\_\{k\}=\|D\_\{k\}\|its number of samples\. Conventional FL can be expressed as the minimization of a weighted global objective:

\(1\)min𝐰⁡F⁡\(𝐰\)=∑k=1Knk∑j=1Knj​Fk​\(𝐰\),\\min\_\{\\mathbf\{w\}\}F\(\\mathbf\{w\}\)=\\sum\_\{k=1\}^\{K\}\\frac\{n\_\{k\}\}\{\\sum\_\{j=1\}^\{K\}n\_\{j\}\}F\_\{k\}\(\\mathbf\{w\}\),wherewwrepresents the shared model parameters andFk​\(𝐰\)F\_\{k\}\(\\mathbf\{w\}\)is the local objective evaluated onDkD\_\{k\}\. In FedAvg, the server aggregates the locally optimized parameters as:

\(2\)𝐰t\+1=∑k∈Stnk∑j∈Stnj​𝐰kt\+1,\\mathbf\{w\}^\{t\+1\}=\\sum\_\{k\\in S\_\{t\}\}\\frac\{n\_\{k\}\}\{\\sum\_\{j\\in S\_\{t\}\}n\_\{j\}\}\\mathbf\{w\}\_\{k\}^\{t\+1\},whereStS\_\{t\}is the set of clients participating in communication roundtt\.

The clients in conventional FL are usually heterogeneous, including non\-independent and identically distributed data, computation capacities, storage, and communication\. This case becomes more severe when FL is directly applied to LLMs\. Transmitting and optimizing an entire LLM may exceed the computation, memory, and bandwidth available to many clients\. Therefore, federated LLM systems generally require parameter\-efficient adaptation, compression, model partitioning, or off\-site training rather than straightforward full\-model federated optimization\.

#### 2\.2\.3\.Prompt Learning

The prompt plays a role in bridging the downstream task and the knowledge encoded in the pretrained model\. Prompt learning adapts a pretrained model by modifying or learning the conditioning context while keeping most or all backbone parameters frozen\(zhou2022conditional\)\. This paper treats “prompt tuning" as a class of methods rather than a particular method\.

Prompts can generally be divided into discrete and continuous forms\. A discrete prompt consists of human\-readable tokens, task descriptions, demonstrations, or instructions inserted into the model input\. Continuous prompt learning instead represents the prompt as a set of trainable embedding vectors\.

In prompt tuning, learnable embeddings are appended or prepended to the input sequence\. Prefix tuning introduces trainable vectors into multiple transformer layers, commonly through the attention\(vaswani2017attention\)mechanism\. In vision or vision\-language models, visual prompts may be introduced into image patches or intermediate visual representations, while textual prompts condition the language branch\.

Letfθf\_\{\\theta\}denote a pretrained model with frozen parametersθ\\theta, and letppdenote a trainable prompt\. Prompt learning optimizes:

\(3\)min𝐩⁡ℒ⁡\(f𝜽​\(𝐱,𝐩\),y\),\\min\_\{\\mathbf\{p\}\}\\mathcal\{L\}\\left\(f\_\{\\boldsymbol\{\\theta\}\}\(\\mathbf\{x\};\\mathbf\{p\}\),y\\right\),where\(x,y\)\(x,y\)is a downstream training example\. Becauseθ\\thetaremains fixed and the dimension ofppis substantially smaller than that of the backbone, prompt learning reduces the computation, storage, and communication required for adaptation\.

However, prompt learning remains sensitive to the quality of the pretrained model and the distance between the pretraining and downstream distributions\.

#### 2\.2\.4\.Parameter\-Efficient Fine\-Tuning

Parameter\-efficient fine\-tuning \(PEFT\)\(fu2023effectiveness\)refers to a broader family of adaptation techniques that freeze most pretrained parameters and optimize only a small number of task\-specific parameters\. Prompt learning \([2\.2\.3](https://arxiv.org/html/2608.13844#S2.SS2.SSS3)\) is an important form of PEFT, while other representative techniques include adapters\(hu2023llm\), LoRA, prefix\-based modules\(li2021prefix\), and bias\-only tuning\(zaken2022bitfit\)\.

PEFT is particularly suitable for federated LLM adaptation\. Because only compact parameter subsets are transmitted, communication overhead can be reduced from full\-model scale to prompt\- or adapter\-module scale\. Clients can maintain a common frozen backbone while learning local adaptation modules, and the server can aggregate only the lightweight updates\.

#### 2\.2\.5\.Federated Prompt Learning \(FPL\)

FPL integrates federated optimization with prompt learning or closely related PEFT methods\(3692070\.3692451\)\. It mainly federates lightweight adaptation modules rather than the complete foundation model\. FPL can substantially reduce communication and client\-side optimization costs because only compact modules are trained and exchanged\. In FPL, the frozen foundation model preserves broadly transferable knowledge and reduces the risk of catastrophic forgetting during local adaptation\. Meanwhile, modular prompts and adapters enable the coexistence of shared and personalized capabilities\. A typical FPL training round contains four stages:

1. \(1\)The server distributes the current global prompt or adapter parameters to selected clients\.
2. \(2\)Each client attaches the received module to a shared frozen foundation model and performs local optimization using its private data\.
3. \(3\)Clients transmit only their updated prompt or adapter parameters to the server\.
4. \(4\)The server aggregates the lightweight updates and constructs a new global module\.

Letϕ\\phidenote the federated prompt or adapter parameters, whileθ\\thetadenotes the frozen backbone\. The FPL objective can be represented as:

\(4\)min⁡∑k=1Kϕ⁡nk∑j=1Knj​Fk​\(𝜽,ϕ\)\.\\min\_\{\\phi\}\\sum\_\{k=1\}^\{K\}\\frac\{n\_\{k\}\}\{\\sum\_\{j=1\}^\{K\}n\_\{j\}\}F\_\{k\}\(\\boldsymbol\{\\theta\},\\boldsymbol\{\\phi\}\)\.After local optimization, the server may perform weighted aggregation:

\(5\)ϕt\+1=∑k∈Stnk∑j∈Stnj​ϕkt\+1\.\\phi^\{t\+1\}=\\sum\_\{k\\in S\_\{t\}\}\\frac\{n\_\{k\}\}\{\\sum\_\{j\\in S\_\{t\}\}n\_\{j\}\}\\phi\_\{k\}^\{t\+1\}\.
Different from conventional FL, in which the communicated variablewwmay contain all model parameters, FPL communicates onlyϕ\\phi, whose size is generally much smaller than that of the foundation model\. For example, PROMPTFL\(guo2023promptfl\)replaces conventional full\-model training with cooperative optimization of continuous prompts over a frozen foundation model, while FedPrompt\(zhao2023fedprompt\)extends soft prompt tuning to federated pretrained language models and supports prompt tuning, P\-tuning\(liu2022p\), and prefix tuning\.

Nevertheless, FPL introduces challenges that differ from those of conventional FL\. Prompt parameters learned by heterogeneous clients may encode semantically different concepts, making position\-wise averaging ineffective\. Data heterogeneity can therefore cause prompt misalignment, client drift, and negative transfer\. A single global prompt may favor dominant clients and provide limited adaptation to minority domains\. Personalized FPL addresses this issue by separating global knowledge from client\-specific prompts or by generating customized prompts according to client updates\.

#### 2\.2\.6\.Privacy, Security, and Robustness Considerations

Although keeping raw data local is an important privacy property of FPL, lightweight parameter exchange does not eliminate security risks\. Prompt or adapter updates may still reveal information about local examples through gradient inversion, membership inference, or representation reconstruction\. A malicious client may poison its local data or manipulate its prompt updates to insert a backdoor into the aggregated module\. Conversely, a malicious or compromised server may distribute manipulated prompts, infer client properties, or exploit intermediate activations in split architectures\.

Answers to RQ1:FPL emerges from combining FL with prompt\-based tuning of large pretrained \(often vision\-language\) models, motivated primarily by the practical limitations of applying conventional FL to foundation models: high communication overhead, prohibitive on\-device compute and memory demands, non\-IID data heterogeneity across clients, and risks of overfitting or catastrophic forgetting under full\-model fine\-tuning\. Rather than exchanging full model gradients or weights, FPL keeps the pretrained backbone frozen at both server and clients, and collaboratively learns only lightweight, task\-specific prompts, such as continuous soft\-prompt embeddings, prefix vectors, or prompt\-generating networks\.Then these prompts are aggregated via standard or adapted FL protocols \(e\.g\., FedAvg, FedProx, personalized FL\)\. This design is enabled by advances in parameter\-efficient fine\-tuning \(e\.g\., prompt tuning, prefix\-tuning, P\-tuning v2\) and foundation models such as CLIP, whose strong pretrained representations can be effectively steered through prompts alone\. Different from conventional FL, FPL reduces communication and computation by orders of magnitude, supports client\-level personalization by decomposing prompts into shared and local components, and better preserves general pretrained knowledge\. Compared to full\-model federated fine\-tuning, FPL avoids destructive weight\-averaging under heterogeneous data, lowers system requirements by eliminating the need to store or transmit full gradients and optimizer states, and offers greater flexibility for supporting multiple tasks or domains without maintaining separate full model copies, positioning FPL as a parameter\-efficient, communication\-friendly paradigm for adapting foundation models in federated settings\.

## 3\.Federated Learning for Large Language Models

From the perspective of the interaction between FL and LLMs, this survey broadly categorizes the existing research into three types: \(i\) federated learning for large language models; \(ii\) large language model\-enhanced federated learning; \(iii\) the synergistic integration of federated learning and large language models\.

For the first category, federated learning is applied to address the challenges of training, adaptation, and deployment of large language models on distributed private data\. In this relationship, the large language model is the primary object of learning and optimization, while FL provides a collaborative training mechanism that keeps data local\. Based on the lifecycle of LLMs, this category can be further divided into federated pretraining, federated fine\-tuning, federated instruction tuning, federated alignment, federated prompt learning, federated model compression, and federated continual learning, among other directions\.

### 3\.1\.Larger Language Models Optimization through Federated Learning

Chen*et al*\(che2023federated\)address the fundamental difficulty of applying FL to large language models LLMs, namely the prohibitive communication and computation costs incurred when updating massive model parameters under non\-IID decentralized data\. To overcome the limitations of existing prompt\-based FL methods such as performance degradation, inefficient training, and client drift, the authors propose FedPepTAO by integrating parameter\-efficient prompt tuning with a communication\-efficient adaptive optimization strategy\. FedPepTAO lies in a layer\-importance scoring and lossless selection mechanism, which identifies and synchronizes only a subset of influential prompt layers while keeping other prompts locally updated, thereby significantly reducing communication overhead without sacrificing accuracy\. FedPepTAO is evaluated through extensive experiments on 10 NLP benchmarks using RoBERTa\-Large and multiple decoder\-based LLMs \(e\.g\., GPT\-2, LLaMA\-3B, LLaMA\-7B\)\. The results demonstrate that FedPepTAO consistently achieves state\-of\-the\-art accuracy up to 60\.8% improvement over baselines, while reducing training time by up to 97\.59%\.

Raje\(raje2024communication\)systematically investigates communication\-efficient training of LLMs in FL settings, addressing the dual bottlenecks of limited client\-side computation and expensive wireless communication that hinder practical federated LLM deployment\. Considering that fine\-tuning updates of pretrained LLMs are intrinsically low\-rank, the work integrates LoRA\-based parameter\-efficient fine\-tuning with communication\-only sparsification, proposing federated LoRA with simple sparsity \(FLoSS\)\. Unlike prior approaches that prune adapters during training, FLoSS applies unstructured top\-k sparsity exclusively during download and upload phases, while preserving dense local optimization to maintain model utility\.

To address the largely overlooked limitation of existing federated instruction tuning methods for large language models, Qin*et al\.*\(qin2025federated\)propose FedHDS\. Considering the resource\-constrained edge devices and overfitting to narrow client\-specific domains, FedHDS constructs a hierarchical selection strategy that combines cross\-layer feature fusion from multiple transformer layers with density\-based clustering, enabling both local redundancy removal and global coordination across clients in a privacy\-preserving manner\. FedHDS establishes the first systematic solution for federated data\-efficient instruction tuning and highlights the importance of redundancy\-aware data selection for scalable, efficient, and generalizable federated LLM training\.

Furthermore, FedDQC\(du2025feddqc\)tackles a fundamental yet underexplored challenge in federated instruction tuning of large language models: data quality heterogeneity across decentralized clients, which can severely degrade global model performance despite privacy preservation\. To address the lack of global visibility and the impracticality of centralized data filtering in FL, the authors\(du2025feddqc\)propose FedDQC\. It is a novel federated data quality control framework that operates entirely on the client side with minimal overhead\. The kernel of FedDQC is the instruction response alignment \(IRA\) metric, a lightweight and privacy\-preserving quality estimator that measures how well an instruction conditions its response by comparing conditional and unconditional inference losses\. Building on IRA, FedDQC introduces a quality\-aware hierarchical federated training strategy that progressively fine\-tunes the model from high\-IRA \(easy, high\-quality\) samples to lower\-IRA \(harder, noisier\) data, mirroring human curriculum learning\.

To address the data heterogeneity issue in federated prompt tuning, Chen*et al\.*\(chen2025dualfpt\)propose DualFPT, a federated visual prompt\-tuning framework that jointly pursues generalized and personalized adaptation by decomposing learnable prompts into shared global prompts and client\-specific local prompts\. Because feature shifts and class imbalance impair the transferability of globally aggregated prompts while purely personalized methods risk overfitting to client\-specific distributions\. DualFPT is compose of two mechanisms: \(i\) feature sharing \(FS\); \(ii\) prompt composition scheme \(PCS\)\. First, FS uses a variational autoencoder to disentangle classification\-sensitive features from redundant information and applies differential\-privacy noise before sharing selected features, narrowing inter\-client distribution gaps to improve global\-prompt generalization\. Second, the PCS employs a lightweight adaptive network to estimate similarity between each test instance and client distributions, dynamically combining local prompts into an instance\-specific composite prompt\. A two\-stage alternating strategy optimizes global parameters, local prompts, and the adaptive network\. Nevertheless, DualFPT requires an added feature\-extraction stage, exchange of differentially private intermediate features, and training of auxiliary VAE, classifier, and adaptive\-network components, increasing implementation complexity relative to prompt\-only aggregation\. Evaluation is also limited to image classification with a ViT\-B/16 backbone, controlled feature\- and class\-heterogeneity settings, fixed prompt configurations, and relatively short federated training schedules\.

To address the task and client resource heterogeneity in cross\-device FL, Bai*et al\.*\(bai2024federated\)propose FlexLoRA, a federated fine\-tuning framework for LLMs\. Motivated by the “bucket effect" in conventional FL where all clients are constrained by the least\-capable participant, the authors allow clients to apply heterogeneous LoRA ranks to match their local resources, enabling more powerful clients to contribute richer, less task\-specific knowledge\. FlexLoRA aggregates client updates by reconstructing full LoRA weight matrices, averaging them on the server, and applying SVD\-based decomposition to redistribute rank\-adaptive LoRA parameters back to clients, without additional hyperparameters\. Theoretical analysis links higher local ranks and larger client populations to improved generalization bounds\.

Gao*et al\.*\(gao2025federated\)address the memory overhead and training latency of federated LoRA fine\-tuning for LLMs on resource\-constrained, system\-heterogeneous edge devices by proposing FAH\-QLoRA, a framework that combines heterogeneous base\-model quantization with dynamically adjusted LoRA ranks\. Its core mechanism follows a two\-stage rank\-allocation strategy: first determining the average LoRA rank that maximizes loss\-reduction rate per unit wall\-clock time, then assigning device\-specific ranks based on heterogeneous computation and communication capabilities, allocating lower ranks to slower devices to mitigate the straggler effect\. Truncation and zero\-padding further support local training and global aggregation of heterogeneous LoRA modules\. The main contribution lies in jointly optimizing model precision and LoRA rank allocation to improve time and memory efficiency in federated LLM fine\-tuning, complemented by a convergence analysis under non\-convex, non\-IID settings\.

### 3\.2\.Larger Language Models Application via Federated Learning

FedMRG\(che2025llm\)is the first comprehensive framework for LLM\-driven medical report generation \(MRG\) under federated learning, tackling two key obstacles to multi\-center collaboration: prohibitive LLM communication costs and severe multi\-modal data heterogeneity across institutions\. To enable privacy\-preserving, scalable training, FedMRG employs LoRA to substantially reduce communication overhead\. It further addresses heterogeneity at both visual and textual levels: on the encoder side, Hierarchical Contrasting and Prompting \(HCP\) combines client\-aware contrastive learning with diagnosis\-aware prompting to capture globally consistent yet locally distinctive visual features; on the decoder side, a Dual\-adapter Mutual Boosting \(DMB\) mechanism harmonizes global reporting knowledge with client\-specific linguistic styles via bidirectional knowledge distillation\.

Otoum*et al\.*\(otoum2025llms\)present an LLM\-driven federated learning framework for scalable and secure IoT management, motivated by the latency, privacy, energy, and scalability limitations of centralized cloud\-based IoT architectures\. The authors integrate LLMs with FL in a hybrid edge–cloud architecture, enabling privacy\-preserving decision\-making directly on IoT devices while leveraging cloud resources for global coordination\. A key contribution is the gradient sensing federated strategy \(GSFS\), which adaptively regulates client participation and asynchronous update uploads based on performance shifts and gradient magnitudes, reducing redundant communication and improving convergence efficiency over classical methods, such as FedAvg and FedOpt\.

Evaluations on the IoT\-23 dataset show GSFS achieves higher accuracy and F1\-scores for both central and client models, while significantly cutting response latency \(up to 51% on the client side\) and improving energy efficiency\. Results further indicate that edge\-based LLM inference enables real\-time IoT analytics with reduced cloud dependency, while federated coordination preserves data privacy across heterogeneous devices\.

Agarwal et al\.\(agarwal2023practical\)provide a critical empirical investigation into whether FL with pretrained language models \(PLMs\) genuinely achieves domain adaptation and personalization, or whether its apparent success stems largely from pretraining\-induced regularization\. Through systematic experiments on three NLP tasks: sentiment classification \(SST\-2\), sequence tagging \(OntoNotes\), and text generation \(Gigaword\), the authors analyze key confounders in federated NLP, including the role of pretrained weights, client size imbalance, data partition strategies, and the trade\-off between server generalization and client personalization\. Results show that with PLMs, the performance gap between federated and centralized training stays small even after aggressively ablating client updates, suggesting FL often fails to meaningfully adapt to local client distributions and instead benefits from pretraining’s strong semantic priors\. The proposed personalization–generalization slope \(m​Δ​Pm\\Delta P\) further indicates that local\-data learning provides limited or neutral gains to global generalization, challenging common claims about personalization in federated NLP\. While uniform client data distributions improve convergence speed and stability, they do not fundamentally resolve the lack of true domain adaptation\.

Zhang\(zhang2025fed\)addresses the resource and memory heterogeneity encountered in federated foundation\-model fine\-tuning, where resource\-constrained clients may be unable to update all LoRA layers efficiently\. It proposes Fed\-HeLLo, a federated LoRA framework that assigns different subsets of trainable LoRA layers to clients according to their computational capabilities and layer importance\. Its core design combines Fisher information matrix–based allocation \(FIM\-HLA\) for dynamically estimating layer importance with geometrically defined and randomized allocation strategies \(GD\-HLA/RGD\-HLA\) for stabilizing early\-stage training\. The principal contribution therefore lies in jointly exploiting client resource heterogeneity and layer\-wise importance for efficient federated LoRA fine\-tuning\. Nevertheless, the framework still relies on a server\-side proxy dataset, retains non\-negligible activation\-memory requirements, lacks explicit privacy protection, and does not directly resolve data heterogeneity\.

### 3\.3\.Empowering the Reasoning Process of LLMs in Federated Settings

eFedLLM\(ding2024efedllm\)is an FL\-based framework for LLM inference, targeting the prohibitive computational, memory, and bandwidth requirements that limit LLM accessibility\. Departing from conventional data\-parallel FL, it adopts a transformer\-based model\-parallel architecture, distributing different transformer layers across heterogeneous participants to enable collaborative inference without requiring any single user to host the full model\. To ensure reliability in this chained execution setting, eFedLLM introduces a trust\-based incentive and verification mechanism that evaluates intermediate layer outputs and filters malicious or low\-quality contributors\. It further incorporates transformer\-tailored optimizations, including hierarchical memory access strategies to reduce global memory reads and SVD\-based low\-rank compression of weight matrices to lower communication bandwidth while preserving accuracy\. Analytical and numerical evaluations show these optimizations reduce memory access and bandwidth usage by up to 60% under practical compression ratios, significantly improving inference efficiency\.

Lin\(lin2024splitlora\)introduces SplitLoRA, the first split learning\-based parameter\-efficient fine\-tuning framework for LLMs, addressing the prohibitive computation and communication costs limiting federated LLM adaptation on distributed private data\. Motivated by the scarcity of high\-quality public data and the impracticality of full\-model federated fine\-tuning, SplitLoRA combines split federated learning \(SFL\) with LoRA\-based PEFT, partitioning the LLM between clients and a central server so clients train only shallow layers with lightweight LoRA adapters while the server handles most computation\. This substantially reduces client\-side resource demands and mitigates data heterogeneity by centralizing deeper representations\. SplitLoRA provides a practical balance between performance, efficiency, and scalability\. Meanwhile, it establishes an open\-source benchmark for split LLM fine\-tuning\. However, open challenges remain in optimal model splitting, handling heterogeneous client resources, and strengthening privacy guarantees against split\-learning inference attacks, pointing to promising directions for future research\.