ShadowFPT: Backdooring Federated Prompt Tuning via Shadow Triggers

# ShadowFPT: Backdooring Federated Prompt Tuning via Shadow Triggers
Source: [https://arxiv.org/html/2610.22871](https://arxiv.org/html/2610.22871)
Teng LiAffiliation:Fudan University, Shanghai, ChinaYunhao FengAffiliation:Alibaba Group, Hangzhou, ChinaXingjun Ma††thanks:Corresponding author: xingjunma@fudan\.edu\.cnAffiliation:Fudan University, Shanghai, China

###### Abstract

Federated Prompt Tuning \(FPT\) adapts large vision–language models by freezing the pretrained backbone and optimizing only lightweight prompt parameters across clients\. Although this design improves communication and parameter efficiency, it creates an overlooked security risk\. Since the backbone is shared and frozen, a malicious client can induce backdoor behavior through the representation space while keeping its uploaded prompt updates close to benign ones\. We proposeShadowFPT, a targeted backdoor attack that exploits this frozen\-backbone attack surface\. ShadowFPT first pretrains a learnable*Shadow Trigger*against the frozen CLIP visual encoder, using either auxiliary public data or the malicious client’s local data, so that triggered inputs are steered toward the target class in representation space\. During federated prompt tuning, the malicious client adapts the trigger under the current global prompt and then optimizes its local prompt on both clean and triggered samples\. Only prompt parameters are uploaded to the server, while the trigger and frozen encoders remain local\. By shifting most of the attack burden from prompt updates to trigger\-induced representation steering, ShadowFPT achieves targeted misclassification while preserving prompt\-space stealthiness\. Across multiple datasets, aggregation rules, and non\-IID federated partitions, ShadowFPT increases the attack success rate from 19\.81% to 90\.36% in our main setting, while maintaining clean accuracy\. It remains effective across textual, visual, and joint vision–language prompt tuning\. These results identify frozen backbones as stealthy and underexplored backdoor surfaces in federated prompt tuning, suggesting that defenses based only on prompt\-update anomaly detection are insufficient\.

## 1Introduction

Large vision–language models \(VLMs\), such as CLIP[Radford et al\. \(2021\)](https://arxiv.org/html/2610.22871#bib.bib5), learn transferable image–text representations and have become a standard foundation for multimodal recognition and adaptation\. Adapting these models in distributed environments, however, remains difficult because full fine\-tuning of large pretrained backbones is computationally expensive and communication\-intensive\. Prompt tuning addresses these limitations by keeping the pretrained backbone fixed and optimizing only a small set of learnable prompt parameters[Khattak et al\. \(2023a\)](https://arxiv.org/html/2610.22871#bib.bib7)\. This parameter\-efficient formulation is particularly well suited to federated learning\. In*Federated Prompt Tuning*\(FPT\), clients update lightweight prompts using private local data and communicate only prompt parameters to the server, while the VLM backbone remains frozen\.

Backdoor attacks remain a persistent security threat in federated learning\. Classical attacks, such as BadNets[Gu et al\. \(2017\)](https://arxiv.org/html/2610.22871#bib.bib17)and Blended[Chen et al\. \(2017\)](https://arxiv.org/html/2610.22871#bib.bib18), poison training data with attacker\-designed triggers, causing the trained model to retain high clean accuracy while predicting an attacker\-specified target label on triggered inputs\. In federated learning, malicious clients can further inject backdoors through their local model updates, making attack persistence under server aggregation a central challenge\. Prior work has studied this problem in standard and personalized federated learning, including A3FL[Zhang et al\. \(2023\)](https://arxiv.org/html/2610.22871#bib.bib19)and Bad\-PFL[Fan et al\. \(2025\)](https://arxiv.org/html/2610.22871#bib.bib30)\. Backdoor risks have also been examined in vision–language models and prompt learning\. For example, BadCLIP attacks multimodal contrastive learning by guiding both visual and textual embeddings[Liang et al\. \(2024\)](https://arxiv.org/html/2610.22871#bib.bib36)\. However, backdoor attacks against federated prompt tuning remain insufficiently understood\. In particular, existing studies largely overlook the security risk that arises when the server aggregates only lightweight prompt parameters while all clients rely on a shared frozen VLM backbone\.

Most federated backdoor defenses operate on the updates communicated by clients\. Representative methods identify updates that are close to the client majority[Blanchard et al\. \(2017\)](https://arxiv.org/html/2610.22871#bib.bib14), remove coordinate\-wise outliers[Yin et al\. \(2018\)](https://arxiv.org/html/2610.22871#bib.bib22), downweight clients with highly similar update directions[Fung et al\. \(2018\)](https://arxiv.org/html/2610.22871#bib.bib37), calibrate client updates using trusted server data[Cao et al\. \(2020\)](https://arxiv.org/html/2610.22871#bib.bib26), or combine clustering, clipping, noise injection, and certification to improve robustness[Nguyen et al\. \(2022\)](https://arxiv.org/html/2610.22871#bib.bib25)\. Recent work extends this update\-centered view to federated prompt learning\. For example, SABRE\-FL detects malicious clients through anomaly analysis in embedding space[Khan et al\. \(2025\)](https://arxiv.org/html/2610.22871#bib.bib13)\. Despite their methodological differences, these defenses share a common assumption that backdoor behavior will manifest as detectable anomalies in uploaded updates or in representations derived from them\.

This assumption becomes fragile in FPT\. The server observes only low\-dimensional prompt updates, while the dominant representation geometry is determined by the shared frozen VLM backbone\. A malicious client can therefore shift much of the backdoor effect to the interaction between the trigger and the frozen representation space, while keeping its uploaded prompt updates close to benign ones\. This creates a security gap between where the backdoor behavior is realized and what the server can observe\. The same frozen backbone that enables efficient adaptation also provides a stable and underexplored attack surface for stealthy backdoor injection\.

Motivated by this observation, we proposeShadowFPT, a targeted backdoor attack for federated prompt tuning that exploits the shared frozen CLIP visual encoder as a representation\-level attack surface rather than relying on conspicuous prompt updates\. ShadowFPT consists of two stages\. First, it pretrains a learnable*Shadow Trigger*using either auxiliary public data or the malicious client’s local data\. The trigger is optimized to steer triggered images toward the target class in the frozen CLIP representation space by aligning final image embeddings with the target text anchor and intermediate visual features with target\-class layer\-wise prototypes\. Second, during federated training, the malicious client adapts the Shadow Trigger under the current global prompt and then optimizes its local prompt on both clean and triggered samples to implant the targeted backdoor\. Only prompt parameters are uploaded to the server, while the trigger and frozen encoders remain local\. By shifting much of the attack effect from uploaded prompt updates to trigger\-induced representation steering, ShadowFPT achieves targeted misclassification while keeping malicious prompt updates close to benign updates\.

We evaluate ShadowFPT across multiple datasets, aggregation rules, and non\-IID partitions\. In the main setting, ShadowFPT improves attack success by 70\.55 percentage points while largely preserving clean accuracy\. It remains effective across textual, visual, and joint vision–language prompt tuning, and prompt\-space analyses show that malicious updates stay close to benign ones\. These results reveal frozen backbones as stealthy backdoor surfaces and challenge defenses based solely on prompt\-update anomaly detection\.

The main contributions of this paper are as follows:

- •We identify a frozen\-backbone backdoor surface in FPT, showing that targeted backdoor behavior can be realized through trigger\-induced representation steering while malicious prompt updates remain close to benign updates\.
- •We proposeShadowFPT, a targeted backdoor attack that shifts the attack burden from conspicuous prompt\-space poisoning to representation\-level manipulation of the shared frozen backbone\.
- •We introduce the*Shadow Trigger*, a target\-directed trigger learned through target\-text alignment and layer\-wise visual prototype alignment, enabling triggered images to move toward the target class in the frozen representation space\.
- •We evaluate ShadowFPT across textual, visual, and joint vision–language prompt tuning under non\-IID federated settings, demonstrating high attack success, minimal clean\-accuracy degradation, and strong prompt\-space stealthiness\.

## 2Related Work

Federated Prompt Tuning\.Vision–language models \(VLMs\), such as CLIP, learn transferable image–text representations and provide strong foundations for multimodal adaptation[Radford et al\. \(2021\)](https://arxiv.org/html/2610.22871#bib.bib5)\. Prompt tuning adapts these models efficiently by freezing the pretrained backbone and optimizing only a small set of prompt parameters\. Representative CLIP\-based approaches include textual prompt tuning[Zhou et al\. \(2022b\)](https://arxiv.org/html/2610.22871#bib.bib3);[Zhou et al\. \(2022a\)](https://arxiv.org/html/2610.22871#bib.bib2), visual prompt tuning[Jia et al\. \(2022\)](https://arxiv.org/html/2610.22871#bib.bib12), joint vision–language prompt learning[Khattak et al\. \(2023a\)](https://arxiv.org/html/2610.22871#bib.bib7), and semantic regularization[Khattak et al\. \(2023b\)](https://arxiv.org/html/2610.22871#bib.bib40)\. This parameter\-efficient paradigm naturally extends to federated learning, where clients communicate prompt parameters rather than full model updates\. Recent studies on federated prompt tuning investigate cooperative prompt learning[Guo et al\. \(2023a\)](https://arxiv.org/html/2610.22871#bib.bib6), personalization[Guo et al\. \(2023b\)](https://arxiv.org/html/2610.22871#bib.bib34), text\-driven prompt generation[Qiu et al\. \(2024\)](https://arxiv.org/html/2610.22871#bib.bib4), personalization–generalization trade\-offs[Cui et al\. \(2024\)](https://arxiv.org/html/2610.22871#bib.bib39), mixture\-of\-experts prompts[Luo et al\. \(2025\)](https://arxiv.org/html/2610.22871#bib.bib35), and multimodal visual prompt tuning[Singha et al\. \(2025\)](https://arxiv.org/html/2610.22871#bib.bib15)\. These works primarily focus on communication efficiency, personalization, and generalization under heterogeneous client data\. In contrast, the security risks of federated prompt tuning, especially its vulnerability to backdoor attacks, remain insufficiently understood\.

Backdoor Attacks\.Backdoor attacks aim to make a model behave normally on clean inputs while predicting an attacker\-specified target label on inputs containing a trigger\. Classical attacks achieve this behavior by poisoning training data with visible or blended triggers[Gu et al\. \(2017\)](https://arxiv.org/html/2610.22871#bib.bib17);[Chen et al\. \(2017\)](https://arxiv.org/html/2610.22871#bib.bib18)\. In federated learning, the backdoor must also persist through local training and server aggregation, which has motivated attacks based on stealthy poisoned updates, persistent backdoor injection, adaptive optimization, collusive trigger design, and personalized federated learning[Lyu et al\. \(2023\)](https://arxiv.org/html/2610.22871#bib.bib20);[Zhang et al\. \(2022\)](https://arxiv.org/html/2610.22871#bib.bib21);[Zhang et al\. \(2023\)](https://arxiv.org/html/2610.22871#bib.bib19);[Lyu et al\. \(2024a\)](https://arxiv.org/html/2610.22871#bib.bib33);[Fan et al\. \(2025\)](https://arxiv.org/html/2610.22871#bib.bib30)\.Backdoor vulnerabilities have also been studied in vision–language models\. For example, BadCLIP manipulates multimodal contrastive representations through visual and textual embedding guidance[Liang et al\. \(2024\)](https://arxiv.org/html/2610.22871#bib.bib36)\. Existing attacks, however, are largely built around data poisoning, model\-update manipulation, or direct prompt\-parameter poisoning\. FPT presents a different regime in which both the trainable and communicated parameters are small, and local optimization is lightweight\. These constraints leave limited capacity for conventional backdoor injection through uploaded updates, motivating our investigation of a more concealed representation\-level attack surface\.

Backdoor Defenses\.Federated defenses against malicious clients have largely followed the paradigm of Byzantine\-robust aggregation\. Classical methods, such as Krum[Blanchard et al\. \(2017\)](https://arxiv.org/html/2610.22871#bib.bib14), TrimmedMean[Yin et al\. \(2018\)](https://arxiv.org/html/2610.22871#bib.bib22), and Bulyan[Guerraoui et al\. \(2018\)](https://arxiv.org/html/2610.22871#bib.bib23), mitigate malicious behavior by suppressing abnormal client updates through robust statistics and outlier filtering\. Subsequent approaches improve robustness under heterogeneous or unreliable clients using geometric aggregation, trust bootstrapping, or sparsification[Pillutla et al\. \(2022\)](https://arxiv.org/html/2610.22871#bib.bib29);[Karimireddy et al\. \(2022\)](https://arxiv.org/html/2610.22871#bib.bib28);[Cao et al\. \(2020\)](https://arxiv.org/html/2610.22871#bib.bib26);[Xu et al\. \(2025\)](https://arxiv.org/html/2610.22871#bib.bib27)\. Defenses designed specifically for federated backdoors further target poisoned global models\. FLAME combines client clustering with model clipping[Nguyen et al\. \(2022\)](https://arxiv.org/html/2610.22871#bib.bib25), while CRFL provides certified robustness against federated backdoors[Xie et al\. \(2021\)](https://arxiv.org/html/2610.22871#bib.bib24)\. Recent studies also investigate secure aggregation and detectability, privacy\-constrained robustness under heterogeneity, system\- and optimization\-level mitigation, and dynamic behavior\-based identification[Huang et al\. \(2025\)](https://arxiv.org/html/2610.22871#bib.bib31);[Zhang et al\. \(2025\)](https://arxiv.org/html/2610.22871#bib.bib32)\.

Despite their methodological differences, these defenses largely share the assumption that malicious behavior will appear as detectable abnormalities in uploaded updates or in representations derived from them\. This assumption becomes fragile in FPT\. Under non\-IID data, benign prompt updates may vary substantially across clients, while the backdoor effect can be shifted into the interaction between the trigger and the shared frozen representation space rather than being exposed through conspicuous prompt updates\. This gap motivates our investigation of frozen\-backbone representation spaces as stealthy backdoor surfaces in federated prompt tuning\.

## 3Method

### 3\.1Problem Setup and Threat Model

Federated Prompt Tuning\.We consider federated prompt tuning of a CLIP model composed of a visual encoderfIf\_\{I\}and a text encoderfTf\_\{T\}, both of which remain frozen throughout training\. The system containsNNclients, where clientiiholds a private local dataset𝒟i=\{\(xj,yj\)\}j=1ni\\mathcal\{D\}\_\{i\}=\\\{\(x\_\{j\},y\_\{j\}\)\\\}\_\{j=1\}^\{n\_\{i\}\}drawn from a potentially non\-IID distribution\. Let𝒞\\mathcal\{C\}denote the downstream label set\. For clarity, we first present the formulation for textual prompt tuning\.

Following standard CLIP prompt tuning, each classc∈𝒞c\\in\\mathcal\{C\}is represented by a prompted text input, such as “a photo of a \[CLASS\]”, where the hand\-crafted context tokens are replaced by learnable continuous tokens\. We denote the resulting prompted text feature byfT​\(c,P\)f\_\{T\}\(c;P\)\. Given an imagexxand prompt parametersPP, the prompted CLIP classifier computes the class logit as

ℓc​\(x,P\)=⟨fI​\(x\),fT​\(c,P\)⟩,c∈𝒞,\\ell\_\{c\}\(x;P\)=\\left\\langle f\_\{I\}\(x\),f\_\{T\}\(c;P\)\\right\\rangle,\\qquad c\\in\\mathcal\{C\},\(1\)wherefI​\(x\)f\_\{I\}\(x\)andfT​\(c,P\)f\_\{T\}\(c;P\)are the image and prompted text features, respectively, and⟨⋅,⋅⟩\\langle\\cdot,\\cdot\\rangledenotes cosine similarity\. We writeF⁡\(x,P\)F\(x;P\)for the resulting prompted classifier\.

At communication roundtt, the server broadcasts the current global promptPtP^\{t\}to a subset of clients𝒮t\\mathcal\{S\}\_\{t\}\. Each selected client initializes its local prompt withPtP^\{t\}and optimizes the standard classification objective:

ℒ⁡\(Pi\)=𝔼\(x,y\)∼𝒟i​\[CE⁡\(F⁡\(x,Pi\),y\)\],\\mathcal\{L\}\(P\_\{i\}\)=\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{i\}\}\\left\[\\mathrm\{CE\}\\big\(F\(x;P\_\{i\}\),y\\big\)\\right\],\(2\)whereCE⁡\(⋅,⋅\)\\mathrm\{CE\}\(\\cdot,\\cdot\)denotes the cross\-entropy loss\. After local optimization, clients upload their prompt parameters to the server, which aggregates the received prompts using an aggregation ruleAgg⁡\(⋅\)\\mathrm\{Agg\}\(\\cdot\):

Pt\+1=Agg⁡\(\{Pit\+1:i∈𝒮t\}\)\.P^\{t\+1\}=\\mathrm\{Agg\}\\left\(\\\{P\_\{i\}^\{t\+1\}:i\\in\\mathcal\{S\}\_\{t\}\\\}\\right\)\.\(3\)
Threat Model\.We consider a targeted backdoor attack against FPT with a frozen vision–language backbone\. The adversary aims to preserve the clean accuracy of the global model while causing triggered inputs,xδ=𝒜⁡\(x,δ\)x^\{\\delta\}=\\mathcal\{A\}\(x,\\delta\), to be classified as an attacker\-specified target classyty\_\{t\}\. The malicious client has access to local target\-class samples, denoted byℛyt\\mathcal\{R\}\_\{y\_\{t\}\}, which are used to guide target\-directed trigger optimization\. During local training, the malicious client can optimize both its local prompt parameters and the triggerδ\\delta, but uploads only prompt parameters to the server\. The adversary cannot modify the frozen image encoder, the frozen text encoder, the global training protocol, or the server\-side aggregation rule\. We assume a white\-box threat setting in which the malicious client knows the frozen backbone architecture\.

![Refer to caption](https://arxiv.org/html/2610.22871v1/figures/over.png)Figure 1:Overview of ShadowFPT\. \(A\) The adversary pretrains a Shadow Triggerδshadow\\delta\_\{\\mathrm\{shadow\}\}against the frozen CLIP backbone through target\-text and layer\-wise visual alignment\. \(B\) During FPT, the malicious client adapts the trigger under the current global promptPtP^\{t\}and optimizes its local prompt on clean and triggered samples, while uploading only prompt parameters\. \(C\) By shifting the attack effect to trigger\-induced representation steering, ShadowFPT achieves targeted misclassification with benign\-like prompt updates\.
### 3\.2Overview of ShadowFPT

ShadowFPT backdoors FPT through a two\-stage procedure\. It first learns a*Shadow Trigger*against the frozen CLIP representation space, and then uses this trigger to implant targeted backdoor behavior during malicious local prompt training\. Figure[1](https://arxiv.org/html/2610.22871#S3.F1)provides an overview of the ShadowFPT\.

Before federated training, the adversary optimizes a learnable triggerδ\\deltaon a pretraining dataset𝒟s\\mathcal\{D\}\_\{\\mathrm\{s\}\}, which can be either auxiliary public data or the malicious client’s local data\. The optimized trigger, denoted byδshadow\\delta\_\{\\mathrm\{shadow\}\}, is trained to steer triggered images toward the target class in the frozen CLIP representation space, as described in Section[3\.3](https://arxiv.org/html/2610.22871#S3.SS3)\.

During federated training, the malicious client initializes the trigger withδshadow\\delta\_\{\\mathrm\{shadow\}\}and adapts it under the current global prompt, allowing the trigger to remain aligned with the evolving federated prompt\. The adapted trigger is then applied to a subset of the malicious client’s local data, and the local prompt is optimized on both clean and triggered samples so that triggered inputs are classified as the target class\. After local optimization, the malicious client uploads only the prompt parameters to the server, while the trigger remains local\. This injection procedure is detailed in Section[3\.4](https://arxiv.org/html/2610.22871#S3.SS4)\.

### 3\.3Shadow Trigger Pretraining

ShadowFPT learns a*Shadow Trigger*by optimizing a learnable triggerδ\\deltaagainst the frozen CLIP visual encoder\. Let𝒟s\\mathcal\{D\}\_\{\\mathrm\{s\}\}denote the data used for trigger pretraining, which can be either auxiliary public data or the malicious client’s local data\. For an imagex∼𝒟sx\\sim\\mathcal\{D\}\_\{\\mathrm\{s\}\}, the triggered image is defined asxδ=𝒜⁡\(x,δ\)x^\{\\delta\}=\\mathcal\{A\}\(x,\\delta\)\. During this stage, only the triggerδ\\deltais updated, while both the visual encoderfIf\_\{I\}and the text encoderfTf\_\{T\}remain fixed\. The goal is to obtain a trigger that steers triggered images toward the target class in the frozen representation space\.

Target\-text alignment\.During Shadow Trigger pretraining, we use fixed hand\-crafted CLIP templates instead of the learnable federated prompt\. For each classc∈𝒞c\\in\\mathcal\{C\}, we construct a text anchor using the frozen text encoder:

zc=1\|𝒯\|​∑T∈𝒯fT​\(T⁡\(c\)\),z\_\{c\}=\\frac\{1\}\{\|\\mathcal\{T\}\|\}\\sum\_\{T\\in\\mathcal\{T\}\}f\_\{T\}\(T\(c\)\),\(4\)where𝒯\\mathcal\{T\}is a fixed set of templates, such as “a photo of a \[CLASS\]”\. Letztz\_\{t\}denote the text anchor of the target classyty\_\{t\}\. We optimize the trigger so that the image feature of each triggered input is aligned withztz\_\{t\}:

ℒtext=𝔼x∼𝒟s​\[1−⟨fI​\(xδ\),zt⟩\],\\mathcal\{L\}\_\{\\mathrm\{text\}\}=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{\\mathrm\{s\}\}\}\\left\[1\-\\left\\langle f\_\{I\}\(x^\{\\delta\}\),z\_\{t\}\\right\\rangle\\right\],\(5\)where⟨⋅,⋅⟩\\langle\\cdot,\\cdot\\rangledenotes cosine similarity\. This loss encourages the Shadow Trigger to steer triggered images toward the target\-class semantic region in the frozen CLIP space\.

Layer\-wise visual alignment\.Target\-text alignment constrains the final image–text similarity, but it does not explicitly control how the trigger changes intermediate visual representations\. We therefore align triggered features with target\-class prototypes within the frozen visual encoder\. For each selected visual layerl∈𝒦l\\in\\mathcal\{K\}, we compute a target\-class prototype from the target\-class image setℛyt\\mathcal\{R\}\_\{y\_\{t\}\}:

μlyt=1\|ℛyt\|​∑x∈ℛythl​\(x\),\\mu\_\{l\}^\{y\_\{t\}\}=\\frac\{1\}\{\|\\mathcal\{R\}\_\{y\_\{t\}\}\|\}\\sum\_\{x\\in\\mathcal\{R\}\_\{y\_\{t\}\}\}h\_\{l\}\(x\),\(6\)wherehl​\(x\)h\_\{l\}\(x\)denotes the layer\-llvisual feature\. The layer\-wise alignment loss is defined as

ℒlayer=𝔼x∼𝒟s​\[1\|𝒦\|​∑l∈𝒦\(1−⟨hl​\(xδ\),μlyt⟩\)\]\.\\mathcal\{L\}\_\{\\mathrm\{layer\}\}=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\}\_\{\\mathrm\{s\}\}\}\\left\[\\frac\{1\}\{\|\\mathcal\{K\}\|\}\\sum\_\{l\\in\\mathcal\{K\}\}\\left\(1\-\\left\\langle h\_\{l\}\(x^\{\\delta\}\),\\mu\_\{l\}^\{y\_\{t\}\}\\right\\rangle\\right\)\\right\]\.\(7\)This term encourages triggered images to follow the target\-class representation trajectory through the frozen visual backbone\.

Objective\.The Shadow Trigger is pretrained by minimizing

ℒshadow=ℒtext\+λl​ℒlayer\.\\mathcal\{L\}\_\{\\mathrm\{shadow\}\}=\\mathcal\{L\}\_\{\\mathrm\{text\}\}\+\\lambda\_\{l\}\\mathcal\{L\}\_\{\\mathrm\{layer\}\}\.\(8\)The optimized trigger is denoted byδshadow\\delta\_\{\\mathrm\{shadow\}\}and is used to initialize the malicious client’s trigger during federated prompt tuning\.

### 3\.4Backdoor Injection in FPT

During federated training, ShadowFPT uses the pretrained Shadow Trigger to implant targeted backdoor behavior into the federated prompt\. At communication roundtt, the malicious client receives the current global promptPtP^\{t\}from the server and initializes its local prompt asPa←PtP\_\{a\}\\leftarrow P^\{t\}\. It then performs two sequential local steps\. First, it adapts the trigger under the current global prompt\. Second, it optimizes the local prompt on poisoned local data\.

Trigger adaptation\.In the first attack round, the malicious client initializes the trigger withδshadow\\delta\_\{\\mathrm\{shadow\}\}and then updates it locally across subsequent rounds\. At roundtt, the prompt is fixed to the current global promptPtP^\{t\}, and only the trigger is optimized\. We construct the target text anchor under the current global prompt as:ztarget\(t\)=fT​\(yt,Pt\)z\_\{\\mathrm\{target\}\}^\{\(t\)\}=f\_\{T\}\(y\_\{t\};P^\{t\}\)\. The trigger is adapted using the same representation\-level objective as in Shadow Trigger pretraining:

ℒadapt\(t\)=ℒtext\(t\)\+λl​ℒlayer\(t\)\.\\mathcal\{L\}\_\{\\mathrm\{adapt\}\}^\{\(t\)\}=\\mathcal\{L\}\_\{\\mathrm\{text\}\}^\{\(t\)\}\+\\lambda\_\{l\}\\mathcal\{L\}\_\{\\mathrm\{layer\}\}^\{\(t\)\}\.\(9\)Here,ℒtext\(t\)\\mathcal\{L\}\_\{\\mathrm\{text\}\}^\{\(t\)\}aligns triggered image embeddings with the current target text anchorztarget\(t\)z\_\{\\mathrm\{target\}\}^\{\(t\)\}, whileℒlayer\(t\)\\mathcal\{L\}\_\{\\mathrm\{layer\}\}^\{\(t\)\}keeps intermediate visual features close to the target\-class visual prototypes defined in Section[3\.3](https://arxiv.org/html/2610.22871#S3.SS3)\. This step produces a prompt\-adapted triggerδ\(t\)\\delta^\{\(t\)\}\.

Backdoor prompt optimization\.With the adapted triggerδ\(t\)\\delta^\{\(t\)\}fixed, the malicious client optimizes its local promptPaP\_\{a\}using both clean and triggered local samples\. Let𝒟a\\mathcal\{D\}\_\{a\}denote the malicious client’s local dataset\. The prompt\-level backdoor objective is

ℒback​\(Pa\)=𝔼\(x,y\)∼𝒟a​\[CE⁡\(F⁡\(x,Pa\),y\)\]\+𝔼\(x,y\)∼𝒟a,y≠yt​\[CE⁡\(F⁡\(xδ\(t\),Pa\),yt\)\]\.\\mathcal\{L\}\_\{\\mathrm\{back\}\}\(P\_\{a\}\)=\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{a\}\}\\left\[\\mathrm\{CE\}\\big\(F\(x;P\_\{a\}\),y\\big\)\\right\]\+\\mathbb\{E\}\_\{\(x,y\)\\sim\\mathcal\{D\}\_\{a\},\\,y\\neq y\_\{t\}\}\\left\[\\mathrm\{CE\}\\big\(F\(x^\{\\delta^\{\(t\)\}\};P\_\{a\}\),y\_\{t\}\\big\)\\right\]\.\(10\)The first term preserves clean\-task performance, whereas the second term implants the targeted backdoor into the local prompt\.

After local optimization, the malicious client uploads only the prompt parametersPaP\_\{a\}to the server\. The adapted triggerδ\(t\)\\delta^\{\(t\)\}and the frozen CLIP encoders are never communicated\. The server then aggregates the uploaded prompts using the same aggregation rule as in standard FPT\.

## 4Experiments

### 4\.1Experimental Setup

Federated Setting\.All methods use a frozen CLIP model with ViT\-B/16 as the backbone\. Unless otherwise specified, we use textual prompt tuning as the default FPT setting, denoted by FPT\-L[Li et al\. \(2024\)](https://arxiv.org/html/2610.22871#bib.bib8)\. To evaluate whether ShadowFPT transfers across different prompt parameterizations, we also consider visual prompt tuning \(FPT\-V\)[Jia et al\. \(2022\)](https://arxiv.org/html/2610.22871#bib.bib12)and joint vision–language prompt tuning \(FPT\-VL\)[Khattak et al\. \(2023a\)](https://arxiv.org/html/2610.22871#bib.bib7)\. Following FedTPG[Qiu et al\. \(2024\)](https://arxiv.org/html/2610.22871#bib.bib4), we simulate a non\-IID federation with 20 clients\. Each client holds samples from five classes, and the class sets are disjoint across clients\. All clients participate in every communication round\.

Unless otherwise stated, a single client is malicious, and the target class is set to the first class of each dataset\. All methods are trained for 20 communication rounds with one local epoch per round, batch size 32, and learning rate0\.00350\.0035for local prompt optimization\. We report clean accuracy \(Acc\) on benign test samples and attack success rate \(ASR\) on triggered non\-target test samples\.

ShadowFPT Variants\.We evaluate two variants of ShadowFPT that differ in the data used for Shadow Trigger pretraining\.ShadowFPT\-Ipretrains the Shadow Trigger using only the malicious client’s local data, whereasShadowFPT\-IIuses auxiliary public data, instantiated as 100 randomly sampled ImageNet classes\. We do not assume semantic overlap between the auxiliary ImageNet classes and the downstream target classes\. Both variants use a32×3232\\times 32trigger patch placed at the bottom\-right corner of the image\. The Shadow Trigger is optimized with learning rate0\.010\.01during pretraining and0\.0010\.001during federated trigger adaptation\.

Datasets\.We evaluate ShadowFPT and all baselines on Caltech101[Fei\-Fei et al\. \(2004\)](https://arxiv.org/html/2610.22871#bib.bib9), Oxford Flowers[Nilsback and Zisserman \(2008\)](https://arxiv.org/html/2610.22871#bib.bib10), and UCF101[Soomro \(2012\)](https://arxiv.org/html/2610.22871#bib.bib11)\. These datasets cover generic object recognition, fine\-grained flower classification, and human action recognition, respectively\.

Attack Baselines\.We compare ShadowFPT with three groups of attack baselines\. The first group includes generic backdoor attacks, namelyBadNets[Gu et al\. \(2017\)](https://arxiv.org/html/2610.22871#bib.bib17)andBlended[Chen et al\. \(2017\)](https://arxiv.org/html/2610.22871#bib.bib18)\. The second group includes federated backdoor attacks, namelyNeurotoxin[Zhang et al\. \(2022\)](https://arxiv.org/html/2610.22871#bib.bib21),A3FL[Zhang et al\. \(2023\)](https://arxiv.org/html/2610.22871#bib.bib19),FCBA[Liu et al\. \(2024\)](https://arxiv.org/html/2610.22871#bib.bib16), andCerP[Lyu et al\. \(2023\)](https://arxiv.org/html/2610.22871#bib.bib20)\. The third group includes personalized federated backdoor attacks, namelyBad\-PFL[Fan et al\. \(2025\)](https://arxiv.org/html/2610.22871#bib.bib30)andPFedBA[Lyu et al\. \(2024b\)](https://arxiv.org/html/2610.22871#bib.bib38)\.

Defense Baselines\.We evaluate defenses covering several aggregation and filtering mechanisms\. The first category is standard aggregation, represented byFedAvg[McMahan et al\. \(2017\)](https://arxiv.org/html/2610.22871#bib.bib1)\. The second category is Byzantine\-robust aggregation, includingKrum[Blanchard et al\. \(2017\)](https://arxiv.org/html/2610.22871#bib.bib14),TrimmedMean[Yin et al\. \(2018\)](https://arxiv.org/html/2610.22871#bib.bib22), andBucketing[Karimireddy et al\. \(2022\)](https://arxiv.org/html/2610.22871#bib.bib28)\. The third category includes adaptive or similarity\-based filtering methods, namelyLaSA[Xu et al\. \(2025\)](https://arxiv.org/html/2610.22871#bib.bib27)andFoolsGold[Fung et al\. \(2018\)](https://arxiv.org/html/2610.22871#bib.bib37)\. We also evaluate trust\-based aggregation withFLTrust[Cao et al\. \(2020\)](https://arxiv.org/html/2610.22871#bib.bib26), as well as backdoor\-specific defenses, includingFLAME[Nguyen et al\. \(2022\)](https://arxiv.org/html/2610.22871#bib.bib25)andCRFL[Xie et al\. \(2021\)](https://arxiv.org/html/2610.22871#bib.bib24)\.

### 4\.2Main Results

Table 1:Overall attack performance across datasets and prompt\-tuning settings\. Results are averaged over all aggregation and defense rules\. Best Base\. denotes the strongest non\-ShadowFPT baseline under each dataset\-setting pair, selected by the highest average ASR\.Δ\\DeltaAcc andΔ\\DeltaASR report absolute differences from Best Base\., measured in percentage points\.Overall effectiveness\.We first evaluate whether ShadowFPT can implant targeted backdoors while preserving clean\-task performance\. Table[1](https://arxiv.org/html/2610.22871#S4.T1)reports results averaged over all aggregation and defense rules, with detailed results for all datasets, baselines, and defenses provided in Appendix[6](https://arxiv.org/html/2610.22871#S6)\. Because Best Base denotes the strongest non\-ShadowFPT attack for each dataset\-setting pair, this comparison is conservative\. ShadowFPT consistently outperforms Best Base across all datasets and prompt\-tuning settings\. Averaged over the nine settings, Best Base achieves 34\.56% ASR, whereas ShadowFPT\-I and ShadowFPT\-II reach 95\.63% and 97\.67%, respectively\. ShadowFPT\-II is the strongest method in every setting, improving ASR by 44\.58 to 77\.14 percentage points\.

These gains are achieved with limited degradation in clean accuracy\. On Caltech101, ShadowFPT\-II reduces clean accuracy by at most 1\.05 points\. On UCF101, the drop ranges from 0\.55 to 2\.25 points despite near\-100% ASR\. Oxford Flowers under FPT\-L shows a larger drop of 5\.78 points, suggesting a stronger clean\-attack trade\-off in fine\-grained recognition\. This trade\-off is much weaker under FPT\-V and FPT\-VL, where ShadowFPT\-II maintains comparable clean accuracy while achieving 98\.94% and 99\.18% ASR\. Overall, these results show that ShadowFPT substantially improves targeted attack success without primarily relying on degrading benign performance\.

Table 2:Robustness of ShadowFPT against aggregation and defense rules\. Results are averaged over all datasets and prompt\-tuning settings\.Robustness against defenses\.We next examine whether ShadowFPT remains effective against aggregation and defense rules that act on uploaded client updates\. As shown in Table[2](https://arxiv.org/html/2610.22871#S4.T2), ShadowFPT\-II achieves the highest ASR under every defense, ranging from 96\.63% to 98\.30%\. In contrast, the strongest non\-ShadowFPT baseline reaches only 13\.17% to 60\.93% ASR\. The gains are especially pronounced under robust and backdoor\-specific defenses, including Krum \(\+73\.97 points\), LaSA \(\+76\.55 points\), FLAME \(\+84\.75 points\), and CRFL \(\+74\.55 points\)\.

These results support our claim that update\-centered defenses are insufficient for FPT\. Although such defenses mitigate abnormal client updates through robust aggregation, filtering, clustering, clipping, or certification, ShadowFPT shifts much of the backdoor effect to the interaction between the adapted trigger and the frozen representation space\. ShadowFPT\-II improves ASR by 63\.61 percentage points on average over Best Base, while incurring only a 1\.71\-point average clean\-accuracy drop\. This indicates that its robustness does not primarily come from degrading benign performance\.

![Refer to caption](https://arxiv.org/html/2610.22871v1/figures/ASR_ACC1.png)Figure 2:Transferability of ShadowFPT across prompt modalities\. Results are averaged over Caltech101, Oxford Flowers, UCF101, and all aggregation or defense rules\. Best Base denotes the strongest non\-ShadowFPT baseline under each dataset\-setting pair, selected by the highest average ASR before averaging across datasets\. ShadowFPT achieves consistently high ASR under FPT\-L, FPT\-V, and FPT\-VL, while largely preserving clean accuracy\.Transfer across prompt modalities\.We further evaluate whether ShadowFPT transfers across textual, visual, and joint vision–language prompt tuning\. Figure[2](https://arxiv.org/html/2610.22871#S4.F2)reports results averaged over three datasets and all aggregation or defense rules\. ShadowFPT\-II achieves 96\.42%, 97\.63%, and 98\.95% ASR under FPT\-L, FPT\-V, and FPT\-VL, respectively, substantially outperforming Best Base, which reaches only 27\.65%, 33\.20%, and 42\.83% ASR\. ShadowFPT\-I shows a similar pattern, with ASR consistently above 93% across all prompt modalities\.

Clean accuracy is largely preserved\. Compared with Best Base, ShadowFPT\-II incurs average clean\-accuracy drops of 2\.92, 0\.24, and 1\.45 percentage points under FPT\-L, FPT\-V, and FPT\-VL, respectively\. These results show that ShadowFPT is not specific to a single prompt modality\. Instead, it exploits the shared frozen CLIP representation space, which remains common across textual, visual, and joint vision–language prompt tuning\.

![Refer to caption](https://arxiv.org/html/2610.22871v1/pca1.png)Figure 3:PCA visualization of client prompts under FedAvg\. Blue points denote benign clients, the red cross denotes the malicious client, and the green star denotes the benign centroid\.Prompt\-space stealthiness\.We further examine whether malicious prompt updates can be separated from benign ones\. Figure[3](https://arxiv.org/html/2610.22871#S4.F3)projects client prompts under FedAvg onto the first two principal components\. Existing attacks produce malicious prompts that appear as clear outliers from the benign cluster\. In contrast, the ShadowFPT prompt lies close to benign clients and nearly overlaps with the benign centroid\. This supports our claim that ShadowFPT shifts much of the backdoor effect from uploaded prompt updates to trigger\-induced representation steering in the frozen backbone, thereby preserving prompt\-space stealthiness\.

### 4\.3Ablation Study

![Refer to caption](https://arxiv.org/html/2610.22871v1/figures/aba1.png)Figure 4:Role of Shadow Trigger Pretraining and Backdoor Injection in FPT on Caltech101\. Random Trigger uses a randomly initialized trigger; Shadow Trigger w/o Injection uses the pretrained Shadow Trigger without Backdoor Injection in FPT; ShadowFPT denotes the full method\.
![Refer to caption](https://arxiv.org/html/2610.22871v1/figures/aba2.png)Figure 5:Role of representation\-level alignment on Caltech101\. ASR is reported under nine aggregation and defense rules\. Target\-text Only and Layer\-wise Only use a single alignment objective, whereas ShadowFPT combines both\.

Role of Pretraining and Injection\.We assess the roles of Shadow Trigger Pretraining and Backdoor Injection in FPT on Caltech101\. As shown in Figure[5](https://arxiv.org/html/2610.22871#S4.F5), Random Trigger achieves only 21\.32% average ASR, indicating that an arbitrary trigger is insufficient to induce a strong backdoor\. Shadow Trigger without Injection increases the average ASR to 49\.75%, suggesting that pretraining already learns target\-directed behavior in the frozen CLIP representation space, but remains fragile under defenses such as Krum \(25\.17%\) and FLTrust \(31\.99%\)\. The full ShadowFPT further raises the average ASR to 90\.36%, with large gains under Krum \(\+59\.54\) and FLTrust \(\+60\.55\)\. These results show that pretraining provides a transferable initialization, while Backdoor Injection is necessary to convert it into a robust federated backdoor\.

Role of Representation\-Level Alignment\.We further examine the two representation\-level objectives used in Shadow Trigger Pretraining\. As shown in Figure[5](https://arxiv.org/html/2610.22871#S4.F5), target\-text alignment alone and layer\-wise visual alignment alone achieve 66\.38% and 59\.25% average ASR, respectively, indicating that both provide useful target\-directed guidance\. However, each objective alone remains fragile under defenses, such as CRFL for Target\-text Only \(20\.08%\) and FoolsGold for Layer\-wise Only \(20\.73%\)\. In contrast, the full ShadowFPT objective reaches 90\.36% average ASR, improving over the two single\-objective variants by 23\.98 and 31\.11 percentage points\. These results show that target\-text alignment provides semantic target guidance, while layer\-wise visual alignment stabilizes the triggered representation trajectory, together producing a more robust Shadow Trigger\.

## 5Conclusion

This paper identifies a frozen\-backbone backdoor surface in FPT\. Unlike conventional federated backdoor attacks that rely primarily on anomalous uploaded updates, ShadowFPT exploits the shared frozen CLIP representation space and shifts much of the attack burden to trigger\-induced representation steering\. We propose a two\-stage attack in which a Shadow Trigger is first pretrained through target\-text and layer\-wise visual alignment, and then adapted during FPT while only prompt parameters are uploaded to the server\. Across multiple datasets, aggregation and defense rules, and textual, visual, and joint vision–language prompt tuning, ShadowFPT achieves consistently high targeted attack success with limited clean accuracy degradation\. Prompt\-space analyses further show that its malicious updates remain close to benign prompts, explaining why update\-centered defenses are insufficient\. Our results suggest that securing FPT requires moving beyond prompt anomaly detection and accounting for the interaction between triggers and frozen representation geometry\. More broadly, this work highlights a security risk of FPT: freezing the backbone reduces trainable parameters, but does not eliminate backdoor attack surfaces\.

## Acknowledgments

This work is supported by the National Natural Science Foundation of China \(Grant No\. 62521004\)\.

## References

- Blanchardet al\.\(2017\)P\. Blanchard, E\. M\. El Mhamdi, R\. Guerraoui, and J\. StainerMachine learning with adversaries: byzantine tolerant gradient descent\.NeurIPS\.Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p3.1),[§2](https://arxiv.org/html/2610.22871#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p6.1)\.
- Caoet al\.\(2020\)X\. Cao, M\. Fang, J\. Liu, and N\. Z\. GongFltrust: byzantine\-robust federated learning via trust bootstrapping\.arXiv preprint arXiv:2012\.13995\.Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p3.1),[§2](https://arxiv.org/html/2610.22871#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p6.1)\.
- Chenet al\.\(2017\)X\. Chen, C\. Liu, B\. Li, K\. Lu, and D\. SongTargeted backdoor attacks on deep learning systems using data poisoning\.arXiv preprint arXiv:1712\.05526\.Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p2.1),[§2](https://arxiv.org/html/2610.22871#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p5.1)\.
- Cuiet al\.\(2024\)T\. Cui, H\. Li, J\. Wang, and Y\. ShiHarmonizing generalization and personalization in federated prompt learning\.arXiv preprint arXiv:2405\.09771\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p1.1)\.
- Fanet al\.\(2025\)M\. Fan, Z\. Hu, F\. Wang, and C\. ChenBad\-pfl: exploring backdoor attacks against personalized federated learning\.arXiv preprint arXiv:2501\.12736\.Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p2.1),[§2](https://arxiv.org/html/2610.22871#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p5.1)\.
- Fei\-Feiet al\.\(2004\)L\. Fei\-Fei, R\. Fergus, and P\. PeronaLearning generative visual models from few training examples: an incremental bayesian approach tested on 101 object categories\.In2004 conf\. on computer vision and pattern recognition workshop,pp\. 178–178\.Cited by:[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p4.1)\.
- Funget al\.\(2018\)C\. Fung, C\. J\. Yoon, and I\. BeschastnikhMitigating sybils in federated learning poisoning\.arXiv preprint arXiv:1808\.04866\.Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p3.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p6.1)\.
- Guet al\.\(2017\)T\. Gu, B\. Dolan\-Gavitt, and S\. GargBadnets: identifying vulnerabilities in the machine learning model supply chain\.arXiv preprint arXiv:1708\.06733\.Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p2.1),[§2](https://arxiv.org/html/2610.22871#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p5.1)\.
- Guerraouiet al\.\(2018\)R\. Guerraoui S\. Rouaultet al\.The hidden vulnerability of distributed learning in byzantium\.InICML,Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p3.1)\.
- Guoet al\.\(2023a\)T\. Guo, S\. Guo, J\. Wang, X\. Tang, and W\. XuPromptfl: let federated participants cooperatively learn prompts instead of models\-federated learning in age of foundation model\.IEEE Transactions on Mobile Computing\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p1.1)\.
- Guoet al\.\(2023b\)T\. Guo, S\. Guo, and J\. WangPfedprompt: learning personalized prompt for vision\-language models in federated learning\.InProceedings of the ACM Web Conference 2023,pp\. 1364–1374\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p1.1)\.
- Huanget al\.\(2025\)S\. Huang, Y\. Li, C\. Chen, Y\. Gao, and X\. HuFedid: enhancing federated learning security through dynamic identification\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p3.1)\.
- Jiaet al\.\(2022\)M\. Jia, L\. Tang, B\. Chen, C\. Cardie, S\. Belongie, B\. Hariharan, and S\. LimVisual prompt tuning\.InECCV,Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p1.1)\.
- Karimireddyet al\.\(2022\)S\. P\. Karimireddy, L\. He, and M\. JaggiByzantine\-robust learning on heterogeneous datasets via bucketing\.arXiv preprint arXiv:2006\.09365\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p6.1)\.
- Khanet al\.\(2025\)M\. A\. Khan, Y\. Chandio, and F\. M\. AnwarSABRE\-fl: selective and accurate backdoor rejection for federated prompt learning\.arXiv preprint arXiv:2506\.22506\.Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p3.1)\.
- Khattaket al\.\(2023a\)M\. U\. Khattak, H\. Rasheed, M\. Maaz, S\. Khan, and F\. S\. KhanMaple: multi\-modal prompt learning\.InCVPR,Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p1.1),[§2](https://arxiv.org/html/2610.22871#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p1.1)\.
- Khattaket al\.\(2023b\)M\. U\. Khattak, S\. T\. Wasim, M\. Naseer, S\. Khan, M\. Yang, and F\. S\. KhanSelf\-regulating prompts: foundational model adaptation without forgetting\.InProceedings of the IEEE/CVF international conference on computer vision,pp\. 15190–15200\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p1.1)\.
- Liet al\.\(2024\)L\. Li, H\. Guan, J\. Qiu, and M\. SpratlingOne prompt word is enough to boost adversarial robustness for pre\-trained vision\-language models\.InCVPR,Cited by:[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p1.1)\.
- Lianget al\.\(2024\)S\. Liang, M\. Zhu, A\. Liu, B\. Wu, X\. Cao, and E\. ChangBadclip: dual\-embedding guided backdoor attack on multimodal contrastive learning\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 24645–24654\.Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p2.1),[§2](https://arxiv.org/html/2610.22871#S2.p2.1)\.
- Liuet al\.\(2024\)T\. Liu, Y\. Zhang, Z\. Feng, Z\. Yang, C\. Xu, D\. Man, and W\. YangBeyond traditional threats: a persistent backdoor attack on federated learning\.InAAAI,Cited by:[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p5.1)\.
- Luoet al\.\(2025\)J\. Luo, C\. Chen, and S\. WuMixture of experts made personalized: federated prompt learning for vision\-language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 95458\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p1.1)\.
- Lyuet al\.\(2024a\)X\. Lyu, Y\. Han, W\. Wang, J\. Liu, B\. Wang, K\. Chen, Y\. Li, J\. Liu, and X\. ZhangCoBA: collusive backdoor attacks with optimized trigger to federated learning\.IEEE Transactions on Dependable and Secure Computing22\(2\),pp\. 1506–1518\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p2.1)\.
- Lyuet al\.\(2023\)X\. Lyu, Y\. Han, W\. Wang, J\. Liu, B\. Wang, J\. Liu, and X\. ZhangPoisoning with cerberus: stealthy and colluded backdoor attack against federated learning\.InAAAI,Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p5.1)\.
- Lyuet al\.\(2024b\)X\. Lyu, Y\. Han, W\. Wang, J\. Liu, Y\. Zhu, G\. Xu, J\. Liu, and X\. ZhangLurking in the shadows: unveiling stealthy backdoor attacks against personalized federated learning\.In33rd USENIX Security Symposium \(USENIX Security 24\),pp\. 4157–4174\.Cited by:[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p5.1)\.
- McMahanet al\.\(2017\)B\. McMahan, E\. Moore, D\. Ramage, S\. Hampson, and B\. A\. y ArcasCommunication\-efficient learning of deep networks from decentralized data\.InArtificial intelligence and statistics,pp\. 1273–1282\.Cited by:[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p6.1)\.
- Nguyenet al\.\(2022\)T\. D\. Nguyen, P\. Rieger, H\. Chen, H\. Yalame, H\. Möllering, H\. Fereidooni, S\. Marchal, M\. Miettinen, A\. Mirhoseini, S\. Zeitouni,et al\.\{\\\{flame\}\\\}: Taming backdoors in federated learning\.InUSENIX Security,Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p3.1),[§2](https://arxiv.org/html/2610.22871#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p6.1)\.
- Nilsback and Zisserman \(2008\)M\. Nilsback and A\. ZissermanAutomated flower classification over a large number of classes\.In2008 Sixth Indian conf\. on computer vision, graphics & image processing,pp\. 722–729\.Cited by:[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p4.1)\.
- Pillutlaet al\.\(2022\)K\. Pillutla, S\. M\. Kakade, and Z\. HarchaouiRobust aggregation for federated learning\.IEEE Transactions on Signal Processing70,pp\. 1142–1154\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p3.1)\.
- Qiuet al\.\(2024\)C\. Qiu, X\. Li, C\. K\. Mummadi, M\. R\. Ganesh, Z\. Li, L\. Peng, and W\. LinFederated text\-driven prompt generation for vision\-language models\.InICLR,Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p1.1)\.
- Radfordet al\.\(2021\)A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark,et al\.Learning transferable visual models from natural language supervision\.InICML,Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p1.1),[§2](https://arxiv.org/html/2610.22871#S2.p1.1)\.
- Singhaet al\.\(2025\)M\. Singha, S\. Roy, S\. Mehrotra, A\. Jha, M\. Abdar, B\. Banerjee, and E\. RicciFedMVP: federated multi\-modal visual prompt tuning for vision\-language models\.arXiv preprint arXiv:2504\.20860\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p1.1)\.
- Soomro \(2012\)K\. SoomroUCF101: a dataset of 101 human actions classes from videos in the wild\.arXiv preprint arXiv:1212\.0402\.Cited by:[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p4.1)\.
- Xieet al\.\(2021\)C\. Xie, M\. Chen, P\. Chen, and B\. LiCrfl: certifiably robust federated learning against backdoor attacks\.InICML,Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p6.1)\.
- Xuet al\.\(2025\)J\. Xu, Z\. Zhang, and R\. HuAchieving byzantine\-resilient federated learning via layer\-adaptive sparsified model aggregation\.InWACV,Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p6.1)\.
- Yinet al\.\(2018\)D\. Yin, Y\. Chen, R\. Kannan, and P\. BartlettByzantine\-robust distributed learning: towards optimal statistical rates\.InICML,Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p3.1),[§2](https://arxiv.org/html/2610.22871#S2.p3.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p6.1)\.
- Zhanget al\.\(2023\)H\. Zhang, J\. Jia, J\. Chen, L\. Lin, and D\. WuA3fl: adversarially adaptive backdoor attacks to federated learning\.NeurIPS\.Cited by:[§1](https://arxiv.org/html/2610.22871#S1.p2.1),[§2](https://arxiv.org/html/2610.22871#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p5.1)\.
- Zhanget al\.\(2025\)X\. Zhang, X\. Xue, X\. Du, X\. Xie, Y\. Liu, and M\. SunRuntime backdoor detection for federated learning via representational dissimilarity analysis\.IEEE Transactions on Dependable and Secure Computing\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p3.1)\.
- Zhanget al\.\(2022\)Z\. Zhang, A\. Panda, L\. Song, Y\. Yang, M\. Mahoney, P\. Mittal, R\. Kannan, and J\. GonzalezNeurotoxin: durable backdoors in federated learning\.InICML,Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p2.1),[§4\.1](https://arxiv.org/html/2610.22871#S4.SS1.p5.1)\.
- Zhouet al\.\(2022a\)K\. Zhou, J\. Yang, C\. C\. Loy, and Z\. LiuConditional prompt learning for vision\-language models\.InCVPR,pp\. 16816–16825\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p1.1)\.
- Zhouet al\.\(2022b\)K\. Zhou, J\. Yang, C\. C\. Loy, and Z\. LiuLearning to prompt for vision\-language models\.International Journal of Computer Vision130\(9\),pp\. 2337–2348\.Cited by:[§2](https://arxiv.org/html/2610.22871#S2.p1.1)\.

## 6Appendix

Table 3:Acc and ASR \(%\) on Caltech101, Oxford Flowers, and UCF101 under FPT\-L, FPT\-V, and FPT\-VL\. Bold values indicate the best result among all attack methods under each defense\. The arrowed values below ShadowFPT\-I and ShadowFPT\-II report absolute differences from the strongest baseline, where green and red arrows indicate improvement and degradation, respectively\.
## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]\.
4. Justification: The abstract and introduction clearly state the paper’s scope, threat model, main methodological contribution, and empirical claims\. The claims are supported by the experiments in Section[4](https://arxiv.org/html/2610.22871#S4), including evaluations across datasets, aggregation and defense rules, non\-IID federated settings, and textual, visual, and joint vision–language prompt tuning\.
5. 2\.Limitations
6. Question: Does the paper discuss the limitations of the work performed by the authors?
7. Answer:\[No\]\.
8. Justification: The current version does not include a dedicated limitations discussion\. Important limitations include the white\-box threat model, the focus on CLIP\-style frozen backbones, the use of one malicious client by default, and evaluation on three public image\-recognition datasets\.
9. 3\.Theory assumptions and proofs
10. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete proof?
11. Answer:\[N/A\]\.
12. Justification: The paper does not present formal theoretical results, theorems, or proofs\. The equations in Section[3](https://arxiv.org/html/2610.22871#S3)define the problem setup and optimization objectives rather than stating theoretical guarantees\.
13. 4\.Experimental result reproducibility
14. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper?
15. Answer:\[Yes\]\.
16. Justification: The paper specifies the main information needed to reproduce the experiments, including the CLIP backbone, datasets, federated setting, non\-IID construction, prompt\-tuning modalities, baselines, defenses, training details, trigger configuration, and evaluation metrics\. Additional results and implementation details are provided in the appendix\.
17. 5\.Open access to data and code
18. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
19. Answer:\[No\]\.
20. Justification: The experiments use publicly available datasets and standard pretrained backbones, but the current submission does not provide an open\-source code package or reproduction scripts\. We plan to release anonymized code and detailed reproduction instructions when permitted by the submission policy\.
21. 6\.Experimental setting/details
22. Question: Does the paper specify all the training and test details necessary to understand the results?
23. Answer:\[Yes\]\.
24. Justification: The experimental setup specifies the backbone, datasets, non\-IID client construction, number of clients, malicious\-client setting, communication rounds, local epochs, batch size, learning rates, trigger configuration, attack baselines, defense baselines, and metrics\. These details are provided in Section[4](https://arxiv.org/html/2610.22871#S4)and the appendix\.
25. 7\.Experiment statistical significance
26. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
27. Answer:\[No\]\.
28. Justification: The paper reports results averaged over datasets, prompt\-tuning settings, and aggregation or defense rules, but it does not report error bars, confidence intervals, or statistical significance tests over repeated random seeds\. This is a limitation of the current empirical reporting\.
29. 8\.Experiments compute resources
30. Question: For each experiment, does the paper provide sufficient information on the computer resources needed to reproduce the experiments?
31. Answer:\[No\]\.
32. Justification: The current paper does not provide detailed compute\-resource information such as GPU type, memory, runtime per experiment, or total compute budget\. These details should be added in the appendix for full reproducibility\.
33. 9\.Code of ethics
34. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics?
35. Answer:\[Yes\]\.
36. Justification: The research is conducted as a controlled security study using public datasets and standard experimental settings\. The paper analyzes a dual\-use backdoor threat to improve understanding of vulnerabilities in federated prompt tuning and does not involve unauthorized system access, private user data, or human\-subject data collection\.
37. 10\.Broader impacts
38. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
39. Answer:\[Yes\]\.
40. Justification: The work has both positive and negative societal implications\. It exposes a realistic security risk in federated prompt tuning, which can motivate stronger defenses, but the proposed attack is dual\-use and could be misused if applied outside controlled research settings\.
41. 11\.Safeguards
42. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse?
43. Answer:\[N/A\]\.
44. Justification: The paper does not release a high\-risk pretrained model, image generator, scraped dataset, or deployable system\. The study uses public datasets and evaluates attacks in an offline experimental setting\.
45. 12\.Licenses for existing assets
46. Question: Are the creators or original owners of assets used in the paper properly credited, and are the license and terms of use explicitly mentioned and properly respected?
47. Answer:\[Yes\]\.
48. Justification: The paper cites the original sources for the existing datasets, pretrained CLIP backbone, prompt\-tuning methods, attack baselines, and defense baselines used in the experiments\. The experiments are based on publicly available research assets\.
49. 13\.New assets
50. Question: Are new assets introduced in the paper well documented, and is the documentation provided alongside the assets?
51. Answer:\[N/A\]\.
52. Justification: The paper does not introduce a new dataset, benchmark, pretrained model, or other standalone research asset\. The contribution is a method and empirical security analysis\.
53. 14\.Crowdsourcing and research with human subjects
54. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation?
55. Answer:\[N/A\]\.
56. Justification: The paper does not involve crowdsourcing, user studies, human\-subject experiments, or participant compensation\. All experiments are conducted on existing public image\-recognition datasets\.
57. 15\.Institutional review board approvals or equivalent for research with human subjects
58. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board approvals or an equivalent approval/review were obtained?
59. Answer:\[N/A\]\.
60. Justification: The paper does not involve human subjects, crowdsourcing, private user data, or participant interaction\. Therefore, IRB approval or equivalent human\-subjects review is not applicable\.
61. 16\.Declaration of LLM usage
62. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research?
63. Answer:\[N/A\]\.
64. Justification: The core method, experiments, and scientific claims do not rely on LLMs as an important, original, or non\-standard component\. Any use of LLMs for writing, editing, or formatting does not affect the methodology, experimental design, or scientific conclusions\.