Personalized Federated Direct Preference Optimization via LoRA Adapters

# Personalized Federated Direct Preference Optimization via LoRA Adapters
Source: [https://arxiv.org/html/2610.12885](https://arxiv.org/html/2610.12885)
###### Abstract

Federated Direct Preference Optimization \(FedDPO\) enables privacy\-preserving alignment of language models across distributed agents without centralizing sensitive preference data\. However, existing FedDPO analysis assumes all clients converge to a single shared policy, which is fundamentally at odds with the heterogeneous preference distributions present in practice\. We proposePersonalized FedDPO \(pFedDPO\), in which each client maintains a small private LoRA adapter atop a shared global base model\. In pFedDPO, the base model is synchronized via FedAvg each round, while LoRA adapters remain local and capture client\-specific preference patterns\. We prove that this decomposition replaces the gradient diversityκ2\\kappa^\{2\}in the FedDPO bound with a residual heterogeneityκeff2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}, which is no larger thanκ2\\kappa^\{2\}when adapters absorb client\-specific directions, at the cost of an additive adapter\-bias term that vanishes as adapters converge\. Experiments on preference datasets show that pFedDPO consistently attains a lower gradient norm than vanilla FedDPO while remaining competitive in training loss and per\-client reward accuracy\.

1Translational AI Center, Iowa State University, Ames, IA 50011 USA

2Department of Mechanical Engineering, Iowa State University, Ames, IA 50011 USA

\{zhjiang, soumiks\}@iastate\.edu

## Introduction

Aligning language models \(LMs\)\([Wang et al\. 2023](https://arxiv.org/html/2610.12885#bib.bib5)\)with human preferences is increasingly performed via Direct Preference Optimization \(DPO\)\([Rafailov et al\. 2023](https://arxiv.org/html/2610.12885#bib.bib1)\), which avoids explicit reward modeling by optimizing a classification\-style loss over \(chosen/rejected\) response pairs\. As AI systems proliferate to distributed and privacy\-sensitive environments, e\.g\., healthcare devices\([Nguyen et al\. 2022](https://arxiv.org/html/2610.12885#bib.bib6)\), personal assistants\([Sa’ad et al\. 2025](https://arxiv.org/html/2610.12885#bib.bib7)\), institutional silos\([Huang et al\. 2023](https://arxiv.org/html/2610.12885#bib.bib8)\), preference data is naturally fragmented across users who cannot share their data due to regulatory constraints \(HIPAA\([Moore and Frye 2019](https://arxiv.org/html/2610.12885#bib.bib9)\), GDPR\([Li et al\. 2019a](https://arxiv.org/html/2610.12885#bib.bib10)\)\)\. This motivates distributed DPO, and in particular the federated and decentralized variants recently analyzed theoretically in\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\)\. A critical assumption for FedDPO in\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\)is that allNNclients are driven toward a common global policy\. This is appropriate when preference distributions are homogeneous, but unrealistic in practice: a medical professional in an ICU and a casual user evaluating restaurant suggestions have genuinely different preferences that should not be averaged away\. The standard FedDPO convergence bound contains an irreducible floor proportional to gradient diversity \(κ2\\kappa^\{2\}\), which captures exactly this inter\-client heterogeneity and cannot be reduced by communicating more or training longer, for a fixed step size\.

Parameter\-efficient fine\-tuning, and LoRA\([Hu et al\. 2022](https://arxiv.org/html/2610.12885#bib.bib3)\)in particular, offers a natural solution: decompose each client’s model into a shared global base and a small private adapter\. The base model learns the shared structure of preferences across all agents and is aggregated via FedAvg\([Li et al\. 2019b](https://arxiv.org/html/2610.12885#bib.bib4)\), while the adapter captures client\-specific deviations and is never communicated\. This decomposition is the key contribution of this work, differing from the method in\([Zhu et al\. 2026](https://arxiv.org/html/2610.12885#bib.bib2)\)\.

We make the following contributions: We propose pFedDPO, a personalized federated DPO algorithm in which each client maintains a LoRA adapter atop a shared base model updated via FedAvg\. We prove a convergence bound for pFedDPO in whichκ2\\kappa^\{2\}is replaced byκeff2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}at the cost of an explicit adapter\-bias termC​εϕC\\varepsilon\_\{\\phi\}, so the bound is tighter than vanilla FedDPO whenever the heterogeneity reduction outweighs this bias \(Theorem[1](https://arxiv.org/html/2610.12885#Thmtheorem1)\)\. On two preference datasets, pFedDPO consistently attains a lower gradient norm than vanilla FedDPO with competitive loss and per\-client accuracy\.

## Background and Problem Setup

Federated DPO\.We follow the federated setting of\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\)\.NNclients indexi∈\[N\]=\{1,2,…,N\}i\\in\[N\]=\\\{1,2,\.\.\.,N\\\}each hold a disjoint dataset𝒟i\\mathcal\{D\}\_\{i\}of preference pairs\(τ\+,τ−\)\(\\tau^\{\+\},\\tau^\{\-\}\)\. The per\-client DPO loss is

ℒθi=𝔼\(τ\+,τ−\)∼𝒟i​\[lθ​\(τ\+,τ−\)\]\\mathcal\{L\}^\{i\}\_\{\\theta\}=\\mathbb\{E\}\_\{\(\\tau^\{\+\},\\tau^\{\-\}\)\\sim\\mathcal\{D\}\_\{i\}\}\[l\_\{\\theta\}\(\\tau^\{\+\},\\tau^\{\-\}\)\]\(1\)wherelθ​\(τ\+,τ−\)=−log​σ​\(β⁡\[Δθ​\(τ\+,τ−\)−Δref​\(τ\+,τ−\)\]\)l\_\{\\theta\}\(\\tau^\{\+\},\\tau^\{\-\}\)=\-\\text\{log\}\\sigma\(\\beta\[\\Delta\_\{\\theta\}\(\\tau^\{\+\},\\tau^\{\-\}\)\-\\Delta\_\{\\text\{ref\}\}\(\\tau^\{\+\},\\tau^\{\-\}\)\]\),β\>0\\beta\>0is the inverse temperature parameter,Δθ​\(τ\+,τ−\):=log​πθ​\(τ\+\)−log​πθ​\(τ−\)\\Delta\_\{\\theta\}\(\\tau^\{\+\},\\tau^\{\-\}\):=\\text\{log\}\\pi\_\{\\theta\}\(\\tau^\{\+\}\)\-\\text\{log\}\\pi\_\{\\theta\}\(\\tau^\{\-\}\), andΔref​\(τ\+,τ−\):=log​πref​\(τ\+\)−log​πref​\(τ−\)\\Delta\_\{\\text\{ref\}\}\(\\tau^\{\+\},\\tau^\{\-\}\):=\\text\{log\}\\pi\_\{\\text\{ref\}\}\(\\tau^\{\+\}\)\-\\text\{log\}\\pi\_\{\\text\{ref\}\}\(\\tau^\{\-\}\),πref\\pi\_\{\\text\{ref\}\}is a reference policy\. The global objective isℒθ=\(1/N\)​∑iℒθi\\mathcal\{L\}\_\{\\theta\}=\(1/N\)\\sum\_\{i\}\\mathcal\{L\}\_\{\\theta\}^\{i\}\. FedDPO runsE≥1E\\geq 1local gradient steps per client, aggregates via FedAvg, and repeats forRRrounds\. The convergence bound \(Theorem 5\.1 of\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\)\) contains an irreducible term proportional toκ2\\kappa^\{2\}, the gradient diversity:κ2:=\(1/N\)​∑i𝔼⁡\[‖∇θℒθ−∇θℒθi‖2\]\\kappa^\{2\}:=\(1/N\)\\sum\_\{i\}\\mathbb\{E\}\[\\\|\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\theta\}\-\\nabla\_\{\\theta\}\\mathcal\{L\}^\{i\}\_\{\\theta\}\\\|^\{2\}\], where∥⋅∥\\\|\\cdot\\\|indicates the Euclidean norm\. This term reflects how much each client’s local gradient deviates from the global gradient, and is driven entirely bypreference heterogeneityacross clients\. It cannot be eliminated by algorithmic choices within vanilla FedDPO\.

LoRA\.Low\-Rank Adaptation \(LoRA\)\([Hu et al\. 2022](https://arxiv.org/html/2610.12885#bib.bib3)\)reparameterizes a weight matrixW∈ℝm×nW\\in\\mathbb\{R\}^\{m\\times n\}asW\+B​AW\+BA, withB∈ℝm×rB\\in\\mathbb\{R\}^\{m\\times r\},A∈ℝr×nA\\in\\mathbb\{R\}^\{r\\times n\},r≪min⁡\(m,n\)r\\ll\\min\(m,n\)\. Standard LoRA freezesWW; in pFedDPO,WWbelongs to the shared baseθ\\thetaand is also trained\. For a768×768768\\times 768matrix andr=8r=8, the adapter has 12,288 parameters versus 589,824\. Adapters add no communication since they are never transmitted\.

## Personalized FedDPO \(pFedDPO\)

Algorithm\.Each clientiimaintains a modelπθ,ϕi\\pi\_\{\\theta,\\phi\_\{i\}\}, whereθ∈ℝd\\theta\\in\\mathbb\{R\}^\{d\}is the shared global base andϕi=\(Ai,Bi\)\\phi\_\{i\}=\(A\_\{i\},B\_\{i\}\)is a private LoRA adapter\. The effective policy isπθ\\pi\_\{\\theta\}with its attention weights augmented byBi​AiB\_\{i\}A\_\{i\}\. We writeℒθ,ϕii\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}for clientii’s loss \(1\) underπθ,ϕi\\pi\_\{\\theta,\\phi\_\{i\}\}\. The global baseθ\\thetais synchronized via FedAvg every round, and adaptersϕi\\phi\_\{i\}are never communicated\. Thus, in each roundrr:

- •Server broadcasts current global baseθr\\theta^\{r\}to selected clients𝒮r⊂\[N\]\\mathcal\{S\}\_\{r\}\\subset\[N\],\|𝒮r\|=S\|\\mathcal\{S\}\_\{r\}\|=S\.
- •Each clienti∈𝒮ri\\in\\mathcal\{S\}\_\{r\}loadsθr\\theta^\{r\}, restores its privateϕi\\phi\_\{i\}from local storage, and runsEElocal DPO gradient steps updating bothθ\\thetaandϕi\\phi\_\{i\}jointly\.
- •Client uploads only the base model updateΔ​θi=θir,E−θr\\Delta\\theta\_\{i\}=\\theta\_\{i\}^\{r,E\}\-\\theta^\{r\}\(adapterϕi\\phi\_\{i\}remains private\)\.
- •Server aggregatesθr\+1=\(1/\|𝒮r\|\)​∑i∈𝒮r\(θr\+Δ​θi\)\\theta^\{r\+1\}=\(1/\|\\mathcal\{S\}\_\{r\}\|\)\\sum\_\{i\\in\\mathcal\{S\}\_\{r\}\}\(\\theta^\{r\}\+\\Delta\\theta\_\{i\}\)\.

The client’s effective policy at inference isπθr,ϕi\\pi\_\{\\theta^\{r\},\\phi\_\{i\}\},personalizingto𝒟i\\mathcal\{D\}\_\{i\}while leveraging the shared structure learned across all agents\. Each selected client uploads exactly one vectorΔ​θi∈ℝd\\Delta\\theta\_\{i\}\\in\\mathbb\{R\}^\{d\}per round, the same dimension as vanilla FedDPO\. The per\-round communication volume is thereforeS×dS\\times dparameters in both cases\. pFedDPO’s advantage is not reduced communication volume but improved convergence quality:Δ​θi\\Delta\\theta\_\{i\}reflects gradient work on a base model partially decoupled from client\-specific preference directions byϕi\\phi\_\{i\}, yielding smaller effective heterogeneityκeff2\\kappa^\{2\}\_\{\\text\{eff\}\}\(defined below\) in the convergence bound\. Please see Appendix for the motivation why LoRA adapters are kept private\. Figure[1](https://arxiv.org/html/2610.12885#Sx3.F1)shows the specific detail of how pFedDPO works schematically\.

![Refer to caption](https://arxiv.org/html/2610.12885v1/pfeddpo_diagram.png)Figure 1:Illustration of the pFedDPO framework\. The server maintains a shared global base modelθ\\theta\. Each clientiireceivesθ\\thetafrom the server, combines it with its private LoRA adapterϕi\\phi\_\{i\}\(never shared\), and performs local DPO updates on its private dataset𝒟i\\mathcal\{D\}\_\{i\}\. Only the base model updatesΔ​θi\\Delta\\theta\_\{i\}are uploaded to the server for FedAvg aggregation, while adapters and data remain local\. This design is intended to reduce effective gradient diversity \(κeff2≤κ2\\kappa^\{2\}\_\{\\text\{eff\}\}\\leq\\kappa^\{2\}under the conditions in the Appendix\) and enables personalization without additional communication overhead\.Convergence Analysis\.We analyze pFedDPO under the same assumptions as\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\)\(Assumptions 3\.4\-4\.1\)\. The key insight is thatthe LoRA adapter absorbs a portion of the inter\-client gradient divergence, reducing the effective heterogeneity seen by the base model update\. LetFi​\(θ\):=minϕ⁡ℒθ,ϕiF\_\{i\}\(\\theta\):=\\min\_\{\\phi\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\}andF:=1N​∑iFiF:=\\frac\{1\}\{N\}\\sum\_\{i\}F\_\{i\}\. By the envelope theorem,∇Fi​\(θ\)=∇θℒθ,ϕi∗i\\nabla F\_\{i\}\(\\theta\)=\\nabla\_\{\\theta\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}^\{\*\}\}withϕi∗∈arg⁡minϕ⁡ℒθ,ϕi\\phi\_\{i\}^\{\*\}\\in\\arg\\min\_\{\\phi\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\}, i\.e\., the base gradient after clientii’s adapter has absorbed what it can express\. We define the residual heterogeneityκeff2:=1N​∑i𝔼⁡\[‖∇F​\(θ\)−∇Fi​\(θ\)‖2\]\\kappa^\{2\}\_\{\\mathrm\{eff\}\}:=\\frac\{1\}\{N\}\\sum\_\{i\}\\mathbb\{E\}\[\\\|\\nabla F\(\\theta\)\-\\nabla F\_\{i\}\(\\theta\)\\\|^\{2\}\]\. Since the adapter removes client\-specific directions from the base gradient,κeff2≤κ2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}\\leq\\kappa^\{2\}to first order under a shared\-subspace condition \(Appendix\); we also track it empirically\. To characterize the main result, we impose an assumption in the following\.

###### Assumption 1\(Adapter Approximate Stationarity\)\.

Along the local iterates,𝔼⁡\[‖∇ϕℒθ,ϕii‖2\]≤εϕ\\mathbb\{E\}\[\\\|\\nabla\_\{\\phi\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}\\\|^\{2\}\]\\leq\\varepsilon\_\{\\phi\}; moreover,ℒθ,⋅i\\mathcal\{L\}^\{i\}\_\{\\theta,\\cdot\}isμϕ\\mu\_\{\\phi\}\-PL inϕ\\phiand∇θℒθ​ϕi\\nabla\_\{\\theta\}\\mathcal\{L\}^\{i\}\_\{\\theta\\phi\}isLθ,ϕL\_\{\\theta,\\phi\}\-Lipschitz inϕ\\phi\.

Because adapters are trained onlyEEsteps per round,∇θℒθ,ϕii\\nabla\_\{\\theta\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}is a biased estimate of∇Fi​\(θ\)\\nabla F\_\{i\}\(\\theta\)\. Assumption 1 bounds this bias by\(Lθ​ϕ/μϕ\)2​εϕ\(L\_\{\\theta\\phi\}/\\mu\_\{\\phi\}\)^\{2\}\\varepsilon\_\{\\phi\}, givingC​εϕC\\varepsilon\_\{\\phi\}withC=𝒪⁡\(Lθ​ϕ2/μϕ2\)C=\\mathcal\{O\}\(L\_\{\\theta\\phi\}^\{2\}/\\mu\_\{\\phi\}^\{2\}\)\.

###### Theorem 1\.

\(pFedDPO Convergence\) Under Assumptions 3\.4\-4\.1 of\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\)and with LoRA rankr≥1r\\geq 1, letη≤min​\{1/\(8​L​E\),S/\(16​L​N​E\)\}\\eta\\leq\\text\{min\}\\\{1/\(8LE\),S/\(16LNE\)\\\}, whereL\>0L\>0is smoothness constant ofFF, then afterRRrounds of pFedDPO:

1R​∑r=0R−1𝔼⁡\[‖∇F​\(θr\)‖2\]≤2​\(F⁡\(θ0\)−F∗\)η​E​R\+8​L​η​ζg2S\+16​L2​η2​E​κeff2\+16​L2​η2​E2​ζg2S\+C​εϕ,\\begin\{split\}&\\frac\{1\}\{R\}\\sum\_\{r=0\}^\{R\-1\}\\mathbb\{E\}\[\\\|\\nabla F\(\\theta^\{r\}\)\\\|^\{2\}\]\\leq\\frac\{2\(F\(\\theta^\{0\}\)\-F^\{\*\}\)\}\{\\eta ER\}\+\\frac\{8L\\eta\\zeta^\{2\}\_\{g\}\}\{S\}\\\\ &\+16L^\{2\}\\eta^\{2\}E\\kappa^\{2\}\_\{\\text\{eff\}\}\+\\frac\{16L^\{2\}\\eta^\{2\}E^\{2\}\\zeta^\{2\}\_\{g\}\}\{S\}\+C\\varepsilon\_\{\\phi\},\\end\{split\}\(2\)whereζg2\\zeta^\{2\}\_\{g\}is the variance of gradients,F∗:=infθ​F​\(θ\)F^\{\*\}:=\\text\{inf\}\_\{\\theta\}F\(\\theta\)\.

Proof sketch\.We apply Thm\. 5\.1 of \(Jiang 2026\) toFF\. Each client’s base gradient decomposes as∇θℒθ,ϕii=∇Fi​\(θ\)\+bi\\nabla\_\{\\theta\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}=\\nabla F\_\{i\}\(\\theta\)\+b\_\{i\}with‖bi‖2≤\(Lθ​ϕ/μϕ\)2​εϕ\\\|b\_\{i\}\\\|^\{2\}\\leq\(L\_\{\\theta\\phi\}/\\mu\_\{\\phi\}\)^\{2\}\\varepsilon\_\{\\phi\}\(Assumption 1\)\. The heterogeneity of\{∇Fi\}\\\{\\nabla F\_\{i\}\\\}isκeff2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}, so Lemmas C\.1–C\.5 of \(Jiang 2026\) apply withκ2→κeff2\\kappa^\{2\}\\to\\kappa^\{2\}\_\{\\mathrm\{eff\}\}, and the bias addsC​εϕC\\varepsilon\_\{\\phi\}after telescoping\. Since adapters are never aggregated,∇ϕℒi\\nabla\_\{\\phi\}\\mathcal\{L\}^\{i\}does not enter the drift\. Full proof in Appendix\.

Compared to vanilla FedDPO \(Theorem 5\.1 of\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\)\),κ2\\kappa^\{2\}is replaced byκeff2\\kappa^\{2\}\_\{\\text\{eff\}\}\(improvement\), andC​εϕ≥0C\\varepsilon\_\{\\phi\}\\geq 0is added \(cost\)\. The net bound is tighter than vanilla FedDPO whenever the reduction16​L2​η2​E​\(κ2−κeff2\)16L^\{2\}\\eta^\{2\}E\(\\kappa^\{2\}\-\\kappa^\{2\}\_\{\\text\{eff\}\}\)exceedsC​εϕC\\varepsilon\_\{\\phi\}\. We expect both the improvement and the cost to grow withrr: higherrrcan reduceκeff2\\kappa^\{2\}\_\{\\text\{eff\}\}further, but larger adapters may be harder to converge inEEsteps, increasingεϕ\\varepsilon\_\{\\phi\}\. This suggests a non\-monotone relationship betweenrrand performance, with an optimal rank balancing the two effects\.

## Experiments

We evaluate on Stanford SHP\([Esfandiari et al\. 2021](https://arxiv.org/html/2610.12885#bib.bib31)\)\(main result\) and Anthropic hh\-rlhf\([Bai et al\. 2022](https://arxiv.org/html/2610.12885#bib.bib20)\)\(in Appendix\), each withN=5N=5agents holding 120 disjoint non\-IID preference pairs\. The backbone is DistilGPT\-2 \(≈\\approx82M parameters\)\. LoRA targets the attention projection \(“c\_attn”\) with default rankr=8r=8andα=16\\alpha=16\(the scaling parameter\)\. We compare three methods: Centralized DPO \(pooled data\), Vanilla FedDPO \(full model aggregation,S=5S=5\), and pFedDPO \(base \+ private LoRA,S=5S=5\)\. All methods use AdamW \(η=10−4\\eta=10^\{\-4\}\),β=0\.2\\beta=0\.2, gradient clipping at 1\.0, batch size 4,E=3E=3local steps, andR=80R=80communication rounds\. A full list is in Appendix\.

Comparative study\.As Figure[2](https://arxiv.org/html/2610.12885#Sx4.F2)shows, centralized DPO \(dashed grey\) shows the highest gradient norm, highest and most unstable training loss \(oscillating between 0\.0 and 1\.4\), and the lowest preference accuracy \(∼\\sim0\.75 at round 80, accuracy on each agent’s training pairs\)\. Both vanilla FedDPO and pFedDPO substantially outperform it\. This is not a failure of centralized training in principle, but a consequence of the experimental regime: centralized DPO processes batch size 4 per gradient step, while each FedDPO round aggregates acrossS=5S=5agents \(effective batch 20\)\. This places the experiment in the computation\-dominated regime of Theorem 5\.1 in\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\), where larger effective local computation per round accelerates convergence\. Centralized DPO is therefore not an oracle in this setup\. It thus serves as a single\-learner reference, consistent with theζg2/S\\zeta^\{2\}\_\{g\}/Sterm in Theorem[1](https://arxiv.org/html/2610.12885#Thmtheorem1)\. pFedDPO attains a lower gradient norm than vanilla FedDPO with comparable loss and accuracy\.

Figure 2:Comparison of baselines with SHP data\.Figure 3:Per\-client preference accuracy with SHP data\.Additionally, Figure[3](https://arxiv.org/html/2610.12885#Sx4.F3)depicts that all five agents improve from∼\\sim0\.30–0\.60 at round 0 to∼\\sim0\.75–0\.85 by round 80 under pFedDPO\. The spread at round 80 is roughly 10 percentage points \(Agent 5 highest at∼\\sim0\.85, Agent 2 lowest at∼\\sim0\.75\), consistent with heterogeneous preferences across agents rather than a single dominant agent driving the average\. The mean accuracy \(dashed\) reaches approximately 0\.80\.

Figure 4:Low rankrrablation with SHP data\.LoRA Rank Ablation\.Figure[4](https://arxiv.org/html/2610.12885#Sx4.F4)ablates LoRA rankr∈\{2,4,8,16\}r\\in\\\{2,4,8,16\\\}\. In panel \(a\), reward accuracy curves for all four ranks are broadly similar \(∼\\sim0\.78–0\.82 at round 80\), with modest differences across ranks\. Panel \(b\) reveals a more nuanced picture:r=4r=4achieves the highest final reward accuracy \(∼\\sim0\.82\) but with the highest gradient norm, whiler=8r=8achieves the lowest final gradient norm \(∼\\sim0\.02\)\.r=16r=16shows a higher gradient norm thanr=8r=8, andr=16r=16has the lowest bar in final accuracy\. This non\-monotone relationship between rank and accuracy is consistent with the trade\-off suggested by Theorem[1](https://arxiv.org/html/2610.12885#Thmtheorem1): higher rank reducesκeff2\\kappa^\{2\}\_\{\\text\{eff\}\}\(improving convergence\) but increases the adapter\-bias termC​εϕC\\varepsilon\_\{\\phi\}\(harder to converge a larger adapter inE=3E=3local steps\)\. The optimal rank balances these two competing effects, and the data suggestsr=4r=4orr=8r=8is near\-optimal for this model and data regime\. We user=8r=8as the default since it achieves the best gradient norm, which is the primary metric in our theoretical analysis\.

Figure 5:Gradient diversityκ2\\kappa^\{2\}ablation with SHP data\.Figure 6:ParticipationSSablation with SHP data\.Gradient Diversity and Participation Ablation\.As shown in Figure[5](https://arxiv.org/html/2610.12885#Sx4.F5)\(a\), pFedDPO consistently achieves a lower gradient norm than vanilla FedDPO across all 80 rounds, with the gap largest in the later rounds as adapters accumulate client\-specific knowledge\. Figure[5](https://arxiv.org/html/2610.12885#Sx4.F5)\(b\) shows the empiricalκ2\\kappa^\{2\}measured every 10 rounds on pFedDPO’s base model gradients\. Starting from∼\\sim1500 at round 0,κ2\\kappa^\{2\}drops sharply to∼\\sim350 by round 10 and continues to decrease gradually to approximately 150–200 by round 70\. The decrease is not monotone \(minor fluctuations around rounds 40–50\) and partly reflects overall training progress\. At initialization the adapters are zero \(Bi=0B\_\{i\}=0\), so the starting value coincides with vanilla FedDPO’sκ2\\kappa^\{2\}\. The∼\\sim85–90% reduction is consistent with theκ2→κeff2\\kappa^\{2\}\\to\\kappa^\{2\}\_\{\\mathrm\{eff\}\}mechanism of Theorem[1](https://arxiv.org/html/2610.12885#Thmtheorem1)\. Figure[6](https://arxiv.org/html/2610.12885#Sx4.F6)shows the participation ablationS∈\{1,3,5\}S\\in\\\{1,3,5\\\}for both pFedDPO \(left\) and vanilla FedDPO \(right\)\. In both cases the orderingS=5\>S=3\>S=1S=5\>S=3\>S=1holds, consistent with theζg2/S\\zeta\_\{g\}^\{2\}/Svariance floor in Theorem 5\.1 of\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\)\. AtS=1S=1, both methods struggle: the gradient norm remains near 1\.0 throughout with high variance for pFedDPO, and similarly for vanilla FedDPO\. The advantage of personalization is clearest atS=5S=5andS=3S=3, where pFedDPO’s lower gradient norm floor is visible\. AtS=1S=1theζg2/S\\zeta^\{2\}\_\{g\}/Svariance term dominates and the personalization benefit fromκeff2\\kappa^\{2\}\_\{\\text\{eff\}\}is masked\.

## Related Work

DPO\([Rafailov et al\. 2023](https://arxiv.org/html/2610.12885#bib.bib1)\)and its extensions\([Amini et al\. 2024](https://arxiv.org/html/2610.12885#bib.bib11);[Park et al\. 2024](https://arxiv.org/html/2610.12885#bib.bib12);[Xiao et al\. 2024](https://arxiv.org/html/2610.12885#bib.bib13)\)focus on the centralized, single\-model setting\. FedDPO and DecDPO \(decentralized DPO\) were proposed and analyzed in\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\), establishing the first convergence guarantees for preference optimization in distributed settings, where prior work focuses primarily on learning\([Saadati et al\. 2024](https://arxiv.org/html/2610.12885#bib.bib30);[Saadati et al\. 2025](https://arxiv.org/html/2610.12885#bib.bib23);[Esfandiari et al\. 2021](https://arxiv.org/html/2610.12885#bib.bib31)\)and more recently agentic AI systems\([Rillo et al\. 2026](https://arxiv.org/html/2610.12885#bib.bib29)\)\. Our work extends this by addressing the personalization gap they identify as a limitation\. Existing federated DPO approaches\([Spadea and Seneviratne 2025](https://arxiv.org/html/2610.12885#bib.bib14);[Zhu et al\. 2026](https://arxiv.org/html/2610.12885#bib.bib2)\)do not address this theoretical gap: they either treat all clients symmetrically without personalization\([Spadea and Seneviratne 2025](https://arxiv.org/html/2610.12885#bib.bib14)\)or introduce personalization empirically without bounding how it improves convergence\([Zhu et al\. 2026](https://arxiv.org/html/2610.12885#bib.bib2)\)\. Personalized federated learning with LoRA has been studied in\([Yi et al\. 2023](https://arxiv.org/html/2610.12885#bib.bib15);[Qi et al\. 2024](https://arxiv.org/html/2610.12885#bib.bib16);[Song et al\. 2026](https://arxiv.org/html/2610.12885#bib.bib17);[Gu et al\. 2026](https://arxiv.org/html/2610.12885#bib.bib18);[Lee et al\. 2026](https://arxiv.org/html/2610.12885#bib.bib19)\), but not in the context of preference optimization, where the trajectory\-level DPO objective introduces additional variance and heterogeneity that require explicit analysis\. We refer interested readers to Appendix for more related work and include a methodological comparison table\.

## Conclusion

We propose pFedDPO, a personalized federated DPO framework where each client trains a private LoRA adapter on top of a shared global base model\. The base model is synchronized across clients each round, while adapters remain local, preserving personalization without increasing communication compared with standard FedDPO\. Our analysis shows this design better handles client heterogeneity while introducing only a small bias that diminishes over training\. Empirically, pFedDPO consistently lowers the gradient norm relative to vanilla FedDPO while keeping preference accuracy competitive across all clients\.

## Acknowledgments

This work was supported by the COALESCE: COntext Aware LEarning for Sustainable CybEr\-Agricultural Systems \(NSF Frontier \#1954556\)\.

## References

- Aminiet al\.\(2024\)A\. Amini, T\. Vieira, and R\. CotterellDirect preference optimization with an offset\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 9954–9972\.Cited by:[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1)\.
- Baiet al\.\(2022\)Y\. Bai, A\. Jones, K\. Ndousse, A\. Askell, A\. Chen, N\. DasSarma, D\. Drain, S\. Fort, D\. Ganguli, T\. Henighan,et al\.Training a helpful and harmless assistant with reinforcement learning from human feedback\.arXiv preprint arXiv:2204\.05862\.Cited by:[Datasets\.](https://arxiv.org/html/2610.12885#Sx13.SSx8.SSS0.Px1.p1.1),[Experiments](https://arxiv.org/html/2610.12885#Sx4.p1.1)\.
- Bianet al\.\(2025\)J\. Bian, L\. Wang, L\. Zhang, and J\. XuLoRA\-fair: federated lora fine\-tuning with aggregation and initialization refinement\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 3737–3746\.Cited by:[Discussion on pFedDPO](https://arxiv.org/html/2610.12885#Sx9.p1.1)\.
- Bianet al\.\(2026\)J\. Bian, L\. Wang, L\. Zhang, and J\. XuFedalt: federated fine\-tuning through adaptive local training with rest\-of\-world lora\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 19728–19736\.Cited by:[Discussion on pFedDPO](https://arxiv.org/html/2610.12885#Sx9.p1.1)\.
- Choet al\.\(2024\)Y\. J\. Cho, L\. Liu, Z\. Xu, A\. Fahrezi, and G\. JoshiHeterogeneous lora for federated fine\-tuning of on\-device foundation models\.InProceedings of the 2024 conference on empirical methods in natural language processing,pp\. 12903–12913\.Cited by:[Discussion on pFedDPO](https://arxiv.org/html/2610.12885#Sx9.p1.1)\.
- Esfandiariet al\.\(2021\)Y\. Esfandiari, S\. Y\. Tan, Z\. Jiang, A\. Balu, E\. Herron, C\. Hegde, and S\. SarkarCross\-gradient aggregation for decentralized learning from non\-iid data\.InInternational conference on machine learning,pp\. 3036–3046\.Cited by:[Experiments](https://arxiv.org/html/2610.12885#Sx4.p1.1),[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1)\.
- Ethayarajhet al\.\(2021\)K\. Ethayarajh, Y\. Choi, and S\. SwayamdiptaUnderstanding dataset difficulty with v\-usable information \(2021\)\.URL https://arxiv\. org/abs/2110\.08420\.Cited by:[Datasets\.](https://arxiv.org/html/2610.12885#Sx13.SSx8.SSS0.Px1.p1.1)\.
- Guet al\.\(2026\)H\. Gu, J\. Li, M\. Peng, X\. Zhang, Y\. Ma, and X\. HuangD\-lora: a dual low\-rank adaptation framework for cost\-efficient personalized federated learning\.InICASSP 2026\-2026 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1046–1050\.Cited by:[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.Lora: low\-rank adaptation of large language models\.\.International Conference on Learning Representations1\(2\),pp\. 3\.Cited by:[Introduction](https://arxiv.org/html/2610.12885#Sx1.p2.1),[Background and Problem Setup](https://arxiv.org/html/2610.12885#Sx2.p2.1)\.
- Huanget al\.\(2023\)C\. Huang, M\. Tang, Q\. Ma, J\. Huang, and X\. LiuPromoting collaboration in cross\-silo federated learning: challenges and opportunities\.IEEE Communications Magazine62\(4\),pp\. 82–88\.Cited by:[Introduction](https://arxiv.org/html/2610.12885#Sx1.p1.1)\.
- Jiang \(2026\)Z\. JiangDistributed direct preference optimization\.arXiv preprint arXiv:2605\.20696\.Cited by:[Introduction](https://arxiv.org/html/2610.12885#Sx1.p1.1),[Proof of Theorem](https://arxiv.org/html/2610.12885#Sx11.p1.1),[Table 3](https://arxiv.org/html/2610.12885#Sx13.T3),[Table 3](https://arxiv.org/html/2610.12885#Sx13.T3.1.3.1.1.1),[Limitations](https://arxiv.org/html/2610.12885#Sx14.p1.1),[Background and Problem Setup](https://arxiv.org/html/2610.12885#Sx2.p1.1),[Background and Problem Setup](https://arxiv.org/html/2610.12885#Sx2.p1.2),[Personalized FedDPO \(pFedDPO\)](https://arxiv.org/html/2610.12885#Sx3.p2.1),[Personalized FedDPO \(pFedDPO\)](https://arxiv.org/html/2610.12885#Sx3.p5.1),[Experiments](https://arxiv.org/html/2610.12885#Sx4.p2.1),[Experiments](https://arxiv.org/html/2610.12885#Sx4.p5.1),[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1),[Additional Related Work](https://arxiv.org/html/2610.12885#Sx8.p1.1),[Table 1](https://arxiv.org/html/2610.12885#Sx9.T1.1.2.1.1.1),[Discussion on pFedDPO](https://arxiv.org/html/2610.12885#Sx9.p3.1),[Theorem 1](https://arxiv.org/html/2610.12885#Thmtheorem1.p1.1.1)\.
- Kuoet al\.\(2024\)K\. Kuo, A\. Raje, K\. Rajesh, and V\. SmithFederated lora with sparse communication\.arXiv preprint arXiv:2406\.05233\.Cited by:[Discussion on pFedDPO](https://arxiv.org/html/2610.12885#Sx9.p1.1)\.
- Leeet al\.\(2026\)R\. Lee, M\. Kim, F\. Rezk, R\. Li, S\. I\. Venieris, and T\. HospedalesFedP2\{\}^\{2\}eft: federated learning to personalize peft for multilingual llms\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 22805–22813\.Cited by:[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1)\.
- Liet al\.\(2019a\)H\. Li, L\. Yu, and W\. HeThe impact of gdpr on global technology development\.Journal of global information technology management22\(1\),pp\. 1–6\.Cited by:[Introduction](https://arxiv.org/html/2610.12885#Sx1.p1.1)\.
- Liet al\.\(2019b\)X\. Li, K\. Huang, W\. Yang, S\. Wang, and Z\. ZhangOn the convergence of fedavg on non\-iid data\.arXiv preprint arXiv:1907\.02189\.Cited by:[Introduction](https://arxiv.org/html/2610.12885#Sx1.p2.1)\.
- Moore and Frye \(2019\)W\. Moore and S\. FryeReview of hipaa, part 1: history, protected health information, and privacy and security rules\.Journal of nuclear medicine technology47\(4\),pp\. 269–272\.Cited by:[Introduction](https://arxiv.org/html/2610.12885#Sx1.p1.1)\.
- Nguyenet al\.\(2022\)D\. C\. Nguyen, Q\. Pham, P\. N\. Pathirana, M\. Ding, A\. Seneviratne, Z\. Lin, O\. Dobre, and W\. HwangFederated learning for smart healthcare: a survey\.ACM Computing Surveys \(Csur\)55\(3\),pp\. 1–37\.Cited by:[Introduction](https://arxiv.org/html/2610.12885#Sx1.p1.1)\.
- Parket al\.\(2024\)R\. Park, R\. Rafailov, S\. Ermon, and C\. FinnDisentangling length from quality in direct preference optimization\.InFindings of the Association for Computational Linguistics: ACL 2024,pp\. 4998–5017\.Cited by:[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1)\.
- Qiet al\.\(2024\)J\. Qi, Z\. Luan, S\. Huang, C\. Fung, H\. Yang, and D\. QianFdlora: personalized federated learning of large language model via dual lora tuning\.arXiv preprint arXiv:2406\.07925\.Cited by:[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1)\.
- Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.Advances in neural information processing systems36,pp\. 53728–53741\.Cited by:[Introduction](https://arxiv.org/html/2610.12885#Sx1.p1.1),[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1)\.
- Rilloet al\.\(2026\)L\. N\. Rillo, Z\. Jiang, N\. Saadati, A\. Balu, B\. Ganapathysubramanian, C\. Hegde, and S\. SarkarADKO: agentic decentralized knowledge optimization\.arXiv preprint arXiv:2605\.07863\.Cited by:[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1)\.
- Saadatiet al\.\(2025\)N\. Saadati, Z\. Jiang, A\. Balu, C\. Liu, C\. Hegde, and S\. SarkarFoundation model efficient fine\-tuning in centralized and federated settings\.In2025 IEEE International Conference on Big Data \(BigData\),pp\. 1857–1866\.Cited by:[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1),[Discussion on pFedDPO](https://arxiv.org/html/2610.12885#Sx9.p1.1)\.
- Saadatiet al\.\(2024\)N\. Saadati, M\. Pham, N\. Saleem, J\. R\. Waite, A\. Balu, Z\. Jiang, C\. Hegde, and S\. SarkarDimat: decentralized iterative merging\-and\-training for deep learning models\.In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),pp\. 27507–27517\.Cited by:[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1)\.
- Sa’adet al\.\(2025\)A\. Sa’ad, M\. Felemban, and A\. TabbakhCEFEEL: communication\-efficient federated learning for personal assistant applications\.IEEE Open Journal of the Computer Society\.Cited by:[Introduction](https://arxiv.org/html/2610.12885#Sx1.p1.1)\.
- Songet al\.\(2026\)D\. Song, H\. Cai, H\. Zhang, B\. Sheng, J\. Zhou, M\. Ye, and F\. XiaoCA\-pfl: client\-adaptive parameter\-efficient fine\-tuning for personalized federated learning\.InProceedings of the ACM Web Conference 2026,pp\. 5600–5611\.Cited by:[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1)\.
- Spadea and Seneviratne \(2025\)F\. Spadea and O\. SeneviratneFederated fine\-tuning of large language models: kahneman\-tversky vs\. direct preference optimization\.InCompanion Proceedings of the ACM on Web Conference 2025,pp\. 1757–1760\.Cited by:[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1),[Additional Related Work](https://arxiv.org/html/2610.12885#Sx8.p2.1),[Additional Related Work](https://arxiv.org/html/2610.12885#Sx8.p4.1),[Table 1](https://arxiv.org/html/2610.12885#Sx9.T1.1.3.1.1.1)\.
- Sunet al\.\(2024\)Y\. Sun, Z\. Li, Y\. Li, and B\. DingImproving lora in privacy\-preserving federated learning\.arXiv preprint arXiv:2403\.12313\.Cited by:[Discussion on pFedDPO](https://arxiv.org/html/2610.12885#Sx9.p1.1)\.
- Wanget al\.\(2023\)Y\. Wang, Y\. Kordi, S\. Mishra, A\. Liu, N\. A\. Smith, D\. Khashabi, and H\. HajishirziSelf\-instruct: aligning language models with self\-generated instructions\.InProceedings of the 61st annual meeting of the association for computational linguistics \(volume 1: long papers\),pp\. 13484–13508\.Cited by:[Introduction](https://arxiv.org/html/2610.12885#Sx1.p1.1)\.
- Xiaoet al\.\(2024\)T\. Xiao, Y\. Yuan, H\. Zhu, M\. Li, and V\. G\. HonavarCal\-dpo: calibrated direct preference optimization for language model alignment\.Advances in Neural Information Processing Systems37,pp\. 114289–114320\.Cited by:[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1)\.
- Xieet al\.\(2026\)S\. Xie, D\. Wen, C\. You, Q\. Chen, M\. Bennis, and K\. HuangFedLoDrop: federated lora with dropout for generalized llm fine\-tuning\.IEEE Journal on Selected Areas in Communications\.Cited by:[Discussion on pFedDPO](https://arxiv.org/html/2610.12885#Sx9.p1.1)\.
- Yiet al\.\(2023\)L\. Yi, H\. Yu, G\. Wang, X\. Liu, and X\. LiPFedLoRA: model\-heterogeneous personalized federated learning with lora tuning\.arXiv preprint arXiv:2310\.13283\.Cited by:[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1),[Discussion on pFedDPO](https://arxiv.org/html/2610.12885#Sx9.p1.1)\.
- Zhuet al\.\(2026\)K\. Zhu, L\. Yi, Z\. Zhao, Z\. Qi, H\. Yu, and Q\. HuFedPDPO: federated personalized direct preference optimization for large language model alignment\.arXiv preprint arXiv:2603\.19741\.Cited by:[Introduction](https://arxiv.org/html/2610.12885#Sx1.p2.1),[Limitations](https://arxiv.org/html/2610.12885#Sx14.p1.1),[Related Work](https://arxiv.org/html/2610.12885#Sx5.p1.1),[Additional Related Work](https://arxiv.org/html/2610.12885#Sx8.p3.1),[Table 1](https://arxiv.org/html/2610.12885#Sx9.T1.1.4.1.1.1)\.

## Additional Related Work

FedDPO\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\)establishes the first convergence guarantees for distributed DPO under full and partial participation, staleness, and decentralized topologies, but targets a single shared policy and does not support personalization\. Its bound contains an irreducibleκ2\\kappa^\{2\}floor that our work directly reduces\.

Spadea and Seneviratne\([Spadea and Seneviratne 2025](https://arxiv.org/html/2610.12885#bib.bib14)\)compare Kahneman\-Tversky Optimization \(KTO\) and DPO in a federated LLM fine\-tuning setting using Alpaca\-7B\. They show that KTO outperforms DPO across MT\-Bench\-1, Vicuna, and AdvBench benchmarks, particularly under redistributed \(non\-paired\) data where DPO cannot be applied\. Their work is purely empirical with no convergence analysis, and does not address personalization: all clients train toward a single global model\. The key contribution is demonstrating KTO’s resilience to non\-IID data due to its single\-response feedback requirement, which sidesteps the paired\-response constraint of DPO\.

FedPDPO\([Zhu et al\. 2026](https://arxiv.org/html/2610.12885#bib.bib2)\)uses a globally shared LoRA adapter with a personalized client\-specific LLM head, a client\-specific explicit reward head, and a bottleneck adapter to balance global and local features\. Its theory is described as establishing a probabilistic foundation and soundness\. It transmits shared LoRA updates of dimension𝒪⁡\(r⁡\(m\+n\)\)\\mathcal\{O\}\(r\(m\+n\)\)per client, achieving up to 4\.80% average accuracy improvement\. However, its adapters are globally shared rather than private, and its theory does not provide a rate\-explicit bound relating personalization toκ2\\kappa^\{2\}\. pFedDPO makes the opposite trade\-off: fulldd\-dimensional base updates \(same volume as vanilla FedDPO\) in exchange for fully private adapters and an explicit convergence guarantee\.

The key differentiator of pFedDPO is therefore with a rate\-explicit convergence bound that makes the effect of personalization explicit\. Unlike FedPDPO, pFedDPO never transmits adapter weights, keeping all personalization parameters on\-device, at a higher upload volume than FedPDPO\. Unlike Spadea and Seneviratne’s work\([Spadea and Seneviratne 2025](https://arxiv.org/html/2610.12885#bib.bib14)\), pFedDPO explicitly personalizes to each client’s preference distribution\. Please see Table[1](https://arxiv.org/html/2610.12885#Sx9.T1)for methodological comparison\.

## Discussion on pFedDPO

Figure[1](https://arxiv.org/html/2610.12885#Sx3.F1)shows the specific detail of how pFedDPO works schematically\. A closely related line of work\([Cho et al\. 2024](https://arxiv.org/html/2610.12885#bib.bib22);[Saadati et al\. 2025](https://arxiv.org/html/2610.12885#bib.bib23);[Kuo et al\. 2024](https://arxiv.org/html/2610.12885#bib.bib24);[Sun et al\. 2024](https://arxiv.org/html/2610.12885#bib.bib25);[Yi et al\. 2023](https://arxiv.org/html/2610.12885#bib.bib15);[Bian et al\. 2025](https://arxiv.org/html/2610.12885#bib.bib26);[Bian et al\. 2026](https://arxiv.org/html/2610.12885#bib.bib27);[Xie et al\. 2026](https://arxiv.org/html/2610.12885#bib.bib28)\)applies parameter\-efficient federated fine\-tuning by training LoRA adapters locally and periodically aggregating or averaging those adapters across clients\. In contrast,pFedDPO intentionally keeps LoRA adapters private and only synchronizes the shared base model\. This design is motivated by the nature of preference alignment, where client heterogeneity reflects genuine differences in user values, style, or alignment objectives rather than noise to be averaged away\.

When LoRA adapters are shared globally, they must simultaneously encode both:

- •task\-level adaptation of the shared model, and
- •client\-specific preference information\.

Under heterogeneous preference distributions, these objectives can conflict\. Averaging adapters across clients may therefore dilute personalized signals, causingpreference interference: updates that improve alignment for one client can harm another\. This is especially pronounced in DPO, where gradients are driven by pairwise preference comparisons and are highly sensitive to local preference distributions\. pFedDPO avoids this by separating these roles:

- •the base model captures globally transferable alignment knowledge shared across clients;
- •the private LoRA adapter captures client\-specific residual preferences\.

This decomposition makes the global model responsible for learning common alignment structure, while the adapters absorb personalization\. As a result, the server never aggregates incompatible preference\-specific parameters\.

A second advantage is privacy\. LoRA adapters trained on preference data can encode highly sensitive information about user behavior, style, or preference distributions\. Uploading them, even though lightweight, may leak personalization signals\. Keeping adapters local provides a stronger privacy boundary than methods that transmit adapters to the server\. From an optimization perspective, private adapters also reduce effective client drift\. Since some heterogeneity is absorbed locally, the shared model sees less conflicting gradient signal during aggregation, which leads to more stable federated optimization\. This is reflected in our analysis through a smaller effective heterogeneity term relative to vanilla FedDPO\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\)\. Finally, communication remains unchanged relative to standard FedDPO: only updates to the shared base model are transmitted\. Compared with adapter\-sharing methods, pFedDPO therefore offers a better tradeoff between:

- •global coordination \(via shared base model\),
- •local personalization \(via private adapters\),
- •privacy preservation, and
- •robustness to preference heterogeneity\.

Table 1:Comparison of federated fine\-tuning methods\. ✓=yes; ✗=no;∼\\sim=partial\. Comm\.Vol\.=per\-round communication volume per selected client\. pFedDPO transmitsdd\-dimensional base updates \(same as FedDPO\), not LoRA dimensions\. Its distinguishing property is a rate\-explicit bound capturing how private adapters reduceκ2\\kappa^\{2\}toκeff2\\kappa^\{2\}\_\{\\text\{eff\}\}\.
## Hyperparameters

The specific hyperparameters used in the empirical evaluation are listed in Table[2](https://arxiv.org/html/2610.12885#Sx10.T2)\.

Table 2:Hyperparameter Setup\.
## Proof of Theorem[1](https://arxiv.org/html/2610.12885#Thmtheorem1)

The proof follows the structure of Theorem 5\.1 in\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\)\. We highlight only the modification due to LoRA personalization\.

###### Proof\.

Letbir,e:=∇θℒθir,e,ϕir,ei−∇Fi​\(θir,e\)b\_\{i\}^\{r,e\}:=\\nabla\_\{\\theta\}\\mathcal\{L\}^\{i\}\_\{\\theta\_\{i\}^\{r,e\},\\phi\_\{i\}^\{r,e\}\}\-\\nabla F\_\{i\}\(\\theta\_\{i\}^\{r,e\}\)\. \(i\) Bias:μϕ\\mu\_\{\\phi\}\-PL implies the error bounddist⁡\(ϕ,arg⁡min\)≤‖∇ϕℒi‖/μϕ\\mathrm\{dist\}\(\\phi,\\arg\\min\)\\leq\\\|\\nabla\_\{\\phi\}\\mathcal\{L\}^\{i\}\\\|/\\mu\_\{\\phi\}; with cross\-Lipschitzness,𝔼​‖bir,e‖2≤\(Lθ​ϕ/μϕ\)2​εϕ\\mathbb\{E\}\\\|b\_\{i\}^\{r,e\}\\\|^\{2\}\\leq\(L\_\{\\theta\\phi\}/\\mu\_\{\\phi\}\)^\{2\}\\varepsilon\_\{\\phi\}\. \(ii\) Descent: byLL\-smoothness ofFFand⟨a,a\+b⟩≥12​‖a‖2−12​‖b‖2\\langle a,a\+b\\rangle\\geq\\frac\{1\}\{2\}\\\|a\\\|^\{2\}\-\\frac\{1\}\{2\}\\\|b\\\|^\{2\}, the bias addsη​E2​b¯2\\frac\{\\eta E\}\{2\}\\bar\{b\}^\{2\}per round; dividing byη​E​R/2\\eta ER/2givesc0​\(Lθ​ϕ/μϕ\)2​εϕ=:C​εϕc\_\{0\}\(L\_\{\\theta\\phi\}/\\mu\_\{\\phi\}\)^\{2\}\\varepsilon\_\{\\phi\}=:C\\varepsilon\_\{\\phi\}\. \(iii\) Drift: Lemmas C\.1–C\.5 of \(Jiang 2026\) with∇ℒi→∇Fi\\nabla\\mathcal\{L\}^\{i\}\\to\\nabla F\_\{i\}give heterogeneityκeff2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}; the bias’ drift contribution is𝒪⁡\(L2​η2​E2\)​εϕ\\mathcal\{O\}\(L^\{2\}\\eta^\{2\}E^\{2\}\)\\varepsilon\_\{\\phi\}, absorbed intoC​εϕC\\varepsilon\_\{\\phi\}forη≤1/\(8​L​E\)\\eta\\leq 1/\(8LE\)\.

∎

To facilitate the understanding of the proof in the above, we include another detailed analysis on whyκeff2≤κ2\\kappa^\{2\}\_\{\\text\{eff\}\}\\leq\\kappa^\{2\}and present the intuition behind it\.

## Whyκeff2≤κ2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}\\leq\\kappa^\{2\}: Technical Analysis and Intuition

### Setup

Partition the base parameters asθ=\(θ𝒜,θ𝒩\)\\theta=\(\\theta\_\{\\mathcal\{A\}\},\\theta\_\{\\mathcal\{N\}\}\), whereθ𝒜=\(W1,…,WM\)\\theta\_\{\\mathcal\{A\}\}=\(W\_\{1\},\\dots,W\_\{M\}\)collects the LoRA\-adapted matricesWℓ∈ℝm×nW\_\{\\ell\}\\in\\mathbb\{R\}^\{m\\times n\}andθ𝒩\\theta\_\{\\mathcal\{N\}\}all remaining parameters\. In our experiments,θ𝒜\\theta\_\{\\mathcal\{A\}\}consists of theM=6M=6c\_attnmatrices of DistilGPT\-2 \(m=768m=768,n=2304n=2304\), andθ𝒩\\theta\_\{\\mathcal\{N\}\}contains the MLP, embedding, and LayerNorm parameters\. Clientii’s adapter isϕi=\{\(Bi,ℓ,Ai,ℓ\)\}ℓ=1M\\phi\_\{i\}=\\\{\(B\_\{i,\\ell\},A\_\{i,\\ell\}\)\\\}\_\{\\ell=1\}^\{M\}withBi,ℓ∈ℝm×rB\_\{i,\\ell\}\\in\\mathbb\{R\}^\{m\\times r\},Ai,ℓ∈ℝr×nA\_\{i,\\ell\}\\in\\mathbb\{R\}^\{r\\times n\}, and effective weightsWℓ\+s​Bi,ℓ​Ai,ℓW\_\{\\ell\}\+sB\_\{i,\\ell\}A\_\{i,\\ell\}, wheres=α/rs=\\alpha/r\.

#### Adapters as private low\-rank shifts\.

LetΓi∈ℝd\\Gamma\_\{i\}\\in\\mathbb\{R\}^\{d\}places​Bi,ℓ​Ai,ℓsB\_\{i,\\ell\}A\_\{i,\\ell\}on blockℓ\\ellofθ𝒜\\theta\_\{\\mathcal\{A\}\}and zeros onθ𝒩\\theta\_\{\\mathcal\{N\}\}\. Then the adapted model is simply the base model evaluated at shifted weights:

ℒθ,ϕii=ℒθ\+Γii,∇θℒθ,ϕii=∇ℒθ\+Γii\.\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}=\\mathcal\{L\}^\{i\}\_\{\\theta\+\\Gamma\_\{i\}\},\\qquad\\nabla\_\{\\theta\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}=\\nabla\\mathcal\{L\}^\{i\}\_\{\\theta\+\\Gamma\_\{i\}\}\.\(3\)Vanilla FedDPO corresponds toΓi=0\\Gamma\_\{i\}=0, which is also the standard LoRA initialization \(Bi,ℓ=0B\_\{i,\\ell\}=0\)\.

#### The two heterogeneity constants\.

Writegi:=∇ℒθig\_\{i\}:=\\nabla\\mathcal\{L\}^\{i\}\_\{\\theta\}for clientii’s unadapted gradient andg¯:=∇ℒθ=1N​∑igi\\bar\{g\}:=\\nabla\\mathcal\{L\}\_\{\\theta\}=\\frac\{1\}\{N\}\\sum\_\{i\}g\_\{i\}\. For a family\{vi\}i=1N\\\{v\_\{i\}\\\}\_\{i=1\}^\{N\}of random vectors, define the norm

‖\{vi\}‖N:=\(1N​∑i=1N𝔼​‖vi‖2\)1/2,\\big\\\|\\\{v\_\{i\}\\\}\\big\\\|\_\{N\}:=\\Big\(\\tfrac\{1\}\{N\}\\textstyle\\sum\_\{i=1\}^\{N\}\\mathbb\{E\}\\\|v\_\{i\}\\\|^\{2\}\\Big\)^\{1/2\},which satisfies the triangle inequality\. The vanilla gradient diversity is

κ2=‖\{g¯−gi\}‖N2=1N​∑i=1N𝔼⁡\[‖∇ℒθ−∇ℒθi‖2\]\.\\kappa^\{2\}=\\big\\\|\\\{\\bar\{g\}\-g\_\{i\}\\\}\\big\\\|\_\{N\}^\{2\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{E\}\\big\[\\\|\\nabla\\mathcal\{L\}\_\{\\theta\}\-\\nabla\\mathcal\{L\}^\{i\}\_\{\\theta\}\\\|^\{2\}\\big\]\.\(4\)For pFedDPO, letFi​\(θ\):=minϕ⁡ℒθ,ϕiF\_\{i\}\(\\theta\):=\\min\_\{\\phi\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\}andF:=1N​∑iFiF:=\\frac\{1\}\{N\}\\sum\_\{i\}F\_\{i\}, with minimizerϕi∗\\phi\_\{i\}^\{\*\}and shiftΓi∗\\Gamma\_\{i\}^\{\*\}\. We assume the minimizing shiftΓi∗\\Gamma\_\{i\}^\{\*\}is unique, so thatFiF\_\{i\}is differentiable and, by the envelope theorem,∇Fi​\(θ\)=∇θℒθ,ϕi∗i=∇ℒθ\+Γi∗i\\nabla F\_\{i\}\(\\theta\)=\\nabla\_\{\\theta\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}^\{\*\}\}=\\nabla\\mathcal\{L\}^\{i\}\_\{\\theta\+\\Gamma\_\{i\}^\{\*\}\}\. The pair\(B,A\)\(B,A\)itself is never unique, since\(B​Q,Q−1​A\)\(BQ,Q^\{\-1\}A\)gives the same shift for any invertibleQQ\. Nothing below depends on that choice\. The residual heterogeneity is

κeff2=‖\{∇F−∇Fi\}‖N2=1N​∑i=1N𝔼⁡\[‖∇F​\(θ\)−∇Fi​\(θ\)‖2\]\.\\kappa^\{2\}\_\{\\mathrm\{eff\}\}=\\big\\\|\\\{\\nabla F\-\\nabla F\_\{i\}\\\}\\big\\\|\_\{N\}^\{2\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbb\{E\}\\big\[\\\|\\nabla F\(\\theta\)\-\\nabla F\_\{i\}\(\\theta\)\\\|^\{2\}\\big\]\.\(5\)Finally, we splitκ2=κ𝒜2\+κ𝒩2\\kappa^\{2\}=\\kappa^\{2\}\_\{\\mathcal\{A\}\}\+\\kappa^\{2\}\_\{\\mathcal\{N\}\}into the heterogeneity carried by adapted blocks and by non\-adapted parameters, respectively\.

### An Exact Identity at Adapter Stationarity

For blockℓ\\ellof clientii, letPi,ℓP\_\{i,\\ell\}be the orthogonal projector onto the column space ofBi,ℓB\_\{i,\\ell\}andQi,ℓ=Ai,ℓ⊤​\(Ai,ℓ​Ai,ℓ⊤\)\+​Ai,ℓQ\_\{i,\\ell\}=A\_\{i,\\ell\}^\{\\top\}\(A\_\{i,\\ell\}A\_\{i,\\ell\}^\{\\top\}\)^\{\+\}A\_\{i,\\ell\}the projector onto the row space ofAi,ℓA\_\{i,\\ell\}\. The adapter’s tangent space is

𝒯i,ℓ:=\{Bi,ℓX\+YAi,ℓ:X∈ℝr×n,Y∈ℝm×r\},\\mathcal\{T\}\_\{i,\\ell\}:=\\\{B\_\{i,\\ell\}X\+YA\_\{i,\\ell\}:\\ X\\in\\mathbb\{R\}^\{r\\times n\},\\ Y\\in\\mathbb\{R\}^\{m\\times r\}\\\},of dimensionr⁡\(m\+n\)−r2r\(m\+n\)\-r^\{2\}whenBi,ℓ,Ai,ℓB\_\{i,\\ell\},A\_\{i,\\ell\}have full rankrr\. It depends only on the column and row spaces, and hence is invariant to the reparameterization above\. In the Frobenius inner product,Z⟂𝒯i,ℓZ\\perp\\mathcal\{T\}\_\{i,\\ell\}if and only if there exists a matrixZZsuch thatBi,ℓ⊤​Z=0B\_\{i,\\ell\}^\{\\top\}Z=0andZ​Ai,ℓ⊤=0ZA\_\{i,\\ell\}^\{\\top\}=0\. The orthogonal projectors onto𝒯i,ℓ⟂\\mathcal\{T\}\_\{i,\\ell\}^\{\\perp\}and𝒯i,ℓ\\mathcal\{T\}\_\{i,\\ell\}are

Πi,ℓ⟂​\(Z\)=\(I−Pi,ℓ\)​Z​\(I−Qi,ℓ\),Πi,ℓ​\(Z\)=Pi,ℓ​Z\+\(I−Pi,ℓ\)​Z​Qi,ℓ\.\\begin\{split\}&\\Pi^\{\\perp\}\_\{i,\\ell\}\(Z\)=\(I\-P\_\{i,\\ell\}\)\\,Z\\,\(I\-Q\_\{i,\\ell\}\),\\\\ &\\Pi\_\{i,\\ell\}\(Z\)=P\_\{i,\\ell\}Z\+\(I\-P\_\{i,\\ell\}\)ZQ\_\{i,\\ell\}\.\\end\{split\}\(6\)LetΠi\\Pi\_\{i\}act asΠi,ℓ\\Pi\_\{i,\\ell\}on each block ofθ𝒜\\theta\_\{\\mathcal\{A\}\}and as zero onθ𝒩\\theta\_\{\\mathcal\{N\}\}, and letΠi⟂:=I−Πi\\Pi\_\{i\}^\{\\perp\}:=I\-\\Pi\_\{i\}, which is the identity onθ𝒩\\theta\_\{\\mathcal\{N\}\}\. SinceBi,ℓ​Ai,ℓ∈𝒯i,ℓB\_\{i,\\ell\}A\_\{i,\\ell\}\\in\\mathcal\{T\}\_\{i,\\ell\}, we haveΓi∈range⁡\(Πi\)\\Gamma\_\{i\}\\in\\mathrm\{range\}\(\\Pi\_\{i\}\)\.

###### Lemma 1\(Stationarity identity\)\.

LetGi,ℓG\_\{i,\\ell\}denote blockℓ\\ellof∇θℒθ,ϕii\\nabla\_\{\\theta\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}\. Then∇Bi,ℓℒθ,ϕii=s​Gi,ℓ​Ai,ℓ⊤\\nabla\_\{B\_\{i,\\ell\}\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}=s\\,G\_\{i,\\ell\}A\_\{i,\\ell\}^\{\\top\}and∇Ai,ℓℒθ,ϕii=s​Bi,ℓ⊤​Gi,ℓ\\nabla\_\{A\_\{i,\\ell\}\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}=s\\,B\_\{i,\\ell\}^\{\\top\}G\_\{i,\\ell\}\. Consequently, if∇ϕℒθ,ϕii=0\\nabla\_\{\\phi\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}=0, then

∇θℒθ,ϕii=Πi⟂​∇θℒθ,ϕii\.\\nabla\_\{\\theta\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}=\\Pi\_\{i\}^\{\\perp\}\\,\\nabla\_\{\\theta\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}\.\(7\)In particular,∇Fi\(θ\)=Πi⟂∇Fi\(θ\)\\nabla F\_\{i\}\(\\theta\)=\\Pi^\{\\perp\}\_\{i\}\\nabla F\_\{i\}\(\\theta\)withΠi\\Pi\_\{i\}built fromϕi∗\\phi\_\{i\}^\{\*\}\. That is, the base gradient after adaptation has no component in the adapter’s tangent space\.

###### Proof\.

The loss depends onWℓW\_\{\\ell\},Bi,ℓB\_\{i,\\ell\},Ai,ℓA\_\{i,\\ell\}only through the effective weightWℓ\+s​Bi,ℓ​Ai,ℓW\_\{\\ell\}\+sB\_\{i,\\ell\}A\_\{i,\\ell\}, so∇Wℓℒi=Gi,ℓ\\nabla\_\{W\_\{\\ell\}\}\\mathcal\{L\}^\{i\}=G\_\{i,\\ell\}and the chain rule gives the two adapter gradients\. If both vanish, thenGi,ℓ​Qi,ℓ=\(Gi,ℓ​Ai,ℓ⊤\)​\(Ai,ℓ​Ai,ℓ⊤\)\+​Ai,ℓ=0G\_\{i,\\ell\}Q\_\{i,\\ell\}=\(G\_\{i,\\ell\}A\_\{i,\\ell\}^\{\\top\}\)\(A\_\{i,\\ell\}A\_\{i,\\ell\}^\{\\top\}\)^\{\+\}A\_\{i,\\ell\}=0andPi,ℓ​Gi,ℓ=Bi,ℓ​\(Bi,ℓ⊤​Bi,ℓ\)\+​\(Bi,ℓ⊤​Gi,ℓ\)=0P\_\{i,\\ell\}G\_\{i,\\ell\}=B\_\{i,\\ell\}\(B\_\{i,\\ell\}^\{\\top\}B\_\{i,\\ell\}\)^\{\+\}\(B\_\{i,\\ell\}^\{\\top\}G\_\{i,\\ell\}\)=0\. HenceGi,ℓ=\(I−Pi,ℓ\)​Gi,ℓ​\(I−Qi,ℓ\)=Πi,ℓ⟂​Gi,ℓG\_\{i,\\ell\}=\(I\-P\_\{i,\\ell\}\)G\_\{i,\\ell\}\(I\-Q\_\{i,\\ell\}\)=\\Pi^\{\\perp\}\_\{i,\\ell\}G\_\{i,\\ell\}\. Onθ𝒩\\theta\_\{\\mathcal\{N\}\},Πi⟂\\Pi\_\{i\}^\{\\perp\}is the identity\. ∎

### From Stationarity to Reduced Heterogeneity

###### Proposition 1\.

LetΠ\\Pibe any orthogonal projector onℝd\\mathbb\{R\}^\{d\}whose range lies in the adapted blocksθ𝒜\\theta\_\{\\mathcal\{A\}\}, and letΠ⟂=I−Π\\Pi^\{\\perp\}=I\-\\Pi\. Define

ei:=∇Fi​\(θ\)−Π⟂​gi,δ:=‖\{ei\}‖N,κ∥2:=‖\{Π⁡\(g¯−gi\)\}‖N2\.e\_\{i\}:=\\nabla F\_\{i\}\(\\theta\)\-\\Pi^\{\\perp\}g\_\{i\},\\;\\delta:=\\big\\\|\\\{e\_\{i\}\\\}\\big\\\|\_\{N\},\\;\\kappa^\{2\}\_\{\\parallel\}:=\\big\\\|\\\{\\Pi\(\\bar\{g\}\-g\_\{i\}\)\\\}\\big\\\|\_\{N\}^\{2\}\.Then

κ𝒩−δ≤κeff≤κ2−κ∥2\+δ\.\\kappa\_\{\\mathcal\{N\}\}\-\\delta\\ \\leq\\ \\kappa\_\{\\mathrm\{eff\}\}\\ \\leq\\ \\sqrt\{\\kappa^\{2\}\-\\kappa^\{2\}\_\{\\parallel\}\}\+\\delta\.\(8\)In particular,κeff2<κ2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}<\\kappa^\{2\}wheneverδ<κ−κ2−κ∥2\\delta<\\kappa\-\\sqrt\{\\kappa^\{2\}\-\\kappa^\{2\}\_\{\\parallel\}\}\. This threshold is approximatelyκ∥2/\(2​κ\)\\kappa^\{2\}\_\{\\parallel\}/\(2\\kappa\)whenκ∥≪κ\\kappa\_\{\\parallel\}\\ll\\kappa\.

###### Proof\.

Averaging the definition ofeie\_\{i\}gives∇F=Π⟂​g¯\+e¯\\nabla F=\\Pi^\{\\perp\}\\bar\{g\}\+\\bar\{e\}withe¯=1N​∑iei\\bar\{e\}=\\frac\{1\}\{N\}\\sum\_\{i\}e\_\{i\}\. Hence

∇F−∇Fi=Π⟂​\(g¯−gi\)\+\(e¯−ei\)\.\\nabla F\-\\nabla F\_\{i\}=\\Pi^\{\\perp\}\(\\bar\{g\}\-g\_\{i\}\)\+\(\\bar\{e\}\-e\_\{i\}\)\.*Upper bound\.*By the triangle inequality for∥⋅∥N\\\|\\cdot\\\|\_\{N\},κeff≤‖\{Π⟂​\(g¯−gi\)\}‖N\+‖\{e¯−ei\}‖N\\kappa\_\{\\mathrm\{eff\}\}\\leq\\\|\\\{\\Pi^\{\\perp\}\(\\bar\{g\}\-g\_\{i\}\)\\\}\\\|\_\{N\}\+\\\|\\\{\\bar\{e\}\-e\_\{i\}\\\}\\\|\_\{N\}\. SinceΠ\\Piis an orthogonal projector,‖Π⟂​v‖2=‖v‖2−‖Π​v‖2\\\|\\Pi^\{\\perp\}v\\\|^\{2\}=\\\|v\\\|^\{2\}\-\\\|\\Pi v\\\|^\{2\}\. Hence, the first term equalsκ2−κ∥2\\sqrt\{\\kappa^\{2\}\-\\kappa^\{2\}\_\{\\parallel\}\}\. For the second, pointwise1N​∑i‖e¯−ei‖2=1N​∑i‖ei‖2−‖e¯‖2≤1N​∑i‖ei‖2\\frac\{1\}\{N\}\\sum\_\{i\}\\\|\\bar\{e\}\-e\_\{i\}\\\|^\{2\}=\\frac\{1\}\{N\}\\sum\_\{i\}\\\|e\_\{i\}\\\|^\{2\}\-\\\|\\bar\{e\}\\\|^\{2\}\\leq\\frac\{1\}\{N\}\\sum\_\{i\}\\\|e\_\{i\}\\\|^\{2\}; taking expectations, it is at mostδ\\delta\.

*Lower bound\.*Onθ𝒩\\theta\_\{\\mathcal\{N\}\},Π⟂=I\\Pi^\{\\perp\}=I, so\(∇F−∇Fi\)𝒩=\(g¯−gi\)𝒩\+\(e¯−ei\)𝒩\(\\nabla F\-\\nabla F\_\{i\}\)\_\{\\mathcal\{N\}\}=\(\\bar\{g\}\-g\_\{i\}\)\_\{\\mathcal\{N\}\}\+\(\\bar\{e\}\-e\_\{i\}\)\_\{\\mathcal\{N\}\}\. Henceκeff≥‖\{\(∇F−∇Fi\)𝒩\}‖N≥κ𝒩−δ\\kappa\_\{\\mathrm\{eff\}\}\\geq\\\|\\\{\(\\nabla F\-\\nabla F\_\{i\}\)\_\{\\mathcal\{N\}\}\\\}\\\|\_\{N\}\\geq\\kappa\_\{\\mathcal\{N\}\}\-\\delta\. ∎

Proposition[1](https://arxiv.org/html/2610.12885#Thmproposition1)holds for any such projector\. Its content lies in choosingΠ\\Piso thatδ\\deltais small\. The next result identifies what controlsδ\\delta\.

###### Corollary 1\(Sources ofδ\\delta\)\.

Supposeℒi\\mathcal\{L\}^\{i\}has aρ\\rho\-Lipschitz Hessian, and letHi:=∇2ℒθiH\_\{i\}:=\\nabla^\{2\}\\mathcal\{L\}^\{i\}\_\{\\theta\}\. LetΠi\\Pi\_\{i\}be built fromϕi∗\\phi\_\{i\}^\{\*\}\. Then

‖ei‖≤‖Πi⟂​Hi​Πi‖op​‖Γi∗‖⏟cross\-curvature\+ρ2​‖Γi∗‖2⏟higher order\+‖\(Πi−Π\)​gi‖⏟subspace misalignment\.\\\|e\_\{i\}\\\|\\ \\leq\\ \\underbrace\{\\\|\\Pi\_\{i\}^\{\\perp\}H\_\{i\}\\Pi\_\{i\}\\\|\_\{\\mathrm\{op\}\}\\,\\\|\\Gamma\_\{i\}^\{\*\}\\\|\}\_\{\\text\{cross\-curvature\}\}\+\\underbrace\{\\tfrac\{\\rho\}\{2\}\\\|\\Gamma\_\{i\}^\{\*\}\\\|^\{2\}\}\_\{\\text\{higher order\}\}\+\\underbrace\{\\\|\(\\Pi\_\{i\}\-\\Pi\)\\,g\_\{i\}\\\|\}\_\{\\text\{subspace misalignment\}\}\.\(9\)

###### Proof\.

By Lemma[1](https://arxiv.org/html/2610.12885#Thmlemma1),∇Fi=Πi⟂∇Fi\\nabla F\_\{i\}=\\Pi\_\{i\}^\{\\perp\}\\nabla F\_\{i\}, so

ei=Πi⟂​\(∇Fi−gi\)\+\(Πi⟂−Π⟂\)​gi\.e\_\{i\}=\\Pi\_\{i\}^\{\\perp\}\(\\nabla F\_\{i\}\-g\_\{i\}\)\+\(\\Pi\_\{i\}^\{\\perp\}\-\\Pi^\{\\perp\}\)g\_\{i\}\.By \([3](https://arxiv.org/html/2610.12885#Sx12.E3)\) and Taylor’s theorem,∇Fi=∇ℒθ\+Γi∗i=gi\+Hi​Γi∗\+qi\\nabla F\_\{i\}=\\nabla\\mathcal\{L\}^\{i\}\_\{\\theta\+\\Gamma\_\{i\}^\{\*\}\}=g\_\{i\}\+H\_\{i\}\\Gamma\_\{i\}^\{\*\}\+q\_\{i\}with‖qi‖≤ρ2​‖Γi∗‖2\\\|q\_\{i\}\\\|\\leq\\frac\{\\rho\}\{2\}\\\|\\Gamma\_\{i\}^\{\*\}\\\|^\{2\}\. SinceΓi∗∈range⁡\(Πi\)\\Gamma\_\{i\}^\{\*\}\\in\\mathrm\{range\}\(\\Pi\_\{i\}\), we haveΠi⟂​Hi​Γi∗=Πi⟂​Hi​Πi​Γi∗\\Pi\_\{i\}^\{\\perp\}H\_\{i\}\\Gamma\_\{i\}^\{\*\}=\\Pi\_\{i\}^\{\\perp\}H\_\{i\}\\Pi\_\{i\}\\Gamma\_\{i\}^\{\*\}\. Finally,Πi⟂−Π⟂=Π−Πi\\Pi\_\{i\}^\{\\perp\}\-\\Pi^\{\\perp\}=\\Pi\-\\Pi\_\{i\}\. ∎

#### Interpretation\.

- •Shared subspaces\.If the clients’ adapters share column and row spaces, so thatΠi=Π\\Pi\_\{i\}=\\Pifor allii, the misalignment term vanishes\. If, in addition, the Hessian couples𝒯\\mathcal\{T\}and𝒯⟂\\mathcal\{T\}^\{\\perp\}only weakly, \([8](https://arxiv.org/html/2610.12885#Sx12.E8)\) gives κeff2≤κ2−κ∥2\+𝒪⁡\(κ​δ\+δ2\),\\kappa^\{2\}\_\{\\mathrm\{eff\}\}\\leq\\kappa^\{2\}\-\\kappa^\{2\}\_\{\\parallel\}\+\\mathcal\{O\}\(\\kappa\\,\\delta\+\\delta^\{2\}\),whereδ\\deltais second order in the adapter shift\. This is the precise sense in which “κeff2≤κ2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}\\leq\\kappa^\{2\}to first order\.”
- •Misaligned subspaces\.Independently trained adapters do not share subspaces in general\. ThenΠ\\Pican be taken as the projector onto a common subspace \(e\.g\.,⋂i𝒯i\\bigcap\_\{i\}\\mathcal\{T\}\_\{i\}or a principal common subspace\), and the misalignment term is paid explicitly\. Outside these conditions,κeff2≤κ2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}\\leq\\kappa^\{2\}is not guaranteed and must be checked empirically\.

### Role of the LoRA Rankrr

#### A ceiling set by which modules are adapted\.

Becauserange⁡\(Π\)⊆θ𝒜\\mathrm\{range\}\(\\Pi\)\\subseteq\\theta\_\{\\mathcal\{A\}\}, we haveκ∥2≤κ𝒜2\\kappa^\{2\}\_\{\\parallel\}\\leq\\kappa^\{2\}\_\{\\mathcal\{A\}\}, and \([8](https://arxiv.org/html/2610.12885#Sx12.E8)\) gives the floorκeff≥κ𝒩−δ\\kappa\_\{\\mathrm\{eff\}\}\\geq\\kappa\_\{\\mathcal\{N\}\}\-\\delta\. Adapting onlyc\_attncan never remove heterogeneity carried by the MLP, embedding, or LayerNorm parameters, regardless ofrr\.

#### Growth withrr\.

Increasingrrenlarges𝒯i,ℓ\\mathcal\{T\}\_\{i,\\ell\}, whose dimension isr⁡\(m\+n\)−r2r\(m\+n\)\-r^\{2\}\. For nested adapter subspaces,κ∥2\\kappa^\{2\}\_\{\\parallel\}is therefore nondecreasing inrr\.

#### Saturation\.

LetZi,ℓZ\_\{i,\\ell\}be blockℓ\\ellofg¯−gi\\bar\{g\}\-g\_\{i\}\. A block lies in𝒯ℓ\\mathcal\{T\}\_\{\\ell\}if and only if\(I−Pℓ\)​Z​\(I−Qℓ\)=0\(I\-P\_\{\\ell\}\)Z\(I\-Q\_\{\\ell\}\)=0\. A sufficient condition is that the column space ofBℓB\_\{\\ell\}contains the column space ofZZ\. Henceκ∥2=κ𝒜2\\kappa^\{2\}\_\{\\parallel\}=\\kappa^\{2\}\_\{\\mathcal\{A\}\}as soon as the column space ofBℓB\_\{\\ell\}contains∑icol⁡\(Zi,ℓ\)\\sum\_\{i\}\\mathrm\{col\}\(Z\_\{i,\\ell\}\)for everyℓ\\ell, which is possible whenr≥dim∑icol⁡\(Zi,ℓ\)r\\geq\\dim\\sum\_\{i\}\\mathrm\{col\}\(Z\_\{i,\\ell\}\)\. The rank required is thus governed by the \(numerical\) ranks of theNNmean\-zero heterogeneity blocks, not bydd\. Beyond this point, increasingrrremoves no further heterogeneity\.

#### The cost side\.

We hypothesize that larger adapters are harder to bring near stationarity inEElocal steps\. This would raiseεϕ\\varepsilon\_\{\\phi\}, and hence the biasC​εϕC\\varepsilon\_\{\\phi\}in Theorem 1, suggesting a non\-monotone trade\-off inrr\. The SHP rank ablation is consistent with this\. On hh\-rlhf, the gradient norm decreases monotonically up tor=16r=16, which suggests that saturation is not reached in that range\.

### DPO\-Specific Structure

Consider a linear layer with effective weightW~\\widetilde\{W\}\. Letxtx\_\{t\}be its input at tokenttandutu\_\{t\}the back\-propagated signal at its output\. Then∂log⁡π⁡\(τ\)/∂W~=∑t∈τut​xt⊤\\partial\\log\\pi\(\\tau\)/\\partial\\widetilde\{W\}=\\sum\_\{t\\in\\tau\}u\_\{t\}x\_\{t\}^\{\\top\}, up to the transpose convention of the implementation\. With the implicit\-reward marginhθ​\(τ\+,τ−\):=Δθ​\(τ\+,τ−\)−Δref​\(τ\+,τ−\)h\_\{\\theta\}\(\\tau^\{\+\},\\tau^\{\-\}\):=\\Delta\_\{\\theta\}\(\\tau^\{\+\},\\tau^\{\-\}\)\-\\Delta\_\{\\mathrm\{ref\}\}\(\\tau^\{\+\},\\tau^\{\-\}\), the per\-client DPO gradient block is

∇W~ℒi=−β𝔼\(τ\+,τ−\)∼𝒟i\[σ\(−βhθ\)\(∑t∈τ\+utxt⊤−∑t∈τ−utxt⊤\)\]\.\\begin\{split\}&\\nabla\_\{\\widetilde\{W\}\}\\mathcal\{L\}^\{i\}=\-\\beta\\,\\mathbb\{E\}\_\{\(\\tau^\{\+\},\\tau^\{\-\}\)\\sim\\mathcal\{D\}\_\{i\}\}\\Big\[\\sigma\\big\(\-\\beta h\_\{\\theta\}\\big\)\\\\ &\\Big\(\\textstyle\\sum\_\{t\\in\\tau^\{\+\}\}u\_\{t\}x\_\{t\}^\{\\top\}\-\\sum\_\{t\\in\\tau^\{\-\}\}u\_\{t\}x\_\{t\}^\{\\top\}\\Big\)\\Big\]\.\\end\{split\}\(10\)This structure has two consequences for the quantities above\.

1. 1\.Numerical rank of the heterogeneity blocks\.EachZi,ℓZ\_\{i,\\ell\}is a weighted sum of token\-level outer products over a small local dataset \(\|𝒟i\|=120\|\\mathcal\{D\}\_\{i\}\|=120pairs, 64 tokens\)\. When the sigmoid weights concentrate on a few pairs,Zi,ℓZ\_\{i,\\ell\}is numerically low rank, and a smallrrsuffices for the saturation condition above\. This is a hypothesis, and it can be checked directly from the singular values ofZi,ℓZ\_\{i,\\ell\}\.
2. 2\.Source of heterogeneity and the role ofβ\\beta\.The weightσ⁡\(−β​hθ\)\\sigma\(\-\\beta h\_\{\\theta\}\)emphasizes pairs with small or negative margin\. Clients differ in which pairs these are, and this is where their gradients diverge\. Asβ\\betagrows,σ⁡\(−β​hθ\)\\sigma\(\-\\beta h\_\{\\theta\}\)approaches an indicator of misordered pairs \(hθ<0h\_\{\\theta\}<0\), concentrating the weights further\. We conjecture that this lowers the numerical rank ofZi,ℓZ\_\{i,\\ell\}and increases the absorbed fractionκ∥2/κ2\\kappa^\{2\}\_\{\\parallel\}/\\kappa^\{2\}\. All our experiments useβ=0\.2\\beta=0\.2, so this is untested\.

### When Is the Reduction Largest?

By Proposition[1](https://arxiv.org/html/2610.12885#Thmproposition1)and Corollary[1](https://arxiv.org/html/2610.12885#Thmcorollary1), the gapκ2−κeff2\\kappa^\{2\}\-\\kappa^\{2\}\_\{\\mathrm\{eff\}\}is large when:

1. 1\.Heterogeneity sits in adapted modules, i\.e\.,κ𝒜2≈κ2\\kappa^\{2\}\_\{\\mathcal\{A\}\}\\approx\\kappa^\{2\}\. Ifκ𝒩2\\kappa^\{2\}\_\{\\mathcal\{N\}\}dominates, adapting more modules \(e\.g\., MLP projections\) helps more than raisingrr\.
2. 2\.The adapted\-block heterogeneity lies in the adapters’ tangent spaces, i\.e\.,κ∥2≈κ𝒜2\\kappa^\{2\}\_\{\\parallel\}\\approx\\kappa^\{2\}\_\{\\mathcal\{A\}\}\. This requiresrrto be large relative to the numerical rank of theZi,ℓZ\_\{i,\\ell\}\.
3. 3\.δ\\deltais small: weak cross\-curvature, small adapter shifts‖Γi∗‖\\\|\\Gamma\_\{i\}^\{\*\}\\\|, and aligned adapter subspaces across clients\.

The gap is small when preferences are nearly homogeneous \(κ2≈0\\kappa^\{2\}\\approx 0\), when heterogeneity resides mainly in non\-adapted parameters, or when adapter subspaces are strongly misaligned\.

These conditions concernκeff2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}itself\. For the algorithm, it additionally matters that adapters are near stationarity \(smallεϕ\\varepsilon\_\{\\phi\}\)\. Otherwise the base gradients actually transmitted differ from∇Fi\\nabla F\_\{i\}by the biasC​εϕC\\varepsilon\_\{\\phi\}of Theorem 1, and Lemma[1](https://arxiv.org/html/2610.12885#Thmlemma1)holds only approximately \(Remark[1](https://arxiv.org/html/2610.12885#Thmremark1)\)\.

### Intuition

By \([3](https://arxiv.org/html/2610.12885#Sx12.E3)\), each client’s adapter is a private, low\-rank shift of the shared weights\. In vanilla FedDPO, every client pushes the shared base in its own direction, and these directions partially cancel under FedAvg, leaving the diversityκ2\\kappa^\{2\}\.

In pFedDPO, each client first moves its private shift until the loss can no longer be reduced within the adapter’s tangent space \(Lemma[1](https://arxiv.org/html/2610.12885#Thmlemma1)\)\. Whatever part of the client’s disagreement with the population can be expressed there is resolved locally and never reaches the server\. What the client uploads is the remainder: the part of its gradient orthogonal to that space, plus non\-adapted parameters\.

The adapter thus acts as a client\-specific pre\-filter on the base gradient\. The filter removes heterogeneity only where it can express it: inside the adapted modules \(κ𝒜2\\kappa^\{2\}\_\{\\mathcal\{A\}\}\) and inside its rank budget \(κ∥2\\kappa^\{2\}\_\{\\parallel\}\)\. It removes it cleanly only when clients’ filters are roughly aligned and the curvature does not leak the shift back into the orthogonal complement \(smallδ\\delta\)\.

### Measuringκ2\\kappa^\{2\}andκeff2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}on the Same Trajectory

The empirical gradient diversity in Figures 5\(b\) and 8\(b\) is computed from pFedDPO’s base gradients with the current adapters, i\.e\.,κ^eff2:=1N​∑i‖1N​∑j∇θℒθ,ϕjj−∇θℒθ,ϕii‖2\\widehat\{\\kappa\}^\{2\}\_\{\\mathrm\{eff\}\}:=\\frac\{1\}\{N\}\\sum\_\{i\}\\\|\\frac\{1\}\{N\}\\sum\_\{j\}\\nabla\_\{\\theta\}\\mathcal\{L\}^\{j\}\_\{\\theta,\\phi\_\{j\}\}\-\\nabla\_\{\\theta\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}\\\|^\{2\}\. This estimatesκeff2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}up to the adapter bias of Assumption 1, but on its own it cannot separate the adapter effect from ordinary training progress\.

Proposition[1](https://arxiv.org/html/2610.12885#Thmproposition1)can be tested directly by computing, at each logging round and at the*same*baseθr\\theta^\{r\}:

1. \(i\)per\-client base gradients with adapters disabled, givingκ2\\kappa^\{2\};
2. \(ii\)the same gradients with adapters enabled, givingκ^eff2\\widehat\{\\kappa\}^\{2\}\_\{\\mathrm\{eff\}\};
3. \(iii\)the block split of \(i\), givingκ𝒜2\\kappa^\{2\}\_\{\\mathcal\{A\}\}andκ𝒩2\\kappa^\{2\}\_\{\\mathcal\{N\}\};
4. \(iv\)1N​∑i‖∇ϕℒθ,ϕii‖2\\frac\{1\}\{N\}\\sum\_\{i\}\\\|\\nabla\_\{\\phi\}\\mathcal\{L\}^\{i\}\_\{\\theta,\\phi\_\{i\}\}\\\|^\{2\}, an estimate ofεϕ\\varepsilon\_\{\\phi\}\.

The ratioκ^eff2/κ2\\widehat\{\\kappa\}^\{2\}\_\{\\mathrm\{eff\}\}/\\kappa^\{2\}then isolates the reduction due to the adapters, andκ𝒩2\\kappa^\{2\}\_\{\\mathcal\{N\}\}gives the floor predicted by \([8](https://arxiv.org/html/2610.12885#Sx12.E8)\)\.

## Additional Results

#### Datasets\.

Stanford Human Preferences \(SHP\)\([Ethayarajh et al\. 2021](https://arxiv.org/html/2610.12885#bib.bib32)\)contains about 385K pairs of human\-written Reddit comments answering the same post, drawn from 18 subject\-area subreddits; commentAAis labeled preferred overBBif it received a higher score despite being posted later, so labels reflect the collective votes of Reddit users rather than designated annotators\. Anthropic hh\-rlhf\([Bai et al\. 2022](https://arxiv.org/html/2610.12885#bib.bib20)\)contains multi\-turn human–assistant dialogues in which a crowdworker chose the more helpful \(helpfulness subsets\) or less harmful \(harmlessness subset\) of two model responses; the chosen and rejected dialogues share the conversation history and differ in the final assistant turn\. For each dataset we formN=5N=5agents with 120 disjoint pairs each\. Sequences are truncated to 64 tokens, and reward accuracy is computed on training pairs of each agent\.

SHP Table\.Table[3](https://arxiv.org/html/2610.12885#Sx13.T3)summarizes round\-80 results on SHP\. Centralized DPO has the highest gradient norm and lowest accuracy, reflecting its smaller per\-round gradient work in this \(non\-compute\-matched\) regime\. Vanilla FedDPO attains the lowest training loss \(0\.05\) and highest reward accuracy \(0\.82\)\. pFedDPO attains the lowest gradient norm \(0\.02\), with training loss \(0\.06\) and accuracy \(0\.80\) close to vanilla FedDPO\. Its measured base\-gradient diversity falls from∼\\sim1500 to∼\\sim150–200\. As discussed in the above on measuringκ2\\kappa^\{2\}andκeff2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}, this trajectory alone does not separate the adapter effect from training progress\.

Table 3:Comparison of DPO variants\. Results at round 80 on SHP\. Centralized DPO performs worst in this regime: with batch size 4 it receives far less gradient work per round than distributed methods \(effective batch5×4=205\\times 4=20for FedDPO\), consistent with the computation\-dominated analysis in\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\)\.κ2\\kappa^\{2\}is measured on base model gradients only\.HH\-RLHF\.This appendix reports the full experimental results on Anthropic hh\-rlhf using the identical setup as the SHP main results \(N=5N=5agents, 120 pairs each, DistilGPT\-2, AdamWη=0\.0001\\eta=0\.0001,E=3E=3,R=80R=80rounds\)\. Figures[7](https://arxiv.org/html/2610.12885#Sx13.F7)\-[11](https://arxiv.org/html/2610.12885#Sx13.F11)correspond directly to Figures[2](https://arxiv.org/html/2610.12885#Sx4.F2)\-[6](https://arxiv.org/html/2610.12885#Sx4.F6)in the main text\. The qualitative findings are broadly consistent with SHP, with several notable quantitative differences that we highlight below\.

Comparative Study\.Figure[7](https://arxiv.org/html/2610.12885#Sx13.F7)compares pFedDPO, vanilla FedDPO, and centralized DPO on hh\-rlhf over 80 rounds\.

Gradient norm \(panel a\)\.All three methods begin near 1\. pFedDPO descends most rapidly and settles at the lowest floor \(∼\\sim0\.2 by round 80\)\. Vanilla FedDPO \(blue\) is slightly higher but follows a similarly smooth descent\. Centralized DPO \(dashed grey\) remains the worst throughout, fluctuating around0\.6−1\.00\.6\-1\.0and never converging stably\. This replicates the computation\-dominated regime observed on SHP: centralized DPO processes batch size 4 per step while each FedDPO round aggregates acrossS=5S=5agents \(effective batch 20\), giving distributed methods substantially more gradient work per round\.

Training loss \(panel b\)\.pFedDPO and vanilla FedDPO both decrease from∼\\sim0\.85\-0\.90 to∼\\sim0\.10\-0\.15 by round 80, with pFedDPO reaching a marginally lower final value\. Centralized DPO oscillates between 0\.3 and 1\.4 throughout, never achieving a stable low loss\. The instability of centralized DPO is more pronounced on hh\-rlhf than on SHP, consistent with its small per\-step batch \(4 pairs\)\.

Preference accuracy \(panel c\)\.All three methods achieve notably higher accuracy on hh\-rlhf than on SHP\. pFedDPO reaches∼\\sim0\.95 by round 80, vanilla FedDPO reaches∼\\sim0\.92, and centralized DPO reaches∼\\sim0\.80\. Absolute accuracies are substantially higher on hh\-rlhf than on SHP \(all methods≥0\.80\\geq 0\.80vs\.≤0\.82\\leq 0\.82\)\. We do not attribute this to label quality, since hh\-rlhf labels are single crowdworker judgments and SHP labels are inferred from Reddit votes\. A more plausible factor is pair structure: hh\-rlhf pairs share the dialogue history and differ only in the final assistant turn, and harmlessness pairs often contain lexically distinctive rejected responses, whereas SHP compares two independently written human comments whose quality differences are often subtle\.

Effect of LoRA Personalization on Client Drift\.In Figure[8](https://arxiv.org/html/2610.12885#Sx13.F8)\(a\), pFedDPO \(green\) maintains a consistently lower gradient norm than vanilla FedDPO \(blue\) throughout all 80 rounds on hh\-rlhf, replicating the main SHP finding\. The gap is visible from the first few rounds and persists to round 80, with pFedDPO settling near 0\.2 versus vanilla FedDPO near 0\.4\-0\.6\. The magnitude of the improvement is slightly smaller than on SHP, consistent with hh\-rlhf’s lower initialκ2\\kappa^\{2\}\.

In Figure[8](https://arxiv.org/html/2610.12885#Sx13.F8)\(b\), the empirical gradient diversityκ2\\kappa^\{2\}on hh\-rlhf starts at∼\\sim900 at round 0 \(compared to∼\\sim1500 on SHP\), indicating lower inter\-client heterogeneity under our hh\-rlhf partition than under our SHP partition\. The trajectory is substantially more volatile than on SHP:κ2\\kappa^\{2\}does not drop cleanly but instead oscillates across the range 400–900 throughout training, with a spike near round 30 and fluctuations of±\\pm200–300 around subsequent measurements at rounds 20, 30, 40, 50, 60, and 70\. Despite the oscillations, the overall trend is downward, ending near 400–500 by round 70, which is a reduction of approximately 45\-55% from the initial value\. This is notably smaller than the 85–90% reduction observed on SHP, and reflects two factors: \(i\) lower initialκ2\\kappa^\{2\}leaves less room for absolute reduction; \(ii\) the adapters may be further from stationarity afterE=3E=3local steps, increasingC​εϕC\\varepsilon\_\{\\phi\}, and measuring‖∇ϕℒi‖2\\\|\\nabla\_\{\\phi\}\\mathcal\{L\}^\{i\}\\\|^\{2\}would test this\. Rawκ2\\kappa^\{2\}values are not directly comparable across datasets because gradient scales differ\.κ2/‖g¯‖2\\kappa^\{2\}/\\\|\\bar\{g\}\\\|^\{2\}would be the comparable quantity\.

the higher gradient variance of hh\-rlhf’s pairs makes the adapter stationarity condition \(Assumption[1](https://arxiv.org/html/2610.12885#Thmassumption1)\) harder to satisfy withinE=3E=3local steps, increasingC​εϵC\\varepsilon\_\{\\epsilon\}and moderating theκeff2\\kappa^\{2\}\_\{\\text\{eff\}\}reduction\.

Per\-Client Reward Accuracy\.Figure[9](https://arxiv.org/html/2610.12885#Sx13.F9)shows per\-client accuracy for all five agents under pFedDPO on hh\-rlhf\. All agents improve steadily from∼\\sim0\.5\-0\.82 at round 0 to∼\\sim0\.88\-0\.97 by round 80, with the average \(dashed\) reaching approximately 0\.90\-0\.92\. Agent 1 \(red\) is the strongest performer, reaching∼\\sim0\.95\-0\.97 by round 80, while Agent 4 \(orange\) is the slowest, plateauing around 0\.86–0\.88\. The remaining agents \(2, 3, 5\) cluster between 0\.90 and 0\.95\. At round 80 the spread between the highest \(Agent 1,∼\\sim0\.97\) and lowest \(Agent 4,∼\\sim0\.88\) agents is about 9–10 percentage points, similar to SHP\. The result also confirms that all agents improve under pFedDPO\.

LoRA Rank Ablation\.Figure[10](https://arxiv.org/html/2610.12885#Sx13.F10)shows the rank ablationr∈\{2,4,8,16\}r\\in\\\{2,4,8,16\\\}on hh\-rlhf\. In panel \(a\), the accuracy curves are nearly indistinguishable; at round 80 all ranks reach∼\\sim0\.90–0\.92, withr=8r=8marginally highest\. Panel \(b\) shows that the final gradient norm decreases monotonically with rank, from∼\\sim0\.36 atr=2r=2to∼\\sim0\.19 atr=16r=16\. Unlike SHP, wherer=8r=8attained the lowest gradient norm, the gradient\-norm benefit on hh\-rlhf has not saturated byr=16r=16\(see “Role of the LoRA Rank”\)\. Since accuracy is essentially flat across ranks, we keepr=8r=8as the default\. On both datasets, rank affects gradient norm more than accuracy, and the best rank appears to be dataset\-dependent\.

Participation Ablation\.In Figure[11](https://arxiv.org/html/2610.12885#Sx13.F11), it shows the participation ablationS∈\{1,3,5\}S\\in\\\{1,3,5\\\}for pFedDPO \(left\) and vanilla FedDPO \(right\) on hh\-rlhf\.S=3S=3\(blue\) achieves the lowest gradient norm, descending to∼\\sim0\.25 by round 80\.S=5S=5\(green\) is intermediate \(∼\\sim0\.32\)\.S=1S=1\(red\) is the most volatile, fluctuating around 0\.4\-0\.8 throughout without clear convergence\. TheS=5S=5vs\.S=3S=3gap is moderate; theS=1S=1vs\.S=3S=3gap is larger, consistent with theζg2\\zeta^\{2\}\_\{g\}variance floor growing sharply asSSdecreases below the number of agents\. The gap between participation levels is more compressed for vanilla FedDPO on hh\-rlhf\.S=5S=5\(green\) converges to∼\\sim0\.6, whileS=3S=3\(blue\) converges to∼\\sim0\.7\.S=1S=1\(red\) stays near 1 throughout\. Notably, the difference betweenS=5S=5personalized pFedDPO \(∼\\sim0\.32\) andS=5S=5vanilla FedDPO \(∼\\sim0\.65\) is substantial, which is a factor of∼2×\\sim 2\\timesin gradient norm, and is the clearest evidence on hh\-rlhf that personalization provides a meaningful convergence benefit beyond what increased participation alone achieves\.

Cross\-dataset comparison of participation results: on SHP, the advantage of pFedDPO over vanilla FedDPO atS=5S=5was visible but atS=1S=1both methods struggled similarly\. On hh\-rlhf, pFedDPO maintains a clear advantage over vanilla FedDPO even atS=3S=3\. One possible explanation is that adapters converge more reliably withinE=3E=3local steps on hh\-rlhf, reducingC​εϕC\\varepsilon\_\{\\phi\}, but we have not measuredεϕ\\varepsilon\_\{\\phi\}directly\. With a single seed, the S = 3 vs\. S = 5 difference is within run\-to\-run variation\.

Figure 7:Comparison of baselines with HH\-RLHF data\.Figure 8:Gradient diversityκ2\\kappa^\{2\}ablation with HH\-RLHF data\.Figure 9:Per\-client preference accuracy with HH\-RLHF data\.Figure 10:Low rankrrablation with HH\-RLHF data\.Figure 11:ParticipationSSablation with HH\-RLHF data\.

## Limitations

Several limitations should be acknowledged\. The theoretical analysis inherits the log\-linear softmax policy assumptions of \(Jiang 2026\)\. Extending the bounds to transformer architectures requires controlling Fisher information spectra and is left for future work\. Theorem 1[1](https://arxiv.org/html/2610.12885#Thmtheorem1)further assumes that each local loss isμϕ\\mu\_\{\\phi\}\-PL in the adapter and that the minimizing low\-rank shift is unique, which are strong for bilinear LoRA parameterizations, andκeff2≤κ2\\kappa^\{2\}\_\{\\mathrm\{eff\}\}\\leq\\kappa^\{2\}is established only to first order under a shared\-subspace condition\. The constantCCin the adapter\-bias termC​εϕC\\varepsilon\_\{\\phi\}is not available in closed form in terms ofEE,η\\eta, and local curvature; such an expression would let practitioners chooseEEso that the bound is guaranteed to improve on vanilla FedDPO\. pFedDPO does not reduce communication relative to vanilla FedDPO as both transmitdd\-dimensional base updates, whereas FedPDPO\([Zhu et al\. 2026](https://arxiv.org/html/2610.12885#bib.bib2)\)transmits only shared low\-rank modules, which may be preferable under tight bandwidth\. A variant that transmits low\-rank updates while keeping adapters private is an open direction\. Empirically, we use DistilGPT\-2 \(≈\\approx82M parameters\), 120 randomly partitioned pairs per agent, a single seed for ablations, and report accuracy on training pairs; scaling to 7B\+ models, domain\-based partitions, held\-out evaluation, and multiple seeds are important future work\. The rank ablations show that the best rank is dataset\-dependent\. Finally, pFedDPO assumes synchronous updates, and extending the asynchronous staleness analysis of \(\([Jiang 2026](https://arxiv.org/html/2610.12885#bib.bib21)\), Thm\. 5\.4\) to private adapters warrants dedicated analysis\.

## Ethics Statement

Federated DPO, including pFedDPO, is designed to enable preference\-based alignment of language models in privacy\-preserving settings where raw data cannot be centralized\. We identify the following ethical dimensions\.

Privacy and data minimization\.pFedDPO transmits only base model parameter updates, not raw preference data or LoRA adapters\. This avoids sharing raw data and adapter weights, although full base\-model updates are higher\-dimensional than the low\-rank updates shared by adapter\-sharing methods and may leak more information\. However, model updates can still leak information about local data through gradient inversion attacks\. Combining pFedDPO with differential privacy mechanisms is a natural extension that we do not implement here due to the interaction between DP noise and the gradient diversity bound, which warrants dedicated analysis\.

Preference heterogeneity and value plurality\.A central motivation for personalization is that different users have genuinely different preferences, and a single global policy may suppress minority preferences by averaging toward the majority\. pFedDPO mitigates this by allowing each client’s adapter to capture deviations from the shared base, potentially preserving value diversity\. At the same time, private adapters could also encode harmful or biased preferences that would be moderated by global averaging\. Deployment should include adapter\-level content monitoring or periodic auditing\.

Adversarial clients and Byzantine robustness\.pFedDPO, like FedDPO, is vulnerable to clients that submit malicious base model updates to steer the global model\. Private adapters cannot be inspected by the server, which may enable model poisoning through the base model update while concealing the intent in the local adapter\. Robust aggregation methods should be considered in adversarial settings\.

Scope of application\.Our experiments use two open academic preference datasets \(hh\-rlhf, SHP\) for research validation\. Deployment of pFedDPO in production systems, particularly in high\-stakes domains such as healthcare or legal advice, requires additional safeguards including human oversight of alignment targets, regular evaluation against safety benchmarks, and mechanisms for clients to audit or contest the preference optimization process\.